Hadoop Developer Resume
Charleston, SC
PROFESSIONAL SUMMARY:
- An extensive experience of 9 years in all phases of Software Development Life Cycle on Java/J2EE, Big Data, and cloud - based applications spanning across technologies, and business domains.
- 4+ years of experience in Big Data Technologies with full project development, implementation, and deployment in agile - scrum, and waterfall models
- Hands on experience in different phases of big data applications like data ingestion, data analytics, and data visualization.
- Hands on experience in installing, configuring, and using ecosystem components like Hadoop MRV1/MRV2, HDFS, HBase, ZooKeeper, Hive, Oozie, Sqoop, Pig, Flume, Spark.
- Good Understanding of Hadoop Gen1/Gen2 architecture, and Hands-on experience with Hadoop components such as Job Tracker, Task Tracker, Name Node, Data Node, Map Reduce concepts, and YARN architecture.
- Experienced with Cloudera, andHortonworks open source distribution.
- Hands on experience in big data ingestion tools like Flume, Kafka, and Sqoop.
- Experienced in transferring Streaming data, data from different data sources into HDFS, No SQL databases using Apache Flume, andKafka.
- Well versed with Developing, and Implementing Map Reduce programs using Hadoop to work with Semi, and Unstructured data.
- Experienced in implementing different analytical algorithms using map reduce programs with the help of map reduce design patterns.
- Good knowledge analyzing data using Python development, and scripting for Hadoop Streaming.
- Experience in getting data from various sources into HDFS, and building reports using matplotlib python Library
- Experience in using different file formats like Sequence, AVRO, ORC files, and CSV using different compression Techniques like Snappy, zLib, gZip etc
- Expert in implementing advanced procedures like text analytics, and processing using the in-memory computing capabilities like Apache Spark written in Python.
- Good knowledge of NoSQL databases like MongoDB, andHBase.
- Working knowledge of Web/Application Servers like Apache Tomcat, andIBM Web Sphere.
- Good understanding, and experience with Software Development methodologies like Agile, and Waterfall.
- Articulate in written, and verbal communication along with strong interpersonal, analytical, and organizational skills. Highly motivated team player with the ability to work independently, and adapt quickly to new, and emerging technologies.
TECHNICAL SKILLS:
Hadoop/Big Data: HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Oozie, Apache Spark, Flume, Kafka, Scala
IDE Tools: Eclipse, NetBeans
Programming languages: Java/J2EE, Python, Linux shell scripts, C++
Databases: Oracle 11g/10g/9i, MySQL, DB2
Web Technologies: HTML, Java Script, XML, ODBC, JDBC, JSP, Servlets, Struts, JUnit, REST API, Spring
Visualization: MS Excel, RAW, Tableau
PROFESSIONAL EXPERIENCE:
Confidential, Charleston, SC
Hadoop Developer
Responsibilities:
- Imported all the Customer specific personal data to Hadoop using SQOOP component of Hadoop.
- Created tasks for incremental load into staging tables, and schedule them to run.
- Used Apache Kafka as a messaging system to load log data, data from UI applications into HDFS system.
- Designed, and developed solution for ESP pump real time-data ingestion using Kafka, Storm, and HBase. This involves Kafka, and storm cluster design, installation, and configuration.
- Designed, and configured Kafka cluster to accommodate a heavy throughput of 1 million messages per second.
- Used Kafka producer 0.8.3 API's to produce messages.
- Hands on knowledge of writing code in Scala.
- Expert in implementing advanced procedures like text analytics, and processing using the in-memory computing capabilities like Apache Spark written in Scala.
- Designed & assisted Informatica ETL developer to create new Aggregate.
- Created Informatica mappings for initial load, and daily updates.
- Used Scala to extract information from XML files.
- Wrote Scala routines to log into the websites, and fetch data for selected options.
- Worked with Cassandra, and utilized NoSQL for non-relation data storage, and retrieval.
- Installed, and configured Cassandra In-depth knowledge about Cassandra architecture, query, read, and write path.
- Written Map Reduce code that will take input as customer related flat file, and parse the same data to extract the meaningful (domain specific) information for further processing.
- Created Hive External tables with partitioning to store the processed data from Map Reduce.
- Implemented Different analytical algorithms using map reduce programs to apply on top of HDFS data.
- Implemented Hive optimized joins to gather data from different sources, and run ad-hoc queries on top of them.
- Wrote Hive Generic UDF's to perform business logic operations at a record level.
- Implemented Data classification algorithms using map reduce design patterns.
- Extensively worked on creating combiners, Partitioning, Distributed cache to improve the performance of Map-Reduce jobs.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS, and pre-processing with Pig.
- Involved in loading data from LINUX file system to HDFS.
- Used Pig to do data transformations, event joins, filter, and some pre-aggregations before storing the data onto HDFS.
- Experienced in implementing test cases, and writing test classes using MRUnit, and mocking frameworks.
- Implemented test scripts to support test-driven development, and continuous integration.
- Analyzed large amounts of data sets to determine the optimal way to aggregate, and report on it.
Environment: Cloudera, Hadoop, MapReduce, HDFS, Hive, Pig, Oozie, Sqoop, Apache Kafka, Linux, JUnit, MRUnit.
Confidential, Houston, TX
Hadoop Developer
Responsibilities:
- Closely worked with business analysts, and senior-level Architects to design, and configure Master/slave architecture in testing, development, and production using configuration tool puppet, andHortonworks distribution.
- Data ingestion from large data sets from DB2, andRDBMS into Hadoop edge node using Map Reduce, andSqoop (vice-versa), andFTP shell Scripts.
- Developed Map Reduce Programs in Java for filtering out the unstructured data, and also written custom input, and output format classes onto HDFS storage layer.
- Extensively involved in loading, filtering, transforming, and combined data using customs UDFs, and Generic UDF's, Pig loader, and Storage classes from Piggybank by using Pig.
- Involved in resolving performance issues in Pig, andHive with the understanding of Map Reduce physical plan execution, and using debugging commands to run code in the optimized way.
- Involved in Unit testing by analyzing Informatica mappings that compare with Pig Scripts that performs the same set of transformed data as per business user requirements.
- Developed scripts using MAVEN build tools in Jenkins to move source code from one environment to another environment, and responsible for managing code in SVN distributed version control, and the access control strategies.
- Involved in regular commissioning, and decommissioning of nodes in-order to balance Hadoop cluster, and also archiving of name node.
- Involved in creating user interactive sheets, and reports as per business requirements, and written SQL Scripts to load the data in QlikView application.
Environment: Hortonworks, HDFS, Core Java, MapReduce, Hive, Informatica Power Centre, HQL, Pig, Flume, Perl scripting, Python, UNIX, Oozie, Shell Scripting, Maven, Teradata, Qlikview.
Confidential, Norman, OK
Hadoop Developer
Responsibilities:
- Setting the cluster, configuration, and maintenance, install components of the Hadoop ecosystem.
- Exported the analyzed data to the relational databases using Sqoop for visualization, and to generate reports for the BI team.
- Stored data from HDFS to respective Hive tables for business analysts to conduct further analysis in identifying data trends.
- Developed Hivead-hoc queries, and filtered data in order to increase the effectiveness of the process execution by using functions like Joins, Group By,, andHaving.
- Increased the time efficiency of the Hive QL using partitioning of data, and reduced the time difference of executing the sets of data by applying the compression techniques like SNAPPY for Map-Reduce Jobs.
- Created Hive Partitions for storing data for different trends under different partitions.
- Connected the Hive tables to data analysis tools like Tableau for graphical representation of the trends.
- Assisted project manager in problem shooting relevant to Hadoop technologies for data integration between different platforms like Sqoop-Sqoop, Hive-Sqoop,, and Sqoop-Hive.
Environment: Java 7, HBase, HDFS, MapReduce, Hadoop 2.0, Hive, Pig, Eclipse, Linux, Sqoop, MySQL, Agile, Kafka, Cognos.
Confidential, Durham, NC
Hadoop Developer
Responsibilities:
- Implemented on Hadoop scaling from 6 nodes in POC environment to 10 nodes in the development and ended up with 40 nodes of clusters in a pilot environment (prod).
- Included in complete Implementation lifecycle, spent significant time in composing customized MapReduce, Pig and Hive programs.
- Solid involvement with Big data processing using Hadoop technologies HDFS, MapReduce, Crunch, Hive, and Pig.
- Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
- Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map-reduce way. Broadly used Hive/HQL or Hive queries to query or search for a particular string in Hive tables in HDFS.
- Did various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in hive and Map Side joins.
- Expert in developing customized user define functions (UDF's) in java to extend Hive and Pig Latin functionality.
- Developed Pig program for loading and filtering the streaming data into HDFS using Flume.
- Applied Pig to do transformations, event joins, filter boot traffic and some pre-aggregations before storing the data onto HDFS.
- Experience on tuning the performance of Pig queries.
- Involved in developing Pig Scripts for change data capture (CDC) and delta record processing between newly arrived data and already existing data in HDFS.
- Scheduling Jobs Managing to remove the duplicate log data files in HDFS using Oozie.
- Used Flume extensively in gathering and moving log data files from Application Servers to a central location in Hadoop Distributed File System (HDFS).
- Created Mappings using Talend Open Studio for Evaluation and POC.
- Designed Risk Audit process for a Healthcare client and created Risk assessment database on hive for performing Risk Assessment and Audits and used Tableau for visualization.
Environment: Hadoop 2.x, HDFS, Map Reduce, Flume, Hive 0.10, Pig 0.11, Sqoop, HBase, YARN, Shell Scripting, Maven, Git Hub, Ganglia, Apache Solr, AWS, Talend Open studio for Big data, Java and Cloudera.
Confidential, Philadelphia, PA
Java/Hadoop Developer
Responsibilities:
- Design and creation of GUI screens using JSP, Servlets, and HTML based on Struts MVC Framework.
- Operated JDBC to access Database.
- Manipulated JavaScript for client side validation.
- Validations were performed using Struts Validation Framework.
- Commit and Rollback methods were provided for transactions processing.
- Designed and developed the action from beans and action classes and implemented MVC using Struts framework.
- Written Oracle SQL Stored procedures, functions and triggers.
- Developed both Session and Entity beans representing different types of business logic abstractions.
- Maintained the server log document.
- Performed Unit /Integration testing for the test cases.
- Implemented and designed user interface for web based customer application.
- Understanding business needs, analyzing functional specifications and map those to develop and designing MapReduce programs and algorithms.
- Written Pig and Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data. Also, have hands on Experience on Pig and Hive User Define Functions (UDF).
- Execution of Hadoop ecosystem and Applications through Apache HUE.
- Optimizing Hadoop MapReduce code, Hive/Pig scripts for better scalability, reliability, and performance.
Environment: Java, JSP, HTML, CSS, Java Script, JQuery, Struts 2.0, MySQL, Oracle, Hibernate, JDBC, Eclipse, SQL Stored Procedures, Tomcat, Hive, Pig, Sqoop, Flume and Cloudera.
Confidential, Bothell, WA
Associate Java Developer
Responsibilities:
- Developed web pages using Struts framework, JSP, XML, JavaScript, Html/ DHTML and CSS, configure struts application, use tag library.
- Embedded a custom-built Java application in Sales Cloud using JSON Web Token (JWT) as the security mechanism.
- Developed Application using Spring and Hibernate, Spring batch, Web Services like Soap and restful Web services.
- Used Spring Framework at Business Tier and also Spring's Bean Factory for initializing services.
- Used AJAX, JavaScript to create interactive user interface.
- Implemented client side validations using JavaScript & server side validations.
- Developed Single Page application using angular JS & backbone JS.
- Developed app using Front Controller, Business delegate, DAO and Session Facade Patterns.
- Implemented Hibernate to persist the data into Database and wrote HQL based queries to implement CRUD operations on the data.
- Used Hibernate annotations and created Hibernate POJOs.
- Developed Web Services to communicate to other modules using XML-based SOAP and WSDL.
- Designed and implemented (SOA, SOAP) next generation system on distributed platform.
- Designed and developed most of the application's GUI screens using GWT framework.
Environment: Java/J2EE, JSP, Servlets, EJB, XML, XSLT, Struts, Rational Rose, Apache Struts Framework, Web Services, DB2, Beyond Compare, Web Services, CVS, JUnit, Log4j, Windows XP, Red Hat LINUX.
Confidential
Associate Java Developer
Responsibilities:
- Responsible for the Requirement Analysis and Design of Smart Systems Pro (SSP)
- Involved in Object Oriented Design (OOD) and Analysis (OOA).
- Analysis and Design of the Object models using JAVA/J2EE Design Patterns in various tiers of the application.
- Worked with Restful Web Services and WSDL.
- Worked with Maven build tool to build the Project.
- Involved in Coding JavaScript code for UI validation and worked on Struts validation frameworks.
- Analyzing the Client Requirements and designing the specification document based on the requirements.
- Worked on implementing directives and scope values using AngularJs for an existing webpage.
- Familiar with the state-of-the-art standards, processes, design processes used in creating and designing optimal UI using Web 2.0 technologies like Ajax, JavaScript, CSS, and XSLT.
- Involved in the Preparation of Program Specification and Unit Test Case Document.
- Designed the Proto according to the Business requirements.
- Developed the web tier using JSP, Struts MVC to show account details and summary.
- Used Struts Tiles Framework in the presentation tier.
- Designed and developed the UI using Struts view component, JSP, HTML, CSS and JavaScript.
- Used AJAX for asynchronous communication with server
- Utilized Hibernate for Object/Relational Mapping purposes for transparent persistence onto the SQL Server database.
Environment: Java, J2EE Servlet, JSP, JUnit, AJAX, XML, JSON, CSS, JavaScript, Spring, Struts, Hibernate, Eclipse, Apache Tomcat, and Oracle.
