Hadoop Developer Resume
Plano, TexaS
SUMMARY
- Highly acumen and experienced IT professional with 8 years of experience with deep expansive knowledge of Hadoop Ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems and spanning Big data.
- A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
- Result oriented and hands on, skillfully balances between meeting resources and time constraints, while doing it right.
- Experience in Object - oriented concepts with complete software development cycle (SDLC) experience - Requirements gathering, Detail design, Development, System and User Acceptance Testing.
- Hands-on development and implementation experience on Big Data Management Platform(BMP) using Hadoop 2.x, HDFS, MapReduce/Yarn/Spark, Hive, Pig, Oozie, Sqoop and other Hadoop eco-system components.
- Experience in analyzing different types of data that flow from Data lakes to Hadoop cluster.
- Experience in developing data pipelines using Big Data Technologies.
- Proficient in big data ingestion and streaming tools like Apache Flume, Sqoop, Kafka, Storm and Spark.
- Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa.
- Migrated existing MapReduce programs to Spark using Scala and Python.
- Experience creating real-time data streaming solutions using Apache Spark core, Spark SQL, Kafka, spark streaming and Apache Storm.
- Use RDD’s to perform transformation on datasets as well as to perform actions like count, reduce, first.
- Good knowledge on Spark platform parameters like memory, cores and executors.
- Used Kafka to populate log transactions to HDFS and handling the clusters for processing.
- Used Storm for distributed real time processing for large Data sets.
- Proficient in working with MapReduce programs using Apace Hadoop for working with Big Data.
- Hands-on experience in developing MapReduce programs and User defines functions (UDF’s) for Hive and Pig.
- Hands on experience with HiveQL to query data from Hive Tables in HDFS.
- Good working exposure with ETL and Query big data tools like Pig Latin and HiveQL.
- Experience on working with data transformation from HDFS, Hive, Pig, HBase and MYSQL.
- Responsible for developing data pipelines to load data from servers such as SQL servers using Sqoop along with Kafka.
- Experienced in creating UDF’s, UDAF’s to analyze and parse large amount of data sets and assisted with performance tuning.
- Expertise in development of Web Applications using J2EE technologies like Servlets, JSP, Web Services, Spring, Hibernate, HTML, JQuery, Ajax etc.
- Experienced in working with Hadoop/Big-Data storage and analytical frameworks over Amazon AWS cloud using tools like SSH, Putty and Mind-Term.
- Good Knowledge on Amazon AWS cloud services (EC2, EBS, and S3).
- Design and Strong programming experience with Java technologies using JAVA, J2EE, JDBC and Web based development tools.
- Experience working on NoSQL technologies like HBase Developer in Internet Applications and Client/Server, Cassandra and MongoDB.
- Excellent at scripting for monitoring and automation using Shell and Perl scripts.
- Extensively development experience in different IDE’s like Eclipse, NetBeans, and Forte.
- Expertise in Oozie for configuring job work flows based on time driven and data driven.
- Proficiency in using Linux and Unix shell.
- Hands on experience with Agile and Scrum methodologies.
- Successfully working on a fast-paced environment, both independently and in collaborative team environment.
TECHNICAL SKILLS
Big Data EcoSystems: HDFS, MapReduce, Spark, Yarn, Hive, Pig, HBase, Sqoop, Flume, Kafka, Oozie, Zookeeper, Impala.
Hadoop Technologies: Apache Hadoop1.x, Apache Hadoop2.x, Cloudera CDH4/CDH5, Horton Works.
Programming Languages: Java, C/C++, MATLAB, Scala, Shell Scripts, Pig Latin, HiveQL.
Operating Systems: Windows (xp/7/8/10), Linux (Ubuntu, Centos)
NoSQL Databases: HBase, Cassandra, MangoDB.
Databases: RDBMS, MYSQL, Terabyte, DB2, Oracle.
BI Tool: Tableau, Talend.
Web Development: HTML, XML, JavaScript s.
IDE Tools: Eclipse, NetBeans and Forte.
Application Servers: Apache Tomcat, J2EE, JDBC, ODBC.
PROFESSIONAL EXPERIENCE
Hadoop Developer
Confidential, Plano, Texas
Responsibilities:
- Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Zookeeper, SQOOP, flume,Spark, Impala, Cassandra.
- Real time streaming of data usingSparkwith Kafka.
- Responsible for developing data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.
- Experienced with batch processing of data sources using ApacheSparkand Elastic search.
- Experienced in implementingSparkRDD transformations, actions to implement business analysis.
- ImplementedSparkusing Scala andSparkSQL for faster testing and processing of data.
- Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.
- Integrating user data from Cassandra to data in HDFS. Integrating Cassandra with Storm for real time user attributes look up.
- Involved in converting Hive/SQL queries intoSparktransformations usingSparkRDDs, Scala.
- Involved in importing the real-time data to Hadoop using Kafka and implemented the Oozie job for daily imports.
- Automated the process for extraction of data from warehouses and weblogs by developing work-flows and coordinator jobs in Oozie.
- Created Hive tables and involved in data loading and writing Hive UDFs.
- Working with Talend Open Studio for ETL process.
Environment: Hadoop, MapReduce, HDFS, Pig,Spark, Hive,SparkStreaming,SparkSQL, Apache Kafka, Sqoop, Java, Scala, CDH4, CDH5, AWS, Eclipse, Oracle, Git, Shell Scripting and Cassandra.
Hadoop Developer
Confidential, San Diego, CA
Responsibilities:
- Involved in the complete Software Development Life Cycle (SDLC) to develop the application.
- Worked on analyzingHadoopcluster and different Big Data analytics tools including Pig, Hive, HBase and Sqoop.
- Used Pig Latin and Hive to work on unmodified data and produce reports for client's purposes for large data analysis.
- Involved in loading data from local UNIX file system to HDFS.
- Experience in managing and reviewingHadooplog files.
- Responsible for Cluster maintenance, adding and removing Cluster nodes, Cluster Monitoring, and Troubleshooting, Manage and review backups and log files.
- Exported the analyzed data to relational databases using Sqoop for visualization and to generate reports for the BI team.
- Developed Pig UDF's to pre-process data for analysis.
- Developed Java MapReduce programs for analysis of log files stored in cluster.
- Importing and exporting data into HDFS and Hive using Sqoop.
- Installed Oozie workflow engine to run multiple Hive and Pig Jobs.
- Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
- Migration of ETL processes from RDBMS to HIVE to test easy data manipulation.
- Implemented test scripts to support test driven development and continuous integration.
- Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.
- Developed Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.
- Mentored analyst and test team for writing Hive queries.
- Cluster co-ordination services through Zookeeper.
Environment: Hadoop, HDFS, MapReduce, Hive, Pig, HBase, Sqoop, Spark, Oozie, Zookeeper, RDBMS/DB, MySQL, CSV, AVRO data files.
Hadoop Developer
Confidential, New York
Responsibilities:
- Importing and exporting data into HDFS and Hive using Sqoop and Kafka.
- Develop different components of system likeHadoopprocess that involves Map Reduce, and Hive.
- Developed interface for validating incoming data into HDFS before kicking offHadoopprocess.
- Written hive queries using optimized ways like user-defined functions, customizingHadoopshuffle & sort parameters.
- Worked on tuning Hive and Pig to improve performance and solve performance related issues in Hive and Pig scripts with good understanding of Joins, Group, and aggregation and how it does Map Reduce jobs.
- Developing map reduce programs for different types of Files using Combiners with UDF's and UDAF's.
- Experience working on multiple node cluster tool which offer several commands to return HBase usage.
- Experience in creating tables, dropping, and altered at run time without blocking updates and queries using HBase and Hive.
- Experience on pre-processing the logs and semi structured content stored on HDFS using PIG.
- Experience in structured data imports and exports into Hive warehouse which enables business analysts to write Hive queries.
- Experience in managing and reviewingHadooplog files.
- Experience on Unix shell scripts for business process and loading data from different interfaces to HDFS.
- Involved in creating Hive tables, Pig tables, and loading data and writing hive queries and pig scripts.
- Hands on experience in eclipse, Putty, WinSCP, VNCviewer, etc.
Environment: Linux 6.7, CDH5.5.2, MapReduce, Hive 1.1, PIG, HBase, Shell Script, SQOOP 1.4.3, Eclipse, Java 1.8.
Java Developer
Confidential
Responsibilities:
- Developing front-end screens using JSP, HTML and CSS.
- Developing server side code using Struts and Servlets.
- Developing core java classes for exceptions, utility classes, business delegate, and test cases.
- Developing SQL queries using MySQL and established connectivity.
- Working with Eclipse using Maven plugin for Eclipse IDE.
- Designing the user interface of the application using HTML5, CSS3, JSP, and JavaScript.
- Tested the application functionality with JUnit Test Cases.
- Developing all the User Interfaces using JSP framework.
- Writing Client Side validations using JavaScript.
- Extensively used JQuery for developing interactive web pages.
- Developed the user interface presentation screens using HTML, XML, and CSS.
- Developed the Shell scripts to trigger the Java Batch job, Sending summary email for the batch job status and processing summary.
- Co-ordinate with the QA lead for development of test plan, test cases, test code and actual testing responsible for defects allocation and those defects are resolved.
- Application was developed in Eclipse IDE and was deployed on Tomcat server.
- Involved in Agile scrum methodology.
- Supported for bug fixes and functionality change.
Environment: Java/J2EE, Oracle 10g, SQL, PL/SQL, JSP, Hibernate, WebLogic 8.0, HTML, AJAX, Java Script, JDBC, XML, UML, JUnit, Eclipse.
Jr. Java/J2EE Developer
Confidential
Responsibilities:
- Interacting with the client on a regular basis to gather requirements.
- Understanding the business, technical, and functional requirements.
- Checking for timely delivery of various milestones.
- Using Spring Framework, Axis, developed web services including design of the XML request/response structure.
- Implemented Hibernate/Spring framework for Database and business layer.
- Configured Oracle with Hibernate, wrote hibernate mapping and configuration files for database processing (Create, Update, select) operations.
- Involved in creating Oracle stored procedures for data/business logic.
- Created PL/SQL stored procedures for Contract generation module.
- Involved in configuring and deploying of code to different environments Integration, QA and UAT.
- Preparing and designing system/acceptance test cases and executing them.
- Created ant build script to build Artifacts.
- Worked on fine-tuning the response time of Web Service components.
Environment: Java, JSP, EJB, Servlets, Struts, Tomcat, Web logic, Oracle 10g.
Java Developer
Confidential
Responsibilities:
- Developed the applications using Java, J2EE, Structs, JDBC.
- Built applications for scale using Java Script.
- Used SOAP UI Pro version for serving the Web Services.
- Involved in preparing High level and Detail level design of the system using J2EE.
- Created Structs form beans, action classes, JSPs following Struts framework standards.
- Implemented the database connectivity using JDBC with Oracle database as backend.
- Involved in the development of underwriting process, which involves communications without side systems using IBM, MQ and JMS.
- Created a deployment procedure utilizing Jenkins CI to run the unit tests.
- Worked with JMS Queues for sending messages in point-to-point mode.
- Used PL/SQL stored procedures for applications that needed to execute as part of a scheduling mechanisms.
- Developed SOAP based XML web services.
- Used JAXB to manipulate XML documents.
- Created XML document using STAX XML API to pass the XML structure to Web Services.
- Used Rational Clear Case for version control and JUnit for unit testing.
- Provided troubleshooting and error handling support in multiple projects
Environment: JSP1.2, Jasper reports, JMS, XML, SOAP, JDBC, JavaScript, XML, UML, HTML, JNDI, Apache Tomcat, ANT, and JUnit.
