Sr. Hadoop Developer Resume
Denver, CO
SUMMARY
- Over 8 years of professional IT experience in all phases of Software Development Life Cycle which includes 4 years of experience in Big Data, Hadoop Development and Ecosystem Analytics in Banking and Retail industries.
- Hands on experience in using Hadoop Ecosystem components like HDFS, Map Reduce, Hive, Pig, HBase, Zookeeper, Sqoop, Flume, Kafka, Oozie and Spark.
- Hands on experience in Apache Hadoop Map Reduce Programming and Pig Scripting.
- Expert level understanding of Map Reduce internals, including shuffling, troubleshooting and partitioning. The bottlenecks in performance of a map reduce program.
- Expertise in extending Hive and Pig functionalities by writing customized UDF’S.
- Good knowledge on building Apache Spark applications.
- Implemented Spark using Scala and SparkSQL for faster processing and testing of data.
- Used Impala to read, write, and query the data in HDFS.
- Experience with Oozie workflow Engine in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.
- Experience in scripting for automation and monitoring using Python.
- Excellent hands on experience with migrating data from different Relational Database Systems like MySQL and Oracle into HDFS and Hive and vice versa using Sqoop.
- Good hands on experience in migrating unstructured data into HDFS using Flume/Kafka.
- Imported data from Kafka consumer into HBase using spark streaming.
- Good experience in NoSQL databases like HBase and MongoDB.
- Involved in complete flow of the application, starting from data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data.
- Involved in ETL, Data Integration, Ingestion and Migration.
- Used Maven for building jar files of MapReduce programs and deployed to Cluster.
- Generated various marketing reports using Tableau and Spotfire.
- Implemented Continuous Integration using Jenkins.
- Experience in Amazon AWS cloud services.
- Configured HDFS to run on top of S3 to reduce the cost and leveraging the choice of a wide variety Amazon EC2 instances and Amazon EBS volumes for cluster's hardware, backup - and-restore to Amazon S3.
- Good knowledge in Object Oriented Design Pattern concepts.
- Expertise in Java Multithreading, Exception Handling, JSP, Servlets, JavaScript, jQuery, AJAX, CSS, HTML, Spring, Hibernate, Enterprise Java Beans, JDBC, RMI and XML related technologies.
- Experience with version control systems like GitHub, SVN and CVS.
- Good knowledge on SDLC, Agile and Waterfall Methodologies.
- Involved in daily SCRUM meetings to discuss the development/progress and was active in making Scrum meetings more productive.
- Excellent analytical, problem solving and interpersonal skills.
- Ability to learn recent technologies and methodologies quickly. Consistent team player with excellent communication skills.
TECHNICAL SKILLS
Big Data Technologies: HDFS, YARN, Map Reduce, Pig, Hive, Sqoop, Flume, HBase, MongoDB, Oozie, Cassandra, Zookeeper, Spark, Kafka
Languages: C, C++, Java, J2EE, JavaScript, XML, PL/SQL, Python, Scala, Pig Latin, HiveQL
Frameworks: Spring, Hibernate, Struts
Web Technologies: HTML, CSS, Ajax, XML, jQuery
Web Servers: Apache Tomcat, WebSphere, WebLogic
Databases: Oracle, DB2, SQL Server, MySQL
IDE: Eclipse, NetBeans, SQL Server, MS Office Suite, Notepad++
Operating Systems: UNIX, LINUX, Windows, Mac, CentOS
PROFESSIONAL EXPERIENCE
Sr. Hadoop Developer
Confidential, Denver, CO
Responsibilities:
- Wrote Map Reduce programs for data cleaning and preprocessing.
- Involved in writing HiveQL to extract, transform and load the data into Database.
- Wrote HiveQL queries and Pig Latin Scripts.
- Wrote custom UDF's for Hive and Pig using Java.
- Performed Hive partitioning, bucketing and executing different types of joins on Hive tables and implementing Hive Serves like JSON and Avro.
- Fine tuning hive jobs for optimized performance.
- Used Spark Streaming to divide streaming data into batches as an input to spark engine for batch processing.
- Developed SparkSQL to load tables into HDFS to run select queries on top.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs.
- Imported data from flume consumer into HBase using spark streaming.
- Created Hive External tables and loaded data in to tables and queried data using HiveQL.
- Experienced in using Zookeeper and Oozie Operational Services for coordinating the cluster and scheduling workflows.
- Automated all the jobs, for ingesting data from relational databases to load data into Hive tables, using Oozie workflows.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java MapReduce, Hive and Sqoop as well as system specific job.
- Experience in HBase database manipulation with structured, unstructured and semi-structured datasets.
- Configured HDFS to run on top of S3 to reduce the cost and leveraging the choice of a wide variety Amazon EC2 instances and Amazon EBS volumes for cluster's hardware, backup-and-restore to Amazon S3.
- Involved in ETL, Data Integration, Ingestion and Migration.
- Implemented ETL pipelines to ingest data from traditional EDW Hadoop.
- Generated various marketing reports using Tableau with Hadoop as a source for data.
- Implemented Continuous Integration using Jenkins.
- Involved in daily SCRUM meetings to discuss the development/progress and was active in making Scrum meetings more productive.
Environment: Apache Hadoop, Apache Hive, HDFS, Map Reduce, Pig, Spark, Oozie, Flume, Kafka, Tableau, Jenkins, Java, Python, Linux, Agile.
Hadoop Developer
Confidential, Atlanta, GA
Responsibilities:
- Worked on Hive for exposing data for further analysis and for generating transforming files from different analytical formats to text files.
- Expert level understanding of Map Reduce internals, including shuffling, troubleshooting and partitioning. The bottlenecks in performance of a map reduce program.
- Developed Shell Scripts to perform Data Profiling on the ingested data with the help of Hive bucketing.
- Involved in creating Hive tables for customer/Transaction, loading data into tables and partitioned tables using Hive Index and bucket to make ease data analytics.
- Developed Pig Scripts, Pig and Hive UDFs to process the data.
- Implemented Spark using Scala and SparkSQL for faster testing and processing of data.
- Developed Map Reduce jobs for Log Analysis, Analytics and to generate reports for the number of activities created.
- Used Impala to read, write, and query the data in HDFS.
- Created HBase tables to load large data sets from UNIX, NoSQL and a variety of portfolios.
- Used Flume to collect, aggregate, and store the web log data from various sources like web servers, network devices and pushed to HDFS.
- Importing and exporting data into HDFS and Hive using Flume.
- Used AWS services like EC2 and S3 for small data sets.
- Created and involved in scheduling Oozie workflow engine to run Sqoop jobs.
- Used Zookeeper and Oozie Operational Services for coordinating the cluster and scheduling workflows.
- Implemented Continuous Integration using Jenkins.
- Used Maven for building jar files of MapReduce programs and deployed to Cluster.
- Used GitHub for version control across common source code used by developers.
- Analyzing the issues and proactively identifying potential problems and their solutions.
Environment: Apache Hadoop, Hive, Map Reduce, HDFS, Pig, HBase, Cassandra, Oozie, Impala, AWS, Maven, GitHub, Jenkins, Linux.
Hadoop Developer
Confidential, Boca Raton, FL
Responsibilities:
- Developing multiple MapReduce programs in java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other file formats.
- Wrote Map Reduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration.
- Experience in writing aggression logics in different combinations to perform complex data analytics to the business needs.
- Experience in creating tables, dropping and altered at run time without blocking updates and queries using HBase and Hive.
- Developed Java programs to process huge JSON files received from marketing team to convert into format standardized for the application.
- Worked on different Big Data file formats like txt, sequence, Avro, Parquet and Snappy compression.
- Involved in complete Big Data flow of the application starting from data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data.
- Performed analysis of data using Pig scripting.
- Implemented solutions for ingesting data from various sources and processing the data utilizing Big Data Technologies such as Hive, Pig, Sqoop, HBase, Map reduce, etc.
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as map-reduce Hive, Pig, and Sqoop.
- Used Oozie to organize the MapReduce jobs and worked with HCatalog to open access to Hive's Metastore.
Environment: Apache Hadoop, Hive, Map Reduce, Pig, HBase, Sqoop, Oozie, Linux, Java, Agile.
Java Developer
Confidential
Responsibilities:
- Involved in all the phases of SDLC process.
- JDBC concept was used for database transaction management using create statements, prepared statements and callable statements to interact with the underlying MySQL Database.
- Used various Core Java concepts such as Multi-Threading, Exception Handling, Collection APIs to implement various features and enhancements.
- Application development using JSP, Servlets, JavaScript and JDBC Technologies.
- Developed service layer logic for core modules using JSPs and Servlets.
- Extensively used JavaScript to provide dynamic User Interface and for the client-side validations.
- Developed the front-end screens using HTML, CSS, XML and JSP.
- Used the Struts Validation Framework for validation and Struts Tiles Framework for reusable presentation components at the presentation tier.
- Involved in using Hibernate for data persistence tier.
- Worked on writing SQL Queries and PL/SQL Stored Procedures using Oracle. Used JDBC for connectivity.
- Involved in creating software requirement specification document.
- Implemented Maven as build and configuration tool.
- Used Eclipse IDE for development.
- Used GitHub for version control across common source code used by developers.
Environment: HTML, CSS, JSP, Oracle 10g, SQL, PL/SQL, Hibernate, Maven, GitHub, Eclipse, J2EE, CVS, Linux.
Java Developer
Confidential
Responsibilities:
- The application is designed using J2EE design patterns and technologies based on MVC architecture.
- Used HTML, jQuery, Java script and AJAX for implementing dynamic media play outs.
- Extensively used various collections classes like Array List, Hash Map, Hash Table and Hash Set.
- Generated Use case diagrams, Class diagrams, and Sequence diagrams using Rational Rose.
- Involved in using Core java concepts - Collections, Exception Handling, Multi-Threading, Serialization, Thread pools, lock interface.
- Used simple Struts Validation for validation of user input as per the business logic and initial data loading.
- Implemented Struts Framework for configuration of action mappings and presentation logic in JSPs.
- Developed Swing components such as JSlider, JButton and JCheckbox and AWT labels, textields and scrollbars.
- Designed GUI screens using usability and design principles.
- Multithreading is used to continuously keep track of downlink messages and send corresponding uplink messages.
- Locks are used to synchronize resources sharing between these threads.
- Writing database queries on Oracle 9i and Involved in the JDBC queries as part of implementation.
- Involved in Unit Testing, User Acceptance Testing and Bug Fixing.
- Used NetBeans IDE for development.
- Involved in writing JUNIT test cases and Code version controlling using SVN.
- Excellent written and verbal communication skills, inter-personal skills and self-learning attitude.
Environment: J2EE, JSP, NetBeans, Struts, HTML, Java Script AJAX, Oracle 9i, Swing, Linux, SVN.
