SUMMARY

• 5+ years of IT experience in which 3+years as hadoop developer worked in building the scalable bigdata applications, designing and implementing.
• Extensive working experience on Hadoop eco - system components like HDFS, MapReduce, Hive, Pig, Sqoop, Spark, Flume, Kafka and Zookeeper.
• Experience in importing and exporting data using Sqoop from HDFS/Hive/HBase to Relational Database Systems and vice-versa.
• Experience in working with Flume/Kafka to load the log data from different sources into HDFS.
• Experience in optimization of MapReduce algorithm using Combiners and Partitioner to deliver best results.
• Extensive working knowledge on writing and implementing UDF’s in HIVE and PIG.
• Experience in working with Hive tool creating tables, data distribution by implementing partitioning and bucketing, writing and optimizing the Hive SQL queries.
• Strong knowledge of data warehousing technologies such as Hive, including Extract, Transform, and Load Processes.
• Hands-on experience in writing Pig Latin scripts.
• Extensively worked on Spark Streaming and Spark Sql, consuming petabytes of real time data from Kafka, and performed various transformations on data and queried using Spark Sql.
• Hands on in converting Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL and Scala/Python.
• Strong knowledge in NoSQL databases like MongoDB, HBase, and Cassandra.
• Experience in installing, configuring, and administrating Hadoop cluster for distributions like Cloudera.
• Hands on experience in setting up a workflow using Apache Oozie workflow engine for managing and scheduling hadoop jobs.
• Hands on experience in using BI tool like Tableau.
• Experienced in Application Development using Java, Hadoop, RDBMS and Linux shell scripting and performance tuning.
• Comprehensive knowledge of Software Development Life Cycle coupled with excellent communication skills. Strong technical and interpersonal skills combined with great commitment towards meeting deadlines.

TECHNICAL SKILLS

Hadoop Technologies and Distributed system: Apache Hadoop, Cloudera Hadoop Distribution.

Hadoop Ecosystem: HDFS, Hive, Pig, Sqoop, Oozie, Flume, SparkZookeeper, Map-Reduce, Kafka.

NoSQL Databases: Cassandra, Mongo DB.

Programming: C, C++, Python, Java, SQL, J2EE.

RDBMS: ORACLE, MySQL, SQL Server.

Web Development: HTML, JSP, Servlets, XML.

IDE: Eclipse4.x, NetBeans.

Operating Systems: Linux (Red Hat, CentOS), Windows XP/7/8.

Web Servers: Apache Tomcat.

Cluster Management Tools: Cloudera Manager.

PROFESSIONAL EXPERIENCE

Confidential, Atlanta-GA

Spark/Hadoop Developer

Responsibilities:

• Involved in end to end data processing like ingestion, processing, and quality checks and splitting.
• Worked on structured data and semi structured data with daily incremental loads of 1 TB in size and monthly, quarterly loads of several TB’s.
• Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data.
• Developed a data pipeline using Kafka, Spark and Hive to ingest, transform and analyzing data.
• Developed Hive scripts for end user / analyst requirements to perform ad hoc analysis.
• Solved performance issues in Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
• Writing spark programs creating data frames by dynamically applying schema to the flat source files by reading target hive table schema using hive context.
• Writing data to parquet tables both non-partitioned and partitioned tables by adding dynamic data to partitioned tables using spark.
• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
• Worked on Oozie and Info works for workflow automation and for scheduling jobs.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Hadoop YARN, Spark Core, Spark Streaming, Spark SQL, Scala, Python, Kafka, Hive, Sqoop, Oozie, Cloudera, Oracle, Linux.

Confidential, Herndon-VA

Hadoop Developer

Responsibilities:

• Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Kafka and stored the data into HDFS for analysis.
• Worked in Importing and exporting data into HDFS and Hive using Sqoop.
• Involved in collecting, aggregating and moving data from servers to HDFS using Apache Flume.
• Developed UDF’s in Java as and when necessary to use in PIG and HIVE queries.
• Involved in creating Pig tables, loading with data and writing Pig Latin queries which will run internally in Map Reduce way.
• Very good understanding of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
• Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.
• Downloading the data from hdfs to local server and to extract the data as per requirement and visualize it for business.

Environment: Hadoop, Map Reduce, HDFS, Sqoop, Pig, Apache Flume, Oozie, Hive. Oracle.

Confidential

Big Data Engineer

Responsibilities:

• Design and develop data ingestion framework using Hadoop stacks and expertise in analyzing the logs and diagnosis the issues.
• Used Sqoop to extract data from multiple structured sources and to export data to other external RDBMS tables for querying and reporting.
• Developed multiple MapReduce jobs in java for data cleaning and pre-processing.
• Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
• Involved in creating Hive tables, and loading and analyzing data using hive queries.
• Implemented Join optimizations in Pig using Skewed and Merge joins for large datasets.
• Created custom re-usable UDFs, UDAFs, UDTFs and macros in Pig Latin/Hive and used in various reporting queries.
• Exploring the different methods in design for performance optimization.

Environment: Hadoop, Map reduce, Linux, Sqoop, HBase, Pig, Flume, Pig, Hive.

Confidential

Java Developer

Responsibilities:

• Designed object model diagrams and data model diagrams to achieve the business needs.
• Worked on creating custom components by using HTML, JavaScript’s, CSS and used Controls and Repeaters for the data sets.
• Extensively worked on Spring framework for developing the control logic and interacts with Domain.
• Worked in developing the presentation layer of the application by using JSP, servlets.
• Participated in writing stored procedures and added new paper types also for finishing options to the database.
• Implemented a Web based Application using Servlets, JSP, Spring, JDBC, XML.
• Implemented Webservices for most of the modules.

Environment: Java, Servlets, JSP, HTML, PHP, JavaScript’s, JDBC, Spring, CSS, Eclipse.

Confidential

Jr. Java Developer

Responsibilities:

• Involved in design phase meetings for Business Analysis and Requirements gathering.
• Worked with business functional lead to review and finalize requirements and data profiling analysis.
• Responsible for gathering the requirements, designing and developing the applications.
• Worked on UML diagrams for the project use case.
• Worked with Java String manipulations, to parse CSV data for applications.
• Connected Java applications to Java database to read, write data.
• Developed static and dynamic Web Pages using JSP, HTML and CSS.

Environment: Java, Servlets, JSP, HTML, PHP, CSS, Eclipse.