SUMMARY:

• 3 years of practical software development as Hadoop developer in Big Data/Hadoop/Spark technology development.
• Experience in developing applications that perform large scale distributed data processing using big data ecosystem tools like HDFS, YARN, Sqoop, Kafka, MapReduce, Pig, Hive, Spark, Spark SQL, Spark Streaming, HBase, Oozie.
• Good functional experience in using various Hadoop distributions like Cloudera and MapR
• Good understanding in using data ingestion tools - such as Kafka and Sqoop.
• Experienced in performing in-memory real time data processing using Apache Spark.
• Good experience in developing multiple Kafka Producers and Consumers as per business requirements.
• Extensively worked on Spark components like Spark SQL, and Spark Streaming.
• Configured Spark Streaming to receive real time data from Kafka and store the stream data to HDFS and process it using Spark and Scala.
• Developed quality code adhering to Scala coding standards and best practices.
• Experience in migrating map reduce programs into Spark RDD transformations, actions to improve performance.
• Great experience in data analyzation using HiveQL, Pig Latin, HBase and custom MapReduce programs in Java.
• Expertise in writing Hive-ACID tables and Pig queries for data analysis to meet the business requirement.
• Extensively worked on Hive transactions and Sqoop for sourcing and transformations.
• Extensive work experience in creating UDFs, UDAFs in Pig and Hive.
• Good experience in using Impala for data analysis.
• Expertise in relational databases like MySQL, SQL Server, and Oracle.
• Experience in migrating historical data from HBase to Oracle.
• Expertise in writing map reduce programs in Java for data extraction, transformation, and aggregation from various file formats such as XML, JSON, CSV, Avro, Parquet and other formats.
• Good knowledge in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Involved in identifying job dependencies to design work flow for Oozie & YARN resource management.
• Experience working with Core Java, J2EE, JDBC, ODBC, JSP, Java Eclipse, EJB and Servlets.
• Experience in using bug tracking and ticketing systems such as JIRA, and Rally.
• Hands on experience on build tools like Maven, JUnit, and Ant.
• Highly involved in all facets of SDLC using Agile Scrum methodologies.

TECHNICAL SKILLS:

Big Data/ Hadoop: HDFS, MapReduce, Pig, Hive, Spark, Kafka, Sqoop, Flume, Hbase Impala, Oozie, Zookeeper, YARN, Hue.

Hadoop Distributions: Cloudera, MapR

Programming Languages: Java, Scala.

Database/NoSQL: HBase, MySQL, Oracle

Scripting Languages: JSP, Servlets, JavaScript, XML, HTML

ETL Tools: Informatica

PROFESSIONAL EXPERIENCE

Confidential, Phoenix, AZ

Big Data Engineer

Responsibilities:

• Worked on Kafka and Spark integration for real time data processing.
• Responsible for design & deployment of Spark SQL scripts and Scala shell commands based on functional specifications.
• Experience in Configure, Design, Implement and monitor Kafka Cluster and connectors.
• Developed Kafka producer and consumer components for real time data processing.
• Developed spark streaming jobs by developing RDD’s (Resilient Distributed Datasets) and Spark SQL as required.
• Implemented Spark using Java, and Spark SQL API for faster processing data.
• Worked with Spark to create structured data from the pool of unstructured data received.
• Performed unit testing using JUnit.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Java.
• Analyzed the data by performing Hive queries on existing database. Designed and Implemented partitioning (Static, Dynamic), Buckets in Hive.
• Created Hive Generic UDF’s to process business logic that varies based on policy.
• Load and transform large sets of structured, semi structured using Hive .
• Involved in creating Hive-ACID tables, loading the data and writing Hive queries that will run internally in Map Reduce way.
• Extended Hive-ACID transactions and Pig core functionality by writing custom UDFs.
• Implemented intermediate functionalities like events or records count from the Kafka topics by writing Spark programs in Java and Java.
• Involved in identifying job dependencies to design workflow for Oozie and YARN resource management.
• Dealt with Rally, ServiceNow as ticket tracking and work flow.
• Involved in sprint planning, code review and daily standup meetings to discuss the progress of the application.
• Effectively followed Agile Scrum methodology to design, develop, deploy and support solutions that leverage the client big data platform.

Environment : Apache Spark, Spark SQL, Java, Hive , MapR, Apache Kafka, MySQL, Oracle, Spark Streaming, Java 8, Agile, Rally, ServiceNow..

Confidential, Philadelphia

Hadoop Developer

Responsibilities:

• Migrated historical data from HBase to Oracle.
• Designed and implemented Spark jobs to support distributed data processing.
• Worked with Spark ecosystem using Spark SQL and Scala queries on different formats like text file, and CSV file.
• Implemented Scala scripts, UDF’s using both Data frames/SQL and RDD in Spark for data aggregation.
• Involved in developing Impala scripts to do Adhoc queries.
• Implemented Spark RDD’s transformmations and performed actions to implement business analysis.
• Involved in ETL process for design, development, testing and migration to production environments.
• Involved in writing the ETL test scripts and guided the testing team in executing the test scripts.
• Involved in loading data from UNIX/LINUX file system and FTP to HDFS.
• Used Flume to stream through the log data from various sources.
• Configured Flume to extract the data from the web server output files to load into HDFS.
• Set up Elastic Search and Logstash and Kibana for searching and routing the log data.
• Implemented counters on HBase data to count total records on different tables.
• Created HBase tables to store variable data formats coming from different portfolios. Performed real time analytics on HBase using Java API and Rest API.
• Developed different kind of custom filters and handled pre-defined filters on HBase data using API.
• Oozie and were used to automate the flow of jobs and coordination in the cluster respectively.
• Used Jira for ticket tracking, Git & Bitbucket to check in and check out code changes.

Environment : Apache Spark, Spark SQL, Scala, Hive , Flume, Hbase, Impala, Agile, ETL, Oozie, Linux, Git, MobXterm, Jira.

Confidential

Java/Hadoop Developer

Responsibilities :

• Developed solutions to process data into data lake HDFS, process within Hadoop and emit the summary results from Hadoop to downstream systems.
• Installed and configured Hadoop MapReduce, developed multiple MapReduce jobs for cleansing and preprocessing.
• Worked and written Hadoop MapReduce jobs and creating workflows for running jobs.
• Worked on Sqoop extensively to ingest data from various source systems into data lake HDFS.
• Imported data from different relational data sources like Oracle, MySQL to HDFS using Sqoop.
• Analyzed substantial data steps using Hive queries and Pig scripts.
• Written Pig scripts for sorting, joining, and grouping data.
• Integrated multiple sources of data MySQL into Hadoop cluster and analyzed data by Hive-HBase integration.
• Involved in writing optimized Pig Script along with developing and testing Pig Latin Scripts.
• Played a major role in working with the team to leverage Sqoop for extracting data from Oracle.
• Solved small file problem using Sequence files processing in MapReduce.
• Implemented the project using the Spring framework (used Spring IOC and dependency injection, Spring MVC)
• Designed and developed server side J2EE components and used Struts framework for the web application to adopt MVC architecture.
• Created new and maintained existing web pages built in JSP, Servlet .
• Worked on different file formats like Text files, Parquet, Sequence Files, Avro, Record columnar files (RC).
• Understood complex data structures of different type (structured, semi structured) and de-normalizing for storage in Hadoop.
• Experienced with working on Avro Data files using Avro Serialization system.
• Kerberos security was implemented to safeguard the cluster.

Environment: HDFS, Pig, MapReduce, Sqoop, Oozie, Java, Eclipse, MySQL, Oracle, Shell Scripting, Kerberos, Oozie, Cloudera, MySQL.