SUMMARY:

• Around 3+ years of hands - on experience in working on Apache Hadoop ecosystem components like Map-Reduce, Sqoop, Flume Pig, Hive, HBase, Oozie, Kafka, and Zookeeper.
• Excellent understanding/ knowledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager and Map Reduce.
• Experience in analyzing data using HiveQL, Pig Latin.
• Knowledge in job work-flow scheduling and monitoring tools like Oozie.
• Experience in different Hadoop distributions like Cloudera 5.3(CDH4, CHD 5) and Horton Works Distributions(HDP).
• Proficient knowledge on Apache Spark and programming SCALA to analyze large datasets using Spark and Storm & Kafka to process real time data.
• Worked on writing custom UDF’s in java for Hive and Pig.
• Involved in building, evolving and reporting framework on top of the Hadoop cluster to facilitate data mining, analytics and dash-boarding.
• Support a wide variety of ad hoc data needs.
• Hands on experience in writing map-reduce programs in java.
• Have a strong ability to prepare and present data in a visually appealing and easy to understand manner.
• Build high-volume real-time data processing applications using Hadoop platform.
• Performed Importing and exporting data into HDFS, Hive and HBase using Sqoop.
• Hands-on experience on full life cycle implementation using MapR, CDH (Cloudera) and HDP (Hartonworks Data Platform).
• Strong end-to-end experience in Hadoop Development.
• Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa.
• Experience in data transformations using Map-Reduce, HIVE and Pig scripts for different file formats.
• Expertise in analyzing the data using HIVE and writing custom UDF's in JAVA for extended HIVE and PIG core functionality.
• Hands on experience in configuring and administering the Hadoop Cluster.
• Good understanding of HDFS Designs, Daemons and HDFS high availability (HA).
• Experience with various scripting languages like Linux/Unix shell scripts, Python.
• Experience in understanding and managing Hadoop Log Files.
• Experience in managing the Hadoop infrastructure with Cloudera Manager.
• Experienced with data warehousing and ETL processes.
• Expert knowledge of data warehousing concepts, with hands-on in developing ETL applications in a dimensional data mart/data warehouse environment.
• Involved in creating MVC architecture using java, validating files, Struts frame Work.
• Participated in Review of Test Plan, Test Cases and Test Scripts prepared by system integration testing team.
• Monitored the performance and identified performance bottlenecks in ETL code.
• Strong experience in client interaction and understanding business application, business data flow and data relations.
• Have flair to adapt to new software applications and products, self-starter, have excellent communication skills and good understanding of business work flow.

TECHNICAL SKILLS:

Script Languages: Shell Scripting, Python, Unix script

Big Data Technologies: HDFS,MapReduce,Hive,Hql,Pig,Sqoop, Flume, Spark, Zookeeper, Oozie, Kafka

RDBMS: MySQL, Oracle, Teradata, MSSQL

Programming language: Python, SQL, Java

IDE’s: NetBeans, Eclipse

Tools: Maven

Virtual Machines: VMWare, Virtual Box

OS: Cent OS 5.5, Unix, RedHat Linux,Windows7,Debian, Kali

PROFESSIONAL EXPERIENCE:

Sr. Big Data/Hadoop Developer

Confidential, Hoffman Estate, IL

Responsibilities:

• Expert in implementing advanced procedures like text analytics and processing using the in-memorycomputing capabilities like Apache Spark written in Python.
• Developed and executed shell scripts to automate the jobs.
• Wrote complex Hive queries and UDFs.
• Worked on reading multiple data formats on HDFS using PySpark.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python.
• Developed multiple POCs using PySpark and deployed on the Yarn cluster, compared the performance ofSpark, with Hive and SQL/Teradata.
• Analyzed the SQL scripts and designed the solution to implement using PySpark.
• Involved in loading data from UNIX file system to HDFS
• Extracted the data from Teradata into HDFS using Sqoop.
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce.
• Spark and loaded data into HDFS.
• Manage and review Hadoop log files.
• Involved in analysis, design, testing phases and responsible for documenting technical specifications.
• Worked hand-in-hand with the Architect; enhanced and optimized product Spark code to aggregate, group and run data mining tasks using Spark framework
• Hands on experience in joining raw data with the reference data using Pig scripting.
• Written custom UDF’s in Hive.
• Hands on extracting data from different databases and to copy into HDFS file system using Sqoop.
• Created Oozie coordinated workflow to execute Sqoop incremental job daily.
• Used Oozie workflow engine to run multiple Hive and Pig jobs.
• Involved in installing and configuring Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster.
• Used Kafka to consume data into Hadoop.
• Very good understanding of Partitions, bucketing concepts in Hive and designed both managed and External tables in Hive to optimize performance.
• Worked on Hive context and SQLContext of spark extensively.
• Experienced in running Hadoop streaming jobs to process terabytes of data.
• Involved in importing the real time data to Hadoop using Kafka and implemented the Oozie job for dailyimports.

ENVIRONMENT: Hadoop, HDFS, Hive, Python, Spark, SQL, Teradata, Yarn, Sqoop, Kafka, UNIX Shell Scripting.

Big Data/Hadoop Developer

Confidential, Columbus, OH

Responsibilities:

• Worked on Spark SQL, Spark Streaming, Reading/Writing data from JSON file, text file, parquet file, Schema RDD.
• Worked extensively with HIVE DDLS and Hive Query language(HQLs).
• Developed PIG Latin for handling business transformations.
• Responsible writing PIG script and Hive queries for data processing.
• Developed Pig Latin scripts using operators such as LOAD, STORE, DUMP, FILTER, DISTINCT, FOREACH, GENERATE, GROUP, COGROUP, ORDER, LIMIT, UNION.
• Involved in implementing the job workflows and scheduling for the end to end application processing.
• Written spark programs in Python and ran spark jobs on YARN.
• Worked with HBase databases for non-relational data storage and retrieval on enterprise usecases.
• Wrote Map Reduce jobs using Java API and Pig Latin.
• Loaded the data from Teradata to HDFS using Teradata Hadoop connectors.
• Issued SQL queries via Impala to process the data stored in HDFS and HBase.
• Involved in developing Impala scripts for extraction, transformation, loading of data in to data warehouse.
• Used Flume to collect, aggregate and store the web log data onto HDFS.
• Wrote Pig scripts to run ETL jobs on the data in HDFS.
• Used Hive to do analysis on the data and identify different correlations.
• Written lots PIG UDF to process some complex data.
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop.
• Configured MySQL Database to store Hive metadata.
• Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
• Written Hive queries for data analysis to meet the business requirements.
• Automated all the jobs, for pulling data from FTP server to load data into Hive tables, using Oozie workflows.
• Involved in creating Hive tables and working on them using Hive QL.
• Loaded data into the cluster from dynamically generated files using Flume and from relational databasemanagement systems using Sqoop.
• Utilized Agile Scrum Methodology to help manage and organize a team of 4 developers with regular codereview sessions.
• Weekly meetings with technical collaborators and active participation in code review sessions with seniorand junior developers.

ENVIRONMENT: Hadoop, Map Reduce, HDFS, Pig, Hive, Spark, Yarn, Sqoop, Flume, Zookeeper,CDH 5.4, Oozie, ETL, MYSQL, agile, Windows, UNIX Shell Scripting, Teradata.

Data Stage Developer

Nationwide, Columbus, OH

Responsibilities:

• Designed jobs involving various cross reference lookups and joins, shared containers which can be usedin multiple jobs.
• Sequencers are created at job level to include multiple jobs and a layer level sequence which include alljob level sequences.
• Extensively employed Data Stage Director to validate, run, schedule, monitor the jobs and followed job logcarefully to debug the jobs.
• Carefully monitored the performance statistics and involved in fine tuning of jobs for the improved processingtime.
• Involved in developing UNIX scripts to call Data stage jobs.
• Involved in fine tuning, trouble shooting, bug fixing, defect analysis and enhancement of the multiple adminsystems Data stage jobs.
• Involved in the designing of marts and dimensional and fact tables.

ENVIRONMENT: Data stage 7.5, Teradata, Mainframe system.

JAVA DEVELOPER

Stryker, Kalamazoo, MI

• The application was developed in J2EE using an MVC based architecture.
• Implemented MVC design using Struts1.3 frameworks, JSP custom tag Libraries and various in-house customtag libraries for the presentation layer.
• Created tile definitions, Struts-config files, validation files and resource bundles for all modules using Struts framework.
• Wrote prepared statements and called stored Procedures using callable statements in MySQL.
• Executing SQL queries to check the customer records are updated appropriately.
• Used Apache Tomcat as the application server for deployment.
• Used Web services for transmission of large blocks of XML data over HTTP.

ENVIRONMENT: Java/J2EE, JSP, MySQL, Struts 1.3, Apache Tomcat, Eclipse, XML.