SUMMARY

• Software Engineer having more than 8 years of experience in Java, J2EE, Big Data Management, Design, Development, SDLC and clear objective to work, grow and learn as much as an engineer.
• Over 8 plus years of experience in Software Development Life Cycle (SLDC) including requirements and system analysis, design, programming, testing, implementation, and application maintenance
• Involved in all phases of Software Development Life Cycle (SDLC) and Worked on all activities related to the development, implementation, administration and support for Hadoop.
• Experience in designing, installing and configuring complete Hadoop ecosystem (components such as HDFS, Map reduce, pig, hive, oozie, flume, zookeeper).
• Experience in upgrading the existing Hadoop infrastructure to latest releases, monitoring, tuning and administrating Hadoop cluster.
• Experience in writing Map Reduce programs using Java to perform Data processing and analysis.
• Experience in understanding Big Data business requirements and providing them Hadoop based solutions.
• Experience in Importing and exporting data from different databases like MySQL, Oracle, Netezza, Teradata, DB2 into HDFS using Sqoop.
• Good experience in design the jobs and transformations and load the data sequentially & parallel for initial and incremental loads.
• Good experience in using various PDI / Kettle (Pentaho Data Integrator) steps in cleansing and load the data as per the business needs
• Experience in writing UDF'S in Pig and Hive for processing and analyzing large datasets.
• Experience in working with different file formats and compression techniques in Hadoop.
• Experience in developing and scheduling ETL workflows in Hadoop using Oozie.
• Experience in setting up Nagios and Ganglia for monitoring Hadoop infrastructure.
• Experience in deploying and managing the Hadoop cluster using Cloudera Manager.
• Experience in tools like puppet to automate Hadoop installation, configuration and monitoring.
• Experience in writing Low Level Design (LLD) and High Level Design (HLD) documents.
• Experience in using NFS (Network File Systems) for backing up Name node metadata.
• Experience in PL/SQL programming including SQL queries using stored procedures and triggers in Oracle, SQL Server using TOAD and Query Manager.
• Experience in managing the cluster resources by implementing fair scheduler and capacity scheduler.
• Experience in developing Pig Latin scripts for data processing on HDFS.
• Installation, patching, upgrading, tuning, configuring and troubleshooting Linux based operating systems Red Hat and Centosand virtualization in a large set of servers.
• Experience with data warehousing tools Ab Initio GDE, Ab Initio EME, Data Profiler.
• Actively involved in successfully migration project without affecting Ab Initio applications data and processing with through testing.
• Excellent team player with good communication skills and effective time management.

TECHNICAL SKILLS

Hadoop Ecosystem Development: HDFS, Map - Reduce, Hive, Pig, Oozie, HBase, Sqoop, Flume, Zookeeper

Languages: Java, C, C++

Scripting: Shell Script, Perl

Web UI: HTML, XML, JSP, PHP

Database: Oracle 11g, MySQL, IBM DB2

Operating Systems: Linux, Unix, Windows

Development Tools: Eclipse, Putty, Tectia

ETL Tools: PDI ( Pentaho Data Integration / Kettle )

PROFESSIONAL EXPERIENCE

Hadoop Developer

Confidential, CA

Responsibilities:

• Developed Map Reduce jobs in java for data cleansing and preprocessing.
• Moving data from DB2, Oracle Exadata to HDFS and vice-versa using SQOOP.
• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis
• Worked with different file formats and compression techniques to determine standards
• Developed hive queries and UDFS to analyze/transform the data in HDFS.
• Developed hive scripts for implementing control tables logic in HDFS.
• Designed and Implemented Partitioning (Static, Dynamic), Buckets in HIVE.
• Developed Pig scripts and UDF’s as per the Business logic.
• Developed user defined functions in pig using Python.
• Analyzing/Transforming data with Hive and Pig.
• Developed Oozie workflows and they are scheduled through a scheduler on a monthly basis.
• Designed and developed read lock capability in HDFS.
• Implemented Hadoop Float equivalent to the DB2 Decimal.
• Using the data Integration tool Pentaho for designing ETL jobs in the process of building Data warehouses and Data Marts.
• Participation in Performance tuning in database side, transformations, and jobs level.
• Involved in End to End implementation of ETL logic.
• Effective coordination with offshore team and managed project deliverable on time.
• Worked on QA support activities, test data creation and Unit testing activities.
• Imported, exported file to the HDFS, Using NOSQL MongoDB, Hive, Hbase, Cassandra.

Environment: CDH, Hadoop, HDFS, MapReduce, Hive, Sqoop, NOSQL, Pig, XML, ETL, DB2, QA and Pentaho (PDI/Kettle)

Hadoop Developer

Confidential, OR

Responsibilities:

• Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to hdfs.
• Developed Java Map Reduce programs on log data to transform into structured way to find user location, age group, spending time.
• Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most purchased product on website.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports by our BI team.
• Dumped the data using Sqoop into HDFS for analyzing.
• Developed data pipeline using Pig and Hive from Teradata, DB2 data sources. These pipelines had customized UDF’S to extend the ETL functionality.
• Developed job flows in Oozie to automate the workflow for extraction of data from Teradata and DB2
• Developed data pipeline into DB2 containing the user purchasing data from Hadoop
• Implemented Partitioning, Dynamic Partitions, buckets in Hive and wrote map reduce programs to analyze and process the data

Environment: Hadoop, HDFS, MapReduce, Hive, Sqoop, Pig, DB2, Oracle, XML, Cloudera Manager

Hadoop and Java Consultant

Confidential, San Diego, CA

Responsibilities:

• Installation & configuration of a Hadoop cluster along with Hive.
• Developed Map Reduce application using Hadoop map reduce programming, a framework for processing.
• Large data sets in parallel across the Hadoop cluster for pre-processing.
• Developed the code for Importing and exporting data into HDFS and Hive using Sqoop.
• Responsible for writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL).
• Involved in defining job flows using Oozie for scheduling jobs to manage apache Hadoop jobs by directed.
• Acyclic graph (DAG) of actions with control flows.
• Developing Hive User Defined Functions in java, compiling them into jars and adding them to the HDFS and executing them with Hive Queries.
• Experienced in managing and reviewing Hadoop log files.
• Responsible to manage data coming from different sources.
• Assisted in monitoring the Hadoop cluster using Ganglia tool.
• Dealing with high volume of data in the cluster.
• Tested and reported defects in an Agile Methodology perspective.
• Consolidate all defects, report it to PM/Leads for prompt fixes by development teams and drive it to closure.
• Installed and configured Hadoop Cluster (3 Node Cluster) in fully distributed mode.
• Installed hadoop ecosystems(Hive, Pig, Sqoop, HBase, Oozie) on top of hadoop cluster
• Importing data from Oracle to HDFS & Hive for analytical purpose.
• Analyzing imported data in HDFS & Hive using HiveQL and custom Map Reduce programs in Java

Environment: Java, CDH, Hadoop, HDFS, Map Reduce, Hive and Sqoop

Associate JAVA Developer

Confidential

Responsibilities:

• Involved in the complete SDLC software development life cycle of the application from requirement analysis to testing.
• Developed the modules based on struts MVC Architecture.
• Developed The UI using JavaScript, JSP, HTML, and CSS for interactive cross browser functionality and complex user interface.
• Created Business Logic using Servlets, Session beans and deployed them on Weblogic server.
• Used MVC struts framework for application design.
• Created complex SQL Queries, PL/SQL Stored procedures, Functions for back end.
• Prepared the Functional, Design and Test case specifications.
• Involved in writing Stored Procedures in Oracle to do some database side validations.
• Performed unit testing, system testing and integration testing
• Developed Unit Test Cases. Used JUNIT for unit testing of the application.
• Provided Technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
• Resolved more priority defects as per the schedule.

Environment: Java, HTML, Java Script, CSS, Oracle, JDBC, Swing and Eclipse.