SUMMARY

• Over 8+ years of diverse IT experience on multiple projects in software requirement analysis, design and implementation including more than Three Years of excellent experience on Big Data and Hadoop Development.
• Experienced in supporting the Technical and System Architects to design the Application and System.
• Good experience in supporting Testing Team on drafting Test Plans.
• Health Care &Retail Industry Project sectors with multiprogramming language expertise like Java, Python and Scala.
• Working experience of various phases of SDLC such as Requirement Analysis, Design, Code Construction and Test
• Having experience as a Hadoop Developer/Administrator
• Created Talend custom components for the various use cases and worked on XML components, Data quality, Processing and Log & Error components.
• Having experience in the field of software development in creating solutions for Enterprise Applications and Web based Applications using JAVA & J2EE Technologies
• Built and Deployed Industrial scale Data Lake on on premise and Cloud platforms.
• Having Experience on HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, HBase, Impala, Zookeeper, Hue
• Good Knowledge on Hadoop EcoSystem components, Cluster architecture and monitoring the cluster.
• Having basic knowledge about real - time processing tools Storm.
• Experience in analyzing data using HiveQL, Pig Latin, and custom MapReduce programs in Java
• Having Experience on monitoring tools Ganglia, Cloudera Manager, and Ambari
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems, Teradata and vice versa
• Experience in providing Business Intelligence Solutions, data Extraction Transform and Loading (ETL) using SQL Server Integration Services (SSIS)
• Real time experience on Production deployment, code fixes.
• Basic knowledge on Machine Learning and Predictive Analysis.
• Worked on Talend data visualization tools
• Wrote custom UDF’s for extending Hive and Pig core functionality
• Worked on MySQL, Oracle 10g
• Development experience with IDE’s Eclipse and NetBeans
• Having experience with Schema design using like HBase, Cassandra
• Passionate about working on the most cutting-edge Big Data technologies.
• Ability to adapt to evolving technology, strong sense of responsibility and accomplishment
• Willing to update my knowledge and learn new skills according to business requirements.

TECHNICAL SKILLS

Hadoop Technologies: HDFS,MapReduce,Hive,Pig,Sqoop,Flume,Oozie,Zookeeper,Ambari,Hue,Strom,Talend.

Operating System: Windows, Unix, Linux

Languages: Java, J2EE, SQL, PL/SQL, Shell Script, COBOL, Python

Project Management / Tools: MS Project, MS Office, TFS, HP Quality Center Tool

ETL Technologies: Microsoft SQL Server Integration Services (SSIS 2008Microsoft SQL Server Integration Services (SSIS 2008)

Front - End: HTML, JSTL, DHTML, JavaScript, CSS, XML, XSL, XSLT

Databases: MySQL, Oracle 11g/10g/9i, SQL Server, DB2

NoSQL Databases: HBase, Cassandra

File System: HDFS

Reporting Tools: Jasper Reports, Tableau

IDE Tools: Eclipse, NetBeans

Application Server: Apache Tomcat, Web Logic

PROFESSIONAL EXPERIENCE

Confidential, Englewood, CO

Hadoop Developer

Responsibilities:

• Created multiple topics for producing and publishing the data to Consumer.
• Working on real time streaming data using Kafka streaming with Java API
• Using Storm topology for publishing data into Apache Accumulo to get the real time data.
• Loading data from different sources like Teradata and Oracle into Hadoop Distribution File System.
• Re-processing the data after loading into HDFS using Hive and Pig.
• Doing multiple layer journaling to remove all junk and unwanted data and storing into final data stage.
• Worked on new customized Hadoop framework DIFA for processing the data in Prep, Stage and Journal stages.
• Created different process and application for processing data through DIFA.
• Used Atscale as BI tool for moving data out of Hadoop cluster.
• Using Github for code promoting and sharing.
• Designing, building, installing, configuring and supportingHadoop frame work DIFA.
• Preparing Runbook and LLD after development for deployment purpose.
• Performing analysis of vast data stored and help ETL team in preparing reports through Tableau.
• Ingesting the Data into HDFS using Sqoop and writing pig scripts to transform it.
• Designed/Created HDFS data lake by drawing relationship between different sources of data from various systems
• Extensively worked on creating partitions, UDF’s, Performance tuning, Compression - related properties in Hive.
• Experienced in Talend Data Integration, Talend Platform Setup on Windows and Unix systems.
• Used UC4 for scheduling jobs for streaming data into HDFS.
• Created jobs in UC4 for pushing the data from landing zone to different stages which help data quality.
• Scheduled multiple UC4 jobs for processing the data.
• Maintaining security and data privacy by handling in Hadoop cluster.
• Created scalable and high-performance web services for data tracking.
• Created High-speed queries for instant output for Business users.
• Managing and deploying Apache Accumulo with HDFS and creating tables on it in Hive.
• Being a part of a POC effort to help build new Hadoop clusters with 310 nodes.
• Test prototypes and oversee handover to operational teams.
• Establishment of Data Lake using Hadoop platform is aiming to create an Enterprise grade ecosystem that enables Analytics and Global Reporting resulting into data driven Decision Making, Flexible Management, Financial Reporting, Data Governance across all business units.
• Monitoring the production issue with deployed jobs, verifying logs for accuracy and optimizing it.

Environment: DIFA, Apache Hadoop, Map Reduce, HDFS, Pig, Hive, Sqoop, UC4, Java, Storm, Kafka, Linux, Teradata, Oracle.

Confidential, Bellevue, WA

Hadoop Developer

Responsibilities:

• Monitoring the issue with the production jobs, verifying logs for accuracyand timeliness of execution.
• Reporting the production issues and suggesting theenhancementsor fix plan according.
• Developed theenhancementsand bug fixes for production issues.
• Co-ordinate with build and release team for production readiness of fixes and enhancements.
• Ingesting the data from different databases like SQL, Oracle into Hadoop Data Lake using Sqoop. Writing Pig and Hive scripts to process the HDFS data.
• Extensively worked on creating partitions, UDF’s, Performance tuning, Compression- related properties in Hive.
• Debugging and performance tuning of Pig and hive jobs.
• Gathering Business requirements and preparing detailed specifications that follows project guidelines which are required to develop programs.
• Writing several Shell scripts for cleansing and preparing the data before it is moved to Hadoop Data Lake.
• Performance tuning of Sqoop for optimizing the time taken to importing the data into Hadoop.
• Exporting data from Hadoop to different data sets.
• Transforming the data according to the data set before exporting the data from Hadoop.
• Writing pig scripts at transformation stage which take place before exporting data.
• Writing Sqoop queries for exporting the data in Teradata and Sql server databases.
• Created jar files to handle many limitations in the framework.
• Involved in loading data from Teradata database into HDFS using Sqoop queries.
• Involved in submitting and tracking MapReduce jobs using Job Tracker.
• Involved in creating Oozie workflow and Coordinator jobs to kick off the jobs on time for data availability.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations
• Implemented business logic by writing Pig UDFs in Java and used various UDFs from Piggybanks and other sources
• Implemented Hive Generic UDF's to implement business logic.
• Implemented test scripts to support test driven development and continuous integration.

Environment: DMF, Apache Hadoop, Map Reduce, HDFS, Pig, Hive, Sqoop, Oozie, Java, Python, Linux, Teradata.

Confidential, Bellevue, WA

Hadoop Developer

Responsibilities:

• Worked on writing transformer/mapping Map-Reduce pipelines using Java.
• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.
• Designed and implemented Incremental Imports into Hive tables.
• Worked in Loading and transforming large sets of structured, semi structured and unstructured data
• Involved in collecting, aggregating and moving data from servers to HDFS.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data
• Experienced in managing and reviewing the Hadoop log files. Implemented the workflows using Apache Oozie framework to automate tasks
• Worked on different file formats like Sequence files, XML files and Map files using MapReduce Programs.
• Enhanced existing module written in Pythonscripts.
• Involved in Unit testing and delivered Unit test plans and results documents using Junit and MRUnit.
• Developed scripts and automated data management from end to end and sync up between all the clusters.
• Involved in Setup and benchmark of Hadoop /HBase clusters for internal use.
• Created and maintained Technical documentation for launching Hadoop Clusters and for executing Pig Scripts.

Environment: Hadoop, HDFS, Map Reduce, Sqoop, Oozie, Pig, Hive, Flume, Nettezza, Python, Net prophet, LINUX, Java, Eclipse.

Confidential, Chicago, IL

Hadoop Developer

Responsibilities:

• Responsible for building scalable distributed data solutions using Hadoop.
• Written multiple Map Reduce programs in Java for Data Analysis.
• Wrote Map Reduce job using Pig Latin and Java API.
• Performed performance tuning and troubleshooting of Map Reduce jobs by analyzing and reviewing Hadoop log files.
• Developed Pig scripts for analyzing large data sets in the HDFS.
• Collected the logs from the physical machines and the OpenStack controller and integrated into HDFS using Flume.
• Designed and presented plan for POC on Impala.
• Experienced in migrating HiveQL into Impala to minimize query response time.
• Knowledge on handling Hive queries using Spark SQL that integrates with Spark environment.
• Implemented Avro and parquet data formats for Apache Hive computations to handle custom business requirements.
• Responsible for creating Hive tables, loading the structured data resulted from MapReduce jobs into the tables and writing Hive queries to further analyze the logs to identify issues and behavioral patterns.
• Worked on Sequence files, RC files, Map side joins, bucketing, partitioning for Hive performance enhancement and storage improvement.
• Implemented Daily Cron jobs that automate parallel tasks of loading the data into HDFS using Oozie coordinator jobs.
• Responsible for performing extensive data validation using Hive
• Sqoop jobs, PIG and Hive scripts were created for data ingestion from relational databases to compare with historical data.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations
• Used Visualization tools such as Power view for excel, Tableau for visualizing and generating reports.
• Exported data to Tableau and excel with Power view for presentation and refining
• Implemented business logic by writing Pig UDFs in Java and used various UDFs from Piggybanks and other sources
• Implemented Hive Generic UDF's to implement business logic.
• Implemented test scripts to support test driven development and continuous integration.
• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: Apache Hadoop, Map Reduce, HDFS, Pig, Hive, Sqoop, Flume, Oozie, Java, Linux, Maven, Teradata, Zookeeper, Tableau.

Confidential, Dearbon, MI

Hadoop Developer

Responsibilities:

• Involved in analysis of end user requirements and business rules based on given documentation and worked closely with tech leads and Business analysts in understanding the current system
• Assisted Business Analyst in documenting business requirements, technical specifications and implementation of various ETL standards in the mappings
• Analyzed the business requirements and involved in writing Test Plans and Test Cases.
• Involved in the design of Data-warehouse using Star-Schema methodology and converted data from various sources to oracle tables.
• Developed views, functions, procedures, and packages using PL/SQL & SQL to transform data between source staging area to target staging area.
• Worked on writing transformer/mapping Map-Reduce pipelines using Java.
• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.
• Designed and implemented Incremental Imports into Hive tables.
• Worked in Loading and transforming large sets of structured, semi structured and unstructured data
• Involved in collecting, aggregating and moving data from servers to HDFS using Apache Flume
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data
• Experienced in managing and reviewing the Hadoop log files.
• Migrated ETL jobs to Pig scripts do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Implemented the workflows using Apache Oozie framework to automate tasks
• Worked with Avro Data Serialization system to work with JSON data formats.
• Worked on different file formats like Sequence files, XML files and Map files using MapReduce Programs.
• Involved in Unit testing and delivered Unit test plans and results documents using Junit and MRUnit.
• Developed scripts and automated data management from end to end and sync up between all the clusters.
• Involved in Setup and benchmark of Hadoop /HBase clusters for internal use.
• Created and maintained Technical documentation for launching Hadoop Clusters and for executing Pig Scripts.
• Wrote SQL queries to perform Data Validation and Data Integrity testing.
• Created SQL*Loader scripts to load legacy data into Oracle staging tables.
• Created staging tables necessary to store validated customer data prior to loading data into customer interface tables.
• Coordinated with the Business Objects developers in building universes and developed various reports like Cross Tab, Master Detail and different charts including Line, Column, Area, and Pie Charts for analysis
• Developed UNIX shell scripts to run the batch jobs.
• Documented the process procedures and flow for the process

Environment: Hadoop, HDFS, Map Reduce, Sqoop, Oozie, Pig, Hive, Flume, LINUX, Java, Eclipse, Cassandra.

Confidential, Charlotte, NC

Hadoop Admin/ Developer

Responsibilities:

• Loaded customer data such as service installations, technical help line calls and interaction from the DISH Network web site in to HDFS using Flume.
• Implemented 26 node CDH4 Hadoop cluster on Red hat Linux using Cloudera Manager.
• Import data from Teradata, Oracle to HDFS using Sqoop.
• Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Handled importing of data from various data sources, performed transformations using Simple to complex MapReduce jobs, and loaded data into HBase.
• Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
• Setup Flume agent on the application servers to capture the log data.
• Setup flume agent on the web servers to capture the server logs.
• Used Avro sink and sources to move the file into HDFS.
• Analysis of the top errors occurring post an Integrated release using Hive and Pig.
• Used Sqoop to move the analyzed data to the MySql DB for report generation.
• Setup Amazon web services (AWS) to check whether Hadoop is a feasible solution or not.
• Setup Hadoop cluster using EC2 (Elastic MapReduce) on managed Hadoop Frame Work.
• Used Maven extensively for building MapReduce jar files and deployed it to Amazon Web Services (AWS) using EC2 virtual Servers in the cloud.
• Used S3 Bucket to store the jar’s, input datasets and used DynamoDB to store the processed output from the input data sets.
• Setup MongoDB to store the ever growing application config entries in JSON format.
• Use MongoDB as a contingency DB for the current Oracle clusters.
• Setup a MongoDB cluster with 12 shards.
• Setup Hadoop cluster using CDH5.0
• Used Sqoop to and mongoDump to move the data between MongoDB and HDFS.

Environment: CDH4/5, Cloudera Manager, MapReduce, HDFS, Hive, Pig, HBase, Flume, MySQL, Sqoop, Oozie, AWS, MongoDB.

Confidential

JAVA/J2EE Developer

Responsibilities:

• Used Hibernate ORM tool as persistence Layer - using the database and configuration data to provide persistence services (and persistent objects) to the application.
• Implemented Oracle Advanced Queuing using JMS and Message driven beans.
• Responsible for developing DAO layer using Spring MVC and configuration XML's for Hibernate and to also manage CRUD operations (insert, update, and delete).
• Implemented Dependency injection of spring frame work.
• Developed and implemented the DAO and service classes.
• Developed reusable services using BPEL to transfer data.
• Participated in Analysis, interface design and development of JSP.
• Configured log4j to enable/disable logging in application.
• Developed Rich user interface using HTML, JSP, AJAX, JSTL, JavaScript, JQuery and CSS.
• Implemented PL/SQL queries, Procedures to perform data base operations.
• Wrote UNIX Shell scripts and used UNIX environment to deploy the EAR and read the logs.
• Implemented Log4j for logging purpose in the application.
• Involved in code deployment activities for different environments.
• Implemented agile development methodology.
• Used IBM As400 as back end application for storing the data and maintaining it.
• DB2 was the database and UI was Siebel CRM.
• Java was the mid-layer and we have written many web services and xml for input and output data.
• Responsible for many help insurance products which developed in mobile portal using Java Web services.
• Involved in IBM As400 back end application and understand and programmed it using COBOL
• Involved in month end, quarter-end and Year-end batch processing event and provided multi-level support for both Web services and IBM As400 application for the smooth run.

Environment: Java, Spring, Hibernate, JMS, EJB, Web logic Server, SQL Developer, Maven, XML, CSS, JavaScript, JSON, IBM AS400, COBOL.

Confidential

Java Developer

Responsibilities:

• Responsible and active in the analysis, design, implementation and deployment of full Software Development Lifecycle (SDLC) of the project.
• Designed and developed user interface using JSP, HTML and JavaScript.
• Developed Struts action classes, action forms and performed action mapping using Struts framework and performed data validation in form beans and action classes.
• Extensively used Struts framework as the controller to handle subsequent client requests and invoke the model based upon user requests.
• Defined the search criteria and pulled out the record of the customer from the database. Make the required changes and save the updated record back to the database.
• Validated the fields of user registration screen and login screen by writing JavaScript validations.
• Developed build and deployment scripts using Apache ANT to customize WAR and EAR files.
• Used DAO and JDBC for database access.
• Developed stored procedures and triggers using PL/SQL in order to calculate and update the tables to implement business logic.
• Design and develop XML processing components for dynamic menus on the application.
• Involved in post-production support and maintenance of the application.

Environment: Oracle 11g, Java 1.5, Struts, Servlets, HTML, XML, SQL, J2EE, JUnit, Tomcat 6.