SUMMARY:

• 8+ years of professional experience in IT industry, with 4 years of experience in Hadoop ecosystem's implementation, maintenance, and Big Data analysis operations.
• Excellent understanding of Hadoop architecture and underlying framework including storage management.
• Excellent understanding of Hadoop Gen 2 and Gen 1 concepts.
• Experience in using various Hadoop infrastructures such as MapReduce, Pig, Hive, ZooKeeper, HBase, Sqoop and Oozie for data storage and analysis.
• Experience with Oozie Scheduler in setting up workflow jobs with Map/Reduce and Pig jobs.
• Knowledge of architecture and functionality of NOSQL DB like HBase and MongoDB. Experience in troubleshooting errors in HBase Shell/API, Pig, Hive and MapReduce.
• Experience in importing and exporting data between HDFS and Relational Database Management systems using Sqoop.
• Experience with AutoSys as a scheduler to run jobs on the Hadoop cluster.
• Experience in Streaming the Data to HDFS using Flume.
• Experience in working with Hadoop clusters using Cloudera (CDH - 5.9) distributions.
• Hands on experience in using Map reduce programming model for Batch processing of data stored in HDFS
• Expertise in writing ETL Jobs for analyzing data using Pig.
• Experience in NoSQL Column-Oriented Databases like HBase and its Integration with Hadoop cluster
• Experience in Build and Release tools for deployment on HDFS. Like RLM and RTC .
• Experience in security enhancement using Apache SENTRY for enforcing fine grained role based authorization.
• Mainframe Application Developer experience in System study, Requirements gathering, Analysis, Design, Development, Testing, Troubleshooting and Implementation of full life cycle mainframe applications.
• Excellent time management skills with proven ability to work accurately and quickly to prioritize, coordinate and consolidate tasks, whilst simultaneously managing the diverse range of functions.
• Diligent team builder with ability to objectively evaluate results; also possesses excellent written, oral and interpersonal communication skills.
• Strong logical and analytical skills leading to problem resolutions within timelines.
• Committed and dedicated for continuous improvement. Quick learner. Excellent team player with strong team leading capabilities.

TECHNICAL SKILLS:

Languages: Pig Latin, HiveQL, NOSQL, Shell script, Map Reduce (Core Java),Python and COBOL

Big Data Tools: SQOOP, ZOO Keeper, Apache OOZIE, HDFS, HUE, SPARK, SENTRY and its components

TOOLS: PIG, HIVE Query tool,TALEND,RELEASE TEAM CONCERT(RTC-IBM),RELEASE LIFECYCLE MANAGEMENT(RLM-BMC),AutoSys

FRAMEWORKS: Hadoop HDFS (Apache, Cloudera)

Other Languages: Core Java, Excel VBA Automation

Databases: DB2, HIVE, HBASE and VSAM,MONGODB

Operating Systems: OS/390, Windows and Unix

Build Tools: Ant, Maven,RLM,RTC

CI Tool: Jenkins

PROFESSIONAL EXPERIENCE:

Confidential, Warren, NJ

Sr. Hadoop Developer

Responsibilities:

• Developed and executed custom HIVE queries. Used Hadoop scripts for HDFS (Hadoop File System) data loading and manipulation.
• Performed Hive test queries on local sample files and HDFS files.
• Written Hive queries for data analysis to meet the business requirements. Created Hive tables, views and worked on them using Hive QL.
• Create Oozie workflows and super workflows to automate the data loading process
• Used AutoSys extensively as a scheduler to run the jobs on HIVE for data loading and data manipulation
• Assisted in loading large sets of data (Structure, Semi Structured, and Unstructured) to HDFS using sqoop.
• Developed the application on Eclipse.
• Worked on a POC to test Apache Sentry and successfully implemented it on the HDFS cluster
• Ingested data from various RDBMS databases using tools like TALEND onto HDFS
• Extensively used Pig for data cleaning and optimization.
• Developed Hive queries to analyze data and generate results.
• Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization and user report generation.
• Monitoring the clustered data sets using Cloudera Data Navigator and Cloudera Manager
• Extensively used Rational Team Concert (RTC) for source control and as a build management tool for development
• Deployed code by mitigating risk and automating using Release Lifecycle Management Tool ()RLM-BMC) .
• Analyzed user request patterns and implemented various performance optimization measures including implementing partitions and buckets in HiveQL.
• Created and maintained Technical documentation for launching HADOOP Clusters and for executing Hive queries and Pig Scripts
• Worked with a data from various regions and have complete understanding of PII fields and masking of the same
• Wrote MapReduce jobs to generate reports for the number of activities created on a particular day, during a dumped from the multiple sources and the output was written back to HDFS. Reviewed the HDFS usage and system design for future scalability and fault-tolerance.

Environment: Java, Linux, Apache Hadoop Framework, HDFS, Map Reduce V1, HUE, YARN (MR2.X), PIG, HIVE, HBASE, Zoo Keeper, Oozie, SQOOP & Flume,AutoSys.

Confidential, El Segundo, CA

Sr. Hadoop Developer

Responsibilities:

• Developed and executed custom MapReduce programs, PigLatin scripts and HQL queries.
• Used Hadoop scripts for HDFS (Hadoop File System) data loading and manipulation. Performed Hive test queries on local sample files and HDFS files.
• Written Hive queries for data analysis to meet the business requirements. Created Hive tables and worked on them using Hive QL.
• Assisted in loading large sets of data (Structure, Semi Structured, and Unstructured) to HDFS.
• Developed the application on Eclipse.
• Extensively used Pig for data cleaning and optimization. Developed Hive queries to analyze data and generate results.
• Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization and user report generation.
• Worked in an onsite-offshore environment.
• Completely responsible for creating data model for storing & processing data and for generating & reporting alerts. This model is being implemented as standard across all regions as a global solution.
• Involved in discussions and guiding other region teams on Confidential Big data platform and AML cards data model and strategy.
• Responsible for technical design based on data dictionary (Business requirement). Responsible for providing technical solutions and work arounds.
• Migrating the needed data from Data warehouse and Product processors into HDFS using Talend and Sqoop and importing various formats of flat files in to HDFS.
• Switched from Sqoop to Flume for all batch & real time data ingestion.
• Using Spark Streaming to bring all credit card transactions in the Hadoop environment. Involved in design of overall Confidential Group Big data architecture.
• Involved in discussion with source systems for issues related to DQ in data.
• Comfortable with Scala functional programming idioms and very familiar with Iterate / Enumerate streaming patterns. Almost entire DQ and end to end reconciliation is done in Scala & Spark.
• Implemented partitioning, dynamic partitions, indexing and bucketing in HIVE.
• Analyzed user request patterns and implemented various performance optimization measures including implementing partitions and buckets in HiveQL.
• Created and maintained Technical documentation for launching HADOOP Clusters and for executing Hive queries and Pig Scripts
• Wrote MapReduce jobs to generate reports for the number of activities created on a particular day, during a dumped from the multiple sources and the output was written back to HDFS. Reviewed the HDFS usage and system design for future scalability and fault-tolerance.

Environment: Java, Linux, Apache Hadoop Framework, HDFS, Map Reduce V1, YARN (MR2.X), PIG, HIVE, HBASE, Zoo Keeper, Oozie, SQOOP & Flume.

Confidential, San Ramon, CA

Hadoop Developer

Responsibilities:

• Coordinated with business customers to gather business requirements. Importing and exporting data into HDFS from database and vice versa using SQOOP.
• Responsible to manage data coming from different sources.
• Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, HBase database and SQOOP.
• Load and transform large sets of structured and semi structured data.
• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
• Analyzed data using Hadoop components Hive and Pig.
• Involved in running Hadoop streaming jobs to process terabytes of data. Gained experience in managing and reviewing Hadoop log files.
• Involved in writing Hive queries for data analysis to meet the business requirements.
• Worked on streaming the analyzed data to the existing relational databases using SQOOP for making it available for visualization and report generation by the BI team.
• Involved in creating the workflow to run multiple Hive and Pig jobs, which run independently with time and data availability.
• Developed PIG Latin scripts for the analysis of semi structured data.
• Imported data using SQOOP to load data from MySQL to HDFS on regular basis.
• Worked hands on with ETL process.
• Prepared Project maintenance, Test summary, Test result, Test case and Go-Live plan documents for project release

Environment: Hadoop, HDFS, MapReduce, Pig, Hive, Sqoop, HBase and Java Eclipse.

Confidential, Boston, MA

Hadoop Developer

Responsibilities:

• Coordinated with business customers to gather business requirements. Importing and exporting data into HDFS from database and vice versa using SQOOP.
• Responsible to manage data coming from different sources.
• Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, HBase database and SQOOP.
• Load and transform large sets of structured and semi structured data.
• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
• Analyzed data using Hadoop components Hive and Pig. Involved in running Hadoop streaming jobs to process terabytes of data.
• Gained experience in managing and reviewing Hadoop log files.
• Involved in writing Hive queries for data analysis to meet the business requirements.
• Worked on streaming the analyzed data to the existing relational databases using SQOOP for making it available for visualization and report generation by the BI team.
• Involved in creating the workflow to run multiple Hive and Pig jobs, which run independently with time and data availability.
• Developed PIG Latin scripts for the analysis of semi structured data.
• Imported data using SQOOP to load data from MySQL to HDFS on regular basis.
• Worked hands on with ETL process.
• Prepared Project maintenance, Test summary, Test result, Test case and Go-Live plan documents for project release

Environment: Hadoop, MapReduce, HDFS, Hive, Java (jdk1.6), Hadoop distribution of Horton Works, Cloudera, MapR, DataStax, Flat files, Oracle 11g/10g, PL/SQL, SQL*PLUS, Toad 9.6, Windows NT, UNIX Shell Scripting, Autosys r11.0.

Confidential

J2EE Developer

Responsibilities:

• Experience with creating and revising class diagrams, sequence diagrams, and entity relationship diagrams.
• Read Fleet transactions from DB2 databases in the remote database and created the transaction information into delimited files and also updated Oracle database in local system.
• Used XML for interchange of data between various layers of the application - Beans, User interface (JSP, HTML), Servlets, and EJBs.
• Responsible for checking all the front-end validations using JavaScript
• Worked in web service SOAP response using payload XML.
• Wrote XSLT code, which displays XML data from customer insight. Also created portlets using portal builder.
• Using DOM API developed payload and envelop parser to parse XML data.
• Written ANT scripts to copy compile, deploy the java classes (EAR) in to WebLogic.
• Written stored procedure (PL/SQL) and used Complex Queries with Joins in the conversion process.
• Tuned database and SQL statements and schemas for optimal performance. Expertise in SQL queries for the cross verification of data.

Environment: Java, JSP, Servlets, JDBC, ANT, UML, Eclipse, WebLogic 8.1, DB2, Oracle8i, CVS, XML, XSLT, XSD, UNIX, Korn shell scripting, PERL, MVS, JCL, ISPF, VSAM Files, OS/390