SUMMARY:

• 8 years of experience in software development life cycle design, development and support of systems application architecture.
• 4+ Years of Big Data Hadoop Ecosystems experience in ingestion, storage, querying, processing and analysis of big data.
• Experience in working with Hadoop clusters using Cloudera (CDH3) distributions.
• Good knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and MapReduce concepts.
• In depth knowledge and hands on experience in installing, configuring, monitoring and integration of Hadoop ecosystem components Hadoop (HDFS, MapReduce, Pig, Hive, Scoop, Flume, Hbase, Oozie).
• Good experience in Cloudera, Hortonworks & Apache Hadoop distributions.
• Extensively worked on MRV1 and MRV2 Hadoop architectures.
• Designing and creating Hive external tables using shared meta - store instead of derby with partitioning, dynamic partitioning and buckets.
• Exposure on Spark, Kafka and Scala Programming.
• Expertise in database design, creation and management of schemas, writing Stored Procedures, Functions, DDL, DML, SQL queries & Modeling.
• Proficient in using RDMS concepts with Oracle, SQL Server and MySQL.
• Strong experience in database design, writing complex SQL Queries and Stored Procedures.
• Familiar with Java virtual machine (JVM) and multi-threaded processing.
• Extensively used ETL methodology for supporting Data Extraction, transformations and load.
• Experience in analyzing data using HiveQL, Pig Latin, HBase and custom Map Reduce programs in Java and Python.
• Experience in writing with Map Reduce programs using Apache Hadoop for working with Big Data.
• Experience on NoSQL databases including HBase, Cassandra.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Extending Hive and Pig core functionality by writing customUDFs.
• Experience with Eclipse/ RSA.
• Knowledge of job workflow scheduling and monitoring tools like oozie and Zookeeper, of NoSQL databases such as HBase, Cassandra, and of administrative tasks such as installing Hadoop, Commissioning and decommissioning, and its ecosystem components such as Flume, Oozie, Hive and Pig.
• Excellent teamwork and communication skills, research-minded, technically competent and result-oriented with problem solving abilities.

TECHNICAL SKILLS:

Programming Languages: Core Java, J2EE, Scala, XML, DB2, CICS, SQL, PL/SQL, HiveQL, Pig Latin

Hadoop Eco System: HDFS YARN, MapReduce, Pig, Hive, Sqoop, Flume, zookeeper, Oozie, Apache Kafka and Kerberos

Hadoop Distributions: Cloudera, Hortonworks

Operating Systems: Linux, Unix, MVS, Windows

NonRelational Databases: MongoDB, Cassandra, HBase

Relational Databases: DB2 V 9.0, MySQL, Microsoft SQL Server

Scripting Languages: Python, Shell Scripting

Application/Web Servers: Apache Tomcat, JBoss, Websphere, MQ Series, Data power, Web services

Tools: Endeavour, Data Power XI150 Appliance, SoapUI, Jmeter, XML Harness, Labs testing tool

QA Tools: Quality Center

IDE: Intellij, Eclipse, Net Beans

PROFESSIONAL EXPERIENCE:

Senior Hadoop Developer

Confidential, IL

Responsibilities:

• Developed data pipeline using Kafka, Flume, Sqoop, Pig and Spark to ingest customer behavioural data and financial histories into HDFS for analysis.
• Developed Turbocow framework for filtering data from raw to enrich with business rules.
• Framework contains different actions like lookups, replace null with zero and also simple copy for further uses.
• Involved in sqoop that to bring data from teradata into hdfs to do lookups with dimensional data.
• Used Hive to analyse the partitioned and bucketed data and compute various metrics for reporting. Involved in developing Hive UDFs for the needed functionality that is not out of the box available from Apache Hive.
• Responsible for developing data pipeline using flume, sqoop and pig to extract the data from weblogs and store in HDFS Designed and implemented various metrics that can statistically signify the success of the experiment.
• Used Intellij to build the application.
• Hands on experience in hadoop cluster 5.5 and 5.4.
• Involved in extracting customer’s Big data from various data sources into Hadoop HDFS. This included data from mainframes, databases and also logs data from servers.
• Responsible and managed entire Hive warehouse.
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java MapReduce, Hive and Sqoop as well as system specific jobs.
• Created Hive tables, loaded data and wrote Hive queries that run within the map.
• Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.
• Involved in streaming data to ingest from kafka cluster on json format and also teradata tables imported on periodic basis as batch jobs.
• Taken care of flume agents to ingest the ALS event data stream from kafka to hdfs as compressed for batch processing with spark and also streams raw data to spark streaming.
• Used sqoop to import tables and also data from teradata to hdfs periodically.
• Implemented automatic failover Zookeeper and zookeeper failover controller.
• Worked on impala performance tuning with different workloads and file formats.

Environment: Hadoop, Spark, Yarn, Hive, Pig, HBase, Oozie, Sqoop, Flume, Kafka, Cloudera 5.5, Oozie, Impala, Tableau, Eclipse, Intellij

Senior Hadoop Developer

Confidential,WI

Responsibilities:

• Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioural data and financial histories into HDFS for analysis.
• Involved in writing MapReduce jobs.
• Involved in SQOOP, HDFS Put or Copy from Local to ingest data.
• Used Pig to do transformations, event joins, filter bot traffic and some pre-aggregations before storing the data onto HDFS.
• Involved in developing Pig UDFs for the needed functionality that is not out of the box available from Apache Pig.
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Involved in developing Hive DDLs to create, alter and drop Hive TABLES.
• Involved in developing Hive UDFs for the needed functionality that is not out of the box available from Apache Hive.
• Involved in using HCATALOG to access Hive table metadata from Map Reduce or Pig code.
• Computed various metrics using Java MapReduce to calculate metrics that define user experience, revenue etc.
• Responsible for developing data pipeline using flume, sqoop and pig to extract the data from weblogs and store in HDFS Designed and implemented various metrics that can statistically signify the success of the experiment.
• Used Eclipse and ant to build the application.
• Involved in using SQOOP for importing and exporting data into HDFS and Hive.
• Involved in processing ingested raw data using MapReduce, Apache Pig and Hive.
• Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
• Involved in pivot the HDFS data from Rows to Columns and Columns to Rows.
• Involved in emitting processed data from Hadoop to relational databases or external file systems using SQOOP, HDFS GET or CopyToLocal.
• Involved in developing Shell scripts to orchestrate execution of all other scripts (Pig, Hive, MapReduce) and move the data files within and outside of HDFS.

Environment: Hadoop, MapReduce, Yarn, Hive, Pig, HBase, Oozie, Sqoop, Flume, Oracle 11g, Core Java Cloudera HDFS, Eclipse

Senior Hadoop Developer

Confidential,MI

Responsibilities:

• Development and ETL Design in Hadoop
• Developed Mapreduce Input format to read visa specific data format.
• Performance tuning of Hive Queries written by data analysts.
• Developing Hive queries and udf’s as per requirement.
• Involved in extracting customer’s Big data from various data sources into Hadoop HDFS. This included data from mainframes, databases and also logs data from servers.
• Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers.
• Developed MapReduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
• The Hive tables created as per requirement were managed or external tables defined with appropriate static and dynamic partitions, intended for efficiency.
• Implemented Partitioning, Bucketing in Hive for better organization of the data.
• Developed UDFs in Pig and Hive
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java MapReduce, Hive and Sqoop as well as system specific jobs.
• Worked with BI teams in generating the reports on Tableau
• Installed and configured various components of Hadoop ecosystem and maintained their integrity.
• Implemented Fair Scheduler on the job tracker to allocate the fair amount of resources to small jobs.
• Upgraded Hadoop Versions using automation tools.
• Deployed high availability on the Hadoop cluster quorum journal nodes.
• Implemented automatic failover Zookeeper and zookeeper failover controller.
• Migrating existing Ab initio transformation logic to Hadoop Pig Latin and Udf's.
• Used Sqoop to efficiently Transfer data from DB2 to HDFS, Oracle Exadata to HDFS
• Designing ETL flow for several newly on boarding Hadoop Applications.
• Worked on implementing Hadoop Streaming, Python mapreduce for visa analytics.
• Implemented NLine Input Format to split a single file into multiple small files.
• Designed and Developed oozie workflows, integration with Hcatalog/Pig.
• Documented ETL Best Practices to be implemented with Hadoop
• Managing and Supporting Infrastructure.
• Monitoring and Debugging Hadoop jobs/Applications running in production.
• Worked on Hadoop Cloudera upgrade from CDH4.x to CDH5.x.
• Worked on Providing User support and application support on Hadoop Infrastructure.
• Reviewing ETL application use cases before on boarding to Hadoop.
• Worked on Evaluating, comparing different tools for test data management with Hadoop.
• Helping testing team to get up to speed on Hadoop Application testing.
• Worked on Integration of Hiveserver2 with Tableau.
• Worked on impala performance tuning with different workloads and file formats.
• Worked on Installing 20 node UAT Hadoop cluster.
• Worked on POC of Talend integration with hadoop

Environment: Hadoop, HDFS, MapReduce, Hive, Sqoop, Pig, DB2, Oracle, XML, Cloudera Manager.

Hadoop Developer

Confidential,NY

Responsibilities:

• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data.
• Developed MapReduce programs to parse the raw data and store the refined data in tables.
• Designed and Modified Database tables and used HBASE Queries to insert and fetch data from tables.
• Involved in moving all log files generated from various sources to HDFS for further processing through Flume.
• Involved in loading and transforming large sets of structured, semi structured and unstructured data from relational databases into HDFS using Sqoop imports.
• Responsible for analyzing and cleansing raw data by performing Hive queries and running Pig scripts on data.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Created Hive tables, loaded data and wrote Hive queries that run within the map.
• Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
• Involved in fetching brands data from social media applications like Facebook, twitter.
• Developed and updated social media analytics dashboards on regular basis.
• Performed data mining investigations to find new insights related to customers.
• Involved in forecast based on the present results and insights derived from data analysis.
• Create a complete processing engine, based on Cloudera's distribution, enhanced to performance.
• Manage and review Hadoop log files.
• Developed and generated insights based on brand conversations, which in turn helpful for effectively driving brand awareness, engagement and traffic to social media pages.
• Involved in identification of topics and trends and building context around that brand.
• Involved in the identifying, analyzing defects, questionable function error and inconsistencies in output.

Environment: Java, HBase, Hadoop, HDFS, Map Reduce, Hive, Sqoop, Flume, Oozie, Zookeeper, MySQL, and eclipse.

Hadoop Developer

Confidential,IL

Responsibilities:

• Responsible for designing and implementing ETL process to load data from different sources, perform data mining and analyze data using visualization/reporting tools to leverage the performance of OpenStack.
• Collected the logs from the physical machines and the OpenStack controller and integrated into HDFS using Flume.
• Partitioned the collected logs by date/timestamps and host names.
• Worked on the HBase for data optimization.
• Designing and creating Hive external tables using shared meta-store instead of derby with partitioning, dynamic partitioning and buckets.
• Hands on experience in loading data from UNIX file system to HDFS.
• Experienced on loading and transforming of large sets of structured, semi structured and unstructured data from HBase through Sqoop and placed in HDFS for further processing.
• Responsible and managed entire Hive warehouse.
• Developed custom MapReduce programs to extract the required data from the logs.
• Performed performance tuning and troubleshooting of MapReduce jobs by analyzing and reviewing Hadoop log files.
• Responsible for creating Hive tables, loading the structured data resulted from MapReduce jobs into the tables and writing hive queries to further analyze the logs to identify issues and behavioural patterns.
• Imported data frequently from MySQL to HDFS using Sqoop.
• Supported operations team in Hadoop cluster maintenance activities including commissioning and decommissioning nodes and upgrades.
• Used Tableau for visualizing and to generate reports.
• Tested impala integration.

Environment: Hadoop, MapReduce, Hive, HBase, Sqoop, kafka, Impala, Tableau

Java Developer

Confidential

Responsibilities:

• Develop GUI related changes using JSP, HTML and client validations using Java script.
• Designed and developed front end using HTML, JSP and Servlets
• Implemented client side validation using JavaScript
• Used Hibernate in persistence layer of the application
• Created UML class diagrams that depict the code’s design and its compliance with the functional requirements.
• Developed user interface using JSP to simplify the complexities of the application.
• Developed the Web Interface using Servlets, Java Server Pages, HTML and CSS.
• Extensively used the JDBC Prepared Statement to embed the SQL queries into the java code. Implemented the DAO pattern.
• Developed business logic using Stateless session beans for calculating asset depreciation on Straight line and written down value approaches.
• Involved coding SQL Queries, Stored Procedures and Triggers.
• Created java classes to communicate with database using JDBC.

Environment: Java 1.4, Servlets, JSP, EJB, J2EE 1.4, XML, XSLT, Java Script, SQL, PL/SQL, Ms Visio, Eclipse, JDBC, Win CVS, Windows XP.

Java Developer

Confidential

Responsibilities:

• Responsible for gathering and analyzing requirements and converting them into technical specifications.
• Used Rational Rose for creating sequence and class diagrams.
• Developed presentation layer using JSP, Java, HTML and JavaScript.
• Participated in the design and development of database schema and Entity-Relationship diagrams of the backend Oracle database tables for the application.
• Designed and Developed Stored Procedures, Triggers in Oracle to cater the needs for the entire application. Developed complex SQL queries for extracting data from the database.
• Designed and built SOAP web service interfaces implemented in Java.
• Used Apache Ant for the build process.

Environment: Java, JDK 1.5, Servlets, Ajax, Oracle 10g, Eclipse, Apache Ant, Web Services (SOAP), Apache Axis, Apache Ant, Web Logic Server, JavaScript, HTML, CSS, XML.