SUMMARY:

• 9years of comprehensive IT experience in Big Data & ETL domain with tools like Apache Hadoop,Apache Spark and other open source tools/technologies.
• Hadoop distributions like Hortonworks Data Platform (HDP), Cloudera and IBM Big Insights experinance.
• Hands on experience in development of Big Data projects using Hadoop, Hive,Sqopp,Oozie, PIG, Flume,kafka,Apache Nifiand MapReduce open source tools/technologies.
• Hands on experience in installing, configuring Hadoop using ecosystem components like Apache Hadoop MapReduce, HDFS, Hbase, Zoo Keeper, Oozie, Hive, HDP, Cassandra, Sqoop, PIG, Flume,Spark.
• Good Experience in importing and exporting data between HDFS and Relational Database Management systems using Sqoop.
• Developed pig latin scripts.
• Good knowledge of NoSQL databases such as Hbase and Mangodb.
• Strong knowledge of Spark for handling large data processing in streaming process along with Scala/Python.
• Strong knowledge of ETL tool Data Stage and reporting tools like Cognos, Qlickview and Tableau for handling DWH applications.
• Experienced in loading the huge data from local file system and HDFS to Hive and writing complex queries to load data into internal tables.
• Experience using different file formats like Text Files, Sequence Files,AVRO Files and ORC Files.
• Worked on JSON files processing.
• Experienced in loading data to Hive partitions and bucketing.
• Background with traditional databases such as Oracle, Teradata, Netezza, ETL tools / processes and data warehousing architectures.
• Experience in working with Iterative Agile/ Scrum Methodologies, Waterfall methodology.
• Extensive Knowledge on source code version control using SVN.
• Strong development skills in Object Oriented and functional programming.
• Ability to manage and deliver results on multiple tasks by effectively managing time and priority constrains.
• Able to work within a team environment as well as independently.
• Proficient in collaborating with Team members, Scoping and planning project estimates, Requirement Analysis, Design and Architecture review, Development and Unit Testing.
• Have a Strong customer focus and excellent interpersonal, verbal, and written communication skills.

TECHNICAL SKILLS:

Bigdata Technologies: Hadoop, Map Reduce, Hive,Pig, Zookeeper, Sqoop,Oozie, Flume, Scala, IMPALA, HBASE

Big Data Frameworks: HDFS, YARN, Spark

Hadoop Distributions: Cloudera(CDH3,CDH4,CDH5), Hortonworks (2.2,2.4,2.6), Amazon EMR

Programming Languages: C, C++, Java, Python

Scripting Languages: XML, HTML, Python, Shell Scripting, AngularJS

Web Services: SOAP, Restful

Databases: RDBMS, HBase, MySQL, Oracle,Netezza, SAP HANA,EPICORE

Tools: Eclipse, Net Beans, Tableau, Visual Studio, MS - Project, MS-Office

Platforms: Windows, Linux, Unix

Reporting Tools: Tableau, Jasper Reports, SAS Reporting, R

Methodologies: Agile, SDLC,Waterfall

ETL Tools: Oracle Data Integrator, Informatica

PROFESSIONAL EXPERIENCE:

Confidential, New Jersey

Hadoop Developer

Responsibilities:

• Responsible for complete end-to-end project delivery.
• Develop code to pick email source feed files to Hadoop file system using python code.
• Pre-processing all input files with python code to make ready to load into table.
• Design meta data for all sources like Amazon, Walmart,Peapod,WalmartPickup,Target.com,Jet.com,Freshdirect,Samsclub,Samsclub Pickup, Boxed.com, Quidsi, Costco and Safeway Albertson.
• Design data model from stage table to final tables for all sources .
• Create Hive External tables to access daily data from sources.
• Create Hive Internal partitioned and Bucketing tables to load data.
• Develop a Hive UDF to make common date format for all sources dates.
• Create Oozie workflow for all actions to execute all actions in sequence.
• Refresh Tableau dashboards with fresh data after each load.

Environment: Hortonworks Data Platform (HDP) 2.4, Tableau, Hive, Oozie, Python

Big Data Developer

Responsibilities:

• Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters.
• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data.
• Responsible for ingesting structured data residing on our traditional back-end databases on to Hadoop and HIVE using SQOOP.
• Experience writing Spark SQL queries using Data frames.
• Worked on implementing applications on Spark frameworks using Scala.
• Good knowledge on converting Hive/SQL queries to Spark transformations and actions using SparkSQL in Scala.
• Experience in performance tuning of Hive queries.
• Worked on implementing analytics on structured data using Ad-hoc queries with Hive.
• Involved in Installing, Configuring Hadoop Eco System and Cloudera Manager using CDH4 Distribution.
• Developed several advanced MapReduce programs to process data files received.
• Hands on experience in writing Linux/Unix Shell scripting.
• Worked extensively with Hive DDLs and Hive Query language (HQLs).
• Developed business logic using Hive UDF's to perform ad-hoc queries on structured data.
• Hands on experience in working with IMPALA.
• Worked on writing Hive queries for data analysis to meet the business requirements.
• Worked extensively with defining, partitioning, bucketing, compressing Hive tables to meet according business requirements.
• Used Pig to write transformations, event joins, filtering and some pre-aggregations before storing the data onto HDFS by writing and executing pig scripts.
• Configured Oozie work flows to automate data flow, preprocess and cleaning tasks using Hadoop Actions.
• Developed a Pig program for loading and filtering the streaming data into HDFS using Flume.
• Responsible for managing data coming from disparate data sources.
• Experience with configuration of Hadoop Ecosystem components: Map Reduce, Hive, HBase, Pig, Sqoop, Oozie, Zookeeper, Flume, Storm, Spark, Yarn, Tez.
• Worked on transporting, and processing real time event streaming using Kafka and Spark Streaming.
• Implemented test scripts to support test driven development and continuous integration.

Environment:: Hadoop, Hive, Map Reduce, HDFS, Pig, Sqoop, Java, Eclipse, Oracle 10g, PL/SQL, Linux, flat files, Microsoft SQL Server, Spark,Scala, CRON Tab,IMPALA, Cloudera, Cloudera Manager, Netezza, Tableau, Agile.

Hadoop Developer

Responsibilities:

• Converting the existing relational database model to Hadoop ecosystem.
• Generate datasets and load to HADOOP Ecosystem.
• Worked with Linux systems and RDBMS database on a regular basis to ingest data using Sqoop.
• Developed Schedulers that communicated with the Cloud based services (AWS) to retrieve the data.
• Worked with Spark to create structured data from the pool of unstructured data received.
• Implemented intermediate functionalities like events or records count from the flume sinks or Kafka topics by writing Spark programs in java and Scala.
• Managed and reviewed Hadoop and HBase log files.
• Developed multiple Kafka Producers and Consumers from scratch to as per the software requirement specifications.
• Also created, altered and deleted topics (Kafka Queues) as and when required with varying configurations involving replication factors, partitions and TTL.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• Involved in review of functional and non-functional requirements.
• Implemented Frameworksusing Javaand python to automate the ingestion flow.
• Responsible to manage data coming from different sources.
• Loaded the CDRs from relational DB using Sqoopand other sources to Hadoop cluster by using Flume.
• Experience in processing large volume of data and skills in parallel execution of process using Talend functionality.
• Involved in loading data from UNIX file system and FTP to HDFS.
• Designed and implemented HIVE queries and functions for evaluation, filtering, loading and storing of data.
• Creating Hive tables and working on them using HiveQL.
• Developed data pipeline using Kafka and Storm to store data into HDFS.
• Implemented Kafka Custom encoders to store custom objects, build Storm topologies as part of preprocessing.
• Created reporting views in Impala using Sentry policy files.
• Developed Hive queries to analyze the output data.
• Had to do the Cluster co-ordination services through ZooKeeper.
• Collected the logs data from web servers and stored in to HDFS using Flume.
• Used HIVE to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
• Design and implement Spark jobs to support distributed data processing.
• Supported the existing MapReduce Programs those are running on the cluster.
• Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
• Wrote Java code to format XML documents; upload them to Solr server for indexing.
• Involved in Hadoop cluster task like Adding and Removing Nodes without any effect to running jobs and data.
• Followed agile methodology for the entire project.
• Installed and configured Apache Hadoop, Hive and Pig environment.
• Worked on HBase in creating HBase tables to load large sets of semi structured data coming from various sources.
• Performed data analysis with HBase using Hive External tables
• Exported the analyzed data to HBase using Sqoop and to generate reports for the BI team.
• Prepared technical design documents, detailed design documents.

Environment: Hadoop, HDFS, pig, Hive, Flume, Sqoop, Oozie, Python,Talend, Spark, HBase, Elastic search,Linux- Ubuntu, Solr, Kafka.

Hadoop Developer

Responsibilities:

• Analyze and understand the technical design document.
• Develop the shell scripts to check files and move files from edge node to HDFS location.
• Implement Hive external / internal tables and Hive queries to extract data for reports.
• Develop Sqoop Scripts according to business rules.
• Design Oozie workflow to execute actions in control order.
• Analysis of SIT defects and proving solutions.
• Accountable for understanding and analyzing the defects and proving solution.
• Handle importing of data from various data sources using Sqoop, perform transformations after loading data to Hive.
• Created UDF’s to implement business logic properly in Hadoop .
• Used Impala for faster querying purposes.
• Created Autosys jobs for automation for workflows.

Environment: Cloudera 5.3.3,Unix,Hive,Impala,Sqoop,Oozie,Hue,XML, Autosys,Qlikview

Big data Administrator/Developer

Responsibilities:

• Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters.
• Installed and configured Hadoop Map Reduce, HDFS, Developed multiple Map Reduce jobs in java for data cleaning and preprocessing. Python streaming in Hive and Pig.
• Used Impala for optimization of query performance instead of Hive.
• Experience in installing, configuring and using Hadoop Architecture components.
• Experience in Importing and exporting data into HDFS and Hive using Sqoop.
• Experienced in defining job flows.
• Experienced in managing and reviewing Hadoop log files.
• Participated in development/implementation of Cloudera Hadoop environment.
• Load and transform large sets of structured, semi structured and unstructured data.
• Experience in working with various kinds of data sources such as Oracle.
• Successfully loaded files to Hive and HDFS from Oracle DB.
• Gained good experience with NOSQL database like Cassandra and HBase.
• Supported Map Reduce Programs those are running on the cluster.
• Involved in loading data from UNIX file system to HDFS.
• Installed and configured Hive and also written Hive UDFs.
• Involved in creating Hive tables, loading with data and writing hivequeries, which will run internally in map, reduce way. Shell Scripting for automation of data cleaning and reporting purposes.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Cloudera HDFS, Hadoop, MapReduce, Hive, Hive UDF Pig Latin, Java, SQL, Sqoop, Centos, NOSQL database,Cassandra,Shell Scripting.

Hadoop Developer

Responsibilities:

• Analyze and understand the technical design document.
• Develop the shell scripts to check files and move files from edge node to HDFS location.
• Key role in the design of Hadoop scripts for Application.
• Develop Pig Scripts according to business rules.
• Develop mapper and reducer jobs where ever required.
• Used UDF’s to implement business logic in Hadoop.
• Design Oozie workflow to execute actions.
• Analysis of SIT defects and proving solutions.
• Accountable for understanding and analyzing the defects and proving solution.
• Analyse the data by performing Hive queries and running Pig scripts to study customer behaviour.
• Responsible to play as Hadoop SME role for customer in absence of onshore SME.

Environment: IBM Big Insights, Qlikview