SUMMARY

• 9+ years’ experience in Big Data development.
• Experience in large scale distributed systems with extensive experience as Hadoop Developer and Big Data Analyst.
• Primary technical skills in HDFS, YARN, Pig, Hive, Sqoop, HBase, Flume, Oozie, Zookeeper.
• Have good experience in extracting and generating statistical analysis using Business Intelligence tool
• Facilitation of meetings following Scrum processes such as Sprint Planning, Backlog, Sprint Retrospective, Requirements Gathering and providing planning and documentation for project; ensuring project is on track with stakeholder wishes.
• QlikView for better analysis of data.
• In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, Job
• Tracker, Task Tracker, Name Node, Data Node and MapReduce concepts and experience in working with
• Used Apache Hadoop for working with Big Data to analyze large data sets efficiently.
• Hands on experience in working with Ecosystems like Hive, Pig, Sqoop, MapReduce, Flume, Oozie. Strong knowledge of Pig and Hive's analytical functions, extending Hive and Pig core functionality by writing custom UDFs.
• Strong Experience in working with Databases like Teradata and proficiency in writing complex SQL, PL/SQL for creating tables, views, indexes, stored procedures and functions.
• Hands - on experience developing Teradata PL/SQL Procedures and Functions and SQL tuning of large databases.
• Experience in importing and exporting Terabytes of data between HDFS and Relational Database Systems using Sqoop.
• ETL from databases such as SQL Server … Oracle11G to Hadoop HDFS in Data Lake.
• Experience in writing SQL queries, Stored Procedures, Triggers, Cursors and Packages.
• Experience working with various tools.
• Experience in handling XML files and related technologies.
• Performed the performance and tuning at source, Target and Data Stage job levels using Indexes, Hints and Partitioning in DB2, ORACLE and DataStage.

TECHNICAL SKILLS

APACHE: Flume, Hadoop, Yarn, Hive, Kafka, Maven, Oozie, Pig, Spark, Tez, Zookeeper

SCRIPTING: PIG/Pig Latin, HiveQL, Python, UNIX, Shell scripting, LINUX, HTML5

OPERATING SYSTEMS: Unix/Linux, Windows

FILE FORMATS: Parquet, Avro, JSON, ORC, CSV, SequenceFile

DISTRIBUTIONS: Cloudera, Hortonworks, MapR, AWS, Elastic, Elastic Cloud, Elasticsearch, Cloudera CDH 4/5, Hortonworks HDP 2.3/2.4, Amazon Web Services (AWS)

DATA PROCESSING (COMPUTE) ENGINES: Apache Spark, Flink, MapReduce, Databricks

DATA VISUALIZATION TOOLS: Pentaho, QlikView, Tableau, PowerBI

DATABASE And DATA STRUCTURES: Apache Cassandra, Datastax Cassandra, Amazon Redshift, DynamoDB, Apache HBase

PROJECT MANAGEMENT: Management experience as team lead, project planning, facilitation and coordination using both waterfall and Agile processes such as Agile Scrum. Experience managing costs

PROFESSIONAL EXPERIENCE

Cloud Data Engineer

Confidential, Boston MA

Responsibilities:

• Created automated Python scripts to convert the data from different sources and to generate ETL pipelines.
• Used Hive optimization techniques such as partitioning, bucketing, Map Join, and parallel execution.
• Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Big Data technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive.
• Developed maintenance and software installation scripts.
• Extracted data from different databases and scheduled Oozie workflows to execute daily tasks.
• Developed distributed query agents for performing distributed queries against Hive.
• Loaded data from different sources such as HDFS or HBase into Spark data frames and implemented in-memory data computation to generate the output response.
• Monitored resources such as Amazon DB and CPU Memory using CloudWatch.
• Collaborated on a Hadoop cluster (CDH) and reviewed log files of all daemons.
• Used Spark SQL to realize quicker results compared to Hive throughout information Analysis.
• Created Hive external tables and designed information models in Hive.
• Developed multiple Spark Streaming and batch Spark jobs using Python.

BIG DATA ENGINEER

Confidential, Maplewood, MN

Responsibilities:

• Worked with Spark to create structured data from the pool of unstructured data received.
• Implemented advanced procedures like text analytics and processing using the in-memory computing capabilities like Apache Spark written in Scala.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala.
• Documented the requirements including the available code which should be implemented using Spark, Hive, HDFS and Elasticsearch.
• Maintained ELK (Elasticsearch, Logstash, Kibana) and wrote Spark scripts.
• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
• Fine-tuned resources for long running Spark Applications to utilize better parallelism and executor memory for more caching.
• Used Apache Spark and Scala on large datasets using Spark to process real time data.
• Transferred Streaming data from different data sources into HDFS and HBase using Apache Spark.
• Fetched the live stream data from DB2 to Hbase table using Spark Streaming and Apache Kafka.
• Involved in complete Big Data flow of the application starting from data ingestion from upstream to HDFS, processing the data into HDFS using Spark Streaming.
• Developed ETL pipelines using Spark and Hive for performing various business specific transformations.
• Automated the pipelines in Spark for bulk loads as well as incremental loads of various datasets.
• Worked on building input adapters for data dumps from FTP Servers using Apache spark.
• Integration of Kafka with Spark for real time data processing.
• Performed streaming data ingestion to the Spark distribution environment, using Kafka.
• Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load the data into Cassandra. Elasticsearch and Logstash performance and configure tuning.
• Responsible for designing and deploying new ELK clusters (Elasticsearch, Logstash, Kibana), Beats, Kafka, zookeeper etc.
• Bash source databases and creating ETL pipeline into Kibana and Elasticsearch. Involved in the process of data acquisition, data pre-processing and data exploration of project in Scala.
• Developed Spark applications for the entire batch processing by using Scala.
• Developed Spark scripts per the requirement and used Scala for proof of concept.
• Implemented Hadoop using Hortonworks Data Platform (HDP).
• Worked on continuous Integration with Jenkins and Git and automated jar files at end of day.

Hadoop Cloud Engineer

Confidential, New York, NY

Responsibilities:

• Involved in migrating MapReduce jobs to Spark, using Spark SQL andDataFramesAPI to load structured data into Spark clusters
• Implemented Spark using Scala, utilizedDataFramesand Spark SQL API for faster processing of data.
• Involved in converting HiveQL/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
• Wrote shell scripts for automating the process of data loading.
• Documented Technical Specs, Dataflow, Data Models and Class Models.
• Installed and configured various components of the Hadoop ecosystem.
• Created UNIX shell scripts to automate the build process, and to perform regular jobs like file transfers between different hosts.
• Developed a task execution framework on EC2 instances using SQL and DynamoDB.
• Captured and transformed real-time data from Amazon Aurora into a suitable format for scalable analytics.
• AWS Cloud services planning, designing and DevOps support likeIAM users, group, roles & policy management. AWS Access Key management
• Manipulated and analyzed complex, high volume, and high dimensional data in AWS using various querying tools.
• Created a POC involved in loading data from LINUX file system to AWS S3 and HDFS.
• Responsible for installing and configuring Apache Hadoop and tools on the cloud.
• Managed, configured, tuned and continuous deployment of 80 Hadoop nodes in a Red Hat Enterprise edition 5; configured via the AWS console for 2 medium scale AMI instances for the Name Nodes
• Configured Zookeeper ensemble of 3 nodes in standalone and multi-node cluster.
• Established Java based shell; reconfigured Zookeepernodes-ephemeral, sequential and persistentnodes.
• Worked on streaming analyzed data to HBase using Sqoop to make it available for visualization and report generation by the BI team.

Big Data Developer

Confidential, New York, NY

Responsibilities:

• Exported data from DB2 to HDFS using Sqoop.
• Installed and configured Pig and also wrote Pig Latin scripts.
• Wrote Map Reduce jobs using Pig Latin.
• Developed workflow using Oozie for running MapReduce jobs and Hive Queries.
• Worked on Cluster coordination services through Zookeeper.
• Worked on loading log data directly into HDFS using Flume.
• Involved in loading data from LINUX file system to HDFS.
• Wrote multiple MapReduce programs in Java for data extraction, transformation, and aggregation from multiple file formats.
• Responsible for managing data from multiple sources.
• Responsible to manage data coming from different sources.
• Assisted in exporting analyzed data to relational databases using Sqoop.
• Implemented JMS for asynchronous auditing purposes.
• Involved in developing Message Driven and Session beans for claimant information integration with MQbased JMS queues.
• Created and maintained technical documentation for launching Cloudera Hadoop Clusters and for executing
• Hive queries and Pig Scriptsto make UDFs.
• Worked on developing custom MapReduce programs and User Defined Functions (UDFs) in Hive to transform large volumes of data with respect to business requirements.
• Worked on installing cluster, commissioning & decommissioning of data node, name node recovery, capacity planning, and slots configuration.
• CreatedHBasetables to store variable data formats of PII data coming from different portfolios.