SUMMARY

• 7+ years of IT experience in Analysis, Design, Development in Big Data technologies like Spark, MapReduce, Hive, Kafka and HDFS including programming languages like Java, Scala and Python.
• Strong experience building data pipelines and performing large scale data transformations.
• In - Depth Dknowledge in working with Distributed Computing Systems and parallel processing techniques to efficiently deal with Big Data.
• Firm understanding of Hadoop architecture and various components including HDFS, Yarn, MapReduce, Hive, Pig, HBase, Kafka, Oozie etc.,
• Strong experience building Spark applications using scala and python as programming language.
• Good experience troubleshooting and fine-tuning long running spark applications.
• Strong experience using Spark RDD Api, Spark DataFrame/Dataset Api,Spark-SQL and Spark ML frameworks for building end to end data pipelines.
• Good experience working with real time streaming pipelines using Kafka and Spark-Streaming.
• Strong experience working with Hive for performing various data analysis.
• Detailed exposure with various hive concepts like Partitioning, Bucketing, Join optimizations, Ser-De’s, built-in UDF’s and custom UDF’s.
• Good experience in automating end to end data pipelines using Oozie workflow orchestrator.
• Good experience working with Cloudera, Hortonworks and AWS big data services.
• Strong experience using and integrating various AWS cloud services like S3, EMR, Glue Metastore, Athena, Redshift into the data pipelines.
• Experience in analyzing, designing, and developing ETL Strategies and processes, writing ETL specifications.
• Excellent understanding of NoSQL databases like HBASE, Cassandra, MongoDB.
• Proficient knowledge and hand on experience in writing shell scripts in Linux.
• Experienced in requirement analysis, application development, application migration and maintenance using Software Development Lifecycle (SDLC) and Python/Java technologies.
• Excellent technical and analytical skills with clear understanding of design goals and development for OLTP and dimensions modeling for OLAP.
• Adequate knowledge and working experience in Agile and Waterfall Methodologies.
• Defining user stories and driving the agile board in JIRA during project execution, participate in sprint demo and retrospective.
• Have good interpersonal, communication skills, strong problem-solving skills, explore/adopt to new technologies with ease and a good team member.

TECHNICAL SKILLS

Big Data Ecosystem: Hadoop, HDFS, MapReduce, Hive, Sqoop, Pig, HBase, Flume, Oozie, Impala, Kafka, Spark

Programming Languages: Python, Scala and Java

Databases: MySQL, Teradata, Oracle

IDE & ETL Tools: Eclipse, Intellij, Maven, Jenkins

Other Tools: Putty, WinSCP, Amazon AWS Console, Apache Amabari, Cloudera Manager.

Version Control: GitHub, SVN, CVS

Methodologies: Agile, Waterfall

Operating Systems: Windows, Mac, Linux

PROFESSIONAL EXPERIENCE

Confidential, Bloomfield, CT

Data Engineer

Responsibilities:

• Designedrobust, reusable and scalable data driven solutions and data pipelines to automate the ingestion, processing and delivery of both structured and semi structured batch and real time data streaming data.
• Applied efficient and scalable data transformations on the ingested data using Spark framework.
• Built end-to-end Spark applications for performing data cleansing, tranformations and aggregations of variets of data sources ingested from down stream applications.
• Worked closely with machine learning teams to deliver feature datasets in an automated manner to help them with model training and mode scoring.
• Gainedgood knowledge introubleshooting and performance tuningSpark applications and Hive scripts to achieve optimal performance.
• Developed various custom UDF’s in spark for performing transformations on date fields, complex string columns and encrypting PI fields etc.
• Performed Spark join optimizations, troubleshooted, monitored and wrote efficient codes using Scala.
• Written complex hive scripts for performing various data analysis and creating various reports requested by business stakeholders.
• Used AWS Simple workflows for automating and scheduling our data pipelines.
• Utilized Glue metastore as common metastore between EMR clusters and Athena query engine with S3 as the storage layer for both.
• Worked extensively on migrating our existing on-Prem data pipelines to AWS cloud for better scalability and infrastructure maintenance.
• Worked extensively in automating creation/termination of EMR clusters as part of starting the data pipelines.
• Worked extensively on migrating/rewriting existing oozie jobs to AWS simple workflow.
• Loaded the processed data into Redshift tables for allowing downstream ETL and reporting teams to consume the processed data.
• Good experience working on analysis tools like Tableau, Splunk for regression analysis, pie charts and bar graphs.

Environment: AWS Cloud, S3, EMR, Redshift, Athena, Scala, Spark, Kafka, Hive, Yarn, HBase, Jenkins, Docker

Confidential, Charlotte, NC

Hadoop Developer

Responsibilities:

• Involved in developing roadmap for migration of enterprise data from multiple data sources like SQL Server, provider databases into S3 which serves as a centralized datahub across the organization.
• Loaded and transformed large sets of structured and semi structured data from various downstream systems.
• Developed ETL pipelines using Spark and Hive for performing various business specific transformations.
• Building Applications and automating the pipelines in Spark for Bulk loads as well as Incremental Loads of various Datasets.
• Worked closely with our team’s data scientists and consumers to shape the datasets as per the requirements.
• Automated the data pipeline to ETL all the Datasets along with full loads and incremental loads of data.
• Utilized AWS services like EMR, S3, GlueMetastore and Athena extensively for building the data applications.
• Worked on building input adapters for data dumps from FTP Servers using Apache spark.
• Wrote spark applications to perform operations like data inspection, cleaning, load and transforms the large sets of structured and semi-structured data.
• Developed Spark with Scala and Spark-SQL for testing and processing of data.
• Reporting the spark job stats, Monitoring and running Data Quality Checks are made available for each Datasets.

Environment: AWS Cloud Services, Apache Spark, Spark-SQL, Unix, Kafka, Scala, SQL Server.

Confidential, Nashville, TN

Hadoop Developer

Responsibilities:

• Responsible for the design, implementation and architecture of very large-scale data intelligence solutions around big data platforms.
• Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper.
• Developed multiple POC’s using Spark, Scala and deployed on the Yarn Cluster, compared the performance of Spark, with Hive and SQL.
• Use Amazon Elastic Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage mechanism.
• Capable of using AWS utilities such as EMR, S3 and Cloud Watch to run and monitor Hadoop and Spark jobs on AWS.
• Worked on SQL queries in dimensional data warehouses and relational data warehouses. Performed Data Analysis and Data Profiling using Complex SQL queries on various systems.
• Troubleshoot and resolve data processing issues and proactively engaged in data modelling discussions.
• Worked on RDD Architecture and implementing spark operations on RDD and optimizing transformations and actions in Spark.
• Written programs in Spark using Python (PySpark) packages for performance tuning, optimization and data quality validations.
• Worked on developing KafkaProducers and Kafka Consumers for streaming millions of events per second on streaming data.
• Implemented a distributing messaging queue to integrate with Cassandra using Apache Kafka.
• Hands on experience on fetching the live stream data from UDB into HBase table using PySpark streaming and Apache Kafka.
• Worked on Tableau to build customized interactive reports, worksheets, and dashboards.

Environment: HDFS, Python, SQL, Spark, Scala, Kafka, Hive, Yarn,Sqoop, Tableau, AWS Cloud, GitHub, Shell Scripting.

Confidential, Cranberry Township, PA

Hadoop Developer

Responsibilities:

• Developed shell scripts to create staging, landing tables with the same schema as the source and generate the properties which are used by Oozie jobs.
• Developed Oozie workflow for executing Sqoop and Hive actions and worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.
• Performance optimizations on Spark and Python.
• Diagnose and resolve performance issues in Spark.
• Responsible for developing Python wrapper scripts which will extract specific date range using Sqoop by passing custom properties required for the workflow.
• Developed scripts to run Oozie workflows, capture the logs of all jobs that run on cluster and create a metadata table which specifies the execution times of each job.
• Convertedexisting MapReduce applications toPySparkapplicationas part of overall effort to stream line legacy jobs and create new framework.

Environment: Hadoop, HDFS, Map Reduce, Hive, HBase, Oozie, Impala, Java (jdk1.8), Cloudera, Python, UNIX Shell Scripting, Flume, Scala, Spark, Sqoop,Kafka, Oracle

Confidential

Java Developer

Responsibilities:

• End to End designing of Critical Core Java Components using Java Collections and Multithreading.
• Analysis of different database schemas Transaction and Data warehouse to build
• Extensive reports to Business using SQL & Joins.
• Development of multiple reports to business in quick turn-around time,
• Created one of the best programs to notify the operational team on Downtime of one of 250 pharmacies on the AP network in a few seconds.
• Created an interface using JSP, Servlet, and MVC Struts architecture for the pharmacy team to resolve stuck orders in different pharmacies.
• Performance tuned the IMS report to memory leaks and best practices in java to boost the performance and reliability of the application.

Environment: Java 1.5, Servlets, J2EE 1.4, JDBC, Oracle, 10g, PL SQL, HTML, JSP, Eclipse, UNIX, J2EE (JSP, Servlets, Java Beans, JDBC, Multi-Threading), LINUX, (Shell & Perl Scripting), and SQL