SUMMARY

• IT professional as a data engineer with elite experience in Big Data Technologies, ETL, Hadoop Ecosystem, AWS, Map Reduce, RDBMS, business reporting
• Hands on experience on Hadoop ecosystem components like HDFS, Hive, YARN, Sqoop, Hbase, impala, oozie, zookeeper, Python, kafka, pyspark.
• In depth understanding of Hadoop Architecture including YARN and various components such as HDFS, Resource Manager, Node Manager, Name Node, Data Node.
• In - depth understanding of Spark Architecture includinresg spark core, spark sql, data frames, spark streaming, spark mlib and Spark Real time Streaming.
• Hands on experience in Analysis, Design, Coding and Testing phases of Software Development Life Cycle (SDLC).
• Proficient with Apache Spark ecosystem such as Spark, Spark Streaming using Scala and Python.
• Hands on experience with AWS (Amazon Web Services), Elastic Map Reduce (EMR), Storage S3, EC2 instances, lambda, kinesis.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Hands on experience in various Bigdata application phases like data ingestion, data analytics and data visualization.
• Experience in usage of Hadoop distribution like Cloudera, Hortonworks distribution & Amazon AWS.
• Experience in transferring data from RDBMS to HDFS and HIVE table using pyspark.
• Migrating the coding from Hive to Apache Spark and Scala using Spark SQL, RDD.
• Understanding of flume architecture to load the data from multiple sources directly into HDFS.
• Very well versed in workflow scheduling and monitoring tools such as CAWA, Autosys.
• Good knowledge on impala, sparksql, storm, avro, kafka, hue and aws and knowledge on ide tools such as eclipse, netbeans, and maven.
• Strong knowledge of version control systems like SVN and GITHUB.
• Good understanding of Spark -Streaming using kafka. Experience in writing kafka consumers.
• Basic Knowledge on nifi, kylin and zeppelin with Apache Spark.
• Experience in NoSQL Column-Oriented Databases like Hbase, Cassandra and its Integration with Hadoop cluster.
• Involved in Cluster coordination services through Zookeeper.
• Good level of experience in Core Java, J2EE technologies as JDBC, Servlets, and JSP.
• Hands-on knowledge on core Java concepts like Exceptions, Collections, Data-structures, Multi-threading, Serialization, and deserialization.
• Excellent interpersonal skills in areas such as teamwork, communication and presentation to business users or management teams.
• Strong problem-solving skills, quick learner and able to work independently as well as a team member of varying sized teams.
• Experience working with AWS EMR, Sagemaker or other cloud-based platforms
• Developed the Pyspark code for AWS Glue jobs and for EMR.
• Ability to plan, manage, motivate, and work efficiently as an independent or collaboratively in a team.
• Self-motivated, enthusiastic, and always keen to learn new methodologies and techniques.

TECHNICAL SKILLS

Databases: Oracle, Mysql, Hive, SQL Server

BigData Technologies: HDFS, Hive, pySpark, NiFi, REST API, Map Reduce, Pig, YARN, Sqoop, Oozie, Zookeeper, Flume, Spring Boot, Spring REST

Programming Languages: Python, Java, SQL, R, PL/SQL, Scala, json, xml, C#

Cloud Computing: AWS EC2, S3, RDS, DynamoDB, SNS, SQS, kinesis, step functions, lambda functions

Techniques: Datamining, Clustering, Data Visualization, Data Analytics

NoSQL Databases: Hbase, Cassandra

Container Platforms: Docker, Kubernetes, CD/CI, Jenkins

PROFESSIONAL EXPERIENCE

Confidential, Denver, CO

Sr. Big Data Engineer (AWS)

Responsibilities:

• Developed the different ingestion pipeline components using Spark and Python.
• Writing spark sql scripts to load and process data into Hadoop File System.
• Design and created common templates for ETL processing jobs.
• Involved in designing and deploying multi-tier applications using all the AWS services like (EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, IAM) focusing on high-availability, fault tolerance, and auto-scaling in AWS Cloud Formation
• Worked on json formats and converted all json files to hive tables using spark/java.
• Implemented reactive REST API’s using Spring Cloud and FLUX to aggregate data from external services and formulate data in well-defined contracts to consumers.
• Written multiple Map Reduce programs to extract data for extraction, transformation, and aggregation from more than 20 sources having multiple file formats including XML, JSON, CSV & other compressed file formats.
• Developed Java MapReduce jobs for aggregation and interest matrix calculations for users.
• Implemented Terraform to manage the AWS infrastructure and managed servers using configuration management tools like Chef and Ansible.
• Implemented Spark Core in Scala to process data in memory.
• Performed job functions using Spark APIs in Scalaforreal time analysis and for fast querying purposes.
• Involved in creating Spark applications in Scalausing cache, MapReduce by Key etc. functions to process data.
• Develop complex Hadoop and Java applications (serverless & containerized) in AWS.
• Make data ingestion modules using AWS Glue for stacking data in different layers in S3 and reporting utilizing Athena and Quicksight.
• Parsing XML, JSON files, extensively used Java API connectors to call web services for MDM.
• Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.
• Transferred the data from AWS S3 to AWS Redshift.
• Wrote various data normalization jobs for new data ingested into Redshift.
• Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
• POC to explore AWS Glue capabilities on Data cataloging and Data ingestion.
• Involved in spark performance tuning for spark long running jobs.
• Implemented server less architecture using Lambda, Dynamo Db.
• Created a Lambda deployment function, and configured it to receive events from S3 bucket.
• Automated lambda ETL jobs using S3 events and AWS Lambda functions.
• Designed and implemented data ingestion Spark streaming framework from various data source like REST API, Kafka using spark streaming Scala API and Kafka.
• Responsible for delivering datasets from Snowflake to One Lake Data Warehouse and built CI/CD pipeline using Jenkins and AWS lambda and Importing data from DynamoDB to Redshift in Batches using Amazon Batch using TWS scheduler.
• Developed the UNIX shell/Python scripts for creating the reports from Hive data.
• Good knowledge and extensively using Web HDFS REST API commands in Perl scripting.
• Implemented confluent KAFKA consumers to stream the data to HDFS using python.
• Designed and developed micro services with Spring Boot and Spring Rest
• Designed and developed Spring Rest API with Spring Boot
• Involved in migrating oozie workflow code to spark flows for greater performance.
• Developed Spark code and Spark-SQL for faster testing and processing ofdata.
• Involved in converting Hive or SQL queries into Spark transformations using Python and Scala.
• Experience in creating spark applications using RDD, Data frames
• Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the PYSPARK API.
• Great Understanding of AWS compute services such as EC2,Elastic MapReduce(EMR), EBS and accessing Instance metadata.
• Worked extensively on PYSPARK to build Big Data flow.

Confidential, Denver, CO

Data Engineer

Responsibilities:

• Worked with internal and external data sources on improving data accuracy / coverage and generate recommendation on the process flow to accomplish the goal.
• Involved in spark performance tuning.
• Designed/Developed C# ASP.NET web applications for online survey, data collection, data analysis.
• Designed and implemented data ingestion Spark streaming framework to pull data from multiple data sources using REST API and PySpark.
• Created EC2 instances and implemented large multi node Hadoop clusters in AWS cloud from scratch using automated scripts such as terraform.
• Write a back-end logic by using C# language with ASP.NET core web API.
• Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for increasing performance benefit and helping in organizing data in a logical fashion.
• Written Hive queries for data analysis to meet the business requirements. Designed and developed User Defined Function (UDF) for Hive.
• Developed spark applications in python (PySpark) on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables.
• Web development using Blazor framework with reusable UI components like C#
• Worked on NoSQL databases likeHBaseand imported data fromOracleand processed data using Hadoop Tools and exported toCassandraNoSQL database.
• Involved in creating Hive tables (Managed tables and External tables), loading, and analyzing data using hive queries.
• Build, test, and deploy predictive models through batch & API in an AWS ecosystem (e.g., Sagemaker).
• This entire pipeline was orchestrated using state machine from AWS Step Functions.
• Good knowledge about the configuration management tools like SVN/CVS/GitHub.
• Exposure on Spark Architecture and how RDD’s work internally by involving and processing the data from Local files,HDFS and RDBMS sources by creating RDD and optimizing for performance.
• Working on Airflow which is a job trigger where the code is written in hive query language and Scala. This helps to read, back fill, and write the data for particular time Frame from hive tables to HDFS locations.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD, Scala, and Python.
• Configured Spark streaming to receive real time data from theApache KAFKA and store the stream data to the HDFS usingScala.
• Developed framework for converting existing PowerCenter mappings and to PySpark(Python and Spark) Jobs.
• Developed several classes using C# and experienced in creating assemblies and Name Spaces.
• Involved in converting MapReduce programs into Spark transformations usingSpark RDDin Scala
• Worked towards creating real time data streaming solutions using ApacheSpark / Spark Streaming, Kafka.
• Good Knowledge about scalable, secure cloud architecture based on Amazon Web Services (leveraging AWS cloud services:EC2, Cloud Formation,VPC, S3, etc.
• Create external tables with partitions using Hive, AWS Athena and Redshift.
• Migrated on premise database structure to confidential Redshift
• Collected the logs data from web servers and integrated in toHDFSusingFlume.
• Analyzed the data by performing Hive queries (HiveQL)and running Pig Scripts (Pig Latin).
• Expert in creating the SFTP Connection to the internal and external source to get data in secured manner without any breakage.
• Handle the production Incidents assigned to our workgroup promptly and fix the bugs or route it to the respective teams and optimized the SLA’s.

Confidential

Data Engineer

Responsibilities:

• Created various hive external tables, staging tables and joined the tables as per the requirement.Implemented static Partitioning, Dynamic partitioning and Bucketing in Hive using internal and externaltables.
• Written Spark programs in Python and ran Spark jobs on YARN.
• Involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS.
• Involved in converting Hive/SQL queries into Spark transformations usingSpark RDDs, Python.
• Import the data from different sources like HDFS/HBase into SparkRDD.
• Experienced with Spark Context, Spark -SQL, Data Frame, Pair RDD's, Spark YARN.
• Involved in converting Hive/SQL queries into Spark transformations using SparkRDD and Scala.
• Involved in loading data from UNIX file system to HDFS, Importing and exporting data into HDFS using Sqoop, experienced in managing and reviewing Hadoop log files.
• Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creating ETL pipelines using Pig, and Hive
• Implemented the workflows using Apache Oozie framework to automate tasks.
• Experience in managing and reviewing Hadoop log files.
• Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems like Oracle and MySQL.
• Implemented Ad - hoc query using Hive to perform analytics on structured data.
• Created Airflow Scheduling scripts in Python.
• Worked on SQL queries in dimensional datawarehouses and relational datawarehouses.
• Developed Spark SQL to load tables into HDFS to run select queries on top.
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
• Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing.
• Performed troubleshooting, fixed and deployed many Python bug fixes of the two main applications that were a main source of data for both customers and internal customer service team.
• Evaluate, Refine and Improve the Efficiency of existing Hive Scripts by using partitioning, bucketing, map-side joins, different file format, compression techniques such as parquet, orc, JSON, snappy to increase the performance of the hive queries and reduce the memory leakage and wastage of the resources.
• Develop bash scripts to append the logs data and part files generated in the process to ingest into warehouse and HBase.
• Created data pipelines for different events to load the data from DynamoDB to AWS S3 bucket and then into HDFS location.
• Develop dynamic data pipeline with the help of Flume, Kafka, Oozie Operational Services, and shell scripting as part of batch processing.
• Optimizing Hive queries by tuning configuration parameters. Worked on different data formats such as Parquet, AVRO, text, json and XML file formats.

Confidential

ETL/Hadoop Developer

Responsibilities:

• Developed and delivered crucial pilot project successfully which resulted in major project win for organization.
• Increased ETL performance by 55% by implementing best ETL practices after analyzing inherent operational pattern.
• Developed code to import data SQL Server into HDFS and created Hive views on data in HDFS using Spark in Python.
• Designed & implemented data loading process and BPMN flows throughout data reconciliation phase.
• Designed ETL workflows in Informatica PowerCenter for data cleansing, enrichment and implemented quality checks.
• Acted as point of contact between product users, onsite team, dev team and SMEs resulting in successful communication.
• Improved project performance by 70% by seamlessly auditing and moving ETL jobs from DEV to QA environment.
• Good experience in handling data manipulation using python Scripts
• Developed various algorithms for generating several data patterns.Used JIRA for bug tracking and issue tracking.
• Created Airflow workflows to run multiple MR, Hive, and pig jobs.
• Supported in setting up QA environment and updating configurations for implementing scripts with Hive and Sqoop.
• Migrated existing java application into Microservices using spring boot and spring cloud.
• Implemented test scripts to support test driven development and continuous integration.
• Worked on tuning the performance of MapReduce Jobs.
• Involved in loading data from LINUX file system to HDFS.
• Analyze latest Big Data Analytic technologies and their innovative applications in both business intelligence analysis and new service offerings.