SUMMARY

• 9+ years of experience in designing and building scalable data pipelines to collect, parse, clean and transform data from multiple source systems and generate high - quality data sets for advanced analytics, dashboards, alerts, and visualizations.
• Experience with Big Data/ Hadoop Ecosystem: Spark, Hadoop, Hive, Airflow, Sqoop, Kafka, Oozie, Databricks.
• In-depth understanding of Spark Architecture and performed several batch and real-time data stream operations using Spark (Core, SQL, Structured Streaming).
• Experienced in handling large datasets using Spark in-memory capabilities, Partitions, Broadcast variables, Accumulators, Effective & Efficient Joins.
• Tested and Optimized Spark applications.
• Experience in using different data engineering frameworks on Cloudera, AWS and Azure.
• Performed Hive operations on large datasets with proficiency in writing HiveQL queries using transactional and performance efficient concepts: Partitioning, Bucketing, Windowing, etc.
• Hands on work experience in writing applications on NoSQL database -HBase.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Experienced the integration of various data sources like RDBMS, Spreadsheets, Text files.
• Expertise in developing and scheduling jobs using Airflow, Azure Data Factory, Oozie, Crontab and used Elasticsearch to index, fetch, and filtering of log data.
• Set up standards and processes for Hadoop based application design and implementation.
• Experience in developing solutions to analyze large data sets efficiently.

TECHNICAL SKILLS

Data Processing and ETL: Hadoop, Spark, PySpark, Apache Hive, Impala, Databricks, Sparksql, Python, SQL

Data Storage: Azure ADLS, Blob Storage, HDFS, AWS S3

Data Orchestration & Scheduling: Azure Data Factory, Airflow, Pentaho DI, shell scripting

Source data formats in Data Pipelines: RDBMS, CSV, XML, JSON, Rest APIs

On-prem & Cloud platforms: Azure, Cloudera, AWS

Languages: Python, Scala, SQL

Databases: SQL Server, MySQL, PostgreSQL, Oracle

PROFESSIONAL EXPERIENCE

Confidential, Deerfield, IL

Data Engineer

Responsibilities:

• Designed and implemented generic historical and incremental data ingestion framework using Spark, Python, Delta Lake and Azure Cloud offerings.
• Designing and developing the ETL process, to ingest, transform and store data using Hive, Snowflake, Presto, Airflow and Python.
• Performing transformations on the data by creating an ETL tables on top of the staging tables which helps in faster accessing of the data for reporting.
• Implemented generic JSON parsing logic using Spark and Python
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse.
• Extracted data from multiple sources including FTP, RDBMS and REST API, applied complex transformations and loaded it to ADLS.
• Experience in using Metadata PostgreSQL which is used to schedule the entire process like transferring the files into S3, loading the data into Staging, Perform the ETL operations and updating the ETL table on the daily basis which is used by Revenue Analytics team to create their dashboards in Looker.
• Implemented Apache Airflow, a workflow management platform, with scheduled Python scripts to automate workflows.
• Designed the scalable infrastructure required for optimal ETL of data using Airflow, to move data from a variety of data sources to the data warehouse.
• Automated the data pipeline using Azure Data Factory.
• Developed Azure Logic Apps for sending the automated emails to business users after completion of data pipelines.
• Used Azure DevOps for CI/CD and deployment of data pipelines

Environment: Spark, Hive, HDFS, ETL, Python, Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Delta Lake, Azure SQL DB, SQL DW

Confidential - Cincinnati, OH

Data Engineer

Responsibilities:

• Performed Data analysis, Data Profiling and Requirement Analysis.
• Analyzed massive and highly complex HIVE data sets, performing ad-hoc analysis and data manipulation.
• Designed and developed custom data integration pipelines on Facebook’s big data stack such as python, YAML, Hive and Data swarm.
• Developed Oozie workflow for scheduling and orchestrating the ETL process.
• Worked on building streaming data pipeline using Spark Streaming and Kafka.
• Worked on Spark based Ingestion framework to ingest data from RDBMS to Hadoop data lake.
• Ingested data from various RDBMS including MySQL, Oracle, SQL Server using spark-based ingestion framework.
• Worked on spark optimization and performance tuning to minimize the execution time to achieve targeted ingestion time.

Environment: Spark, Python, Hive, ADLS, Azure Data Factory, Databricks, SQL Server, SQL warehouse

Confidential, Palo Alto, CA

Data Engineer

Responsibilities:

• Developed Spark framework for parsing and processing files from different source systems.
• Developed Spark pipelines to call REST APIs and pull JSON, XML Data into Datalake.
• Created Hive DDLs and DMLs to load and transform CSV files.
• Developed ETL mappings and workflows using Informatica and Data swarm.
• Developed framework in PySpark to parse JSON files and insert the same into specific tables dynamically.
• Developed UDF’s in Python to meet specific business requirements.
• Involved in migration of ETL process from Vertica to Presto.
• Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of
• Involved in different phases of building the Data Marts like analysing business requirements: ETL process design, performance enhancement, go-live activities, and maintenance.

Environment: Python, Spark, Hive, AWS S3, AWS EMR, SQL, Airflow.

Confidential

Data Engineer

Responsibilities:

• Partitioned the collected logs by date/timestamps and host names. Developed programs to extract the required data from the logs.
• Extracted data from SQLServer, Oracle, Teradata into HDFS using Sqoop jobs with incremental/full load.
• Extensive experience in writing Pig (version 0.11) scripts to transform raw data from several data sources into forming baseline data.
• Developed Hive (version 0.10) scripts for end user / analyst requirements to perform ad hoc analysis.
• Excellent understanding of Partitions concepts in Hive and designed External tables in Hive to optimize performance.
• Developed HBase tables for storing meta data to be used by other Hadoop/application specific elements.

Environment: AWS, S3, Hive, Pig, HBase, Oracle, Oozie, Hadoop, EMR, Spark, Sqoop, Redshift, Shell Scripting.