SUMMARY

• Over 6+ years of IT experience as a Data Engineer with a good understanding of Big Data ecosystems, Hadoop Distributions, python, scala, NoSQL and SQL databases, and ETL processes.
• Sound Experience with AWS cloud EMR, EC2, EBS, S3, Lambda.
• Experience in working with Azure cloud platform (Databricks, Data Lake, Blob, Data Factory, SQL DB, SQL DWH).
• Expertise in building PySpark and Spark - Scala applications for interactive analysis, batch processing, and stream processing.
• Experienced in transporting and processing real-time event streaming using Kafka and Spark Streaming.
• Extensive experience using Sqoop to ingest data from RDBMS - Oracle, MS SQL Server, Teradata, PostgreSQL, and MySQL.
• Sound experience in building production ETL pipelines between several source systems and Enterprise Data Warehouse by leveraging SSIS, SSAS, and SSRS.
• Experience with all aspects of development from initial implementation and requirement discovery, through release, enhancement, and support (SDLC & Agile techniques).
• Having experience in Development, Data Migration, Testing, Support, and Maintenance using Redshift Databases.
• Having experience with ApacheHadooptechnologies likeHadoopdistributed file system (HDFS), Map Reduce framework, Hive, PIG, Python, Sqoop, Oozie, HBase, Spark, Scala, and Python.
• Extensive experience in migrating on-premise Hadoop platforms to cloud solutions using AWS and Azure.
• Experience in writing python as ETL framework and Pyspark to process huge amounts of data daily.
• Strong experience in implementing data models and loading unstructured data using HBase, Dynamo Db, and Cassandra.
• Solid knowledge of Dimensional Data Modeling with Star Schema and Snowflake for FACT and Dimensions Tables using Analysis Services.
• Extensive knowledge in Data transformations, Mapping, Cleansing, Monitoring, Debugging, performance tuning, and troubleshooting Hadoop clusters.

TECHNICAL SKILLS

Big Data Ecosystem: HDFS, MapReduce, Yarn, Spark, Hive, Impala, StreamSets, Sqoop, HBase, Pig, Oozie, Zookeeper.

Hadoop Distributions: Apache Hadoop 2.x/1.x, Cloudera CDP, Hortonworks HDP, Amazon EMR (EMR, EC2, RDS, S3, Athena)

Programming Languages: Python, Scala, Java, R, Shell Scripting, Pig Latin, HiveQL.

NoSQL Database: Cassandra, MongoDB.

Database: MySQL, Oracle, MS SQL SERVER, PostgreSQL, DB2.

ETL/BI: Informatica, SSIS, SSRS, SSAS, Tableau, Power BI.

Web Development: Spring, J2EE, JDBC, .Net MVC, Tomcat, JavaScript, Node.js, HTML, CSS.

Operating systems: Linux (Ubuntu), Windows (XP/7/8/10)

Others: Machine learning, Spring Boot, Jupyter Notebook, Jira, Service Now

PROFESSIONAL EXPERIENCE

Confidential, Palo Alto, CA

Data Engineer

Responsibilities:

• Involved in developing batch and stream processing applications that require functional pipelining using Spark Scala and Streaming API.
• Analyzed and understood the architectural design of the project in a step-by-step process along with the data flow.
• Supported analytical platform, handled data quality, and improved the performance using Scala’s higher-order functions, lambda expressions, pattern matching, and collections.
• Implemented scalable microservices to handle concurrency and high traffic. Optimized existing Scala code and improved the cluster performance.
• Designed and automated Custom-built input adapters using Spark, Sqoop, and Oozie to ingest and analyze data from RDBMS to Azure Data Lake.
• Developed an end-end (Ingestion to Monitoring) NIFI data flow to process a huge volume of streaming XML and JSON events data from IoT sensors positioned on Locomotive engines, Base stations, and wayside detectors.
• Design and architecture of creating model-ready data for data scientists and advanced analytics.
• Involved in Optimizing Spark applications and tuning hive tables.
• Extensive knowledge in Data transformations, Mapping, Cleansing, Monitoring, Debugging, performance tuning, and troubleshooting Hadoop clusters.

Environment: Hadoop, Hive HBASE, Spark Scala, Python, Jenkins, Code Cloud, Sqoop, AWS, Apache Airflow, Shell Scripting, Oracle, SQL Server, and Teradata

Confidential, New York, NY

Data Engineer

Responsibilities:

• Analyzed and understood the architectural design of the project in a step-by-step process along with the data flow.
• Used Scoop API to perform data ingestion to the raw zone.
• Involved in the design and development of several big data implementations such as Patient Account receivables efficiency System, Claims Denial Discovery system, DRG analysis Engine, billing coding efficiency, Customer service productivity, etc. in the Hadoop platform.
• Have extensively used Apache Spark features such as RDD operations (mapping, merging, combining, aggregation of data, vectorization of data, etc.) and Data frames and datasets for transformation, enrichment of data, data storage operations, applying descriptive statistics, and aggregation of data.
• Worked on a big data development platform consisting of 124 nodes in the MapR distribution Cluster.
• Created and worked Sqoop jobs with incremental load to populate Hive External tables
• Scripts were written for the distribution of queries for performance test jobs in Amazon Data Lake.
• Have extensively worked on developing Spark Scala and Python scripts for data ingestion, transformations, and building data pipeline for Data Scientists, Data Analysis and Business Analysts.
• Developed optimal strategies for distributing the web log data over the cluster importing and exporting the stored web log data into HDFS and Hive using Sqoop.

Environment: Hadoop, Hive, Spark Scala, Java, Python, Apache Nifi, Apache Kafka, HBase, AWS EMR, Jenkins, GitHub, GitLab, Oozie, Sqoop, IntelliJ, Shell Scripting.

Confidential

Data Engineer

Responsibilities:

• Performed ETL operations using Python, SparkSQL, S3, and Redshift on terabytes of data to obtain customer insights.
• Involved in writing Python scripts to automate ETL pipeline and DAG workflows using Airflow.
• Involved in creating Hive tables, loading, and analyzing data using hive scripts by implementing Partitioning and Bucketing in Hive.
• Analyzed and understood the architectural design of the project in a step-by-step process along with the data flow.
• Implementing new dimensions into spark application upon the business requirements.
• Migrated Java analytical applications into Scala. Used Scala where performance and logic are critical
• Fetched live data from Oracle database using Spark Streaming and Kafka using the feed from API Gateway REST service.

Confidential

Associate Data Engineer

Responsibilities:

• Created SSIS Packages using Pivot Transformation, Fuzzy Lookup, Derived Columns, Condition Split, Term extraction, Aggregate, Execute SQL Task, Data Flow Task, and Execute Package
• Created Datasets using different data sources like Microsoft SQL Server, and Excel.
• Worked on DTS Packages, and DTS Import/Export for transferring data from Heterogeneous Database. Created Triggers to enforce data and referential integrity.
• Created Automated ETL jobs in Talend and pushed the data to Azure SQL data warehouse.
• Involved in the full life cycle of the project from Design, Analysis, logical and physical architecture modeling, development, Implementation, and testing.
• Moving data from Oracle to HDFS using Sqoop
• Data profiling on critical tables from time to time to check for the abnormalities
• Developed POC to execute Machine Learning models using SparkML library.
• Used Git for version control and Jira for project management, tracking issues and bugs.