SUMMARY

• 8 years of IT industry knowledge with hands - on working experience in Data Engineering & Data Analysis.
• Good knowledge of Data Quality & Data Governance practices & processes.
• Well-versed with Agile with Scrum, Waterfall Model, and Test-driven Development (TDD) methodologies.
• Proficient in SQLite, MySQL, and SQL databases with Python.
• Practical understanding of the Data modeling (Dimensional & Relational) concepts like Star - Schema modeling, Snowflake Schema Modelling, and Fact and Dimension tables.
• Experience in handling python and spark context when writing PySpark programs for ETL.
• Strong knowledge in data visualization using Power BI and Tableau.
• Hands-on experience with NoSQL databases like Snowflake, HBase, Cassandra, and MongoDB.
• Experience in building and architecting multiple Data pipelines, end-to-end ETL, and ELT processes for Data ingestion and transformation in GCP and coordinating tasks among the team.
• Experience in GCP Dataproc, GCS, Cloud functions, Big Table, and BigQuery.
• Experience with Apache Spark ecosystem using Spark-Core, SQL, Data Frames, and RDDs.
• Experienced in data manipulation using python.
• Hands-on experience working with Amazon Web Services (AWS) using Elastic Map Reduce (EMR), Redshift, and EC2 for data processing.
• Proficient in installing, configuring, and using Apache Hadoop ecosystems such as MapReduce, Hive, Pig, Flume, Yarn, HBase, Sqoop, Spark, Storm, Kafka, Oozie, and Zookeeper.
• Strong experience in designing Big data pipelines such as Data Ingestion, Data Processing (Transformations, enrichment, and aggregations), and Reporting.
• Experience in integrating Kafka with Spark streaming for high-speed data processing.
• Experience in implementing Azure data solutions, provisioning storage accounts, Azure Data Factory, Azure Databricks, Azure Blob Storage, Azure Synapse, and Azure Cosmos DB.
• An excellent team member with an ability to perform individually, good interpersonal relations, strong communication skills, hardworking and high level of motivation.

PROFESSIONAL EXPERIENCE

Sr. Data Engineer

Confidential, Weehawken, NJ

Responsibilities:

• As a Data Engineer responsible for building scalable distributed data solutions using Hadoop.
• Involved in Agile Development process (Scrum and Sprint planning).
• Handled Hadoop cluster installations in Windows environment.
• Migrated on-premise environment in GCP (Google Cloud Platform)
• Migrated data warehouses to Snowflake Data warehouse.
• Defined virtual warehouse sizing for Snowflake for different types of workloads.
• Involved in porting the existing on-premise Hive code migration to GCP (Google Cloud Platform) BigQuery.
• Involved in migration of an Oracle SQL ETL to run on Google cloud platform using cloud Dataproc & BigQuery, cloud pub/sub for triggering the Apache Airflow jobs.
• Extracted data from data lakes, and EDW to relational databases for analyzing and getting more meaningful insights using SQL Queries and PySpark.
• Designed, developed, and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems.
• Developed MapReduce programs to parse the raw data, populate staging tables, and store the refined data in partitioned tables in the EDW.
• Wrote Sqoop Scripts for importing and exporting data from RDBMS to HDFS.
• Set up Data Lake in Google cloud using Google cloud storage, BigQuery, and Big Table.
• Developed scripts in BigQuery and connected them to reporting tools.
• Designed workflows using Airflow to automate the services developed for Change data capture.
• Carried out data transformation and cleansing using SQL queries and PySpark.
• Used Kafka and Spark Streaming to ingest real-time or near real-time data in HDFS.
• Worked related to downloading BigQuery data into Spark data frames for advanced ETL capabilities.
• Built reports for monitoring data loads into GCP and driving reliability at the site level.
• Participated in daily stand-ups, bi-weekly scrums, and PI panning.

Environment: Hadoop 3.3, GCP, BigQuery, Big Table, Spark 3.0, Sqoop 1.4.7, ETL, HDFS, Snowflake DW, Oracle SQL, MapReduce, Kafka 2.8, and Agile process.

Sr. Data Engineer

Confidential, Boston, MA

Responsibilities:

• Built reporting data warehouse from ERP system using Order Management, Invoice & Service contracts modules.
• Extensive work in Informatica Powercenter.
• Acted as SME for Data Warehouse related processes.
• Performed Data analysis for building Reporting Data Mart.
• Worked with Reporting developers to oversee the implementation of report/universe designs.
• Tuned performance of Informatica mappings and sessions for improving the process and making it efficient after eliminating bottlenecks.
• Worked on complex SQL Queries, and PL/SQL procedures and convert them to ETL tasks
• Worked with PowerShell and UNIX scripts for file transfer, emailing, and other file-related tasks.
• Worked with deployments from Dev to UAT, and then to Prod.
• Worked with Informatica Cloud for data integration between Salesforce, RightNow, Eloqua, and WebServices applications
• Expertise in Informatica cloud apps Data Synchronization (ds), Data Replication (dr), Task Flows & Mapping configurations.
• Worked on migration project which included migrating web methods code to Informatica cloud.
• Implemented Proof of concepts for SOAP & REST APIs
• Built web services mappings and expose them as SOAP WSDL
• Worked with Reporting developers to oversee the implementation of reports/dashboard designs in Tableau.
• Assisted users in creating/modifying worksheets and data visualization dashboards in Tableau.
• Tuned and performed optimization techniques for improving report/dashboard performance.
• Assisted report developers with writing required logic and achieving desired goals.
• Met End Users for gathering and analyzing the requirements.
• Worked with Business users to identify root causes for any data gaps and develop corrective actions accordingly.
• Created Ad hoc Oracle data reports for presenting and discussing the data issues with Business.
• Performed gap analysis after reviewing requirements.
• Identified data issues within the DWH dimension and fact tables like missing keys, joins, etc.
• Wrote SQL queries to identify and validate data inconsistencies in the data warehouse against the source system.
• Validated reporting numbers between source and target systems.
• Finding a technical solution and business logic for fixing any missing or incorrect data issues identified
• Coordinating and providing technical details to reporting developers

Environment: Informatica Power Center 9.5/9.1, Informatica Cloud, Oracle 10g/11g, SQL Server 2005, Tableau 9.1, Salesforce, RightNow, Eloqua, Web Methods, PowerShell, Unix

Confidential, Scottsdale, AZ

Data Engineer

Responsibilities:

• As a Data Engineer was responsible to build a data lake as a cloud-based solution in AWS using Apache Spark and Hadoop.
• Involved in Agile methodologies, daily Scrum meetings, and Sprint planning.
• Objective of this project is
• Installed and configured Hadoop and was responsible for maintaining cluster and managing and reviewing Hadoop log files.
• Used AWS Cloud and On-Premise environments with Infrastructure Provisioning/ Configuration.
• Contributed to the development of key data integration and advanced analytics solutions leveraging Apache Hadoop.
• Wrote complex Hive queries to extract data from heterogeneous sources (Data Lake) and persist the data into HDFS.
• Developed Big Data solutions focused on pattern matching and predictive modeling.
• Developed the code for Importing and exporting data into HDFS and Hive using Sqoop
• Developed a data pipeline using Kafka, HBase, Spark, and Hive to ingest, transform, and analyze customer behavioral data.
• Developed Spark jobs and Hive Jobs to summarize and transform data.
• Developed reconciliation process to make sure elastic search index document count match to source records.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data.
• Implemented Sqoop to transform the data from Oracle to Hadoop and load back in parquet format
• Developed incremental and complete load Python processes to ingest data into Elastic Search from the oracle database
• Used Hive to analyze data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboard
• Created Hive External tables to stage data and then move the data from Staging to main tables
• Pulled the data from the data lake (HDFS) and massaged the data with various RDD transformations.
• Load the data through HBase into Spark RDD and implement in-memory data computation to generate the output response.
• Continuously tuned Hive UDFs for faster queries by employing partitioning and bucketing.

Environment: Hadoop 2.7, Spark 2.7, Hive, Sqoop 1.4.6, AWS, HBase, Kafka 2.6.2, Python 3.6, HDFS, Elastic Search & Agile Methodology

Hadoop Developer

Confidential

Responsibilities:

• Importing and sending out information into HDFS and Hive utilizing Sqoop.
• Experienced in running Hadoop stream jobs to process terabytes of xml group information with the help of Map Reduce programs.
• Used parquet file format for published tables and created views on the tables.
• In-charge of managing data coming from different sources.
• Support in running MapReduce Programs in the cluster.
• Cluster coordination services through Zoo Keeper.
• Involved in loading information from UNIX document framework to Hadoop Distributed File System.
• Installed, configured Hive and furthermore composed Hive UDFs.
• Automated every one of the jobs for pulling information from FTP server to stack information into Hive tables, utilizing Oozie work processes.
• Writing data to parquet tables both non-partitioned and partitioned tables by adding dynamic data to partitioned tables using Spark.
• Wrote User Defined functions (UDFs) for special functionality for Spark.
• Used SQOOP Export functionalities and scheduled the jobs on daily basis with Shell scripting in Oozie.
• Worked with SQOOP jobs to import the data from RDBMS and used various optimization techniques to optimize Hive and SQOOP.
• Used SQOOP import functionality for loading Historical data present in a Relational Database system into Hadoop File System (HDFS).

Environment: - Hadoop, MapReduce, HDFS, Hive, Java, R, Sqoop, Spark, Sqoop

Data Analyst

Confidential

Responsibilities:

• Involved in understanding the legacy applications & data relationships.
• Identified problematic areas and conduct research to determine the best course of action to correct the data.
• Analyzed problems and solved issues with current and planned systems as they relate to the integration and management of order data.
• Involved in Data Mapping activities for the data warehouse.
• Analyzed reports of data duplicates or other errors to provide ongoing appropriate inter-departmental communication and monthly or daily data reports.
• Monitor for timely and accurate completion of select data elements.
• Collected, analyze, and interpret complex data for reporting and/or performance trend analysis.
• Monitor data dictionary statistics.
• Involved in analyzing and adding new features of Oracle 10g like DBMS SHEDULER, Create Directory, Data pump, and CONNECT BY ROOT in the existing Oracle 10g application.
• Archived the old data by converting them into SAS data sets and flat files.
• Extensively used Erwin tool in forwarding and reverse engineering, following the Corporate Standards in Naming Conventions, using Conformed dimensions whenever possible.
• Enhance smooth transition from legacy to a newer system, through the change management process.
• Planned project activities for the team based on project timelines using Work Breakdown Structure.
• Compare data with original source documents and validate Data accuracy.
• Used reverse engineering to create Graphical Representation (E-R diagram) and to connect to an existing database.
• Generate weekly and monthly asset inventory reports.
• Created Technical Design Documents, Unit Test Cases.
• Written SQL Scripts and PL/SQL Scripts to extract data from the Database to meet business requirements and for Testing Purposes.
• Written complex SQL queries for validating the data against different kinds of reports generated by Business Objects XIR2
• Involved in Test case/ data preparation, execution, and verification of the test results
• Created user guidance documentations.
• Created reconciliation report for validating migrated data.

Environment: UNIX, Shell Scripting, XML Files, XSD, XML, SAS, PL/SQL, Oracle 10g, Erwin 9.5, Autosys.