SUMMARY

• IT professional wif 9+ years of combined experience in teh fields of Data Engineering.
• High Exposure on Big Data technologies and Hadoop ecosystem, in - depth understanding of Map Reduce and Hadoop Infrastructure.
• Expertise in writing end to end Data processing Jobs to analyze data using MapReduce, Spark and Hive.
• Experience wif Apache Spark ecosystem using Spark-Core, SQL, Data Frames, RDD's and noledge on Spark MLLib.
• Hands on experience working Amazon Web Services (AWS) using Elastic Map Reduce (EMR), Redshift, and EC2 for data processing.
• Hands on experience in SQL and NOSQL database such as Snowflake, HBase, Cassandra and MongoDB.
• Hands on experience in using Google Stack driver for monitoring teh logs of GKE and GCP, Big Query, instances and configured alerts from Stack driver.
• Good experience in conducting Joint Application Development (JAD) sessions wif SMEs, Stakeholders
• Hands on experience in writing Ad-hoc Queries for moving data from HDFS to HIVE and analyzing teh data using HIVE QL.
• Excellent noledge in Data Analysis, Data Validation, Data Cleansing, Data Verification and identifying data mismatch.
• Excellent SQL programming skills and developed Stored Procedures, Triggers, Functions, Packages using SQL/PL SQL.
• Experience in BI/DW solution (ETL, OLAP, Data mart), Informatica, BI Reporting tool like Tableau and Power BI.
• Strong experience in Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export.
• Experience in importing and exporting Terabytes of data between HDFS and Relational Database Systems using Sqoop
• Experience in object-oriented analysis and design (OOAD), used modeling language (UML) and design patterns.
• Extensive experience in shell scripting Python & PySpark scripting language.
• Excellent working experience in Scrum/Agile framework and Waterfall project execution methodologies.
• Extensive experience working wif business users/SMEs as well as senior management.

TECHNICAL SKILLS

Big Data & Hadoop Ecosystem: MapReduce, Spark 3.3, HBase 2.3.4, Hive 2.3, Flume 1.9, Sqoop 1.4.6, Kafka 2.6, Oozie 4.3, Hue, Cloudera Manager, Neo4j, Hadoop 3.3, Apache NiFI 1.6

NOSQL Database: Snowflake, Mongo DB, Azure Sql DB, Cassandra 3.11.10

Databases: , Microsoft SQL Server 2017, Teradata 15.0, Oracle 12c, and MS Access

Cloud Platforms: GCP, Google big-query, AWS, EC2, EC3, Redshift & MS Azure

BI Tools: Tableau 10, SSRS, Crystal Reports, Power BI.

Programming Languages: PySpark, Python, SQL, PL/SQL, UNIX shell Scripting, AWK

RDBMS: Microsoft SQL Server 2017, Teradata 15.0, Oracle 12c, and MS Access

Operating Systems: Microsoft Windows 8 and 10, UNIX and Linux.

Methodologies: Agile, RAD, JAD, RUP, UML, System Development Life Cycle (SDLC), Waterfall Model.

PROFESSIONAL EXPERIENCE

Confidential, Worcester, MA

Sr. Data Engineer

Responsibilities:

• Working as Data Engineer to collaborate wif other Product Engineering team members to develop, test and support data-related initiatives.
• Developed understanding of key business, product and user questions.
• Followed agile methodology for teh entire project.
• Defined teh business objectives comprehensively through discussions wif business stakeholders, functional analysts and participating in requirement collection sessions.
• Provided a summary of teh Project's goals, and teh specific expectation of business users from BI and how it aligns wif teh project goals.
• Lead teh estimation, review teh estimates, identify teh complexities and communicate to all teh stakeholders.
• Assemble large, complex data sets that meet functional / non - functional business requirements.
• Migrated on-primes environment on Cloud using MS Azure.
• Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
• Created pipeline jobs, scheduling triggers, mapping data flows using Azure Data Factory (V2) and using Key Vaults to store credentials.
• Performed data flow transformation using teh data flow activity.
• Worked on creating tabular models on Azure analysis services for meeting business reporting requirements.
• Ingested data using Polybase.
• Performed ongoing monitoring, automation, and refinement of data engineering solutions.
• Creating pipelines, data flows and complex data transformations and manipulations using ADF and PySpark wif Databricks.
• Designed relational and non-relational data stores on Azure.
• Performed analyses on data quality and apply business rules in all layers of data extraction transformation and loading process.
• Developed mapping document to map columns from source to target.
• Created azure data factory (ADF pipelines) using Azure polybase and Azure blob.
• Performed ETL using Azure Data Bricks.
• Wrote UNIX shell scripts to support and automate teh ETL process.
• Worked on python scripting to automate generation of scripts. Data curation done using azure data bricks.
• Used stored procedure, lookup, executes pipeline, data flow, copy data, azure function features in ADF.
• Worked on Kafka to bring teh data from data sources and keep it in HDFS systems for filtering.
• Created several Databricks Spark jobs wif Pyspark to perform several tables to table operations.
• Working on building visuals and dashboards using Power BI reporting tool.
• Providing 24/7 On-call Production Support for various applications.

Environment: Hadoop 3.1, Spark 3.1.1, Kafka 3.0, Azure Data Bricks, ADF, Python 2.7, PySpark 2.4, HDFS, ETL, Agile & Scrum meetings

Confidential, North Wales, PA

Data Engineer

Responsibilities:

• As a Data Engineer provided technical expertise and aptitude to Hadoop technologies as they relate to teh development of analytics.
• Assisted in leading teh plan, building, and running states wifin teh Enterprise Analytics Team.
• Provided thought leadership for architecture and teh design of Big Data Analytics solutions for customers, actively drive Proof of Concept (POC) and Proof of Technology (POT) evaluations and to implement a Big Data solution.
• Engaged in solving and supporting real business issues wif your Hadoop distributed File systems and Open Source framework noledge.
• Performed detailed analysis of business problems and technical environments and use dis data in designing teh solution and maintaining data architecture.
• Built teh data pipelines that will enable faster, better, data-informed decision-making wifin teh business.
• Implementing Microservices in Scala along wif Apache Kafka.
• Migrated 9 Microservices to Google Cloud Platform and have one more big release planned wif 4 more Microservices.
• Worked on POC to check various cloud offerings including Google Cloud Platform (GCP).
• Compared Self hosted Hadoop wif respect to GCPs Data Proc, and explored Big Table (managed HBase) use cases, performance evolution.
• Wrote complex Hive queries to extract data from heterogeneous sources (Data Lake) and persist teh data into HDFS.
• Developed data pipelines to consume data from Enterprise Data Lake (MapR Hadoop distribution - Hive tables/HDFS) for analytics solution.
• Involved in various phases of development analyzed and developed teh system going through Agile Scrum methodology.
• Loaded and transformed large sets of structured, semi structured and unstructured data using Hadoop/Big Data concepts.
• Performed Data transformations in Hive and used partitions, buckets for performance improvements.
• Continuously monitor and manage data pipeline (CI/CD) performance alongside applications from a single console wif GCP.
• Designed Google Compute Cloud Data Flow jobs that move data wifin a 200 PB data lake.
• Implemented scripts that load Google Big Query data and run queries to export data.
• Implemented business logic by writing UDFs and configuring CRON Jobs.
• Extensively involved in writing PL/SQL, stored procedures, functions and packages.
• Lead architecture and design of data processing, warehousing and analytics initiatives.
• Developed Spark scripts by using python and bash Shell commands as per teh requirement.
• Used Google SDK and BQ command line tools to extract data from Big Query tables and load into Google Cloud Storage.
• Developed Simple to complex streaming jobs using Python and Hive.
• Optimized Hive queries to extract teh customer information from HDFS.
• Involved in scheduling Oozie workflow engine to run multiple Hive and Pig jobs.
• Analyzed data using Hive teh partitioned and bucketed data and compute various metrics for reporting.

Environment: GCP, Big Query, Hive 2.3, Hadoop 3.0, HDFS, Flume 1.8, Sqoop 1.4, Oozie 4.3, Python, PL/SQL, NoSQL

Confidential, Greensboro, NC

Data Engineer

Responsibilities:

• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Worked wif Data Governance, Data Quality and Metadata Management team to understand project.
• Used SDLC Methodology of Data Warehouse development using Kanbanize.
• Architected, Designed and Developed Business applications and Data marts for reporting.
• Developed Big Data solutions focused on pattern matching and predictive modeling
• Objective of dis project is to build a data lake as a cloud based solution in AWS using Apache Spark.
• Implemented Installation and configuration of multi-node cluster on Cloud using Amazon Web Services (AWS) on EC2.
• Developed reconciliation process to make sureelasticsearchindex document count match to source records.
• Used AWS S3 Buckets to store teh file and injected teh files into Snowflake tables using Snow Pipe and run deltas using Data pipelines.
• Worked on complex SNOW SQL and Python Queries in Snowflake.
• Configured Spark streaming to receive real time data from teh Kafka and store teh stream data to HDFS.
• Developed Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.
• Pulled teh data from data lake (HDFS) and massaging teh data wif various RDD transformations.
• Used AWS Cloud wif Infrastructure Provisioning / Configuration.
• Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting on teh dashboard.
• Involved in PL/SQL query optimization to reduce teh overall run time of stored procedures.
• Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting on teh dashboard.
• Worked on configuring and managing disaster recovery and backup on Cassandra Data.
• Implemented Kafka producers create custom partitions, configured brokers and implemented High level consumers to implement data platform.
• Developed Rest services to write data intoElasticSearchindex using Python Flask specifications
• Developed incremental and complete load Python processes to ingest data intoElasticSearch from Hive.

Environment: Spark 2.7, AWS, EC2, Kafka 2.6, Python 3.6, Scala, Hive, PL/SQL, Cassandra, HDFS and SDLC

Confidential

Data Analyst

Responsibilities:

• Worked extensively along wif business analysis team, scrum masters in gathering requirements and understanding teh workflows of teh organization
• Involved in Data mapping specifications to create and execute detailed system test plans. Teh data mapping specifies wat data will be extracted from an internal data warehouse, transformed and sent to an external entity.
• Analyzed business requirements, system requirements, data mapping requirement specifications, and responsible for documenting functional requirements and supplementary requirements in Quality Center.
• Wrote and executed unit, system, integration and UAT scripts in a data warehouse projects.
• Wrote and executed SQL queries to verify that data has been moved from transactional system to DSS, Data warehouse, data mart reporting system in accordance wif requirements.
• Created teh test environment for Staging area, loading teh Staging area wif data from multiple sources.
• Used and supported database applications and tools for extraction, transformation and analysis of raw data.
• Assisted in defining business requirements for teh IT team and created BRD and functional specifications documents along wif mapping documents to assist teh developers in their coding.
• Involved in building various logics to handle Slowly Changing Dimensions, Change Data Capture, and Deletes for Incremental Loads in to teh Data warehouse.
• Involved in designing fact, dimension and aggregate tables for Data warehouse Star Schema.
• Performed Reverse Engineering of teh legacy application using DDL scripts in Erwin, and developed Logical and Physical data models for Central Model consolidation
• Worked on data profiling and data validation to ensure teh accuracy of teh data between teh warehouse and source systems.
• Monitored teh Data quality of teh daily processes and ensure integrity of data was maintained to ensure TEMPeffective functioning of teh departments.
• Developed data mapping documents for integration into a central model and depicting data flow across systems & maintain all files into electronic filing system.
• Worked on SQL Server concepts SSIS (SQL Server Integration Services), SSAS (Analysis Services) and SSRS (Reporting Services).
• Developed database objects including tables, Indexes, views, sequences, packages, triggers and procedures to troubleshoot any database problems
• Wrote SQL scripts to test teh mappings and Developed Traceability Matrix of Business Requirements mapped to Test Scripts to ensure any Change Control in requirements leads to test case update.
• Involved in extensive data validation by writing several complex SQL queries and Involved in back-end testing and worked wif data quality issues.
• Created or modifying teh T-SQL queries as per teh business requirements.
• Developed and optimized stored procedures for use as a data window source for complex reporting purpose.
• Performed teh batch processing of data, designed teh SQL scripts, control files, batch file for data loading.
• Delivered file in various file formatting system (ex. Excel file, Tab delimited text, Coma separated text, Pipe delimited text etc.)
• Performed ad hoc analyses, as needed, wif teh ability to comprehend analysis as needed

Environment: Sql Server 2008, SQL, XML, ad hoc, Excel 2008, data validation