OBJECTIVE

• Goal - oriented, dedicated to high levels of customer satisfaction and meeting aggressive business goals. Passionate, motivated with a drive for excellence knowledge in Big data and Hadoop eco systems.

SUMMARY

• Overall 7+ years of experience in IT with 4 years in Hadoop Ecosystem, 3 years in ETL and data with tools such as Teradata, Informatica and related automation tools.
• Worked as Business Systems Analyst, which involved tasks such as data validation, data checks, manipulations, make the data available with right grain by use of sql queries. Basically, looking for answers in the data that makes difference.
• Strong technical background on the analysis of large datasets for the understanding of complex dynamics and their impact on business.
• Have performed activities such as development, cleaning and transforming data and descriptive and predictive analysis and visualizing results to ensure actionable insights.
• Comfortable with technologies such as Hive, Sqoop, Spark, Python, AWS, S3, EMR, Snowflake
• Worked in Agile as well as waterfall methodologies.

TECHNICAL SKILLS

Bigdata: Hadoop, HDFS, MapReduce, Hive, Spark, AWS S3, EMR, EC2, Airflow

Tools: Jenkins, Quality Center, UFT (QTP), Selenium, Informatica, Autosys, Alteryx

Domain: Benefits, Finance, Digital

Languages: Python, Scala, C#, Java, HTML, SQL, Java Script, VB Script, Shell Scripting

Databases: Oracle, Teradata, MySQL, SQL Server, Snowflake

Version Control: Bitbucket, GITHUB.

PROFESSIONAL EXPERIENCE

Confidential, Beaverton, OR

Data Engineer

Responsibilities:

• Work with Product team to understand requirements.
• Developed data ingestion pipelines using Airflow and pyspark and loaded the data into Hive and snowflake tables.
• Implemented workflows using Airflow for managing and scheduling Hadoop jobs.
• Worked on GitHub/Bit Bucket as code repository and Jenkins as deployment to different environments.
• Ingested data from API response (json files) and parsed the json files to store in S3/Hive tables.
• Experience in building real time data ingestion pipelines using NSP ( Confidential Streaming Platform).
• Developed Jenkins pipeline for deployments.
• Developed and designed SPARK jobs for data ingestion/aggregates.
• Used HDFS and AWS S3 as storage to build Hive tables.
• Developed copy scripts to move aggregated/integrated tables data to Snowflake and Athena.
• Experience in working with multiple data formats - Parquet, Avro, Json, Xml, Csv
• Using Confluence for Project Documentation.

Environment: Spark, AWS, EMR, S3, Hive, NSP, KAFKA, Spark Streaming, Python, airflow, Teradata, Presto, Athena, snowflake

Confidential, Beaverton, OR

Data Engineer/Snowflake Engineer

Responsibilities:

• Work with Product team to understand requirements.
• Good experience on Airflow dag creation.
• Very Good experience on Hive and Spark jobs creation.
• Monitor, support and troubleshoot Airflow and Spark jobs.
• Hands on experience on Bitbucket and GitHub.
• Hands on experience using Jenkins.
• Creating Spark jobs efficiently with data cache, coalesce, repartition methods to improve performance.
• Good experience on Agile.

Environment: Spark, AWS, EMR, S3, Hive, Python, airflow, Teradata, Sqoop, snowflake

Confidential, Hillsboro, OR

Data Engineer/Snowflake Engineer

Responsibilities:

• Develop, monitoring and troubleshoot Hive & Spark on AWS EMR.
• Working on ingesting data from Teradata to snowflake.
• Snowflake user support.
• Co-ordinated with report developers to refresh tableau data sources.
• Using GitHub/Bit Bucket as code repository.
• Sending airflow production jobs status report to stakeholders.
• Update confluence page with job status.
• Building Jenkin pipelines as part of DevOps model.
• Developed Canary queries and integrated with MCF to find issues before business users.

Environment: Spark, AWS, EMR, S3, Hive, Python, airflow, Teradata, Sqoop, snowflake

Confidential, Beaverton, OR

Data Engineer

Responsibilities:

• Monitoring and troubleshoot Hive & Spark on AZURE.
• Developed wrapper scripts in Shell and Python.
• Co-ordinated with report developers to refresh tableau data sources.
• Using GitHub/Bit Bucket as code repository.
• Sending airflow production jobs status report to stakeholders.
• Prepare production jobs daily status report.
• Prepare failed jobs status report and reason for failures.
• Update confluence page with job status.
• Using Confluence for Project Documentation.
• Building Jenkin pipelines as part of DevOps model.
• Developed Canary queries and integrated with MCF to find issues before business users.

Confidential, Beaverton, OR

Product support Analyst

Responsibilities:

• Collaborate with users and technical teams to understand requirements.
• Monitoring and troubleshoot Hive & Spark on AWS EMR.
• Scheduled daily jobs in Airflow.
• Developed DAG’s for daily production run.
• Developed wrapper scripts in Shell and Python.
• Co-ordinated with report developers to refresh tableau data sources.
• Using GitHub/Bit Bucket as code repository.
• Helped the tableau team integrate with HiveServer2 for reporting.
• Prepare production jobs daily status report.
• Prepare failed jobs status report and reason for failures.
• Update confluence page with job status.
• Using Confluence for Project Documentation.

Environment: Spark, AWS, EMR, S3, Hive, Python, airflow, Teradata, Sqoop, Snowflake

Hadoop Developer

Responsibilities:

• Data ingesting from Teradata, Oracle and SAP, GSS JSON files into AWS S3.
• ETL process implemented in Spark using data frames on EMR.
• Each etl process implemented with bop and eop quantities to avoid multiple query executions while reporting.
• Creating Airflow DAGs for backfill, history and incremental loads.
• Creating Spark jobs efficiently with data cache, coalesce, repartition methods to improve performance.
• Creating HDFS staging tables to improve throughput to write into S3 parquet files.
• Developing Unit test functions in Spark.
• Building Jenkin pipelines as part of DevOps model.
• Developed PyUnit framework for unit testing.
• Developed Canary queries and integrated with MCF to find issues before business users.

Environment: CDH, Spark, AWS, EMR, S3, Hive, Python, Zookeeper, Oracle, Teradata, Sqoop

Confidential

Informatica developer

Responsibilities:

• Handled batch processing for export/import functionalities into various heterogeneous systems.
• Design and development of Informatica mappings to move data from SAP to Teradata.
• Created unit test cases for testing Informatica mappings.
• Involved in batch design.
• Migrating code from one environment to other using labels query and deployment group.
• Helped in setting up development assurance process.
• Created complex sql’s for data validation as well as for data analysis.
• Provide database administration for testing and development database servers. Creation of database objects like tables, indexes, stored procedures and views etc.
• Wrote extensive SQL and PL/SQL Procedures, Functions.
• Monitored database activities, performance, and size. Increase the size if required and identified and resolved performance issues.
• Tuned & Optimized SQL queries using indexes and table partitioning.
• Analyzed the database tables and indexes and then rebuilt the indexes if there is fragmentation in indexes.
• Planned Backup and Recovery using Cold Backup. Responsible for building & deploying the DB Utilities required in QA Environment
• Built the Client Restore/back-up utility for QA Clients to enable QA Automation Executions by bringing the clients to desired App State.
• Responsible for applying the DB dumps in to Development Environment from Production (typically based on Customer Tickets/requests).
• Responsible for applying DB Scripts/Patches as part of Hot Fixes/Version Migrations.

Environment: Teradata R13, Informatica 9.1, unix, SAP BW, Oracle