Data Engineer Resume

PROFESSIONAL SUMMARY:

Currently associated with confidential and having experience in Designing and Developing complex pipeline for the ETL process in AWS, Spark Environment and Snowflake Datawarehouse. Responsible to maintain Snowflake as Admin, Design and develop new Data Model in Snowflake and Redshift. Working in Spark with Scala and Databrick for every day scheduled job.
10 years of experience as ETL and Data Engineer in multiple data warehouse and environment.
Extensive knowledge of design and development in AWS components such as EC2, S3, IAM, EBS etc.
Worked in multiple project as Snowflake Architect.
Working knowledge of all cloud Environment such as Azure and GCP.
Design and develop Spark jobs with Scala and python.
Experience building and maintaining Data Pipeline in Airflow and Nifi Job Scheduler.
Develop and deploy the docker of Airflow, Metabase, Nifi and many more in AWS Ec2 instances.
Develop and schedule the spark job in Databricks
Develop, Deploy and Maintain Snowflake Data Model.
Extensive experience in working with multiple Database Hive, MYSQL, Progress 4GL, HBASE, mongo DB.
Working experience in Agile, Waterfall and Agile Hybrid model.
Experience in developing Front End Web App using React and JavaScript.

PROFESSIONAL EXPERIENCE:

Confidential

Data Engineer

Responsibilities:

Overview
Project Details
Design Develop and Maintain Data Pipeline.
Design End to End Data pipeline in Airflow to fetch data from API, AWS S3, Different Database(Postgres, MongoDB, Snowflake, etc), model Data and load to end DW. To Be exposed to Tableau for BI tools.
Technologies and Tools used
SQL, Python, Scala, Snowflake, Airflow, Nifi, AWS, DataBricks

Confidential

Developer

Responsibilities:

Corporate Technology Application Development teams of Confidential support a large number of diverse applications across the Corporate Technology department providing development and maintenance services to business units, having a plethora of very complex and sophisticated applications with an Enterprise - wide reach.
A self-service tool using SDE has been developed, that will enable the business users to customize and create the ad-hoc reports on their own through Batch Jobs.The data from the source system (IFAST Mainframe Application) has been extracted (Data Extraction) and pushed to the BIG DATA platform through the sqoop tool.
The extracted data (full refresh file and delta file) has been loaded into a big data platform and the required views (using HIVE and BigSql) have been created (Data Aggregation). These views will be read by the Qlikview tool for creating the reports.
Technologies and Tools used

Confidential

Application Developer

Responsibilities:

Project work involved working within the IT team of clients and developing modules based on the requirement provided by business users.
Maintaining the End to End Data Pipeline.
SnowFlake DBA jobs to maintain the warehouse, users, optimizing the codes, Clustering and Re-Clustering and to always work on cost-saving new approaches.
Interacting directly with business users to gather requirements and report progress.
Providing KT of the project to the Production Management team.
Support system testing and defect fixing.
Reporting via regular client calls and a team meeting to brief about projects progress and proactively suggest process changes to enhance efficiency.
Working independently and in collaboration with the release management team in order to ensure a successful and smooth release.
Working on automation of manual tasks to improve efficiency.
Develop spark script and HQL scripts.
Design and create HIVE data structures.
Optimize Batch jobs.

TECHNICAL SKILLS:

Programming languages
Databases& Datawarehouse
Technology& Tools
Servers
Scala, Python, SnowSQL, SQL, Shell Script, JavaScript
SnowFlake, MS SQL, Progress 4GL, PostgresSpark, Hive, Shell Script, Airflow, Docker, Nifi, DataBricks
Linux