We provide IT Staff Augmentation Services!

Data Engineer Resume

2.00/5 (Submit Your Rating)

St Louis, MO

SUMMARY

  • Skilled and enthusiastic professional Data engineer with around 7 plus years of as a Data Engineer, Analyst & Operations with expertise in Design, Develop and Implement projects with great analytical & Technical skills in Cloud platforms like Azure & AWS. Looking forward to working in a challenging, well organized, motivated, and innovative environment which gives abundant potential opportunities to deliver excellence with continuous learning and career growth.
  • Experience inData Modeling, Database Design, SQL Scripting, Development, and Implementation of Client - Server & Business Intelligence(SQL, SSIS, SSAS)applications.
  • Extensive experience in working with AWS cloud Platform (EC2, S3, EMR, Redshift, Lambda and Glue).
  • Experienced in scripting T-SQL queries and writing and debugging Stored Procedures and Triggers in Azure SQL and Microsoft SQL Server.
  • Experience inData Extraction, Transforming and Loading(ETL) between Homogenous and Heterogeneous System using SSIS, Hive, Spark etc.
  • Experience inDesigning, Developing, Deploying Azure Data Factory Pipelines to orchestration the data between various sources to destinations and integrate the further data processing systems.
  • Proficient knowledge of Data Warehousing methodologies and concepts, including star schemas, snowflakes schemas, dimensional modeling and ETL processes.
  • Extensive knowledge ofdesigning, developing, and deployingvarious kinds of reports withTableau and Power BItools usingrelational and multidimensional data.
  • Experienced working with various services in Azure like Azure Data Lake to store, process and analyze data using multiple compute features like Databricks, Synapse and Data Lake Analytics.
  • Experienced in SDLC, Agile Methodologies of operating projects/initiatives.
  • Excellent skills in a wide variety of technologies and a proven Ability and willingness to learn new technology and deliver results in a timely manner.
  • Sound knowledge and hands on experience in SQL, Spark, and Python.
  • Lead, Design and developed ML systems for predicting and control of bio-chemical reaction dynamics (Penicillin production).
  • Excellent organizational skills and ability to prioritize workload. Problem solving, Strong work ethics, critical thinking, and collaborative methodologies to coordinate with the teams to streamline the efforts as needed.
  • A well-organized, goal-oriented, highly motivated effective team player.
  • Empowered to be as continuous learner and develop my interpersonal skills and tech knowledge strongly.

TECHNICAL SKILLS

Databases: SQL Server, MySQL, Aurora, Redshift, NoSQL(DynamoDB)

BI tools: Tableau, Power BI

Cloud: AWS (Amazon EC2, Amazon EMR, Amazon LAMBDA, Amazon GLUE, Amazon S3, Amazon ATHENA)Azure (Azure Data Lake, Azure Data Factory, Azure Databrick, Azure SQL Database, Azure SQL data Warehouse)

Big Data: Spark, Kafka, Airflow, Hadoop

Algorithms: ML and DNN algorithms

Programming Languages: SQL, Python, JavaScript

Operating Systems: Windows, Linux

PROFESSIONAL EXPERIENCE

Confidential, St.Louis, MO

Data Engineer

Responsibilities:

  • Designed and implemented dimensional models, data pipelines and BI solutions for key initiatives as per client requirements by consolidating data sources from various third-party vendors. Solutions helped organizations realize the financial KPIs (coupon usage, trend, forecast etc..) and build a campaign strategy by marketing.
  • Worked with finance team closely to deliver prepaid voucher sales, trends and perform monthly audits, cost forecast and reconciliations to close ledger books on time. Data pipelines helped the finance team to view the KPIs for making informed decisions.
  • Actively involved in all aspects of ETL and data integration processes, development including acquisition, staging, transformations, cleansing and maintenance.
  • Written AWS Lambda code in Python for nested Json files, converting, comparing, sorting etc.
  • Created AWS Data pipelines using various resources in AWS including AWS API Gateway to receive response from AWS Lambda and retrieve data from Snowflake using Lambda function and convert the response into Json format using database as Snowflake, DynamoDB, AWS Lambda function and AWS S3.
  • Migrated an existing on-premises application to AWS.
  • Used AWS services like EC2 and S3 for small data sets processing and storage.
  • Maintained the Hadoop cluster on AWS EMR.
  • Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's
  • Assessed and evaluated the process to migrate pipelines, compute, and storage from on-premises to AWS cloud for building scalable, managed, cost-effective large-scale distribution systems.
  • Experienced in Providing support on AWS Cloud infrastructure automation with multiple tools including Gradle, Chef, Nexus, Docker and monitoring tools such as Splunk and CloudWatch.
  • Written complex to stored procedures to handle the incremental data and doing a merge process to handle SCD type -II.
  • Actively worked on implementing 3NF and dimensional data models. Took part in building ad-hoc and canned reports as needed for business users using Tableau.
  • Support Ad-hoc requests using T-SQL queries for reporting from SQL Server

Confidential, Atlanta, GA

Data Engineer

Responsibilities:

  • Designing and implementing pipelines in cloud for multiple structured & semi structured data sources utilizing azure services.
  • Develop notebooks using SQL, Spark, Python to process the data in Azure Data Bricks Clusters sourced from Data Lake Store and structured the data to consume for multiple reporting solutions.
  • Created Pipelines in Azure Data Factory (ADF) using Linked Services, Datasets, Pipeline to extract, transform and load data from different sources like Azure SQL, Blob storage, Azure SQL DW.
  • Created Application Interface Document for the downstream to create new interface to transfer and receive the files through Azure Data Share.
  • Explored the options of Power BI embedded models to determine the optimal solution for hosting larger datasets for quick insights to make data driven decisions.
  • Optimizing Python, Spark, and SQL notebooks to gradually reduce the intake and processing times for analytical data models.
  • Scheduling Spark/Scala jobs using Oozie workflow in Hadoop Cluster and generated detailed design documentation for the source-to-target transformations.
  • Developed Kafka consumer’s API in python for consuming data from Kafka topics.
  • Used Kafka to consume XML messages and Spark Streaming to process the XML file to capture UI updates.
  • Created live real-time Processing and core jobs using Spark Streaming with Kafka as a data pipe-line system.
  • Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems (RDBMS) and vice-versa.
  • Designing and developing Tabular cubes with complex measures and dimensions for reporting needs.
  • Build multiple pipelines using Azure Data Factory services to extract and load data from On-prem to Azure sources, also used the Transformation features to structure the data based on the business requirements.
  • Cloud migration from OnPrem to Azure SQL database where we have to read from Blob storage resulting in annual savings of $300K and an increase in performance by 20%
  • Enforced strong data quality patterns in pipelines and reduced errors or populating incorrect numbers before they could occur at the reporting layer.
  • Enforced strong data architecture, data modeling and data governance principles to implement the systems at scale and maintain the single source of truth where applicable.

Confidential

Data Engineer

Responsibilities:

  • Launching and Setup of HADOOP/ HBASE Cluster which includes configuring different components of HADOOP and HBASE Cluster.
  • Hands on experience in loading data from UNIX file system to HDFS.
  • Experienced on loading and transforming of large sets of structured, semi structured and unstructured data from HBase through Sqoop and placed in HDFS for further processing.
  • Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
  • Managing and scheduling Jobs on a Hadoop cluster using Oozie.
  • Involved in creating Hive tables, loading data, and running hive queries in those data.
  • Extensive Working knowledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
  • Launching Amazon EC2 Cloud Instances using Amazon Images (Linux/ Ubuntu) and configuring launched instances with respect to specific applications.
  • Involved in writing optimized Pig Script along with involved in developing and testing Pig Latin Scripts.
  • Good knowledge in writing Pig's Load and Store functions.

Confidential

Full Stack Developer, Data Engineer

Responsibilities:

  • Developed frontend and backend modules using Python on Django Web Framework.
  • Used Django framework for application development.
  • Designed and developed the UI of the website using HTML, AJAX, CSS and JavaScript
  • Worked on CSS Bootstrap to develop web applications.
  • Used update strategy to effectively migrate data from source to target.
  • Moved the mappings from development environment to test environment.
  • Designed ETL Process using Informatica to load data from Flat Files, and Excel Files to target Oracle Data Warehouse database.
  • Interacted with the business community and database administrators to identify the Business requirements and data realties.
  • Created various transformations according to the business logic like Source Qualifier, Normalizer, Lookup, Stored Procedure, Sequence Generator, Router, Filter, Aggregator, Joiner, Expression and Update Strategy.
  • Created Informatica mappings using various Transformations like Joiner, Aggregate, Expression, Filter and Update Strategy.
  • Improving workflow performance by shifting filters as close as possible to the source and selecting tables with fewer rows as the master during joins.
  • Used connected and unconnected lookups whenever appropriate, along with the use of appropriate caches.
  • Created tasks and workflows in the Workflow Manager and monitored the sessions in the Workflow Monitor.
  • Perform Maintenance, including managing Space, Remove Bad Files, Remove Cache Files and monitoring services.
  • Set up Permissions for Groups and Users in all Development Environments.
  • Migration of developed objects across different environments.
  • Designed and developed Web services using XML and jQuery.
  • Improved performance by using more modularized approach and using more in-built methods.
  • Experienced in Agile Methodologies and SCRUM Process.
  • Maintained program libraries, user’s manuals and technical documentation.
  • Wrote unit test cases for testing tools.
  • Involved in entire lifecycle of the projects including Design, Development, and Deployment, Testing and Implementation and support.
  • Built various graphs for business decision making using Python matplotlib library.
  • Worked in development of applications especially in UNIX environment and familiar with all its commands.
  • Handling the day to day issues and fine tuning the applications for enhanced performance.
  • Implement code in Python to retrieve and manipulate data
  • Environment: Python, Django, MySQL, Linux, Informatica Power Centre, PL/SQL, HTML, CSS, AJAX, JavaScript, NO SQL, jQuery.

We'd love your feedback!