We provide IT Staff Augmentation Services!

Data Engineer Resume

5.00/5 (Submit Your Rating)

St Louis, MO

SUMMARY

  • Skilled and enthusiastic professional Data engineer wif around 7 plus years of as a Data Engineer, Analyst & Operations wif expertise in Design, Develop and Implement projects wif great analytical & Technical skills in Cloud platforms like Azure & AWS. Looking forward to working in a challenging, well organized, motivated, and innovative environment which gives abundant potential opportunities to deliver excellence wif continuous learning and career growth.
  • Experience inData Modeling, Database Design, SQL Scripting, Development, and Implementation of Client - Server & Business Intelligence(SQL, SSIS, SSAS)applications.
  • Extensive experience in working wif AWS cloud Platform (EC2, S3, EMR, Redshift, Lambda and Glue).
  • Experienced in scripting T-SQL queries and writing and debugging Stored Procedures and Triggers in Azure SQL and Microsoft SQL Server.
  • Experience inData Extraction, Transforming and Loading(ETL) between Homogenous and Heterogeneous System using SSIS, Hive, Spark etc.
  • Experience inDesigning, Developing, Deploying Azure Data Factory Pipelines to orchestration teh data between various sources to destinations and integrate teh further data processing systems.
  • Proficient knowledge of Data Warehousing methodologies and concepts, including star schemas, snowflakes schemas, dimensional modeling and ETL processes.
  • Extensive knowledge ofdesigning, developing, and deployingvarious kinds of reports wifTableau and Power BItools usingrelational and multidimensional data.
  • Experienced working wif various services in Azure like Azure Data Lake to store, process and analyze data using multiple compute features like Databricks, Synapse and Data Lake Analytics.
  • Experienced in SDLC, Agile Methodologies of operating projects/initiatives.
  • Excellent skills in a wide variety of technologies and a proven Ability and willingness to learn new technology and deliver results in a timely manner.
  • Sound knowledge and hands on experience in SQL, Spark, and Python.
  • Lead, Design and developed ML systems for predicting and control of bio-chemical reaction dynamics (Penicillin production).
  • Excellent organizational skills and ability to prioritize workload. Problem solving, Strong work ethics, critical thinking, and collaborative methodologies to coordinate wif teh teams to streamline teh efforts as needed.
  • A well-organized, goal-oriented, highly motivated TEMPeffective team player.
  • Empowered to be as continuous learner and develop my interpersonal skills and tech knowledge strongly.

TECHNICAL SKILLS

Databases: SQL Server, MySQL, Aurora, Redshift, NoSQL(DynamoDB)

BI tools: Tableau, Power BI

Cloud: AWS (Amazon EC2, Amazon EMR, Amazon LAMBDA, Amazon GLUE, Amazon S3, Amazon ATHENA)Azure (Azure Data Lake, Azure Data Factory, Azure Databrick, Azure SQL Database, Azure SQL data Warehouse)

Big Data: Spark, Kafka, Airflow, Hadoop

Algorithms: ML and DNN algorithms

Programming Languages: SQL, Python, JavaScript

Operating Systems: Windows, Linux

PROFESSIONAL EXPERIENCE

Confidential, St.Louis, MO

Data Engineer

Responsibilities:

  • Designed and implemented dimensional models, data pipelines and BI solutions for key initiatives as per client requirements by consolidating data sources from various third-party vendors. Solutions halped organizations realize teh financial KPIs (coupon usage, trend, forecast etc..) and build a campaign strategy by marketing.
  • Worked wif finance team closely to deliver prepaid voucher sales, trends and perform monthly audits, cost forecast and reconciliations to close ledger books on time. Data pipelines halped teh finance team to view teh KPIs for making informed decisions.
  • Actively involved in all aspects of ETL and data integration processes, development including acquisition, staging, transformations, cleansing and maintenance.
  • Written AWS Lambda code in Python for nested Json files, converting, comparing, sorting etc.
  • Created AWS Data pipelines using various resources in AWS including AWS API Gateway to receive response from AWS Lambda and retrieve data from Snowflake using Lambda function and convert teh response into Json format using database as Snowflake, DynamoDB, AWS Lambda function and AWS S3.
  • Migrated an existing on-premises application to AWS.
  • Used AWS services like EC2 and S3 for small data sets processing and storage.
  • Maintained teh Hadoop cluster on AWS EMR.
  • Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's
  • Assessed and evaluated teh process to migrate pipelines, compute, and storage from on-premises to AWS cloud for building scalable, managed, cost-TEMPeffective large-scale distribution systems.
  • Experienced in Providing support on AWS Cloud infrastructure automation wif multiple tools including Gradle, Chef, Nexus, Docker and monitoring tools such as Splunk and CloudWatch.
  • Written complex to stored procedures to handle teh incremental data and doing a merge process to handle SCD type -II.
  • Actively worked on implementing 3NF and dimensional data models. Took part in building ad-hoc and canned reports as needed for business users using Tableau.
  • Support Ad-hoc requests using T-SQL queries for reporting from SQL Server

Confidential, Atlanta, GA

Data Engineer

Responsibilities:

  • Designing and implementing pipelines in cloud for multiple structured & semi structured data sources utilizing azure services.
  • Develop notebooks using SQL, Spark, Python to process teh data in Azure Data Bricks Clusters sourced from Data Lake Store and structured teh data to consume for multiple reporting solutions.
  • Created Pipelines in Azure Data Factory (ADF) using Linked Services, Datasets, Pipeline to extract, transform and load data from different sources like Azure SQL, Blob storage, Azure SQL DW.
  • Created Application Interface Document for teh downstream to create new interface to transfer and receive teh files through Azure Data Share.
  • Explored teh options of Power BI embedded models to determine teh optimal solution for hosting larger datasets for quick insights to make data driven decisions.
  • Optimizing Python, Spark, and SQL notebooks to gradually reduce teh intake and processing times for analytical data models.
  • Scheduling Spark/Scala jobs using Oozie workflow in Hadoop Cluster and generated detailed design documentation for teh source-to-target transformations.
  • Developed Kafka consumer’s API in python for consuming data from Kafka topics.
  • Used Kafka to consume XML messages and Spark Streaming to process teh XML file to capture UI updates.
  • Created live real-time Processing and core jobs using Spark Streaming wif Kafka as a data pipe-line system.
  • Worked extensively wif Sqoop for importing and exporting teh data from HDFS to Relational Database systems (RDBMS) and vice-versa.
  • Designing and developing Tabular cubes wif complex measures and dimensions for reporting needs.
  • Build multiple pipelines using Azure Data Factory services to extract and load data from On-prem to Azure sources, also used teh Transformation features to structure teh data based on teh business requirements.
  • Cloud migration from OnPrem to Azure SQL database where we have to read from Blob storage resulting in annual savings of $300K and an increase in performance by 20%
  • Enforced strong data quality patterns in pipelines and reduced errors or populating incorrect numbers before they could occur at teh reporting layer.
  • Enforced strong data architecture, data modeling and data governance principals to implement teh systems at scale and maintain teh single source of truth where applicable.

Confidential

Data Engineer

Responsibilities:

  • Launching and Setup of HADOOP/ HBASE Cluster which includes configuring different components of HADOOP and HBASE Cluster.
  • Hands on experience in loading data from UNIX file system to HDFS.
  • Experienced on loading and transforming of large sets of structured, semi structured and unstructured data from HBase through Sqoop and placed in HDFS for further processing.
  • Installed and configured Flume, Hive, Pig, Sqoop and Oozie on teh Hadoop cluster.
  • Managing and scheduling Jobs on a Hadoop cluster using Oozie.
  • Involved in creating Hive tables, loading data, and running hive queries in those data.
  • Extensive Working knowledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
  • Launching Amazon EC2 Cloud Instances using Amazon Images (Linux/ Ubuntu) and configuring launched instances wif respect to specific applications.
  • Involved in writing optimized Pig Script along wif involved in developing and testing Pig Latin Scripts.
  • Good knowledge in writing Pig's Load and Store functions.

Confidential

Full Stack Developer, Data Engineer

Responsibilities:

  • Developed frontend and backend modules using Python on Django Web Framework.
  • Used Django framework for application development.
  • Designed and developed teh UI of teh website using HTML, AJAX, CSS and JavaScript
  • Worked on CSS Bootstrap to develop web applications.
  • Used update strategy to TEMPeffectively migrate data from source to target.
  • Moved teh mappings from development environment to test environment.
  • Designed ETL Process using Informatica to load data from Flat Files, and Excel Files to target Oracle Data Warehouse database.
  • Interacted wif teh business community and database administrators to identify teh Business requirements and data realties.
  • Created various transformations according to teh business logic like Source Qualifier, Normalizer, Lookup, Stored Procedure, Sequence Generator, Router, Filter, Aggregator, Joiner, Expression and Update Strategy.
  • Created Informatica mappings using various Transformations like Joiner, Aggregate, Expression, Filter and Update Strategy.
  • Improving workflow performance by shifting filters as close as possible to teh source and selecting tables wif TEMPfewer rows as teh master during joins.
  • Used connected and unconnected lookups whenever appropriate, along wif teh use of appropriate caches.
  • Created tasks and workflows in teh Workflow Manager and monitored teh sessions in teh Workflow Monitor.
  • Perform Maintenance, including managing Space, Remove Bad Files, Remove Cache Files and monitoring services.
  • Set up Permissions for Groups and Users in all Development Environments.
  • Migration of developed objects across different environments.
  • Designed and developed Web services using XML and jQuery.
  • Improved performance by using more modularized approach and using more in-built methods.
  • Experienced in Agile Methodologies and SCRUM Process.
  • Maintained program libraries, user’s manuals and technical documentation.
  • Wrote unit test cases for testing tools.
  • Involved in entire lifecycle of teh projects including Design, Development, and Deployment, Testing and Implementation and support.
  • Built various graphs for business decision making using Python matplotlib library.
  • Worked in development of applications especially in UNIX environment and familiar wif all its commands.
  • Handling teh day to day issues and fine tuning teh applications for enhanced performance.
  • Implement code in Python to retrieve and manipulate data
  • Environment: Python, Django, MySQL, Linux, Informatica Power Centre, PL/SQL, HTML, CSS, AJAX, JavaScript, NO SQL, jQuery.

We'd love your feedback!