We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

5.00/5 (Submit Your Rating)

Chicago, IL

SUMMARY

  • Over 10 years of experience in IT industry and researching wif core competency in development as Data Engineer wif technical skills ofHadoop (Hive, Spark, HDFS, PySpark, AWS, DynamoDB, S3, Glue, Atana, DATABRICKS(SparkEngine), RDS, Python, CI/CD, Apache Kafka, Informatica, UNIXand other client tools and strong noledge in data warehousing.
  • Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Experienced working on Big-Data technologies like Hive, Sqoop, HDFS and Spark streaming Kafka, and No-SQL Databases like MongoDB, Cassandra, Good exposure on cloud technologies like AWS S3, AWS EC2, AWS Redshift, EMR, Atana, Lambda, SNS, SQS, Step Functions and AWS Glue.
  • Proficient in Hive Query language and experienced in hive performance optimization using Static-Partitioning,Dynamic-Partitioning, Bucketing and Parallel Execution concepts.
  • Highly proficient inDataModeling retaining concepts of RDBMS, Logical and PhysicalDataModeling until 3NormalForm (3NF) and MultidimensionalDataModeling Schema (Star schema, Snow-Flake Modeling, Facts and dimensions).
  • Expertise in theDataAnalysis, Design, Development, Implementation and Testing usingData Conversions, Extraction, Transformation and Loading (ETL) and SQL Server, ORACLE and other relational and non-relational databases and experience in developing ETL applications on large volumes of data using different tools: MapReduce, Spark-Scala, PySpark, Spark-Sql, and Pig.
  • Experience wif commercial BI tools, such as Tableau, Qlikview, Cognos, Microstrategy), as well as open source and experienced working wif variousdatasources such as Oracle, SQL Server, DB2, Teradata &Netezza
  • Demonstrable architecture experience, specifically wifin both Standard CRM packages & ideally wifin the financial services industry.
  • Experience wif various ETL engines, such as Informatica, Oracle OWB, and Microsoft (SQL Server Integration Services), as well as strong experience writing the complete ETL process wif PL/SQL, along wif scripting languages (Bash/Python/Perl)
  • Experienced in Consolidating and auditing Metadata from disparate tools and sources, including business intelligence (BI), extract, transform, and load (ETL), relational databases, modeling tools, and third-party metadata into a single repository.
  • Experienced in using distributed computing architectures such as AWS products (e.g. S3, EC2, Redshift, and EMR), Hadoop and TEMPeffective use of Map-Reduce, SQL and Cassandra to solve Big Data type problems.
  • Acquired extensive experience in business analysis, entity relationship, and dimensional data modeling, data warehousing (Kimball Methodology), GUI design, OLTP system development based on relational and hierarchical databases, as well as experience relevant to OOD and structured systems analysis, design, programming, and testing and used complete noledge ofdatawarehouse methodologies (Ralph Kimball, Inmon), ODS, EDW and Metadata repository.
  • Expertise in writing SQL Queries, Dynamic-queries, sub-queries and complex joins for generating Complex Stored Procedures, Triggers, User-defined Functions, Views and Cursors and extensive experience in advanced SQL Queries and PL/SQL stored procedures.
  • Experienced in deploying and scheduling Reports using SSRS to generate all daily, weekly, monthly and quarterly Reports including current status and experienced in designing and deploying reports wif Drill Down, Drill Through and Drop-down menu option and Parameterized and Linked reports.
  • Extensive working experience in Normalization and De-Normalization techniques for both OLTP and OLAP systems in creating Database Objects like tables, Constraints (Primary key, Foreign Key, Unique, Default), Indexes.

TECHNICAL SKILLS

Programming Languages: Python, Pyspark, SQL, PL/SQL, UNIX shell Scripting and PERL

DataWarehousing & Reporting Tools: Erwin, Informatica, Hive, Redshift, Snowflake, Tableau and PowerBI.

Databases: HBase, MongoDB, Teradata, SQL Server, MySQL and PostgreSQL

Cloud Technologies: AWS S3, EMR, Redshift, Lambda, SNS, SQS, AWS Glue, Atana and Step Functions.

Big-Data Technologies: Hive, MondoDB, Cassandra, Oozie, Spark, Spark SQL, Spark Streaming, Kafka, K-SQL, Sqoop and HDFS.

Operating Systems: Windows, Linux and UNIX

ETL Tools: Informatica, SSIS and AWS Glue

Scheduling Tools and CICD: Autosys, Airflow, Docker, Kubernetes, Control M and Oozie.

PROFESSIONAL EXPERIENCE

Confidential - Chicago IL

Sr. Data Engineer

Responsibilities:

  • Analyze, design and build Modern data solutions using AWS Cloud service to support visualization of data and understand current Production state of application and determine the impact of new implementation on existing business processes.
  • Created AWS Glue job for archiving data from Redshift tables to S3 (online to cold storage) as per data retention requirements and automated the process of transforming and ingesting terabytes of monthly data inParquetformat usingKinesis, S3, LambdaandAirflow.
  • Involved in complete remodeling of data processing pipeline by changing the design of the data flow and created pipelines to migrate the data from on-prem resources through the data lake and load the data into the Snowflake Datawarehouse.
  • DevelopedSparkjobs onDatabricksto perform tasks like data cleansing, data validation, standardization, and tan applied transformations as per the use cases.
  • Manage metadata alongside the data for visibility of where data came from, its linage to ensure and quickly and efficiently finding data for customer projects using AWS Data lake and its complex functions like AWS Lambda, AWS Glue.
  • Create Notebooks inDatabricksto pull the data from S3 and process wif the transformation rules and load back the data to persistence area in S3 in Apache Parquet format
  • Worked on predictive and wat-if analysis using Python from HDFS and successfully loaded files to HDFS from Teradata and loaded from HDFS to HIVE.
  • Design and construct of AWS Data pipelines using various resources in AWS including AWS API Gateway to receives response from AWS lambda and retrieve data from snowflake using lambda function and convert the response into Json format using Database as Snowflake, DynamoDB, AWS Lambda function and AWS S3.
  • UsedKafkaandKafka brokers, initiated the spark context and processed live streaming information wif RDD and used Kafka to load data into HDFS and NoSQL databases.
  • Developed Pyspark/Python modules to parse highly complex data types and shared the module wif the rest of the team and implemented complex ETL design patterns in PySpark/Python such as Change Data Capture, Delta load using relevant APIs.
  • Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3 buckets, performed folder management in each bucket, managed logs and objects wifin each bucket.
  • UsedSpark-Streaming APIsto perform necessary transformations and actions on the fly for building the common learner data model which gets the data fromKafkain near real time and persists intoCassandra.
  • Created Snowpipe for continuous data load and created Snow pipe for continuous data load from staged data residing on cloud gateway servers.
  • ImplementedSparkusingpythonandSpark SQLfor faster testing processing the data and developedsparkscripts by usingpython shellcommands as per the requirement.
  • Responsible for estimating the cluster size, monitoring and troubleshooting of the Spark databricks cluster.
  • Created the ETL mapping specs and high-level test plan for Source to target transformations and designed and tested the ETL mapping along wif Development of high-level test scenarios.
  • Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Created data pipeline for different events of ingestion, aggregation and load consumer response data in AWS S3 bucket into Hive external tables in HDFS location to serve as feed for tableau dashboards.
  • Implemented a CI/CD pipeline using Kubernetes in both cloud and on-premises wif GIT, Docker, along wif Jenkins’s plugins.

Environment: Spark, Spark-SQL, Spark Streaming, Kafka, Hive, HDFS, Hadoop, AWS S3, Lambda, Redshift, Glue, Informatica, SNS, SQS, Step Functions, SQL, PySpark, Python, Snowflake, Docker, Kubernetes, RDS, PostgreSQL, MongoDB, Tableau and DynamoDB.

Confidential - Irving TX

Sr. Data Engineer

Responsibilities:

  • Participated in the design, development, and support of the corporate operation data store and enterprise data warehouse database environment and manage the technical delivery of custom development, integrations, and data migration elements of a Microsoft Dynamics CRM implementation.
  • Worked in Agile Data Modeling methodology and creating data models in sprints in SOA architecture and involved wif delivery of complex enterprise data solutions wif comprehensive understandings in Architecture, Security, Performance, Scalability, and Reliability.
  • Build ETL pipeline end to end from AWS S3 to Key, Value store DynamoDB, and Snowflake Datawarehouse for analytical queries and specifically for cloud data.
  • Developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for Analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Involved in building an EnterpriseDataLakeusing Data Factory and Blob storage, enabling other teams to work wif more complex scenarios and ML solutions.
  • Convert into different Data formats for user/business requirements by streaming data pipeline from various sources Snowflake and unstructured data, Dynamo-db.
  • Responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark.
  • Kafka producer API and consumer API configuration, upgrading, rolling upgrade, Topic level Configs, Kafka connect configs, stream configs, consumer rebalancing, operations, replication, message delivery semantics, end - to - end batch compression etc and worked on streaming data to consume data fromKAFKA topicsand load the data to landing area for reporting in near real time
  • Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS.
  • Analyzed existing systems and propose improvements in processes and systems for usage of modern scheduling tools like Airflow and migrating the legacy systems into an Enterprise data lake built on AWS Cloud.
  • Worked wif the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD's, Spark YARN.
  • Used Hive to form an abstraction on top of structured data resides in HDFS and implemented Partitions, Dynamic Partitions, Buckets on HIVE tables.
  • Loaded data intoS3buckets usingAWS GlueandPySpark. Involved in filtering data stored inS3buckets usingElasticsearchand loaded data intoHiveexternal tables and utilizedSpark’sin memory capabilities to handle large datasets onS3 Data-lake.
  • Spun up HDInsight clusters and used Hadoop ecosystem tools like Kafka, Spark and Databricks for real-time analytics streaming, Sqoop, pig, hive and CosmosDB for batch jobs.
  • Responsible for building Confidential data cube using SPARK framework by writing Spark SQL queries in Scala to improve efficiency of data processing and reporting query response time.
  • Implement Continuous Integration and Continuous Delivery process using GitLab along wif Python and Shell scripts to automate routine jobs, which includes synchronize installers, configuration modules, packages and requirements for the applications.
  • Performed ETL operations usingPython, SparkSQL, S3andRedshifton terabytes of data to obtain customer insights and performed interactive Analytics like cleansing, validation and quality checks on data stored inS3buckets usingAWS Atana.
  • Designed and developed T-SQL stored procedures to extract, aggregate, transform, and insert data and developed SQL Stored procedures to query dimension and fact tables in data warehouse.
  • DevelopedTableauvisualizations and dashboards usingTableauDesktop, Tableauworkbooks from multipledatasources usingDataBlending.
  • Incorporating GIT to keeping track of the history, merging code between the different versions of the software, check-in/check-out options.

Environment: AWS S3, Redshift, Lambda, Step Functions, Glue, Airflow, Tableau, Kafka, Spark, Spark Streaming, PySpark, SQL, Atana, Kinesis, K-SQL, hive, HDFS, Hadoop, MongoDB, HBase, RDS, PostgreSQL, DynamoDB, Snowflake, Databricks and Control-M

Confidential -Charlotte NC

Sr. Data Modeler/Data Analyst

Responsibilities:

  • Collaboratively worked wif the Data modeling architects and other data modelers in the team to design the Enterprise Level Standard Data model.
  • Worked wif Business Analysts team in requirements gathering and in preparing functional specifications and translating them to technical specifications.
  • Extensively involved in developing logical and physical data models that fit in exactly for the current state and the future state data elements and data flows using ER/Studio.
  • Utilized existing Informatica, Teradata, and SQL Server to deliver work and fix production issues on time in fast paced environment.
  • Responsible for full data loads from production to AWS Redshift staging environment and worked on migrating of EDW to AWS using EMR and various other technologies.
  • Defined the key columns for the Dimension and Fact tables of both the Warehouse and Data Mart and conduct Design discussions and meetings to come out wif the appropriate Data Mart at the lowest level of grain for each of the Dimensions involved.
  • Created Data Quality Scripts using SQL and Hive to validate successful data load and quality of the data. Created various types of data visualizations using Python and Tableau.
  • Involved in ETL processing using Pig & Hive in AWS EMR, S3 and Data Profiling, Mapping and Integration from multiple sources to AWS S3.
  • Extensively used Star Schema methodologies in building and designing the logical data model into Dimensional Models.
  • Implemented several DAX functions for various fact calculations for efficient data visualization in Power BI.
  • Created and maintained Logical Data Model (LDM) for the project. Includes documentation of all entities, attributes, data relationships, primary and foreign key structures, allowed values, codes, business rules, glossary terms, etc.
  • Writing ad-hoc queries based on schema noledge for various reporting requirements Writing / Tuning data ingestion procedures from external suppliers and partners using PL/SQL, Teradata SQL Assistant, SQL loader, 3rd party tools.
  • Designed Normalized data up to 3rd Normal form and participated in brain storming sessions wif application developers and DBAs to discuss about various de-normalization, Partitioning and Indexing Schemes for Physical Model.
  • Created Hive architecture used for real time monitoring and HBase used for reporting and worked for map reduce and query optimization for Hadoop hive and HBase architecture.
  • Collaborated wif the Reporting Team to design Monthly Summary Level Cubes to support the further aggregated level of detailed reports.
  • Designed Data Flow Diagrams, E/R Diagrams and enforced all referential integrity constraints.
  • Worked on designing Conceptual, Logical and Physical data models and performed data design reviews wif the Project team members.
  • Worked on SQL Server concepts SSIS (SQL Server Integration Services), SSAS (Analysis Services) and SSRS (Reporting Services).
  • Involved in Teradata utilities (BTEQ, Fast Load, Fast Export, Multiload, and Tpump) in both Windows and Mainframe platforms.

Environment: ER/Studio, OLTP, SQL, SSIS, SSAS, SSRS, PL/SQL and 3NF, Hadoop, Hive, Pig, MapReduce, MongoDB, HBase, AWS S3, DAX, AWS Redshift, Python, BigData, Spark, XML, Tableau, SSRS, Teradata, Netezza and Teradata SQL Assistance.

Confidential -Knoxville TN

Sr. Data Modeler/ Data Analyst

Responsibilities:

  • Developed normalized Logical and Physical database models to design OLTP system for finance applications. Created dimensional model for reporting system by identifying required dimensions and facts using Erwin.
  • Developed a Conceptual model using Erwin based on business requirements and produced functional decomposition diagrams and defined logicaldatamodel.
  • Generated ad-hoc SQL queries using joins, database connections and transformation rules to fetch datafrom legacy Oracle and SQL Server database systems
  • Extensive use of DAX (Data Analysis Expressions) functions for the reports and for the tabular models.
  • Used Erwin for reverse engineering to connect to existing database and ODS to create graphical representation in the form of Entity Relationships and elicit more information.
  • Managed the historical data in the data warehouse from various data sources and generated various drill down, drill through, matrix and chart reports using SSRS.
  • Worked on designing the OLAP Model/Dimension model for BI Reporting sourcing from SAP Transactions.
  • Worked wif ETL teams and used Informatica Designer, Workflow Manager and Repository Manager to create source and target definition, design mappings, create repositories and establish users, groups and their privileges.
  • Used forward engineering to create a PhysicalDataModel wif DDL, based on the requirements from the LogicalDataModel and implemented Referential Integrity using primary key and foreign key relationships.
  • Performed Data validation, Data cleansing, Data integrity, Data Quality checking before delivering data to operations, Business, Financial analyst by using Oracle,Teradata.
  • Used Model Mart of Erwin for TEMPeffective model management of sharing, dividing and reusing model information and design for productivity improvement and identified and tracked the slowly changing dimensions and determined the hierarchies in dimensions.
  • Designed high level ETL architecture for overalldatatransfer using SSIS from the source server to theWarehouse and defined various facts and dimensions in thedatamart including Fact less Facts and designed theDataMart defining Entities, Attributes and relationships between them.
  • Worked on designing the OLAP Model/Dimension model for BI Reporting sourcing from SAP Transactions.
  • Consulted wif client management and staff to identify and document business needs and objectives, current operational procedures for creating the logicaldatamodel.
  • Design Source to target mapping as per the requirement of the client and business rules to load thedatafrom Source to Stage and from Stage toDataWarehouse
  • Provided technical guidance for re-engineering functions of Teradata warehouse operations into Netezza.
  • Facilitated in developing testing procedures, test cases and User Acceptance Testing (UAT) and applieddatanaming standards, created thedatadictionary and documenteddatamodel translation decisions and also maintained DW metadata.
  • Extractdatafrom Hadoop environment and NoSQLdatafrom MongoDB to Stage DB's.
  • Developed complex jobs using various stages like Lookup, Join, Transformer, Dataset, Row Generator, Column Generator, Datasets, Sequential File, Aggregator and Modify Stages.
  • Worked on different tasks in Informatica Power Center Workflow Manager like Sessions, Events raise, Event wait, Decision, E-mail, Command, Worklets, Assignment, Timer and Scheduling of the workflow.
  • Conducted complex ad-hoc programming and analysis including statistical programming and analysis and worked wif ad-hoc reporting using Crystal Reportsand T-SQL.
  • Worked on enterprise logicaldatamodeling project (in 3NF) to gatherdata requirements for OLTP enhancements.
  • Involved in extensiveDatavalidation by writing several complex SQL queries and Involved in back-end testing and worked wifdataquality issues.
  • Worked in importing and cleansing ofdatafrom various sources like Teradata, Oracle, flatfiles, SQLServer wif high volumedata.
  • Review and analyze database performance, SSAS cubes and reports generated using SSRS.

Environment: Erwin 9.x, SQL, SQL Server 2008, Rational Rose, Windows XP, Oracle 10g, TOAD, PL/SQL, Flat Files, Teradata, T-SQL, NetezzaA ginity, MDM, informatica Power Centre, DB2, SSRS, SAS, SQL Server, SSIS, SSAS, SSRS, Visio, SharePoint, Informatica, Tableau

Confidential, Dewitt-NY

Sr. Data Analyst

Responsibilities:

  • Createdconceptual,logicaland physical relationalmodelsfor integration and base layer; created logicaland physical dimensionalmodelsfor presentation layer and dim layer for a dimensionaldata warehouse in Power Desinger.
  • Involved in reviewing business requirements and analyzingdatasources form Excel/Oracle SQL Server for design, development, testing, and production rollover of reporting and analysis projects.
  • Analyzing, designing, developing, implementing and maintaining ETL jobs using IBM Info sphere Data stage andNetezza.
  • Extensively worked in Client-Server application development usingOracle 10g, Teradata 14, SQL, PL/SQL, OracleImport and Export Utilities.
  • Coordinated wif DB2 on database build and table normalizations and de-normalizations.
  • Conducted brain storming sessions wif application developers and DBAs to discuss about various de-normalization, partitioning and indexing schemes for Physical Model.
  • Involved in several facets ofMDMimplementations including Data Profiling, metadata acquisition and data migration.
  • Extensively used SQL Loader to load data from the Legacy systems into Oracle databases using control files and used Oracle External Tables feature to read the data from flat files into Oracle staging tables.
  • Involved in extensive Data validation by writing several complex SQL queries and involved in back-end testing and worked wifdataquality issues.
  • Used SSIS to create ETL packages to validate, extract, transform and load data to data warehouse databases, data mart databases, and process SSAS cubes to store data to OLAP databases
  • Strong understanding of Data Modeling (Relational, dimensional,StarandSnowflakeSchema), Data analysis, implementations of Data warehousing using Windows and UNIX.
  • Extensively worked wifNetezzadatabase to implement data cleanup, performance tuning techniques.
  • Created ETL packages using OLTP data sources (SQL Server, Flat files, Excel source files, Oracle) and loaded the data into target tables by performing different kinds of transformations using SSIS.
  • MigratedSQLserver2008 toSQLServer2008 R2 in Microsoft WindowsServer2008 R2 Enterprise Edition.
  • Developing reusable objects like PL/SQL program units and libraries, database procedures and functions, database triggers to be used by the team and satisfying the business rules.
  • Performed data validation on the flat files that were generated in UNIX environment using UNIX commands as necessary.
  • Worked wif NZ Load to load flat file data intoNetezza, DB2 and Architect to identify proper distribution keys forNetezzatables.

Environment: Power Designer, Teradata, Oracle, PL/SQL, MDM, SQL Server 2008, ETL, Netezza, DB2, SSIS, SSRS, SAS, SPSS, Datastage, Informatica, SQL, T-SQL, UNIX, Netezza, Aginity, SQL assistance etc.

We'd love your feedback!