Sr. Data Engineer Resume
Raleigh, NC
SUMMARY
- Over 7+ years of working experience as Data Engineer with high proficient noledge in Data Modeling and Data Analysis.
- Experience in Google Cloud components, Google container builders and GCP client libraries and cloud SDK’s
- Experience with all stages of teh SDLC and Agile Development model right from teh requirement gathering to Deployment and production support.
- Experience in Big Data/Hadoop, Data Analysis, Data modeling professional with applied information Technology.
- Strong understanding and strong noledge in NoSQL databases like HBase and MongoDB.
- Experience of building Data Warehouse in Azure platform using Azure data bricks and data factory.
- Good working experience on using Sqoop to import data into HDFS from RDBMS and vice - versa.
- Experience in understanding business requirements for analysis, database design & development of applications
- Expertise in building PySpark and Spark-Scala applications for interactive analysis, batch processing, and stream processing.
- Excellent working with data modeling tools like Erwin, Power Designer and ER/Studio.
- Extensive experience in creating complex mappings in Talend using transformation and big data components
- Strong experience working with HDFS, MapReduce, Spark, Hive, Sqoop, Kafka, Oozie, Pig and HBase.
- Excellent Programming skills at a higher level of abstraction using Scala and Python.
- Good noledge in streaming applications using Apache Kafka.
- Hands on experience of clouding computing, like using AWS and Google Cloud to run big data analysis
- Extensive experience in Technical consulting and end-to-end delivery with data modeling, data governance.
- Extensive experience in using PL/SQL to write Stored Procedures, Functions and Triggers.
- Experience in Google cloud ecosystem like BigQuery, BigTable, CloudProc and cloud storage.
- Excellent technical and analytical skills with clear understanding of design goals and development for OLTP and dimension modeling for OLAP.
- Experience in extract transform and load (ETL) processing large datasets of different forms including structured, semi - structured and unstructured data
- Hands on experience in Normalization and De-normalization techniques for TEMPeffective and optimum performance in OLTP and OLAP environments.
- Hands on experience in creating various database objects like tables, stored procedures, functions, and triggers using SQL, PL/SQL, DB2.
- Hands on experience on data modeling with Star schema and Snowflake schema.
- Hands on experience of Git to track changes in different versions of source code and coordinating work on those versions with other peoples
- Extensive experience in development of T-SQL, Oracle PL/SQL Scripts, Stored Procedures and Triggers for business logic implementation.
- Experience in Data transformation, Data mapping from source to target database schemas, Data Cleansing procedures.
- Experience in designing interactive dashboards, reports, performing ad-hoc analysis and visualizations using Tableau and Power BI.
- Experience in working with different scripting technologies like Python, UNIX shell scripts.
- Experience working both independently and collaboratively to solve problems and deliver high quality results in a fast-paced, unstructured environment.
- Ability to work TEMPeffectively in cross-functional team environments, excellent communication and interpersonal skills.
TECHNICAL SKILLS
Big Data & Hadoop Ecosystem: MapReduce, Spark 3.3, HBase, Hive 2.3, Sqoop 1.4, Kafka 2.6, Oozie 4.3.
Cloud Platforms: GCP, Google big-query, AWS, EC2, EC3, Redshift & MS Azure
Data Modeling Tools: Erwin R9.8/9.67, ER Studio V17 and Power Designer
Databases: , Microsoft SQL Server 2017, Teradata 15.0, DB2, Oracle 12c, and MS Access
NOSQL Database: Mongo DB, Azure Sql DB, HBase
BI Tools: Tableau 10, SSRS, Crystal Reports, Power BI.
Programming Languages: SQL and PL/SQL
Methodologies: Agile, JAD, System Development Life Cycle (SDLC).
Operating Systems: Microsoft Windows Vista7/8 and 10 and UNIX.
PROFESSIONAL EXPERIENCE
Confidential - Raleigh, NC
Sr. Data Engineer
Responsibilities:
- As a Sr. Data Engineer I was responsible to build a data lake as a cloud based solution in GCP using Apache Spark and Hadoop.
- Highly Involved into Data Architecture and Application Design using Cloud and Big Data solutions on GCP.
- Involved in Designing and building data pipelines to load teh data into GCP platform.
- Worked on End to End Software Development Life Cycle process in Agile Environment using SCRUM methodologies.
- Provided production support to Power BI users and wrote custom SQL to support business requirements.
- Created Active Batch jobs to automate teh Pyspark and SQL functions as daily run jobs.
- Converted Talend Job lets to support teh snowflake functionality.
- Used rest API with Python to ingest Data from and some other site to BIGQUERY.
- Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries.
- Implemented ETL code to load data from multiple sources into HDFS using Pig Scripts.
- Worked on monitoring and troubleshooting teh Kafka-HDFS data pipeline for real time data ingestion in data lake in HDFS
- Involved in Using DataProc, Big Query to develop and maintain GCP cloud base solution.
- Developed complete end to end Big-data processing in Hadoop eco system.
- Involved in Design and document CI/CD tools configuration management.
- Used Python to preprocess data and attempt to find insights.
- Developed MapReduce programs to parse teh raw data, populate staging tables and store teh refined data in partitioned tables in teh EDW.
- Modified existing Talend mappings to load to Snowflake DB.
- Involved in converting Hive queries into Spark transformations using Spark RDD and Pyspark concepts.
- Highly skilled in GCP cloud Platform.
- Utilized Oozie workflow to run Pig and Hive Jobs Extracted files from MongoDB through Sqoop and placed in HDFS and processed.
- Developed data warehouse model in snowflake for over 100 datasets using whereScape
- Used GIT for code reviews during stages of branching, merging and staging.
- Worked on data pre-processing and cleaning teh data to perform feature engineering and performed data imputation techniques for teh missing values in teh dataset using Python.
- Compared Self hosted Hadoop with respect to GCPs Data Proc, and explored Big Table use cases, performance evolution.
- Imported data from different data source to create Power BI reports and dashboards.
- Created complex program unit using PL/SQL Records, Collection types
- Worked on data cleaning and reshaping, generated segmented subsets using Numpy and Pandas in Python
- Responsible for building scalable distributed data solutions using Hadoop.
- Built, maintained and tested infrastructure to aggregate critical business data into Google Cloud Platform (GCP) Big Query and GCP Storage for analysis.
- Developed stored procedures/views in Snowflake and use in Talend for loading Dimensions and Facts.
- Managed External tables in Hive for optimized performance using Sqoop jobs.
- Developed Spark scripts for data analysis in both python and Scala.
- Involved in teh development of Talend Jobs and preparation of design documents, technical specification documents
- Involved in porting teh existing on-premise Hive code migration to GCP (Google Cloud Platform) BigQuery.
- Handled performance requirements for databases in OLTP and OLAP models.
- Used Google BigQuery to run complex queries to perform ETL.
- Created Tableau dashboards, datasets, data sources and worksheets
Environment: Big Data, Talend, GCP, Power BI, Pyspark, SQL, CI/CD, Snowflake, Rest API, Python, BigQuery, HDFS, MapReduce, Spark, Hive, Pyspark, PL/SQL, Pig, Git, MongoDB, Sqoop, Scala, OLAP, Tableau, Oozie, OLTP.
Confidential - Columbus, OH
Data Engineer
Responsibilities:
- Worked as a Data Engineer designed and deployed scalable, highly available, and fault tolerant systems on Azure.
- Collaborated with and across agile teams to design, develop, test, implement, and support technical solutions in Microsoft Azure Data Platform.
- Collaborated with business and technical subject matter experts to develop graphical representations of data and work flows from a business and system level perspective.
- Implemented teh Big Data solution using Hadoop, Hive to pull/load teh data into teh HDFS system.
- Integrated Custom visuals based on business requirements using power BI desktop.
- Designed and implemented database solutions in Azure SQL Data Warehouse and Azure SQL.
- Developed JSON Scripts for deploying teh Pipeline in Azure Data Factory (ADF) that process teh data using teh SQL Activity.
- Extensively used DataStage for extracting, transforming and loading databases from sources including Oracle and Flat files.
- Generated JSon files from teh JSon models created for Zip Code, Group and Claims using Snowflake DB.
- Used Azure Data Bricks to do ETL (Workspace, Hive, RDD, MapReduce so on).
- Implemented of data movements from on-premises to cloud in Azure.
- Involved in writing SQL-Scripts for validation and writing User defined SQL for DataStage job.
- Built teh model on Azure platform using Python and Spark for teh model development and Dash by plotly for visualizations.
- Implemented Azure Data bricks clusters, notebooks, jobs and auto scaling.
- Worked at conceptual/logical/physical data model level using Erwin according to requirements.
- Analyzed teh SQL scripts and designed teh solution to implement using Scala.
- Involved in developing PySpark to transform data from one Data Lake to other data lake.
- Loaded and transformed large sets of structured, semi structured, and unstructured data
- Designed relational and non-relational data stores on Azure
- Implemented Continuous integration/continuous development best practice using Azure Devops, ensuring code versioning
- Used JIRA to track issues and Change Management.
- Loaded teh tables from teh azure data lake to azure blob storage for pushing them to snowflake
- Extensively used Pig and Hive in teh analysis of data.
- Worked on data processing and transformations and actions in spark by using Python (Pyspark) language.
- Involved in Monitoring end to end integration using Azure monitor.
- Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
- Worked in importing and cleansing ofdatafrom various sources like Teradata, Oracle, and PL/SQL with high volumedata.
- Transform and analyze teh data using HIVE, based on ETL mappings
- Worked on creating tabular models on Azure analysis services for meeting business reporting requirements.
- Involved in Importing and exporting data into HDFS and Hive.
- Involved in working with Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW).
- Used DataStage Designer to extract cleanse transform integrate and load data into teh data Mart
- Implemented Kafka High level consumers to get data from Kafka partitions and move into HDFS.
- Involved in writing T-SQL programming for implement Stored Procedures and Functions for different tasks.
- Worked on managing and reviewing Hadoop log files.
- Used Azure Data Lake as Source and pulled data using Azure Polybase.
- Involved in Azure Data Lake storage gen2 to store excel files, parquet files and retrieve using data using Blob API.
- Defined job work flows as per their dependencies in Oozie.
- Tuned DataStage jobs to enhance their performance.
- Supported cloud-based deployments using containers within Azure hosting environments.
- Built OLAP cubes to carry out multidimensional data analysis.
- Created and published multiple dashboards and reports using Tableau server.
Environment: BigData3.0, DataStage, Azure Data Bricks, Hadoop3.0, SnowflakeDB, Hive2.3, Power BI, Python, Spark, Erwin9.8, SQL, MapReduce, OLAP, Scala, Pyspark, Oracle12c, PL/SQL, JiRA, Teradata15, Kafka1.1, T-SQL, Oozie4.3, Tableau.
Confidential - Columbus, GA
Data Engineer/Data Modeler
Responsibilities:
- Used teh Agile Scrum methodology to build teh different phases of Software development life cycle.
- Worked with data governance team to maintain data models, Data Dictionaries; define source fields and its definitions.
- Worked on scalable distributed data system using Hadoop ecosystem in AWS EMR.
- Involved in requirement gathering and database design and implementation of star-schema, snowflake schema using ER/Studio.
- Worked on AWS Data Pipeline to configure data loads from S3 to into Redshift
- Used AWS Glue for teh data transformation, validate and data cleansing.
- Created and reviewed teh conceptual model for teh EDW (Enterprise Data Warehouse) with business users.
- Skilled in AWS Compute such as EC2, Lambda, and Beanstalk.
- Implemented AWS Lambdas to drive real-time monitoring dashboards from system logs.
- Extracted teh data from other data sources into HDFS using Sqoop
- Designed developed and unit test Informatica Power Center transformations.
- Defined teh strategy and architecture required to integrate data across multiple systems, improve teh existing data warehouse architecture and support our transition to AWS.
- Documented logical, physical, relational and dimensional data models.
- Designed teh Data Marts in dimensional data modeling using star and snowflake schemas.
- Involved in Integration of various data sources like DB2 and XML Files.
- Used AWS glue catalog with crawler to get teh data from S3 and perform SQL query operations
- Worked on data profiling and data validation to ensure teh accuracy of teh data between teh warehouse and source systems.
- Designed and develop end to end ETL processing from Oracle to AWS using Amazon S3, EMR, and Spark.
- Uploaded data to Hadoop Hive and combined new tables with existing databases.
- Involved in Creating, modifying and executing DDL in table AWS Redshift and snowflake tables to load data
- Developed Oracle stored packages, procedures, functions and database triggers using PL/SQL for ETL process.
- Worked in development and maintenance using SQLLoader and Informatica Power Center.
- Created S3 buckets also managing policies for S3 buckets and Utilized S3 bucket and Glacier for storage and backup on AWS.
- Transferred data from AWS S3 to AWS Redshift.
- Monitored teh Data quality and integrity of data was maintained to ensure TEMPeffective functioning of department.
- Created Hive tables on top of HBase using Storage Handler for TEMPeffective OLAP analysis.
- Created external tables with partitions using AWS Athena and Redshift
- Worked on Normalization and De-Normalization techniques for both OLTP and OLAP systems.
- Involved in Importing and exporting data into HDFS and Hive using Sqoop.
- Created teh DDL scripts using ER/Studio and source to target mappings to bring teh data from Source to teh warehouse.
- Created databases, tables, stored procedures, DDL/DML triggers, views, functions and cursors.
- Implemented teh Slowly changing dimension scheme (Type II & Type I) for most of teh dimensions.
Environment: ER/Studio, AWS Lambda, SQL,AWS EC2, Oracle, Informatica Power Center, AWS Athena, PL/SQL, HDFS, Sqoop, EDW, DB2, Hive, Oracle, OLAP, HBase and OLTP.
Confidential, Highland park, NJ
Data Analyst/Data Modeler
Responsibilities:
- Worked with Data Analyst/Data Modeler for requirements gathering, business analysis and project coordination.
- Involved in story-driven agile development methodology and actively participated in daily scrum meetings.
- Created dimensional model for teh reporting system by identifying required dimensions and facts using Power Designer.
- Participated in JAD sessions, gathered information from Business Analysts, end users and other stakeholders to determine teh requirements.
- Defined and deployed monitoring, metrics, and logging systems on AWS.
- Performed Data Analysis, Data Migration and data profiling using complex SQL on various sources systems including Oracle and Teradata.
- Added support for Amazon AWS S3 and RDS to host static/media files and teh database into Amazon Cloud.
- Worked in generating and documenting Metadata while designing OLTP and OLAP systems environment.
- Developed all teh required stored procedures, user defined functions and triggers using T-SQL and SQL.
- Produced report using SQL Server Reporting Services (SSRS) and creating various types of reports.
- Written PL/SQL Packages and Stored procedures to implement business rules and validations.
- Worked in data management performing data analysis, gap analysis, and data mapping.
- Involved in developing teh data warehouse for teh database using teh Ralph Kimball's Dimensional Data Mart modeling methodology.
- Implemented teh Slowly changing dimension scheme (Type II) for most of teh dimensions.
- Involved inNormalization/De-normalization, Normal Form and database design methodology.
- Used extensively BaseSAS,SAS/Macro,SAS/SQL, and Excel to develop codes and generated various analytical reports.
- Worked with teh ETL team to document teh transformation rules for data migration from OLTP to Warehouse environment for reporting purposes.
- Used forward engineering approach for designing and creating databases for OLAP model.
- Developed teh code as per teh client's requirements usingSQL,PL/SQLand Data Warehousing concepts.
- Developed teh required data warehouse model using Star schema for teh generalized model
- Managed teh infrastructure required for ETL processing of data based on defined schema using appropriate AWS technologies
- Enforced referential integrity in teh OLTP data model for consistent relationship between tables and efficient database design
- Created graphical representation in teh form of Entity Relationships and elicit more information.
- Developed and maintained data dictionary to create metadata reports for technical and business purpose.
Environment: Power Designer, Amazon Redshift, Oracle, SQL, Teradata, PL/SQL, OLAP, OLTP, T-SQL, ETL and MS Excel.
Confidential - Plano TX
Data Analyst
Responsibilities:
- Performed data analysis and data profiling using complex SQL on various sources systems and answered complex business questions by providing data to business users.
- DevelopedSQLandPL/SQLscripts for migration of data between databases.
- Involved in Analysis of functional and non-functional categorized data elements for data profiling and mapping from source to target data environment.
- Extensively used ETL methodology for supporting data extraction, transformations and loading processing, in a complex EDW using Informatica.
- Performed data reconciliation between integrated systems.
- Worked with T-SQL to create Tables, Views, Triggers and Stored Procedures.
- Created complex SSIS packages using appropriate control and data flow elements with SSIS Standards like error handling, logging and working with package config files.
- Generated customized reports usingSAS/MACRO facility, PROC REPORT, PROC TABULATE and PROC SQL.
- Involved in data mining, transformation and loading from teh source systems to teh target system.
- Generated multiple enterprise reports like SSRS and Crystal reports.
- Retrieved data from cubes and generated OLAP reports using SSRS
- Created SSIS package to extract data from OLTP to OLAP and scheduled jobs to call teh packages and stored procedures.
- Documented data dictionaries and business requirements for key workflows and process points
- Performed Backend Testing by executing SQL queries.
- Involved with data cleansing/scrubbing and validation.
- TEMPEffectively used data blending feature in tableau.
- Used teh automated process for uploading data in production tables by using UNIX.
Environment: PL/SQL, SAS, Informatica, T-SQL, SSIS, SSRS, OLAP, OLTP, Tableau, UNIX.
