Data Engineer Resume
Seattle, WashingtoN
SUMMARY
- Experienced as a Data Warehouse consultant spent on projects for different multinational companies with extensive experience in Software Development Life Cycle (SDLC) including System Analysis, Design, Data Modelling, Dimensional Modelling, Requirements Gathering, Implementation and Testing of Enterprise Data Warehousing and OLAP Reporting Applications.
- Good experience of the ETL processes applied to different industries. Data Warehousing experience using Informatica Power Center (Workflow Manager, Workflow Monitor, Repository Manager, Source Analyser, Warehouse Designer, Mapping Designer, Mapplet Designer, Transformation developer), Data mart, ETL, OLAP, OLTP.
- Experience in Development of Informatica Mappings, corresponding Sessions and Workflows.
- Experience in creating the resource groups, setting up the subscription, storage account and accessing the key vaults from the account.
- Experience in using Microsoft Systems Engineer in supporting environments that including Windows Server, Azure, Hyper - V, VMware, Active Directory Services, etc.
- Experience inAzureDevelopment, worked onAzure web application,App services,Azure storage,Azure SQL Database,Virtual machines,Fabric controller,Azure AD, Azure search, and notification hub.
- Experience on Migrating SQL database toAzure data Lake, Azure data lake Analytics,Azure SQL Database, Data BricksandAzure SQL Data warehouseand controlling and granting database accessandMigrating On premise databases toAzure Data Lake storeusing Azure Data factory.
- Experience in database design and development using SQL Azure, Microsoft SQL Server, Microsoft Access.
- Experience in Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
- Experience in Testing Azure Data components such as Azure SQL Database, Azure SQL Warehouse, SYNAPSE Analytics.
- Experience in writing the test cases to meet the functionality of the business requirements.
- Performed end to end testing on delta tables on azure sql.
- Responsible for analysis of current programs including performance, diagnosis and troubleshooting of problem programs in Azure
- Run and validate current scripts provided and modify, expand, and improve necessary test scripts for reuse and automation purpose.
- Analysed the existing Teradata scripts to enhance the performance of the components and updated the scripts by understanding the functionality of the scripts.
- Experience in performance tuning of Informatica Sources, Targets, Mappings and Sessions.
- Extensively worked with XML files as a source and target, used transformations like XML generator and XML parser to transform XML files, and used oracle XML type to store XML files.
- Extracted data from Homogeneous and Heterogeneous sources and developing the mapping needs and to push the data as per ETL Specification document into Target Tables.
- Has good hands-on knowledge in configuring various transformations like Source Qualifier, Expression, Filter, Joiner, Aggregator, Router, Unconnected and Connected Lookups, Update Strategy, Sequence Generator, Stored Procedure.
- Comfortable in implementing the CDC (Change data capture) for the slowly changing dimensions of types SCD-Type1, SCD-Type2, SCD-Type3.Used effective Start-Date, End-Date, Version, flagging to capture the change for the records.
- Familiar with end to end of Agile and Scrum as sprint refinement, planning, review, daily stand-up’s, retro.
TECHNICAL SKILLS
ETL: Informatica Power Center Informatica Power Exchange, IDQ, IICS, Azure Cloud, Alteryx, Data Lake, Data Bricks
Databases: Oracle 12g/11g/10g, DB2, SQL server 2008R2/2012, MS Access and Teradata, Snowflake
Interfaces: SQL server Management Studio, SQL server Query Analyzer, TOAD, SQL Loader
Scheduling: Informatica Scheduler, Autosys, ESP, Control-M, TWS,Tidal
Concepts: RDBMS.Data warehousing, Data marts
BI Tools: Qlik, Tableau, Alteryx
PROFESSIONAL EXPERIENCE
Confidential, Seattle, Washington
Data Engineer
Responsibilities:
- Extensive hands-on experience with Azure, Databricks, Data Lake, Spark, and Python
- Experience on Migrating SQL database toAzure data Lake, Azure data lake Analytics,Azure SQL Database, Data BricksandAzure SQL Data warehouseand controlling and granting database accessandmigrating on premise databases toAzure Data Lake storeusing Azure Data factory.
- Worked on could infrastructure on Azure which include Blob storage, Data Factory V2, Databricks, Data Lake.
- Migrated the data from on-premises data warehouse to Azure SQL Datawarehouse.
- Worked on data modelling of DB objects on Azure DW.
- Worked on testing the delta tables and adf pipeline in azure to verify the end-to-end process.
- Developed Spark applications usingPysparkandSpark-SQLfor data extraction, transformation, and aggregation from multiple file formats for analysing & transforming the data to uncover insights into the customer usage patterns.
- Created pipelines and the datasets in data factory with the copy activities.
- Processed event-based schedules to run the pipelines.
- Understanding of the job clusters and interactive clusters and worked with admin teams to make use of given clusters nodes, worker type, driver type along with run time versions and autoscaling, runtime instances.
- Created standard data bricks notebooks to run and validate the data queries and executed the Scala code from data bricks.
- Created an automated notebooks to read the code from the storage CSV files and update the tables.
- Created the linked servers to connect to the multiple databases.
Environment: Azure Data Factory (ADF), Azure SQL, Data Bricks, DataLake, MS SQL Server
Confidential, Tampa, Florida
Data Engineer
Responsibilities:
- Worked closely with business clients to understand the requirement and created mapping based on that.
- Extensively used ETL to load data using Power Centre from source systems like SAP Tables into staging tables and load the data into the target database Oracle and Analysed the existing systems and made a Feasibility
- Extensively worked with Designer tools like Source Analyser, Target designer, Mapping designer, mapplet designer, Transformation Developer.
- Transformed the data based on business requirement using Informatica Power Center and loaded data into target, by creating sessions followed by workflows.
- Designed ETL processes and develop source-to-target data mappings, integration workflows and load processes.
- Followed and maintained source definitions, transformation rules and targets definitions using Informatica repository Manager. Used transformations like Sorter, Joiner, Lookup, Filter, Aggregator, Rank, etc...
- Troubleshooting the Informatica Mappings and validating the data based on the User Interface.
- Experience in using Microsoft Azure tool to maintain and track the tasks and code migration for developers.
- Created pipelines and data flows in Azure
- Worked on creating the ADF Data pipelines for copy activity in Azure
- Created the incremental load using ADF activities to capture the changed records.
- Comfortable with Azure SQL to create tables, views.
- SQL performance tuning techniques including query plan understanding and management, SQL tracing, DB stats in Azure SQL
Environment: Informatica Power Center 10.0.1, Oracle 12g, MS Access, MS SQL Server, Unix, Shell Scripting, Toad for Oracle, WinSCP, ADF
Confidential, Los Angeles, California
ETL Developer
Responsibilities:
- Worked closely with business analyst and Data Warehouse architect to understand the source data and the need of the Warehouse.
- Involved in Unit and Integrating testing of Informatica Sessions, Batches, and the Target Data
- Involved in migration of mappings and sessions from development repository to test and production repository.
- Running the daily jobs for Raw and clean data files and loading it to the ABC tables.
- Created the Autosys jobs and scheduled those as per the requirement.
- Implementing the code in wherescape red and sending the data to the extract files.
- Extensively involved in tuning the mappings, sessions, and the Source Qualifier query.
- Designed and created ETL mappings using Informatica mapping designer.
- Worked extensively on performance tuning by making changes to SQL in source qualifier.
- Creation of facts and dimensions according to the business requirements.
- Experienced in identifying and documenting data integration issues, challenges such as duplicate data, non-conformed data, and unclean data.
- Worked with business clients to understand the requirement and created mapping in IICS based the business requirement.
- Extensively used ETL to load data using IICS from source systems like SAP Tables into staging tables and load the data into the target database Snowflake and Analysed the existing systems and made a Feasibility
- Have an exposure to WSDL files in IICS to read data from third party websites through API calls and load it into snowflake.
- Worked on data modelling of DB objects on Azure DW.
Environment: Informatica Power Center 10.0.1, IICS, Snowflake, Wherescape 2.1.0, Autosys, Oracle 12g, MS Access, MS SQL Server 2008, Unix, Shell Scripting, Greenplum, DB2
Confidential, Columbus, Ohio
ETL\Data Warehouse Developer
Responsibilities:
- Extract ETL jobs from sources like JD Edwards SQL server XML, CSV file, Excel file.
- Validated, debugged old Mappings tested Workflows & Sessions and figured out the better technical solutions. Identified the Bottlenecks in old/new Mappings and tuned them for better Performance.
- Involved in profiling, cleansing, standardization, and Match/Merge.
- Involved in creating IDQ detail level design plans.
- Design and development of Informatica Data quality plans.
- Worked on Profiling Report, Expression Rules, Creating and Running Scorecards on claims.
- Worked with IDQ toolkit, Analysis, datacleansing, data matching, dataconversion, exception handling, and reporting and monitoring capabilities of IDQ.
- Identifying and acquiring new data sources.
- Creating and maintaining consistent referencedataand masterdatadefinitions.
- Developed reusable mapplet called as plans in IDQ and imported into power centre.
- Extensively worked with Designer tools like Source Analyser, Target designer, Mapping designer, mapplet designer, Transformation Developer
- Created Batch scripts for active batch job scheduling and supporting them.
- Extensively used informatica PowerCenter to extract data from various sources, which included Flat files, SQL server, Oracle, Ma-Access, and XML.
- Developed various complex SCD Type1/Type2 Mappings to maintain the data history in different layers. Used Mapping & Sessions Variables/Parameters, Parameters files, Reusable Transformations & Mapplets to maintain the life cycle development. Designed and coded of major change requests as per new requirements.
Environment: Informatica Power Center 10.0.1 and 9.5.1, Data Quality, Power Exchange, AutoSys, Oracle 10g/11g, MS Access, MS SQL Server 2008, Unix, Shell Scripting, Perl Scripting