Etl Tech Lead/hadoop Resume
Richmond, VA
EXPERIENCE SUMMARY:
- 8+ Years of experience in IT - Data warehouse implementations usingInformatica PowerCenter9.x/8.x/7.x,Oracle11g/10g/9i,Teradata,Hadoop,Mainframes, XML and Flat Files.
- Extensive experience in Analysis, Design, Development, Implementation, Enhancement and Support of BI applications which includes strong experience in Data Warehousing (ETL & OLAP) environment as a Data Warehouse Consultant.
- Proficiency in utilizing ETL tool Informatica Power Center 9.x/8.x/7.x for developing the Data warehouse loads with work experience focused in Data Acquisition and Data Integration.
- Expertise in designing confirmed and traditional ETL Architecture involving Source databases Main Frame systems (COBOL files), Oracle, Flat Files (fixed width, delimited), DB2, SQL server, XML and Target databases Oracle, SQL server, XML and Flat Files (fixed width, delimited).
- Have Good understanding of ETL/Informatica standards and best practices, Slowly Changing Dimensions (SCD1, SCD2, and SCD3).
- Strong knowledge of Dimensional Modeling, Star and Snowflake schema. Designed Fact and Dimension Tables as per the reporting requirements and ease of future enhancements.
- Extensive Experience in designing and developing complex mappings applying various transformations such as lookup, source qualifier, update strategy, router, sequence generator, aggregator, rank, stored procedure, filter joiner and sorter transformations.
- Extensive experience in developing the Workflows, Worklets, Sessions, Mappings, and configuring the Informatica Server using Informatica Power Center .
- Excellent knowledge in identifying performance bottlenecks and also in tuning the Mappings and Sessions by implementing various techniques like partitioning techniques and pushdown optimization.
- Experience in creating Sessions, Workflows and Worklet by using the Workflow Manager tools like Task Developer, Worklet and Workflow Designer.
- Experience in optimizing query performance, session performance and fine tuning the mappings for optimum performance.
- Createdreusable transformation and mappletsin the designer using transformation developer and mapplet designer.
- Experience with pre-session and post-session SQL commands to drop indexes on the target before session runs, and then recreate them when the session completes.
- Extensively worked for monitoring scheduled, running, completed and failed sessions. Involved in debugging the failed mappings and developing error handling methods.
- Strong Knowledge in Relational Database Concepts, Entity Relation Diagrams, Normalization and De normalization Concepts.
- Provided 24/7 Active Data Warehouse Production Support as part of on-call rotation for both Incremental and Complete refresh.
- SME for application, provide assistance to support team during issues in production
- Lead and provided guidance to both on-site and off-shore teams for various EDW initiatives with design and coding standards and best ETL practices for using normalized data structures, de-normalized structures, and dimensional structures, best practices for the design, development, deployment, change management, support and successfully present the ETL Data Integration efforts to external departments to influence, appreciate and gain support for the projects from client departments.
- Provided extensive Production Supportfor Data Warehouse for internal and external data flows toTeradata, OracleDBMS fromETL serversvia remote servers.
- Experienced as point of contact to client for enhancement and support related activities.
- Support for code deployment in QA and PROD environment and also responsible for code validation
- Responsible for Production support activities like Jobs monitoring, log the load statistics, analysis and resolve in case of any production issues, coordinate with business to fix any source file issues, coordinate with DBA to resolve any table space or file system related issues.
- Responsible for deliverable out of monthly/Quarterly/Half Yearly/Yearly jobs
- Verystrong SQL skills, under Windows and Unix shell script environment including use of nested table expressions, unions, multi-table joins, scalar functions, outer joins, derived columns and creative use of the EXISTS operator.
- Extensively worked with Teradata utilities likeBTEQ, Fast Export, Fast Load, Multi Loadto export and load data to/from different source systems including flat files.
- Experience in Development and Documentation throughout the entire SDLC using Water fall and Agile methodologies.
- Excellent communication skills, Businessinteraction skillsandstrong interpersonal skillsto deal effectively with customer groups and clients to the top management.
- Knowledge in ETL techniques Analysis and Reporting tools.
- Having Experience with working with Insurance, Banking and Telecom domain clients.
- Excellent team player and self-starter with good ability to work independently and possess good analytical, problem solving and logical skills.
TECHNICAL SKILLS:
Hadoop: HDFS, Pig, Hive, Sqoop and Oozie
ETL tools: Informatica Power Center (9.x/8.x/7.x), Informatica IDQ 9.xInformatica Power Exchange 9.x
Databases: Oracle (9i,10g,11g),DB2,Teredata
Tools: and Utilities: Teradata utilities like BTEQ, Multiload, FastLoad, FastExportSQL Assistant, WINSCP, SQL*PLUS,TOAD
Scheduling Tool: Control M
Reporting Tool: Cognos
Operating Systems: Windows 2000/XP/2007,UNIX and Shell Scripts
PROFESSIONAL EXPERIENCE:
Confidential - Richmond, VA
ETL Tech Lead/Hadoop
Responsibilities:
- Working on building new data warehouse for customer data information.
- Experienced working with Product Owners, Scrum Masters, Interfaced with business analysts.
- Interacted with Business Analyst to gather requirements and translated them into ETL technical specifications.
- Worked with data modelers to understand financial data model and provided suggestions to the logical and physical data model
- Involved in data quality profiling, design, development, unit testing and deployments of ETL Informatica components.
- Document unit test cases and provide QA support to make testers understand the business rules and code implemented in Informatica.
- Extracted data from various sources like XML files, flat files, RDBMS loaded into target data warehouse Oracle.
- Created mapping using various transformations like Joiner, Filter, Aggregator, Lookup, Router, Sorter, Expression, Normalizer, Sequence Generator and Update Strategy, data standardization, address validator etc.
- Implemented SCD Type 1 and SCD Type 2 for loading data into data warehouse dimension tables.
- Implemented error handling for invalid and rejected rows by loading them into error tables.
- Implementing the Change Data Capture Process using Informatica Power Exchange.
- Handled Production issues and monitored Informatica workflows in production.
- Involved in Performance Tuning of ETL code by addressing various issues during extraction, transformation and loading of data.
- Worked with Support team to migrate the ETL code to Production servers from development servers by creating a mapping document and Request for Change (RFC) of the process.
- Used IDQ (Informatica Data Quality) as data Cleansing tool to create the IDQ Mapplets and to correct the data before loading into the Target tables.
- Created IDQ Mapplets for address cleansing, telephone cleansing and SSN cleansing and used them as Informatica Mapplets in Power Center.
- Involved in Knowledge Transfer sessions to Support team after completion of UAT signoff.
- Handled Production issues and monitored Informatica workflows in production.
- Extensively worked on batch frame work to run all Informatica job scheduling.
- Involved in the Unit Testing and Integration testing of the workflows developed.
- Used Teradata utilities like Multi Load and Fast Load to load data into Teradata data warehouse from Oracle.
- Resolving the bottlenecks in the sources, targets, mappings and sessions.
- CreatedMapplets, reusable transformationsand used them in different mappings.
- Assisted in Batch processes using Fast Load, BTEQ, UNIX Shell and Teradata SQL to transfer, cleanup and summarize data
- Involved in Performance Tuning of mappings in Informatica.
- Involved in developing the Hive Reports, Partitions of Hive tables.
- Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.
- Involved in using SQOOP for importing and exporting data into HDFS and Hive.
- Hands on experience in loading data from UNIX file system to HDFS.
Environment: Informatica Power Center 9.6.1, Shell Scripting Unix, Teradata, HDFS, Hive SQL, Pig, Sqoop, Oozie, Informatica Power Exchange, HP ALM Quality Control, Informatica Data Quality
Confidential
Technical Lead
Responsibilities:
- Studied project plans and analyze various phase of work.
- Allocate work to team member, keep track of the progress of work and provided timely report to project manager.
- Provide technical guidance to team members.
- To interact with the onsite Project Manager and keep them updated on the work progress.
- Worked closely with ETL Architect and QC team for finalizing ETL Specification document and test scenarios. Scheduling, automating the jobs to meet user needs.
- Managing the adhoc work requests and tactical work requests.
- Created complex mappings in PowerCenter Designer using Aggregate, Expression, Filter Sequence Generator, Update Strategy, Union, Lookup, Joiner, XML Source Qualifier and Stored procedure transformations.
- Developed mappings/Reusable Objects/mapplets by using mapping designer, transformation developer and mapplet designer in Informatica PowerCenter.
- Worked extensively with different caches such as Index cache, Data cache and Lookup cache (Static, Dynamic, Persistence and Shared).
- Developed UNIX shell scripts to automate applications, Schedule jobs and for audit purposes.
- Worked with Informatica Data Quality IDQ toolkit, Analysis, data cleansing, data matching, data conversion, exception handling, Score cards, reporting and monitoring capabilities of Informatica Data Quality IDQ.
- Developed and modified complex SQL Queries and Stored Procedures as per business requirements
- Worked on performance tuning to increase the data load speed.
- Prepared scripts to create new tables, views and queries for new enhancement in the project using TOAD.
- Developed various Mappings with the collection of all Sources, Targets and Transformations.
- Extracted the data from various sources like DB2, Flat files and COBOL files.
Environment: Informatica Power Center 9.1, Shell Scripting Unix, Mainframe, Oracle 11g,DB2,Informatica IDQ, Informatica PowerExchange 9.1.0
Confidential
Technical Lead
Responsibilities:
- Design and development of new data warehouse (Analytical Data Warehouse) for better reporting and analysis.
- Worked with several vendors in sending outbound extracts and load in bound files into the warehouse.
- Worked on Developed mappings/Reusable Objects/Transformation/Mapplet by using mapping designer, transformation developer and Mapplet designer inInformaticaPower Center 9.6
- Created complex mappings in Power Center Designer using Aggregate, Expression, Filter, and Sequence Generator, Update Strategy, Union, Lookup, Joiner, XML Source Qualifier and Stored procedure transformations.
- Used Teradata utilities FastLoad, MultiLoad to load data and building, testing, implementing Teradata FastLoad, MultiLoad and BTEQ scripts, DML and DDL.
- Solved Various Defects in Set of wrapper Scripts which executed the Teradata BTEQ, MLOAD, FLOAD utilities and Unix Shell scripts.
- Performed tuning and optimization of complex SQL queries using Teradata Explain.
- Expert level knowledge of complex SQL using Teradata functions, macros and stored procedures.
- Analyzed the source data, made decisions on appropriate extraction, transformation, and loading strategies.
- Fast Load jobs to load data from various data sources and legacy systems to Teradata Staging.
- Modified the existing BTEQ script to enhance performance by using Volatile tables, incorporating parallelism, collect stats when needed and using the index techniques.
- Used BTEQ and SQL Assistant (Query man) front-end tools to issue SQL commands matching the business requirements to Teradata RDBMS.
- UsedInformaticadebugging techniques to debug the mappings and used session log files and bad files to trace errors occurred while loading.
- Provided on call support during the release of the product to low level to high level Production environment.
- Worked with scheduling tool for jobs scheduling.
- Involved in Unit testing, User Acceptance testing to check whether the data is loading into target, which was extracted from different source systems according to the user requirements.
- Created and Developed re-usable transformations, mappings and Mapplets confirming to the business rules.
- Documented ETL test plans, test cases, test scripts, test procedures, assumptions, and validations based on design specifications for unit testing, system testing, expected results.
- Involved in developing mapping to populate data from various systems.
Environment: Informatica Power Center 8.6, Shell Scripting Unix, ESP Scheduler, MainframeTeradata
Confidential
Module Lead
Responsibilities:
- Responsible for definition, development and testing of processes/programs necessary to extract data from operational databases, Transform and cleanse data, and Load it into data warehouse using Informatica Power center.
- Worked on Informatica PowerCenter Tool. Extensively worked with SQL, Expression, Filter, Lookup, Normalizer etc.
- Worked on the change data capture (CDC) process for extract the data.
- Developed various Mappings with the collection of all Sources, Targets and Transformations.
- Worked with Business users in consolidating requirements and data mapping of reporting requirements.
- Performed unit testing on the Informatica code by running it in the Debugger and writing simple test scripts in the database thereby tuning it by identifying and eliminating the bottlenecks for optimum performance.
- Made use of Post-Session success and Post-Session failure commands in the Session task to execute scripts needed for clean up and update purposes.
- Extensively worked with various lookup caches like Static Cache, Dynamic Cache, and Persistent Cache.
- Requirements gathering and analysis, involved to creating High-Level design documents, Application Design documents.
- Use various transformations like Source Qualifier, Filter, Aggregator, Expression, Lookup, Sequence Generator, Joiner, Union, Router, Sorter and Update strategy to create mapping.
- Using ETL Process to Extract, Transform and Load the data into stage area and data warehouse.
- Creating reusable transformations, mapping, mapplet, various tasks like session, Command, Email, Timer, Control, Event Wait, Event raise, Assignments, Worklets and workflows.
- Coordinating the team and resolving the issues of the team technically as well as functionally.
- Scheduling the jobs and monitoring its daily run in Production.
Environment: Informatica 8.6, Unix, Oracle 10g, HP Quality Center,SFTP,Putty
Confidential
Senior Developer
Responsibilities:
- Analyzing user requirements and designing and documenting functional specification and validation documents.
- Designing ETL mapping as per functional specification document.
- Developing ETLs using Informatica mapping, UNIX Shell and providing alternative solutions to common problems.
- Developing Data Mart Mappings for Reporting Purpose.
- Performance optimization on Teradata SQL Queries, designed/Created several BTEQ/MLOAD/FLOAD scripts with data transformations for loading the base tables to apply the business rules manipulate and/or massage the data according to the requirements.
- Developed several complex mappings in Informatica a variety of PowerCenter transformations, Mapping Parameters, Mapping Variables, Mapplets and Parameter files in Mapping Designer using Informatica PowerCenter.
- Performing extensive testing of ETL mapping with various test scenarios.
- Resolving various technical issues with ETL mapping.
Environment: Informatica PowerCenter 8.1/8.6, Oracle10 g, UNIX, Teradata
Confidential
Developer
Responsibilities:
- Extensively used ETL to load data from Flat Files, XML, Oracle
- Designing ETL mapping as per functional specification document.
- Involved in Designing of Data Modeling for the Data warehouse.
- Involved in Requirement gathering and Business Analysis.
- Created various mappings using Aggregate, Filter, Join, Expression, Lookup, Update Strategy
Environment: Informatica PowerCenter 7.1, Oracle 9i, UNIX,SQL PLUS