We provide IT Staff Augmentation Services!

Senior Data Engineer Resume

SUMMARY

  • Data Engineer having 13 years of experience with strong background in end - to-end implementation and execution of enterprise data warehousing and big data projects.
  • Worked in large scale data integration projects in all phases of SDLC which supports multiple platforms and technologies like Big Data and Hadoop Ecosystem components,Teradata,Informatica Power Centre,Informatica Cloud, Python and Snowflake.
  • Hands on experience in Hive, Teradata, SQL, Python, My SQL and Snowflake.
  • Proficient in gathering and analyzing business requirement, designing the overall solution, development, end to end testing, maintenance and support of the complete data management & processing systems.
  • Adept in Process documentation and creation of ETL technical and design documents.
  • Adept at multitasking, working independently and as part of a team as required. Very flexible at adapting to changing client needs and deadlines. Possessing strong problem solving and communication skill.

TECHNICAL SKILLS

Big Data Ecosystems: Hadoop, HDFS, Hive,Sqoop

Programming Languages: Python

Scripting: UNIX Shell Scripting (ksh)

Databases: Oracle 11g/9i, Sybase 12.5, Teradata 13.1,My Sql

ETL Tools: Informatica Power Center and Informatica Cloud

Reporting Tools: Tableau, Business Object XIR1 and Business Object XI R2

Scheduling tools: Autosys R4, Informatica Power Center scheduler, UC4

Other Tools: SQL*Plus, TOAD, Teradata SQL Assistant, Rapid SQL.7.3.1, Putty, WinSCP, oracle SQL developer, MySQL Workbench

Operating Systems: Windows, UNIX, MacOs

Data ware house Methodologies: Dimensional modeling (Star/Snowflake)

Ticketing Tool: IBM rational synergy7.1, HP ALM 12.01, Pac 200 v7.6,Jira

Versioning Tool: Tortoise SVN, GitHub

PROFESSIONAL EXPERIENCE

Confidential

Senior Data Engineer

Responsibilities:

  • Leading the team to design, develop, testing and providing end -end deliverables
  • Develop solutions by leverage the capabilities of ETL tools and identify opportunities for process improvements and implemented using Informatica and Python
  • Conduct root cause analysis and resolve production problems and data issues
  • Conduct performance analysis and optimize data processes. Make recommendations for continuous improvement of the data processing environment
  • Develop a data platform from scratch and took part in requirement gathering and analysis phase of the project to document the business requirements.
  • Design and implement multiple ETL solutions with various data sources by extensive SQL Scripting, ETL tools, Python, Shell Scripting and scheduling tools. Data profiling and data wrangling of XML, JSON,Web feeds and file handling using python, Unix and Sql.
  • Loading data from different sources to a data ware house to perform some data aggregations for business Intelligence using python.
  • Processing structured and semi structured data sources like JSON,XML using python.
  • Created Snow Pipe for continuous data load and used COPY to bulk load the data.
  • Creating Data Pipelines using combination of Snowflake and Python.

Environment: Python, Postgres DB,Tableau,Unix,SQL, Snowflake, MY Sql

Confidential

Senior Data Engineer

Responsibilities:

  • As a Senior Data Engineer, my role includes analyzing and evaluating the business rules, data sources, data volume and come up with estimation, planning,development and execution plan to ensure deliveries meets the business requirements.
  • Working Closely with Stakeholders and Solution Architect for understanding the requirements and designing the overall ETL solutions including analyzing data, preparation of high level and detailed design documents, test plans, error-handling document and deployment strategy to build highly scalable, robust and fault-tolerant system.
  • Migrated the existing data pipeline in Teradata and SAP HANA to Hadoop platform and perform ETL operations on it and build analytical reports on it.
  • Designed the historical load from Teradata and SAP HANA into hive.
  • Worked on on-call production issues- scrubbing the data source, resolve hive query issues, workaround for defects within SLA duration.
  • Managed the imported data from different data sources, performed transformation using Hive and Map- Reduce and loaded data in HDFS.
  • Actively involved throughout the SDLC Life cycle and managing task completion across all the implementation phases.
  • Supporting other Data Engineers, providing mentoring, technical assistance, troubleshooting and alternative development solutions

Environment: Teradata, Hive, Sqoop, SAP HANA, Informatica Power Centre

Confidential

Senior Teradata Developer

Responsibilities:

  • Designed and Build ELT architecture to load data from Source System to Core EDW tables.
  • Gathered requirements and created functional and technical design documents.
  • Worked in a Techno Functional role to resolve the existing issues and enhancing and incorporating enhancements in the existing data pipelines.
  • Development of scripts for loading the data into the base tables in EDW and to load the data from source to staging and staging area to target tables using Fast Load, Multiload and TPUMP utilities of Teradata.
  • Implement multiple software components using various Teradata utilities including Teradata parallel transporter, Multiload, Fast Load and Tpump and performance tuning of ETL solutions, UNIX scripts and database code according to the new requirements and enhancements.
  • Involved in deploying the code between various environments (Dev, QA, and Prod) after development and scheduling the code for daily and monthly runs.
  • Identifying, designing and implementing internal process improvements which includes automating manual data load processes,optimizing data delivery,re-designing infrastructure for greater scalability etc
  • Supporting other Data Engineers, providing mentoring, technical assistance, troubleshooting and alternative development solutions.

Environment: Teradata 15.10, UC4, UNIX KSH, Teradata SQL Assistant, Putty, Winscp,Python

Confidential

Senior ETL Developer

Responsibilities:

  • Assisting in designing the overall ETL solutions including analyzing data, preparation of high level and detailed design documents, test and data validation plans and deployment strategy.
  • Prepared the technical mapping specifications, process flow and error handling document.
  • Developed complex mappings implementing complex business logic using variety of transformation logic like Unconnected and connected Lookups, Router, Filter, Expression, Aggregator, Joiner, Update Strategy, Unconnected and Connected Stored procedures, normalizer and more.
  • Created various tasks like Pre\Post Session Commands, Timer, Event Wait, Event Raise, Email and Command task.
  • Writing autosys Jil files to run workflow components and deploy the files.
  • Extensively worked in database components like SQL, PL/SQL, Stored Procedures, Stored Functions, Packages and Triggers.
  • Performs Code review and troubleshooting of existing Informatica mappings and deployment of code from Development to test to production environment.
  • Supporting other ETL developers, providing mentoring, technical assistance, troubleshooting and alternative development solutions

Environment: Informatica 9.6.1, Teradata, Oracle 11g, Autosys, UNIX KSH, SQL*Plus, Putty, Winscp, Oracle SQL Developer

Confidential

Senior ETL Developer

Responsibilities:

  • Assisted in designing the overall ETL solution including analyzing data, preparation of high level and detailed design documents, test and data validation plans and deployment strategy.
  • Prepared the process flow and error-handling document.
  • Developed mappings implementing complex business logic using variety of transformation logic like Unconnected and Connected lookups, Router, Filter, Expression, Aggregator, Joiner, Update Strategy, Unconnected and Connected Stored Procedure, Normalizer and more.
  • Created various tasks like Pre\Post Session Commands, Timer, Event Wait, Event raise, Email and Command Task.
  • Extensively worked in database components like SQL, PL/SQL, Stored Procedures, Stored Functions, Packages and Triggers.
  • Developed re-usable transformations, mapplets and worklets confirming to the business rules. Used Debugger wizard to remove bottlenecks at source level, transformation level and target level for the optimum usage of source, transformations and target loads.
  • Monitored batches and sessions for weekly and Monthly extracts from various data sources to the target database.
  • Involved in development and performance tuning of ETL and database code and deploying the code between various environment (Dev, QA, Prod) using Synergy and Service Centre Tickets after data validation.
  • Worked extensively on Informatica Partitioning when dealing with huge volumes of data.
  • Used Teradata External Loaders like Multi Load, T Pump and Fast Load to load data into Teradata database.
  • Writing Teradata SQL Queries, procedures and macros to join or any modifications in the table
  • Involved in writing UNIX Shell scripts (Pre/Post Sessions commands) for the Sessions & wrote Shell scripts to kickoff workflows and packages, deleting old files and taking backup of source files and doing FTP files.

Environment: Teradata 13.1, Informatica 9.6.1, Autosys, Unix KSH, IBM rational synergy7.1

Hire Now