We provide IT Staff Augmentation Services!

Senior Data Engineer/ Architect Resume

5.00/5 (Submit Your Rating)

PROFESSIONAL SUMMARY:

  • Highly qualified Senior Data Engineer/ Architect with 11+ Years of professional experience in the field of Data Management, involved in development of various Data projects which helped enterprises in transforming their data into information, information into insight and insight into business decisions
  • Extensive knowledge on Big Data Stack, Data Lake, Database design, Oracle SQL/PLSQL, DWH, Data Security, Data Lineage, Data Modeling, ETL - Talend/Informatica / Abinitio, Google Big Query (BQ), Python, Spark SQL, Cloud Migration, Unix Shell scripting and Agile process.
  • Experience in designing and implementing of Data & Analytics solutions including ODS, Data Warehouses, Lakes, Pipelines, BI, Reporting, Master & Data, Metadata, Data Quality, Modelling, Catalogues & Governance.
  • Extensive experience building & operating highly available, distributed systems of data extraction, ingestion, and processing of large data sets using Hadoop and Spark (Scala, Java, SQL & Python).
  • Proficiency in MPP cloud data-warehouse-as-a-service offerings, such as Snowflake, BigQuery.
  • Experience in design and development of Data Solutions using RDBMS (Oracle, SQL Server, Teradata) & NoSQL databases (HBase, Cassandra)
  • Experience building highly scalable real-time Data Pipelines using Apache Kafka and Hudi.
  • Sound understanding of various data solution patterns and when to use them: ETL/ELT, RDBMS, Normalization/De-normalization, Key-Value, In-Memory, Wide Column, Columnar, Graph, Text Indexing, Streaming & Messaging.
  • Extensive experience in building Data Lakes, HUB & Data Pipelines using Big Data technologies & tools such as Apache Hadoop, Cloudera, HDFS, MapReduce, Spark, YARN, Delta-lake, Hive.
  • Experience with scheduling and workflow orchestration tools such as Automic & Control-M and Tivoli.
  • Experience with query optimization including performance tuning, utilizing indexes, and materialized views to improve query performance. Expert grasp of data warehouse design techniques including slowly changing dimensions, aggregation, partitioning and indexing strategies.
  • Demonstrated ability and willingness to act as subject matter expert, tracking technology/industry trends and to provide data-driven reasoning for recommending technology paths.
  • Good hands-on migration of databases like oracle, Teradata onto Big data like Hadoop, hive, GCP and BigQuery.
  • Good knowledge of Data Warehousing concepts and complete life-cycle implementation experience in a Business. Extensive knowledge of data modeling in support of data warehouse and data mart design, implementation and measurement.
  • Extensive experience in designing theData modelsfor OLTP & OLAP database system experience usingER diagram. Design and construction of ETL processing and data marts. Experience in optimizing performance in relational and dimensional database environments by making proper use of Indexes and Partitioning techniques.
  • Good Experience in data modeling and understanding of different data structures and migrating the Relational DWH Data Models into Hadoop Hive and loading the data through SQOOP

TECHNICAL SKILLS:

Languages: PL/SQL, SQL, Shell Scripting, Windows, Scala

Databases: Big Query, Oracle 9i/10g/11g/12c, SQL SERVER, TERADATA 14

Tools: &Utilities: SQL*Plus, Toad, PL/SQL Developer, SQL * Loader, Explain Plan and CitrixData Modelling Tools: ERWIN

Operating Systems: Windows NT 4.0/ XP/ 2000, Unix/Linux

Data Warehousing Tools: Informatica 9.6.1, Ab- initio, Big Query (BQ)

Big Data: HDFS, Hive, Sqoop, Kafka, Cassandra, HBase, Spark, BQCloud Technologies

PROFESSIONAL EXPERIENCE:

Senior Data Engineer/ Architect

Confidential

Responsibilities:

  • Lead the definition and implementations of Data Capabilities like Data Discovery & Classification, Catalog, Lineage, Integration, Mesh and Lakes leveraging Google cloud and on-premises data technologies
  • Partnered with product owners & business SMEs to analyze the business needs and provided supportable and sustainable Data solutions. Ensured overall technical solutions are aligned with business needs and adheres to Confidential architectural guiding principles.
  • Designed and development of the Confidential Ecommerce—Customer, Gift Registry, Tax Exempt, Capital One and the CAU pipelines.
  • Design & implemented CDC pipeline for Confidential Ecommerce to process Consumer data real-time using Kafka Streaming, Scala, Spark, HIVE and Big Query extensively. The platform optimized to scale peta-byte of data; support fast insert/update throughput & fast seek queries.
  • Collaborated with stakeholders from Risk & Compliance and Engineering offices in designing technical solutions for managing data classification, governance and security in MITI.
  • Lead the team in achieving the AAIP targets and implemented the code coverage using Funsuite and Lineage using Spline.
  • Define data (metadata), identify systems of record and authoritative sources, created data quality rules, created data flow diagrams, reconciliation and applied standards and controls for ETL pipelines.
  • Developed ETL solution to load data from Cosmic Pipeline & Datamart into BQ and performed ELT using UDP to transformed into domain optimized data.
  • Lead the team in migrating the Customer Pipelines from Cloudbreak to Google Dataproc.
  • Build data strategy to improve data quality and operational efficiencies. Enabled enhanced reporting capabilities including business intelligence dashboards and data visualization software (Tableau).
  • Proposed and lead POCs of latest cloud services & data platform technology to enhanced, optimized operations and management of the Enterprise Data Platform.
  • Created strategies & plans for data capacity planning, life cycle data management, scalability, backup, archiving and ensured data security & privacy standards are implemented including: role-based security, encryption & tokenization.
  • Actively participated in Agile Scrum development processes including continuous integration, prototyping and testing in a strongly collaborative environment.

Associate Vice President

Confidential

Responsibilities:

  • Work closely with business in Requirement Analysis, Design, and Development.
  • Support code/design analysis, strategy development and project planning.
  • Understating the source and target systems to implement migration.
  • Using Sqoop & Teradata Fast Export to export data from Teradata and ingest into Hadoop Data Reservoir.
  • Automate the daily activities using Python, build Tableau dashboards for reporting.
  • Ingest the SOR data into Hadoop Staging - Conformed Zone after performing data cleansing and TDQ checks
  • Creating HIVE tables on top of Hadoop, tuning the HIVE table loads, long running queries by reviewing queries and data to save resources and meet SLA
  • Working closely with BI teams to build Cognos & Tableau dashboards
  • Perform Data Management activities for LOB/Tenant level to adhere Risk compliances and getting the clearances from Internal & external Audit Teams
  • Played a key role in adopting the Agile methodology and implementing Scrum
  • Rewarded with Tech-Ace for decommissioning the legacy system and saving the Teradata space.

EDW to ICDW Migration

Confidential

Responsibilities:

  • Requirement Study by analyzing the Business requirement documents & Functional Specifications based on the usage pattern and user discussions
  • Developing Informatica mappings to implement business logic by using appropriate transformations in an efficient way by using Pushdown Optimization technique
  • As a senior ETL developer, managing ETL Development Team and Involving in code/ peer reviews and tuning the code with best practices
  • Preparing the supporting documents like Design Document, Data Flow Diagram, Unit Test Results, Peer Review and Sign Off documents
  • Point of contact for business reviews, getting sign-off, project documentation, common SharePoint ownership and code promotions to higher environments.
  • Using control M for scheduling and running the jobs as per the frequency and dependency to suit the requirement
  • Monitoring the Job runs, validating the reports, perform primary checks before submitting to regulatory Board
  • Support Audit & Risk Governance teams by providing the required metrics on ad-hoc basis.
  • Decommissioning the legacy systems after migrating the data to new warehouse and got ‘kill the tail ’ from LOB head.

Associate

Confidential

Responsibilities:

  • Build Data pipeline for loading the OFAC data from the mainframe system into oracle.
  • Co-ordinate with the business team and the third-party audit teams for verification of the OFAC data.
  • Communicating with source, development teams for the test data requirements.
  • Sensitive data will never be exposed to testers.
  • Sophisticated mask data based on condition, compound, deterministic.
  • The Data is masked while loading from Production to Testing and Development environments.
  • Sub-Set the data and reduce the amount of data in the test and developments environments.
  • Generate Synthetic Data whenever required for testing new applications where the data is not available.
  • Implement Test matching and help the testers by providing the attributes for each testing scenario and their corresponding test cases.
  • Complete documentation of the Test Data management.

Software Engineer

Confidential

Responsibilities:

  • Developed data pipelines and load the data into target DB.
  • Created various mappings using informatica and helped the business in their DSS.
  • Analyzed the sources, targets, transformed the data, mapped the data and loading the data into Targets using Informatica.
  • Designed and developed data pipelines using slowly changing dimension 1.
  • Extensively used many transformations and developed complex mappings.
  • Designed the target database as per the client requirements for staging and loading of data.

Software Engineer

Confidential

Responsibilities:

  • Developed simple and complex mappings.
  • Created various Packages, Stored procedures, Cursors and Triggers for the generation of reports using oracle PL/SQL.
  • Analyzed the sources, targets, transformed the data, mapped the data and loading the data into Targets using Informatica.
  • Developed SCD1, SCD2 and SCD3 Mappings.
  • Extensively used transformations like Lookup, Filter, Aggregator, Sorter, Expression, Stored Procedures, Router, Sequence generator transformations.
  • Designed the target database as per the client requirements for staging and loading of data.
  • Designed and developed the common reports for India Post using iReport tool.
  • Created many packages and stored procedures for the generation of reports.
  • Experienced in trouble shooting the various issues related to the ETL mappings and report generation.
  • Provided a detailed documentation for the release.
  • Resolved issues raised in the SIT and UAT phases.
  • Appreciation from client for in time delivery of complex modules.

Software Engineer

Confidential

Responsibilities:

  • Understanding existing Business Model and Customer Requirements.
  • Involved in developing front end menus using FSDP and FSS.
  • Responsible for the design and development of ETL processes using Informatica.
  • Designed various transformations Filter, Aggregator, Joiner, Expression, Router, Stored Procedure, Sequence generator as per business logic.
  • Designed various reusable transformation mappings (mapplets).
  • Designed and developed various slowly changing dimension models.
  • Involved in designing and developing various consolidated reports.
  • Created various Packages, Stored procedures, Cursors and triggers for the generation of reports using oracle PL/SQL.
  • Prepared Unit Test Plan.
  • Resolved issues raised in ST and UAT phase.
  • Providing the detailed documentation for the release.

We'd love your feedback!