We provide IT Staff Augmentation Services!

Data Engineer Lead Resume

Tempe, AZ

SUMMARY

  • Over 25 years of experience in IT with various positions as a Developer, Data Engineer, ETL/Data Architect, Technical Lead. Involved with Architecture and design, development, testing, Implementation and support phases of software development life cycle. Proven ability to provide innovative IT solutions and meet the demanding business needs of customers. I am a self - motivated, a fast learner and take pride in the honesty and integrity with which I do my job. Recognized team player working effectively with individuals at all levels.
  • A versatile team player with strong architectural, data engineering, analytical, communication and negotiation skills with the ability to deliver quickly on time sensitive projects.
  • Hands on work experience using Big Data/Hadoop on-premises and Azure cloud technology.
  • Extensive experience using Apache NiFi ETL tool for data ingestion into HDFS file system.
  • Experience using Scala and Spark SQL to perform data transformation and data cleansing on a Hadoop Distributed File System (HDFS).
  • Good experience using Talend ETL tool for data integration and data management using relational databases, delimited files and files residing on Hadoop file system.
  • Extensively used Hive/Impala database to write queries and monitor jobs on Hadoop User Interface (HUE).
  • Good knowledge with data processing using MapReduce, Kafka and PIG scripts.
  • Strong experience with DataStax Cassandra NoSQL database to perform data analytics on real time data feed.
  • Knowledge using Sqoop for bulk import/export of data between relational database and Big Data residing on Hadoop Distributed File System.
  • Experience with IntelliJ IDE used for front end development of Scala code.
  • Good knowledge using Development Operations (DevOps) model for software Development and Operations.
  • High experience using Agile and SCRUM methodologies to perform sprint planning, release backlog, backlog grooming sessions, retrospective, daily standup sessions.
  • Extensive experience with Test Automation and Test Driven Development (TDD) using pySpark, Gherkin language using Cucumber and shell scripts on Linux.
  • Hands on experience using Azure Data Factory (ADF) to perform data ingestion into Azure Data Lake Storage (ADLS).
  • Good experience using Azure Data bricks to perform data transformation/data cleansing using Python and Spark SQL on databricks notebook.
  • Experience with Azure Storage Explorer (ASX) to perform data operations on Binary Large Object Storage (BLOB) and ADLS.
  • Hands on experience using Delta Lake operations on Azure Data Lake using Azure databricks notebook.
  • Experience using Azure Functions to perform serverless operations and Azure Logic Apps to create workflows.
  • Exposure to Azure snowflake data warehouse used for a project POC.
  • Familiarity with CI/CD pipelines in Azure DevOps (ADO) and adoption of best practices and standards for development.
  • Experience using Tableau for Reports and dashboard development.

PROFESSIONAL EXPERIENCE

Confidential, Tempe, AZ

Data Engineer Lead

Responsibilities:

  • Active participation in all phases of the Software Development Life Cycle (SDLC) starting with requirements gathering sessions, technical Architecture, design, development, testing and Implementation.
  • Extensive experience using NiFi ETL tool for batch data ingestion into Hive/Impala database residing in Big Data on premises environment.
  • Experience with batch file ingestion handling files in both semi-structured and unstructured file formats - CSV, Parquet, XML, JSON and Avro including PII and sensitive data.
  • Code development with IntelliJ IDE using Scala and Spark SQL to perform data transformations/data cleansing and loading data into target tables.
  • Experience with using Spark SQL DataFrames, Spark SQL tuning, partition management and stats collection on Hive/Impala tables.
  • Extensive experience dealing with data migration activities between Big Data On-premises and Azure Cloud environment.
  • Experience using Azure Data Factory (ADF) on a day-to-day basis to perform data ingestion onto Azure Data Lake Storage (ADLS).
  • Extensive experience with data transformation and data cleansing using Python notebook and Spark SQL within Azure databricks.
  • Hands on experience using Delta Lake functionality for data transformation and data loading into target tables.
  • Worked on Azure functions for Encryption/Decryption of data files along with using eMFT file transfer mechanism.
  • Involved in Data Architecture to analyze the data engineering patterns(batch versus real time data ingestion) into Azure Cloud Services
  • Experience using Azure Data Studio (ADS) to perform operations between relational and cloud storage and develop notebooks.
  • Good knowledge about Azure Active Directory Authentication, Subscription, Resource Group, Azure Key Vault, Service Principal.
  • Familiarity with Azure Streaming Analytics (ASA) / Azure Event Hub (AEH).
  • Proficiency with Enterprise Data Integration (EDI) consolidating data from various subject areas and build OLAP tables used for business analytics and reporting.
  • Experience with DevOps transformation model for data development, testing and deployment operations.
  • Worked on Talend ETL tool for data integration using relational databases, delimited files and files residing on Hadoop file system.
  • Experience with Cassandra NoSQL database used as a speed layer for storing real time data.
  • Working with end users and product owner hand-in-hand to write better queries using Hive/Impala and using Tableau for reports and dashboard.
  • Extensive experience with data quality testing:
  • Tools Development for Test data preparation, Test Automation using Python, Shell scripts for jobs monitoring and Alerts.
  • Mentored teams to focus on Test Driven Development (TDD)
  • Behavior Driven Development (BDD) using Cucumber with test code using Gherkin/Groovy
  • Strong experience writing complex SQLs using Teradata, Oracle, SQL Server data testing in a data warehouse environment.
  • Experience in using various version control tools like GitHub, CVS, Clear Case, SVN
  • Active participation in design review and code review sessions, technical conversations within internal and cross-functional teams to solve complex issues.
  • Utilized Agile tools / methods to track and report project progress and coordinate project activities.

Environment: Business Objects Data Integrator (BODI) / Informatica ETL Business Objects XI R3 for reporting, Oracle 10g, MySQL, SQL Server 2005, Linux, Windows XP

Confidential

ETL Architect

Environment: Business Objects Data Integrator (BODI) / Informatica ETL Business Objects XI R3 for reporting, Oracle 10g, MySQL, SQL Server 2005, Linux, Windows XP

Responsibilities:

  • Experience following Software Development Life Cycle (SDLC) starting with Requirements gathering sessions, technical Architecture, design, development, testing and Implementation.
  • Extensive experience with leading the development team for ETL development using Business Objects Data Integrator (BODI)/Data Services to build rapid mart using Sales Force Application and Oracle database.
  • Experience with using PowerDesigner to develop logical data model and build Entity Relationships
  • Experience with Business Objects reporting tool to perform universe tuning, usage of merge dimensions and checking the integrity of the business object universes.
  • Experience with unit testing, system testing and user acceptances test (UAT) for universe, reports and security management .

Confidential

Team Lead / Database Administrator

Environment: SAP BI 7.0, ECC6.0, Oracle 10g, Business Objects

Responsibilities:

  • Responsible for design, development, testing and deployment using SAP source system and flat file extracts load into APO Info Cubes.
  • Implemented the business contents for APO - demand planning, sales forecast and demand planning forecast.
  • Led production support/on-call team and managed to work well under pressure during critical time and thereby support team work and collaboration.
  • Data migration experience from various legacy systems using Oracle, SQL Server, Teradata along with flat files/XML etc., to load into SAP ERP system using Oracle.

Confidential

Sr. Programmer Analyst / Data Modeler

Environment: Ingres 4GL, Ingres and Oracle 7, C, ESQLC, Legacy ERP

Responsibilities:

  • Technical team led in the design and development of Legacy ERP applications.
  • Developed the code using Ingres 4GL and ESQLC.
  • Supported Finance modules built using J.D. Edwards ERP.

Hire Now