We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

3.00/5 (Submit Your Rating)

Dallas, TX

SUMMARY

  • Senior Data Engineer with 7 years of experience in Implementing highly scalable, processing data pipelines,dataLakes and Analytics platforms using AWS, Presto, Apache Hudi, Apache Spark and other Hadoop opensource frameworks.
  • Strong experience in working in Safe Agile Environment, working in a big group of dev and support leading juniors.
  • Expert in Designing large distributed systems and technology implementation in Cloud, BigData.
  • Strong experience in implementing automation workflows with zero - touch build, deployment, and remediation.
  • Hands-on experience with Hadoop distributions like Cloudera, AWS EMR, Apache Hadoop and other technology.
  • Experience with cloud tools for ingesting and processingdata. SkillsCollaborated on ETL (Extract, Transform, Load) tasks SQL maintainingdataintegrity and verifying pipeline stability.
  • ExcellentKnowledge and hands-on experience working on cloud platforms like GCP, AWS, IBM Cloud, Big Query DataPipeline or Azure.
  • Experience in designing, architecting, and managing Integrations inPeopleSoft, Workday andDell Boomi Applications.
  • Experience in .Net Framework, developing web applications using ASP.Net, C#.Net, MVC, SQL Server, SSRS, PL/SQL, Ajax, jQuery,JavaScript, LINQ, JSON.
  • Experience in working with Apache Spark Batch Processing, Streaming and SparkSQLAPIin various contexts using Scala and Python (Spark).
  • Hands-on experience in Python and Spark.
  • Experience analyzingdatato identify gaps and inconsistencies.
  • Strong experience using and integrating various AWS cloud services like S3, EMR, Glue Megastore, Athena, Redshift into thedatapipelines.
  • Experienced in managingAzureDataLakes (ADLS) andDataLake Analytics and an understanding of how to integrate with otherAzureServices.
  • Excellent understanding of NoSQL databases like HBase, Cassandra, MongoDB.
  • Proficient noledge and hand on experience in writing shell scripts in Linux.

TECHNICAL SKILLS

Bigdata Eco Systems: Hadoop, HDFS, MapReduce, Hive, Snowflake

Programming: Python, Kafka .NET

Data Warehousing: Informatica Power Center, Informatica Cloud, Talend Open studio & Integration suite

Applications: Salesforce, RightNow, Eloqua

Databases: Oracle (9i/10g/11g), SQL Server

Cloud: Azure

Query Languages: C, C++, Python, Scala, UNIX Shell Script, COBOL, SQL and PL/SQL

Scripting Languages: Unix, Python, Windows PowerShell

RDBMS Utility: Toad, SQL Plus, SQL Loader

PROFESSIONAL EXPERIENCE

SR. DATA ENGINEER

Confidential, DALLAS, TX

RESPONSIBILITIES:

  • Responsible for ingesting large volumes of user behavioraldataand profiledataAnalyticsDatastore.
  • Developed Spark applications for performingdatacleansing, event enrichment,dataaggregation, denormalization anddatapreparation needed for machine learning and reporting teams to consume.
  • Gatheird and createdDataSets, SourceData, Source MetaData,DataDefinitions and identifiedDataFormats.
  • Worked with COVID claimsdatafor provider reimbursements.
  • Developed and Implemented end-to-end ETL process for ingesting large claimsdatafrom blob storage AzureDataFactory (ADF) and Synapse very recently.
  • Wrote Kafka producers to stream thedatafrom external rest APIs to Kafka topics.
  • Wrote Spark-Streaming applications to consume thedatafrom Kafka topics and write teh processed streams to HBase via Phoenix JDBC connector.
  • CreatedAzureLogic Apps for various monitoring and automated purpose.
  • Created alert to monitorAzureSQL Database and scaled up/down depending on teh usage.
  • Designing and customizingdatamodels forDatawarehouse supportingdatafrom multiple sources on real time.
  • DesignedDataAnalytics platform monitoring systems via Azure Logic App, AzureBuilt ETL and ELTdatapipelines for ingesting & cleansing live and batchperformingdataexploration,datavalidation & transformation, using SQL, U-SQL,
  • Implemented Partitioning, Dynamic Partitions, Buckets in Hive
  • Designed financial productivity tool for CFO staff which included timeclock, Peoplesoftwage, Allscripts patient census, and financial benchmarkdatasources into a dimensionaldatamart using SSIS, T-SQL, and C#.
  • Designed and implemented APIs on AWSAPIGateway using AWS Lambda methods written in Python
  • Collaborated with teh infrastructure, network, database, application, and BA teams to ensuredataquality and availability.
  • Developed Python based framework for Dynamic Scaling on Metadata in Hive.
  • Designed, documented operational problems by following standards and procedures using JIRA.
  • Optimized S3 writes for writing large datasets via Pyspark.
  • Built complex ADF pipelines to migratedatafrom various sources (Azure SQLDatabases, on prem SQL Servers, IBM DB2, Shared Drives, SharePoint, Azure Blob storages, etc.) to load into Azure SQL databases and Azure Synapse Dedicated SQLPools, AzureDataStorage via AzureDataFactory (ADF), Databricks, U-SQL, LogicApps, Function Apps.

Environment: Apache Spark, Apache Hive, Python, S3, Pyspark, NIFI,Zookeeper, Kafka, Oracle Primavera Hadoop,DataLake, MySQL.

DATA ENGINEER

Confidential, Chicago, IL

RESPONSIBILITIES:

  • Collaborated with business user's/product owners/developers to contribute to teh analysis of functional requirements.
  • Handled importing ofdatafrom variousdatasources, performed transformations using Hive, PIG, and loadeddatainto HDFS.
  • Develop, refine, and scaledatamanagement and analytics procedures, systems, workflows, and best practices.
  • Worked ondatausing Sqoop from HDFS to Relational Database Systems and vice-versa. Maintaining and troubleshooting.
  • Created a Python script that called teh Cassandra Rest API, transformed thedata, and loaded it into Hive.
  • Designed, developeddataintegration programs in a Hadoop environment with NoSQLdatastore Cassandra fordataaccess and analysis.
  • Involved in writing Spark Scala scripts and UDF's to perform transformations on large datasets.
  • Enabling other teams to work with more complex scenarios and machine learning solutions.
  • Worked on AzureDataFactory to integratedataof both on-prem (MY SQL, Cassandra) and cloud (Blob storage, Azure SQL DB) and applied transformations to load back to Azure Synapse.
  • Involved in addressing issues related withPeopleSoftsecurity related to Users, Roles, PermissionLists, Password related, Object and Query Security.
  • Involved in converting Hive/SQL queries into transformations using Python
  • Performed complex joins on tables in hive with various optimization techniques
  • Created Hive tables as per requirements, internal or external tables defined with appropriate static and dynamic partitions, intended for efficiency
  • Worked extensively with HIVE DDLS and Hive Query language (HQLs)
  • Involved in loadingdatafrom edge node to HDFS using shell scripting.
  • Understand and manage Hadoop Log Files.
  • Manage Hadoop infrastructure with Cloudera Manager.
  • Created and maintained technical documentation for launching Hadoop cluster and for executingHive queries.
  • Build Integration between applications primarily Salesforce.
  • Ingesteddatainto Azure Blob storage and processed thedatausing Databricks.
  • Involved in Importing and exportingdatainto HDFS and Hive using Sqoop, Delimited File &Kafka.
  • Designed and developed Tableau dashboards to help teh business users has dimensions of teh customer demographics for marketing campaigns.
  • Involved in design and development of a front-end application usingJavaScript, CSS, HTML and JSPs and spring MVC for registering a new entry and management and configured it to connect to database using Hibernate.
  • Used Flume sink to write directly to indexers deployed on cluster, allowing indexing during ingestion.
  • Migrated from Oozie to Apache Airflow. Involved in developing Oozie and Airflow.
  • Managed resources and scheduling across teh cluster using Azure KubernetesService. AKS can be used to create, configure, and manage a cluster of Virtual machines.
  • Worked on different file formats like csv,JSON, AVRO, ORC, and Parquetproject.
  • DevelopedRestAPIs using Spring MVC Framework.
  • Developed a web-based application that automates teh complete process of granting privileges to a database,usingPL/SQLand Oracle APEX.

ENVIRONMENT: Azure HDInsight, Data Bricks, Data Lake, Cosmos Db, MySQL, Snowflake,MongoDB, Teradata, Ambari, Flume, VSTS, Tableau, Power BI, Azure DevOps, Ranger,Kafka

DATA ENGINEER

Confidential, Atlanta, GA

RESPONSIBILITIES:

  • Utilized Airflow to develop ETL solutions that helped enhance teh conversion rates by 18%.
  • Used Python, Google Analytics API, and SQL to ingestdatafrom disparate sources.
  • Collaborated with otherdataengineers to document best practices for developingdatapipelines and transformation strategies to be used for TEMPeffectivedatavisualization.
  • Collaborated in adatamining project to assess teh TEMPeffectiveness of various medical treatments.
  • Translated business needs into SQL queries.
  • Estimated teh project costs and completion time by developing Polynomial regression model.
  • TEMPEffectively used model regularization techniques to avoid over fitting of teh ML model.
  • Generated server-side PL/SQL scripts fordatamanipulation and validation and materialized views for remote instance.
  • Worked with product owners to establish design of experiment and teh measurement system for TEMPeffectiveness of product improvements.
  • Used Scoop to export teh evaluateddatainto relational databases for visualization and report generation for teh BI teams.
  • Used Hive queries to analyze massivedatasets of structured, unstructured, and semi-structureddata.
  • Used HQL in Hive to write and use complicateddatatypes in storing and retrievingdata.
  • Used Hive queries to analyze massivedatasets of structured, unstructured, and semi-structureddata.
  • Using advanced techniques such as bucketing, partitioning, and optimizing self joins, worked with structureddatain Hive to increase performance.
  • Used Cloudera Manager for installation and management of Hadoop Cluster.
  • Integrateddatasources from Kafka (Producer and Consumer API) fordatastream-processing in Spark using AWS network

Environment: AzureDataLake, Azure SQL, AzureDataFactory(V2), Azure Databricks, Informatica, Python 2.0, SSIS, Azure Blob Storage, Spark 2.0, Hive, Five Tran.

DATA ANALYST

Confidential, Philadelphia, PA

RESPONSIBILITIES:

  • Involved in requirements analysis and legacy systemdataanalysis to design and implement jobs using Microsoft SQL Server Integration Services (SSIS)
  • Gatheird business requirements and designed a roadmap to assess and implement predictive analytics across teh team handlingDatawarehouse projects
  • Documents,Datamodels,Dataflow Diagrams, User stories and Test cases to has an understanding of teh business and technical requirements across teh multiple involved.
  • Analyzed thedatasources from Oracle, SQL Server for design, development, testing, production rollover of reporting and analysis projects within TableauDesktop.
  • Partnered with Development and Quality Assurance teams to ensure teh product quality is always intact.
  • Resolved Production Support issues using SQL, PL SQL to prevent any interference caused to teh business. Analyzed SQL scripts and design it by using Spark SQL for faster performance.
  • Performed ETL Transformation activities in SSIS and built several packages and loadeddatatoDatawarehouse.
  • Involved in writing stored procedures in T-SQL do teh transformations of thedata.

ENVIRONMENT: Erwin, SQL, Oracle, PL/SQL, Bigdata, Hadoop, AzureDataLake, Spark, Scala, APIs, Pig, Python, Kafka, HDFS, ETL.

We'd love your feedback!