We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

5.00/5 (Submit Your Rating)

Fargo, ND

SUMMARY

  • Over 7 years of experience in Data Engineering,Data Pipeline Design,Development and Implementation as a Sr.Data Engineer/Data Developer and Data Modeler.
  • Strong experience in Software Development Life Cycle (SDLC) includingRequirements Analysis, Design Specification and Testing as per Cycle in bothWaterfall and Agile methodologies.
  • Hands - on experience with Spark, AWS, Azure,Talend, Hadoop, HDFS, Hive, Impala, Oozie, Sqoop, HBase, Scala, Python, Kafka, Kudu, NoSQL Databases like Cassandra, Mongo DB.
  • Experience in BIGDATA using HADOOP framework and Analysis, Design, Development,Documentation, Deployment, and Integration using SQL and BigData technologies.
  • Expertise in BigData Hadoop Ecosystem components Map - Reduce, HDFS, Yarn/MRv2,Pig, Hive, HDFS, HBase, Spark, Kafka, Sqoop, Flume, Avro, Sqoop, AWS, Avro, Solr and Zookeeper.
  • Experience in creating data pipelines to move data from RDBMS to HDFS/HDFS toRDBMS using Sqoop for improved Business Intelligence and Reports.
  • Expertise in Development, deployment and Managing Hadoop clusters using distributions like Cloudera (CDH4), Horon Works (HDP 2.3.0, HDP 2.6.0)
  • Experience with Oozie Workflow Engine in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.
  • Hands-on experience in developing SPARK applications using Spark tools like RDD transformations, Spark core, Spark MLlib, Spark Streaming and Spark SQL.
  • Strong experience and knowledge of real timedataanalytics using SparkStreaming, Kafka, and Flume.
  • Experience in job/workflow scheduling and monitoring tools like Oozie, AWSDatapipeline & Autosys
  • Experience working on creating and running Docker images with multiple micro - services.
  • Strong understanding ofDataModeling and ETL process indatawarehouse environment such as star schema, snowflake schema.
  • Experience working with version control systems like Subversion, GIT and usedSource code management tools GitHub, GitLab, Bitbucket including command line applications.
  • Experience in using Apache Spark Streaming API to process multiple event batches and generating analytics using it.
  • Experience in creating on demand Spark Cluster usingDataBricks on AWS and run heavy tasks that can process large amounts ofdataand also to setup to bring down the cluster when the jobs is done to reduce resource usage and saving cost for clients.
  • Good understanding of Spark internals and have worked on performance tuning, troubleshooting and debugging spark applications.
  • Good Knowledge in Amazon Web Services (AWS) concepts like EC2, S3, EMR, ElasticCache, DynamoDB, Redshift, Aurora.
  • Experienced with cloud: AWS/EMR or working directly with Hadoop installed on EC2Instance (non EMR) using Cloudera manager.
  • Experience in developing scripts using Python or Shell Scrips to do Extract,Load and Transform.
  • Experience in using integrated development environment like Eclipse, PyCharm, and Visual Studio Code.
  • Experience in conducting requirement gathering/analysis, proposing solution with design diagrams, implementing POC to test various ideas, developing and productionizing solutions. Well versed with Software Development Lifecycle (SDLC).

PROFESSIONAL EXPERIENCE

Sr. Data Engineer

Confidential, Fargo, ND

Responsibilities:

  • Developed Spark scripts by using Scala as per the requirement.
  • Involved in Requirement Gathering, Business Analysis and Translated Business requirements into Technical Design in Hadoop and BigData
  • Implementation ofdatamovements from on-premises to cloud in Azure.
  • Develop batch processing solutions by usingDataFactory and AzureDatabricks
  • Implement AzureDatabricks clusters, notebooks, jobs and auto scaling.
  • Design fordataauditing anddatamasking.
  • Design fordataencryption fordataat rest and in transit.
  • Design relational and non-relationaldatastores on Azure.
  • Efficient usage of SAS tools such as SAS/Base, SAS/Access, SAS/Connect, SAS/SQL,SAS/STAT, SAS/Graph and SAS Macro.
  • Scrapingdatafrom diverse platform’s and applying business logics and loadingdatato Teradata database
  • Implemented Incremental load approach in spark for huge tables.
  • Created and deployed Lambda functions in AWS using pre-built AWS LambdaLibraries, as well as Lambda functions in Scala using custom libraries.
  • Involved in SAS Programming, using Base SAS, Macro Facility, Proc SQL, SASProcedures, SAS Functions, SAS Formats, and ODS facility indatascrubbing, manipulation, and preparation to produce summary datasets and reports.
  • Used Spark API over Cloudera Hadoop YARN to perform analytics ondatain Hive
  • Worked on different file formats like JSON, CSV, XML using spark SQL.
  • Imported and exporteddatainto HDFS from database and vice versa using Sqoop.
  • Developed MapReduce/Spark Python modules for machine learning & predictive analytics in Hadoop on AWS
  • Worked with CDH4 as well as CDH5 applications. PerformedDatatransfer of largedataback and forth from development and production clusters.
  • Created Partitions and Buckets in Hive for both Managed and External tables inHive for optimizing performance.
  • Implemented Hive Join queries to join multiple tables of a source system and load them to Elastic search tables
  • Built reusable HiveUDFlibraries for business requirements which enabled users to use these UDF's in Hive Querying.
  • Developed DataLake as aDataManagement Platform for Hadoop.
  • Used Talend to run the ETL processes instead of Hive queries.
  • DesignedDataMarts by following Star Schema and Snowflake Schema Methodology, using industry leadingDatamodeling tools like ER Studio.
  • Involved in Designing Star Schema (identification of facts, measures, and dimensions), Snowflake Schema forDataWarehouse, ODS architecture by using tools likeDataModel, ERwin.
  • Worked on Key performance Indicators (KPIs), design of Star schema and Snowflake schema in Analysis Services (SSAS).
  • Well versed in designing and buildingdatasolutions to migrate existing sourcedatain Teradata and DB2 to Big Query (Google Cloud Platform), extracting batch and Real timedatafrom DB2, Oracle, Sql server, Teradata, Netezza to Hadoop(HDFS) using Teradata TPT.
  • Design scoop scripts to load from Teradata and DB2 to Hadoop environment and design Shell scripts to transferdatafrom Hadoop to Google Cloud Storage (GCS) and from GCS to Big Query.
  • Handled Data Movement,Data transformation, Analysis and visualization across the lake by integrating it with various tools.

Environment: Hadoop YARN, Spark Core, Spark Streaming, Spark SQL,Teradata Python,Kafka, Hive, Sqoop, Amazon AWS, Elastic Search, Impala, Cassandra, Tableau,Talend, Snowflake, Oozie, Jenkins, Cloudera, Oracle 12c, Linux.

Data Engineer

Confidential, Chicago, IL

Responsibilities:

  • Responsible for Building Scalable DistributedDatasolutions using Hadoop.
  • Real timedataprocessing (Kafka, Spark Streaming & Spark Structured Streaming),Worked on Spark SQL, Structured Streaming, MLlib and using Core Spark API to explore Spark features to builddatapipelines, Implemented Spark streaming applications & fine tune to reduce shuffling.
  • Development of Spark jobs in PySpark and SparkSQL to run on top of hive tables and create transformeddatasets for downstream consumption.
  • Handled large datasets using partitions, Spark In-Memory capabilities,Broadcasts in Spark, Effective & Efficient Joins, transformations and other during ingestion process itself.
  • Worked in Performance Tuning of Spark Applications for setting right batch interval time, correct level of parallelism and memory tuning.
  • Design and Implement Database Schema importdataand build stored procedures onSQL Azure.
  • Set-up databases in GCP using RDS, storage using S3 bucket and configuring instance backups to S3 bucket. Prototype CI/CD system with GitLab on GKE utilizingKubernetes and Docker for the runtime environment for the CI/CD systems to build and test and deploy.
  • Worked on a Cluster of Size 105 nodes.
  • Involved in converting Hive/SQL queries into Spark transformations using SparkRDDs and Python.
  • Design and build ETL workflows, leading the efforts of programmingdataextraction from various sources into Hadoop file system, implement end to end ETL workflows using Teradata, SQL, TPT, SQOOP and load to HIVEdatastores.
  • Implemented schema extraction for Parquet and Avro file formats in Hive.
  • Used Hive and created Hive tables, loadeddatafrom Local file system to HDFS
  • Used hive to do transformations, event joins and pre-aggregations before storing thedatato HDFS
  • Implemented Partitioning, Dynamic Partitions, and Buckets on huge datasets to analyze and compute various metrics for reporting.
  • Involved in HBase setup and storingdatainto HBase for future analysis.
  • Used Oozie workflow to coordinate pig and Hive scripts.
  • Create Notebooks in Databricks to pull thedatafrom S3 and process with the transformation rules and load back thedatato persistence area in S3 in ApacheParquet format.
  • Used Stash Git-Bucket for Code Control and Worked on AWS Components such asAirflow, Elastic Map Reduce (EMR), Athena and Snowflake.
  • Used apache airflow in GCP composer environment to builddatapipelines and used various airflow operators like bash operator, Hadoop operators and python callable and branching operators.Used apache airflow in GCP composer environment to builddatapipelines and used various airflow operators like bash operator, Hadoop operators and python callable and branching operators.
  • Built reports for monitoringdataloads into GCP and drive reliability at the site level
  • Written Hive Queries for ad hocdataanalysis to meet business requirements.

Environment: HDFS, MapReduce, Hive, PIG, Sqoop, HBase, Oozie, Flume, Sqoop, Kafka,Zookeeper, Amazon AWS, SparkSQL, Spark Dataframes, PySpark, Python, Java, JSON,SQL Scripting and Linux Shell Scripting, Avro, Parquet, Hortonworks.

Software Consultant

Confidential, Dallas, TX

Responsibilities:

  • Responsible for setting up networks through Design and Development.
  • Engineered various custom modules for the integration of various types of network circuits.
  • Optimized the code using smart pointers, profilers, and C++ Standard template library.
  • Directly responsible for various enhancements of Equipment updates/transmission of signals on OC/STS cables.
  • Used Purify, GDB, and DBX for debugging/crash logs which helped in performance enhancement and optimization.
  • Used various Design Patterns in provisioning the networks.
  • Designed and implemented the orders for VNI, VHO, VSO, ONI etc., and their controllers as demandthe design of the application.
  • Created Forms, views, models using Django.
  • Used the Python's modules NumPy, matplotlib etc for generating complex graphical data, the creation of histograms etc.
  • UsedPython's XML parser architectures (SAX) and DOMAPI for tracking small amounts of data without requiring the DB.
  • Profiledpythoncode for optimization and memory management.
  • Designed and led big data algorithms using Hadoop and Java to improve forecasting and used HiveQL to generate reports.

Environment: Python2.6/2.7, Linux, Shell Scripting, JavaScript, Django Framework, CSS, SQL, MySQL, LAMP, JQuery, Apache web server.

We'd love your feedback!