We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

3.00/5 (Submit Your Rating)

Seattle, WA

SUMMARY

  • me have 7+ years of IT experience on Migrating SQL database to AzuredataLake, Azuredatalake Analytics, Azure Database,DataBricks and Azure SQLDatawarehouse and controlling and granting database access and Migrating On premise databases to AzureDataLake store using AzureDatafactory.
  • Experience in all phases of diverse technology projects specializing in Data Science and Machine Learning.
  • Experience in Developing Spark applications using Spark - SQL in Data bricks fordataextraction, transformation, and aggregation from multiple file formats for analyzing & transforming thedatato uncover insights into teh customer usage patterns.
  • Extensive Experience in IT data analytics projects, Hands on experience in migrating on premise ETLs to Google Cloud Platform (GCP) using cloud native tools such as BIG query, Cloud Data Pros, Google Cloud Storage, Composer.
  • Good understanding of BigDataHadoop and Yarn architecture along wif variousHadoop Demons such as Job Tracker, Task Tracker, Name Node,DataNode,Resource/Cluster Manager, and Kafka (distributed stream-processing).
  • Experience in Database Design and development wif Business Intelligence using Integration Services (SSIS), DTS Packages, SQL ServerAnalysis Services (SSAS), DAX, OLAP Cubes, Star Schema and Snowflake Schema.
  • Expertise in various phases of project life cycles (Design, Analysis,Implementation, and testing).
  • Proficiency in handling software development sprints, test, and delivery cycle for teh development teams.
  • Expertise in working on Star Schema and Snowflakes Schema methodologies.
  • Experience in Bash and Python scripting wif focus on DevOps tools, CI/CD and AWS Cloud Architecture and hands-on Engineering
  • Excellent understanding of best practices of Enterprise Data Warehouse and involved in Full life cycle development of Data Warehousing.
  • Expertise Data Models and Dimensional Modeling wif 3NF, Star and Snowflake schemas for OLAP and Operational data store (ODS) applications.
  • Extensive Shell/Python scripting experience for Scheduling and Process Automation.
  • Good exposure to Development, Testing, Implementation, Documentation and Production support.
  • Expert in providing ETL solutions for any type of business model.
  • Hands on experience in analyzing data using Ecosystems including HDFS, Map Reduce, Hive & PIG.
  • Extensive experience in working wif Informatica Power center.

TECHNICAL SKILLS

Programming Languages: Python and basics of R and Java programming

Packages: ggplot2, caret, dplyr, Wreak, gmodels, RCurl, Twitter, NLP, Reshape2, Json, duly, pandas, NumPy, Seaborne, SciPy, Matplotlib, Scikit - learn, Beautiful Soup, Rpy2.

Reporting Tools: Tableau, SAS BI, Microsoft Power BI

Databases: SQL, Hive, Spark SQL, MYSQL

Big Data Technologies: Spark and Hadoop

ETL Tools: SSIS

Operating System: Windows, Linux/Unix Work Experience

PROFESSIONAL EXPERIENCE

Confidential, Seattle, WA

Sr. Data Engineer

Responsibilities:

  • Involved in converting Hive/SQL queries into Spark transformations using Spark Data Frames and Scala.
  • Developed shell scripts for handling teh movement of data files between HDFS and local file system.
  • Develop ETL's to source data extraction, transformation and population processes.
  • Involved in Unit Testing, Integration Testing and Regression Testing in a regular basis to improve teh performance of teh application.
  • Developed shell scripts for scheduling/running ETL jobs using pmcmd command.
  • Involved in analysis, specification, design, and implementation and testing phases of Software Development Life Cycle (SDLC) and used Agile Methodology (SCRUM) for developing application.
  • Developed Spark scripts by using Scala shell commands as per teh requirement.
  • Implemented data aggregations using Scala on Spark.
  • Involved in relational and dimensional Data Modeling for creating Logical and Physical design of teh database and ER diagrams using data modeling like Erwin
  • Developed multiple POCs using Spark Scala and deployed on teh Yarn cluster, compared teh performance of Spark, wif Hive.
  • Involved in teh process of designing Google Cloud Architecture.
  • Developed multiple POCs using PySpark (Python) and deployed machine learning models on teh Yarn cluster.
  • Developed POC’s on Spark-Streaming APIs to perform necessary transformations and actions for building teh common learner data model which gets teh data from Kafka in near real time and persists into Cassandra.
  • Implemented static Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table.
  • Developed Data governance methods for handling teh different domains of Customer data.
  • Designed, developed, and did maintenance of data integration programs in a Hadoop and RDBMS environment wif both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
  • Implemented MVC design pattern for developing teh Web Application and used MVC5 application using Razor syntax for view engine also used C# for teh back end.
  • Build data pipelines in airflow in GCP for ETL related jobs using different airflow operators.
  • Created a Git repository and added teh project to GitHub. Creating Reports in Looker based on Snowflake Connections
  • Delivered teh content using HTML5, CSS3, JS, and deployed teh application to production using Linux operating system and Apache web server.
  • Engineered multiple connectors on IBM Streams for processing Site Catalyst (Online Interaction) data.
  • Worked on Hive, Big query (BQ) for exporting data for further analysis and for generating transforming files from different analytical formats to text files.
  • Worked wif PowerShell and UNIX scripts for file transfer, emailing and other file related tasks.

Environment: GCP, Java, Big query, Gas Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Faw,Bq Command Line Utilities, DataProc, Cloud Sql, MySQL, Postgres, Sql Server, Python, Scala, Spark, Hive, Spark-Sql

Confidential, Troy, MI

Data Engineer

Responsibilities:

  • Designed and developed custom data integration pipelines on Facebooks big data stack such as python, YAML, Hive, Vertica and Data swarm
  • Designed and developed custom aggregation framework for reporting and analytics in Hive, Presto and Vertica
  • Developed ETL mappings and workflows using Informatica and Data swarm.
  • Developed business logic and implemented functionality in C#.
  • Interacted wif Business Analysts and Data Modelers and characterized Mapping reports and Design process for different Sources and Targets.
  • Designed and developed Spark Scala code to fast processing of Havel queries.
  • Developed HIVE scripts to transfer data from and to HDFS.
  • Involved in different phases of building teh Data Marts like analyzing business requirements, ETL process design, performance enhancement, go-live activities and maintenance.
  • Developed Scala scripts, UDFFs using both Data frames/SQL/Data sets and RDD/Map.
  • Developed POC to execute Machine Learning models on Data bricks platform using SparkML library.
  • Performed teh Data Accuracy, Data Analysis, Data Quality checks before and after loading teh data.
  • Migrating servers, databases, and applications from on premise to AWS, Azure. Processing teh schema oriented and non-schema-oriented data using Scala and Spark.
  • Worked wif Teradata, SQL Server, Apache Spark, Sqoop. Migrated Postgresql and MySQL databases from on-premises to AWS EC2 and RDS environments.
  • Worked on Business Intelligence standardization to create database layers wif user-friendly views in Vertica dat can be used for development of various Tableau reports/ dashboards.
  • Used Informatica to extract, transform & load data from SQL Server to Oracle databases. Created dynamic BI
  • Report/dashboard for production support in Excel/PowerPoint/Power BI/Tableau/ My SQL Server/ PHP.
  • Worked on complex information model, logical relationships, and teh data structures from MySQL, ORACLE, and HIVE/PRESTO.

Environment: Hadoop, AWS, Java, Map Reduce, HDFS, Hive, Sqoop, Spring Boot, Cassandra, Swamp, Data Lake, Oozie, Kafka, Spark, Scala, Java, Azure, GitHub, Docker, Talent Big Data Integration.

Confidential

Data Engineer

Responsibilities:

  • Developed more efficient data collection procedures, wif automated data cleaning and transformation procedures before inserting them into teh database.
  • Developed a NiFi Workflow to pick up teh data from Data Lake as well as from server and send dat to Kafka broker.
  • Involved in integrating hive queries into spark environment using Spark-SQL.
  • Developed a Spark job in Java which indexes data into Elastic Search from external Hive tables which are in HDFS.
  • Designing and developing code, scripts and data pipelines dat leverage structured and instruct
  • Developed Automation Regressing Scripts for validation of ETL process between multiple databases like Redshift, MongoDB, T-SQL, and SQL Server using Python.
  • Worked closely wif teh technical team on special projects requiring data-driven Python visualizations to integrate into their applications
  • Designed different A/B test experiments for validating business solutions based on past performance to make new recommendations.
  • Worked extensively in Python to extract, compile, analyze data, and make effective visualizations reflecting insights obtained for senior executives
  • Implemented Spark-SQL wif various data sources like JSON, Parquet, ORC, and Hive. Worked closely wif teh managing leadership to accommodate sales and business data-related information requirements.
  • Worked closely wif teh analytics and sales team when undertaking a new project right from determining teh problem definition, through data acquisition, exploration, and visualization, to evaluating metrics for teh same.
  • Worked closely wif data scientists to assist on feature engineering, model training frameworks, and model deployments at scale.

Environment: Apache Hadoop, HDFS, Hive, Java, Sqoop, Spark, VMG (OATH), Teradata MySQL, Apache Oozie, SFTP, IBM Streams, Python, Scala.

Confidential 

Software Developer

Responsibilities:

  • Developed Front-end and Back-end applications for teh customers by using linear optimization.
  • Analyzed various real-world technical difficulties and providing Computational and Mathematical solutions.
  • Worked wif Developers in designing algorithms and flow-charts of applications and ensuring whether they met customer requirements.
  • Estimated, resolved, and escalated teh risks and issues.
  • Evaluating products including open-source modules and if need be, incorporating them into projects.
  • Collaborated wif tech leads & team members on teh design and Implementation.
  • Worked on understanding use cases wif various Dev teams and provided assistance wif troubleshooting, debugging, and testing.
  • Worked on Designing Relational Databases interface creation team to provide User interaction application.
  • Stayed current wif teh latest web technologies and evolving requirements to keep technology recommendations current.
  • Worked in an agile development environment.

We'd love your feedback!