Sr. Data Engineer Resume
Columbus, OH
SUMMARY
- Qualified professional with over 8+ years of extensive experience in IT industry including Big Data, Hadoop environments on premise and Cloud environments for hosting cloud - based data warehouses and databases using Redshift, Cassandra and RDBMS sources.
- Adept in writing Python scripts to parse XML documents and load the data in database.
- Extensive experience of implementing solutions using AWS services like (EC2, S3, and Redshift), Hadoop HDFS architecture and Map-Reduce framework.
- Worked in AWS environment for development and deployment of custom Hadoop applications
- Strong understanding of developing MapReduce programs using Apache Hadoop for working with Big Data.
- Hands on experience in working of big data analysis using Pig and Hive.
- In-depth knowledge of various Cloudera distributions like (CDH 4/CDH 5), Knowledge of working on Hortonworks and Amazon EMR Hadoop distributors.
- Developed Pyspark Scripts to process streaming data from data lakes using Spark Streaming.
- Experienced in creating Spark jobs that run in EMR clusters using AMR Notebooks.
- Hands on experience in Python Boto3 for developing Lambda functions in AWS.
- Analysis was done using Python libraries such as PySpark.
- Hands on expertise in writing different RDD (Resilient Distributed Datasets) transformations and actions using Scala and Python.
- Strong experience in working with ELASTIC MAP REDUCE (EMR) and setting up environments on Amazon AWS EC2 instances.
- Involved in development of data processing framework (flow master and DCT) using Pyspark.
- Hands on experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4), Yarn distributions (CDH 5.X).
- Experience working with big data and real time/near real time analytics using the big data platforms like Hadoop and Spark using Python.
- Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.
- Solid programming knowledge on Python and shell scripting.
- Experience in migrating on premise to Windows Azure in DR on cloud using Azure Recovery Vault and Azure backups.
- Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and Migrating On premise databases to Azure Data Lake store using Azure Data factory.
- Strong understanding of Data Modelling and experience with Data Cleansing, Data Profiling and Data analysis.
- Good Experience on SDLC (Software Development Life cycle) like Agile, Scrum.
- Experience in all phases of SDLC like Requirement Analysis, Implementation and Maintenance
- Prepare technical reports by collecting, analyzing, and summarizing information and trends.
- Expert in performing business analytical scripts using HiveQL.
- Extensively used Spark Data Frames API over Cloudera platform to perform analytics on Hive data and used Spark Data Frame Operations to perform required to Validations in the data.
PROFESSIONAL EXPERIENCE
Sr. Data Engineer
Confidential, Columbus, OH
Responsibilities:
- Developed various data loading strategies and performed various transformations for analyzing the datasets by using Hortonworks Distribution for Hadoop ecosystem.
- Implemented solutions for ingesting data from various sources utilizing Big Data technologies such as Hadoop, Map Reduce Frameworks, Hive
- Developing the PySpark applications for Spark SQL, Data frames and transformations using Python APIs to perform the business requirement on Hive staging tables, and load the final transformed data into Hive master tables
- Worked as a Hadoop consultant on technologies like Map Reduce, Pig, and Hive.
- Involved in ingesting large volumes of credit data from multiple provider data sources to AWS S3. Created modular and independent components for AWS S3 connections, datareads.
- Implemented Data warehouse solutions in AWS Redshift by migrating the data to Redshift from S3.
- Developed Spark code using Python to run in the EMR clusters.
- Created User Defined Functions (UDF) using Scala to automate some business logic in the applications.
- Automated the jobs and data pipelines using AWS Step Functions, AWS Lambda and configured various performance metrics using AWS Cloud watch.
- Worked using Apache Hadoop ecosystem components like HDFS, Hive, Pig, and Map Reduce.
- Designed AWS Glue pipelines to ingest, process, and store data interacting with different services in AWS.
- Implemented usage of Amazon EMR for processing Big Data across Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
- Developed a process to migrate local logs to CloudWatch for better integration and monitoring.
- Executed the program by using python API written in python to support Apache Spark or PySpark.
- Helped Dev ops Engineers for deploying code and debug issues.
- Worked in writing Hadoop Jobs for analyzing data like Text format files, sequence files, Parquet files using Hive and Pig.
- Worked on analyzing Hadoop cluster and different Big Data components including Pig, Hive, Spark, and Impala.
- Populated database tables via AWS Kinesis Firehose and AWS Redshift.
- Developed Spark code using Python and Spark-SQL for faster testing and data processing.
- Created Hive External tables and loaded the data into tables and query data using HQL.
- Developed ETL modules and data workflows for solution accelerators using PySpark and Spark SQL.
- Used Spark SQL to process the huge amount of structured data.
- Extracted the data from MySQL and AWS RedShift into HDFS using Kinesis.
- Developed Pyspark application for creating reporting tables with different masking in both Hive and MySQL DB and made available for newly build fetch API’s.
- Wrote numerous Spark code in Scala for information extraction, change, and conglomeration from numerous record designs.
Environment: Big Data, Spark, Hive, Pig, Python, Hadoop, AWS, Databases, AWS RedShift, Agile, SQL, HQL, Impala, CloudWatch, AWS Kinesis
Cloud Data Engineer
Confidential, Englewood, CO
Responsibilities:
- Created Pipelines in ADF using Linked Services/Datasets/Pipeline to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Datawarehouse, write-back tool, and backward.
- Extracted, Transformed and Loaded data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and U-SQL Azure Data Lake Analytics.
- Data is Ingested to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
- Worked on Azure Services like IaaS, PaaS and worked on storage like Blob (Page and Block), SQL Azure.
- Implemented OLAP multi-dimensional functionality using Azure SQL Data Warehouse.
- Retrieved data using Azure SQL and Azure ML which is used to build, test, and predict the data.
- Worked on Cloud databases such as Azure SQL Database, SQL managed instance, SQL Elastic pool on Azure, and SQL server.
- Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB)
- Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
- Designed and developed Azure Data Factory pipelines to Extract, Load and Transform data from difference sources systems (Mainframe, SQL Server, IBM DB2, Shared Drives, etc.) to Azure Data Storage services using a combination of Azure Data Factory, Azure Databricks (PySpark, Spark-SQL), Azure Stream Analytics and U-SQL Azure Data Lake Analytics. Data Ingestion into various Azure Storage Services like Azure Data Lake, Azure Blob Storage, Azure Synapse Analytics (formerly known as Azure Data Warehouse).
- Configured and deployed Azure Automation Scripts for a multitude of applications utilizing the Azure stack (including Compute, Web & Mobile, Blobs, ADF, Resource Groups, Azure Data Lake, HDInsight Clusters, Azure Data Factory, Azure SQL, Cloud Services, and ARM), Services and Utilities focusing on Automation.
- Involved in Migrating Objects from Teradata to Snowflake and created Snow pipe for continuous data load.
- Increased consumption of solutions including Azure SQL Databases, Azure Cosmos DB, Azure SQL.
- Created continuous integration and continuous delivery (CI/CD) pipeline on Azure that helps to automate steps in the software delivery process.
- Deploying and managing applications in Datacenter, Virtual environment, and Azure platform as well.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDD's and PySpark.
- Log data Stored in HBase DB is processed and analyzed and then imported into Hive warehouse, which enabled end business analysts to write HQL queries.
- Handled importing of data from various data sources, performed transformations using Hive, and loaded data into HDFS.
- Design, development, and implementation of performant ETL pipelines using PySpark and Azure Data Factory.
Environment: Azure Data Factory(V2), Azure Data Bricks (Pyspark, Spark SQL), Azure Data Lake, Azure BLOB Storage, Azure ML, Azure SQL, Hive, Git, GitHub,JIRA, HQL, Snowflake, Teradata
Data Engineer
Confidential, Charlotte, NC
Responsibilities:
- Worked on AWS Cloud Formation templates for using Terraform with existing plugins.
- Used AWS Cloud Formation to ensure successful deployment of database.
- Configured Sqoop to import/export data from database to HDFS and Data Lake on AWS.
- Implemented Spark in EMR for processing Big Data across our Data Lake in AWS System.
- Developed Docker images to support Development and Testing Teams and their pipelines and distributed images like Jenkins, Selenium, JMeter and ElasticSearch, LogStash and Kibana (ELK), and handled the containerized deployment using Kubernetes.
- Installed, configured, and managed the ELK (ElasticSearch, Logstash and Kibana) for Log management within AWS EC2/ Elastic Load balancer for Elastic search involving in cloud automation with configuration management system Ansible.
- Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregations to build the data model and persists the data in HDFS.
- Configured AWS IAM and Security Group as per requirement and distributed them as groups into various availability zones of the VPC.
- Implemented a Spark-Streaming consumer job to consume data in near real time from AWS Kinesis and sink to S3 for downstream systems consumptions.
- Involved in deploying spark and hive applications in AWS stack.
- Worked on architecting Serverless design using AWS API, Lambda, S3 and Dynamo DB with optimized design with Auto scaling performance.
- Populated database tables via AWS Kinesis Firehose and AWS Redshift.
- Led many critical on-prem data migration to AWS cloud, assisted in performance tuning and providing successful path towards Redshift Cluster and AWS RDS DB engines.
- Set up Scala scripts to create snapshots on AWS S3 buckets and deleted old snapshots.
- Worked on AWS S3 bucket integration for application and development projects.
- Utilized Agile Scrum Methodology to help manage and organize a team of 4 developers with regular code review sessions.
- Experience managing and reviewing Hadoop log files in AWS S3.
Hadoop Developer
Confidential
Responsibilities:
- Installed and configured Flume, Hive, Pig, and Oozie on the Hadoop cluster.
- Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.
- Also used Spark SQL to handle structured data in Hive.
- Involved in making Hive tables, stacking information, composing hive inquiries, producing segments and basins for enhancement.
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and extracted data from Teradata into HDFS using Sqoop.
- Written and Implemented Teradata Fast load, Multiload and Bteq scripts, DML and DDL.
- Involved in migrating tables from RDBMS into Hive tables using Sqoop and later generate particular visualizations using Tableau.
- Analyzed substantial data sets by running Hive queries and Pig scripts.
- Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
- Involved in transforming data from Mainframe tables to HDFS, and HBase tables using Sqoop.
- Defined the Accumulo tables and loaded data into tables for near real-time data reports.
- Created the Hive external tables using Accumulo connector.
- Ability to spin up different AWS instances including EC2-classic and EC2-VPC using cloud formation templates.
- Imported the data from different sources like AWS S3, LFS into Spark RDD.
- Involved in creating Shell scripts to simplify the execution of all other scripts (Pig, Hive, Impala and MapReduce) and move the data inside and outside of HDFS.
- Creating files and tuned the SQL queries in Hive utilizing HUE.
- Experienced in working with spark eco system using Spark SQL and Scala queries on different formats like Text file, CSV file.
- Designed Power BI data visualization utilizing cross tabs, maps, scatter plots, pie, bar and density charts.
- Expertized in implementing Spark using Scala and Spark SQL for faster testing and processing of data responsible to manage data from different sources.
- Worked with NoSQL databases like HBase in making the tables to load expansive arrangements of semi structured data.
- Designed the ETL process and created the high-level design document including the logical data flows, source data extraction process, the database staging, job scheduling and Error Handling
- Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.
Environment: Hadoop, Cloudera, HDFS, MapReduce, YARN, Hive, Pig, Sqoop, HBase,Apache Spark, Accumulo, Oozie Scheduler, Kerberos, AWS, Tableau, Talend, HUE, HCATALOG, Flume, Git, Maven.
Data Analyst
Confidential
Responsibilities:
- Involved in the complete life cycle of the project performing various tasks like Data Understanding by performing Exploratory Data Analysis, Data Cleansing, Data Modeling for creating visual representation, statistics, and marketing from several sources with the business, Evaluating and Deploying with deep knowledge of consumer and marketing analysis
- Closely worked with business requirements converting them into technical requirements and working with data owners and stewards to gather all the data requirements for analysis for management of reports with help of excel pivot tables and Presentation charts with applied knowledge of HEDIS metrics.
- Developed, analyzed, reported, and interpreted complex data for ongoing activities/projects while ensuring that all data are accurate.
- Worked with Data Governance tools and extract-transform-load (ETL) processing tool for data mining, data warehousing, and data cleaning using SQL.
- Used data to identify trends, needs, and opportunities, and prepares reports, visualizations, and recommendations to help the district determine what’s working and what needs to change.
- Integrating Word, Excel, and PowerPoint for business communication more effective by organizing separated information into one place for easy access and analysis.
- Optimized SQL performance, integrity, and security of the project’s databases/schemas.
- Performed Data cleaning, Data pre-processing, and Manipulation using Python.
- Developed intricate algorithms based on deep-dive statistical analysis and predictive and time-series data modeling.
- Devised effective metrics, KPIs, Visualizations, and built Dashboards for reporting the Daily, Weekly, and Monthly summary, trending, and benchmark reports to the Management in Tableau Desktop.
- Have good working knowledge of Michigan reporting requirements, including the Registry of Educational Personnel (REP) reporting
- Statistical analysis and data mining for Utilizing software to look for patterns in large batches of data so can learn more about customers.
Environment: Tableau (Power BI), R, Python, SQL, PL/SQL, PostgreSQL, Excel, Spark,Databricks (Jupyter Notebooks), Redshift, ETL, AWS, Nifi, Netezza,Hadoop, Hive, Quantitative Analysis.
