Data Engineer Resume
San Francisco, CA
SUMMARY:
- Big Data and Cloud engineering professional with 9 years of experience, passionate to work in the intersection of cloud computing, machine learning and scalable distributed systems. Experienced in building distributed data solutions, data analytical applications, predictive modeling and ETL and streaming pipelines leveraging big data ecosystem components, Databricks platform, AWS and Azure Cloud services.
TECHNICAL SKILLS:
Programming Languages: Python, R language, YAML, Bash scripting, SAS Base, SAS Enterprise Miner, Regular Expressions and SQL
Databases: Relational - Oracle 11g, SQL Server, MySQL, No-SQL - Elastic, Cosmos DB.
Cloud Infrastructure: AWS Cloud - AWS S3, AWS Lambda, Step functions, AWS Athena, Code pipelines, AWS EMR, AWS EC2.
Big Data Tools: Snowflake, Databricks, Spark/PySpark, Kafka streaming, Hive, Snowflake, Map Reduce, HDFS, Sqoop, Flume, Kafka and Oozie
Machine Learning & AI Frameworks: Data Mining- Logistic Regression, Linear Regression, Support Vector Machines, Decision trees, Time Series, Neural Networks and other advances statistical techniques.
Natural Language Processing: Text Pre-Processing, Information Retrieval, Classification, Text Clustering, Word2Vec, Sentiment Analysis and Natural Language Understanding (NLU).
Business Intelligence: Kibana, Tableau, Qlik Sense, Google Cloud Data Studio, Advanced Microsoft Excel and Power BI.
Version Control: Git, SVN, Bit bucket
PROFESSIONAL EXPERIENCE:
Data Engineer
Confidential, San Francisco, CA
Responsibilities:
- Designed and implemented near real time data pipeline using framework based on Kafka, Spark structured streaming. Leveraged Kafka a publish-subscribe messaging system by creating topics using consumers and producers to ingest data into the AWS S3 for Spark to process the data and create data into DATABRICKS delta.
- Developed ETL pipelines using AWS S3, AWS Athena, AWS Redshift and AWS EMR to Extract, Transform and load data from various third party and microservices. Built data migration jobs into enterprise AWS cloud and Snowflake data warehouse.
- Built and implemented big data ingestion pipelines to process TB size data from various data sources using Kafka, Pyspark and Snow pipe. Developed data audit checks and stored as efficient columnar storage formats enabling data wrangling for a variety of downstream analytics purposes.
Azure Data Engineer
Confidential, Sunnyvale, CA
Responsibilities:
- Designed and developed scalable data processing and analytics framework and integrated with Azure cloud services. Migrated on-prem databases to Azure data Lake, Azure Blob, Azure SQL Database, Azure SQL leveraging Azure Data bricks and Azure Data Factory (ADF).
- Extracted, Transformed and Loaded (ETL) data from disparate sources to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and Azure Data Lake Analytics.
- Created ETL/ELT pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back to and backwards. Built data migration jobs into enterprise Azure cloud and Snowflake.
- Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats and loaded into Delta lake layer.
- Responsible for estimating the cluster size, monitoring and troubleshooting of the Azure data bricks cluster.
- Analyzed and build cloud data solutions using Azure PaaS service to support visualization of data. Performed as-is analysis on current production state of application and determined the impact of cloud implementation on existing business processes.
Data Engineer
Confidential, Chicago, IL
Responsibilities:
- Designed and built Spark/Pyspark based ETL pipelines for migration of credit card transactions, account and customer data into enterprise Hadoop data lake. Developed strategies in handling large datasets using partitions, Spark SQL, broadcast joins and performance tuning.
- Developed and implemented automated ETL/ELT pipelines using Databricks/Delta Lake, AWS S3, AWS Athena and AWS EMR. Automated data management in AWS S3 using AWS Lambda and Step Functions.
- Built and implemented performant data pipelines using Apache Spark on AWS EMR. Performed maintenance of data integration programs into Hadoop and RDBMS environments from both structured and semi- structured data source systems.
- Developed performance tuning on existing Hive queries and UDF’s to analyze the data. Used Pig to analyze datasets and perform transformation according to requirements.
Data Engineer
Confidential, Dallas, TX
Responsibilities:
- Achieved reduction of recurring incidents up to 60% by implementing a natural language processing and statistical modeling-based approach. Identified nearest-neighbor NCIs (Non-Conformance Incidents) reported for products/process that were manufactured across global manufacturing sites.
- Developed a text classification model that categorize incidents to product related or process related for further root cause analysis. Used Python for text mining.
- Build to Max Forecast: Reduced overbuilding of finished goods inventory (FGI) to about 40% compared to as-is model. Built a Time Series Forecasting model to forecast Build to Max for each product for different theatres. Provided recommendations for building optimal safety stock in order to avoid overbuild or under build situations. Blended lead times for damaged goods processing at different repair sites with demand variability for calculating safety stock.
- Predominantly used Python, HDFS, Sqoop, Hive (HQL) and Oracle SQL developer in providing insights to the business users.
Data Engineer
Confidential, Dallas, TX
Responsibilities:
- Designed and developed enterprise scale cloud alert mechanism using azure data bricks, Spark/Spark UI data processing framework (Python/Scala) and azure Data Factory. Built data pipelines to transform, aggregate and process data using Azure Databricks, Azure ADLS, Blob, Azure Delta and Airflow.
- Built and maintained data transformation jobs and productionalized ETL pipelines for cloud data migration. Performed data loading and transformation tasks using external sources, performed data enrichment and load into target data destinations. Debugged and optimized Spark jobs for performance and scalability.
- Performed data wrangling and developed data tables for excess inventory, stock outs, future high orders, forecast inaccuracy and delayed purchase orders on supply chain data. Performed extensive analyses on club-item and store level attributes to determine impacts on club-item replenishment.
- Developed and automated processes for predictive model validation, deployment, and productization of machine learning model for stock outs prediction.
Data Science Engineer
Confidential, Charlotte, NC
Responsibilities:
- Designed and developed a machine learning modelling framework in python which showed potential of 7% improvement in sales per customer with an incremental revenue of ~3M. Leveraged rule-based extraction engines and ensemble of machine learning models to predict likelihood of a customer making a purchase.
- Used big data technology tools to interact with Hadoop data lake for data transformation. Performed ETL on disparate data sources like online transactional data, web services data (RESTful), payment, orders history & marketing campaigns data to provide customer insights.
- Utilized distributed systems and built data pipelines (Python, SQL, Hadoop, Hive,) to determine product impacts on different marketing channels, and proposed solutions to drive impact for intensive analyses.
- Marketing Analytics: Designed a robust physician segmentation framework based on physician prescribing potential and adoption rate of branded drugs. Predicted physician lifetime value for each segment groups leveraging APLD patient level data from Symphony, IMS Xponent, IMS Sales and Distribution data (DDD) and various internal datasets.
- Performed A/B testing by sending emails to certain physician segments in the categories while maintaining a control population to observe the incremental impact of the emails.
Senior Data Analyst
Confidential
Responsibilities:
- Developed interactive dashboards using Tableau and made recommendations utilizing exploratory analysis that facilitated evaluation of quality and monitor the performance of BA/BE trial sites that contribute to potential risk.
- Worked closely with Regulatory Affairs team, and interacted with ETL developers, Project Managers, and members of the QA teams for successful reporting across enterprise and ensured consistency on Key Performance Metrics (KPI’s). Created custom Function's (Date range, Time functions, Logical functions) for the reports. Designed, developed, tested, and maintained functional reports based on user requirements.
Data Analyst
Confidential
Responsibilities:
- Performed data analysis on large relational datasets using optimized diverse SQL queries, such as DML and DDL statements including joins, group functions, string functions, and ranking functions.
- Developed queries to create, modify, delete and update the Oracle database and to analyze the data.
