SUMMARY

• 7+ years of experience in Big Data frameworks, Data Pipeline Design, Development and Implementation as a Sr. Data Engineer/Data Analyst.
• Recommended and used various best practices to improve dashboard performance for Tableau server users.
• Expert wif the design of custom reports using data extraction and reporting tools, and development of algorithms based on business cases.
• Extensively usedPythonLibraries PySpark, Pandas, OS, NumPy, Snowflake.connector, shutil,glob etc.
• Hands on in performance tuning and reporting for optimization using various methods like Extracts, Context filters, writing efficient calculations, Data source filters, Indexing, and Partitioning over SQL.
• Strong experience in migrating other databases to Snowflake.
• Created dashboards in Tableau using various features of Tableau like Custom - SQL, Multiple Tables, Blending, Extracts, Parameters, Filters, Calculations, Context Filters, Data source filters, Hierarchies, Filter Actions, Maps, etc.
• Experience wif Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source systems which include loading nested JSON formatted data into snowflake table.
• Expert wif BI tools like Tableau and PowerBI, data interpretation, modeling, data analysis, and reporting wif the ability to assist in directing planning based on insights.
• Hands-on experience wif Snowflake utilities, SnowSQL, SnowPipe, Big Data model techniques using Python
• ETL pipelines in and out of data warehouses using a combination of Python and Snowflakes SnowSQL Writing SQL queries against Snowflake.
• Experienced in working wif Amazon Web Services (AWS) using EC2 for computing and S3 as storage mechanism.
• Capable of using AWS utilities such as EMR, S3 and cloud watch to run and monitor Hadoop and Spark jobs on AWS.
• Hands-on experience in using other Amazon Web Services like Autoscaling, DynamoDB, Route53.
• Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement.
• Proficiency in SQL across several dialects (we commonly write MySQL, PostgreSQL,, SQL Server, and Oracle)
• Experience in designing star schema, Snowflake schema for Data Warehouse, ODS architecture. Creative skills in developing elegant solutions to challenges related to pipeline engineering
• Hands-on knowledge on Coding skills wif PySpark, Spark Context, and Spark SQL.
• Strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies.

TECHNICAL SKILLS

Operating System: Windows, LINUX, UNIX,macOS

Languages: C, C++, HQL, PySpark, Scala, Python, PL/SQL, SQL, Unix, and Linux

Databases: Oracle, SQL Server, MySQL

Bigdata Ecosystems: Hadoop, HBase MapReduce, Sqoop, HIVE, HDFS, Pig, Spark, Kafka Oozie, Flume, Zookeeper

Methodologies: Agile, JIRA, Scrum,Trello

Cloud: Google Cloud (VMs, Big Query, Dataflow, Dataproc, Fusion, CloudSQL). AWS (EC2, S3, VPCIAM, RDS).

ETL Tools: DataStage, Talend

Data Visualization: Tableau, Power BI

Data Formats: Parquet, Sequence, AVRO, ORC, CSV, JSON

Version Control: GITLAB,GITHUB, IntelliJ

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, Allen, TX

Responsibilities:

• Installed, configured and maintained Data Pipelines, Developed Data Pipeline wif Kafka and Spark.
• Architect and implement ETL and data movement solutions using Azure Data Factory (ADF), SSIS.
• Developed processes for loading the data into snowflakes. Designed data modeling on the data and joined them wif other DIM tables for tableau reporting.
• Authored Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks.
• Experienced wif cloud platforms like Amazon Web Services, Azure, Databricks (both on Azure as well as AWS integration of Databricks)
• Architect and implement ETL and data movement solutions using Azure Data Factory, SSIS create and run SSIS Package ADF V2 Azure-SSIS IR.
• Hands - on experience in Azure Cloud Services (Paas & laaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Evaluated Snowflake Design considerations for any change in the application, and built the Logical and Physical data model for snowflake as per the changes required.
• Redesigned the Views in snowflake to increase the performance and Unit tested the data between and Snowflake.
• Spark for ETL follower, Databricks Enthusiast, Cloud Adoption & Data Engineer enthusiast in open source community.
• Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB).
• Proficient wif Azure Data Lake Services (ADLS), Databricks & iPython Notebooks formats, Databricks Deltalakes & Amazon Web Services (AWS).
• Developed data warehouse model in snowflake for over 100 datasets using whereScape and Created Reports in Looker based on Snowflake Connections
• Migration of on-premise data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake Store (ADLS) using Azure Data Factory (ADF V1/V2).
• Have a good experience working wif Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW).
• Developed solutions to leverage ETL tools and identify opportunities for process improvements using Scheduling tool and Python
• Designed and implemented for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Used AWS services like EC2 and S3 for small data sets processing and storage. Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Worked on Dimensional and Relational Data Modeling using Star and Snowflake Schemas, OLTP/OLAP.
• Developed Automation Regressing Scripts for validation of ETL process between multiple databases like AWS, SQL Server usingPython.

Environment: AWS,Snowflake, Kafka, Spark, Lambda, Hadoop, Tableau, Python, Hive, SQL, Oracle, scheduling tool, Shell scripting.

Data Engineer

Confidential

Responsibilities:

• Designed the business requirement collection approach based on the project scope and SDLC methodology.
• Installed, configured and maintained Data Pipelines, Developed Data Pipeline wif Kafka and Spark.
• Transforming business problems into Big Data solutions and defining Big Data strategy and Roadmap.
• Authored Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks.
• Evaluated Snowflake Design considerations for any change in the application, and built the Logical and Physical data model for snowflake as per the changes required.
• Redesigned the Views in snowflake to increase the performance and Unit tested the data between and Snowflake.
• Developed data warehouse model in snowflake for over 100 datasets using whereScape and Created Reports in Looker based on Snowflake Connections.
• Written Pig Scripts to generate Map Reduce jobs and performed ETL procedures on the data in HDFS.
• Developed solutions to leverage ETL tools and identify opportunities for process improvements using Informatica and Python.
• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
• Designed and implemented for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.
• Used Sqoop to channel data from different sources of HDFS and RDBMS.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Created Tables, Stored Procedures, and extracted data using PL/SQL for business users whenever required and used SSIS to build automated multi-dimensional cubes.
• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra.
• Used Spark for interactive queries, processing of streaming data and integration wif popular NoSQL databases for huge volumes of data.
• Expansively worked wif Partitions, Dynamic Partitioning, bucketing tables in Hive, designed both Managed and External tables, also worked on optimization of Hive queries.
• Developed Spark Applications by using Scala, Python and Implemented Apache Spark data processing Project to handle data from various RDBMS and Streaming sources.
• Using SparkContext, Spark-SQL, Spark MLlib, Data Frame, Pair RDD, Spark YARN.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common.
• Learner data model which gets the data from Kafka in near real time and persists it to Cassandra.
• Developed Kafka consumer API in Scala for consuming data from Kafka topics.
• Experienced in writing live Real-time Processing and core jobs using Spark Streaming wif Kafka as a data pipe-line system. Migrated an existing on-premises application to AWS.

Environment: AWS, MapReduce, Cloudera, Snowflake Kafka, Spark, Lambda, Hadoop, Hbase, Tableau, Informatica, Python, Hive, PL/SQL, Oracle, Unix, Shell Scripting.

Data Engineer

Confidential

Responsibilities:

• Headed negotiations to find optimal solutions wif project teams and clients
• Mapped client business requirements to internal requirements of trading platform products
• Supported revenue management using statistical and quantitative analysis, developed several statistical approaches and optimization models.
• Led the business analysis team of four members, in absence of the Team Lead
• Added value by providing innovative solutions and delivering improved methods of data presentation by focusing on the Business need and the Business Value of the solution.
• Responsible for data extraction, Exploratory data analysis, understand the ‘Why’ behind data
• Statistical analysis, Data analysis, anomaly/Outlier detection using R programming.
• Developed Tableau dashboards & reports to show the analysis insights.
• Extensively used SQL for accessing and manipulating database systems
• Experience in using various packages in R and python-like ggplot2, caret, dplyr, Rweka, gmodels, twitter, NLP, Reshape2, rjson, plyr, pandas, NumPy, Seaborn, SciPy, Matplotlib, sci-kit-learn, Beautiful Soup.
• Extensive experience in Text Analytics, generating data visualizations using R, Python.
• Hands on experience wif big data tools like Hadoop, Spark, Hive, Pig, PySpark, Spark SQL, PySpark
• Hands-on experience in implementing LDA, Naive Bayes and skilled in Random Forests, Decision Trees, Linear and Logistic Regression, SVM, Clustering, neural networks, TEMPPrincipal Component Analysis.

Environment: Exploratory data analysis, R, python, SQL, Tableau, Hadoop, Spark, Hive, Pig, PySpark, Spark SQL.