SUMMARY

• Around 8 years of extensive IT experience as a data engineer with expertise in designing data - intensive applications using Hadoop Ecosystem and Big Data Analytical, Cloud Data engineering (AWS, Azure), Data Visualization, Data Warehouse, Reporting, and Data Quality solutions.
• Strong experience in end-to-end data engineering including data ingestion, data cleansing, data transformations, data validations/auditing and feature engineering.
• Strong experience troubleshooting failures in spark applications and fine-tuning spark applications and hive queries for better performance.
• Hands on experience using Hadoop tools like HDFS, Hive, Apache Spark, Apache Sqoop, Flume, Oozie, Apache Kafka, Apache storm, Yarn, Impala, Zookeeper, Hue.
• Hands on knowledge creating Amazon EC2 instances, S3 buckets on Amazon EMR.
• Hands on experience with AWS (Amazon Web Services), Elastic Map Reduce (EMR), Storage S3, EC2 instances and Data Warehousing.
• Experience in Migrating SQL database to Azure data Lake, Azure Synapse, Azure data lake Analytics, Azure SQL Database, Data Bricks, and Azure SQL Data warehouse and controlling and granting database access and migrating on-premises databases to Azure Data Lake store using Azure Data factory.
• Expertise in developing Spark applications utilizing Spark-Core, Data frames, Spark-SQL, Spark-ML and Spark-Streaming API's.
• Good exposure on usage of NoSQL databases column oriented HBase, Cassandra and MongoDB (Document Based DB).
• Worked extensively on building real time data pipelines using Kafka for streaming data ingestion and Spark Streaming for real time consumption.
• Experience in building, deploying, and integrating applications in Application Servers with ANT, Maven and Gradle.
• Experience with multiple cloud service providers AWS and Azure.
• Created Java apps to handle data in MongoDB and HBase.

TECHNICAL SKILLS

Big Data Technologies: HDFS, Hive, MapReduce, Pig, Phoenix, Sqoop, Flume, Zookeeper, Spark, Oozie

Devops Tools: Jenkins, Git

Monitoring Tools: Cloudera Manager, Ambari, Ganglia, Splunk

Scripting Languages: Shell Scripting

Programming Languages: Java, Scala and Python

Application Servers: Apache Tomcat, WebLogic Server, Web sphere

Cloud Platforms: Amazon Web Services (AWS), Azure

Databases: Oracle 11g, MySQL, MS SQL Server, IBM DB2, Teradata

NoSQL Databases: HBase, Cassandra, MongoDB.

PROFESSIONAL EXPERIENCE

Confidential, Cincinnati, OH

Big Data Developer

Responsibilities:

• Get Business requirements and get data from Data Science Team and apply transformations and validations and process it to downstream teams
• Daily deal with ~60 m records based on number of Stores.
• SME on Master Catalog, base pricing and Inventory Selection
• Implement code in Java based on requirements and run the job locally in IntelliJ and in Azure Databricks.
• Data is Ingested to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Created continuous integration and continuous delivery (CI/CD) pipeline on Azure that helps to automate steps in the software delivery process.
• Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
• Involved in Migrating Objects from Teradata to Snowflake and created Snowpipe for continuous data load.
• Worked on both batch processing jobs and streaming jobs.
• Worked on Azure Data Factory to build pipelines and schedule jobs
• Working on Different Azure Services like MS Azure Cosmos Db MongoDB API, MS Azure Databricks, Kafka, Apache Spark, Spark Structured Streaming and Other PaaS components such as Azure Key Vault.
• Increased consumption of solutions including Azure SQL Databases, Azure Cosmos DB, Azure SQL.
• Created Jenkins pipelines with YAML script to assist with frequent product releases across multiple teams. These pipelines reduced the man hours by 90%.
• Handled importing of data from various data sources, performed transformations using Hive, and loaded data into HDFS.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD's and PySpark.
• Design, development, and implementation of performant ETL pipelines using PySpark and Azure Data Factory.
• Neatly include information in the confluence pages regarding the implemented processors.

Environment: Azure Data Factory(V2), Azure Data Bricks, Azure Data Lake, Azure BLOB Storage, Azure ML, Azure SQL, PySpark, Hive, Git, GitHub, JIRA.

Confidential - Boise, ID

Big Data Developer

Responsibilities:

• Performed end-to-end Architecture and implementation assessment of various AWS services like Amazon EMR, Redshift, S3, Athena, Glue, and Kinesis.
• Created AWS RDS (Relational database services) to work as Hive metastore and could combine EMR cluster's metadata into a single RDS, which avoids the data loss even by terminating the EMR.
• Used AWS Athena extensively to ingest structured data from S3 into multiple systems, including RedShift, and to generate reports.
• Created on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark.
• Involved in code migration of quality monitoring tool from AWS EC2 to AWS Lambda and built logical datasets to administer quality monitoring on snowflake warehouses.
• Designed and implemented ETL pipelines on S3 parquet files on data lake using AWS Glue.
• Data pipeline consists of Spark, Hive and Sqoop and custom build Input Adapters to ingest, transform and analyze user behavior (clickstream) data.
• Involved in converting Hive/SQL queries into Spark transformations using Spark Data Frames and Scala.
• Created monitors, alarms, notifications, and logs for Lambda functions, Glue Jobs, EC2 hosts using CloudWatch, and used AWS Glue for the data transformation, validate and data cleansing.
• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kinesis in near real-time.
• Worked with Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source system which include loading nested JSON formatted data into snowflake table.
• Used AWS glue catalog with crawler to get the data from S3 and perform SQL query operations and JSON schema to define table and column mapping from S3 data to Redshift.
• Worked on EMR Security Configurations, to store the self-signed certificates as well as KMS keys created into it. This makes it spin up a cluster in an easy manner without modifying permissions after the call.
• Involved in continuous Integration of application using Jenkins.
• Involved in importing the real-time data using Kafka and implemented Oozie jobs for daily imports.
• Performed partitioning and Bucketing concepts in Apache Hive database, which improves the retrieval speed when someone performs a query.

Environment: Snowflake, HDFS, PySpark, Spark Streaming, Kafka Spark SQL, Python, Hive, Sqoop, Tableau, AWS S3, Athena, Lambda, CloudWatch Glue, Redshift, EMR, EC2, Linux.

Confidential

Big Data Engineer

Responsibilities:

• Collaborated with business user’s/product owners/developers to contribute to the analysis of functional requirements.
• Implemented Spark SQL queries that combine Hive queries with Python programmatic data manipulations supported by RDDs and data frames.
• Used Kafka Streams to Configure Spark streaming to get information and then store it in HDFS.
• Extract Real-time feed using Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data in HDFS.
• Developing Spark scripts, UDFS using Spark SQL query for data aggregation, querying, and writing data back into RDBMS through Sqoop.
• Worked on analyzing Hadoop clusters using different big data analytic tools including HBase database and Sqoop.
• Worked on importing and exporting data from Oracle, and DB2 into HDFS and HIVE using Sqoop for analysis, visualization, and generating reports.
• Creating and inserting data into Hive tables for dynamically inserting data into data tables using partitioning and bucketing for EDW tables and historical metrics.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations, and others during the ingestion process itself.
• Created ETL packages with different data sources (SQL Server, Oracle, Flat files, Excel, DB2, and Teradata) and loaded the data into target tables by performing different kinds of transformations using SSIS.
• Designed, developed data integration programs in a Hadoop environment with NoSQL data store Cassandra for data access and analysis.
• Created partitions, bucketing across the state in Hive to handle structured data using Elastic search.
• Performed Sqooping for various file transfers through the HBase tables for processing of data to several NoSQL DBs- Cassandra, MongoDB.

Environment: Hadoop, MapReduce, HDFS, Hive, python, Kafka, HBase, Sqoop, NoSQL, Spark 1.9, PL/SQL, Oracle, Cassandra, MongoDB

Confidential

Associate Big Data Developer

Responsibilities:

• Developed ETL pipelines using Spark and Hive for performing various business specific transformations.
• Loaded and transformed large sets of structured and semi structured data from various downstream systems.
• Building data applications and automating the pipelines in Spark for bulk loads as well as Incremental Loads of various Datasets.
• Automated the data pipeline to ETL all the Datasets along with full loads and incremental loads of data.
• Involved in developing roadmap for migration of enterprise data from multiple data sources like SQL Server, provider databases into S3 which serves as a centralized datahub across the organization.
• Wrote spark applications to perform operations like data inspection, cleaning, load and transforms the large sets of structured and semi-structured data.
• Developed Spark with Scala and Spark-SQL for testing and processing of data.
• Used Broadcast variables in Spark, effective & efficient Joins, transformations and other capabilities for data processing.
• Created Kafka producer API to send live stream json data into various Kafka topics.
• Developed Spark-Streaming applications to consume the data from Kafka topics and to insert the processed streams to HBase.

Environment: Spark, MapReduce, Hive, SQL Server, Scala, Kafka API, JSON, HBase

Confidential

Java/J2EE Developer

Responsibilities:

• Designed Use Case and Sequence Diagrams according to UML standard using Rational Rose.
• Implemented Model View Controller (MVC-2) architecture and developed Form classes, Action Classes for the entire application using Struts Framework.
• Performed client-side validations using JavaScript and server-side validations using in built Struts Validation Framework.
• Implemented the data persistence functionality of the application by using Hibernate to persist java objects to the relational database.
• Used Hibernate Annotations to reduce time at the configuration level and accessed Annotated bean from Hibernate DAO layer.
• Worked on various SOAP and RESTful web services used in various internal applications.
• Used SOAP UI tool for testing the RESTful web services.
• Used HQL statements and procedures to fetch the data from the database.
• Transformed, Navigated and Formatted XML documents using XSL, XSLT.
• Used LAMBDA EXPRESSION OF JAVA 1.8 features extensively to remove the boiler plate code and to extend the functionality.
• Used a LAMBDA EXPRESSION to improve Sack Employees further and avoid the need for a separate class.
• Used JMS for asynchronous exchange of message by applications on different platforms.
• Developed the view components using JSP, HTML, Struts Logic tags and Struts tag libraries.
• Used JUnit Testing Framework for performing Unit testing.
• Deployed application in WebSphere Application Server and developed using Rational Application RAD.

Environment: Struts 2.0 Hibernate 3.0, JSP, JDK 1.7, RAD, JMS, CVS, JavaScript, XSL, XSLT, lambda expression, Servlets 2.5, WebSphere Application Server, Oracle 10g.