SUMMARY

• Over 8+ years of experience as a Software developer with strong emphasis on building data engineering pipelines using Big Data tools.
• Hands on experience in designing and implementing data engineering pipelines and analyzing data using Hadoop ecosystem tools like HDFS, MapReduce, Yarn, Spark, Sqoop, Hive, Flume, Kafka, Impala, Oozie and HBase.
• Sound knowledge on architecture of Distributed Systems and parallel processing frameworks.
• Designed and implemented end - to-end data pipelines to extract, cleanse, process and analyze huge amounts of behavioral data and log data.
• Good working knowledge of Amazon Web Services (AWS) Cloud Platform which includes services like EC2, S3, VPC, ELB, IAM, DynamoDB, Cloud Front, Cloud Watch, Route 53, Elastic Beanstalk (EBS), Auto Scaling, Security Groups, EC2 Container Service (ECS), Code Commit, Code Pipeline, Code Build, Code Deploy, Dynamo DB, Auto Scaling, Security Groups, Red shift, CloudWatch, CloudFormation, CloudTrail, Ops Works, Kinesis, IAM, SQS, SNS, SES.
• Experienced in developing production ready spark application using Spark RDD API’s, Data frames, Spark-SQL and Spark-Streaming API's.
• Hands - on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Experience in building data pipelines using Azure Data factory, Azure Databricks and loading data to Azure data Lake, Azure SQL Database, Azure SQL Data warehouse and controlling and granting database access.
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Worked extensively on fine tuning spark applications to improve performance and troubleshooting failures in spark applications.
• Strong experience in using Spark Streaming, Spark Sql, and other components of spark like accumulators, Broadcast variables, different levels of caching and optimization techniques for spark jobs
• Experience in importing and exporting data from HDFS to RDBMS systems like Teradata (Sales Data Warehouse), SQL-Server, and Non-Relational Systems like HBase using Sqoop by efficient column mappings and maintaining the uniformity.
• Used Hive extensively to performing various data analytics required by business teams.
• Worked on Spark Streaming and Structured Spark streaming along with Kafka for real time data processing.
• Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.
• Experience working with Impala tables and accessing hive tables through for faster and Interactive access to data
• Experience in design and development of Web forms using Spring MVC, Java Script, and JSON and JQ plotter. Good level of experience in Core Java, JEE technologies, JDBC, Servlets and JSP.

TECHNICAL SKILLS

Hadoop/Big Data Technologies: Hadoop, HDFS, MapReduce, HBase, Hive, Sqoop, Impala, Oozie, Yarn, NIFI, Kafka

Programming Language: Java, Scala, Python, SQL

Databases: Oracle, Teradata, MySQL, Postgres

IDE and Dev Tools: Eclipse, Intellij, Maven, Jenkins, GitHub

NoSQL Databases: HBase, Cassandra, MongoDB, DynamoDB

PROFESSIONAL EXPERIENCE

Confidential

Data Engineer

Responsibilities:

• Worked on building centralized Data Lake on AWS Cloud utilizing primary services like S3, EMR, Redshift and Athena.
• Worked on migrating datasets and ETL workloads from On-prem to AWS Cloud services.
• Built series of Spark Applications and Hive scripts to produce various analytical datasets needed for digital marketing teams.
• Worked extensively on building and automating data ingestion pipelines and moving terabytes of data from existing data warehouses to cloud.
• Worked extensively on fine tuning spark applications and providing production support to various pipelines running in production.
• Worked closely with business teams and data science teams and ensured all the requirements are translated accurately into our data pipelines.
• Worked on full spectrum of data engineering pipelines: data ingestion, data transformations and data analysis/consumption.
• Worked on automating the infrastructure setup, launching and termination EMR clusters etc.,
• Created Hive external tables on top of datasets loaded in S3 buckets and created various hive scripts to produce series of aggregated datasets for downstream analysis.
• Build real time streaming pipeline utilizing Kafka, Spark Streaming and Redshift.
• Worked on creating Kafka producers using Kafka Java Producer Api for connecting to external Rest live stream application and producing messages to Kafka topic.
• Developed Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.
• Using Kafka and integrating with the Spark Streaming. Developed data analysis tools using SQL and Python code.

Confidential, TN

Data Engineer

Responsibilities:

• Experienced in developing Spark applications using Spark-SQL inDatabricksfor data extraction, transformation, and aggregation from multiple file formats for Analyzing& transforming the data to uncover insights into the customer usage patterns.
• Extract Transform and Load data from sources Systems to Azure Data Storage services using a combination of Azure Data factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data ingestion to one or more Azureservices (Azure Data Lake, Azure Storage, Azure SQL, Azure DW)and processing the data inAzure Databricks.
• Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark databricks cluster.
• Using Sqoop to import and export data from Oracle and PostgreSQL into HDFS to use it for the analysis.
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Created many Sparks UDF and UDAFs in Hive for functions that were not preexisting in Hive and Spark Sql.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.
• Implementing different performance optimization techniques such as using distributed cache for small datasets, partitioning, and bucketing in hive, doing map side joins etc.
• Responsible for Designing Logical and Physical data modelling for various data sources on Confidential Redshift.
• Designed and Developed ETL jobs to extract data from Salesforce replica and load it in data mart in Redshift.
• Done data validation between data present in Data Lake and S3 bucket.
• Used Spark Data Frame API over Cloudera platform to perform analytics on hive data.
• Designed batch processing jobs using Apache Spark to increase speeds by ten-fold compared to that of MR jobs.
• Used Kafka for real time data ingestion. Created different topic for reading the data in Kafka.
• Created database objects like Stored Procedures, UDFs, Triggers, Indexes and Views using TSQL in both OLTP and Relational data warehouse in support of ETL.
• Developed complex ETL Packages using SQL Server 2008 Integration Services to load data from various sources like Oracle/SQL Server/DB2 to Staging Database and then to Data Warehouse.
• Creating Hive tables, loading and analyzing data using hive scripts. Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Created report models from cubes as well as relational data warehouse to create ad-hoc reports and chart reports.

Confidential, PA

Hadoop Developer

Responsibilities:

• Experience in supporting and managing Hadoop Clusters using Hortonworks distributions.
• Collected, aggregated large amount of web log data from different sources such as web servers, mobile and network devices using Apache Kafka.
• Ingestion framework was developed in python Big Data technologies with data stores such as DynamoDB, Cassandra.
• Creating the RDD’s, Data frames for faster execution and performing data transformations and actions using Spark.
• Analyzed the SQL scripts and designed the solution to implement using PySpark.
• Experienced in writingSpark ApplicationsinScala and Python (PySpark).
• Developed optimal strategies for distributing the web log data over the cluster.
• Implemented Hive Generic UDF's to in corporate business logic into Hive Queries.
• Configuring Spark Streaming to receive real time data from the Kafka for high-speed data processing and store the stream data to HDFS.
• Configured EC2 instances and configured IAM users and roles and created S3 data pipe using Boto API to load data from internal data sources.
• Worked on Spark SQL for faster execution of Hive queries using Spark SQL Context.
• Implemented complex big data with a focus on collecting, parsing, managing, analyzing, and visualizing large sets of data to turn information into business insights using multiple platforms in the Hadoop ecosystem.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
• Involved in source system analysis, data analysis, and data modelling to ETL (Extract, Transform and Load).
• Written Spark programs to model data for extraction, transformation, and aggregation from multiple file-formats including XML, JSON, CSV& other compressed file formats.
• Imported data from the structured data source into HDFS using Sqoop incremental imports.
• Created Hive tables, partitions and implemented incremental imports to perform ad-hoc queries on structured data.
• Build Hive tables using list partitioning and hash partitioning and created Hive Generic UDF's to process business logic with HiveQL.
• Developed SQL scripts using Spark for handling different data sets and verifying the performance over Map Reduce jobs.
• Supported MapReduce Programs that are running on the cluster and Wrote MapReduce jobs using Java API.
• Designed unit test Data models and applications for data analytics solutions on streaming data

Confidential, Charlotte, NC

Big Data Developer

Responsibilities:

• Designed and built end-to-end Data Warehouse infrastructure from the ground up on Redshift for large scale data handling.
• Implemented and managed ETL solutions and automating operational processes.
• Optimized and tuned the Redshift environment, enabled queries to perform up to 100x faster for
• Tableau and SAS Visual Analytics.
• Migrated on premise database structure to Redshift data warehouse.
• Responsible for ETL and data validation using SQL Server Integration Services.
• Defined and deployed monitoring, metrics, and logging systems on AWS.
• Worked publishing interactive data visualizations dashboards, reports /workbooks on Tableau and SAS Visual Analytics.
• Responsible for designing logical and physical data modelling on Redshift.
• Designed and developed ETL jobs to extract data from salesforce replica and load it in data mart in Redshift.

Confidential

Java Developer

Responsibilities:

• Responsible for developing and modifying the existing service layer based on the business requirements.
• Involved in designing & developing web-services using SOAP and WSDL.
• Created tables, stored procedures in SQL for data manipulation and retrieval, Database Modification using SQL, PL/SQL, Stored procedures, triggers, Views in Oracle 9i.
• Involved in integration testing the Business Logic layer and Data Access layer.
• Designed user-interface and checking validations using JavaScript.
• AWS API Gateway protection strategies like Resource Policies, IAM, Lambda, Cognito Authentications.
• Involved in design of JSP's and Servlets for navigation among the modules.
• Designed cascading style sheets and XML also performed client-side validations with java script using Business Objects, XML, and JDBC.
• Developed various EJBs for handling business logic and data manipulations from database.
• Integrated JSF with JSP and used JSF Custom Tag Libraries to display the value of variables defined in configuration files.
• Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components.
• Involved in JUnit testing of the application using JUnit framework
• Used Maven build to wrap around Ant build scripts
• SVN Tortoise is used for version control of code and project documents.