SUMMARY

• Around 10+ Years of Overall Experience in Software development lifecycle and 7+ years of Experience in Data Pipeline Development, Data Transformation, Data Modeling, & Data cleansing, and Spark Streaming API’s.
• Expertise in creating Dashboards and alerts using Splunk Enterprise, Tableau, ELK & monitoring using DAG’s. Good Understanding of Programming languages like C#, Java, Scala, and Python.
• Experience in Cloud environments like Azure, AWS & GCP.
• Multiple years of hands - on experience in Hadoop Eco system technologies such as in Pig, Hive, HBase, Zookeeper, Map Reduce, Oozie, Flume and Sqoop.
• Organizing data into tables, performing spark transformations, and simplifying complex queries with Hive.
• Performing real-time interactive analysis on massive data sets stored in HDFS. In depth understanding/knowledge of Hadoop Architecture and various components such as MapReduce, Hadoop GEN2 Federation, High Availability and YARN architecture and good understanding of workload management, schedulers, scalability, and distributed platform architectures
• Hands on experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa. Good knowledge in using job scheduling tools like Oozie.
• Good experience in designing, developing database to create its Objects like Tables, Stored Procedures, Triggers and Cursors using PL/SQL
• Responsible for analyzing big data and provide technical expertise and recommendations to improve current existing systems.
• Overall responsibility of all technical business requirements gathering deliverables and implementation.
• Hands on experience in Capacity planning, Installation, monitoring and Performance Tuning of Hadoop Clusters.
• Proficient Knowledge on Apache Spark and Apache Storm to process real time data.
• Hands on experience in analyzing data using Hive QL, Pig Latin, and custom MapReduce programs in Java.
• Build Data pipelines, maintaining existing pipelines, migrating on-prem to the cloud, ad-hoc projects, and Production support.
• Developed Kafka API to stream the data from external source to ingest into spark for data transformations.
• Developed pipeline to process data from Hive to snowflake as well as to publish Kafka topic using spark in Scala and deploy it on Kubernetes Cluster.
• Developed pipelines to migrate the data from on-prem to cloud using stream-sets. Migrated legacy Java pipelines to spark applications in Scala and will validated on Databricks.
• Implemented custom Sinks and Interceptors for Flume. Support a wide variety of ad hoc data needs.
• Involved in building, evolving, and reporting framework on top of the Hadoop cluster to facilitate data mining, predictive Modeling, analytics, and dash-boarding.
• Have a strong ability to prepare and present data in a visually appealing and easy to understand manner. Build high-volume real-time data processing applications using Hadoop platform.
• Performed Importing and exporting data into HDFS, Hive and HBase using Sqoop.
• Hands-on experience on full life cycle implementation using HDP (Hartonworks Data Platform) Platform.
• Hands on Experience in designing and developing scripts for customer ETL processes and Automation in Azure data factory, Azure databricks, Python, PySpark etc.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Provide assistance in developing strategies for integrating data from various SaaS Applications.
• Proficient in designing efficient and robust ETL workflows
• Extensively worked on Informatica PowerCenter 9.1/8.6 Designer, Repository Manager, Workflow Manager, Workflow Monitor including Administration performance tuning of transformations/mappings/ sessions/workflows
• Strong background in Data warehousing concepts and implementing dimensional Modeling concepts.
• Configured and deployed various AWS services including EC2, VPC, S3, IAM, ELB, AWS Lambda, DynamoDB, CloudWatch, CloudFormation, AWS Auto Scaling, Route53.
• Launching Amazon EC2 Cloud Instances using Amazon Images (Linux/ Ubuntu) and configuring launched instances with respect to specific applications
• Proficient in interacting with the business users by conducting meetings with the clients during requirements Analysis phase
• Hands on Experience in all the phases of Data warehouse life cycle involving Requirement Analysis, Design, Coding, Testing, and Deployment
• Involved in Performance Tuning and Productivity Improvement activities
• Experienced with Google Cloud Platform (GCP) services like Compute engine, cloud load balancing, Cloud Storage, Cloud SQL, Stack driver monitoring, and cloud Deployment manager.
• Good knowledge with Unit Testing, Regression Testing, Integration Testing, User Acceptance Testing, Production implementation and Maintenance.
• Demonstrated ability to communicate and gather requirements, partner with Enterprise Architects, Business Users, Analysts, and development teams to deliver rapid iterations of complex solutions. cloud storage, cloud SQL, stack driver monitoring, and cloud deployment manager.
• Implemented PYSPARK scripts to classified data Organizations into different classified based on different types of records. Assisted monitoring Spark cluster.
• Created PySpark frame to bring to bring data from DB2 to Amazon S3 to run on Kubernetes
• Optimize the Pyspark jobs to run on Kubernetes cluster for faster data Processing.
• Worked on reading and writing multiple data formats like JSON, ORC, Parquet on HDFS using Pyspark.
• Responsible for enhancing data pipelines to analyze the device logs and dispatch information which includes more aggregations and transformations using AWS lambda function with spark streaming and SQL.
• Developed spark job to consume topics from Kafka and process it before pushing them out to AWS S3 buckets.
• Worked on enhancing dispatch applications to process CAD Communication messages from BOS using Java API.
• Developed hive ingest application based on H-catalog streaming and involved in spark streaming API development to process near real-time logs from plane to update status of a plane to costumers.
• Experienced in writing terra templates that can spin up infra for infrastructure for multi-tier applications and created and provisioned boot strapped software demo on cloud with terraform and deployed infrastructure in multi cloud environment to deploy a fault tolerant application. Terraform works just like cloud formation but it is available for all cloud platforms and not designed to a specific cloud platform like cloud formation to AWS.
• Expertise in Python, Perl, Bash, Ruby scripting languages for deploying applications on servers. Good knowledge of maintaining application servers like Apache, Tomcat, JBoss
• Wrote Ansible playbooks with Python SSH as a wrapper to manage configurations and the test playbooks using Python. Handled the Ansible Tower errors that occurred while deploying, by updating the Playbook with missing.
• Involved in converting Hive/SQL queries into Spark (RDDs, Data frame and Dataset) using Python and Scala.

TECHNICAL SKILLS

Big Data Technologies: HDFS, Hive, MapReduce, Pig, Sqoop, Flume, Oozie, Hadoop distribution, and HBase, Spark, Yarn, Zookeeper, Kafka, Ranger, Grafana, Atlas.

Cloud: Azure, Aws & GCP

Programming languages: Java, C#, Scala, Python, PySpark.

Databases: MySQL, SQL/PL-SQL, MS-SQL Server 20012/16, Oracle 10g/11g/12c

NoSQL Databases: HBASE, MongoDB, Snowflake.

DevOPs: GitHub, Jenkins, Ansible, Docker, Nagios & Kubernetes.

Software Life Cycle: SDLC, Waterfall and Agile models.

Data Visualization Tools: Tableau, BI, SSRS, Splunk.

PROFESSIONAL EXPERIENCE

Confidential, Chicago, IL

Data Engineer

Responsibilities:

• Implemented Hadoop framework to capture user navigation across the application to validate the user interface and provide analytic feedback/result to the UI team.
• Built Data pipelines, maintaining existing pipelines, migrating on-prem to the cloud, ad-hoc projects, and Production support.
• Developed Kafka API to stream the data from external source to ingest into spark for data transformations.
• Developed pipeline to process data from Hive to snowflake as well as to publish Kafka topic using spark in Scala and deploy it on Kubernetes Cluster.
• Worked on Hadoop Cluster with current size of 56 Nodes and 896 Terabytes capacity. Wrote Map Reduce Jobs, HIVEQL, Pig, Spark.
• Migrated an existing on-premises application to AWS and used AWS services like EC2 and S3 for small data sets processing and storage, experienced in maintaining the Hadoop cluster on AWS EMR.
• Imported data from AWS S3 into Spark RDD, performed transformations and actions on RDD's.
• Performing metadata conversion from Oracle to Redshift using Schema Conversion Tool - AWS SCT
• Used AWS SCT sort keys and distribution keysRedshift database optimization
• Replicate database objects in AWSRedshift via Change Data Capture - CDC
• Worked on the large-scale Hadoop Yarn cluster for distributed data processing and analysis using Spark, Hive, and HBase.
• Involved in creating data-lake by extracting customer's data from various data sources to HDFS, which include data from Excel, databases, and log data from servers.
• Used Apache Solr to index the documents and used free-form queries to search the indexed documents.
• Developed Spark applications by using Scala and Python and implemented Apache Spark for data processing from various streaming sources.
• Developed Spark applications using Scala & Python to do the analytics on the data stored on HDFS.
• Worked as a Spark Expert and performance Optimizer.
• Written MapReduce (Hadoop) programs to convert text files into AVRO and loading into Hive (Hadoop) table.
• Worked with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames, and pair RDD's.
• Developed preprocessing job using Spark Data frames to transform Json documents to flat file.
• Loaded D-Stream data into Spark RDD and did in-memory data computation to generate output response.
• Involved in loading data from rest endpoints to Kafka producers and transferring the data to Kafka brokers.
• Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds.
• Worked with Grid Gain in-memory file system to share state of spark job and applications when working with files instead of RDD's.
• Import data using Sqoop into Hive and Hbase from existing SQL Server. Support code/design analysis, strategy development and project planning.
• Create reports for the BI team using Sqoop to export data into HDFS and Hive.
• Develop multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Involve in Requirement Analysis, Design, and Development.
• Export and Import data into HDFS, HBase and Hive using Sqoop.

Confidential, Burlington, NJ

Data Engineer

Responsibilities:

• Implemented Hadoop framework to capture user navigation across the application to validate the user interface and provide analytic feedback/result to the UI team.
• Build Data pipelines, maintaining existing pipelines, migrating on-prem to the cloud, ad-hoc projects, and Production support.
• Developed Kafka API to stream the data from external source to ingest into spark for data transformations.
• Developed pipeline to process data from Hive to snowflake as well as to publish Kafka topic using spark in Scala and deploy it on Kubernetes Cluster.
• Developed pipelines to migrate the data from on-prem to cloud using stream-sets. Migrated legacy Java pipelines to spark applications in Scala and will validated on Databricks.
• Good SQL/PL SQL programming skills - Writing Complex procedures and queries for Data frames and Notebooks.
• Performed analysis on the unused user navigation data by loading into HDFS and writing MapReduce jobs.
• Good Experience on Data Lake, Data Warehouse and Integration Strategy, BI technologies Data Governance and service lifecycle Management.
• Wrote MapReduce jobs using Java API and Pig Latin.
• Loaded the data from Teradata to HDFS using Teradata Hadoop connectors.
• Used Flume to collect, aggregate and store the web log data onto HDFS.
• Wrote Pig scripts to run ETL jobs on the data in HDFS.
• Used Hive to do analysis on the data and identify different correlations.
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop.
• Written Hive queries for data analysis to meet the business requirements.
• Automated all the jobs, for pulling data from FTP server to load data into Hive tables, using Oozie workflows.
• Hands on Experience in designing and developing scripts for customer ETL processes and Automation in Azure data factory, Azure databricks, Python, PySpark etc.
• Involved in creating Hive tables and working on them using Hive QL.
• Supported Map Reduce Programs those are running on the cluster.
• Maintaining and monitoring clusters. Loaded data into the cluster from dynamically generated files using
• Developed Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Implemented PySpark scripts to classified data organizations into different classified based on different types of records. Assisted monitoring Spark Cluster.
• Involved in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Designed and Developed Real time Stream processing Application using Spark, Kafka, Scala, and Hive to perform Streaming ETL and apply Machine Learning.
• Created Partitioned and Bucketed Hive tables in Parquet File Formats with Snappy compression and then loaded data into Parquet hive tables from Avro hive tables.
• Involved in running all the hive scripts through hive, Impala, Hive on Spark, and some through Spark SQL.
• Involved in converting Hive/SQL queries into Spark (RDDs, Data frame and Dataset) using Python and Scala.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• As a lead I am mostly involved in design meetings to refine the stories and also creating tasks for the rest of the team and assigning the tasks as per the availability.
• Developed an algorithm model to process quick books data from customers using Python API’s which reads data from Kafka topics and ingest oracle DB.
• And I was leading a team of Developers in our team making sure the tasks are assigned and completed in time.
• Developed Java Application to Parse XML tags from Black box files to filter necessary fields to process
• Developed Python Scripts for data analytics and aggregations. Developed QuickBooks related streaming API using storm.
• Along with it I also be overseeing the code by code reviewing the change in the report form, in our pull request process.
• Migrated the EDL data to GCP, using as much as possible the existing work in HiveQL’s (data integration) to the Google Dataproc cluster for historical and recurring/delta loads. Using as much as the lift and shift architecture pattern.
• Hands on Experience in designing and developing scripts for customer ETL processes and Automation in Azure data factory, Azure data bricks, Python, PySpark etc.
• Implemented large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML and Power BI.
• Designed end to end scalable architecture to solve business problems using various Azure Components like HDInsight, Data Factory, Data Lake, Storage and Machine Learning Studio. Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the SQL Activity.
• Used Airflow for ETL pipelines to automatically Organize, Execute & monitor Data Flow.
• Configured & deployed AWS services including EC2, VPC, S3, IAM, ELB, AWS Lambda, DynamoDB, CloudWatch, CloudFormation, AWS Auto Scaling, Route53

Confidential - Plano, TX

Hadoop developer

Responsibilities:

• Launching and Setup of HADOOP/ HBASE Cluster which includes configuring different components of HADOOP and HBASE Cluster.
• Hands on experience in loading data from UNIX file system to HDFS.
• Responsible for enhancing data pipelines to analyze the device logs and dispatch information which includes more aggregations and transformations using AWS lambda function with spark streaming and SQL.
• Developed spark job to consume topics from Kafka and process it before pushing them out to AWS S3 buckets.
• Worked on enhancing dispatch applications to process CAD Communication messages from BOS using Java API.
• Experienced on loading and transforming of large sets of structured, semi structured and unstructured data from HBase through Sqoop and placed in HDFS for further processing.
• Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
• Managing and scheduling Jobs on a Hadoop cluster using Oozie.
• Involved in creating Hive tables, loading data, and running hive queries in those data.
• Extensive Working knowledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
• Launching Amazon EC2 Cloud Instances using Amazon Images (Linux/ Ubuntu) and configuring launched instances with respect to specific applications.
• Involved in writing optimized Pig Script along with involved in developing and testing Pig Latin Scripts.
• Good knowledge in writing Pig's Load and Store functions

Confidential

ETL Developer

Responsibilities:

• Analyzing the existing ETL Sas process
• Worked closely with data modelers on Erwin
• Preparing functional, technical document.
• Created new streams for handling the new feeds
• Developed hive ingest application based on H-catalog streaming and involved in spark streaming API development to process near real-time logs from plane to update status of a plane to costumers.
• Design and develop Abinitio Graph for processing Account/Statement /Transactional data.
• Worked on performance improvement of Abinitio graphs.
• Worked on Enhancement of graphs to in corporate new requirement and client additions.
• Metadata changes are made to handle the new columns and new feeds.
• Preparing various test cases for programs.
• End-to-End testing of the process for verifying the impacts caused due to the enhancements.

Confidential

Tech Lead

Responsibilities:

• Providing ITIL Change mgmt. / technical coordination on multiple tasks.
• Reviewing the Change’s on a Daily basis and coordinating with the teams accordingly.
• Responsible for Health checks done on all servers which are scheduled for Patching.
• Responsible for the communication between onsite and offshore teams
• Logging PR tickets during any issues and follow with the respective team.
• Ensure smooth allocation and coordination of all activities.
• Participate in the review and reporting of Major issues during patching.
• Responsible to create and send alerts to management for the servers which are not patched.
• Generating Reports on Daily, Weekly, Fortnightly and Monthly basis.