SUMMARY

• Around 7+ years of profession experience in Software Systems Development, Business Systems, experience in Big Data ecosystem related technologies with a master's degree in Information systems.
• Experience in data management and implementation of Big Data applications using Spark and Hadoop frameworks.
• Worked in analyzing data using Spark SQL, Hive QL and PIG Latin.
• Hands on experience building streaming applications using Spark Streaming and Kafka with minimal/no data loss and duplicates.
• Excellent technical and analytical skills with clear understanding of design goals and development for OLTP and dimension modeling for OLAP.
• Strong experience and knowledge of HDFS, Map Reduce and Hadoop ecosystem components like Hive, Pig, Sqoop, NoSQL databases such as Mongo DB and Cassandra.
• Familiarity with Amazon Web Services along with provisioning and maintaining AWS resources such as EMR, S3 buckets, EC2 instances, RDS and others.
• Extensive work in Text Analytics, developing different Statistical Machine Learning, Data Mining solutions to various business problems and generating data visualizations using R, Python and Tableau.
• Hands on experience in implementing LDA, Naïve Bayes and skilled in Random Forests, Decision Trees, Linear and Logistic Regression, SVM, Clustering, neural networks, Principle Component Analysis and good knowledge on Recommender Systems.
• Expertise in transforming business requirements into analytical models, designing algorithms, building models, developing datamining and reporting solutions that scales across massive volume of structured and unstructured data.
• Performed statistical & graphical analytics using NUMPY, PANDAS, MATPLOTLIB and BI tools such as Tableau.
• Experience in using visualization tools like Tableau, ggplot2 and d3.js for creating dashboards.
• Statistical Modelling with ML to bring Insights in Data under guidance of Principal Data Scientist.

TECHNICAL SKILLS

Big Data Tools: HDFS, MapReduce, Hive, Pig, Hadoop Streaming

Languages: Pyhon, Scala, R

Tools: & Utilities: PyCharm, DataBricks, SQL server management studio

No SQL Databases: Cassandra & MongoDb

Machine learning: Decision trees, Random forest, Linear & Logistic regression, PCA, K - means, XG Boost and predictive analytics-based algorithms

Data -Streaming: Batch Processing & Real-time streaming using KAFKA

AWS platform: Familiarity with cluster deployment using EC2 and EMR apart from using Storage platforms such as S3 along with a basic understanding of AWS Redshift.

O. S: Linux, Windows, Shell Scripting.

PROFESSIONAL EXPERIENCE

Confidential, Philadelphia, PA

Big Data Engineer

Responsibilities:

• Developed various data loading strategies and performed various transformations for analyzing the datasets by using Hortonworks Distribution for Hadoop ecosystem.
• Worked in Loading and transforming large sets of structured, semi structured and unstructured data.
• Involved in collecting, aggregating and moving data from servers to HDFS using Flume.
• Collecting data from various Flume agents that are imported on various servers using Multi-hop Flow.
• Knowledge on various flume sources, channels and sink by which data is ingested into HDFS
• Responsible for performing various transformations like sort, join, aggregations, filter in-order to retrieve various datasets using apache spark.
• Experience in extracting appropriate features from datasets in-order to handle bad, null, partial records using spark SQL.
• Worked on storing the dataframe into hive as table using Python (PySpark).
• Experienced in ingesting data into HDFS from various Relational databases like Teradata using sqoop and exported data back to Teradata for data storage.
• Hands on experience in developing apache SPARK applications using Spark tools like RDD transformations, Spark core, Spark MLlib, Spark Streaming and Spark SQL.
• Experience in developing various spark application using Spark-shell (Scala).
• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.
• Designed and implemented Incremental Imports into Hive tables and writing Hive queries to run on TEZ.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Migrated ETL jobs to Pig scripts do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Involved in writing optimized Pig Script along with developing and testing Pig Latin Scripts
• Implemented the workflows using Apache Oozie framework to automate tasks
• Worked on different file formats like Sequence files, XML files and Map files using MapReduce Programs. Worked with Avro Data Serialization system to work with JSON data formats.
• Exported data to Cassandra (NoSQL) database from HDFS using sqoop and performed various CQL commands on Cassandra to obtain various datasets as required.
• After performing all the transformations data is stored in MongoDB (NOSQL)using Sqoop.
• Created and imported various collections, documents into MongoDB and performed various actions like query, project, aggregation, sort, limit.
• Involved in Unit testing and delivered Unit test plans and results documents using Junit and MRUnit.
• Created Hive UDFs and UDAFs using python scripts & Java code based on the given requirement
• Automated all the jobs to pull the data and load into Hive tables, using Oozie workflows
• Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
• Knowledge on microservices architecture in spring Boot integrating with various restful webservices.
• Created and maintained Technical documentation for launching Hadoop Clusters and for executing Pig Scripts.

Environment: Hadoop, HDFS, Map Reduce, spark, Sqoop, Oozie, Pig, Kerberos, Hive, Flume, TEZ, LINUX, Java, Eclipse, Cassandra, python, MongoDB.

Confidential, Valley Forge, PA

Big Data Engineer

Responsibilities:

• Simulated Credit risk scenarios and used logistic regression along with decision tree-based ML algorithms to predict aforesaid output.
• Transformed Kafka loaded data using Spark-streaming with Scala and Python.
• Used Sci-kit learn, Pandas, Numpy and Tensor flow to determine insights from data and created a trained credit fraud detection model for batch data.
• Created the framework for the dashboard using Tableau and optimized the same using open source Google optimization tools.
• Trained a Fraud Prediction model and created a customizable dashboard for credit monitoring agencies.
• Developed SQOOP scripts to migrate data from Oracle to Big data Environment.
• Extensively worked with Avro and Parquet files and converted the data from either format Parsed Semi Structured JSON data and converted to Parquet using Data Frames in Spark.
• Developed a Python Script to load the CSV files into the S3 buckets and created AWS S3 buckets, performed folder management in each bucket, managed logs and objects within each bucket.
• Created Airflow Scheduling scripts in Python to automate the process of Sqooping wide range of data sets.
• Involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS
• Converted all Hadoop jobs to run in EMR by configuring the cluster according to the data size
• Collated Real-time streaming data from credit agencies such as Transunion & Experian, performed data cleaning and fed the data into Kafka.
• Deployed model using RESTful APIs and used Dockers to facilitate multi-environment transition.
• Streaming data was stored using Amazon S3 deployed over EC2 and EMR cluster framework apart from in-house tools.

Environment: Podium Data, Data Lake, HDFS, Hue, AWS S3, Impala, Spark, Scala, Kafka, Looker, AWS EC2 and EMR

Confidential, New York, NY

Big Data Engineer

Responsibilities:

• Cleaned and congruous data was then streamed using Kafka into Spark and manipulations were performed on real time data with Python and Scala.
• Built the Machine learning based coupon purchase recommendation engine by training the model on historical purchase data of customers across the retail hemisphere.
• Simulated real-time scenarios using the Sci-kit learn and Tensor flow libraries on Batch data for training model with the resulting model being used in real-time models.
• Wrote Spark applications for Data validation, cleansing, transformations and custom aggregations.
• Imported data from various sources into Spark RDD for processing.
• Developed custom aggregate functions using Spark SQL and performed interactive querying.
• Worked on installing cluster, commissioning & decommissioning of Data node, Name node high availability, capacity planning, and slots configuration.
• Developed Spark applications for the entire batch processing by using Scala.
• Automatically scale-up the EMR instances based on the data.
• Stored the time-series transformed data from the Spark engine built on top of a Hive platform to Amazon S3 and Redshift.
• Facilitated deployment of multi-clustered environment using AWS EC2 and EMR apart from deploying Dockers for cross-functional deployment.
• Visualized the results using Tableau dashboards and the Python Seaborn libraries were used for Data interpretation in deployment.

Environment: Spark, AWS, EC2, EMR, Hive, MS SQL Server, Genie Logs, Kafka, Sqoop, Spark SQL, Spark Streaming, Scala, Python, Tableau

Confidential

Big Data engineer

Responsibilities:

• Responsible for penetration testing of corporate networks and simulated virus infections on computers to assess network security & Presented a report to the Development Team to assess the intrusions.
• Used Spark-SQL to load JSON data and create schema RDD and loaded it into the Hive tables and handled structured data using Spark SQL.
• Loaded the data into Spark RDD and did the memory data computation to generate the Output response.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Worked on the Pub-Sub system using Apache Kafka during the ingestion process.
• Worked on automating the flow of data between software systems using Apache NiFi.
• Prepared workflows for scheduling the load of data into Hive using IBIS Connections.
• Worked on a robust automated framework in Data Lake for metadata management that integrates various metadata sources, consolidates and updates podium with latest and high-quality metadata using the big data technologies like Hive and Impala.
• Coordinated with product leads to identify problems with Norton products and acted as a liaison between the Development Team and Quality Team to ascertain the efficiency of the product
• Handled network intrusion data and manipulated spark jobs on the same data to identify most common threats and analyze the aforesaid issues.

Environment: Hadoop (Cloudera Stack), Hue, Spark, Kafka, HBase, HDFS, Hive, Pig, Sqoop

Confidential

Data Engineer

Responsibilities:

• Worked as Data Engineer with Dell Technical Support for over 10000 US customers
• Reported common issues faced with Dell products and fostered feasible solutions making least changes in physical design of the product and rendering maximum throughput by alleviating the defects
• Created a data-profiling dashboard by leveraging podium internal architecture, which drastically reduced the time to analyze data quality using Looker reporting.
• Worked on an input agnostic framework for data stewards to handle their ever-emerging work group datasets and created a business glossary by consolidating them using Hive.
• Created technical design documentation for the data models, data flow control process and metadata management.
• Reversed Engineered and generated the data models by connecting to their respective databases.
• Worked on a robust comparison process to compare data modelers' metadata with data stewards' metadata and identify anomalies using Hive and Podium Data.

Environment: Hadoop (Hortonworks stack), HDFS, Oozie, Pig, Hive, MapReduce, Sqoop, Linux