SUMMARY:

• Over 8+ years of experience in Data Engineering, Data Pipeline Design, Development and Implementation as a Sr. Data Engineer/Data Developer and Data Modeler.
• Strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies.
• Strong experience in writing scripts using Python API, PySpark API and Spark API for analyzing the data.
• Extensively used Python Libraries PySpark, Pytest, Pymongo, cxOracle, PyExcel, Boto3, Psycopg, embedPy, NumPy and Beautiful Soup.
• Migrated an existing on - premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Hands On experience on Spark Core, Spark SQL, Spark Streaming and creating the Data Frames handle in SPARK wif Scala.
• Experience in NoSQL databases and worked on table row key design and to load and retrieve data for real time data processing and performance improvements based on data access patterns.
• Extensive experience in Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Map Reduce concepts.
• Experience in building large scale highly available Web Applications. Working knowledge of web services and other integration patterns.
• Developed Simple to complex Map/reduce and Streaming jobs using Java and Scala language.
• Developed Hive scripts for end user / analyst requirements to perform ad hoc analysis.
• EMR wif Hive to handle less important bulk ETL jobs.
• Hands-on use of Spark and Scala API's to compare the performance of Spark wif Hive and SQL, and Spark SQL to manipulate Data Frames in Scala.
• Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python.
• Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement.
• Hands on Spark

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Data Engineer

Responsibilities:

• Performed data analysis and developed analytic solutions. Data investigation to discover correlations / trends and the ability to explain them. Worked wif Data Engineers, Data Architects, to define back - end requirements for data products (aggregations, materialized views, tables - visualization) Developed frameworks and processes to analyze unstructured information. Assisted in Azure Power BI architecture design Experienced wif machine learning algorithm such as logistic regression, random forest, XGboost, KNN, SVM, neural network, linear regression, lasso regression and k - means Implemented Statistical model and Deep Learning Model (Logistic Regression, XGboost, Random Forest, SVM, RNN, CNN). Designing and Developing Oracle
• PL/SQL and Shell Scripts, Data Import/Export, Data Conversions and Data Cleansing. Responsible for importing data from PostgreSQL to HDFS, HIVE using SQOOP tool. Experienced in migrating HiveQL into Impala to minimize query response time. Implemented Avro and parquet data formats for apache Hive computations to handle custom business requirements. Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2. Used Sqoop to channel data from different sources of HDFS and RDBMS Developing Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.
• Using Kafka and integrating wif the Spark Streaming. Developed data analysis tools using SQL and Python code. Authoring Python (PySpark) Scripts for custom UDF's for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks. Migrate data from on-premises to AWS storage buckets. Developed various Mappings wif the collection of all Sources, Targets, and Transformations using Informatica Designer Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2. Used Sqoop to channel data from different sources of HDFS and RDBMS. Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS. Used Apache NiFi to copy data from local file system to HDP. Worked on Dimension

Confidential

Big Data Engineer

Responsibilities:

• Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management Strong understanding of AWS components such as EC2 and S3 Created yaml files for each data source and including glue table stack creation Worked on a python script to extract data from Netezza databases and transfer it to AWS S3 Developed Lambda functions and assigned IAM roles to run python scripts along wif various triggers (SQS, EventBridge, SNS) Created a Lambda Deployment function, and configured it to receive events from S3 buckets
• Writing UNIX shell scripts to automate the jobs and scheduling cron jobs for job automation using commands wif Crontab. Developed various Mappings wif the collection of all Sources,
• Targets, and Transformations using Informatica Designer Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data Installing IBM Http Server, WebSphere Plugins and WebSphere Application Server Network Deployment (ND). Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes. Provide troubleshooting and best practices methodology for development teams. This includes process automation and new application onboarding. Produce unit tests for Spark transformations and helper methods. Design data processing pipelines. Configuring IBM Http Server, WebSphere Plugins and WebSphere Application Server Network Deployment (ND) for user work - load distribution. Multiple batch jobs were written for processing hourly and daily data received through multiple sources like Adobe, No-SQL databases. Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, such as AWS, GCP
• Designed and implemented configurable data delivery pipeline for scheduled updates to customer facing data stores built wif Python Proficient in Machine Learning techniques (Decision Trees,
• Linear/Logistic Regressors) and Statistical Modeling Compiled data from various sources to perform complex analysis for actionable results Measured Efficiency of Hadoop/Hive environment ensuring SLA is met Optimized the TensorFlow Model for efficiency Analyzed the system for new enhancements/functionalities and perform Impact analysis of the application for implementing ETL changes Implemented a Continuous Delivery pipeline wif Docker, and Git Hub and AWS Built performant, scalable ETL processes to load, cleanse and validate data Collaborate wif team members and stakeholders in design and development of data environment Preparing associated documentation for specifications

Confidential

Data Engineer

Responsibilities:

• Gathered business requirements, definition and design of the data sourcing, worked wif the data warehouse architect on the development of logical data models.• Created sophisticated visualizations, calculated columns and custom expressions and developed Map Chart, Cross table, Bar chart, Tree map and complex reports which involves Property Controls,
• Custom Expressions.• Investigated market sizing, competitive analysis and positioning for product feasibility. Worked on Business forecasting, segmentation analysis and Data mining.• Automated Diagnosis of Blood Loss during Emergencies and developed Machine Learning algorithm to diagnose blood loss.• Extensively used Agile methodology as the Organization Standard to implement the data Models. Used Micro service architecture wif Spring Boot based services interacting through a combination of REST and Apache Kafka message brokers.• Created several types of data visualizations using Python and Tableau. Extracted Mega Data from AWS using SQL Queries to create reports.
• Performed reverse engineering using Erwin to redefine entities, attributes, and relationships existing database.• Analyzed functional and non-functional business requirements and translate into technical data requirements and create or update existing logical and physical data models. Developed a data pipeline using Kafka to store data into HDFS.• Performed Regression testing for Golden Test Cases from State (end to end test cases) and automated the process using python scripts.• Developed Spark jobs using Scala for faster real-time analytics and used Spark SQL for querying• Generated graphs and reports using ggplot package in RStudio for analytical models. Developed and implemented R and Shiny application which showcases machine learning for business forecasting.• Developed predictive models using Decision Tree, Random Forest, and Naïve Bayes.• Used pandas, NumPy, seaborne, SciPy, Matplotlib, Scikit-learn, NLTK in Python for developing various machine learning algorithms.
• Expertise in R, Matlab, python and respective libraries.• Research on Reinforcement Learning and control (TensorFlow, Torch), and machine learning model (Scikit-learn).• Hands on experience in implementing Naive Bayes and skilled in Random Forests, Decision Trees, Linear, and Logistic Regression, SVM, Clustering, Principal Component Analysis.• Performed K-means clustering, Regression and Decision Trees in R. Worked on data cleaning and reshaping, generated segmented subsets using NumPy and Pandas in Python.• Implemented various statistical techniques to manipulate the data like missing data imputation, principal component analysis and sampling.• Worked on R packages to interface wif Caffe Deep Learning Framework. Perform validation on machine learning output from R.• Applied different dimensionality reduction techniques like principal component analysis (PCA) and t-stochastic neighborhood embedding(t-SNE) on feat

Confidential

Data Analyst / Hadoop Developer

Responsibilities:

• Imported Legacy data from SQL Server and Teradata into Amazon S3. Created consumption views on top of metrics to reduce the running time for complex queries. Exported Data into
• Snowflake by creating Staging Tables to load Data of different files from Amazon S3. As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake. Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN,Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for the Project Worked on to retrieve the data from FS to S3 using spark commands Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS Created performance dashboards in Tableau/ Excel / Power point for the key stakeholders Incorporated predictive modeling (rule engine) to evaluate the Customer/Seller health score using python scripts, performed computations and integrated wif the Tableau viz. Worked wif stakeholders to communicate campaign results, strategy, issues or needs. Analyzed marketing campaigns from various perspectives including CTR, conversion rates, seasonal/geographical trends, search queries, landing page, conversion funnel, quality score, competitors, distribution channel, etc. to achieve maximum ROI for clients.
• Understood Business requirements to the core and Came up wif Test Strategy based on Business rules Implemented Defect Tracking process using JIRA tool by assigning bugs to Development Team Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing and performance Testing. Tested Hadoop Map Reduce developed in python, pig, Hive
• Created Metric tables, End user views in Snowflake to feed data for Tableau refresh. Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs. Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts Experienced in working wif spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file. Developed spark code and spark - SQL/streaming for faster testing and processing of data. Closely involved in scheduling Daily, Monthly jobs wif Precondition/Postcondition based on the requirement.

Environment: Snowflake,Hadoop, Map Reduce,Spark SQL, Python, Pig,AWS, GitHub, EMR, Nebula Metadata,Teradata, SQL Server, Apache Spark, Sqoop

Confidential

Java/Hadoop Developer

Responsibilities:

• Involved in review of functional and non - functional requirements. Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs in java for data cleaning and preprocessing. Installed and configured Pig and also written Pig Latin scripts. Wrote MapReduce job using Pig Latin. Involved in ETL, Data Integration and Migration. Imported data using Sqoop to load data from Oracle to HDFS on regular basis. Developing Scripts and Batch Job to schedule various Hadoop Program. Written Hive queries for data analysis to meet the business requirements.
• Creating Hive tables and working on them using Hive QL. Experienced in defining job flows. Utilized various utilities like Struts Tag Libraries, JSP, JavaScript, HTML, & CSS. Build and deployed war file in WebSphere application server. Implemented Patterns such as Singleton, Factory, Facade, Prototype, Decorator, Business Delegate and MVC. Involved in frequent meeting wif clients to gather business requirement & converting them to technicalspecification for development team. Adopted agile methodology wif pair programming technique and addressed issues during system testing. Involved in Bug fixing and Enhancement phase, used find bug tool. Version Controlled using SVN. Developed application in Eclipse IDE. Experience in developing spring Boot applications for transformations.
• Primarily involved in front-end UI using HTML5, CSS3, JavaScript, jQuery, and AJAX. Used struts framework to build MVC architecture and separate presentation from business logic.Environment: Hadoop, MapReduce, HDFS, Hive, Java, Hadoop distribution of Cloudera, Pig, HBase, Linux, XML, Java 6, Eclipse, Oracle 10g, PL/SQL, MongoDB, Toad