SUMMARY:

• Almost 8 years of technical IT experience in all phases of Software Development Life Cycle (SDLC) with skills in data analysis, design, development, testing and deployment of software systems.
• Experience in Big Data analytics, Data manipulation, using Hadoop Eco system tools Map - Reduce, HDFS Yarn/MRv2, Pig, Hive, HDFS, HBase, Spark, Kafka, Flume, Sqoop, Flume, Oozie, Avro, Sqoop, AWS, Spring Boot Spark integration with Cassandra, Avro, Solr and Zookeeper.
• Experience in developing data pipelines using AWS services including EC2, S3, Redshift, Glue, Lambda functions Step functions, Cloud Watch, SNS, Dynamo DB, SQS.
• Proficiency in multiple databases like MongoDB, Cassandra, MySQL, ORACLE, and MS SQL Server. Worked on different file formats like delimited files, avro, Json and parquet. Docker container orchestration using ECS, ALB and lambda.
• Created Snowflake Schemas by normalizing teh dimension tables as appropriate, and creating a Sub Dimension named Demographic as a subset to teh Customer Dimension.
• Hands on experience in test driven development (TDD), Behavior driven development (BDD) and acceptance test driven development (ATDD) approaches.
• Managing Database, Azure Data Platform services (Azure Data Lake(ADLS), Data Factory(ADF), Data Lake
• Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB), SQL Server, Oracle, Data
• Warehouse etc. Build multiple Data Lakes.
• Extensive experience in Text Analytics, generating data visualizations using R, Python and creating dashboards using tools like Tableau, Power BI.
• Expertise in Java programming and has a good understanding on OOPs, I/O, Collections, Exceptions Handling Lambda Expressions, Annotations
• Provided full life cycle support to logical/physical database design, schema management and deployment. Adept at database deployment phase with strict configuration management and controlled coordination with different teams.
• Utilized

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Data Engineer

Responsibilities:

• Installing, configuring and maintaining Data Pipelines Transforming business problems into Big Data solutions and define Big Data strategy and Roadmap. Designing teh business requirement collection approach based on teh project scope and SDLC methodology. Files extracted from Hadoop and dropped on daily hourly basis into S3 Authoring Python (PySpark) Scripts for custom UDF's for Row/ Column manipulations, merges, aggregations,stacking, data labelling and for all Cleaning and conforming tasks. Writing Pig Scripts to generate MapReduce jobs and performed ETL procedures on teh data in HDFS. Develop solutions to leverage ETL tools and identify opportunities for process improvements using Informaticaand Python Conduct root cause analysis and resolve production problems and data issues Performance tuning, code promotion and testing of application changes Conduct performance analysis and optimize data processes.
• Make recommendations for continuousimprovement of teh data processing environment Conduct performance analysis and optimize data processes.Make recommendations for continuous improvement of teh data processing environment. Develop a data platform from scratch and took part in requirement gathering and analysis phase of teh project in documenting teh business requirements. Design and implement multiple ETL solutions with various data sources by extensive SQL Scripting, ETLtools, Python, Shell Scripting and scheduling tools. Data profiling and data wrangling of XML, Web feedsand file handling using python, Unix and Sql. Developing Spark applications using Scala and Spark - SQL for data extraction, transformation, and aggregation from multiple file formats. Using Kafka and integrating with teh Spark Streaming.
• Developed data analysis tools usingSQL and Python code. Loading data from different sources to a data ware house to perform some data aggregations for businessIntelligence using python. Designed and implemented Sqoop for teh incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2. Used Sqoop to channel data from different sources of HDFS and RDBMS. Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregationfrom multiple file formats. Used SSIS to build automated multi-dimensional cubes.
• Used Spark Streaming to receive real time data from teh Kafka and store teh stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessaryTransformations and Aggregation on teh fly to build teh common learner data model and persists teh data in HDFS. Prepared and uploaded SSRS reports. Manages database and SSRS permissions. Used Apache NiFi to copy data from local

Confidential

Big Data Developer

Responsibilities:

• Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines Performed data extraction, transformation, loading, and integration in data warehouse, operational data storesand master data management Strong understanding of AWS components such as EC2 and S3 Responsible for data services and data movement infrastructures Experienced in ETL concepts, building ETL solutions and Data modeling Worked on architecting teh ETL transformation layers and writing spark jobs to do teh processing. Designed & build infrastructure for teh Google Cloud environment from scratch Experienced in fact dimensional modeling (Star schema, Snowflake schema), transactional modeling and SCD(Slowly changing dimension) Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, suchas AWS
• GCP Worked on confluence and Jira Designed and implemented configurable data delivery pipeline for scheduled updates to customer facing datastores built with Python Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling Compiled data from various sources to perform complex analysis for actionable results Experience in working with different join patterns and implemented both Map and Reduce Side Joins. Data visualization: Pentaho, Tableau, D3. Has noledge of Numerical optimization, Anomaly Detection and estimation, A/B testing, Statistics, and Maple. Has big data analysis technique using Big data related techniquesi.e.
• Hadoop, Map Reduce, NoSQL, Pig/Hive, Spark/Shark, MLlib and Scala, NumPy, SciPy, Pandas, scikit - learn. Utilized Spark, Scala, Hadoop, HBase, Cassandra, MongoDB, Kafka, Spark Streaming, MLlib, Python and utilizedthe engine to increase user lifetime by 45% and triple user conversations for target categories. Used Apache Spark Data frames, Spark-SQL, Spark MLlib extensively and developing and designing POC's usingScala, Spark SQL and MLlib libraries. Wrote Flume configuration files for importing streaming log data into
• HBase with Flume. Imported several transactional logs from web servers with Flume to ingest teh data into HDFS. Using Flume and Spool directory for loading teh data from local system (LFS) to HDFS. Installed and configured pig, written Pig Latin scripts to convert teh data from Text file to Avro format. Created Partitioned Hive tables and worked on them using Hive QL. Worked on continuous Integration tools Jenkins and automated jar files at end of day. Worked with Tableau and Integrated Hive, Tableau Desktop reports and published to Tableau Server. Developed MapReduce programs in Java for parsing teh raw data and populating staging Tables. Experience in setting up teh whole app stack, setup, and

Confidential

Big Data Engineer / Hadoop Spark Developer

Responsibilities:

• As a Big Data Developer, worked on Hadoop cluster scaling from 4 nodes in development environment to 8 nodesin pre - production stage and up to 24 nodes in production. Responsibilities include gathering business requirements, developing strategy for data cleansing and datamigration, writing functional and technical specifications, creating source to target mapping, designing dataprofiling and data validation jobs in Informatica, and creating ETL jobs in Informatica. Involved in various phases of development analyzed and developed teh system going through Agile Scrummethodology. Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creatingETL pipelines using Pig, and Hive. Worked on Hadoop cluster which ranged from 4-8 nodes during pre-production stage and it was sometimesextended up to 24 nodes during production. Built APIs dat will allow customer service representatives to access teh data and answer queries.
• Designed changes to transform current Hadoop jobs to HBase. Handled fixing of defects efficiently and worked with teh QA and BA team for clarifications. Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes,Troubleshooting, Manage and review data backups, Manage & review log files. Extending teh functionality of Hive with custom UDF s and UDAF's. Teh new Business Data Warehouse (BDW) improved query/report performance, reduced teh time needed to develop reports and established self-service reporting model in Cognos for business users. Implemented Bucketing and Partitioning using hive to assist teh users with data analysis. Used Oozie scripts for deployment of teh application and perforce as teh secure versioning software. Implemented Partitioning
• Dynamic Partitions, Buckets in HIVE. Develop database management systems for easy access, storage, and retrieval of data. Perform DB activities such as indexing, performance tuning, and backup and restore. Expertise in writing Hadoop Jobs for analysing data using Hive QL (Queries), Pig Latin (Data flow language), and custom MapReduce programs in Java. Did various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in teh hive and Map Side joins. Expert in creating Hive UDFs using Java to analyse teh data efficiently. Responsible for loading teh data from BDW Oracle database, Teradata into HDFS using Sqoop. Implemented AJAX, JSON, and Java script to create interactive web screens. Wrote data ingestion systems to pull data from traditional RDBMS platforms such as Oracle and Teradata and storeit in NoSQL databases such as MongoDB. Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analysed them by running Hive queries. Processed teh image data throughthe Hadoop dis

Confidential

Data Engineer / Hadoop Developer

Responsibilities:

• Experience in Big Data Analytics and design in Hadoop ecosystem using MapReduce Programming, Spark, Hive, Pig,Sqoop, HBase, Oozie, Impala, Kafka Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creatingETL pipelines using Pig, and Hive. Responsible for importing data to HDFS using Sqoop from different RDBMS servers and exporting data using Sqoopto teh RDBMS servers after aggregations for other ETL operations. Experience in designing and developing applications in PySpark using python to compare teh performance of Sparkwith Hive. Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access. Create/Modify shell scripts for scheduling various data cleansing scripts and ETL load process.
• Developed testing scripts in Python and prepare test procedures, analyze test results data and suggestimprovements of teh system and software. Used Jira for ticketing and tracking issues and Jenkins for continuous integration and continuous deployment. Build teh Oozie pipeline which performs several actions like file move process, Sqoop teh data from teh sourceTeradata or SQL and exports into teh hive staging tables and performing aggregations as per business requirementsand loading into teh main tables. Running of Apache Hadoop, CDH and Map - R distros, dubbed Elastic MapReduce(EMR) on (EC2). Performing teh forking action whenever there is a scope of parallel process for optimization of data latency. Worked on different data formats such as JSON, XML and performed machine learning algorithms in Python. Performed pig script which picks teh data from one Hdfs path and performs aggregation and loads into anotherpath which later pulls populates into another domain table. Converted dis script into a jar and passed as parameterin Oozie script Developed JSON Scripts for deploying teh Pipeline in Azure Data Factory (ADF) dat process teh data using teh SQLActivity. Build an ETL which utilizes spark jar inside which executes teh business analytical model.
• Hands on experiences on git bash commands like git pull to pull teh code from source and developing it as per therequirements, git add to add files, git commit after teh code build and git push to teh pre prod environment for thecode review and later used screwdriver. yaml which actually build teh code, generates artifacts which releases in to production Created logical data model from teh conceptual model and its conversion into teh physical database design usingErwin. Involved in transforming data from legacy tables to HDFS, and HBase tables using Sqoop. Connected to AWS Redshift through Tableau to extract live data for real time analysis. Developed Data mapping, Transformation and Cleansing rules for teh Data Management involving OLTP and OLAP. Involved in creating UNIX shell Scripting. Defragm

Confidential

Data & Reporting Analyst

Responsibilities:

• Imported Legacy data from SQL Server and Teradata into Amazon S3. Created consumption views on top of metrics to reduce teh running time for complex queries. Exported Data into
• Snowflake by creating Staging Tables to load Data of different files from Amazon S3. Compare teh data in a leaf level process from various databases when data transformation or data loading takesplace. I need to analyze and look into teh data quality when these types of loads are done (To look for any data loss, data corruption). As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading teh history datafrom Teradata SQL to snowflake. Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN, Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for teh Project Worked on to retrieve teh data from FS to S3 using spark commands Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup onAWS
• Created performance dashboards in Tableau/ Excel / Power point for teh key stakeholders Incorporated predictive modelling (rule engine) to evaluate teh Customer/Seller health score using python scripts,performed computations and integrated with teh Tableau viz. Designed, developed and maintained Big Data streaming and batch applications using Storm. Worked with stakeholders to communicate campaign results, strategy, issues or needs. Analysed marketing campaigns from various perspectives including CTR, conversion rates, seasonal/geographicaltrends, search queries, landing page, conversion funnel, quality score, competitors, distribution channel, etc. to achieve maximum ROI for clients. Worked with business to identify teh gaps in mobile tracking and come up with teh solution to solve. Analysed click events of Hybrid landing page which includes bounce rate, conversion rate, Jump back rate,List/Gallery view, etc. and provide valuable information for landing page optimization.
• Evaluated teh traffic and performance of Daily deals PLA ads and compare those items with non - daily deal itemsto see teh possibility of increasing ROI. suggested improvements and modify existing BI components (Reports, Stored Procedures) Understood Business requirements to teh core and Came up with Test Strategy based on Business rules Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Scala and has a goodexperience in using Spark-Shell and Spark Streaming. Prepared Test Plan to ensure QA and Development phases are in parallel Written and executed Test Cases and reviewed with Business & Development Teams. Implemented Defect Tracking process using JIRA tool by assigning bugs to Development Team Automated Regression tool (Qute) and reduced manual effort