SUMMARY

• More than 14 years of experience in IT industry, including big data environment, data engineering, Hadoop ecosystem and Design, Developing, and Maintenance of various applications. Experience on Migrating SQL database to Azure Data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and migrating on premise and real - time databases to Azure Data Lake store using Azure Data factory.
• Strong experience of leading multiple Azure Big Data and Data transformation implementations in Banking and Financial Services, and Healthcare industries. Hands on experience in Hadoop ecosystem including Spark, Kafka, HBase, Scala, Pig, Impala, Sqoop, Oozie, Flume, Storm, big data technologies and experienced in developing custom UDFs for Pig and Hive to incorporate methods and functionality of Python/Scala, Java into Pig Latin and HQL (HiveQL).
• Experience in implementing large Lambda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML and Power BI. Hands on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Designed and Developed ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into Snowflake and involve with Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark.
• Built ETL pipeline end to end from AWS S3 to Key, Value store Dynamo DB, and Snowflake Data warehouse for analytical queries and specifically for cloud data. Design and construct of AWS Data pipelines using various resources in AWS including AWS API Gateway to receives response from AWS lambda and retrieve data from snowflake using lambda function and convert the response into Json format using Database as Snowflake, Dynamo DB, AWS Lambda function and AWS S3.
• Experience in Developing Spark applications using Spark - SQL in Data bricks for data extraction, trans-formation and aggregation from multiple file formats for analyzing & transforming the data to uncover in-sights into the customer usage patterns. Experience in build scripts using Maven and do continuous integrations (CI/CD) systems like Docker,
• Kubernetes and Jenkins. Good experience in Tableau for Data Visualization and analysis on large data sets, drawing various conclusions and leveraged and integrated Google Cloud Storage and Big Query applications, which connected to Tableau for end user web-based dashboards and reports. Expertise in synthesizing Machine learning, Predictive Analytics and Big data technologies into integrated solutions and worked on Spark SQL, Spark Streaming and using Core Spark API to explore Spark features to build data pipelines.
• Experience in Database Design and development with Business Intelligence using SQL Server 2014/2016, Integration Services (SSIS), DTS Packages, SQL Server Analysis Services (SSAS), DAX, OLAP Cubes, Star Schema and Snowflake Schema. Exploring on Azure Cognitive Services (LUIS) and Machine Learning and IOT technologies. Experienced in development of Big Data projects using Hadoop, Hive, HDP, Pig, Flume, Storm and Map Reduce open-source tools and experience in installation, configuration, supporting and managing Hadoop clusters
• Experience in Data Modeling &Data Analysis experience using Dimensional Data Modeling and Relational Data Modeling, Star Schema/Snowflake Modeling, FACT & Dimensions tables, Physical & Logical Data Modeling. Excellent Knowledge in understanding Big Data infrastructure, distributed file systems -HDFS, parallel processing - Map Reduce framework. Experience in installation, configuration, supporting and managing - Cloudera Hadoop platform along with CDH4 & CDH5 clusters and Hortonwors.
• Experience in database design, entity relationships, database analysis, programming SQL, stored procedures PL/ SQL, packages and triggers in Oracle and experience in using PL/SQL to write Stored Procedures, Functions and Triggers.

TECHNICAL SKILLS

Big Data Ecosystem: Map Reduce, HDFS, HIVE, Pig, Sqoop, Flume, HDP, Oozie, Zookeeper, Spark, Kafka, storm, Hue Hadoop Distributions Cloudera (CDH3, CDH4, CDH5), Hortonworks and Flume

SQL and NoSQL Databases: Oracle 12c/11g, MySQL, MS-SQL, Teradata, HBase, MongoDB, Cassandra.

Version Control: GIT, GitLab, SVN

Data Warehousing & Reporting Tools: Informatica, Erwin, ER Studio, Data bricks, Snowflake, SSIS, SSRS, Power BI and Tableau.

Cloud Technologies: Azure Data Lake, AWS, Azure Data Factory, Azure Data bricks, Synapse, Azure SQL, Azure DW and Storage Blob, AWS S3, Redshift, Glue, Kinesis, KMS, Lambda, Athena and RDS

Web Technologies: HTML, XML, JavaScript, AJAX, SOAP, WSDL

Programming Languages: Java, Python, Pyspark, SQL, PL/SQL, HiveQL, UNIX Shell Scripting, Scala, Linux shell scripts

Methodologies: Software Development Lifecycle (SDLC), Waterfall Model and Agile, STLC (Software Testing Life cycle) & UML, Design Patterns (Core Java and J2EE)

Databases: Oracle, SQL Server, Cassandra, MongoDB, PostgreSQL, Teradata and DB2.

Operating Systems: Windows, UNIX/Linux and Mac OS.

CI/D: Docker, Jenkins and Kubernetes.

PROFESSIONAL EXPERIENCE

Confidential

Lead Data Engineer/Analyst/Architect

Responsibilities:

• Analyze, design, and build Modern data solutions using Azure PaaS service to support visualization of data. Understand current Production state of application and determine the impact of new implementation on existing business processes.
• Implemented solutions for ingesting data from various sources and processing the Data-at-Rest utilizing Big Data technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive.
• Involved in Extracting Transforming and Loading data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in in Azure Data bricks.
• Managed and lead the development effort with the help of a diverse internal and overseas group and design/ architected and implemented complex projects dealing with the considerabledatasize (TB/ PB) and with high complexity.
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
• Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine learning use cases under Spark ML and Mllib and Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL,DataFrame, Pair RDD's, Spark YARN.
• Performed data profiling and transformation on the raw data using Pig, Python, and Java and developed predictive analytic using Apache Spark Scala APIs.
• Developed Spark applications usingPysparkandSpark-SQLfor data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the SQL Activity.
• Analyzed data using HiveQL to generate payer by reports for transmission to payer's form payment summaries and involved in working of big data analysis using Pig and User defined functions (UDF) and created Hive External tables and loaded the data into tables and query data using HQL.
• Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in CSV format and used Data Frame API in Scala for converting the distributed collection of data organized into named columns.
• Worked on catapulting data from Teradata to snowflake to consume on Data bricks and worked on Teradata SQL queries, Teradata Indexes, Utilities such as Mload, Tpump, Fast load and Fast Export.
• Designed and Developed Real Time Stream Processing Application using Spark, Kafka, Scala, and Hive to perform Streaming ETL and apply Machine Learning and design and implement streaming solutions using Kafka or Azure Stream Analytics.
• Creating Spark clusters and configuring high concurrency clusters using Azure Data bricks to speed up the preparation of high-quality data and worked on Azure Streaming Analytics with Event Hubs and sending output to PowerBi Dashboard.
• Exploring DAG's, their dependencies and logs usingAirflowpipelines for automationand useApacheAirflowto schedule and run theairflowdags to execute code and involved in schedulingAirflowworkflow engine to run multiple Hive and pig jobs using python.
• Enhancements to traditional data warehouse based on STAR schema, update data models, perform Data Analytics and Reporting using Tableau and involved in migration of data from existing RDBMS (oracle and SQL server) to Hadoop using Sqoop for processing data.
• Developed Shell, Perl, and Python scripts to automate and provide Control flow to Pig scripts and Developed HiveQL scripts for performing transformation logic and also loading the data from staging zone to landing zone and Semantic zone.
• Maintain and work with ourdatapipeline that transfers and processes several terabytes ofdatausing Spark, Scala, Python, Apache Kafka, Pig/ Hive & Impalaand Developed prototype for Big Data analysis using Spark, RDD, Data Frames, and Hadoop ecosystem with .csv, JSON, parquet and HDFS files.
• Involved in creating Oozie workflow and Coordinator jobs for Hive jobs to kick off the jobs on time for data availability and worked on Oozie scheduler to automate the pipeline workflow and orchestrate the Sqoop, hive and pig jobs that extract the data on a timely manner.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Virtualized the servers using Docker for the test environments and dev-environments needs and Utilized Kubernetes and Docker for the runtime environment for the CI/CD system to build, test, and deploy

Confidential

Big Data Engineer/Developer/Analyst

Responsibilities:

• Evaluate deep learning algorithms for text summarization using Python, Keras and TensorFlow on Cloudera Hadoop System and Use Spark API for Machine learning and translate a predictive model from SAS code to Spark and used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Recreating existing application logic and functionality in the Azure Data Lake, Data Factory, SQL Database and SQL data warehouse environment
• Exploring with the Spark for improving the performance and optimization of the existing algorithms inHadoopusing Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Utilized the Azure polybase utility to run t-sql queries on external data in Hadoop and as well as to import and export data from Azure blob storage.
• Developed Sqoop scripts for the extractions of data from various RDBMS databases into HDFS and developed scripts to automate the workflow of various processes using python and shell scripting and collected and aggregate large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
• Developed data pipeline using Pig and Hive from Teradata, DB2 data sources and these pipelines had customized UDF'S to extend the ETL functionality and extensively used ETL methodology for supporting Data Extraction, transformations and loading processing, using Hadoop.
• Created real-time streaming Operational Data Store environment with Azure event hubs and stream analytics which directly streamed into MS Power BI for corporate reporting
• Developed Scala scripts, UDF's using bothDataframes/SQL and RDD/Map Reduce in Spark forDataAggregation, queries and writingdataback into RDBMS through Sqoop.
• Wrote Hive join query to fetch info from multiple tables, writing multiple Map Reduce jobs to collect output from Hive and used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard.
• Deploying Azure Resource Manager JSON Templates from PowerShell worked on Azure suite: Azure SQL Database, Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, Azure Analysis Service
• Developed Map-Reduce programs using java andpythonto parse the raw data and store the refined data in Hive and Used UDF's to implement business logic in Hadoop by using Hive to read, write and query the Hadoop data in HBase.
• Implemented micro services, data lake ingestion of data and BI functions using the power U-SQL scripting language provided by Azure and python.
• Worked on writing Perl scripts covering data feed handling, implementing mark logic, communicating with web services through SOAP Lite module and WSDL.
• Used Oozie workflow engine to run multiple Hive and Pig Scripts with the help of Kafka for the real-time processing of data to navigate through data sets in the HDFS storage by loading Log File data directly into HDFS using Flume.
• DevelopedPythonMap Reduce programs for log analysis and Designed Algorithm for finding the fake review by usingpython.
• Involved in developing Map-reduce framework, writing queries scheduling map-reduce and developed the code for Importing and exporting data into HDFS and Hive using Sqoop and Used Hive to analyze data ingested into HBase by using Hive-HBase integration and compute various metrics for reporting on the dashboard.
• Using SparkSQL to create data frames by loading JSON data and analyzing it and developed Spark code using Scala and Spark-SQL for faster testing and data processing.

Confidential

Big Data Engineer/Analyst

Responsibilities:

• Gathered business requirements, working closely with business users, project leaders and developers and analyzed the business requirements and designed data models and analyzed existing systems and propose improvements in processes and systems for usage of modern scheduling tools like Airflow and migrating the legacy systems into an Enterprise data lake.
• Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into Snowflake and involve with Data Extraction, aggregations, and consolidation of Adobe data within AWS Glue using PySpark.
• Build ETL pipeline end to end from AWS S3 to Key, Value store Dynamo DB, and Snowflake Data warehouse for analytical queries and specifically for cloud data.
• Designed and implemented a real-time data pipeline to process financial data by integrating ~150 billion raw records from multiple data sources using SQL, SnowSQL, Jenkins and stored processed data in Snowflake Cloud.
• Created Hive Fact tables on top of raw data from different retailer’s which indeed partitioned by Time dimension key, Retailer name, Data supplier name which further processed pulled by analytics service engine.
• Developed highly complex Python and SQL code, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.
• Design and construct of AWS Data pipelines using various resources in AWS including AWS API Gateway to receives response from AWS lambda and retrieve data from snowflake using lambda function and convert the response into Json format using Database as Snowflake, Dynamo DB, AWS Lambda function and AWS S3.
• On Snowflake data migration involved in Bulk loading and unloading data into Snowflake tables using COPY command, Validating the data feed from the source systems to Snowflake DW cloud platform and Ensuring ETL/ELTs succeeded and loaded data successfully in Snowflake DB.
• Worked on creating various types of indexes on different collections to get good performance in Mongo DB database and implemented scripts for import, export, dump and restore.
• Involved in designing optimizing Spark SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS/AWS S3.
• Created data pipeline for different events of ingestion, aggregation, and load consumer response data in AWS S3 bucket into Hive external tables in HDFS location to serve as feed for tableau dashboards.
• Created time series data-store to log changes on Tags in AWS account into RDS Aurora PostgreSQL.
• Create Notebooks in Data bricks to pull the data from S3 and process with the transformation rules and load back the data to persistence area in S3 in Apache Parquet format.
• Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS and Responsible for building scalable distributed data solutions using EMR cluster environment with Amazon EMR.
• Amazon S3 using S3fs fuse, to has KMS encryption enabled on the data reflecting in S3 buckets, Used AWS glue catalog with crawler to get the data from S3 and perform SQL query operations and JSON schema to define table and column mapping from S3 data to Redshift.
• DevelopedTableauvisualizations and dashboards usingTableauDesktop, Tableauworkbooks from multipledatasources usingDataBlending and Scheduled Airflow DAGs to run multiple Hive and Spark jobs, which independently run with time and data availability.

Confidential

ETL Developer

Responsibilities:

• Scrum methodology Agile Development Methodology was followed for enhancement of the project.
• Actively Participated in grooming to discuss/ understand the requirements from the business people. Estimated hours for the Requirements and allocated hours in the sprint planning, acted as scrum master role for this Project.
• Managed small sized off shore team and assigning/gathering task status and update to the client manager in daily scrum calls.
• Created and Modified tables, views, Indexes, Functions, Stored Procedures, cursors in SQL Server 2012.
• Written the complex queries using joints, Sub-queries, common table expressions (CTE) in T-SQL-2012.
• Experienced in writing Dynamic T-SQL in stored procedures.
• Created and Managed production/ Configuration tables using deployment scripts.
• Improved the Scalability and Manageability of large tables by creating Table Partitions and maintained Data Consistency and Data Integrity by applying various constraints like Primary Key, Foreign Key, Unique, Check, and Default.
• Made changes to existing reports and new reports from ICD 9 to ICD 10 codes.
• Given Demos to the business for UAT Testing after at the end of the Sprint, helped business people in UAT testing in UAT environment.
• Supported operations team as tier 3 support to resolve production issues when needed.
• Designed and optimized dynamic SQL batches to perform bulk operations in the OLTP environments.
• Involved in fine tuning queries and stored procedures using SQL Profiler and Execution plan in SQL Server.
• Created package with different data sources (SQL Server, Oracle, Flat Files, Excel files, etc.) and loaded the data into target tables by performing different kinds of transformations using SQL Server Integration Services (SSIS).
• Successfully deployed SSIS Package into QA/UAT/Production environment and used Package configuration to export various package properties to make package environment independent.
• Developed Scheduled Jobs to automate the SSIS packages.
• Used Event Handlers and implemented Error Handling in SSIS packages and notified process results to various user groups.
• Developed a Tabular Reports, matrix Reports, Parameterized Reports, Drill down, Drill through and cascading parameter Reports and ad-hoc reports using SSRS Report Designer.
• Employed many Slowly Changing Dimension methodologies to maintain and update historical data.
• Configured SSIS packages using Package configuration wizard to allow packages run on different environments.
• Task and Script Component in SSIS to implement out-of-the box functionality in SSIS.
• Maintain minimal issue count in UAT and Production release.
• Involved in designing and development of services in C#.NET which communicates the database and render the data back to the client consuming the service.
• Build enhancements on ASP.NETweb application usingC#.
• Deployed code into the production environment. Provided 24/7 production support for the production database.
• Import/export data using Sqoop from Relational Database systems to HDFS.
• Involved in creating Hive tables, loading and analyzing data using hive queries.
• Developed POCs on Hadoop using HIVE, PIG and Sqoop Given Demo to the Team Management as part of the Year Goal.

Confidential

SQL Developer/DBA

Responsibilities:

• Collaborated in analyzing business requirements and understanding the functional work flow of information from source systems to destination systems.
• Expertise in Always-On Availability Groups (AAG) and HA/DR Setup and Configuration
• Migrated SQL server 2005 to SQL Server 2008 in Microsoft Windows Server 2008 Enterprise Edition.
• Installed and administered Eight-Node cluster Active/Passive on Microsoft Windows Server 208 Enterprise Edition on SAN environment.
• Migrated SSIS packages from SQL Server 2005 to SSIS 2008.
• Extensively worked on SQL Server Integration Services (SSIS) and SQL Server Reporting Services (SSRS).
• Expert in Using DMVs, Performance dashboard, Mirroring, database snapshots and tracking Performance Counters.
• Rebuilding / Monitoring the indexes at regular intervals for better performance.
• Recovering the databases from backup in disasters. Implementing Point-In-Time Recovery solutions.
• Involved in trouble shooting and fine-tuning of databases for its performance and concurrency.
• Involved in Source Data Analysis, analysis and designing mappings for data extraction also responsible for Design and Development of SSIS Packages to load the Data from various Databases and Files

Confidential

Python Developer

Responsibilities:

• Developed scripts in python for configuring and tracking progress in one Siebel and B2B CRM.
• Developed web-service for client-handler tier and connected with Database tier in Oracle Flex cube.
• Experience in developing Service Fulfillment, New Service Introduction, Service Provisioning,
• Service Assurance, Reporting, Mediation and Billing journeys related to CRM/BSS/OSS COTS systems like Siebel and B2B CRM.
• Responsible for developing and deploying network configuration on routers like Ericsson SE-800 and CISCO CAT-6509 switches.
• Responsible for automating network configuration deployment in shell script and python.