SUMMARY

• Almost 8+ years of professional experience in IT, which includes work experience in Big Data, Hadoop ecosystem for data processing, Data Warehousing and Data Pipeline design and implementation.
• Expert knowledge in SDLC (Software Development Life Cycle) and was involved in all phases in projects.
• Expertise in using Cloud based managed services for data warehousing in Confidential Azure (Azure Data Lake Storage, Azure Data Factory).
• Strong experience with big data processing using Hadoop technologies Map Reduce, Apache Spark, Apache Hive and Pig.
• Good understanding of cloud configuration in Amazon web services (AWS).
• Have experience in Dimensional Modeling using Star and Snowflake schema methodologies of Data Warehouse and Integration projects
• Worked with teh Apache Airflow engine that can easily schedule and run my complex data pipelines which will make each task to get executed in a correct order
• Excellent proficiency in Agile/Scrum and waterfall methodologies.
• Extensive experience in using ER modeling tools such as Erwin and ER/Studio.
• Experience in integration of various data sources with multiple Relational Databases like SQLServer, Teradata, and Oracle.
• Proficient in data governance, Experience in Data Ingestion projects to inject data into Data Lake using multiple sources systems using Talend BigData.
• Excellent knowledge in Data Analysis, Data Validation, Data Cleansing, Data Verification and identifying data mismatch, data quality, metadata management, master data management.
• Experience in working with creating ETL specification documents, & creating flowcharts, process work flows and data flow diagrams.
• Experience in execution of Batch jobs through teh data streams to SPARK Streaming.
• Good knowledge in streaming applications using Apache Kafka.
• Hands on experience in working with Tableau Desktop, Tableau Server and Tableau Reader in various versions.
• Extending HIVE and PIG core functionality by using custom UDF's.
• Experience in designing both time driven and data driven automated workflows using Oozie.
• Expertise in SQL Server Analysis Services (SSAS), SQL Server Reporting Services (SSRS) and SQL Server Integration Services.
• In - depth knowledge of T-SQL, SSAS, SSRS, SSIS, OLAP, OLTP, BI suite, Reporting and Analytics.
• Strong experience in using MS Excel and MS Access to dump teh data and analyze based on business needs.
• Good communication skills, work ethics and teh ability to work in a team efficiently with good leadership skills.
• Strong experience in migrating data warehouses and databases into Hadoop/NoSQL platforms.
• Designing and Developing Oracle PL/SQL and Shell Scripts, Data Conversions and Data Cleansing.
• Experience with query optimization and ETL loading on Teradata.
• Extensive experience in ETL tools like Teradata Utilities, Informatica, and Oracle.
• In depth understanding of Apache spark job execution Components like DAG, lineage graph, Dag Scheduler, Task scheduler, Stages and Worked on NoSQL databases including HBase and Mongo DB.
• me have created dash boards using Tableau and Power BI for getting valuable insights based up on teh business uses and stakeholder requirement.

TECHNICAL SKILLS

Hadoop/Big Data Technologies: Hadoop, Map Reduce, Sqoop, Hive, Oozie, Spark, Zookeeper and Cloudera Manager, Kafka, Flume

ETL Tools: Informatica

NO SQL Database: HBase, Cassandra, Dynamo DB, Mongo DB.

Monitoring and Reporting: Tableau, Custom shell scripts

Hadoop Distribution: Horton Works, Cloudera

Build Tools: Maven

Programming & Scripting: Python, Scala, JAVA, SQL, Shell Scripting, C, C++

Databases: Oracle, MY SQL, Teradata

Machine Learning & Analytics Tools: Supervised Learning (Linear Regression, Logistic Regression, Decision Tree, Random Forest, SVM, Classification), Unsupervised Learning (Clustering, KNN, Factor Analysis, PCA), Natural Language Processing, Google Analytics Fiddler, Tableau

Version Control: Git, GitHub, SVN, CVS

Operating Systems: Linux, Unix, Mac OS-X, CentOS, Windows 10, Windows 8, Windows 7

Cloud Computing: AWS, Azure

AWS Services: Amazon EC2, Amazon S3, Amazon Simple DB, Amazon MQ, Amazon ECS, Amazon Lambdas, Amazon Sagemaker, Amazon RDS, Amazon Elastic Load Balancing, Elastic Search, Amazon SQS, AWS Identity and access management, AWS Cloud Watch, Amazon EBS and Amazon CloudFormation

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, Nashville, TN

Responsibilities:

• Developed complete end to end Big-data processing in Hadoop eco system.
• Provided application support during teh build and test phases of teh SDLC for their product.
• Used Oozie for automating teh end-to-end data pipelines and Oozie coordinators for scheduling teh work flows.
• Recreated existing application logic and functionality in teh Azure Data Lake, Data Factory, Data Bricks, SQL Database and SQL data warehouse environment.
• Performed data profiling and transformation on teh raw data using Pig, Python, and oracle
• Developed predictive analytics using Apache Spark Scala APIs.
• Created dimensional model for teh reporting system by identifying required dimensions and facts using Erwin.
• Developed and implemented a data pipeline using Kafka and Strom to store data into HDFS.
• Created automated python scripts to convert teh data from different sources and to generate teh ETL pipelines.
• Analyze, design and build Modern data solutions using Azure Pass service to support visualization of data. Understand current Production state of application and
• Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing teh data in In Azure
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming.
• Involved in modeling (Star Schema methodologies) in building and designing teh logical data model into Dimensional Models.
• Created indexes both non clustered and clustered indexes in order to maximize teh query performance in T-SQL.
• Created Hive External tables and loaded teh data into tables and query data using HQL.
• Generated multiple enterprise reports like SSRS and Crystal report, worked on Tableau.
• Managed Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services
• Wrote MapReduce jobs to generate reports for teh number of activities created on a day, during a dumped from teh multiple sources and teh output was written back to HDFS.
• Used Sqoop to efficiently transfer data between databases and HDFS.
• Developed Spark code using Scala and Spark-SQL for faster testing and data processing.

Environment: Erwin, SQL, Oracle12c, PL/SQL, Big data, Hadoop, Azure Data Lake, Spark, Scala, APIs, Pig, Python, Kafka, HDFS, ETL, MDM, OLAP, OLTP, SSAS, T-SQL, Hive, SSRS, Tableau, Map Reduce, Sqoop, Scala, HBase, SSIS.

Data Engineer

Confidential, Negaunee, MI

Responsibilities:

• Developed Spark Streaming application in teh Agile methodology in multiple sprints.
• Developed Consumer application to consume teh data from Kafka to Hive External tables using Scala, Spark.
• Implemented Asynchronous commits in Consumer application to commit Kafka Offsets
• Designed and developed a horizontally scalable APIs using Spark.
• Performed Data validations using Scala-Spark scripts by capturing teh Raw Data on HDFS and comparing with data in final tables.
• Worked with Data ingestion, querying, processing and analysis of big data.
• Performed tuned and optimized various complex SQL queries.
• Developed normalized Logical and Physical database models to design OLTP system.
• Extensively involved in creating PL/SQL objects me.e. Procedures, Functions, and Packages.
• Performed bug verification, release testing and provided support for Oracle based applications.
• Used Model Mart of Erwin for TEMPeffective model management of sharing, dividing and reusing model information and design for productivity improvement
• Worked with Real-time Streaming using Kafka and HDFS.
• Worked with Alteryx a data Analytical tool to develop workflows for teh ETL jobs.
• Designed teh data marts in dimensional data modeling using star and snowflake schemas.
• Used various OLAP operations like slice / dice, drill down and roll up as per business requirements.
• Handled importing of data from various data sources, performed data control checks using Spark and loaded data into HDFS.
• Designed SSRS reports with sub reports, dynamic sorting, defining data source and subtotals for teh report.
• Designed and implemented importing data to HDFS using Sqoop from different RDBMS servers.
• Worked with Sqoop commands to import teh data from different databases.
• Designed and developed Map Reduce jobs to process data coming in different file formats like XML.

Environment: Erwin, SQL, PL/SQL, Kafka1.1, AWS, API's, Agile, ETL, HDFS, OLAP, HDFS, T-SQL, SSIS, Teradata, Hive, SSRS, Sqoop, Tableau, Map Reduce, XML.

Data Engineer

Confidential, Louis MO

Responsibilities:

• Spearheaded analysis, design and development for building a common architecture and data repository for Apparel and Footwear sourcing data across teh Geo's for data management and reporting needs.
• Work cross functionally with analysts, business insights, report developers, and stakeholders to ensure a tight Development/QA/UAT process and quality deployments.
• Evaluate, cleanse, extract/transform data for analytic purpose within teh context of Big data environment and structure large data sets by applying standard data modelling methods.
• Design and develop efficient PySpark programs using cloud-based data platforms (EMR) to extract/transform/load data in between various data warehouse applications.
• Worked on Cloud Platform like AWS and possess good knowledge on different types of instances for optimal usage of clusters based on teh requirement.
• Engineered a solution to optimize teh ETL process of Alteryx to Snowflake ingestion process which takes around
• Designed Hive tables over teh parquet and csv files for loading and analyzing data.
• Involved in developing DAGS using Airflow orchestration tool and monitored teh weekly processes.
• Identify gaps in data processes and drive improvements via continuous improvement loop by ensuring good data flow between databases and backend systems.

Environment: Python, Alteryx, AWS S3, AWS EMR, Spark, Hive, Airflow, Sqoop, Teradata SQL assistant, Snowflake, MS SQL Server 2016, Tableau 2018.3, Snowflake database, R - Programming, SharePoint, SSIS

Hadoop Developer

Confidential

Responsibilities:

• Importing and sending out information into HDFS and Hive utilizing Sqoop.
• Experienced in running Hadoop stream jobs to process terabytes of xml group information with teh halp of Map Reduce programs.
• Used parquet file format for published tables and created views on teh tables.
• In-charge of managing data coming from different sources.
• Support in running MapReduce Programs in teh cluster.
• Cluster coordination services through Zoo Keeper.
• Involved in loading information from UNIX document framework to Hadoop Distributed File System.
• Installed, configured Hive and furthermore composed Hive UDFs.
• Automated every one of teh jobs for pulling information from FTP server to stack information into Hive tables, utilizing Oozie work processes.
• Writing data to parquet tables both non-partitioned and partitioned tables by adding dynamic data to partitioned tables using Spark.
• Wrote User Defined functions (UDFs) for special functionality for Spark.
• Used SQOOP Export functionalities and scheduled teh jobs on daily basis with Shell scripting in Oozie.
• Worked with SQOOP jobs to import teh data from RDBMS and used various optimization techniques to optimize Hive and SQOOP.
• Used SQOOP import functionality for loading Historical data present in a Relational Database system into Hadoop File System (HDFS).

Environment: - Hadoop, MapReduce, HDFS, Hive, Java, R, Sqoop, Spark, Sqoop

Data Analyst/Data Modeler

Confidential

Responsibilities:

• As a Data Analyst /Data Modeler me was responsible for all data related aspects of a project.
• Created report on Cloud based environment using Amazon Redshift and published on Tableau
• Developed of Python APIs to dump teh array structures in teh Processor at teh failure point for debugging.
• Worked extensively on ER/ Studio in several projects in both OLAP and OLTP applications.
• Created SQL tables with referential integrity and developed queries using SQL, SQL*PLUS and PL/SQL.
• Performed Data Analysis and data profiling using complex SQL on various sources systems including Oracle.
• Developed teh required data warehouse model using Star schema for teh generalized model
• Implemented Visualized BI Reports with Tableau.
• Worked on stored procedures for processing business logic in teh database.
• Extensively worked on Viewpoint for Teradata to look at performance Monitoring and performance tuning.
• Performed Extract, Transform and Load (ETL) solutions to move legacy and ERP data into Oracle data warehouse.
• Developed and maintained data dictionary to create metadata reports for technical and business purpose.
• Managed database design and implemented a comprehensive Snow flake-Schema with shared dimensions.
• Worked Normalization and De-normalization concepts and design methodologies.
• Worked on teh reporting requirements for teh data warehouse.
• Worked on SQL Server concepts SSIS (SQL Server Integration Services), SSAS (Analysis Services) and SSRS (Reporting Services).
• Developed complex T-Sql code such as Stored Procedures, functions, triggers, Indexes, and views for teh business application.
• Wrote a complex SQL, PL/SQL, Procedures, Functions, and Packages to validate data and testing process.

Environment: ER-Studio, Oracle 10g, SQL Server, Crystal Reports, Windows 7, MS Office, Teradata, PL/SQL, T-SQL, Informatica, SSRS, SAS, SSIS, XML, Excel, Pivot Tables, Metadata, Cognos