SUMMARY

• Almost 8+ years of professional experience in IT, which includes work experience in Big Data, Hadoop ecosystem for data processing, Data Warehousing and Data Pipeline design and implementation.
• Expert knowledge in SDLC (Software Development Life Cycle) and was involved in all phases in projects.
• Expertise in using Cloud based managed services for data warehousing in Confidential Azure (Azure Data Lake Storage, Azure Data Factory).
• Strong experience with big data processing using Hadoop technologies Map Reduce, Apache Spark, Apache Hive and Pig.
• Good understanding of cloud configuration in Amazon web services (AWS).
• Have experience in Dimensional Modeling using Star and Snowflake schema methodologies of Data Warehouse and Integration projects
• Worked with the Apache Airflow engine that can easily schedule and run my complex data pipelines which will make each task to get executed in a correct order
• Excellent proficiency in Agile/Scrum and waterfall methodologies.
• Extensive experience in using ER modeling tools such as Erwin and ER/Studio.
• Experience in integration of various data sources with multiple Relational Databases like SQLServer, Teradata, and Oracle.
• Proficient in data governance, Experience in Data Ingestion projects to inject data into Data Lake using multiple sources systems using Talend BigData.
• Excellent knowledge in Data Analysis, Data Validation, Data Cleansing, Data Verification and identifying data mismatch, data quality, metadata management, master data management.
• Experience in working with creating ETL specification documents, & creating flowcharts, process work flows and data flow diagrams.
• Experience in execution of Batch jobs through the data streams to SPARK Streaming.
• Good knowledge in streaming applications using Apache Kafka.
• Hands on experience in working with Tableau Desktop, Tableau Server and Tableau Reader in various versions.
• Extending HIVE and PIG core functionality by using custom UDF's.
• Experience in designing both time driven and data driven automated workflows using Oozie.
• Expertise in SQL Server Analysis Services (SSAS), SQL Server Reporting Services (SSRS) and SQL Server Integration Services.
• In - depth knowledge of T-SQL, SSAS, SSRS, SSIS, OLAP, OLTP, BI suite, Reporting and Analytics.
• Strong experience in using MS Excel and MS Access to dump the data and analyze based on business needs.
• Good communication skills, work ethics and the ability to work in a team efficiently with good leadership skills.
• Strong experience in migrating data warehouses and databases into Hadoop/NoSQL platforms.
• Designing and Developing Oracle PL/SQL and Shell Scripts, Data Conversions and Data Cleansing.
• Experience with query optimization and ETL loading on Teradata.
• Extensive experience in ETL tools like Teradata Utilities, Informatica, and Oracle.
• In depth understanding of Apache spark job execution Components like DAG, lineage graph, Dag Scheduler, Task scheduler, Stages and Worked on NoSQL databases including HBase and Mongo DB.
• I have created dash boards using Tableau and Power BI for getting valuable insights based up on the business uses and stakeholder requirement.

TECHNICAL SKILLS

Hadoop/Big Data Technologies: Hadoop, Map Reduce, Sqoop, Hive, Oozie, Spark, Zookeeper and Cloudera Manager, Kafka, Flume

ETL Tools: Informatica

NO SQL Database: HBase, Cassandra, Dynamo DB, Mongo DB.

Monitoring and Reporting: Tableau, Custom shell scripts

Hadoop Distribution: Horton Works, Cloudera

Build Tools: Maven

Programming & Scripting: Python, Scala, JAVA, SQL, Shell Scripting, C, C++

Databases: Oracle, MY SQL, Teradata

Machine Learning & Analytics Tools: Supervised Learning (Linear Regression, Logistic Regression, Decision Tree, Random Forest, SVM, Classification), Unsupervised Learning (Clustering, KNN, Factor Analysis, PCA), Natural Language Processing, Google Analytics Fiddler, Tableau

Version Control: Git, GitHub, SVN, CVS

Operating Systems: Linux, Unix, Mac OS-X, CentOS, Windows 10, Windows 8, Windows 7

Cloud Computing: AWS, Azure

AWS Services: Amazon EC2, Amazon S3, Amazon Simple DB, Amazon MQ, Amazon ECS, Amazon Lambdas, Amazon Sagemaker, Amazon RDS, Amazon Elastic Load Balancing, Elastic Search, Amazon SQS, AWS Identity and access management, AWS Cloud Watch, Amazon EBS and Amazon CloudFormation

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, Nashville, TN

Responsibilities:

• Developed complete end to end Big-data processing in Hadoop eco system.
• Provided application support during the build and test phases of the SDLC for their product.
• Used Oozie for automating the end-to-end data pipelines and Oozie coordinators for scheduling the work flows.
• Recreated existing application logic and functionality in the Azure Data Lake, Data Factory, Data Bricks, SQL Database and SQL data warehouse environment.
• Performed data profiling and transformation on the raw data using Pig, Python, and oracle
• Developed predictive analytics using Apache Spark Scala APIs.
• Created dimensional model for the reporting system by identifying required dimensions and facts using Erwin.
• Developed and implemented a data pipeline using Kafka and Strom to store data into HDFS.
• Created automated python scripts to convert the data from different sources and to generate the ETL pipelines.
• Analyze, design and build Modern data solutions using Azure Pass service to support visualization of data. Understand current Production state of application and
• Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming.
• Involved in modeling (Star Schema methodologies) in building and designing the logical data model into Dimensional Models.
• Created indexes both non clustered and clustered indexes in order to maximize the query performance in T-SQL.
• Created Hive External tables and loaded the data into tables and query data using HQL.
• Generated multiple enterprise reports like SSRS and Crystal report, worked on Tableau.
• Managed Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services
• Wrote MapReduce jobs to generate reports for the number of activities created on a day, during a dumped from the multiple sources and the output was written back to HDFS.
• Used Sqoop to efficiently transfer data between databases and HDFS.
• Developed Spark code using Scala and Spark-SQL for faster testing and data processing.

Environment: Erwin, SQL, Oracle12c, PL/SQL, Big data, Hadoop, Azure Data Lake, Spark, Scala, APIs, Pig, Python, Kafka, HDFS, ETL, MDM, OLAP, OLTP, SSAS, T-SQL, Hive, SSRS, Tableau, Map Reduce, Sqoop, Scala, HBase, SSIS.

Data Engineer

Confidential, Negaunee, MI

Responsibilities:

• Developed Spark Streaming application in the Agile methodology in multiple sprints.
• Developed Consumer application to consume the data from Kafka to Hive External tables using Scala, Spark.
• Implemented Asynchronous commits in Consumer application to commit Kafka Offsets
• Designed and developed a horizontally scalable APIs using Spark.
• Performed Data validations using Scala-Spark scripts by capturing the Raw Data on HDFS and comparing with data in final tables.
• Worked with Data ingestion, querying, processing and analysis of big data.
• Performed tuned and optimized various complex SQL queries.
• Developed normalized Logical and Physical database models to design OLTP system.
• Extensively involved in creating PL/SQL objects i.e. Procedures, Functions, and Packages.
• Performed bug verification, release testing and provided support for Oracle based applications.
• Used Model Mart of Erwin for effective model management of sharing, dividing and reusing model information and design for productivity improvement
• Worked with Real-time Streaming using Kafka and HDFS.
• Worked with Alteryx a data Analytical tool to develop workflows for the ETL jobs.
• Designed the data marts in dimensional data modeling using star and snowflake schemas.
• Used various OLAP operations like slice / dice, drill down and roll up as per business requirements.
• Handled importing of data from various data sources, performed data control checks using Spark and loaded data into HDFS.
• Designed SSRS reports with sub reports, dynamic sorting, defining data source and subtotals for the report.
• Designed and implemented importing data to HDFS using Sqoop from different RDBMS servers.
• Worked with Sqoop commands to import the data from different databases.
• Designed and developed Map Reduce jobs to process data coming in different file formats like XML.

Environment: Erwin, SQL, PL/SQL, Kafka1.1, AWS, API's, Agile, ETL, HDFS, OLAP, HDFS, T-SQL, SSIS, Teradata, Hive, SSRS, Sqoop, Tableau, Map Reduce, XML.

Data Engineer

Confidential, Louis MO

Responsibilities:

• Spearheaded analysis, design and development for building a common architecture and data repository for Apparel and Footwear sourcing data across the Geo's for data management and reporting needs.
• Work cross functionally with analysts, business insights, report developers, and stakeholders to ensure a tight Development/QA/UAT process and quality deployments.
• Evaluate, cleanse, extract/transform data for analytic purpose within the context of Big data environment and structure large data sets by applying standard data modelling methods.
• Design and develop efficient PySpark programs using cloud-based data platforms (EMR) to extract/transform/load data in between various data warehouse applications.
• Worked on Cloud Platform like AWS and possess good knowledge on different types of instances for optimal usage of clusters based on the requirement.
• Engineered a solution to optimize the ETL process of Alteryx to Snowflake ingestion process which takes around
• Designed Hive tables over the parquet and csv files for loading and analyzing data.
• Involved in developing DAGS using Airflow orchestration tool and monitored the weekly processes.
• Identify gaps in data processes and drive improvements via continuous improvement loop by ensuring good data flow between databases and backend systems.

Environment: Python, Alteryx, AWS S3, AWS EMR, Spark, Hive, Airflow, Sqoop, Teradata SQL assistant, Snowflake, MS SQL Server 2016, Tableau 2018.3, Snowflake database, R - Programming, SharePoint, SSIS

Hadoop Developer

Confidential

Responsibilities:

• Importing and sending out information into HDFS and Hive utilizing Sqoop.
• Experienced in running Hadoop stream jobs to process terabytes of xml group information with the help of Map Reduce programs.
• Used parquet file format for published tables and created views on the tables.
• In-charge of managing data coming from different sources.
• Support in running MapReduce Programs in the cluster.
• Cluster coordination services through Zoo Keeper.
• Involved in loading information from UNIX document framework to Hadoop Distributed File System.
• Installed, configured Hive and furthermore composed Hive UDFs.
• Automated every one of the jobs for pulling information from FTP server to stack information into Hive tables, utilizing Oozie work processes.
• Writing data to parquet tables both non-partitioned and partitioned tables by adding dynamic data to partitioned tables using Spark.
• Wrote User Defined functions (UDFs) for special functionality for Spark.
• Used SQOOP Export functionalities and scheduled the jobs on daily basis with Shell scripting in Oozie.
• Worked with SQOOP jobs to import the data from RDBMS and used various optimization techniques to optimize Hive and SQOOP.
• Used SQOOP import functionality for loading Historical data present in a Relational Database system into Hadoop File System (HDFS).

Environment: - Hadoop, MapReduce, HDFS, Hive, Java, R, Sqoop, Spark, Sqoop

Data Analyst/Data Modeler

Confidential

Responsibilities:

• As a Data Analyst /Data Modeler I was responsible for all data related aspects of a project.
• Created report on Cloud based environment using Amazon Redshift and published on Tableau
• Developed of Python APIs to dump the array structures in the Processor at the failure point for debugging.
• Worked extensively on ER/ Studio in several projects in both OLAP and OLTP applications.
• Created SQL tables with referential integrity and developed queries using SQL, SQL*PLUS and PL/SQL.
• Performed Data Analysis and data profiling using complex SQL on various sources systems including Oracle.
• Developed the required data warehouse model using Star schema for the generalized model
• Implemented Visualized BI Reports with Tableau.
• Worked on stored procedures for processing business logic in the database.
• Extensively worked on Viewpoint for Teradata to look at performance Monitoring and performance tuning.
• Performed Extract, Transform and Load (ETL) solutions to move legacy and ERP data into Oracle data warehouse.
• Developed and maintained data dictionary to create metadata reports for technical and business purpose.
• Managed database design and implemented a comprehensive Snow flake-Schema with shared dimensions.
• Worked Normalization and De-normalization concepts and design methodologies.
• Worked on the reporting requirements for the data warehouse.
• Worked on SQL Server concepts SSIS (SQL Server Integration Services), SSAS (Analysis Services) and SSRS (Reporting Services).
• Developed complex T-Sql code such as Stored Procedures, functions, triggers, Indexes, and views for the business application.
• Wrote a complex SQL, PL/SQL, Procedures, Functions, and Packages to validate data and testing process.

Environment: ER-Studio, Oracle 10g, SQL Server, Crystal Reports, Windows 7, MS Office, Teradata, PL/SQL, T-SQL, Informatica, SSRS, SAS, SSIS, XML, Excel, Pivot Tables, Metadata, Cognos