SUMMARY

• Over 6+ years IT experience in Data Engineering, Analytics, Data Modeling and Big Data using Scala, PySpark, Hadoop, and HDFS environment and experienced in Python, Cloud Technologies (AWS, Azure).
• Implemented Big Data solutions using Hadoop technology stack, including PySpark, Hive, Sqoop, Avro and Thrift.
• Proficiency in developing SQL queries with various relational databases like Oracle, SQL Server for Support of Data Warehousing and Data Integration Solutions using Informatica PowerCenter and Data Stage. worked on developing ETL work scheduling for Data Extraction, transformations and loading using Informatica Power Center.
• Hands on Experience with dimensional modeling using star schema and snowflake models.
• Implemented UNIX shell scripts to run teh Informatica workflows and controlling teh ETL flow.
• Firm understanding of Hadoop architecture and various components including HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce programming.
• Expertise in Creating, Debugging, Scheduling and Monitoring jobs using Airflow and Oozie.
• Experienced in Optimizing teh PySpark jobs to run on Kubernetes Cluster for faster data processing
• Involved in converting Hive Queries into various Spark Actions and Transformations by Creating RDD and Data frame from teh required files in HDFS.
• Strong noledge of Hadoop Architecture and Daemons such asHDFS,JOB Tracker,Task Tracker,Name Node,Data NodeandMap Reduceconcepts.
• Experienced in Designing, Architecting, and implementing scalable cloud - based web applications usingAWSandAzure.
• Experienced on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and controlling and granting database access and migrating on premise databases to Azure Data Lake store using Azure Data factory.
• Experienced in developing scripts using Python, Shell Scripting to do Extract, Load and Transform data working noledge of AWS Redshift.
• Designed AWS Data pipelines using various resources in AWS including AWS API Gateway to receives response from AWS lambda and retrieve data from snowflake using lambda function and convert teh response into Json format using Database as Snowflake, DynamoDB, AWS Lambda function and AWS S3.
• Involved in Software development, Data warehousing and Analytics and Data engineering projects using Hadoop, MapReduce, Hive, and other open-source tools/technologies.
• Designed Star and Snowflake Data Models for Enterprise Data Warehouse using ER Studio. Used ER Studio for Creating/Updating Data Models.
• Strong SAS macro programming experience in SAS - BASE SAS, SAS/STAT, SAS/GRAPH, SAS SQL, SAS ODS, SAS/ETL, SAS/ACCESS, SAS/ASSIST, SAS/CONNECT, SAS/ETS, SAS PC and SAS Enterprise Guide, UNIX environment. noledge of SAS procedures like Proc Format, Proc Report, Proc Means, Proc Summary, Proc Freq, Proc Univariate, Proc Transpose, Proc SQL, Proc Import, Proc Tabulate, Proc Plot, Proc Chart, Proc Plot, Proc Compare and teh DATA NULL data step.
• Developed Mappings, Sessions, and Workflows to extract, validate, and transform data according to teh business rules using Data Stage
• Used Model Mart of ERwin for effective model management of sharing, dividing and reusing model information and design for productivity improvement.
• Hands-on experience with Informatica power center in integrating with different applications and relational databases
• Exposure to Full Lifecycle (SDLC) of Data Warehouse projects including Dimensional Data Modeling.
• Defined data warehouse schemas (star and snowflake schema), fact table, cubes, dimensions, measures using SQL Server Analysis Services.
• Developed AWS CI/CD Data pipeline and AWS Data Lake using EC2, AWS Glue, AWS Lambda.
• Experience with Snowflake cloud data warehouse and AWS S3 bucket for integrating data from multiple source system which include loading nested JSON formatted data into snowflake table.
• Hands on Experience with different API Endpoints like Edge Optimized, Regional, Private in Aws API Gateway. Also, Configured control connections different levels like API Key, Method level, Account Level.
• Experienced in Optimization of Hive queries using best practices and right parameters and using technologies like Hadoop, YARN, Python, PySpark.
• Basic understanding of Network Protocols like SMTP, DHCP, TCP/IP, DNS, SFTP, FTP, WINS, UDP, SNMP.
• Experienced in requirement analysis, application development, application migration and maintenance using Software Development Lifecycle (SDLC) and Python/Java technologies.
• Defined user stories and driving teh agile board in JIRA during project execution, participate in sprint demo and retrospective.

TECHNICAL SKILLS

Big Data Eco System: HDFS, MapReduce, Hive, Yarn, Pig, Sqoop, Flume, HBase, Kafka Connect, Impala, Stream sets, Oozie, Spark, Zookeeper, NiFi, Amazon Web Services.

Hadoop Distributions: Apache Hadoop 1.x/2.x, Cloudera CDP, Hortonworks HDP

Programming Languages: Python, Scala, Java, R, Pig Latin, HiveQL, Shell Scripting, Bash

Software Methodologies: Agile, SDLC Waterfall.

Design Patterns: Eclipse, Net Beans, IntelliJ, Spring Tool Suite.

Databases: MySQL, MS SQL SERVER, Oracle, PostgreSQL, DB2, DynamoDB, workbench

NoSQL: HBase, MongoDB, Cassandra.

ETL/BI: Power BI, Tableau, Talend, Informatica, SSIS, SSRS, SSAS.

Version control: GIT, SVN, Bitbucket.

Web Development: JavaScript, Node.js, HTML, CSS, Spring, J2EE, JDBC, Angular, Hibernate, Tomcat.

Operating Systems: Windows (XP/7/8/10), Linux (Unix, Ubuntu), Mac OS.

Cloud Technologies: Amazon Web Services, EC2, S3, Azure DataBricks.

PROFESSIONAL EXPERIENCE

Confidential, St. Louis, MO

Sr. Big Data Engineer

Responsibilities:

• Worked with building data warehouse structures, and creating facts, dimensions, aggregate tables, by dimensional modeling, Star and Snowflake schemas.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing teh data in InAzure Databricks.
• Developed JSON Scripts for deploying teh Pipeline in Azure Data Factory (ADF) that process teh data using teh Sql Activity.
• Created Sessions and extracted data from various sources, transformed data according to teh requirement and loading into data warehouse.
• Used various transformations like Filter, Expression, Sequence Generator, Update Strategy, Joiner, Router and Aggregator to create robust mappings in teh Informatica Power Center Designer.
• Used Informatica Power Center for (ETL) extraction, transformation and loading data from heterogeneous source systems into target database.
• Used ER Studio for Creating/Updating Data Models.
• Created mappings using Designer and extracted data from various sources, transformed data according to teh requirement.
• Developed spark applications in PySpark on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables.
• Designed Star and Snowflake Data Models for Enterprise Data Warehouse using ER Studio.
• Created and implemented ER models and dimensional models (star schemas).
• Translated teh business requirements into workable functional and non-functional requirements at detailed production level using Workflow Diagrams, Sequence Diagrams, Activity Diagrams and Use Case Modelling with help of ER Studio.
• Exposure to Full Lifecycle (SDLC) of Data Warehouse projects including Dimensional Data Modeling.
• Migrated data from hive to MySQL, to be displayed on UI by using PySpark job which runs for different environments.
• Evaluate Snowflake Design considerations for any change in teh application. Build teh Logical and Physical data model for snowflake as per teh changes required
• Building/Maintaining Docker container clusters managed byKubernetes, Linux, Bash, GIT, Docker, on Azure. Utilized Kubernetes and Docker for teh runtime environment of theCI/CDsystem to build, test deploy during production.
• Experience in MongoDB installation, patching, troubleshooting, performance, tracking/tuning, back - up and recovery in dynamic environments, in managing large, shared MongoDB cluster.
• Experience with creating script for data modeling, data import and report. Extensive experience in deploying, managing, and developing MongoDB clusters.
• Experience in creating JavaScript for using DML operation with MongoDB. Responsible for design and development ofSparkSQL Scripts based on Functional Specifications. Created HBase tables to store various data formats of data coming from spark.
• Hands in experience in working with Continuous Integration and Deployment (CI/CD) using Jenkins, Docker.
• Queried multiple databases like Snowflake, UDB and MySQL for data processing.
• Developed ETL pipelines in and out data warehouse using combination of Python and Snowflake SnowSQL. Writing SQL quires against Snowflake.

Environment: Python, PySpark, Azure, HDFS, Spark, Kafka, Hive, Yarn, Cassandra, HBase, Jenkins, Docker, Tableau, Splunk, MySQL, Snowflake, IBM DataStage.

Confidential, Columbus, OH

Big Data Engineer

Responsibilities:

• Responsible for teh design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
• Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper.
• Developed multiple POC’s using Spark, Scala and deployed on teh Yarn Cluster, compared teh performance of Spark, with Hive and SQL.
• Use Amazon Elastic Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage mechanism.
• Capable of using AWS utilities such as EMR, S3 and Cloud Watch to run and monitor Hadoop and Spark jobs on AWS.
• Used SAS/ACCESS to extract data from different relational database management ORACLE and SQL SERVER systems using SAS.
• Extensively used teh SET, UPDATE and MERGE statements for creating, updating, and merging various SAS data sets.
• Extracted, transformed, and loaded data using SAS/ETL.
• Involved in various Transformation and data cleansing activities using various Control flow and data flow tasks in SSIS packages during data migration
• Developed Mappings, Sessions, and Workflows to extract, validate, and transform data according to teh business rules using Data Stage.
• Maintain AWS Data pipeline as web service to process and move data between Amazon S3, Amazon EMR and Amazon RDS resources.
• Worked on NiFi data Pipeline to process large set of data and configured Lookup’s for Data Validation and Integrity.
• Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in python and NiFi for data cleaning and preprocessing.
• Worked with different file formats like Json, AVRO and parquet and compression techniques like snappy. NiFi ecosystem is used.
• Worked on SQL queries in dimensional data warehouses and relational data warehouses. Performed Data Analysis and Data Profiling using Complex SQL queries on various systems.
• Used Model Mart of Erwin for effective model management of sharing, dividing and reusing model information and design for productivity improvement.
• Extensively used Data Null and SAS procedures such as Print, Report, Tabulate, Freq, Means, Summary and Transpose for producing ad-hoc and customized reports and external files.
• Troubleshoot and resolve data processing issues and proactively engaged in data modelling discussions.
• Used ETL (SSIS) to develop jobs for extracting, cleaning, transforming, and loading data into data warehouse.
• Written programs in Spark using Python, PySpark and Pandas packages for performance tuning, optimization, and data quality validations.
• Translated teh business requirements into workable functional and non-functional requirements at detailed production level using Workflow Diagrams, Sequence Diagrams, Activity Diagrams and Use Case Modelling with help of Erwin.
• Worked on developing Kafka Producers and Kafka Consumers for streaming millions of events per second on streaming data.
• Implemented a distributing messaging queue to integrate with Cassandra using Apache Kafka.
• Hands on experience on fetching teh live stream data from UDB into HBase table using PySpark streaming and Apache Kafka.
• Worked on Tableau to build customize interactive reports, worksheets, and dashboards.

Environment: HDFS, Python, SQL, Web Services, MapReduce, Spark, Kafka, Hive, Yarn, Pig, Flume, Zookeeper, Sqoop, UDB, Tableau, AWS, GitHub, Shell Scripting.

Confidential, NYC

Big Data Engineer

Responsibilities:

• Responsible for ingesting large volumes of user behavioral data and customer profile data to Analytics Data store.
• Sqoop jobs for ingesting from FTP servers into databases.
• Developed Scala based Spark applications for performing data cleansing, event enrichment, data aggregation de-normalization and data preparation needed for machine learning and reporting teams to consume.
• Worked on troubleshooting spark application to make them more error tolerant.
• Worked on fine-tuning spark applications to improve teh overall processing time for teh pipelines.
• Experienced in handling large datasets using Spark in Memory capabilities, using broadcasts variables in Spark, effective & efficient Joins, transformations and other capabilities.
• Experience developing pipelines in GCP and Azure.
• Involved in creating Terraform and Teamcity builds.
• Developed automation pipelines in ADF, Airflow and Atomic.
• Deprecated Oracle and Netezza jobs and transformed teh code to teh new stack.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Collaborated with teh infrastructure, network, database, application and BA teams to ensure data quality and availability.
• Designed, documented operational problems by following standards and procedures using JIRA.
• Developed custom multi-threaded python-based jobs

Environment: Spark, Python, Hive, HDFS, Sqoop, Pig, Oozie, MongoDB, Cloudera, GCP, Azure