SUMMARY:

• Over 9+ years of experience in Designing, Developing, and integrating applications using Hadoop, Hive, PIG, in all three Bigdata platforms Cloudera, Hortonworks, MapReduce, Snowflake, Apache Airflow in which 5 years of experience on Data Engineering and 4years of experience on Data Warehouse.
• Experience writing pig and hive scripts.
• Experience in writing Map Reduce programs using Apache Hadoop for analyzing Big Data.
• Hands on experience in writing Ad - hoc Queries for moving data from HDFS to HIVE and analyzing the data using HIVE QL.
• In-depth knowledge of Hadoop Architecture and Hadoop daemons such as Name Node, Secondary Name Node, Data Node, Job Tracker, and Task Tracker.
• Good working knowledge of Snowflake and Teradata databases.
• Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle/Snowflake.
• Expertise in Python data extraction and data manipulation, and widely used python libraries like NumPy, Pandas, and Matplotlib for data analysis.
• Extensively worked on other machine learning libraries such as Seaborn, Scikit learn for machine learning and familiar working with TensorFlow, NLTK for deep learning
• Experienced in creating SnowFlake Multi-cluster Size and Credit Usage.
• Played a key role in Migrating Teradata objects into the SnowFlake environment.
• Experience with Snowflake Multi-Cluster Warehouses
• Experience with Snowflake Virtual Warehouses.
• Having In-depth knowledge of Data Sharing in Snowflake.
• Have a knowledge of Snowflake Database, Schema and Table structures.
• Experience in using Snowflake Clone and Time Travel.
• Solid understanding and experience with extract, transform, load
• Implementing data movement from File system to Azure Blob storage using python API
• Implemented PoC for running ML models on Azure ML studio
• Written Kafka consumer Topic to move data from adobe clickstr

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Data Engineer

Responsibilities:

• Worked on Distributed/Cloud Computing (Map Reduce/Hadoop, Hive, Pig, HBase, SQOOP, Flume, Spark, AVRO, Zookeeper etc.), Handling raw data from various subsystems and load the data from different subsystems into to HDFS for further processing. Developed Apache Pig scripts and UDF's extensively for data transformations and calculating Statement date formats and aggregates the monitory transactions. Integrated Apache Kafka for data ingestion. Created Hive tables to store the processed results in a tabular format. Developed the SQOOP scripts to make the interaction between Pig and MySQL
• Database. Involved in Migrating Objects from Teradata to Snowflake. Created Snowpipe for continuous data load. Developed data warehouse model in snowflake for over 100 datasets using WhereScape. Heavily involved in testing Snowflake to understand best possible way to use the cloud resources. Implemented various Machine learning algorithms to analyze market trends and its effects on the company. Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive querying. Implemented performance - tuning techniques along various stages of the Migration process.
• Developed Hive Scripts for implementing dynamic Partitions. Experienced in handling different optimization join operations like Map join, Sorted Bucketed Map join etc. Implemented Partitioning, Dynamic Partitions and Buckets in Hive for analytical processing by business users. Expertise in performance tuning on Hive Queries, joins and different configuration parameters to improve query response time. Completely involved in the requirement analysis phase. Implemented POC writing programs in Scala and data processed using Spark-SQL. Conducted POC for Hadoop and Spark as part of Next-Gen platform implementation. Implemented recommendation engine using Scala. Setting up cron job to delete Hadoop logs/local old job files/cluster temp files. Monitor System health and logs and respond accordingly to any warning or failure conditions. Tested raw data and executed performance scripts. Setup Hive with MySQL as a Remote metastore. Exported analyzed data to relational databases using Sqoop for visualization to generate reports for the BI team. Moved all log/text files generated by various products into HDFS location. Implemented POC for using APACHE IMPALA for data processing on top of HIVE. Streamlined Hadoop jobs and workflow operations using Oozie workflow. Experienced in writing unit test cases, implement unit test cases using JUnit. Worked with testing teams and resolved defects.

Environment: Hadoop (CDH5), UNIX, Scala, Python, Storm, Databricks, Azure Spark, Spark-SQL, MapReduce, Apache Pig, Hive, Impala, SQOOP, Java, Eclipse, Kafka, MySQL and Oozie.

Confidential

Data Engineer

Responsibilities:

• Developed Spark applications using Scala. Performance analysis of batch jobs by using Spark Tuning parameters. Enhanced and optimized Spark/Scala/ Pyspark jobs to aggregate, group and run data mining tasks using the Spark framework. Worked on aws tools like Kinesis, DynamoDB, and S3. Importing and exporting data into HDFS and hive using Sqoop and Kafka with batch and streaming. Worked on MySQL RDBMs db as backend database to store monitoring information about CCPA project. Used Service now platform to open and close tickets of CCPA project. Involved in complete Big Data flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data using several tools. Imported the data from various formats like JSON, ORC and Parquet to HDFS cluster with compressed for optimization. Experienced on ingesting data from RDBMS sources like -
• Oracle, SQL Server and Teradata into HDFS using Sqoop. Deployed Pyspark applications and developed in Databricks cluster. Experience in managing and reviewing huge Hadoop log files. Importing and exporting data into HDFS and hive using Sqoop and Kafka with batch and streaming. Experienced with Spark-Streaming APIs to perform transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and persists into HBase. Performance analysis of Spark streaming and batch jobs by using Spark tuning parameters. Enhanced and optimized Spark/python jobs to aggregate, group and run data mining tasks using the Spark framework. Installed, configured, and developed various pipeline activities with Apache Nifi using various processors such as Sqoop processor, Kafka processor, HDFS Processor, File Processors etc. Created Data Pipelines as per the business requirements and scheduled it using Oozie. Used Hive to join multiple tables of a source system and load them to Elastic search tables. Involved in complete Big Data flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data using several tools. Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO, and Parquet to HDFS cluster with compressed for optimization. Experienced on ingesting data from RDBMS sources like - Oracle, SQL Server, and Teradata into HDFS using Sqoop. Configured Hive and written
• Hive UDF's and UDAF's Also, created partitions such as Static and Dynamic with bucketing. Experience in managing and reviewing huge Hadoop log files. Expertise in designing and creating various analytical reports and Automated Dashboards to help users to identify critical KPIs and facilitate strategic planning in the organization. Plus 4 months experience indexing data into Elastic search and built dashboards with Kibana and used Kibana as search tool on elastic search. Experience in CI/CD tool Jenkins for code deplo

Confidential

Data Engineer

Responsibilities:

• Responsible for requirements gathering and analyzing the data sources. Responsible for building scalable distributed data solutions using Hadoop. Worked very closely with the Hadoop Administrator to set up the Hadoop cluster. Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive HBase database and SQOOP. Importing and exporting data into HDFS and Hive using SQOOP. Installed Hadoop, Map Reduce, HDFS, and Developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre - processing. Generate OBIEE reports to verify the Hive table's data.
• Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS. Experience working in spark eco system using Spark-Sql and Scala queries on different formats like Text file, CSV file. Involved in loading data from LINUX file system to HDFS. Load and transform large sets of structured and semi structured data. Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose. Created Hive tables to store variable data formats of PII data coming from different portfolios. Implemented a script to transmit sysprin information from Oracle to Hive using Sqoop. Designed and developed MapReduce jobs to analyze the data. Implemented best income logic using Pig scripts and UDFs. Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required. Responsible to manage data coming from different sources. Involved in loading and transforming large sets of structured, semi structured, and unstructured data. Cluster coordination services through Zookeeper. Developed Spark scripts by using Scala Shell commands as per the requirement. Experience in managing and reviewing
• Hadoop log files. Responsible for cluster maintenance, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files. Experience using Amazon Web Services. Installed Oozie workflow engine to run multiple Hive and pig jobs. Analyzed large amounts of data sets to determine optimal way to aggregate and report on it. Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop. Involved in email campaign launch and served as point of contact for anything related to data, pulling customers list, matching and finished in Strong View email platform software.

Environment: MapReduce, Hadoop MapReduce, HDFS, Spark, Hive, Pig, SQL, Sqoop, Flume, Oozie, Java 8, Eclipse HBase, Shell Scripting, Scala.

Confidential

Data Engineer

Responsibilities:

• Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources. Data sources are extracted, transformed, and loaded to generate CSV data files with Python programming and SQL queries. Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself. Designed, developed, and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non - traditional source systes as well as RDBMS and NoSQL data stores for data access and analysis. Worked on a POC to compare processing time of Impala with Apache Hive for batch applications to implement the former in the project. Worked on Cluster of size 130 nodes. Worked extensively with Sqoop for importing metadata from Oracle.
• Analyzed the SQL scripts and designed the solution to implement using Pyspark. Responsible for developing data pipelines with Amazon AWS to extract the data from weblogs and store in HDFS. Involved in creating Hive tables and loading and analyzing data using hive queries. Developed Hive queries to process the data and generate the data cubes for visualizing. Implemented schema extraction for Parquet and Avro file Formats in Hive. Implemented Partitioning, Dynamic Partitions, Buckets in HIVE. Used Reporting tools like Tableau to connect with Hive for generating daily reports of data. Collaborated with the infrastructure, network, database, application, and BI teams to ensure data quality and availability

Environment: MapReduce, Hadoop MapReduce, HDFS, Spark, Hive, Pig, SQL, Sqoop, Flume, Oozie, Java 8, Eclipse HBase, Shell Scripting, Scala.

Confidential

Datawarehouse Developer

Responsibilities:

• Created POC on big data platform to derive the insights of historical trading data. Performed data extraction from RDBMS to HDFS system using Sqoop. Involved in designing data model on big data platform. Interacting with business group to understand the requirements to create analytical platform. Identifying the required data points to create analytical solutions for business. Independently perform moderately to highly complex ETL development tasks and assignments. Using Informatica Power Center and/or Informatica Data Services. Transforming XML data using Informatica Power Center. Responsible for Requirement Gathering Analysis and End user Meetings. Responsible for mentoring Developers and Code Review of Mappings developed by other developers. Highly proficient in Development, Implementation, Administration and Support of ETL processes for Large - scale Data warehouses using Informatica
• Power Center. Loaded (ETL) data from various sources into various targets using Informatica Power Center (Repository Manager, Designer and Workflow Manager) Responsible for tuning mappings, identifying and resolving performance bottle- necks in various levels such as sources, targets, mappings and sessions. Worked on data extraction, transformation and loading with RDBMS and flat files. Designed ETL specification documents. Responsible for Unit Testing and Integration testing of mappings and workflows.

Environment: Cloudera HDP, Linux, Hive, YARN, Oracle, Informatica Power Center, Sybase, UNIX Scripting, Selenium, Maven, Eclipse, TOAD