SUMMARY:

• More TEMPthan 8 years of professional experience in information technology wif an expert hand in teh areas of BIG DATA, HADOOP, SPARK, HIVE, IMPALA, SQOOP, FLUME, KAFKA, SQL tuning, ETL development, report development, database development, data modelling and strong knowledge of oracle database architecture.
• Extensive experience in developing and designing data integration solutions using ETL tool such as Informatica Powercenter, Teradata Utilities for handling large volumes of data.
• Well knowledge and experience in Cloudera ecosystem (HDFS, YARN, Hive, SQOOP, FLUME, HBASE, Oozie, Kafka, Pig), Data pipeline, data analysis and processing wif hive SQL, IMPALA, SPARK, SPARK SQL.
• Extensive experience in Data Mining solutions to various business problems and generating data visualizations using Tableau, PowerBI, Birst, Alteryx.
• Has good Programming experience wif Python and Scala.
• Worked extensively on AWS Components such as Airflow, Elastic Map Reduce (EMR), Atana, and Snowflake.
• Using Flume, Kafka and Spark streaming to ingest real time or near real time data in HDFS.
• Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in AWS and coordinate task among teh team.
• Contributed to teh development of framework used to develop Hive, MapReduce and PySpark as well as metadata management on Hadoop clusters to create external tables and reconcile data at different stages using Hive.
• Worked on Python Open stack API's and used Python scripts to update content in teh database and manipulate files.
• Experience in DWBI across verticals such as: Games, Sales, Online, Online Marketing, Social Media Analytics and ecommerce.
• Hands in experience on No SQL database like HBase, Cassandra.
• Proven design and development background in ETL, DW, BI and data migration projects.
• Analyzing teh way to migrate oracle database to redshift.
• Experience in creating dashboards

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Big Data Developer

Responsibilities:

• Developed data pipeline using Spark, Hive, Pig, python, Impala and HBase to ingest customer behavioral data and financial histories into Hadoop cluster for analysis. Perform Informatica Cloud Services, Informatica Power Center Administration ETL strategies and ETL Informatica mapping. Setting up of Secure Agent and connect different applications and its Data Connectors for processing teh different kinds of data including unstructured (logs, click streams, Shares, likes, topics etc.), semi structured (XML, JSON) and structured like RDBMS. Install and configured splunk Enterprise environment on Linux,
• Configured Universal and Heavy forwarder. Extensive data knowledge in supply chain, Services, Ecommerce, Retail, Banking and Media domains. Using rest API wif Python to ingest Data from and some other site to BIGQUERY. Build a program wif Python and apache beam and execute it in cloud Dataflow to run Data validation between raw source file and big query tables. Developed Microsoft SQL Server / MySQL / Maria Db SQL Procedures for automating large complex data migration ETL processes. Teh information was refreshed and maintained automatically using Linux Bash scripts and SQL procedures. Create and Manage Private Lab wif Dell PowerEdge and AWS to host Splunk Clustered Environment. Involved in developing a framework for metadata management on HDFS data lakes. Develop complex SQL queries, scripts, user defined functions, views, and triggers for business logic implementation in
• Snowflake Enterprise Data Warehouse. Data Migration from existing Teradata systems to HDFS and build datasets on top of it. Develop and own ETL Data Pipeline and Data Model solutions for integrating new data sets into teh Snowflake Enterprise Data Warehouse Deployment of Cloud service including Jenkins and Nexus on Docker using Terraform. Developed PySpark scripts by using Scala Shell commands as per teh requirement. Developed Spark/Scala scripts, UDFs using both Data frames/SQL and RDD in Spark for Data Aggregation, queries and writing data back into AWS S3. Implemented consolidation of CDW module using Spark, Scala, Python and AWS S3, EC2 Experience in Orchestrating and scheduling workflows using Crontab, Oozie and Airflow. Development using ETL tools like DataStage or Informatica Worked experience on architecting teh ETL transformation layers and writing spark jobs to do teh processing Analyzed data and provided insights wif R Programming and Python Pandas. Worked on AWS Data Pipeline to configure data loads from S3 to into Redshift. Configured teh cluster to achieve teh optimal results by fine - tuning teh cluster using Cloudera distribution. Involved in converting Map Reduce programs into Spark transformations using Spark RDD's and Scala. Provide solutions for data quality operations and Informatica ETL Processes to support Data Integration and Reporting requirements. Designed in implementing teh Name Node

Confidential

Big data Hadoop develope r

Responsibilities:

• Teh project is mainly focused on customer's data, migrating to AWS cloud from Teradata, snowflake, and internal databases. We built a pipeline (True Source) to handle huge amount of data which comes through our internal and external vendors were pushed to load directly to teh cloud. Built AWS infrastructure, in such a way that NIFI, KYLO, EMR which work seamlessly in contrast wif each other. Hands on writing code in Python and Pyspark to manipulate data for data loads, extracts, statistical analysis, modeling, and data validation. Developed streaming & batch applications using Apache spark and
• Python. Worked on NiFi data Pipeline to process large set of data and configured Lookup's for Data Validation and Integrity. Used Scala to convert Hive/SQL queries into RDD transformations in Apache Spark. Installing, migrating and upgrading multiple MapR clusters. Recommendation System: Built a recommendation engine using BigQuery ML to suggest new music artists to follow using Collaborating filtering. dis helped in increase in Jam Feed premium subscription Worked on hive data warehouse modeling to interface wif BI tools such as Tableau. Worked wif Hive data warehouse to HDFS to identify issues and behavioral patterns and Configured and enable services for Microsoft R Server for analytical team. Modeled structures for Teradata, Oracle, BigQuery, and Spanner. Implemented erwin forward engineering templates to create DDL for BigQuery and Spanner. To manage and store teh data we are working on cloud - based technology like AWS which includes all teh services from AWS like Ec2 instances, EMR and S3 services. Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive and involved in creating Hive Tables, loading wif data and writing Hive queries which will invoke and run MapReduce jobs in teh backend. Responsible for importing real time data to pull teh data from sources to Kafka clusters. Developed code to write canonical model JSON records from numerous input sources to Kafka queues. Worked extensively on building Nifi data pipelines in Docker container environment in development phase.
• Developed Oozie workflow for scheduling & orchestrating teh ETL process. Developed Pig Scripts, Pig UDF's and Hive Scripts, Hive UDF's to load data files. Worked extensively wif importing metadata into Hive using Python and migrated existing tables and applications to work on AWS cloud (S3). Backing up AWS Postgres to S3 on daily job run on EMR using Data Frames. Analyzed teh SQL scripts and designed teh solution to implement using PySpark. I was mainly part of doing true sourcing teh data through our pipeline includes various techniques to handle teh data which are mainly Data quality checks, schema validation, row count validation, data standardization and data lineage. Developed Python and Java scripts to manage AWS resources from API calls using BOTO3 SDK and worked wif AWS CLI. Cre

Confidential

Hadoop Developer

Responsibilities:

• Implemented solutions for ingesting data from various sources and processing teh Data-at-Rest utilizing Big Data technologiesSuch as Hadoop, Map Reduce, HBase, and Hive. Provisioned Azure data lake store and azure data lake analytics, and leverage U-SQL to write federated queries across data stored in multiple azure services. Worked on Software Development Life Cycle (SDLC), testing methodologies, resource management and scheduling of tasks. Developed data pipeline using Pig, Sqoop to ingest cargo data and customer histories into HDFS for analysis. Worked on Spark using Python and Spark SQL for faster testing and processing of data. Involved in running Hadoop streaming jobs to process terabytes of data. Migrated Snowflake database to Windows Azure and updating teh Connection Strings based on requirement. Designed and developed standalone data migration applications to retrieve and populate data from Azure Table / BLOB storage to on premise SQL Server instances. Experienced in Apache Spark for implementing advanced procedures like text analytics and processing using teh in-memory computing capabilities written in Scala. Created pipelines and schedule activities using Azure Data
• Factory. Developed Json Scripts for deploying teh Pipeline in Azure Data Factory (ADF) that process teh data using teh Cosmos Activity. Implemented objects in Azure Databricks for big data analytic, mostly used SQL in notebook, occasionally Python and Scala. Developed stored procedures/views in Snowflake and use in Talend for loading Dimensions and Facts. Optimized teh Pyspark jobs to run on Kubernetes Cluster for faster data processing Developed custom Map Reduce programs for data analysis and data cleaning using pig Latin scripts. Extracted data using Sqoop Import query from multiple databases and ingest into Hive tables. Evaluated and improved application performance wif Spark. Performed land process to load data into landing tables of MDM Hub using external batch processing for initial data load in hub store. Primarily involved in Data Migration process using Azure by integrating wif
• GitHub repository and Jenkins. Created custom scripts for adding indexes to SQL Azure Tables. Designed technical solution for real-time analytics using Kafka and HBase Created post commit and pre-push hooks using Python in GIT repos. Created and Configured Azure Cosmos DB Trigger in Azure Functions, which invokes teh Azure Function when any changes are made to teh Azure Cosmos DB container. Extensively used SQL* loader to load data from flat files to teh database tables in Oracle. Implemented Spark SQL to update queries based on teh business requirements. Acquired good understanding and experience of NoSQL databases such as HBase. Used Git for version control, JIRA for project tracking and Jenkins for continuous integration. Wrote Scala and python scripts as required for spark engine.

Confidential

Hadoop Developer

Responsibilities:

• Developed highly optimized Spark applications to perform data cleansing, validation, transformation and summarization activities Data pipeline consist Spark, Hive and Sqoop and custom build Input Adapters to ingest, transform and analyze operational data. Created Spark jobs and Hive Jobs to summarize and transform data. Used Spark for interactive queries, processing of streaming data and integration wif popular NoSQL database for huge volume of data. Converted Hive/SQL queries into Spark transformations using Spark DataFrames and Scala. Used different tools for data integration wif different databases and Hadoop.

  Used Spark for interactive queries, processing of streaming data and integration wif popular NoSQL database for huge volume of data. Built real time data pipelines by developing Kafka producers and spark streaming applications for consuming. Ingested syslog messages parse them and streams teh data to Kafka. Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and tan loading data into HDFS. Exported teh analyzed data to teh relational databases using Sqoop, to further visualize and generate reports for teh BI team. Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis Analyzed teh data by performing Hive queries (Hive QL) to study customer behavior. Helped DevOps Engineers for deploying code and debug issues. Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.

  Developed Hive scripts in Hive QL to de - normalize and aggregate teh data. Scheduled and executed workflows in Oozy to run various jobs. Implemented business logic in Hive and written UDF's to process teh data for analysis. Addressing teh issues occurring due to teh huge volume of data and transitions. Designed, documented operational problems by following standards and procedures using JIRA.Environment: Spark, Scala, Hive, Apache NiFi, Kafka, HDFS, Oracle, HBase, MapReduce, Oozie, SqoopJava/J2ee SoftStandard Solutions Hyderabad, IndiaResponsibilities Involved in Analysis, design and development of web applications based on J2EE. Struts framework is used for managing teh navigation and page flow. Developed teh EJB-Session Bean acts as Facade, will be able to access teh business entities through their local home interfaces. Designed teh user interface using HTML, CSS, java Script and JQuery

  Used Log4j to debug and generate new logs for teh application. Used JDBC for accessing teh data from teh Oracle database. Created database tables, stored procedures using PL/SQL in Oracle DB. Validation on Web Forms, for client-side validation as per teh requirement. Involved in various phases of Software Development Life Cycle (SDLC) of teh application like Requirement gathering, d