SUMMARY

• Big data developer with around 7 years of professional IT experience with expertise in Hadoop ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems spanning Big Data.
• A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
• Expertise in Apache Hadoop ecosystem components like Spark, Hadoop Distributed File Systems (HDFS), Hive, MapReduce, Hive, Sqoop, HBase, Airflow, YARN, Python, PySpark, SQL and AWS Ecosystem.
• Hands on experience in creating real - time data streaming solutions using Apache Spark core, Spark SQL & DataFrames, Kinesis, Spark Streaming and Apache Storm.
• Excellent knowledge of Hadoop architecture and daemons of Hadoop clusters, which include Name node, Data node, Resource manager, Node Manager and Job history server.
• Worked on both Cloudera and Horton works in Hadoop Distributions. Experience in managing Hadoop Clusters using the Cloudera Manager tool.
• Well versed in installation, Configuration, Managing of Big Data and underlying infrastructure of Hadoop Cluster.
• Hands on experience in coding MapReduce/Yarn Programs using Pyspark and Python for analyzing Big Data.
• Exposure to Cloudera development environment and management using Cloudera Manager.
• Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle.
• Implemented Spark using PYTHON and utilizing Data frames and Spark SQL API for faster processing of data and handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
• Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
• Working on creating data pipeline for different events of ingestion, aggregation, and load consumer response data into Hive external tables in HDFS location to serve as feed for tableau dashboards.
• Strong Hadoop and platform support experience with all the entire suite of tools and services in major Hadoop Distributions - Cloudera, Hortonworks, Amazon EMR, Azure HDInsight.
• Used Kerberos, Azure AD, Sentry, and Ranger for maintaining security. Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
• Experience in Microsoft cloud and setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud.
• Extensively worked on Spark using Python on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
• Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.
• Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
• Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions.
• Experience in creating Data frames using PySpark and performing operation on the Data frames using Python.
• In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS and MapReduce Programming Paradigm, High Availability and YARN architecture.
• Establishing multiple connections to different Redshift clusters (Bank Prod, Card Prod, SBBDA Cluster) and provide the access for pulling the information we need for analysis.
• Generated various kinds of knowledge reports using Power BI based on Business specification.
• Developed interactive Tableau dashboards to provide a clear understanding of industry specific KPIs using quick filters and parameters to handle them more efficiently.
• Well Experience in projects using JIRA, Testing, Maven and Jenkins build tools.
• Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3,), focusing on high-availability, fault tolerance, and auto-scaling.
• Good experience with use-case development, with Software methodologies like Agile and Waterfall.
• Working knowledge of Amazon's Elastic Cloud Compute(EC2) infrastructure for computational tasks and Simple Storage Service ( S3 ) as Storage mechanism.
• Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark .
• Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
• Hands on experience in Hadoop Big data technology working on MapReduce, Pig, Hive as Analysis tool, Sqoop and Flume data import/export tools.

TECHNICAL SKILLS

Big Data Eco System: HDFS, MapReduce, Hive, Pig, Sqoop, Flume, HBase, Kafka Connect, Impala, Stream sets, Oozie, Airflow, Zookeeper, NiFi, Amazon Web Services.

Hadoop Distributions: Apache Hadoop 1x/2x, Cloudera CDP, Hortonworks HDP

Languages: Python, Scala, Pyspark, Pig Latin, HiveQL, Shell Scripting.

Software Methodologies: Agile, SDLC Waterfall.

Design Patterns: Eclipse, Net Beans, IntelliJ, Spring Tool Suite.

Databases: Teradata,Oracle,PostgreSQL,DynamoDB, Hbase, Snowflake,Redshift.

NoSQL: HBase, MongoDB, Cassandra.

Version control: GIT, SVN, Bitbucket.

Operating Systems: Windows (XP/7/8/10), Linux (Unix, Ubuntu), Mac OS.

Cloud Technologies: Amazon Web Services, EC2, S3, SQS, SNS, Lambda, EMR, Code Build, CloudWatch. Azure HDInsight (Databricks, DataLake, Blob Storage, Data Factory, SQL DB, SQL DWH, CosmosDB, Azure DevOps, Active Directory).

PROFESSIONAL EXPERIENCE

Confidential, Atlanta, GA

Sr. AWS BigData Engineer

Responsibilities:

• Implemented simple to complex transformation on Streaming Data and Datasets. Worked on analyzing Hadoop cluster and different big data analytic tools including Hive, Spark, Python, Sqoop, flume, Oozie.
• Developed Spark Streaming by consuming static and streaming data from different sources.
• Used Spark Streaming to stream data from external sources using AWS Kinesis and responsible for migrating the code base from Cloudera Platform to Amazon EMR and evaluated Amazon eco systems components like RedShift, Dynamo DB. Having good knowledge in NOSQL databases like Dynamo DB, Hbase, Cassandra.
• Setting up and administering DNS system in AWS cloud using Route53.
• Performed configuration, deployment and support of cloud services in Amazon Web Services (AWS).
• Experienced working on cloud AWS using EMR Performed operations on AWS using EC2 instances, S3 storage, performed RDS, analytical Redshift operations and Wrote various data normalization jobs for new data ingested into Redshift by building multi-terabyte of data frame.
• Created action, filters, parameters and calculated sets for dashboards and worksheets in Tableau. Created dual axis chart to show portfolio performance over indexes, used Tree map and scattered chart, calculated fields, Bins, Table calculations, line for visualization.
• Defined best practices for creating Tableau dashboards by matching requirements to the charts to be chosen, color patterns as per user's needs, standardizing dashboard's size, look and feel etc.
• Expert in developing Enterprise BI applications using Tableau Desktop, Actuate, Jasper i-Report and Studio, Business Object, SQL SERVER, Oracle and PL/SQL.
• Developed Oozie workflows for scheduling and orchestrating the ETL process. Involved in writing Python scripts to automate the process of extracting weblogs using Airflow DAGs.
• Worked on creating data pipelines with Airflow to schedule PySpark jobs for performing incremental loads and used Flume for weblog server data. Created Airflow Scheduling scripts in Python.
• Involved in migrating Oozie workflows to Airflow in order to automate data pipelines to extract data and weblogs from DynamoDB and Oracle databases.
• Built and configured a virtual data center in the Amazon Web Services cloud to support Enterprise Data Warehouse hosting including Virtual Private Cloud, Security Groups, Elastic Load Balancer.
• Expert in implementing advanced procedures like text analytics and processing using the in-memory computing capabilities likeApacheSparkwritten inScala. Expertized in implementing Spark usingScalaandSpark SQLfor faster testing and processing of data responsible to manage data from different sources.
• Implemented data ingestion and handling clusters in real time processing using Kinesis.
• Developed a data pipeline using Kafka and Spark Streaming to store data into HDFS.
• Provide all data validation through SQL queries UNIX commands to perform Back-end testing. Performed different levels of testing like Integration testing and System testing.
• Performing Backend testing by creating and executing SQL queries and also Performed Database testing using SQL queries to check the Data Validation and Data Integrity.
• Responsible for data extraction and data integration from different data sources into Hadoop Data Lake by creating ETL pipelines Using Spark, MapReduce, Pig, and Hive.
• Developed Spark programs with Scala and applied principles of functional programming to process the complex unstructured and structured data sets. Processed the data with Spark from Hadoop Distributed File System.
• Design and construct AWS Data pipelines using various resources in AWS like Lambda, SQS, SNS, S3 and EMR by receiving event notifications from AWS S3 and write it to Postgres.
• Developing Informatica Cloud Jobs to integrate legacy ESB (Enterprise System for Bankers) with Snowflake Cloud Datawarehouse. Implemented OLAP multi-dimensional cube functionality using SQL Data Warehouse.
• Used Scala sbt to develop Scala coded spark projects and executed using spark-submit.
• Collaborated with Architects to design Spark model for the existing MapReduce model and migrated them to Spark models using Scala. Worked on writing Scala Programs using Spark-SQL in performing aggregations.
• Developed Web Services in play framework using Scala in building stream data Platform.
• Worked with Apache Spark which provides fast engine for large data processing integrated with Scala.
• Experience working with SparkSQL and creating RDD's using PySpark. Extensive experience working with ETL of large datasets using PySpark in Spark on HDFS. Data Extraction of Adobe data within AWS Glue using PySpark.
• Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift. Was responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark.
• Created monitors, alarms, notifications and logs for Lambda functions, Glue Jobs, EC2 hosts using Cloudwatch
• Used AWS glue catalog with crawler to get the data from S3 and perform sql query operations.
• Developed and deployed Spark application using Pyspark to compute popularity score for all the contents using an algorithm and load the data into Elastic Search for App content management team to consume.
• Developed a Data flow to pull the data from the REST API using Apache Nifi with context configuration enabled and developed entire spark applications in Python (PySpark) on distributed environment. Implemented Micro Services architecture using spring boot framework.
• Used the PC reuse utility to understand the BDM Compatibility to reuse some of the PC ETL Code onto the Informatica Developer. Convert the Power Center code using the informatica developer to BDM Mappings.
• Created pipeline that uses Terraform Cloud to deploy applications from an Amazon S3 bucket for the flow of services in AWS. Written AWS Lambda code in Python for starting EMR clusters which in turn runs Spark Steps.
• Developed Elastic Search Connector using Kafka Connect API with source as Kafka and sink as elastic search.

Sr.Azure BigData Developer

Responsibilities:

• Hands on experience in installing, configuring, and using Hadoop ecosystem components like Hadoop MapReduce, HDFS, HBase, Hive, Spark, Sqoop, Pig, Zookeeper and Flume.
• Data warehouse, Business Intelligence architecture design and develop. Designed the ETL process from various sources into Hadoop/HDFS for analysis and further processing of data modules.
• Responsible for validation of Targetdata in Data Warehouse which are Transformed, Loaded using Hadoop Big.
• Design, develop, test, implement and support of Data Warehousing ETL using Talend and Hadoop Technologies.
• Extensively worked with MySQL for identifying required tables and views to export into HDFS.
• Involved in designing of HDFS storage to have efficient number of block replicas of Data.
• Involved in troubleshooting and performance tuning of reports and resolving issues within Tableau server and generated reports. Developed Tableau data visualization using Cross tabs, Heat maps, Box and Whisker charts, Scatter Plots, Geographic Map, Pie Charts and Bar Charts and Density Chart.
• Responsible for creating Hive tables on top of HDFS and developed Hive Queries to analyze the data.
• Staged data by persisting to Hive and connected Tableau with Spark cluster and developed dashboards.
• Written Stored Procedures using PL/SQL and functions and procedure for common utilities using MySQL.
• Automated the process for extraction of data from warehouses and weblogs by developing workflows and coordinated interdependent Hadoop jobs using Airflow.
• Developed ETL pipelines in Spark using python and workflows using Airflow and also automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.
• Experience in working with Azure cloud platform (HDInsight, Databricks, DataLake, Blob, Data Factory, SQL DB, SQL DWH).Performed data cleansing and applied transformations using Databricks and Spark data analysis.
• Designed and automated Custom-built input adapters using Spark, Sqoop and Oozie to ingest and analyze data from RDBMS to Azure Datalake.
• Designed and Configured Azure Cloud relational servers and databases analyzing current and future business requirements and also have experience working with Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW).
• Worked on migration of data from On-prem SQL server to Cloud databases (Azure Synapse Analytics (DW) & Azure SQL DB). Developed business intelligence solutions using SQL server data tools versions and load data to SQL & Azure Cloud databases.
• Have extensive experience in creating pipeline jobs, scheduling triggers, Mapping data flows using Azure Data Factory(V2) and using Key Vaults to store credentials.
• Worked on creating tabular models on Azure analysis services for meeting business reporting requirements.
• Created Automated ETL jobs in Talend and pushed the data to Azure SQL data warehouse.
• Collected LMS data into Hadoop cluster using SQOOP. Imported and exported data into HDFS and Hive/Impala tables from Relational Database Systems using Sqoop.
• Importing and exporting structured data from different relational databases into HDFS and Hive using Sqoop.
• Involved in importing and exporting data from local and external file system and RDBMS to HDFS.
• Designed and Developed jobs that handles the Initial load and the Incremental load automatically using Oozie workflow. Implemented workflows using Apache Oozie framework to automate tasks.
• Distributed Tableau reports using techniques like Packaged Workbooks, PDF to different user community.
• Developed various pipelines for Mastercard and Visa card data Integration and continuous stream of data to databases and one of the organizations developed Kafka topics using Stream sets Data collector.
• Provide expertise in Kafka brokers, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center. Used Kafka Streaming for data ingestion and cluster handling in real time processing.
• Designed the test scripts and analyzed the results in various levels of the application including integration, regression, functional and acceptance testing using Selenium.
• Used Azure Synapse to manage processing workloads and served data for BI and prediction needs.
• Managed resources and scheduling across the cluster using Azure Kubernetes Service.
• Responsible for using GIT for version control to commit the code developed which further used for deployment using build and release tool Jenkins. Developed CI/CD system with Jenkins on Kubernetes container environment, utilizing Kubernetes and Docker for the CI/CD system to build, test and deploy.
• Worked on implementation and maintenance of Cloudera Hadoop cluster. Experience in working with Cloudera (CDH4 &CDH5), Horton Works, Amazon EMR, Azure HDINSIGHT on multi-node cluster.
• Integrated Oozie with Pig, Hive, Sqoop and developed Oozie workflow for scheduling and orchestrating the Extract, Transform, and Load (ETL) process within the Cloudera Hadoop.
• Involved in Informatica BDM cluster creation, testing connections, importing objects using Hive connection.
• Pulling the data from data lake (HDFS) and massaging the data with various RDD transformations.
• Wrote Pig Scripts to generate Map Reduce jobs and performed ETL procedures on the data in HDFS.
• Migration of code to QA and PROD which includes UNIX shell scripts, source files, and Informatica BDM mapping and parameter files.

Confidential

Sr. Data Engineer

Responsibilities:

• Importing and exporting structured data from different relational databases into HDFS and Hive using Sqoop.
• Experienced in handling large Datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
• Worked with data analysis and visualizing Big Data in Tableau with Spark.
• Developed Spark streaming model which gets transactional data as input from multiple sources and create multiple batches and later processed for already trained fraud detection model and error records.
• Developed Terraform scripts to create the AWS resources such as EC2, Auto Scaling Groups, ELB, Route53, S3, SNS and Cloud Watch Alarms. Developed scripts for loading application call logs to S3.
• Performed Installation and configuration of multi-node cluster on Cloud using Amazon Web Services on EC2.
• Involved in reviewing business requirements and analyzing data sources from Excel, SQL Server for design, development, testing, and produce report and analyze projects within Tableau Desktop.
• Efficient Data migration from various relational Data platforms to Hadoop and building Data warehouse on Hadoop ecosystems such as Hive, Oozie and Sqoop. Worked on loading the data into the cluster from dynamically generated files using Flume and sent the cluster to Relational database management systems using SQOOP.
• Involved in creating Hive tables and working on them using HiveQL and perform data analysis using Hive and Pig.
• Expertise in publishing Power BI reports of dashboards in Power BI server and scheduling the dataset to refresh for live data in Power BI server. Experience with Tableau and Power BI on publishing of created visualizations, dashboards, and workbooks from Tableau Desktop to Tableau Servers, and reports using SSRS.
• Performed Power BI Desktop Data modeling, which cleans, transforms, mash up Data from multiple sources.
• Wrote, tested and implemented Teradata Fastload, Multiload and Bteq scripts
• Created the mainframe datasets and jobs for the BTEQ,FLOAD,MLOAD scripts
• Developed MLoad scripts and shell scripts to move data from source systems to staging and from staging to Data warehouse in batch processing mode.
• Accomplished data movement process that load data from DB2 into Teradata by the development of Korn Shell scripts, using Teradata SQL and utilities such as BTEQ, FASTLOAD, FASTEXPORT, MULTILOAD, and improved the design of Extract and Load specifications.
• Used Pig as ETL tool to do transformations, event joins and pre-aggregations before storing the data onto HDFS.
• Leverage Hadoop ecosystem to design and develop capabilities to deliver our solutions using Spark, Scala, Python.
• Solved performance issues in Hive with understanding of joins, Groups, and aggregation and how does it translate to MapReduce jobs, Control M, Informatica. Extensive experience in working with structured Data using Hive QL, join operations, writing custom UDF's and experienced in optimizing Hive Queries.
• Migration of Teradata and DB2 to Snowflake database using AWS and AWS resources.
• Involved in moving data from HDFS to AWS Simple Storage Service and worked with S3 bucket in AWS.
• Worked on AWS Elastic load balancing for deploying applications in high availability and AWS Auto Scaling for providing high availability of applications and EC2 instances based on the load of applications.
• Developed spark jobs for continuous integration of Error records (read, write and error count) which will pull logs from Kafka Topic to MySQL server Tables in an orderly manner as required.
• Developed the batch scripts to fetch the data from AWS S3 storage and do required transformations in Scala using Spark framework. Used Spark processing framework such as Spark SQL, Data Warehousing and ETL processes.
• Designed a custom referential integrity framework on the No SQL Cassandra tables for maintaining data integrity and relations in the data. Developed multiple spark batch jobs using Spark SQL and performed transformations using many APIs and updated master data in Cassandra database as per the business requirement.

Confidential

Responsibilities:

• Used Informatica as an ETL tool to create source/target definitions, mappings, and sessions to extract, transform and load data into staging tables from various sources.
• Analyzing, designing, and developing ETL strategies and processes, writing ETL specifications, Informatica development, and administration.
• Designed and Developed Informatica processes to extract data from internal check issue systems.
• Used Informatica Power exchange to extract data from one of the EIC s operational system called Datacom.
• Extensive experience in Building, publishing customized interactive reports and dashboards, report scheduling using Tableau Desktop and Tableau Server.
• Extensive experience in Tableau Administration Tool, Tableau Interactive Dashboards, Tableau suite.
• Developed Tableau visualizations and dashboards using Tableau Desktop and published the same on Tableau Server.
• Worked extensively in creating dashboards using Tableau that includes tools like Tableau Desktop, Tableau Server and Tableau Reader in various versions of Tableau 9.0, 8.2and 8.1. Also involved in the administration of Tableau server like installations, upgrades, user, user groups creation and setting up security features.
• Involved in Installation and upgrade of Tableau server and server performance tuning for optimization.
• Used Informatica Power Center for (ETL) extraction, transformation and loading data from heterogeneous source systems into target database.
• Used debugger in Informatica Designer to resolve the issues regarding data thus reducing project delay.
• Designed high-level view of the current state of dealer operation, leads, and website activity using Tableau.
• Performed various types of joins in Tableau for demonstrating integrated data purpose and validated data integrity to examine the feasibility of discussed visualization design.
• Leveraged advanced features of tableau such as calculated fields, parameters, and sets to support data analysis and data mining.
• Developed wrapper shell scripts for calling Informatica workflows using PMCMD command and Created shell scripts to fine tune the ETL flow of the Informatica workflows.
• Worked as ETL developer and Tableau developer and widely involved in Designing, development and debugging of ETL mappings using Informatica designer tool as well as Created advanced chart types, visualizations, and complex calculations to manipulate the data using Tableau Desktop.
• Extensively worked on Informatica B2B Data Exchange Setup from Endpoint creation, Scheduler, Partner setup, Profile setup, Event attributes creation, Event status creation, etc.
• Used informatica to parse out the xml data into the DataMart structures that is further utilized for the reporting needs.
• Utilized Informatica PowerCenter to accomplish full phases of data flow from source data (Oracle, SQL Server, flat files) being analyzed before extracted to transformation.
• Used Custom SQL feature on Tableau Desktop to create overly complex and performance optimized dashboards.
• Connected Tableau to various databases and performed Live data connections, query auto updates on data refresh etc.