SUMMARY:

• Over 6 years of professional experience as BigDataEngineerdealing with Apache
• Hadoop Ecosystem like HDFS, Map Reduce, Hive, Sqoop, Oozie, HBase, Spark - Scala,Kafka and BigDataAnalytics.
• Experience in designing, implementing large scaledatapipelines fordatacuration using Spark/DataBricks along with Python and Scala.
• Excellent understanding / noledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node,DataNode and
• Map Reduce programming paradigm.
• Expertise in developing Pig Latin Script and Hive Query Language.
• Experienced in using distributed computing architectures such as AWS products
• (EC2, Redshift, and EMR, Elastic search, Athena and Lambda), Hadoop, Python, Spark and TEMPeffective use of Map Reduce, SQL and Cassandra to solve bigdatatype problems.
• Extensive noledge in working with Azure cloud platform (HDInsight, Data Lake Databricks, Blob Storage,DataFactory, Synapse, SQL, SQL DB, DWH andDataStorage
• Explorer).
• Experience in developing CI/CD (continuous integration and continuous deployment) and automation using Jenkins, Git, docker, Kubernetes for ML models deployment.
• Expertise inDataMigration,DataProfiling,DataCleansing, Transformation Integration,DataImport, andDataExport using multiple ETL tools such as
• Informatica Power Centre.
• Experience in designing, building, and implementing complete Hadoop ecosystem comprising of Map Reduce, HDFS, Hive, Impala, Pig, Sqoop, Oozie, HBase, MongoDB, and Spark.
• Experience with Client-Server application development using Oracle PL/SQL, SQL
• PLUS, SQL Developer, TOAD, and SQL LOADER.
• Strong experience with architecting highly per formant databases using
• PostgreSQL, PostGIS, MySQL and Cassandra.
• Extensive experience in loading and analyzing large datasets with Hadoop framework (MapReduce, HDFS, PIG, HIVE, Flume and Sqoop)
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting and Object- Oriented Programming (OOPs), multithreading in Core
• Java, JDBC.
• Excellent working experience on NoSQL databases like MongoDB, Cassandra.
• Excellent working experience in Scrum / Agile framework and Waterfall project execution methodologies.
• Hands on experience in schedulingdataingestion process todatalakes using
• Apache Airflow.
• Good noledge and hands on experience on python modules such as Numpy, Pandas Matplotlib, Scikit-learn, Pyspark modules.
• Experience in importing and exporting Terabytes ofdatabetween HDFS and
• Relational Database Systems using Sqoop

WORK EXPERIENCE:

BigDataEngineer

Confidential - Raleigh, NC

Responsibilities:

• Dataingestion intodatalake(S3) and used AWS Glue to expose thedatato
• Redshift.
• Configured EMR cluster fordataingestion and used dbt (databuild tool) to transform thedatain Redshift.
• Scheduled jobs in Airflow for automating teh ingestion process into thedatalake.
• Extract, transform, and load (ETL)datafrom multiple federateddatasources
• (JSON, relational database, etc.) withDataFrames in Spark.
• Worked on writing, testing, debugging SQL code for transformations usingdatabuild tool (dbt).
• Orchestrated multiple ETL jobs using AWS step functions and lambda, also used
• AWS Glue for loading and preparingdataAnalytics for customers.
• Involved in writing Java and Node.js API for Amazon Lambda to manage some of teh
• AWS services.
• Worked on AWS Lambda to run servers without managing them and to trigger run code by S3 and SNS.
• Performed Database activities such as Indexing, performance tuning.
• Collecteddatausing Spark Streaming from AWS S3 bucket in near - real-time and performs necessary Transformations and Aggregations on teh fly to build teh common learnerdatamodel and persistence thedatain HDFS.
• Responsible in loading and transforming huge sets of structured, semi structured and unstructureddata.
• Used AWS EMR clusters for creating hadoop and spark clusters. These clusters are used for submitting and executing python applications in production.
• Designed and develop end to end ETL processing from Oracle to AWS using Amazon
• S3, EMR, and Spark.
• Written SQL Scripts and PL/SQL Scripts to extractdatafrom Database to meet business requirements and for Testing Purposes.
• Facilitated sessions to demo dbt tool for various teams and sent weekly communications on different topics related toDataEngineering.

BigDataEngineer

Confidential, Texas

Responsibilities:

• Used Agile Methodology ofDataWarehouse development using Kanbanize.
• Developeddatapipeline using Spark, Hive and HBase to ingest customer behavioraldataand financial histories into Hadoop cluster for analysis.
• Working Experience on Azure Databricks cloud to organizing thedatainto notebooks and making it easy to visualizedatausing dashboards.
• Created large datasets by combining individual datasets using various inner and outer joins in SAS/SQL and dataset sorting and merging techniques using SAS/Base.
• Worked on QA thedataand addingDatasources, snapshot, caching to teh report
• Involved in troubleshooting at database levels, error handling and performance tuning of queries and procedures.
• Worked on SAS Visual Analytics & SAS Web Report Studio fordatapresentation and reporting.
• Extensively used SAS/Macros to parameterize teh reports so that teh user could choose teh summary and sub - setting variables to be used from teh web application.
• Responsible for translating business anddatarequirements into logicaldatamodels in support Enterprisedatamodels, ODS, OLAP, OLTP and Operationaldatastructures.
• Created SSIS packages to migratedatafrom heterogeneous sources such as MS
• Excel, Flat files and CVS files.
• Provided thought leadership for architecture and teh design of BigData
• Analytics solutions for customers, actively drive Proof of Concept (POC) and Proof of Technology (POT) evaluations and to implement a BigDatasolution

Hadoop Developer

Confidential - Orlando Florida

Responsibilities:

• Performed Hive partitioning, bucketing and executing different types of joins on
• Hive tables and implementing Hive Serves like JSON and Avro.
• Worked with Hadoop Ecosystem components like HBase, Sqoop, Zookeeper, Oozie Hive and Pig with Cloudera Hadoop distribution.
• Experienced in loadingdatafrom teh UNIX file system to HDFS
• Used Google Cloud Platform Services (GCP) to process and managedatafrom streaming sources.
• Extraction of teh streamingdatausing Kafka.
• Created HBase tables to load large sets of structured, semi - structured and unstructureddatacoming from UNIX, NoSQL and a variety of portfolios.
• Used Flume to collect, aggregate, and store teh web logdatafrom different sources like web servers, network devices and pushed to HDFS.
• Scripting to deploy monitors, checks and critical sysadmin functions automation.
• Managing and scheduling Jobs on a Hadoop cluster.
• Performing tuning and troubleshooting of MapReduce jobs by analyzing and reviewing Hadoop log files.
• Involved in defining job flows, managing and reviewing log files.

Confidential - Denver, CO

Responsibilities:

• Worked extensively on ingesting disparatedatasets coming from different sources toDatalake. Used SQOOP to bringdatato landing zone from where ingestion process starts with teh existing automatic process and creates hive tables
• Ingesteddatafrom one tenant to teh other
• Developed Pig queries to loaddatato HBase
• Created ORC tables to improve teh performance for teh reporting purposes.
• Created a MapReduce program which looks intodatain HBase current and prior versions to identify transactional updates. These updates are loaded into Hive external tables which are in turn referred by Hive scripts in transactional feeds generation.
• Created SPARK Scala scripts with UDFs to transform and aggregatedatausing RDD transformations,dataframes etc. and storing result set into OLTP through sqoop.
• Worked on creatingDatamodel to establish parent child relationship for teh tables in Elastic Search
• Loadeddatafrom Hadoop to Elastic Search using Hive context in Spark
• Used spark to generatedataextracts.
• Monitoreddatain ES using Sense.
• Experience in scheduling jobs in Talend.
• Worked on agile methodology using Rally
• Involved in daily SCRUM meetings to discuss teh development/progress and was active in making Scrum meetings more productive