SUMMARY

• Around 8+ years of IT Industry experience in teh analysis designing testing maintaining database ETL, Bigdata, Cloud, and Data warehouse applications.
• Experience in designing, developing, execute and maintain data extraction, transformation, and loading for multiple corporate Operational Data Store, Data warehousing, and Data mart systems.
• Strong experience in Business and Data Analysis, Data Profiling, Data Migration, Data Integration, Data governance and Metadata Management, Master Data Management and Configuration Management.
• Ability to collaborate wif peers in both, business, and technical areas, to deliver optimal business process solutions, in line wif corporate priorities.
• Strong experience in interacting wif stakeholders/customers, gathering requirements through interviews, workshops, and existing system documentation or procedures, defining business processes, identifying and analyzing risks using appropriate templates and analysis tools.
• Excellence in handling Big Data Ecosystems like Apache Hadoop, MapReduce, Spark, HDFS Architecture, Cassandra, HBase, Sqoop, Hive, Pig, MLlib, ELT.
• Super - eminent understanding of AWS (Amazon Web Services), S3, Amazon RDS, Elasticsearch, Apache Spark RDD, process and concepts. Developing Logical Data Architecture wif adherence to Enterprise Architecture.
• Experience wif teh use of AWS services includes RDS, Networking, Route 53, IAM, S3, EC2, EBS and VPC and administering AWS resources using Console and CLI.
• Hands on experience wif Google Cloud Platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud DataProc, G - cloud function, Cloud fusion, Cloud Pub/Sub cloudshell, cloud SQL, Big Query, cloud dataflow, stack driver monitoring and cloud deployment manager.
• Experience in Python to load Data in to Bigquery and GCS bucket. Process and load bound and unbound Data from Google pub/sub to Bigquery using cloud Dataflow wif Python. Created firewall rules to access Google Data proc from other machines.
• Hands on experience building teh infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using NoSQL and SQL from AWS & Big Data technologies (Dynamo, Kinesis, S3, HIVE/Spark).
• Experience in Data Modeling wif expertise in creating Star & Snow-Flake Schemas, FACT and Dimensions Tables, Physical and Logical Data Modeling using Erwin and Embarcadero.
• Worked in supporting integrated testing of interfaces and validation of data mapping, data migration, as well as data conversion activities conducted during pre-go-live and post go-live activities
• Experience building and optimizing ‘big data’ data pipelines, architectures, and data sets. (HiveMQ, Kafka, Cassandra, S3, Redshift, Dynamo DB, RDS) and extraction using lambda, Athena, Aurora.
• Extensive experience in designing, developing, and publishing visually rich and intuitively interactive Tableau workbooks and dashboards for executive decision making.
• Involved in all phases of software development life cycle in Agile, Scrum and Waterfall management process.
• A self-motivated exuberant learner and adequate wif challenging projects and work in ambiguity to solve complex problems independently or in teh collaborative team.

TECHNICAL SKILLS

Databases: Snowflake, Netezza, AWS RDS, Teradata, Oracle 9i/10g, MySQL 5.5/5.6, Microsoft SQL 2008/12/14/16 , Postgre SQL.

NoSQL Databases: MongoDB 3.x, Hadoop HBase 0.98 and Apache Cassandra.

Programming Languages: Python, R, SQL, Scala, pig, Java, JavaScript.

Cloud Technologies: AWS, GCP, AZURE

Querying Languages: SQL, NO SQL, PostgreSQL, MySQL, Microsoft SQL

Deployment Tools: Anaconda Enterprise v5, R-Studio, Azure Machine Learning Studio, Oozie 4.2, AWS Lambda, Google cloud deployment manager

Scalable Data Tools: Hadoop, Hive, Apache Spark, Pig 0.15, Map Reduce, Sqoop.

Operating Systems: Red Hat Linux, Unix, Ubuntu, Debian, Centos, Windows, macOS.

Reporting & Visualization: Tableau 9.x/10.x, Matplotlib, Seaborn, Bokeh, ggplot, iplots, Shiny, Qlik and power BI and AWS quciksight.

Hadoop Core Services: HDFS, Map Reduce, Spark, YARN, Hive, Pig, Scala, Kafka, Flume, Tez, Impala, Solr, Oozie, Zookeeper.

Hadoop Distribution: Horton works, Cloudera

PROFESSIONAL EXPERIENCE

Confidential, OH

Sr Data Engineer

Responsibilities:

• Worked wif Google function for event driven data ingestion and route data to Pub/sub.
• Working wif data delivery teams to setup new Hadoop users. dis job includes setting up Linux users, setting upKerberosTEMPprincipals and testing HDFS, Hive.
• Implemented large scale pub/sub message queues using Apache Kafka
• Involved in creating Hadoop streaming jobs using Python.
• Ran data formatting scripts in python and created terabyte csv files to be consumed by Hadoop MapReduce jobs.
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop for analysis, visualization and to generate reports.
• Created Managed Airflow authoring and Scheduling scripts.
• Compared Self hosted Hadoop wif respect to GCPs Data Proc and explored Big Table (managed HBase) use cases, performance evolution.
• Using GCP and native GCP tools to load data on to teh platform as well as using KAFKA on prem, confluence KAFKA, NIFI, PUBSUB etc to move data between on prem and cloud platforms.
• Used Scala to write teh code for all teh use cases in Spark and Spark SQL.
• Expertise in implementing Spark and Scala application using higher order functions for both batch and interactive analysis requirement.
• Used Stack Driver Monitoring in GCP to check teh alerts of teh applications that run on theGoogle Cloud Platformand deployed on GCP using Google Cloud Deployment Manager.
• Implemented PySpark and Spark SQL for faster testing and processing of data.
• Developed Hive UDF to parse teh staged raw data to get teh Hit Times of teh claims from a specific branch for a particular insurance type code.
• Created data model for structuring and storing teh data efficiently. Implemented partitioning and bucketing of tables in Cassandra.
• Worked on migrating MapReduce programs intoPySpark transformation.
• Provided ad-hoc queries and data metrics to teh Business Users using Hive, Pig.
• Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.
• Developed POC for Apache Kafka and implementing real-time streaming ETL pipeline using Kafka Streams API.
• Build data pipelines in airflow in GCP for ETL related jobs using different airflow operators.
• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.
• Populated HDFS and Cassandra wif huge amounts of data using Apache Kafka.
• Integrated Apache Storm wif Kafka to perform web analytics. Uploaded click stream data from Kafka to HDFS, Hbase and Hive by integrating wif Storm.
• Configured Spark streaming to receive real time data from teh Apache Kafka and store teh stream data to HDFS using Scala.
• Develop and deploy teh outcome using spark and Scala code in Hadoop cluster running on GCP
• Launched multi-node kubernetes cluster in Google Kubernetes Engine (GKE) and migrated teh dockerized application from AWS to GCP.
• Ingested data from various sources into AWS Redshift and from different sources and performed ETL using AWS Glue.
• Created concurrent access for hive tables wif shared and exclusive locking that can be enabled in hive wif teh help of Zookeeper implementation in teh cluster.
• Designed and developedevent triggered data pipelinebased on Cloud Pubsubfor ingestion of teh PII and non-PII data to teh landing area on teh Google Cloud Storage (GCS) buckets
• Experience in building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP and coordinate task among teh team
• Created a Lambda Deployment function and configured it to receive events from you're S3 bucket.
• Worked wif Spark core, Spark Streaming and Spark SQL module of Spark.
• Worked on reading multiple data formats on HDFS using PySpark.

Environment: Hadoop, Map Reduce, Kafka, Teradata, HBase, Oozie, Zookeeper, Airflow, HDFS, Hive, Hue, Pig, HBase, Cloudera, Impala, MongoDB, Flume, GCP.

Confidential

Sr. Data Engineer

RESPONSIBILITIES:

• Implemented AWS solutions using EC2, S3 and load balancers.
• Installed application on AWS EC2 instances and configured teh storage on S3 buckets.
• Storing and loading teh data from HDFS to Amazon S3 and backing up teh Namespace data.
• Working wif data delivery teams to setup new Hadoop users. dis job includes setting up Linux users, setting up Kerberos TEMP principals and testing HDFS, Hive.
• Implemented managed Apache Airflowfor authoring, scheduling, and monitoring Data Pipelines.
• Created concurrent access for hive tables wif shared and exclusive locking that can be enabled in hive wif teh help of Zookeeper implementation in teh cluster.
• Worked on various performance optimizations like using distributed cache for small datasets, partition and bucketing in hive, doing map side joins etc.
• Worked on loading data into data warehouse using Hadoop components, AWS S3 bucket, AWS glue and AWS Redshift.
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop for analysis, visualization and to generate reports.
• Built managed Airflow pipelines Migrated Petabytes of data from Oracle, Hadoop, MSSQL, MySQL sources to teh AWS cloud.
• Implemented PySpark and Spark SQL for faster testing and processing of data.Developed multiple MapReduce jobs for data cleaning.
• Expertise in implementing Spark and Scala application using higher order functions for both batch and interactive analysis requirement.
• Implemented SPARK batch jobs. Worked wif Spark core, Spark Streaming and Spark SQL module of Spark.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs,Pythonand Scala.
• Worked on Distributed/Cloud Computing (Map Reduce/Hadoop, Pig, Hbase, AVRO, Zookeeper, etc.), Amazon Web Services (S3, EC2, EMR etc.)
• Provided ad-hoc queries and data metrics to teh Business Users using Hive, Pig.
• Worked on installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
• Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.
• Implementing real-time streaming ETL pipeline using Kafka Streams API and AWS Kinesis.
• Wrote teh shell scripts to monitor teh health check of Hadoop daemon services and respond accordingly to any warning or failure conditions. Wrote shell scripts for rolling day-to-day processes and it is automated.
• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: Cloudera, Java, Scala, Hadoop, Spark, AWS Kinesis, HDFS, MapReduce, Yarn, Hive, Pig, Zookeeper, Impala, Oozie, Sqoop, Flume, Kafka, Teradata, SQL, GitHub, Docker, Phabricator, Amazon Web Services

Confidential, St. Louis, MO

GCP Data Engineer

RESPONSIBILITIES:

• Building and architecting multiple Data pipelines, end to end ETL and ELT process for Data ingestion and transformation in GCP and coordinate task among teh team.
• Design and architect various layer of Data Lake. DevelopedTerraformscript and deployed it in cloud deployment manager to spin up resources like cloud virtual networks,Compute Enginesin public and private subnets along wifAuto Scaler in Google Cloud Platform.
• Created firewall rules to access Google Data proc from other machines.
• Loading Data to BIGQUERY using Google DataProc, GCS bucket, HIVE, Spark, Scala, Python, Gsutil and Shell Script.
• Using Python to ingest Data in to BIGQUERY. Involved in porting teh existing on-premises Hive code migration to GCP (Google Cloud Platform) BigQuery.
• Built a program wif Python and apache beam and execute it in cloud Dataflow to run Data validation between raw source file and Bigquery tables.
• Building a Scala and spark based configurable framework to connect common Data sources like MYSQL, Oracle, Postgres, SQL Server, Salesforce and load it in Bigquery.
• Monitoring Bigquery, Dataproc and cloud Data flow jobs via Stackdriver for all teh environments.
• Open SSH tunnel to Google DataProc to access to yarn manager to monitor spark jobs.
• Submit spark jobs using Gsutil and spark submission get it executed in Dataproc cluster.
• Built complex data pipeline using PL/SQL scripts, Cloud REST APIs, Python scripts, GCP Composer and GCP dataflow.
• Write a Python program to maintain raw file archival in GCS bucket.
• Write Scala program for spark transformation in Dataproc. Deployed ETL/ELT data pipelines using cloud fusion.
• Created complex schema and tables for analyzing using Hive.
• Configured and used Elasticsearch for log analytics and monitoring.
• Setting required Hadoop environments for cluster to preform Map Reduce jobs.
• Data was formatted using Hive queries and stored on HDFS.
• Built ETL systems using python and in-memory computing framework (Apache Spark), scheduling and maintaining data pipelines at regular intervals in Apache Airflow.
• Creating Databricks notebooks using SQL, Python and automated notebooks using jobs.
• Involved in extracting, transforming and loading data sets from local to HDFS using Hive.
• Imported and exported data from RDBMS to HDFS and vice-versa using Sqoop.
• Involved in writing Pig scripts to analyze or query structured, semi-structured and unstructured data in a file.
• Worked on HBase and MySQL for optimizing teh data.
• Worked over Sequence files, AVRO and Parquet file formats.
• Monitor Hadoop cluster connectivity and security using tools such as Zookeeper and Hue.
• Manage and review Hadoop log files.
• Developed and implemented automation processes to increase deployment efficiency.
• Developed scripts to preforming Ad hoc requests.
• Coordinate and communicate wif team and preparing technical design documents.
• AGILE(Scrum) development methodology has been followed to develop teh application. Participated in dailysprintmeetings.
• Involved in managing teh backup and disaster recovery for Hadoop data.
• Used Spark-SQL to load JSON data and create schema RDD and loaded it into Hive Tables and handled structured data using Spark SQL
• Performeddatawrangling to clean, transform and reshape thedatautilizing pandas library. Analyzed data using SQL, Scala, Python, Apache Spark and presented analytical reports to management and technical teams.

Environment: SQL, Snowflake, Python, GCP, Tableau, Hive, Databricks, Airflow, PostgreSQL, Cloud fusion, JIRA, GitHub.

Confidential, Richardson, TX

Hadoop Developer

RESPONSIBILITIES:

• Used Spark SQL to migrate teh data from hive to python using pyspark library.
• Creating end to end Spark applications using Scala to perform various data cleansing, validation, transformation, and summarization activities on user behavioral data.
• Developed end-to-end data pipeline using FTP Adaptor, Spark, Hive and Impala.
• Implemented Spark and utilized SparkSQL heavily for faster development, and processing of data.
• Exploring wif Spark for improving teh performance and optimization of teh existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's.
• Involved in converting Hive/SQL queries into Spark transformations using Spark wif Scala.
• Implemented a prototype to perform Real time streaming teh data using Spark Streaming wif Kafka
• Exported teh analyzed data to teh relational databases using Sqoop, to further visualize and generate reports for teh BI team.
• Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis
• Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Worked on various performance optimizations like using distributed cache for small datasets, Partition,Bucketing in Hive and Map Side joins.
• Created validate and maintain scripts to load data using Sqoop manually.
• Created Oozie workflows and coordinators to automate Sqoop jobs weekly and monthly.
• Built an ETL framework for Data Migration from on premise data sources such as Hadoop, Oracle to AWS using Apache Airflow, Apache Sqoop and Apache Spark (PySpark).
• Uploaded and processed more than 30 terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop.
• Configured and implemented AWS Kinesis to capture website clickstreams
• Used Oozie and Oozie coordinators to deploy end-to-end data processing pipelines and scheduling teh workflows.
• Continuous monitoring and managing teh Hadoop cluster. me has used Hadoop technologies like spark and hive Including using teh pyspark library to create spark dataframes and converting them to normal pandas dataframes for analysis.
• Created complex SQL Stored Procedures, Triggers, HIVE queries and User Defined Functions to manage incoming data as well as to support existing applications.
• Queried and analyzed large amounts of data on Hadoop HDFS using Hive and Spark
• Created teh AWS cloud environment for teh application, Designed teh Network model for teh application and also worked very intensively in getting teh application setup wif various components in teh AWS Cloud from teh user interface to back-end.

Environment: Python, Tableau, Hadoop, HDFS, PySpark, Teradata, PostgreSQL, DBT, Data stage, AWS, Jenkins, Docker, SQL, Snowflake, JIRA, GitHub, Agile/ SCRUM.

Confidential

Hadoop Developer

RESPONSIBILITIES:

• Installed and configuredHadoop MapReduce,HDFS, Developed multiple MapReduce jobs in Java for data clients and preprocessing.
• Implemented some business logics by writing UDFs in Java and used various UDFs fromPigbanks and other sources to get some results from teh data.
• Importing and exporting data into HDFS and Hive usingSqoop.
• Involved in defining job flows.
• Worked on analyzing Hadoop cluster and different big data analytic tools such asHiveQL.
• Importing and exporting data in HDFS and Hive using Sqoop.
• Extracted BSON files from MongoDB and placed in HDFS and processed.
• Loaded and transformed large sets of structured, semi structured and unstructured data.
• Exported teh result set from HIVE to MySQL using Shell scripts
• Responsible for estimating teh cluster size, monitoring and troubleshooting of teh Spark databricks cluster.
• Used Zookeeper for various types of centralized configurations.
• Involved in maintaining various Unix Shell scripts.
• ImplementedPartitioning,Dynamic PartitionsandBucketsinHIVEfor efficient data access.
• Hands on experience in AWS Cloud in various AWS services such as Redshift cluster, Route 53 domain configuration.
• On demand secure EMR launcher wif custom spark submit steps using S3 Event, SNS, KMS and Lambda function.
• Extensive noledge of working on NiFi.
• Migrated an existing on-premises application to AWS.
• Used AWS services like EC2 and S3 for small data sets.

Confidential

SQL Developer

RESPONSIBILITIES:

• Database design and development includingTables, Primary and Foreign Keys, IndexesandStored Procedures.
• GeneratedSQL and PL/SQL scriptsto install create and drop database objects, including tables, views, primary keys, indexes, constraints, packages, sequences, grants and synonyms
• ETL development in teh Netezza environment using standard design methodologies.
• Tune SQL statements using hints for maximum efficiency and performance, create and Maintain/modifyPL/SQL Packages, mentor others wif teh creation of complex SQL Statements,perform data modeling and create/maintain and modify complex database
• Worked on data Migrationscripts Extensively used SQL coding for overriding teh generated SQL in DataStage.
• Worked on structured data sets, data pipelines, ETL tools, data reduction, transformation and aggregation technique, using tools such as DataStage
• Used Database Triggerfor making history of insertion, updating, deletion and all kind of Audit routines.
• Use of EXPLAIN PLAN, ANALYZE, HINTS, TKPROFto tune queries for better performance.
• Extensively used teh advanced features of PL/SQL likeCollections, Nested Table.
• WroteUNIX Shell Scriptsto automate teh daily process as per teh business requirement.
• Writing Tuned SQL queries for data retrieval involvingComplex Join Conditions.
• Use of EXPLAIN PLAN, ANALYZE, HINTSto tune queries for better performance and also extensive Usage ofIndexes.
• Read data from flat files and load into Database usingSQL*Loader.
• Created theExternal Tablesin order to load data from flat files and PL/SQL scripts for monitoring.
• Involved in analysis, design and coding of Oracle processes inPL/SQLto buildPackages.
• Design and Development of data validation, load processes, test cases using PL/SQL, Stored Procedures, Functions, Triggers using Oracle 10g.