SUMMARY

• Total 9 years of IT experience in Analysis, Design, Development, in that 5 years in Data Engineer technologies like Spark, Map reduce, Hive Yarn and HDFS including programming languages like Java, and Python.
• 4 years of experience in Data warehouse / ETL Developer role.
• Strong experience building data pipelines and performing large - scale data transformations.
• In-Depth knowledge in working with Distributed Computing Systems and parallel processing techniques to efficiently deal with Big Data.
• Firm understanding of Hadoop architecture and various components including HDFS, Yarn, Map Reduce, Hive, Pig, HBase, Kafka, Oozie etc.,
• Strong experience building Spark applications using and python as programming language.
• Good experience troubleshooting and fine-tuning long running spark applications.
• Strong experience using Spark RDD API, Spark Data Frame/Dataset API, Spark-SQL and Spark ML frameworks for building end-to-end data pipelines.
• Extensive hands on experience tuning spark Jobs.
• Experienced in working with structured data using HiveQL, and optimizing Hive queries.
• Good experience working with real time streaming pipelines using Kafka and Spark-Streaming.
• Strong experience working with Hive for performing various data analysis.
• Detailed exposure with various hive concepts like Partitioning, Bucketing, Join optimizations, Sedge, built-in UDF’s and custom UDF’s.
• Good experience in automating end-to-end data pipelines using Oozie workflow orchestrator.
• Good experience working with Cloudera, Hortonworks, and Snowflake and AWS big data services.
• Working on AWS Services IAM, EC2, VPC, AMI, SNS, RDS, SQS, EMR, LAMBDA, GLUE, ATHENA, Dynamo DB, Kinesis, Cloud Watch, Auto Scaling, S3, and Route 53
• Implemented Lambda to configure Dynamo DB Auto scaling feature and implemented Data Access Layer to access AWS Dynamo DB data
• Developed and deployed various Lambda functions in AWS with in-built AWS Lambda Libraries and also deployed Lambda Functions in with custom Libraries
• Experience with developing and maintaining Applications written for AWS S3, AWS EMR ( Confidential Map Reduce), and AWS Cloud Watch
• Experience in analyzing, designing, and developing ETL Strategies and processes, writing ETL specifications.
• Excellent understanding of NoSQL databases like HBASE, and MongoDB.
• Proficient knowledge and hand on experience in writing shell scripts in Linux.
• Experienced in requirement analysis, application development, application migration and maintenance using Software Development Lifecycle (SDLC) and Python/Java technologies.
• Excellent technical and analytical skills with clear understanding of design goals and development for OLTP and dimensions modeling for OLAP.
• Adequate knowledge and working experience in Agile and Waterfall Methodologies.
• Defining user stories and driving the agile board in JIRA during project execution, participate in sprint demo and retrospective.
• Have good interpersonal, communication skills, strong problem-solving skills, explore/adopt to new technologies with ease and a good team member.
• In-depth knowledge of Snowflake Database, Schema and Table structures.
• Experience with gathering and analyzing the system requirements.

TECHNICAL SKILLS

Big Data Technologies: Hadoop, Map Reduce, HDFS, Sqoop, Hive, HBase, Flume, Kafka, Yarn, Apache Spark.

Databases: Oracle, MySQL, SQL Server, MongoDB, Dynamo DB, Cassandra, Snowflake.

Programming Languages: Python, Pyspark, Shell script, Perl script, SQL, Java.

Cloud: AWS (EC2, EMR, Lambda, IAM, S3, Athena, Glue, Kinesis, Cloud Watch, RDS, Redshift)

Tools: PyCharm, Eclipse, Visual Studio, SQL*Plus, SQL Developer, SQL Navigator, SQL Server Management Studio, Eclipse, Postman.

Version Control: SVN, Git, GitHub, Maven

Operating Systems: Windows 10/7/XP/2000/NT/98/95, UNIX, LINUX, OS

Visualization/ Reporting: Tableau, ggplot2, matplotlib

Database Modelling: Dimension Modeling, ER Modeling

Machine Learning Techniques: Linear & Logistic Regression, Decision Trees, Clustering.

PROFESSIONAL EXPERIENCE

Confidential, Frisco, TX

AWS Data Engineer

Responsibilities:

• Designed robust, reusable and scalable data driven solutions and data pipeline frameworks to automate the ingestion, processing and delivery of both structured and semi structured batch and real time data streaming data.
• Develop framework for converting existing Power Center mappings and to Pyspark (Python and Spark) Jobs.
• Create Pyspark frame to bring data from DB2 to Amazon S3.
• Applied efficient and scalable data transformations on the ingested data using Spark framework.
• Gained good knowledge in troubleshooting and performance tuning Spark applications and Hive scripts to achieve optimal performance.
• Developed various custom UDF’s in spark for performing transformations on date fields, complex string columns and encrypting PI fields etc.,
• Written complex hive scripts for performing various data analysis and creating various reports requested by business stakeholders.
• Capable of using AWS utilities such as EMR, S3 and Cloud Watch to run and monitor Hadoop and Spark jobs on AWS.
• Used Oozie and Oozie Coordinators for automating and scheduling our data pipelines.
• Used AWS Athena extensively to ingest structured data from S3 into other systems such as Redshift or to produce reports.
• The Spark-Streaming APIs were used to conduct on-the-fly transformations and actions for creating the common learner data model, which receives data from Kinesis in near real time.
• Implemented data ingestion from various source systems using Sqoop and Pyspark.
• Hands on experience implementing Spark and Hive jobs performance tuning.
• Performed end- to-end Architecture & implementation assessment of various AWS services like Amazon EMR, Redshift, S3, Athena, Glue and Kinesis.
• Hive As the primary query engine of EMR, we have built external table schemas for the data being processed.
• AWS RDS (Relational database services) was created to serve as a Hive Meta store, and it was possible to integrate the Meta data from 20 EMR clusters into a single RDS, avoiding data loss even if the EMR was terminated.
• Worked extensively on migrating our existing on-prem data pipelines to AWS cloud for better scalability and infra structure maintenance.
• Worked extensively in automating creation/termination of EMR clusters as part of starting the data pipelines.
• Worked extensively on migrating/rewriting existing Oozie jobs to AWS simple workflow.
• Loaded the processed data into Redshift tables for allowing downstream ETL and Reporting teams to consume the processed data.
• Good experience working on analysis tools like Tableau, Splunk for regression analysis, pie charts and bar graphs.

Environment: AWS Cloud, Spark, Kafka, Hive, Yarn, HBase, Jenkins, Docker, Tableau, Splunk.

Confidential

AWS Data Engineer

Responsibilities:

• Responsible for the design, implementation, and architecture of very large-scale data intelligence solutions around big data platforms.
• Analyzed large and critical datasets using HDFS, HBase, Hive, HQL, Pig, Sqoop and Zookeeper.
• Developed multiple POC’s using Spark, and deployed on the Yarn Cluster, compared the performance of Spark, with Hive and SQL.
• Use Amazon Confidential Cloud Compute (EC2) infrastructure for computational tasks and Simple Storage Service (S3) as storage mechanism.
• Capable of using AWS utilities such as EMR, S3 and Cloud Watch to run and monitor Hadoop and Spark jobs on AWS.
• Worked on SQL queries in dimensional data warehouses and relational data warehouses. Performed Data Analysis and Data Profiling using Complex SQL queries on various systems.
• Troubleshoot and resolve data processing issues and proactively engaged in data modelling discussions.
• Worked on RDD Architecture and implementing spark operations on RDD and optimizing transformations and actions in Spark.
• Written programs in Spark using Python (Pyspark) packages for performance tuning, optimization, and data quality validations.
• Implemented data ingestion from various source systems using Sqoop and Pyspark.
• Hands on experience implementing Spark and Hive jobs performance tuning.
• Worked on developing Kafka Producers and Kafka Consumers for streaming millions of events per second on streaming data.
• Implemented a distributing messaging queue to integrate with using Apache Kafka.
• Hands on experience on fetching the live stream data from UDB into HBase table using Pyspark streaming and Apache Kafka.
• Evaluate Snowflake Design considerations for any change in the application

Environment: HDFS, Python, SQL, Spark Kafka, Hive, Yarn, Sqoop, Snowflake, Tableau, AWS Cloud, GitHub, Shell Scripting.

Confidential

Bigdata Developer

Responsibilities:

• Designed and developed the applications on the data lake to transform the data according business users to perform analytics.
• Responsible to manage data coming from different sources, involved in HDFS maintenance, and loading of structured and unstructured data.
• Worked on different files like csv, txt, fixed width to load data from various sources to raw tables.
• Conducted data model reviews with team members and captured technical metadata through modelling tools.
• Implemented ETL process wrote and optimized SQL queries to perform data extraction and merging from SQL server database.
• Transferred data from HDFS to Relational Database Systems using Sqoop for Business Intelligence, visualization, and user report generation.
• Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's.
• Maps were used on many occasions like Reducing the number of tasks in Pig and Hive for data cleansing and pre-processing.
• Build Hadoop solutions for big data problems using MR1 and MR2 in YARN.
• Created Hive Tables, used Sqoop to load claims data from Oracle, and then put the processed data into the target database.
• Experience in loading logs from multiple sources into HDFS using Flume.
• Worked with NoSQL databases like HBase in creating HBase tables to store large sets of semi-structured data coming from various data sources.
• Involved in designing and developing tables in HBase and storing aggregated data from Hive tables.
• Developed complex Map reduce jobs for performing efficient data transformations.
• Data cleaning, pre-processing and modelling using Java Map reduce.
• Strong Experience in writing SQL queries.
• Responsible for triggering the jobs using the Control-M.

Environment: Java, SQL, ETL, Hadoop, HDFS, HBase, MySQL, Web Services, Shell Script, Control-M.

Confidential

Hadoop Engineer

Responsibilities:

• Implemented big data technologies such as Hadoop, Map Reduce Frameworks, HBase, and Hive for ingesting data from diverse sources and processing Data-at-Rest.
• Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
• Used Hadoop technologies like spark and hive including using the Pyspark library to create spark data frames and converting them to normal panda’s data frames for analysis.
• Designed and build a Data Lake using Hadoop and its ecosystem components.
• Developed Spark, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources.
• Worked with data investigation, discovery and mapping tools to scan every single data record from many sources.
• Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in ORC format.
• Executed multiple Spark SQL queries after forming the Database to gather specific data corresponding to an image.
• Developed prototype for Big Data analysis using Spark, RDD, Data Frames and Hadoop eco system with CSV, JSON, and distributed files.
• Implemented data ingestion from various source systems using Sqoop and Pyspark.
• Hands on experience implementing Spark and Hive jobs performance tuning.
• Have knowledge on partition of Kafka messages and setting up the replication factors in Kafka Cluster and Implemented to reprocess the failure messages in Kafka using offset id.
• Reviewed Kafka cluster configurations and provided best practices to get peak performance.
• Designed and Implemented Error-Free Data Warehouse-ETL and Hadoop Integration.
• Enhancements to conventional data warehouses based on the STAR schema, data model updates, and Tableau data analytics and reporting.
• Evaluated the performance of Data bricks environment by converting complex Redshift scripts to spark SQL as part of new technology adaption project.

Environment: Spark, Hadoop, YARN, HTML, Python, Data bricks, Kubernetes JDBC, TERADATA, NOSQL, Sqoop, MYSQL.

Confidential

ETL Developer/Data Warehouse Developer

Responsibilities:

• Working as SQL Server Analyst / Developer / DBA using SQL Server 2012, 2015, 2016.
• Created jobs, SQL Mail Agent, Alerts and schedule DTS/SSIS Packages.
• Manage and update the Erwin models - Logical/Physical Data Modeling for Consolidated Data Store (CDS), Actuarial Data Mart (ADM), and Reference DB according to the user requirements.
• Source Controlling, environment specific script deployment tracking using TFS
• Export the current Data Models into PDF out of Erwin and publish them on to SharePoint for various users.
• Developing, Administering, and Managing corresponding databases: Consolidated Data Store, Reference Database (Source for the Code/Values of the Legacy Source Systems), and Actuarial Data Mart
• Writing Triggers, Stored Procedures, Functions, Coding-using Transact-SQL (TSQL), create and maintain Physical Structures.
• Working with Dynamic Management Views, System Views for Indexing and other Performance Problems Analyze the Legacy Data, Import the data into Reference DB that was verified and published by DMRB (Data Management Review Board) Liaison with ETL Team to convert ETL requirements as Standard Database solutions to boost the performance.
• Creating ETL mappings using Informatica Power CenterDay to Day maintenance of databases, Staging, Backup / Restores of databases.
• Deployment of Scripts in different environments according to Configuration Management, Playbook requirements Create / Manage Files/File group - Table/Index association Query Tuning, Performance Tuning.
• Defect tracking and closing by using Quality Center Maintain Users / Roles / Permissions.
• Capacity Planning, Disk Requirements

Environment: SQL Server 2008/2012 Enterprise Edition, SSRS, SSIS, T-SQL, Windows Server 2003, Performance Point Server 2007, C/C++, Oracle 10g, PL/SQL, Visual SourceSafe, visual Studio 2010.