SUMMARY

• 5+ years of IT experience in various domains wif Bigdata, Hadoop Ecosystems and Java J2EE technologies.
• Very good hands - on in Spark Core, Spark Sql and good knowledge on Spark Streaming and Spark machine learning using Scala and Pspark.
• Solid understanding of RDD operations in Apache Spark me.e., Transformations and Actions, Persistence (Caching), Accumulators, Broadcast Variables, Optimizing Broadcasts.
• Tuned Spark RDD parallelism technics to improving the performance and optimization of the spark jobs on Hadoopcluster
• In depth understanding of Apache spark job execution Components like DAG, lineage graph, Dag Scheduler, Task scheduler, Stages and task.
• Hands on experience in testing and implementation phase of all the big data Technologies.
• Experience wif Apache Nifi.
• Solid understanding of open-source monitoring tools like Apache Ambari.
• Analyze and develop Transformation logic for handling large sets of structured, semi structured and unstructured data using Spark and Spark SQL.
• Hands on experience in working wif input file formats like orc, parquet, json, Avro.
• Hands on experience wif NOSQL databases HBase and Cassandra.
• Good working knowledge in creating Hive tables and worked using HQL for data analysis to meet the business requirements.
• Experience in managing and reviewing Hadoop log files.

TECHNICAL SKILLS

Big Data Frameworks: Hadoop, Spark, Pyspark, Scala, Hive, AWS, HBase, Flume, Pig, Sqoop, MapReduce, Kafka

Bigdata distribution: Hortonworks, Amazon EMR, Cloudera

Programming languages: Java, Scala, Python Scripting, SQL, Shell Scripting

Operating Systems: Windows, Linux

Databases: Oracle, SQL Server, Mongo DB, Cassandra, memSQL,Teradata, MS SQL

Designing Tools: Intellij, Eclipse

Java Technologies: JSP, Servlets, Junit, Spring, Hibernate

Web Technologies: XML, HTML, JavaScript, JVM, JQuery, JSON

Linux Experience: System Administration Tools

Web Services: Web Service (RESTfuland SOAP)

Development methodologies: Agile

Logging Tools: Log4j

Application / Web Servers: Nginx, Apache Tomcat, Websphere

Messaging Services: Flume, Kafka

Version Tools: Git, Bitbucket

PROFESSIONAL EXPERIENCE

Confidential, Bentonville, Arkansas

Data Engineer

Responsibilities:

• Performed source analysis on various databases for modelling and loading the customer data into Data lake
• Designed and developed ETL data pipelines for loading data into data lake using Automic.
• Developed history, full and incremental load yamls which consists of HQL queries wif complex joins.
• Developed customized Sqoop scripts for loading data from different databases like Oracle, MSSql, Teradata.
• Developed shell scripts for loading zip files, CSV files from Azure blob to Data lake
• Developed Pyspark count check utility for hive and RDBMS sources for data validation
• Developed Distcp framework shell script for copying the data across different clusters and environments.
• Developed various shell scripts like identifying the landing of zero byte in unix mount and hdfs landing locations, moving files from hdfs landing to archive location etc.
• Developed python wrapper script to optimize the loading of history data by breaking the load into number of small batches
• Implemented pre-defined operators in spark such as map, flat Map, filter, reduceByKey, groupByKey, aggregateByKey and combineByKey etc.
• Used Scala-Maven to develop Scala coded spark projects and executed using spark-submit
• Extract Real time feed using Kafka and Spark Streaming and convert it to RDD and process data in the form of Data Frame and save the data as Parquet format in HDFS.
• Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the Scala API
• Responsible in performing sort, join, aggregations, filter, and other transformations on the datasets using Spark
• Implementes Spark using Scala and Spark SQL for faster testing and processing of data responsible to manage data from different sources.

Environment: Hortonworks, Hive, Azure, Oracle, MSSql, Teradata, CSV files, JSON, Sqoop, HDFS, Automic, Shell scripting, Python scripting, Jenkins, Github, Pyspark, Scala, Maven, Kafka.

Confidential, Chicago, Illinois

Data Engineer

Responsibilities:

• Developed a Pyspark application for creating the Cal report Hive table and made it available for tableau reporting.
• Used Sqoop for loading data to HDFS collection layer.
• Extracted all the Transmit reporting fields from very large Json files stored in Mongo DB by applying various filters.
• Debbuged and recommended few optimizations in creating master dataset which is used for pattern extraction in Rules Genaration Algorithm (RGA) Pyspark model.
• Developed Pyspark application for creating Payfone reporting tables wif different maskings in both Hive and Mysql DB and made available for newly build fetch API’ Confidential .
• Used pre-defined Udf’ Confidential for Protegrity tokenizing and AES encrypting the sensitive fields like Phonenumber, Countrycode, SSN and DOB etc in the reporting tables.
• Successfully migrated multiple Scala and Pyspark applications from old clusters to new LCM clusters.
• Used Distcp for data copying activities across different environments.
• Used Pyspark Dataframe approach for creating the Cal,Dapply and Payfone reporting tables.

Environment: Hortonworks, Pyspark, Scala, Spark-core, Spark-SQL, Hive, MongoDB, Mysql, Sqoop, HDFS, Autosys, Bitbucket, Jenkins.

Confidential, Orlando, Florida

Hadoop Developer

Responsibilities:

• Debugged issues wif WCA event processors code base written in Spark Scala that’ Confidential currently in production and gave recommendations for rewrites.
• Established connection for read and writes between spark edgenode and db2 server using ibmdb2 jdbc jar.
• Worked on Nifi data pipelines that are built for consuming data into Datalake.
• Built Nifi flows for testing and debugging purposes.
• Developed Spark programs using scala API’ Confidential to compare the performance of Spark wif Hive and SQL
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in hive
• Implemented Spark using Scala and SparkSql for faster testing and processing of data.
• Used Json and Xml SerDe’ Confidential for serialization and de-serialization to load Json and Xml data into hive tables.
• Worked on an oozie scheduled application Garment Inventory Management System(Gims) pyspark application to ingest employee garment (costume) information to inform better analytics around the employees, garments, and cleaning processes, mostly focused on cost reduction.
• Worked on an oozie-scheduled kronos pyspark application that generates data for ingestion into Kronos SQL server, to support forecasting of food and beverages sales transaction volumes and security screen (turnstyle) volumes for 15 minute intervals by location.
• Added new tables and new columns for the existing tables in collection and consolidation layers as per the business requirements for both Kronos and Giims pipelines.
• Successfully migrated Kronos and wardrobe pipelines across all the environments.

Environment: Hortonworks, Hive, SparkCore, SparkSql, Nifi, Sqoop, Hdfs, Kafka, Oozie,SQL Server.

Confidential, New York

Bigdata/Hadoop Developer

Responsibilities:

• Analyzed Integrated data facility platform (IDF) and made recommendations to the Scala source code in storing large amount of data in parquet format on S3 buckets.
• Debugged the defects such as count mismatch, missing tables on S3.
• Analyzed nightly, daily, weekly and monthly spark jobs which are triggered using Autosys in IDF platform.
• Used EMR edge nodes in development and QA environments for debugging the issues.
• Analyzed Spears Automated Surveillance which is a rules-based monitoring system built on top of IDF platform.
• Developed various test cases scenarios based on rules provided by the business using Spark-Scala API’ Confidential .
• Worked closely wif Business analysts and Enterprise architects for understanding the rules provided by the business.
• Implemented Spark using Scala and SparkSQL for faster testing and processing of data
• Worked on several semi-structured formats to process datasets in S3 like JSON, XML, CSV
• Writing Spark SQL in Scala for extracting data from AWS S3, transform it and load the data to AWS S3
• Worked extensively on AWS Components such as EC2, S3, Elastic Map Reduce (EMR)
• Used Spark-SQL to load json data and create schema RDD and load it into Hive tables.
• Developed Spark scripts using scala as per requirements.

Environment: SparkCore, Spark-Sql, Scala, Hive, S3, EMR, Autosys, Oracle, Sqoop

Confidential, Sunnyvale, California

Hadoop and Data Science Platform Engineer

Responsibilities:

• Performed benchmarking of federated queries in Spark and compared their performance by running the same queries on Presto.
• Defined Spark confs for optimization of federated queries by maneuvering the number of executors, executor-memory and executor-cores.
• Created partitions and buckets defined Hive tables for data analysis.
• Successfully migrated data from Hive to Memsql db via Spark engine where the largest table being 1.2T.
• Successfully ran benchmarking queries on Memsql database and calculated the performance of each query.
• Compared the performance of each benchmark query among different solutions like Spark, Teradata, Memsql, Presto, Hive (using Tez engine) by creating a bar graph in Numbers.
• Successfully migrated data from Teradata to Memsql using Spark by persisting Dataframe to Memsql.
• Provided a solution using HIVE, SQOOP (to export/ import data), for faster data load by replacing the traditional ETL process wif HDFS for loading data to target tables.
• Developed Spark scripts by using Scala as per the requirement.
• Developed Java scripts using both RDD and Data frames/SQL/Data sets in Spark 1.6 and Spark 2.1 for Data Aggregation, queries and writing data.
• Used Grafana for analyzing the usage of spark executors for different queues on different clusters.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, efficient Joins, Transformations and other during ingestion process itself.
• Developed Hive queries to process the data and generate the data cubes for visualizing
• Converting SQL codes to Spark codes using Java and Spark-SQL/Streaming for faster testing and processing of data and Import and index data from HDFS for secure searching, reporting, analysis and visualizations in Splunk.
• Working extensively on Hive, SQL, Scala, Spark, and Shell.
• Used to complete the Assigned radar' Confidential in time, used to store the code in GIT repository.
• Tested python, R, livy, teradata jdbc interpreters but executing sample paragraphs.
• Performed CI/CD builds of Zeppelin, Azkaban and Notebook using Ansible.
• Built a new version of Zeppelin by applying Git patches, changing the artifacts using Maven.
• Worked on shell scripting to determine the status on various components in data science platform.
• Performed data copying activities in a distributed environment using Ansible.

Environment: SparkCore, SparkSQL, Memsql, Presto, Teradata, Hive, ApacheZeppelin, Maven,Github, Intellij, Nginx, Redis, Monit, Linux, Shell Scripting, Ansible.

Confidential, Orlando, Florida

Bigdata Developer

Responsibilities:

• Extracted the data from Teradata into HDFS using Sqoop.
• Used Flume to collect, aggregate and store the web log data from different sources like web servers, mobile and network devices and pushed into HDFS.
• Implemented MapReduce programs on log data to transform into structured way to find user information.
• Extensive experience in writing Pig scripts to transform raw data from several data sources into forming baseline data.
• Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views and visit duration.
• Extensive work inETLprocess consisting of data transformation, data sourcing, mapping, conversion and loading using Informatica.
• Extensively usedETLprocesses to load data from flat files into the target database by applying business logic on transformation mapping for inserting and updating records when loaded.
• Created Talend ETL jobs to read the data from Oracle Database and import in HDFS.
• Worked on data serialization formats for converting complex objects into sequence bits by using Avro,RC and ORC file formats
• Modified store procedures to setup the control table that used to generate package id, batch id and status for each batch.
• Created DDL’ Confidential for the migration tables residing in mysql and mssql databases
• Performed batch processing on large sets of data.
• Built Apache Nifi flow for migration of data from mssql and mysql databases to the Staging table.
• Performed transformations on large data sets using Apache Nifi expression language.
• Unit tested the migration of mysql and mssql tables using the built Nifi flow.
• Used Dbeaver for connecting to the different databases that are on different sources.
• Performed queries for verifying the data types of different columns that are being migrated to staging table.
• Responsible for monitoring data from source to target.
• Successfully populated the staging tables in mysql database wifout any data mismatch errors.
• Worked on Agile Version One methodology by attending the scrums and scrum plannings.

Environment: Apache Hadoop, Hortonworks HDP 2.0, HDFS, MapReduce, Sqoop, Flume, Pig, Hive, HBaseOozie, Teradata, Avro, Java, Talend, Linux, Apache NiFi, Mysql, Mssql, Agile,Dbeaver.

Confidential

Java Developer

Responsibilities:

• Worked wif the business community to define business requirements and analyze the possible technical solutions.
• Requirement gathering, Business Process flow, Business Process Modeling and Business Analysis.
• Extensively used UML and Rational Rose for designing to develop various use cases, class diagrams and sequence diagrams.
• Used JavaScript for client-side validations, and AJAX to create interactive front-end GUI.
• Developed application using Spring MVC architecture.
• Developed custom tags for table utility component
• Used various Java, J2EE APIs including JDBC, XML, Servlets, and JSP.
• Designed and implemented the UI using Java, HTML, JSP and JavaScript.
• Designed and developed web pages using Servlets and JSPs and also used XML/XSL/XSLT as repository.
• Involved in Java application testing and maintenance in development and production.
• Involved in developing the customer form data tables. Maintaining the customer support and customer data from database tables in MySQL database.
• Involved in mentoring specific projects in application of the new SDLC based on the Agile Unified Process, especially from the project management, requirements and architecture perspectives.
• Designed and developed Views, Model and Controller components implementing MVC Framework.

Environment: JDK 1.3, J2EE, JDBC, Servlets, JSP, XML, XSL, CSS, HTML, DHTML, JavaScript, UML, Eclipse 3.0, Tomcat 4.1, MySQL.