SUMMARY:

• Over 7 Years of experience in Big Data Analytics using Various Hadoop eco - systems tools and Spark Framework and Currently working on Spark and Spark Streaming frameworks extensively using Scala as the main programming dialect
• Experience installing/configuring/maintaining Apache Hadoop clusters for application development and Hadoop tools like Sqoop, Hive, PIG, Flume, Hbase, Kafka, Hue, Storm, Zoo Keeper, Oozie, Cassandra, Sqoop, Python
• Worked with major distributions like Cloudera (CDH 3&4) & Horton works Distributions and AWS. Also worked on Unix and DWH in support for various Distributions
• Hands on experience in developing and deploying enterprise-based applications using major components in Hadoop ecosystem like Hadoop 2.X, YARN, Hive, Pig, MapReduce, Spark, Kafka, Storm, Oozie, HBase, Flume, Sqoop and ZooKeeper
• Experience in developing data pipeline using Pig, Sqoop, and Flume to extract the data from weblogs and store in HDFS and accomplished developing Pig Latin Scripts and using HiveQL for data analytics
• Extensively dealt with Spark Streaming and Apache Kafka to fetch live stream data.
• Experience in converting Hive/SQL queries into Spark transformations using Java and experience in ETL development using Kafka, Flume and Sqoop
• Good experience in writing Spark applications using Scala and Java and used Scala sbt to develop Scala projects and executed using Spark-Submit
• Experience working on NoSQL databases including Hbase, Cassandra and MongoDB and experience using Sqoop to import data into HDFS from RDBMS and vice-versa
• Developed Spark scripts by using Scala shell commands as per the requirement
• Good experience in writing Sqoop queries for transferring bulk data between Apache Hadoop and structured data stores
• Substantial experience in writing Map Reduce jobs in Java, PIG, Flume, Zookeeper, Hive and Storm
• Strong expertise in troubleshooting and performance fine-tuning Spark , MapReduce and Hive applications
• Extensive experience in developing applications that perform Data Processing tasks using Teradata, Oracle, SQL Server and MySQL database
• Worked on data warehousing and ETL tools like Informatica , Tableau , and Pentaho
• Acquaintance with Agile and Waterfall methodologies. Responsible for handling several clients facing meetings with great communication skills

TECHNICAL SKILLS:

Big Data Technologies: HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, Zookeeper, Kafka, Cassandra, Apache Spark, Spark Streaming, HBase, Flume, Impala

Hadoop Distribution: Cloudera, Horton Works, Apache, AWS

Languages: Core Java, SQL, PL/SQL, Python, Pig Latin, HiveQL, Scala, Regular Expressions

Web Technologies: HTML, CSS, JavaScript, XML, JSP, Restful, SOAP

Operating Systems: Windows (xp/7/8/10), UNIX, LINUX, UBUNTU, CENTOS.

Build Automation tools: SBT, Ant, Maven

Version Control: GIT

IDE & Builld Tools, Design: Eclipse, Visual Studio, Net Beans, Rational Application Developer, Junit

Databases: Oracle, SQL Server, MySQL, MS Access, NoSQL Database (Hbase, Cassandra, MongoDB), Teradata.

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Hadoop Developer

Responsibilities:

• Developed programs in Spark based on the application for faster data processing than standard Map Reduce programs
• Developed spark programs using Scala, involved in creating Spark SQL Queries and Developed Oozie workflow for spark jobs
• Developed the Oozie workflows with Sqoop actions to migrate the data from relational databases like Oracle, Teradata to HDFS
• Used Hadoop FS actions to move the data from upstream location to local data locations
• Written extensive Hive queries to do transformations on the data to be used by downstream models
• Developed map reduce programs as a part of predictive analytical model development
• Developed Hive queries to do analysis of the data and to generate the end reports to be used by business users
• Worked on scalable distributed computing systems, software architecture, data structures and algorithms using Hadoop, Apache Spark and ingested streaming data into Hadoop using Spark Framework and Scala
• Expertise working with NOSQL databases like MongoDB
• Extensively used GIT as a code repository and Version One for managing day agile project development process and to keep track of the issues and blockers
• Written spark python for model integration layer
• Implemented Spark using Scala, Java and utilizing Data frames and Spark SQL API for faster processing of data
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data
• Wrote new spark jobs in Scala to analyze the data of the customers and sales history
• Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data
• Developed a data pipeline using Kafka, HBase, Mesos Spark and Hive to ingest, transform and analysing customer behavioural data.

Environment: LINUX, Hadoop, HBase, Hive, Impala, Oracle, Spark, Scala, Python, Pig, Sqoop, Teradata, Zookeeper, Oozie, MongoDB, Map Reduce, GitHub.

Confidential

Spark & Hadoop Developer

Responsibilities:

• Worked on analysing Hadoop cluster and different big data analytical and processing tools including Pig, Hive, Sqoop, python and Spark with Scala & java, Spark Streaming
• Worked on the large-scale Hadoop YARN cluster for distributed data processing and analysis using Spark, Hive, and HBase
• Involved in creating data-lake by extracting customer's data from various data sources to HDFS which include data from Excel, databases, and log data from servers
• Developed Apache Spark applications by using Scala and python for data processing from various streaming sources
• Used Scala to convert Hive/SQL queries into RDD transformations in Apache Spark
• Implemented Spark solutions to generate reports, fetch and load data in Hive
• Experienced in writing real-time processing and core jobs using Spark Streaming with Kafka as a data pipeline system
• Written HiveQL to analyse the number of unique visitors and their visit information such as views, most visited pages, etc.
• Wrote Spark-Streaming applications to consume the data from Kafka topics and wrote processed streams to HBase and steamed data using Spark with Kafka
• Configured Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala
• Monitored workload, job performance and capacity planning using Cloudera Manager
• Designing and implementing complete end-to-end Hadoop Infrastructure including Pig, Hive, Sqoop, Oozie, Flume, and Zookeeper
• Further used pig to do transformations, event joins, elephant bird API and pre -aggregations performed before loading JSON files format onto HDFS
• Involved in resolving performance issues in Pig and Hive with understanding of Map Reduce physical plan execution and using debugging commands to run code in optimized way
• Used Spark to perform analytics on data in Hive and experienced with ETL working with Hive and Map-Reduce.

Environment: Hdp 2.6.0, HDFS, Map Reduce, Spark Streaming, Spark-Core, Spark SQL, Scala, Pig 0.14, Hive 1.2.1, Sqoop 1.4.4, Flume 1.6.0, Kafka, JSON, HBase.

Confidential

Sr. Java Developer

Responsibilities:

• Worked on Horton works-HDP 2.5distribution
• Responsible for building-scalable distribution data solution using Hadoop
• Involved in importing data from MS SQL Server, MySQL and Teradata into HDFS using Sqoop
• Played a key role in dynamic partitioning and Bucketing of the data stored in Hive Metadata
• Wrote HiveQL queries for integrating different tables for create views to produce result set
• Collected the log data from Web Servers and integrated into HDFS using Flume
• Worked on loading and transforming of large sets of structured and unstructured data
• Used Map Reduce programs for data cleaning and transformations and load the output into the Hive tables in different file formats
• Written Map Reduce programs to handle semi structured and unstructured data like JSON, Avro data files and sequence files for log files
• Created data pipelines for different events to load the data from Dynamo DB to AWS S3 bucket and then into HDFS location
• Involved in loading data into HBase NoSQL database
• Building, Managing and scheduling Oozie workflows for end to end job processing
• Worked on extending Hive and Pig core functionality by writing custom UDFs using Java
• Analysing of Large volumes of structured data using SparkSQL
• Wrote Shell Script to execute HiveQL
• Used Spark as ETL tool
• Wrote automated shell scripts in Linux/Unix environment using bash
• Migrated HiveQL queries into SparkSQL to improve performance
• Extracted Real time feed using Spark streaming and convert to RDD and process data into Data Frame and load the data into HBase
• Used Data Stax Spark connector which is used to store the data into Cassandra database or get the data from Cassandra database
• Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load the data into Cassandra

Environment: Horton works, Hadoop, HDFS, Pig, Sqoop, Hive, Oozie, Zookeeper, NoSQL, HBase, Shell Scripting, Scala, Spark, SparkSQL

Confidential

SQL/Java Developer

Responsibilities:

• Involved in the development of use case documentation, requirement analysis, and project documentation
• Developed and maintained Web applications as defined by the Project Lead
• Developed GUI using JSP, JavaScript, and CSS
• Used MS Visio for creating business process diagrams
• Developed Action Servlet, Action Form, Java Bean classes for implementing business logic for the struts Framework
• Developed Servlets and JSP based on MVC pattern using struts Action framework
• Developed all the tiers of the J2EE application. Developed data objects to communicate with the database using JDBC in the database tier, implemented business logic using EJBs in the middle tier, developed Java Beans and helper classes to communicate with the presentation tier which consists of JSPs and Servlets
• Used AJAX for Client-side validations
• Applied annotations for dependency injection and transforming POJO/POJI to EJBs
• Developed persistence layer modules using EJB Java Persistence API (JPA) annotations and Entity manager
• Involved in creating EJBs that handle business logic and persistence of data
• Developed Action and Form Bean classes to retrieve data and process server-side validations
• Designed various tables required for the project in Oracle database and used Stored Procedures in the application. Used PL SQL to create, update and manipulate tables
• Used IntelliJ as IDE and Tortoise SVN for version control
• Involved in impact analysis of Change requests and Bug fixes

Environment: Java 5, Struts, PL/SQL, Oracle, EJB, IntelliJ, Tortoise SVN, MS Visio, Firebug, Apache Tomcat, JSP, Java Script, CSS.