Professional Summary:

• Experience in HadoopEcosystem components like MapReduce, Sqoop, Flume, Kafka, Pig, Hive, Spark, Storm, HBase, Oozie and Zookeeper.
• In depth understanding of HadoopArchitecture including YARN and various components such as HDFS, Resource Manager, Node Manager, Name Node, Data Node and MR v1 & v2 concepts.
• Good experience on creating Data Pipelines inSparkusing SCALA.
• Hands on experience in working onSpark - SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
• Experience in importing and exporting data using Sqoop from RDBMS to HDFS and vice-versa.
• Good experience onSparkcomponents likeSparkSQL and SparkStreaming.
• Expertise in integrating the data from multiple data sources using Kafka.
• Knowledge about unifying data platforms using Kafka producers/ consumers, implement pre-processing using storm topologies.
• Exposure to Waterfall, Agile and Scrum models.
• Strong experience on implementation of data processing onSpark -Core usingSpark SQL and Sparkstreaming.
• Hands-on experience on fetching the live stream data from Webservers and integrate with Kafka and process with spark core.
• Experience with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• DevelopedSpark Streaming applications for Real Time Processing.
• Expertise in writingSpark RDD transformations, actions, Data Frame's, case classes for the required input data and performed the data transformations usingSpark-Core.
• Experience in usingSpark-SQL with various data formats like JSON, Parquet.
• Experience in with HadoopDistributions like Cloudera, Hortonworks and Amazon's EMR.
• Written and implemented custom UDF's in Pig for data filtering.
• Expertise in writing Hive and PIG queries for data analysis to meet the business requirements.
• Hands-on experience in using Impala for data analysis.
• Hands-on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.
• Experience in NoSQL databases like HBase, Cassandra and MongoDB.
• Good noledge in using job scheduling and monitoring tools like Oozie and Zookeeper.
• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Hands on experience on build tools like Maven, Log4j, Junit and Ant.
• Extensive experience with Databases such as Oracle, MySQL, MS-Sql and PL Sql Script.
• Experience in using IDEs like Eclipse and NetBeans.
• Working experience with Linux lineup like Red hat Enterprise Linux.
• Comprehensive noledge of Software Development Life Cycle (SDLC).
• Strengths include handling variety of software systems, capacity to learn and adapt to new technologies, amicable team player and curriculum focused with strong personal, technical and communication skills.

TECHNICAL SKILLS

Big Data Ecosystem: HDFS and Map Reduce, Pig, Hive, Impala, YARN, HUE, Oozie, Zookeeper, Apache Spark, Apache STORM, Apache Kafka, Sqoop, Flume.

Operating Systems: Windows, Ubuntu, RedHat Linux, Unix

Programming Languages: C, Java, SCALA

Scripting Languages: Shell Scripting, Java Scripting

Databases: Oracle 11g/10g/9i, MySQL, DB2, MS-SQL Server, SQL, PL-SQL, Teradata

NoSQL Databases: HBase, Cassandra, and MongoDB

Hadoop Distributions: Cloudera, Hortonworks

Build Tools: Ant, SVN, Maven, Git, GitHub

Development IDEs: NetBeans, Eclipse IDE

Web Servers: Web Logic, Web Sphere, Apache Tomcat 6

Cloud: AWS, Azure

Packages: Microsoft Office, putty, MS Visual Studio

PROFESSIONAL EXPERIENCE:

Sr. Hadoop-Spark Developer

Confidential, Mason, Ohio

Responsibilities:

• Extracted the data from RDBMS into HDFS using Sqoop.
• Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing.
• Used Apache Kafka for importing real time network log data into HDFS.
• Used Flume to collect, aggregate and store the web log data from different sources like web servers, mobile and network devices and pushed into HDFS.
• Created and worked Sqoop jobs with incremental load to populate Hive External tables.
• Implemented MapReduce programs on log data to transform into structured way to find user information.
• Extensive experience in writing Pig scripts to transform raw data from several data sources into forming baseline data.
• Developed UDF functions for Hive and wrote complex queries in Hive for data analysis.
• Export the analyzed data to relational databases using Sqoop for visualizations and to generate reports for the BI team.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
• Involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS.
• Extensively used ETL processes to load data from flat files into the target database by applying business logic on transformation mapping for inserting and updating records when loaded.
• Worked on data serialization formats for converting complex objects into sequence bits by using Avro and ORC file formats.
• Used Hadoop with the AWS EC2 using certain instances to gather and analyzing the data log files.
• Created data partitions on large data sets in S3. Analyzed the SQL scripts and designed the solution to implement using Scala.
• Developed analytical component using Scala, Spark and Spark Stream.
• Development of complex Hive scripts to transform raw data from the staging area.
• Designed and developed Hive tables to store staging and historical data.
• Created Hive tables as per requirement, internal and external tables are defined with appropriate static and dynamic partitions, intended for efficiency.
• Processed large data sets utilizing Hadoop cluster. The data that are stored on HDFS were preprocessed/validated using Pig and tan processed data was stored into Hive warehouse which enabled Business analysts to get the required data from Hive.
• Experience in using ORC file format with Snappy compression for optimized storage of Hive tables.
• Solved performance issues in Hive scripts with understanding of Joins, Group and aggregation.
• Developed Oozie workflow for scheduling and orchestrating the ETL process.
• Involved in migrating MapReduce jobs into Spark jobs and used Spark SQL and Data Frames API to load structured and semi-structured data into Spark clusters.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.

Environment: Apache Hadoop, HDFS, MapReduce, Sqoop, Flume, Pig, Hive, HBase, Oozie, Scala, Spark, Spark Streaming, Kafka, Linux.

Hadoop-Spark Developer

Confidential, Dallas, TX

Responsibilities:

• Responsible for Writing MapReduce jobs to perform operations like copying data on HDFS and defining job flows on EC2 server, load and transform large sets of structured, semi-structured and unstructured data.
• Developed a process for Scooping data from multiple sources like SQL Server, Oracle and Teradata.
• creating the new subscriptions in Azure and giving the necessary privileges as needed.
• Responsible for creation of mapping document from source fields to destination fields mapping.
• Developed a shell script to create staging, landing tables with the same schema like the source and generate the properties which are used by Oozie jobs.
• Developed Oozie workflow's for executing Sqoop and Hive actions.
• Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.
• Used Microsoft Azure for building the applications and for building, testing, deploying the applications.
• Performance optimizations on Spark/Scala. Diagnose and resolve performance issues.
• Responsible for developing Python wrapper scripts which will extract specific date range using Sqoop by passing custom properties required for the workflow.
• Developed scripts to run Oozie workflows, capture the logs of all jobs that run on cluster and create a metadata table which specifies the execution times of each job.
• Developed Hive scripts for performing transformation logic and loading the data from staging zone to final landing zone.
• Worked on Parquet File format to get a better storage and performance for publish tables.
• Involved in loading transactional data into HDFS using Flume for Fraud Analytics.
• Developed Python utility to validate HDFS tables with source tables.
• Designed and developed UDF'S to extend the functionality in both PIG and HIVE.
• Import and Export of data using Sqoop between MySQL to HDFS on regular basis.
• Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.
• Involved in using CA7 tool to setup dependencies at each level (Table Data, File and Time).
• Automated all the jobs for pulling data from FTP server to load data into Hive tables using Oozie workflows.
• Involved in developing Spark code using Scala and Spark-SQL for faster testing and processing of data and exploring of optimizing it using Spark Context, Spark-SQL, Pair RDD's, Spark YARN.
• Migrating the needed data from Oracle, MySQL in to HDFS using Sqoop and importing various formats of flat files in to HDFS.

Environment: MapReduce, HDFS, Hive, Java, Pig, Linux, XML. HBase, Zookeeper, Kafka, Sqoop, Flume, Oozie.

Hadoop Developer

Confidential | Fort Mill, SC

Responsibilities:

• Involved in designing and defining the different ways of data analytics concept using Hadoop.
• Involved in implementation of the ad-clicks (from social networking sites) based data analytics for a keyword.
• Crawled the public posts from Facebook and Twitter.
• Implemented the MapReduce programs, Hive/Impala queries to analyze sales pattern and customer satisfaction index as per the requirement defined by the Data Science team.
• Involved in sizing & scaling the infrastructure requirements by analyzing the daily data storing into the Hadoop system.
• Involved in sqooping the data from RDBMS to HDFS and vice versa by using the Sqoop tool.
• Worked in doing the performance analysis and optimization by adopting/deriving at appropriate design patterns of the MapReduce jobs by analyzing the me/O latency, reduce time etc.
• Well versed with the agile methodology.
• Storing the data to the HDFS via Flume from the various sources like social sites, mobile applications, and from the different websites.
• Created the automated workflows using oziee for the jobs for extracting the data from different Data Sources like MySQL to pushing the result set data to HDFS.

Environment: Hadoop, HBase, HDFS, MapReduce, Pig, Sqoop, Java, Cloudera Manager.

Hadoop Developer

Confidential

Responsibilities:

• Implemented CDH3 Hadoopcluster on RHEL.
• Design and develop a daily process to do incremental import of raw data from Oracle into Hive tables using Sqoop.
• Launching and Setup of Hadoop Cluster which includes configuring different components of HADOOP.
• Hands on experience in loading data from UNIX file system to HDFS.
• Cluster coordination services through Zookeeper.
• Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
• Involved in creating Hive tables, loading data and running hive queries in those data.
• Extensive Working noledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
• Involved in writing optimized Pig Script along with involved in developing and testing Pig Latin Scripts.
• Working noledge in writing Pig's Load and Store functions.

Environment: Apache Hadoop, MapReduce, HDFS, RHEL, Zookeeper, Sqoop, Hive, Pig, Oozie, Java, Eclipse, JSP, Servlets, Oracle.

Java Developer

Confidential

Responsibilities:

• Core Javacoding and development using Multithreading and Design Patterns.
• Designed dynamic user interfaces using AJAX and jQuery to retrieve data without reloading the page and send asynchronous request.
• Developed presentation layer using JSP, HTML, CSS and client validation using JavaScript.
• Developed Servlets and JSP based on MVC pattern using Struts framework.
• Created Action Classes, Form Beans, and Model Objects for the application using Model View Controller (MVC) approach.
• Involved in the integration of spring for implementing Dependency Injection.
• Created connections to database using Hibernate Session Factory, using Hibernate APIs to retrieve and store data to the database with Hibernate transaction control.
• Working on a banking application to create JSON webservice by calling a SOAP service.
• Hands on experience with Web Services including REST and SOAP.
• Optimized SQL queries used in batch processing.
• Extensively written unit test cases using JUnit framework.
• Used JIRA tool for tracking stories progress and follow agile methodology.
• Developed the application using NetBeans as the IDE and used its features for editing, debugging, compiling, formatting, build automation and SVN.
• Used Gradle tool for building and deploying the Web applications in Jboss.
• For Bulk Order Processing, Implemented Functionality to Read Input Data from MS-Excel Files using Javaand JXL API.

Environment: Core Java, Multithreading, Jdk, JDBC, Servlets, JSP, Struts, Hibernate, Spring, Web Services, JSP, jQuery, JSON, AJAX, Html, CSS, JavaScript, log4j, SQL Server, Junit, Gradle, Jboss Server, GIT, NetBeans, DOJO, UNIX, Waterfall.

Java Developer

Confidential | Hyderabad

Responsibilities:

• Created Stored procedures using PL-SQL for data modification (Using DML insert, update, delete) in Oracle.
• Involved in Requirement Analysis, Development and Documentation.
• Used MVC architecture (Jakarta Struts framework) for Web tier.
• Participated in developing form-beans and action mappings required for struts implementation and validation.
• Development of front-end screens with JSP Using Eclipse.
• Involved in Development of Medical Records module. Responsible for development of the functionality using Struts
• Coding for DAO Objects using JDBC (using DAO pattern).
• XML and XSDs are used to define data formats.
• Implemented J2EE design patterns value object singleton, DAO for the presentation tier, business tier and Integration
• Involved in Bug fixing and functionality enhancements.
• Designed and developed excellent Logging Mechanism for each order process using Log4j.
• Involved in writing SQL Queries.
• Involved in Check-in and Checkout process using CVS.
• Created SAP Business Objects Reports.
• Developed additional functionality in the software as per business requirements.
• Involved in requirement analysis and complete development of client-side code.
• Followed Sun standard coding and documentation standards.
• Participated in project planning with business analysts and team members to analyze the Business requirementsand translated business requirements into working software.
• Developed software application modules using disciplined software development process.

Environment: Java, JSP, JDBC, HTML, JavaScript, Tomcat 5.5 and MySQL 5.5, Eclipse.