SUMMARY

• Over 8+ years of professional IT experience including 4+ years Big data ecosystem related technologies: Hadoop HDFS, Map - reduce, Pig, Hive, Oozie, Flume, Hcatalog, Sqoop, Zookeeper, NoSQL. Expertise in Big Data technologies as consultant, proven capability in project based teamwork and also as an individual developer with good communication skills.
• Excellent understanding / noledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce programming paradigm.
• Experience in working with Hadoop clusters using Cloudera (CDH5) and HortonWorks Distributions.
• Hands on experience in installing, configuring, and using Hadoop ecosystem components like Hadoop Map Reduce(MR), HDFS, HBase, Oozie, Hive, Sqoop, Spark, Kafka, Cassandra, Scala, Pig, Knox, SPARK STREAMING and Flume.
• Hands-on development and implementation experience in Big Data Management Platform (BMP) using HDFS, Map Reduce, Hive, Pig, Oozie, Apache Kite and other eco-systems as a Data Storage and Retrieval systems.
• Performed importing and exporting data into HDFS and Hive using Sqoop.
• Experience in managing and reviewing Hadoop log files.
• Experience in analyzing data using HiveQL, Pig Latin, HBase and custom Map Reduce programs in Java.
• Extending Hive and Pig core functionality by writing UDFs.
• Good experience installing, configuring, testing Hadoop ecosystem components.
• Highly noledgeable inWriter Comparable,Writer interfaces, Mapper and Reducer abstract classes,HadoopDataObjects such asIntWritable,ByteWritable, Text objects.
• Well-experienced Mapper, Reducer, Combiner,Partitioner, Shuffling and Sort process along with Custom Partitioning for efficient Bucketing.
• Good experience in writing PIG and Hive UDF’s to solve the purpose of util classes.
• Experience in designing both time driven and data driven automated workflows using Oozie.
• Experience in installation, configuration, supporting and managing - Cloud Era’s Hadoop platform along with CDH4&5 clusters.
• Good understanding of Scrum methodologies, Test Driven Development and continuous integration.
• Expertise in working with databases likes Oracle, MS-SQL Server, Postgres and MS Access 2000 along with exposure to Hibernate for mapping an object-oriented domain model to a traditional relational database.
• Hands on experience in Agile and Scrum methodologies.
• Extensive experience in working with the Customers to gather required information to analyze, provide data fix or code fix for technical problems, and providing Technical Solution documents for the users.
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
• Worked on multiple stages of Software Development Life Cycle including Development, Component Integration, Performance Testing, Deployment and Support Maintenance.
• Have flair to adapt to new software applications and products, self-starter, have excellent communication skills and good understanding of business work flow.
• Expertise in Object-oriented analysis and programming(OOAD) like UML and use of various design patterns
• Working noledge in SQL, PL/SQL, Stored Procedures, Functions,Packages, DB Triggers and Indexes.
• Experience in design the jobs,transformations,load data sequentially & parallel for initial and incremental loads.
• Good experience in using various PDI / Kettle (Pentaho Data Integrator)steps in cleansing and load the data as per the business needs.
• Experience in configuring Zookeeper to provide Cluster coordination services.
• Experience in understanding the security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, creating and managing the realm domain.

TECHNICAL SKILLS

Languages: C,C++,JAVA, SQL and PL/SQL

Big Data Framework and Eco Systems: Hadoop, MapReduce, Hive, Pig, Kafka, Cassandra, HDFS, Zookeeper, Sqoop, Spark, Scala, Apache Crunch, Oozie and Flume

No SQL: Cassandra, HBase and MemBase

Web Technologies: JavaScript, CSS, HTML, XHTML, AJAX, XML, XSLT

Databases: Oracle 8i/9i/10g/11g, MySQL, PostGre SQL and MS-Access

Operating Systems: Windows XP/2000/NT, Linux (Red-Hat, CentOS), Machitosh, UNIX

Tools: Ant, Maven, TOAD, AgroUML, WinSCP, Putty, Lucene

IDE Tools: Eclipse 4.x, Eclipse RCP, NetBeans 6, Editplus

Version Control Tools: CVS, SVN

ETL Tools: PDI / Kettle (Pentaho Data Integration)

BI Tools: Pentaho, QlikView,Tableue, Informatica

WEB Servers: Apache Tomcat and Apache Http web server

PROFESSIONAL EXPERIENCE

Confidential

Sr. Big Data Developer

Responsibilities:

• Responsible for building scalable distributed data solutions using Hadoop.
• Experienced in migrating HiveQL into Impala to minimize query response time.
• Importing different log files using Apache Kafka into HDFS and performed data analytics using spark.
• Involved in importing the data from various data sources into HDFS using Sqoop and applying various transformations using Hive, Spark and then loading data into Hive tables.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in near real time and persist it to Cassandra.
• Collected the logs from the physical machines and the Open Stack controller and integrated into HDFS using Kafka.
• Experience in developing Kafka consumers and Kafka producers by extending low level and high level consumer and producer API’s.
• Involved in converting Hive/SQL queries into spark transformations using spark RDDs and python (pyspark).
• Developed Spark Applications by using Scala, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.
• Experience in developing various Spark Streaming API’s using python. (Pyspark).
• Developing spark code using pyspark to applying various transformations and actions for faster data processing.
• Working noledge onSpark Streaming API dat enables scalable, high-throughput, fault-tolerant stream processing of live data streams.
• Used Spark Stream processing to get data into in-memory, implemented RDD transformations, and performed actions.
• Developed various Kafka Producers and consumers for importing various transaction logs.
• Used Zookeeper to store offsets of messages consumed for a specific topic and partition by a specific Consumer Group in Kafka.
• Involved in integrating HBase with pyspark to import data into HBase and also performed some CRUD operations on HBase.
• Sqoop jobs, PIG and Hive scripts were created for data ingestion from relational databases to compare with historical data.
• Experience in working with Elastic MapReduce (EMR) and setting up environments on amazon AWS EC2 instances.
• Knowledge on handling Hive queries using Spark SQL dat integrates with Spark environment.
• Executed Hadoop/Sparkjobs on AWS EMR using programs, data stored in S3 Buckets.
• Loaded and performed some transform data into Hadoop cluster from large set of structured data using Talend Big data studio.
• Worked with different File Formats like text file, avro, orc for HIVE querying and processing based on business logic.
• Experience in pulling the data from AWS Amazon S3 bucket to Data Lake and built Hive tables on top of it and created data frames inSpark to perform further analysis.
• Involved in writing Custom Talend jobs to ingest, enrich and distribute data in Hadoop ecosystem.
• Implemented Hive, Pig UDF's to implement business logic and Responsible for performing extensive data validation using Hive.
• Implemented Daily Cron jobs dat automate parallel tasks of loading the data into HDFS using Oozie coordinator jobs.
• Involved in loading the structured and semi structured data into spark clusters using Spark SQL and Data Frames API.
• Used Pig as ETL tool to do transformations, event joins, filter and some pre-aggregations
• Used Visualization tools such as Power view for excel, Tableau for visualizing and generating reports.
• Knowledge on Machine Learning algorithms like clustering, classification and regression.
• Written multiple Map Reduce programs for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV& other compressed file formats.
• Implemented various machine learning algorithm based on business logic using Spark MLLib.
• Used Talend Open Studio for data integration and for data migration from various location across the business.
• Integrated data quality plans as a part of ETL processes using Talend.
• Experience in build scripts using Maven and did continuous system integrations like Jenkins.
• Used JIRA for bug tracking and GIT for version control.
• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: Cloudera, Map Reduce, HDFS, Pig, Scala, Hive, Sqoop, Spark, Kafka, Oozie, Java, Linux, Maven, HBase, Zookeeper, Kerberos, Tableau, Python, Talend Open studio, AWS.

Confidential, Bentonville, AR

Sr. Hadoop Developer

Responsibilities:

• Responsible for loading the customer's data and event logs from MSMQ into HBase using Java API.
• Created HBase tables to store variable data formats of input data coming from different portfolios.
• Involved in adding huge volumes of data in columns to store data in HBase.
• Used Sqoop for transferring data from HBase to HDFS and vice versa.
• Responsible for architecting Hadoop clusters with CDH4 on CentOS, managing with Cloudera Manager.
• Involved in initiating and successfully completing Proof of Concept on FLUME for Pre-Processing, Increased Reliability and Ease of Scalability over traditional MSMQ.
• Used Flume to collect the log data from different resources and transfer the data type to hive tables using different SerDe’s to store in JSON, XML and Sequence file formats.
• Managing and scheduling Jobs to remove the duplicate log data files in HDFS using Oozie workflows.
• Involved in creating Hive tables, loading data and running hive queries in those data.
• Extensive Working noledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
• Used Hive to find correlations between customer's browser logs in different sites and analyzed them to build risk profile for such sites.
• End-to-end performance tuning of Hadoop clusters and MapReduce routines against very large data sets.
• Developed the Pig UDF'S to pre-process the data for analysis.
• Monitored Hadoop cluster job performance and performed capacity planning and managed nodes on Hadoop cluster.
• Proficient in using Cloudera Manager, an end to end tool to manage Hadoop operations.

Environment: Hadoop (CDH4), Big Data, HDFS, Pig, Hive, MapReduce, Sqoop, Cloudera manager, LINUX, FLUME, HBase, Pig, Hive

Confidential, Charlotte, NC

Hadoop Developer

Responsibilities:

• Evaluated business requirements and prepared detailed specifications dat follow project guidelines required to develop written programs.
• Worked on a live 60 nodes Hadoop cluster running CDH
• Responsible for building scalable distributed data solutions using Hadoop.
• Extracted the data from Oracle into HDFS using Sqoop
• Experience in Hive-HBase integration by defining external tables in hive and pointing the HBase as data store for better performance and lower I/O.
• Created and worked Sqoop jobs with incremental load to populate Hive External Tables.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Supported Map Reduce Programs to run on cluster.
• Developed Simple to complex Map reduce Jobs using Hive and Pig
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
• Extensively used Pig for data cleansing.
• Created partitioned tables in Hive.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS
• Involved in writing Scala code on REPL by using higher order functions for most of the iterative algorithms.
• Load and transform large sets of structured, semi structured and unstructured data
• Developed Oozie workflow for scheduling and orchestrating the ETL Process.
• Good experience in monitoring and managing the Hadoop cluster using Cloudera Manager

Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, AWS, Flume, HBase, Zookeeper, Oozie work flow, Spark Scala, SQL, Sqoop, Java (Jdk 1.6), Eclipse

Confidential

JavaDeveloper

Responsibilities:

• Involved in creating use case, class, sequence, package dependency diagrams using UML.
• Also involved in analysis and requirements gathering phase.
• Developed Server side code using Servlets, JSPs running on Apache tomcat 3.0 and Enterprise Beans running on IBM Web sphere Application Server.
• Developed web pages using HTML, JSP, DHTML and CSS.
• Used JavaScript for certain form validations, submissions and other client side operations.
• Created Stateless Session Beans to communicate with the client.
• Created the database tables in Oracle 7i; created the required SQL queries and used JDBC to perform database operations.
• Developed a prototype of the application and demonstrated to business users to verify the application functionality.
• Design, develop and implement MVC Pattern based Keyword Driven automation testing framework utilizing Java, JUnit and Selenium Web Driver.
• Used automated scripts and performed functionality testing during the various phases of the application development using Selenium.
• Prepared user documentation with screenshots for UAT (User Acceptance testing).
• Developed and implemented the MVC Architectural Pattern using Struts Framework including JSP, Servlets, EJB, Form Bean and Action classes.
• Used parsers like SAX and DOM for parsing xml documents and used XML transformations using XSLT.
• Worked with QA team for testing and resolving defects.
• Used ANT automated build scripts to compile and package the application.
• Used Jira for bug tracking and project management.

Environment: Java, HTML, JSP, CSS, Servlets, JavaScript, JDBC, Oracle 7i, EJB 1.1, Apache tomcat 3.0, IBM Web sphere