SUMMARY:

• Over 7 years of professional IT experience which includes 4 years of experience in Big data ecosystem related technologies.
• Excellent understanding / knowledge of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce programming paradigm.
• Hands on experience in installing, configuring, and using Hadoop ecosystem components like Hadoop Map Reduce, HDFS, HBase, Oozie, Hive, Sqoop, Zookeeper, Pig, and Flume.
• Experience in managing and reviewing Hadoop log files.
• Experience in analyzing data using HiveQL, Pig Latin, HBase and custom MapReduce programs in Java.
• Extending Hive and Pig core functionality by writing customUDFs.
• Experience in data management and implementation of Big Data applications using Hadoop frameworks.
• Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure.
• Experience in designing, developing and implementing connectivity products that allow efficient exchange of data between our core database engine and the Hadoop ecosystem.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
• Strong experience as a Java Developer in Web/intranet, client/server technologies using Java, J2EE, Servlets, JSP, JSF, EJB, JDBC and SQL.
• Experience in Object Oriented Analysis, Design (OOAD) and development of software using UML Methodology, good knowledge of J2EE design patterns and Core Java design patterns.
• Proficient in UNIX Shell scripting.
• Excellent working knowledge of popular frameworks like Struts, Hibernate, and Spring MVC.
• Experience in Agile Engineering practices.
• Techno-functional responsibilities include interfacing with users, identifying functional and technical gaps, estimates, designing custom solutions, development, leading developers, producing documentation, and production support.
• Excellent interpersonal and communication skills, creative, research-minded, technically competent and result-oriented with problem solving and leadership skills.

TECHNICAL SKILLS:

Big Data Technologies: Hadoop, HDFS, Hive, Map Reduce, Pig, Sqoop, Flume, Zookeeper, oozie

Scripting Languages: JavaScript, HTML, XML, UNIX shell scripting

Programming Languages: C, C++, Java

Java/J2EE Technologies: Java, Java Beans, J2EE (JSP, Servlets, EJB), Struts, spring, JDBC.

DB Languages: SQL, PL/SQL

NoSQL Databases: Hbase, MongoDB

Operating Systems: LINUX, UNIX and Windows Variants

PROFESSIONAL EXPERIENCE:

Confidential - Minnetonka, MN

Hadoop Developer

Responsibilities:

• Worked on importing and exporting data from Oracle into HDFS using Sqoop for analysis, visualization and to generate reports.
• Performing full and incremental imports and created Sqoop jobs.
• Implemented multiple MapReduce jobs for data processing.
• Managing and scheduling batch Jobs on a Hadoop Cluster using Oozie.
• Scheduled the jobs with Oozie workflow engine.
• Used Zookeeper to provide coordination services to the cluster.
• Involved in loading data from edge node to HDFS using shell scripting.
• Created internal table, Externals tables in Hive, and merged the data sets using Hive joins. Involved in integration of Hive and Hbase.
• Designed and Developed Hive managed/external tables using Struct, Maps and Arrays using various storage formats.
• Implemented various performance techniques (Partitioning, Bucketing) in hive to get better performance.
• Worked with different Hadoop file formats like Parquet and compression techniques like gzip & Snappy.
• Built real time pipeline for streaming data using Kafka and Spark Streaming.
• Implemented python script to perform transformations and loaded the data into Hive.
• Worked on the Spark SQL for analyzing the data.
• Load the data into Spark RDD and performed in-memory data computation to generate the output response.
• Experienced in implementing Spark RDD transformations, actions to implement business analysis.
• Developed Pyspark code to read data from Hive, group the fields and generate XML files.
• Implemented Spark using Pyspark and SparkSQL for faster testing and processing of data.
• Involved in HDFS maintenance and loading of structured and unstructured data.

Confidential, Jacksonville FL

Hadoop Developer

Responsibilities:

• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data.
• Developed data pipeline using Flume, Sqoop, Pig and Java MapReduce to ingest behavioral data into HDFS for analysis.
• Responsible for importing log files from various sources into HDFS using Flume.
• Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
• Created POC to store Server Log data in MongoDB to identify System Alert Metrics.
• Extracted files from MongoDB through Sqoop and placed in HDFS and processed.
• Created customized BI tool for manager team that perform Query analytics using HiveQL.
• Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
• Created Hive Generic UDF's, UDAF's, UDTF's to process business logic that varies based on policy.
• Moved Relational Database data using Sqoop into Hive Dynamic partition tables using staging tables.
• Optimizing the Hive queries using Partitioning and Bucketing techniques, for controlling the data distribution.
• Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSON and XML file formats.
• Experienced with different kind of compression techniques like LZO, GZip, and Snappy.
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce Hive, Pig, and Sqoop.
• Created Data Pipeline of Map Reduce programs using Chained Mappers.
• Implemented Optimized join base by joining different data sets to get top claims based on state using Map Reduce.
• Implemented map reduce programs on MapReduceV2 to perform joins on the Map side using Distributed Cache in Java.
• Developed Unit test cases using Junit, Easy Mock and MRUnit testing frameworks.
• Experience in Upgrading hadoop cluster hbase/zookeeper from CDH3 to CDH4.
• Create a complete processing engine, based on Cloudera's distribution, enhanced to performance.
• Experienced in Monitoring Cluster using Cloudera manager.

Environment: Hadoop, HDFS, HBase, MapReduce, Java, Hive, Pig, Sqoop, Flume, Oozie, Hue, SQL, ETL, Cloudera Manager, MySQL, MongoDB.

Confidential, Princeton NJ

Hadoop Developer/Administartor

Responsibilities:

• Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Involved in Installing, Configuring Hadoop Eco System, and Cloudera Manager using CDH3 Distribution.
• Good understanding and related experience with Hadoop stack - internals, Hive, Pig and MapReduce.
• Wrote MapReduce jobs to discover trends in data usage by users.
• Involved in defining job flows.
• Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Involved in managing and reviewing Hadoop log files.
• Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
• Involved in running Hadoop streaming jobs to process terabytes of text data.
• Load and transform large sets of structured, semi structured and unstructured data.
• Responsible to manage data coming from different sources.
• Supported Map Reduce Programs those are running on the cluster.
• Involved in loading data from UNIX file system to HDFS.
• Installed and configured Hive and also written Hive QL scripts.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Experienced with working on Avro Data files using Avro Serialization system.
• Solved small file problem using Sequence files processing in Map Reduce.
• Monitor System health and logs and respond accordingly to any warning or failure conditions.
• Performed cluster co-ordination through Zookeeper.
• Used Clearcase for version control.
• Used JUnit for unit testing and Continuum for integration testing.

Environment: Hadoop, MapReduce, HDFS, Hive, PIG, LinuxJava (jdk1.6), Hadoop distribution of Cloudera, HBase, MapR, UNIX Shell Scripting, Clearcase, JUnit.

Confidential, Houston TX

Big dataDeveloper/Administrator

Responsibilities:

• Understand customer base, buying habits, promotional effectiveness, inventory management, buying decisions, by gathering and analyzing data log files through mapred jobs.
• Hadoop installation, configuration of multiple nodes in AWS-EC2 & Cloudera platform.
• Setup and optimize Standalone-System/Pseudo-Distributed/Distributed Clusters.
• Analyzing data with Hive, Pig and Hadoop Streaming.
• Build/Tune/Maintain Hive QL and Pig Scripts for reporting purpose.
• Help develop Mapreduce programs and define job flows.
• Manage and review Hadoop log files.
• Support/Troubleshoot mapreduce programs running on the cluster
• Load data from Linux/UNIX file system into HDFS.
• Load log data into HDFS using Flume.
• Install and configure Hive and write Hive UDFs.
• Create tables, load data, and write queries in Hive.
• Develop scripts to automate routine DBA tasks using Linux/UNIX Shell Scripts/Python (i.e. database refresh, backups, monitoring etc.)
• Tune/Modify SQL for batch and online processes.
• Commissioning and decommissioning the nodes.
• Manage cluster through performance tuning and enhancement.

Environment: Hortonworks Hadoop (HDFS), MapReduce, AWS, Hive, Java (JDK 1.6), Flat/XML/JSON Files, PSQL, Linux/UNIX Shell Scripting