SUMMARY

• Around 8+ years of professional IT experience including 5+ years in Big Data ecosystem related technologies.
• Expertise in Big Data technologies a consultant, proven capability in project based team work and also as an individual developer with goodcommunications skills.
• Hands - on experience with major components in Hadoop Ecosystem like Map Reduce, HDFS, YARN, Hive, Pig, HBase, Sqoop, Oozie, Cassandra, Impala and Flume.
• Knowledge in installing, configuring, and using Hadoop ecosystem components like Hadoop Map Reduce, HDFS, HBase, Oozie, Hive, Sqoop, Pig, spark, kafka, storm, Zookeeper and Flume
• Experience with new Hadoop 2.0 architecture YARN and developing YARN Applications on it
• Worked on Performance Tuning to Ensure that assigned systems were patched, configured and optimized for maximum functionality and availability. Implemented solutions that reduced single points of failure and improved system uptime to 99.9% availability
• Experience with distributed systems, large-scale non-relational data stores and multi-terabyte data warehouses.
• Firm grip on data modeling, database performance tuning and NoSQL map-reduce systems
• Experience in managing and reviewing Hadoop log files
• Real time experience in Hadoop/Big Data related technology experience in Storage, Querying, Processing and analysis of data
• Analyzed large amounts of data sets using Pig scripts and Hive scripts.
• Developed Apache Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.
• Exploring with various modules of Spark and working with Data Frames, RDD and Spark.
• Worked on multi clustered environment and setting up cloudera Hadoop ecosystem.
• Performed importing and exporting data into HDFS and hive using Scoop
• Experience in processing semi-structured and unstructured datasets
• Hands on experience working on shell scripting.
• Responsible for setting up processes for Hadoop based application design and implementation
• Experience in managing HBase database and using it to update/modify the data
• Experience in running MapReduce and Spark jobs over YARN
• Worked in fast pace environment.
• Extending Hive and PIG core functionality by writing UDFs.
• Used Oozie engine for creating workflow and coordinator jobs that schedule and execute various Hadoop jobs such as MapReduce Jobs, Hive, Pig and sqoop operations.
• Used Hive to create tables in both delimited text storage format and binary storage format.
• Hands-on experience in complete project life cycle (design, development, testing and implementation) of Client Server and Web applications
• Solid experience in writing complex SQL queries. Also, experienced in working with NOSQL databases like Cassandra 2.1.
• Excellent knowledge on Hadoop Architecture and ecosystems such as HDFS, Hive, Pig, Sqoop, Job Tracker, Task Tracker, Name Node, Data.
• Extending Hive and Pig core functionality by writing custom UDFs.
• Experience in Object Oriented Analysis and Design (OOAD) and development of software using UML Methodology.
• Hands on experience in Agile and scrum methodologies
• Hands on experience on JIRA.
• Hands on experience on SVN.
• Hands-on experience with Java.

TECHNICAL SKILLS

Big Data: HDFS, MapReduce, Hive, Pig, Zookeeper, Apache Spark, Core, Yarn, Spark SQL and Dataframes, Scala

Utilities: Sqoop, Flume, Oozie.

No SQL Databases: Hbase,Cassandra

Languages: C, C++, Java, J2EE, PL/SQL, MR, Pig Latin, HiveQL, Unix shell scripting and Scala

Operating Systems: Sun Solaris, RedHat Linux, Ubuntu Linux and Windows XP/Vista/7/8, Linux

Web Technologies: HTML, DHTML, XML, HTML5, CSS.

Databases and Data warehousing: Teradata, DB2, Oracle 9i/10g/11g, SQL Server, MySQL

Tools: and IDE: HP Alm, JDeveloper, SVN,JIRA,Bit Bucket.

Methodologies: Agile Software Development, waterfall

PROFESSIONAL EXPERIENCE

Confidential, Chicago,IL

Hadoop Developer

Responsibilities:

• Worked on a live24 node cluster working on HDP 2.2.
• Workedwith Sqoop (version 1.4.3)jobs with incremental loadto populate Hawq External tables to Internal table.
• Importing and exporting data job's, to perform operations like copying data from HDFS and to HDFS using Sqoop.
• Worked with Spark core, Spark Streaming and spark SQL modules of Spark.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
• Worked with Pig, HBase, NoSQL database HBASE and Sqoop, for analyzing the Hadoop cluster as well as big data.
• Very good understanding ofPartitions, bucketingconcepts in Hive and designed bothManaged and Externaltables in Hive to optimize performance.
• CreatingHive tablesand working on them for data analysis in order to meet the business requirements.
• Developed a data pipeline using Spark and Hive to ingest, transform and analyzing data.
• Created external and internal tables using HAWQ.
• Worked on data cleansing in order to populate into hive external table and internal tables.
• Experience in usingSequence files, RCFile, AVRO and HARfile formats.
• Hands on Experience writing PIG Scripts to Tokenized sensitive information using PROTEGRITY.
• Hands on experience with pivotal Hawq.
• Supporting and building the Data Science team projects on to Hadoop
• Used FLUME to dump the application server logs into HDFS.
• Automating backups by shell for Linux to transfer data in S3 bucket.
• Worked with data modeling teams. Created conceptual, logical & physical models.
• Experience in working with NoSQL database HBASE in getting real time data analytics.
• Hands on experience working as production support Engineer.
• Worked on RCA documentation. In case of any production failures.
• Automated incremental loads to load data into production cluster.
• Ingested the data from various file system to HDFS using Unix command line utilities.
• Hands on experience in moving data from one cluster to another cluster using DISTCP.
• Hands on experience working on JIRA.
• Hands on experience using HP ALM.Created test cases and uploaded into HP ALM.
• Scheduled Jobs on crontab
• Experience in reviewing Hadoop log files to detect failures.
• Worked on EPICuser stories and delivered on time.
• Worked on data ingestion part for Malicious intent model. Automated daily incremental jobs that can run on daily basis.
• Hands on experience in Agile and scrum methodologies

Environment: Hadoop, MapReduce,AWS, HDFS, Hive, HBase, Sqoop, Pig, Flume, Oracle 11/10g, DB2, Teradata, MySQL, Hawq, PL/SQL, Java, Linux, Shell Scripting, SQL Developer, HP ALM.

Confidential, NYC, New York

Hadoop Developer

Responsibilities:

• Worked on a live90 nodes Hadoop clusterrunningCDH4.4
• Worked with highlyunstructured and semi structured data of 90 TBin size (270 TB)
• Extracted the data from Teradata into HDFS usingSqoop.
• Workedwith Sqoop (version 1.4.3)jobs with incremental loadto populate Hive External tables.
• Extensive experience in writingPig (version 0.10)scripts to transform raw datafrom several data sources into forming baseline data.
• Experience in Amazon AWS services such as EMR, EC2, S3, CloudFormation, RedShift which provides fast and efficient processing of Big Data.
• Created data lake on amazon s3
• Implemented scheduled downtime for non-prod servers for optimizing AWS pricing.
• DevelopedHive(version 0.10) scripts for end user / analyst requirements to perform ad hoc analysis
• Very good understanding ofPartitions, Bucketingconcepts in Hive and designed bothManaged and Externaltables in Hive to optimize performance
• Solved performance issuesin Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
• DevelopedUDFsin Java as and when necessary to use in PIG and HIVE queries
• Experience in usingSequence files, RCFile, AVRO and HARfile formats.
• DevelopedOozieworkflow for scheduling and orchestrating the ETL process
• Worked with the admin team in designing and upgrading CDH 3 to CDH 4 Environment Cluster
• Very good experience with both MapReduce 1 (Job Tracker) and MapReduce 2 (YARN) setups
• Worked on Performance Tuning to Ensure that assigned systems were patched, configured and optimized for maximum functionality and availability. Implemented solutions that reduced single points of failure and improved system uptime to 99.9% availability.
• Very good experience in monitoring and managing the Hadoop cluster usingCloudera Manager.
• Experience with professional software engineering practices for the full software development life cycle including coding standards, source control management control and build processes.
• Implemented best income logic using Pig scripts and UDFS.
• Extracted files from CouchDB through Sqoop and placed in HDFS and processed.
• Experience in reviewing Hadoop log files to detect failures.
• Worked on Hive for exposing data for further analysis and for generating transforming files from different analytical formats to text files.
• Imported data fromMySQLserver and other relational databases to Apache Hadoop with the help ofApache Sqoop.
• CreatingHive tablesand working on them for data analysis in order to meet the business requirements.
• Evaluated business requirements and prepared detailed specification’s that follow project guidelines required to develop written programs.
• Responsible for building scalable distributed data solutions using Hadoop.

Environment: Hadoop, MapReduce, HDFS, Hive, HBase, Sqoop, Pig, Flume, Oracle 11/10g, DB2, Teradata, MySQL, Eclipse, PL/SQL, Java, Linux, Shell Scripting, SQL Developer, SOLR.

Confidential, Tampa, FL

Hadoop Developer

Responsibilities:

• Worked on a live90 nodes Hadoop clusterrunningCDH4.1
• Worked with highlyunstructured and semi structured data of 120 TBin size (360 TB)
• Developed hive queries on data logs to perform a trend analysis of user behavior on various online modules.
• Developed the Pig UDF'S to pre-process the data for analysis.
• Involved in the setup and deployment of Hadoop cluster.
• Developed Map Reduce programs for some refined queries on big data.
• Involved in loading data from UNIX file system to HDFS.
• Implemented AWS solutions using E2C, S3, RDS, EBS, Elastic Load Balancer, Auto scaling groups,
• Loaded data into HDFS and extracted the data from MySQL into HDFS using Sqoop.
• Exported the analyzed data to the relational databases using Sqoop and generated reports for the BI team.
• Managing and scheduling jobs on a Hadoop cluster using Oozie.
• Along with the Infrastructure team, involved in design and developed Kafka and Storm based data pipeline
• Designed and configured Kafka cluster to accommodate heavy throughput of 1 million messages per second. Used Kafka producer 0.6.3 API's to produce messages.
• Provided daily code contribution, worked in a test-driven development.
• Installed, Configured Talend ETL on single and multi-server environments.
• Developed Merge jobs in Pythonto extract and load data into MySQL database.
• Created and modified several UNIX shellScripts according to the changing needs of the project and client requirements. Developed UNIX shellscripts to call Oracle PL/SQL packages and contributed to standard framework.
• Developed Simple to complex Map/reduce Jobs using Hive.
• Implemented Partitioning and bucketing in Hive.
• Mentored analyst and test team for writing Hive Queries.
• Involved in setting up of HBase to use HDFS.
• Extensively used Pig for data cleansing.
• Loaded streaming log data from various webservers into HDFS using Flume.
• Along with the Infrastructure team, involved in design and developed Kafka and Storm based
• Performed benchmarking of the No-SQL databases, Cassandra and Hbase streams.
• Implemented Spark using Scala and SparkSQL for faster testing and processing of data.
• Knowledgeable of Spark and Scala mainly in framework exploration for transition from Hadoop/MapReduce to Spark.
• Supported in setting up QA environment and updating configurations for implementing scripts With Pig and Sqoop.
• Involved in collecting and aggregating large amounts of log data using Apache Flumeand staging data in HDFS for further analysis.
• Configured Flume to extract the data from the web server output files to load into HDFS.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.

Environment: Unix Shell Scripting, Python, Oracle 11g, DB2, HDFS, Kafka, Storm, Spark, ETL, 1Java (jdk1.7), Pig, Linux, Cassandra, MapReduce, Ms Access,Toad,SQL,Scala, MySQL Workbench, XML, No-SQL, MapReduce, SOLR, HBase, Hive,Sqoop, Flume, Talend, Oozie

Confidential, New Jersey

Hadoop Developer

Responsibilities:

• Worked on a live 110 nodes Hadoop cluster running CDH4.0
• Processed data into HDFS by developing solutions analyzed the data using MapReduce, pig Hive and produce summary results from Hadoop to downstream systems
• Involved in development of MapReduce job using HiveQL Statements
• Work closely with various levels of individuals to coordinate and prioritize multiple projects, estimate scope, schedule and track projects throughout SDLC
• Worked in Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs in java for data cleaning and processing.
• Involved in MapReduce job using HiveQL query for data stored in HDFS
• Written Storm topology to accept the events from Kafka producer and emit into Cassandra DB.
• Experienced in managing and reviewing Hadoop Log files.
• Designed a data warehouse using Hive
• Handling structured, semi structured and unstructured data
• Developed simple to complex MapReduce jobs using Hive and Pig.
• Extensively used pig for data cleansing
• Created partitioned tables in Hive.
• Managed and reviewed Hadoop log files.
• Cluster coordinating services through ZooKeeper.
• Mentored analyst and test team for writing Hive Queries.
• Designed and configured Kafka cluster to accommodate heavy throughput of 1 million messages per second. Used Kafka producer 0.6.3 API's to produce messages.
• Troubleshooting, debugging & fixing Talendspecific issues, while maintaining the health and performance of the ETL environment.
• Developed Simple to complex Map/reduce Jobs using Hive.
• Implemented Partitioning and bucketing in Hive.
• Mentored analyst and test team for writing Hive Queries.
• Involved in setting up of HBase to use HDFS.
• Supports and assists QA Engineers in understanding, Testing and troubleshooting.
• Exported and analyzed data to the relational databases using sqoop for visualization and to generate reports for the BI team
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and Pre-processing with Pig.
• Involved in the database migrations to transfers to migrations to transfer data from one database to other and other and complete virtualization of many client application.
• Developed the pig UDF’S to pre-process the data for analysis.
• Created HBase tables to store variable data formats of data coming from different portfolios.
• Used sqoop widely in order to import data from various systems/sources (like MYSQL) into HDFS

Environment: Hadoop, MapReduce, HDFS, Hive, HBase, Sqoop, Pig, Teradata, MySQL, Shell Scripting, Kafka, Cassandra.

Confidential, Houston, Texas.

Hadoop Developer

Responsibilities:

• Worked on a live90 nodes Hadoop clusterrunningCDH3.0
• Worked with highlyunstructured and semi structured data of 110 TBin size.
• Developed multiple MapReduce jobs in java for data cleaning and pre-processing.
• Designed and developed Oozie workflows for automation jobs.
• Mainly working on handling big data analytics and infrastructure of Hadoop, Hadoop, MapReduce.
• Got good experience with NoSQL database.
• Performed MapReduce programs those are running on the cluster.
• Created HBase table to store variable data formats of data coming from different portfolios.
• Implemented best income logic using Pig scripts
• Moving data from oracle to HDFS and vice-versa using sqoop.
• Developed Hive queries and UDF’s to analyze/Transform the data in HDFS.
• Developed Hive Scripts for implementing control tables logic in HDFS.
• Designed and implemented partitioning buckets in Hive.
• Developed pig scripts to convert the data from Avro to Textfile format.
• Designed and Developed Read Lock Capabilities In HDFS
• Involved in end-to-end implementation of ETL logic
• Used Hive to process data and Batch data filtering. Used Spark for any other value centric data filtering.
• Worked extensively with Flumefor importing data from various webservers to HDFS.
• Worked on Large-scale HadoopYARN cluster for distributed data processing and analysis using Sqoop, Pig, Hive, Impala and NoSQL databases.
• Develop Hadoop data processes using Hive and/or Impala.
• Designed and implemented Spark test bench application to evaluate quality of recommendations made by the engine.
• Monitored and identified performance bottlenecks in ETL code. Worked on data utilizing a Hadoop and Zookeeper, aiding in the development of specialized indexes for performant queries on big data implementations.
• Worked on Virtual Data Optimizer (VDO) usingHIOPS Compression
• Worked on deploying Hadoop cluster with multiple nodes and different big data analytic tools including Pig, Hbase database and Sqoop.
• Got good experience withNoSQLdatabase.

Environment: Hadoop, HDFS, Hive, Pig, HBase, Python, Big Data, Oozie, Sqoop, Scala, Kafka, Flume, Impala, Zookeeper, MongoDB, MapReduce, Cassandra, Scala, Linux, NoSQL, MySQL Workbench, Java 6,Eclipse, Oracle 10g, PL/SQL, SQL*PLUS.

Confidential, Cincinnati, OH

Java Developer

Responsibilities:

• Involved in gathering business requirements, analyzing the project and created UML diagrams such as Use Cases, Class Diagrams, Sequence Diagrams and flowcharts for the optimization Module using Microsoft Visio.
• Designed and developed Optimization UI screens for Rate Structure, Operating Cost, Temperature and Predicted loads using JSF myfaces, JSP, JavaScript and HTML.
• Configured faces-config.xml for the page navigation rules and created managed and backing beans for the Optimization module.
• Developed JSP web pages for rate Structure and Operating cost using JSF HTML and JSF CORE tags library.
• Designed and developed the framework for the IMAT application implementing all the six phases of JSF life cycle and wrote Ant build, deployment scripts to package and deploy on JBoss application server.
• Designed and developed Simulated annealing algorithm to generate random Optimization schedules and developed neural networks for the CHP system using Session Beans.
• Integrated EJB 3.0 with JSF and managed application state management, business process management (BPM) using JBoss Seam.
• Used Test driven approach for developing the application and Implemented the unit tests using PythonUnit test framework.
• Wrote AngularJS controllers, views, and services for new website features.
• Developed Cost function to calculate the total cost for each CHP Optimization schedule generated by the Simulated Annealing algorithm using EJBs.
• Implemented spring web flow for the Diagnostics Module to define page flows with actions and views and created POJOs and used annotations to map them to SQL Server database using EJB.
• Developed entire frontend and backend modules using Pythonon Django including Tasty pie Web Framework using Git.
• Wrote DAO classes, EJB 3.0 QL queries for Optimization schedule and CHP data retrievals from SQL Serverdatabase.
• Used Eclipse as IDE tool to develop the application and JIRA for bug and issue tracking
• Created combined deployment descriptors using XML for all the session and entity beans.
• Wrote JSF and JavaScript validations to validate data on the UI for Optimization and Diagnostics and Developed Web Services to have access to the external system (WCC) for the
• Developed entire frontend and backend modules using Pythonon Django Web Framework on MySQL.
• Wrote Message Driven Bean to implement the Diagnostic Engine and configured the JMS queue details and involved in performance tuning of the application using JProbe and JProfiler.
• Designed and coded application components in an Agile environment utilizing a test driven development approach.
• Skilled in test driven development and Agile development.
• Created technical design document for the Diagnostics Module and Optimization module covering Cost function and Simulated Annealing approach.
• Involved in code reviews and performed version guidelines.

Environment: Java 1.5, J2EE, Microsoft Vision, Python, EJB 3.0, JSP, JSF, JBoss Seam, JIRA, Webservices, JMS, JavaScript, AngularJs, HTML, ANT, Agile, JUnit, JBoss 4.2.2, MS SQL Server 2005, My ECLIPSE 6.0.1.

Confidential

Java/J2EE Developer

Responsibilities:

• Analysis of system requirements and development of design documents.
• Involved in various client implementations.
• Development of Spring Services
• Development of persistence classes using Hibernate framework.
• Development of SOA services using Apache Axis web service framework.
• Development of user interface using Apache Struts2.0, JSPs, Servlets, JQuery and Java Script.
• Developed client functionality using ExtJS.
• Development of JUnit test cases to test business components.
• Extensively used Java Collection API to improve application quality and performance.
• Vastly used Java 5 features like Generics, enhanced for loop, type safe etc.
• Providing production support and enhancements design to the existing product.

Environment: s: Java 1.5, SOA, Spring, ExtJS, Struts 2.0, Servlets, JSP, GWT, JQuery, JavaScript, CSS, Web Services, XML, Oracle,AngularJS, Web logic Application Server, Eclipse, UML, Microsoft Vision.