SUMMARY

• 7+ years of extensive IT experience in all phases of Software Development Life Cycle with skills in data analysis, design, development, testing and deployment of software systems.
• 3+ years of experience, working on Apache Hadoop ecosystem components like MapReduce, HDFS, HBase, Hive, Sqoop, Pig, Oozie, Zookeeper, Flume, KAFKA, NiFi, Spark with CDH distributions and EC2 cloud computing with AWS.
• Strong knowledge of HDFS Architecture and Cluster concepts.
• Good knowledge of Hadoop Architecture and various daemons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager
• Experience in Map Reduce to process data for extraction, transformation and aggregation.
• Experience in using Map Reduce Combiners and Partitioners for analyzing the big data as per the requirement to deliver the best results.
• Proficient in creating Hive and PIG UDF’s to solve business requirements.
• Expertise in writing Hive Queries & PIG Latin Scripts for data analytics.
• Knowledge on Hive data warehousing infrastructure to analyze large structured datasets.
• Worked with various file formats like sequence files, AVRO, JSON, PARQUET, DAT, RC, ORC, CSV.
• Strong knowledge in NOSQL column oriented databases like HBase, Cassandra, MongoDB, and their integration with Hadoop cluster.
• Strong SQL experience in Teradata from developing the ETL with Complex tuned queries including analytical functions
• Experience with HBase in conducting the quick look ups such as updates, inserts and deletes in Hadoop.
• Experience with Cloudera, Hortonworks distributions.
• Work experience and knowledge with cloud infrastructure like Amazon Web Services (EC2, Lambda, S3, EBS, RDS, Dynamo DB, Red shift, SQS, SNS).
• Good knowledge on managing and reviewing Hadoop log files and experience in transferring Hadoop Log files using Flume.
• Expertise in migrating data from HDFS to relational database and vice versa using SQOOP
• Good experience in handling streaming or near real - time data into HDFS using Kafka
• Work experience in converting Hive/SQL queries into Spark transformations using Java.
• Experience with Apache Spark’s Core, Spark SQL, Streaming and MlLib components.
• Excellent understanding of abstraction using Scala and Spark.
• Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
• Strong knowledge of Oozie for job workflow scheduling and monitoring.
• Experience with Oozie Workflow Engine in scheduling jobs for Map-Reduce, Pig Hive and Kafka.
• Good knowledge on performance tuning of Apache NiFi workflow to optimize the data ingestion speeds.
• Extensive knowledge about Zookeeper process for various types of centralized configurations.
• Experience in designing the zookeeper to facilitate the servers in clusters and to keep up the information consistency.
• Good knowledge in using Splunk for data monitoring and visualization.
• Good in writing Spark scripts by using Scala and Python
• Excellent Java development skills using Springs, J2SE, Servlets, Junit, MR Unit, JSP, JDBC.
• Good understanding of Agile Scrum and Waterfall SDLC methodologies.
• Strong communication, collaboration & team building skills with proficiency at grasping new Technical concepts quickly and utilizing them in a productive manner.
• Strong analytical and Problem-solving skills.

TECHNICAL SKILLS

Big Data Eco-systems: HDFS, YARN, MapReduce, Pig, Hive, Sqoop, Flume, Kafka, Oozie, Zookeeper, Spark, Scala, NiFi

Operating Systems: Windows (95/98/2000/XP/Vista/7/10), Linux (Centos, Ubuntu)

No SQL Database: HBase, Cassandra, MongoDB

Languages: C, C++, Java, SQL, Pig Latin, Hive QL, Shell scripting and Scala

Databases/Data warehouse: MySQL, SQL Server, Oracle, Teradata

Web Development: HTML, XML, JavaScript

ETL tools: Data stage, informatica

BI Tools: Tableau

Cloud Services: Amazon Web Service

PROFESSIONAL EXPERIENCE

Hadoop Developer

Confidential - Thomasville, NC

Responsibilities:

• Responsible for importing data from LINUX file system to HDFS.
• Experience in managing and reviewing Hadoop log files.
• Developed ETL code using Pig Latin as per client requirement for data cleansing and enriching.
• Scheduled Spark streaming jobs using Oozie for continuous job tracking.
• Importing and exporting data into HDFS and Hive using Sqoop from RDBMS and other NoSQL databases.
• Implemented test scripts to support test driven development and continuous integration.
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.
• Developed multiple MapReduce jobs in java for data cleaning and preprocessing.
• Created Pig Latin scripts to sort, group, join and filter the enterprise wise data.
• Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
• Optimized Hive queries to extract the customer information from HDFS
• Designed HBase Schemas based on business requirements and implemented HBase query API in java
• Worked on Hive for further analysis and for generating transforming files from different analytical formats to CSV
• Analyzed large data sets by running Hive queries and Pig scripts.
• Used AWS services like EC2 and S3 for small data sets.
• Installed Oozie workflow engine to run multiple MapReduce jobs.
• Created HBase tables to store variable data formats of data coming from different portfolios
• Used Kafka for Website activity tracking, Stream processing and for auto-scaling the backend servers based on the events throughput.
• Used Flume, Kafka to load log data into HDFS.
• Used Splunk Dashboards for monitoring the pipeline jobs in production.
• Developed analytical components using Scala, Spark and Spark SQL
• Extensively worked on Oozie and UNIX scripts for batch processing and scheduling workflows dynamically.
• Loading data from external data sources like MySQL and Cassandra for Spark applications.
• Strong understanding of Spark real time streaming and Spark SQL.
• Exported the analyzed data to the relational databases using Sqoop and Flume for visualization and to generate reports by our BI team
• Experience in managing and monitoring Hadoop cluster using Cloudera Manager Hue.
• Interact with business analysts to understand the business requirements and translate them to technical requirements

Environment: Hadoop, HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, H-Base, Kafka, Cassandra, AWS, Spark, Scala, Splunk.

Hadoop Developer

Confidential - Milwaukee, WI

Responsibilities:

• Responsible for loading data from various servers to HDFS
• Worked with Map Reduce jobs to generate reports for the number of activities created on a particular day.
• Loaded and transformed large sets of structured, semi structured data using Pig Scripts
• Optimized Hive queries to extract the customer information from HDFS.
• Involved in writing Serdes regular expressions in order to read unstructured data from various sources into Hive tables.
• Used Serdes in Hive for converting JSON format data in CSV format for Loading into tables
• Migrating the needed data from MySQL into HDFS using Sqoop and importing various formats of unstructured data from logs into HDFS using Flume.
• For queuing the message services as per the category is assigned and tracked by using Kafka
• Responsible for continuous scheduling and monitoring using Oozie.
• Involved in checking job status by accessing Ambari
• Configured Zookeeper to coordinate the servers in clusters to maintain the data consistency
• Used Cassandra to support contracts and services that are available.
• Designed workflows by scheduling Hive processes for Log file data, which is streamed into HDFS using Flume.
• Responsible for reporting the data to analysts for further tracking of trends according to various consumers.

Environment: Hadoop, HDFS, Map Reduce, Hive, HBase Flume, Sqoop, Kafka, Oozie, Zookeeper, My SQL, Tableau, Cassandra, Hortonworks

Hadoop Developer

Confidential - Bethpage, NY

Responsibilities:

• Supported MapReduce Programs that are running on the cluster.
• Used Hive QL to do analysis on the data and identify different correlations.
• Responsible for writing Pig Latin scripts and worked on tuning the performance Pig queries.
• Worked on Talend to run ETL jobs on the data in HDFS.
• Experience in creating tables, dropping and altered at run time without blocking updates and queries using HBase and Hive
• Imported data using Sqoop to load data from Oracle to HDFS on a regular basis.
• Developing scripts and batch jobs to schedule various Hadoop Programs.
• Wrote Map Reduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration
• Experience in writing Hive Queries for data analysis to meet the business requirements.
• Wrote and executed various MYSQL database queries from Python using Python-MySQL connector and MYSQL database package.
• Collected the logs data from web servers and integrated in to HDFS using Flume.
• Importing and exporting data into HDFS from Oracle Database, and vice versa using Sqoop.
• Involved in creating Hive Tables, loading the data and writing Hive Queries that will run internally in a map reduce way.

Environment: Hadoop, HDFS, HBase, Pig, Hive, MapReduce, Sqoop, Flume, Java, Oracle, Unix/Linux.

Java Developer

Confidential

Responsibilities:

• Designed the application using the J2EE design patterns such as Session Façade, Business Delegate, Service Locator, Value Object and Singleton.
• Developed presentation tier as HTML, JSPs using Struts 1.1 Framework. Used AJAX for faster page rendering.
• Developed the middle tier using EJBs Stateless Session Bean, Java Servlets.
• Entity Beans used for accessing data from the Oracle database.
• Worked on Hibernate for data persistence.
• Prepared high and low-level design documents for the business modules for future s and updates.
• Deployed the application in JBoss Application Server in development and production environment.
• Implemented CVS as Version control system.
• Code Walkthrough/ Test cases and Test Plans.
• Used ANT as build tool. Used Junit for writing Unit tests

Environment: Eclipse, HTML, Java Script, Core Java, JUnit, JSP, Servlets, JDBC, Oracle 8i, AJAX, CVS and JBoss Application Server.

Java Developer

Confidential

Responsibilities:

• Worked on Requirement analysis, gathered all possible business requirements from end users and business Analysts
• Involved in creation of UML diagrams like Class, Activity, and Sequence Diagrams using modelling tools of IBM Rational Rose
• Worked with core Java code extensively using interfaces and multi-threading techniques
• Involved in production support and documenting the application to provide and knowledge transfer to the user
• Used Log4j for logging mechanism and developed wrapper classes to configure the logs
• Used JUnit and Test cases for testing the application modules
• Developed and configured the Java beans using Spring MVC framework
• Developed the application using Rational Team Concert and worked under Agile Environment
• Developed SQL stored procedures and prepared statements for updating and accessing data from database
• Used C++ to create some libraries used in the application

Environment: C++, Java, JDBC, Servlets, JSP, Struts, Eclipse, Oracle 9i, Apache Tomcat, CVS, JavaScript, Log4J