SUMMARY

• 7+ years of experience in analysis, design, administration, development, deployment and maintenance of critical software and Big Data applications using Hadoop technologies.
• Good understanding of complete Software Development Life Cycle (SDLC) with working knowledge of project implementation methodologies like Agile/SCRUM.
• Hands on experience in installing, configuring and using Hadoop Ecosystem: HDFS, MapReduce, Pig, Hive, Oozie, Flume, HBase, Spark, Sqoop, Flume and Oozie.
• Experience in optimizing Map Reduce algorithms using Mappers, Reducers, combiners and partitions to deliver the best results for the large datasets.
• Skilled in configuring Zookeeper to coordinate servers in clusters & maintain consistency.
• Expertise in loading the data from the different data sources like into HDFS using Sqoop and load into partitioned Hive tables.
• Experience implementing HDFS access controls, directory, and user authorization that facilitates stable, secure access for multiple users in a large multi - tenant cluster.
• Experience in importing and exporting streaming data into HDFS using stream processing platforms like Flume and Kafka messaging system.
• Knowledge on implementing Big Data in Amazon Elastic MapReduce for processing, managing Hadoop framework dynamically scalable Amazon EC2 instances.
• Experience in AWS CloudFront, including creating and managing distributions to provide access to S3 bucket or HTTP server running on EC2 instances.
• Experience in developing data pipeline by using Kafka to store the data into HDFS.
• Experience in migrating data by using Sqoop from HDFS to Relational Database System and vice-versa according to client's requirements.
• Expertise in development of web and enterprise applications using technologies like JSP, Servlets, Struts, JDBC, XML, AJAX, SOAP & REST Services.
• Proficiency in programming with different Java IDE's like Eclipse, JDeveloper, & NetBeans.
• Experience in using of web/application servers Apache Tomcat, WebLogic and WebSphere.
• Expertise in Unit Testing, Integration Testing, System Testing and experience in preparing the Test Cases, Test Scenarios and Test plans.
• Ability to work independently as well as in a team and able to effectively communicate with customers, peers and management at all levels in and outside the organization.

TECHNICAL SKILLS

Languages: Scala, Python, R and Java

Big Data: Apache Spark, Apache SOLR, HDFS, MapReduce, Pig, Sqoop, Hive, HBase, Kafka, Flume, OOZIE, YARN

AWS Components: VPC, EC2, EBS, RedShift, CFT, EMR, CloudWatch

Databases: HBase, MongoDB, Oracle, SQL Server

Operating Systems: Windows, Unix and Linux, CentOS, Solaris

Java Technologies: Struts, J2SE, JDBC, Java Script, HTML, XML

Servers Web: Sphere, WebLogic, Tomcat

PROFESSIONAL EXPERIENCE

Confidential, Waltham, MA

Hadoop Developer

Responsibilities:

• Worked with business partners in discussing the requirements for new projects and enhancements to the existing applications.
• Responsible for Cluster maintenance, monitoring, commissioning/decommissioning data nodes, troubleshooting, managing data backups, & reviewing log files.
• Developed MapReduce programs to parse the raw data, populate tables and store the refined data in partitioned tables.
• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
• Used Spark stream processing to get data into in-memory, implemented RDD transformations, actions to process as units
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Created Hive tables and implemented partitioning, dynamic partitions, buckets and created external tables to optimize performance.
• Developed Sqoop jobs for extracting data from heterogeneous data sources.
• Developed Pig Latin Scripts to process data in a batch to perform trend analysis.
• Configured Spark Streaming to receive real time data from the Kafka and store the stream data to the database.
• Responsible for developing data pipeline by implementing Kafka producers and consumers.
• Connected to AWS EC2 using SSH and ran Spark-submit jobs.
• Responsible for developing data pipeline with AWS to extract the data from weblogs and store in the database.
• Scripted AWS environments using secured VPC, different data pipelines & Redshift cluster.
• Developed Scala functional programs for streaming data and gathered JSON and XML data and passed to Flume
• Developed some utility helper classes to get data from HBase tables.
• Written Hadoop MapReduce jobs using Java for processing data on HDFS.
• Gained familiarity with Hive joins & used HQL for querying the databases eventually leading to complex Hive UDFs.
• Responsible for continuous monitoring Elastic MapReduce cluster through AWS console.
• Involved in Data Validation and fixing discrepancies by working in coordination with the Data Integration.

Confidential, Atlanta, GA

Hadoop Developer

Responsibilities:

• Developed, reviewed and updated architecture and process documentation, server diagrams, requisition documents and other technical documents.
• Followed Agile methodology, attended daily Scrum meetings and Sprint planning meetings.
• Responsible for cluster maintenance, commissioning data nodes, cluster monitoring, troubleshooting, management & review data backups, and Hadoop log files.
• Implemented MapReduce programs to handle unstructured data like XML & JSON files and sequence files for log files
• Developed Spark SQL to load tables into HDFS to run select queries on top.
• Preformed aggregation over large amounts of log data collected using Apache Flume and staging data in HDFS for further analysis.
• Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing.
• Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.
• Scheduled periodic jobs which range from updates on MapReduce jobs to creating Ad-hoc jobs for the business users.
• Migrated Flume with Spark for real time data and developed the Spark Streaming Application with java to consume the data from Kafka and push them into Hive.
• Configured Kafka for efficiently collecting, aggregating and moving large amounts of click stream data from many different sources to HDFS.
• Developed REST-based Microservices supporting both XML and JSON to perform task such as demand response management.
• Used Maven to build/deploy Spring Boot Microservices to inner enterprise Docker registry.
• Used AWS services like EC2 and S3 for small data sets processing and storage.
• Provisioning of Cloudera Director AWS instance and adding Cloudera manager repository to scale up Hadoop Cluster in AWS.
• Monitored workload, job performance and capacity planning using Cloudera Manager.
• Used HiveQL for data analysis like creating tables and import the structured data to specified tables for reporting.
• Used Pig to perform data validation on the data ingested using scoop and flume and the cleansed data set is pushed into HBase.
• Executed speedy reviews and first mover advantages by using workflows like Oozie in order to automate the data.

Confidential, St. Louis, MO

Hadoop Developer

Responsibilities:

• Participated in software development life cycle from requirement gathering to product delivery.
• Involved in developing Hadoop Map Reduce jobs using Java Runtime Environment for the batch processing to search and match the scores.
• Loading process into the Hadoop Distributed File System (HDFS) and Pig language in order to preprocess the data.
• Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.
• Performed analysis on the unused user navigation data by loading into HDFS and writing MapReduce jobs.
• Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box such as Map-Reduce, Pig, Hive, Sqoop, Flume.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Wrote optimized Pig Script along with involved in developing and testing PIG Latin Scripts.
• Working knowledge in writing Pig's Load and Store functions.
• Developed job flows to automate the workflow for PIG and HIVE jobs.
• Created final tables in Parquet format. Use of Impala to create and manage Parquet tables.
• Implemented data ingestion & handling clusters in real time processing using Apache Kafka.
• Involve in creating Hive tables, loading with data and writing Hive queries which will run internally in map reduce way.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Worked on multithreaded middleware using socket programming to introduce whole set of new business rules implementing OOPS design and principles.
• Involved in implementing Java multithreading concepts.
• Developed several REST web services supporting both XML and JSON to perform task such as demand response management.
• Implemented the log functionality by using Log4j and internal logging API's.
• Used J-Unit for server-side testing.

Confidential, Brentwood, TN

Web Developer

Responsibilities:

• Involved in various phases of SDLC such as requirements gathering, analysis, development and testing.
• Generated the use case diagrams, Activity diagrams, Class diagrams and Sequence Diagrams in the design phase using Rational Rose.
• Developed helper classes needed following Core Java multi-threaded programming and Collection classes.
• Developed the application using Struts2 MVC framework.
• Used Struts Tiles Framework in the presentation tier. Developed Struts Action classes.
• Developed interactive web page using Struts, Hibernate, Ajax, XML, JavaScript, & jQuery.
• Used JIRA issue tracker to file the bugs.
• Used jQuery and Ajax to send the request to server side and receive the response.
• Involved in setting up Maven configuration and helping Continuous Integration (CI) Issues.
• Mapped DTOs to Oracle Database tables and Java data types to SQL data types by creating Hibernate mapping XML files.
• Wrote stored procedures for common SQL queries on Oracle database.
• Developed various Database interaction objects by implementing the DAO Patterns and used JDBC as Model Component.
• Used SAX Parser for parsing the XML document.
• Implemented Log4j by enabling logging at runtime without modifying the application binary.
• Involved in Unit Testing and Bug-Fixing and achieved the maximum code coverage using J-Unit test cases.
• Used Java Mail notification mechanism to send confirmation email to customers about scheduled payments.
• Developed Message-Driven beans in collaboration with Java Messaging Service (JMS) to communicate with merchant systems.
• Used ANT Script for building and deploying the application to Apache Tomcat server.
• Tested applications and documented results and bugs including documentation of information useful in the debugging process.