PROFESIONAL SUMMARY:

• Over 7 years of professional IT experience in Software Development Life Cycle including requirement gathering, designing developing and testing web - based applications and enterprise applications using OOAD, OOPS, JAVA/J2EE Technologies.
• Over 3 years of experience of Hadoop Ecosystems such as HDFS, MAP REDUCE, SPARK, HIVE, PIG, SQOOP, HABSE, FLUME and OOZIE.
• In depth understanding of Hadoop Architecture and various components such as HDFS, YARN, Resource Manager, Node Manager, Name Node, Data Node.
• Expertise in writing Spark RDD transformations, actions, Data Frame's, case classes for the required input data and performed the data transformations using Spark-Core.
• Experience in using Spark-SQL to access HIVE tables for fast data processing.
• Hands on experience in working on Spark-SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
• Developed Spark Streaming applications for faster data processing.
• Expertise in developing until tests using Scalatest with Funsuite
• Expertise in extending HIVE core functionality by writing custom UDFs in JAVA and PYTHON.
• Designing and creating Hive tables with dynamic partitioning and bucketing.
• Experience in importing and exporting data using SQOOP from HDFS to Relational Database Systems and vice-versa.
• Experience with NoSQL databases like HBASE and importing / exporting data from legacy systems like Main frames and RDBMS to HBase.
• Good knowledge in using job scheduling and monitoring tools like Oozie and Zookeeper.
• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency
• Hands-on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.
• Developed and executed shell scripts to automate the jobs.
• Expertise in writing Map Reduce Programs and using Apache Hadoop API for analyzing the structured and unstructured data including multiple file formats like XML, JSON, CSV.
• Experience with Testing MapReduce programs using MRUnit, Junit.
• Experienced in Cron and Incron jobs for scheduling and monitoring data movement from Mounted disk to HDFS.
• Configured Jenkins jobs to auto build with unit testing, code coverage for scala and python spark programs.
• Working closely with Test Engineers to automate complete manual testing.
• Developed Oozie workflow for scheduling and orchestrating the ETL process.
• Good working experience in implementing message brokers like Java messaging service and Kafka.
• Experienced in using Agile software methodology.
• Experience in developing distributed Enterprise and Web applications using UML, Java/J2EE, Web technologies that include EJB, JSP, Servlets, Struts, JMS, JDBC, HTML, XML, XSL, XSLT, Java Script, Spring and Hibernate(ORM), AOP(Aspect oriented programming).
• Expertise in using J2EE Application Servers like Web Logic 8.1/9.2, IBM Web Sphere 7.x/6.x and Web Servers like Tomcat 6.x /7.x/8.x
• Designed Use Case diagrams, Class diagrams, Activity diagram, Sequence diagrams, Flow Charts, and deployment diagrams using Rational Rose Tool
• Proficient in writing and handling SQL Queries, Stored Procedures, and triggers.
• Knowledge in different operating systems including Linux, Windows and scripts like Power shell script and UNIX Shell Script.

TECHNICAL SKILLS

Big Data: HDFS, Map Reduce, Spark, Hadoop Streaming, ZooKeeper, Oozie, Sqoop, Hive, Pig, HBase, Sqoop, Hue, Impala, Kafka, Flume cloudera Navigator.

Hadoop Distribution: Cloudera, Hortonworks, MapR

NoSQL: HBase, MongoDB, Aerospike

Programming Languages: Java/ J2EE, Scala, SQL, Shell Scripting, Python, C, C++

Query Languages: SQL, PL/SQL, Hive QL, pig

Web Technologies: HTML, JavaScript, CSS, XML, Servlets, SOAP.

Web/ Application: Server Apache Tomcat Server, JBOSS, IIS, Amazon Ec2

Operating system: Windows, Linux and Unix

Frameworks: Springs, MVC, Hibernate

QA Selenium: , Cucumber, SOAP UI, Junit and TestNG

DBMS / RDBMS: Oracle, SQL Server, MySQL, DB2, Teradata.

Build Tools: SBT and Maven using Jenkins

Reporting Tools: QlikView, Tableau

Version Control: SVN and Git.

PROFESSIONAL EXPERIENCE

Confidential, Chicago, IL

Hadoop Developer

Responsibilities:

• Developing Spark with Scala programs for Data Analysis on different data formats.
• Strong participation with business users from various interface systems to understand the business and data according to the requirements.
• Extracting the data from RDBMS into HDFS using Sqoop.
• Used Flume to collect, aggregate and store the web log data from different sources like web servers, mobile and network devices and pushed into HDFS.
• Implemented Daily Cron and Incron jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Spark using Oozie coordinator jobs
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data and developed POC on Spark in the initial stages of the product.
• Exploring the Spark in improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's.
• Extensively worked on the core and Spark SQL modules to perform in memory data computations.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Extending HIVE core functionality related to data with application compatible format by developing apache Hive UDFs in JAVA and Python.
• Experience in creating Hive tables with partitions and bucketing, loading Data from RDBMS using SQOOP and writing Hive queries in Command.
• Created Hive internal and external tables which are defined with partitions and bucketing appropriately using static and dynamic partitions, for efficiency.
• Used DML statements to perform different operations on Hive Tables
• Developed Hive queries for creating foundation tables from staged data.
• Worked with NoSQL databases like Hbase in creating Hbase tables to load large sets of semi structured data coming from various sources.
• Successfully migrated Legacy application to Big Data application using Hive/Spark/HBase in Production level.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and run Hive and spark jobs.
• Export the analyzed data to relational databases using Sqoop for visualizations and to generate reports for the BI team.
• Involved in designing and developing non-trivial ETL processes within Hadoop using tools like Pig, Sqoop, Flume, and Oozie.
• Developed the Apache Spark, Kafka, and HDFS integration project to do a real time data analyses.
• Used SBT as the build tool and GIT for code management
• Worked on writing RESTful web services for the application.
• Involved in writing, testing, and running Unit test using JUnit.

Environment: Apache Hadoop, SPARK, HBase, Hive, Pig, Sqoop, Oozie, NoSQL, Spark, Map-Reduce, Cloudera, HDFS, Flume, Kafka, SQL Server, GIT, JDBC, Cloudera Manager.

Confidential, Chester, PA

Hadoop Developer

Responsibilities:

• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Responsible for importing data to HDFS using Sqoop from different RDBMS servers and exporting data using Sqoop to the RDBMS servers.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Implemented MapReduce programs on log data to transform into structured way to find user information.
• Implemented Partitioning, Dynamic Partitions and Bucketing in HIVE for controlling the data distribution.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Implemented MapReduce programs on log data to transform into structured way to find user information.
• Developed reusable UDF functions for Hive and wrote complex queries in Hive for data analysis.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing
• Extensive knowledge in writing Pig scripts to transform raw data from several data sources into forming baseline data.
• Created Partitioned and Bucketed Hive tables in Parquet File Formats and then loaded data into Parquet hive tables from Avro hive tables.
• Developed Junit tests for testing MapReduce and performed testing using small sample data.
• Developed a strategy for Full load and incremental load using Sqoop
• Involved in migrating MapReduce jobs into Spark jobs and used Spark core, Spark SQL and Data Frames API using python to load structured and semi-structured data into Spark clusters.
• Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing.
• Used Apache Kafka for importing real time network log data into HDFS.
• Involved in creating workflow to run multiple hive and spark Jobs, which run independently with time and data availability using Oozie.
• Created HBase tables to store variable data formats which needs to be divided based on regions.
• Automating and scheduling the Sqoop jobs in a timely manner using Unix Shell Scripts using Cron and Incron jobs.
• Experienced in using Zookeeper and OOZIE Operational Services for coordinating the cluster and scheduling workflows.
• Developed Testing Framework using Cucumber to validate the data on UI.

Environment: Hadoop, HDFS, HBase, MapReduce, Hortonworks, Zookeeper, AVRO Files, Hive, Pig, Sqoop, Flume, Kafka, Oozie, Spark, Python, Hue, ETL, MySQL, MongoDB, Java, PHP.

Confidential, New York May

Jr Hadoop Developer

Responsibilities:

• Involved in designing and defining the different ways of data analytics concept using Hadoop.
• Involved in loading data from UNIX file system to HDFS.
• Configured various property files like core-site.xml, hdfs-site.xml, mapred-site.xml based upon the requirement.
• Worked on analyzing Hadoop cluster using different big data analytic tools including Pig, Hive and MapReduce.
• Implemented MapReduce programs on log data to transform into structured way to find required information based on business needs.
• Involved in Unit testing and delivered Unit test plans and result documents.
• Responsible for building scalable distributed data solutions using Hadoop.
• Involved in developing Hive UDFs and reused in multiple requirements.
• created Hive tables, partitions, loading data and writing hive queries.
• Implemented job flows using Cron job scheduler.
• Log data is loaded into HDFS using Flume and involved in reviewing the log files.
• Developed Web Automation Testing Framework for QA implementation.
• Define job flows using Cron job scheduler.
• Implemented J2EE design patterns like Singleton Pattern and Factory Pattern.
• Developed and utilized J2EE Web services and JMS components for messaging in Web Logic.
• Developed Hibernate configuration files and java persistence classes for Mapping DB.
• Participated in database schema design and in review meetings.
• Modified UNIX scripts to monitor systems and automation of daily tasks and customer requests.
• Developed Business components using Java Objects, Core Java, Multithreading, Servlet, JSP.
• Implemented Spring MVC framework in the application.

Environment: MapReduce, HDFS, Hive, Java, SQL, Cloudera Manager, Pig, Sqoop, Oozie, flume, UNIX, shell script, J2EE.

Confidential

Java Developer

Responsibilities:

• Involvement in all phases of the Software Development Life Cycle (SDLC).
• Involved in the team discussions regarding the modeling, architectural and performance issues.
• Designed Class diagram, Sequence diagram and analyzed requirements
• Designed functional and technical design documents
• Followed agile methodology and involved in daily SCRUM meetings, sprint planning, showcases and retrospective.
• Understand the business requirement of the project and coding in accordance with the technical design document.
• Designed and developed user interface using JSP, HTML and JavaScript.
• Developed struts action classes, action forms and performed action mapping using Struts Framework and performed data validation in form beans and action classes.
• Involved in multi-tiered J2EE design utilizing Model View Controller architecture with Spring Framework
• Implemented dependency Injection, cross cutting logics, persistence layer using Spring Framework
• Extensively used Spring Framework as the controller to handle subsequent client requests and invoke the model based upon user requests.
• Involved in system design and development in core java using Collections, multithreading.
• Implemented client side validations using Java Script.
• Worked with JavaScript DOM model to interact with HTML objects for developing present logic.
• Defined the search criteria and pulled out the record of the customer from the database. Make the required changes to the record and save the updated information back to the database.
• Responsible for coding SQL Statements and Stored procedures and triggers for back end communication using JDBC.
• Developed Webservices using SOAP, XML based protocol to exchange application level data over HTTP protocol.
• Involved in doing continuous refactoring for increasing productivity and efficiency of code.
• Developed application by following paired programming and joint code ownership methodology for increasing design quality and for decreasing defects in the system.
• Worked on Eclipse IDE to develop the application.

Environment: Java 6, J2ee, Servlet API, JSP, JSP custom tags, HTML, Springs, Ajax, SOAP, WSDL, XML, XPath, XSLT, Oracle 10g, Hibernate 3, WebLogic 9.0, Eclipse, JavaScript, CSS, SVN.

Confidential

Java Developer

Responsibilities:

• Involved in Analysis, Design, Implementation, and Testing of the project.
• Developed Use Case Diagrams, Class Diagrams and Sequence Diagrams to represent the dynamic view of the system.
• Implemented the presentation layer with HTML, XHTML, JavaScript, and CSS.
• Developed web components like JSP, Servlets, JSP tag libraries, tag handlers.
• Developed SQL queries using MySQL and established connectivity.
• Developed user and technical documentation.
• Developed the Shell scripts to trigger the Java Batch job, Sending summary email for the batch job status and processing summary.
• Developed WebServices using SOAP, XML based protocol to exchange application level data over HTTP protocol.
• Implemented client side validations and developed interactive web pages using JavaScript and JQuery.
• Application was developed in Eclipse IDE
• Configured and deployed the application to Tomcat server.
• Involved in fixing defects and unit testing with test cases using Junit.
• Followed agile methodology.

Environment: HTML, CSS, JavaScript, JQuery, JSON, ORACLE, JSP, AJAX, Servlets, Servlets Filters and Listeners, EJB, XML, Hibernate, SOAP, WSDL, Eclipse, Tomcat, and SVN.