SUMMARY:

• Offering 7+ years of overall IT experience in Application development in Java and Big Data Hadoop.
• Expertise in Hadoop, HDFS, Map Reduce and Hadoop Ecosystem including Hive, HBase, HBase - Hive, Integration, PIG, Sqoop, Flume, Oozie, Zookeeper & knowledge of Mapper/Reduce/HDFS Framework.
• Good working experience on Apache Hadoop Map Reduce programming, PIG Scripting and HDFS.
• Knowledge of NO SQL databases like MongoDB and Cassandra.
• Good understanding on Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Application Master, Resource Manager, Node Manager and MapReduce programming paradigm.
• Involved in writing Pig scripts to reduce the job execution time.
• Experienced in loading the huge data from local file system and HDFS to Hive and writing complex queries to load data into internal tables.
• Good hands-on experience in Apache Spark with Scala.
• Used Spark Streaming to divide streaming data into batches as an input to Spark engine for batch processing.
• Developed Spark SQL programs for handling different data sets for better performance.
• Hands on experience in Cloudera and Hortonworks Hadoop environments.
• Knowledge of configuring Ranger authentication for UNIX, LDAP, and Active Directory.
• Experience in Apache NIFI and also Integrating Apache NIFI and Apache Kafka.
• Collected and aggregated large amounts of data from different sources and network using Apache Nifi and stored the data into HDFS for analysis.
• Expertise in AWS services like EMR, EC2 and S3.
• Good exposure and experience in Spark, Scala, Big Data and AWS Stack.
• Experienced in loading data to Hive partitions and bucketing.
• Expertise in deploying applications on public cloud environments like AWS.
• Experience in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera distributions.
• Worked with the Apache Nifi flow to perform the conversion of Raw XML data into JSON, AVRO.
• Uses Talend Open Studio to load files into Hadoop HIVE tables and performed ETL aggregations in Hadoop HIVE.
• Developed and integrated components to run and validate batch jobs on AWS EMR.
• Designing & Creating ETL Jobs through Talend to load huge volumes of data into Cassandra, Hadoop Ecosystem and relational databases.
• Spark SQL and DataFrames for faster execution of Hive queries using Spark and AWS EMR.
• Experience in using Maven 2.0 to compile, package and deploy to the application servers.
• Skilled in data management, data extraction, manipulation, validation, and analyzing huge volume of data.
• Extensive expertise in creating and Automation of workflows using Oozie workflow Engine.
• Scheduled jobs using Oozie Coordinator, to execute jobs on specific days (excluding weekends).
• Very Good understanding of SQL, ETL and Data Warehousing Technologies.
• Extensive experience working in Oracle, SQL Server and MySQL database. Hands on experience in application development using Java and RDBMS.
• Experience in UNIX Shell scripting.
• Strong understanding of Agile Scrum and Waterfall SDLC methodologies.
• Proficient in using OOPs Concepts (Polymorphism, Inheritance, Encapsulation) etc.
• Have sound knowledge on designing data warehousing applications with using Tools like Teradata, Oracle and SQL Server.
• Hands on experience in developing the applications with Java, J2EE, JSP, EJB, SOAP, JDBC2, XML, HTML, XSD, XSLT, PL/SQL, Oracle10g.
• Strong programming skills in designing and implementation of applications using Core Java, J2EE, JDBC, JSP, HTML, Spring Framework, Spring batch framework, Spring AOP, Struts, JavaScript, Servlets.
• Java Developer with extensive experience on various Java Libraries, API's and frameworks.
• Strong knowledge of version control systems like SVN & GIT.

TECHNICAL SKILLS:

Hadoop: HDFS, Map Reduce, Spark Core, Spark Streaming, Spark SQL, Hive, Tez, Pig, Sqoop, Flume, Kafka, Oozie, and ZooKeeper.

Languages: Java, Scala, PL/SQL.

Scripts: JavaScript, Shell Scripting.

Database: Oracle 10g, MySQL. No SQL Database HBase, Cassandra, MongoDB.

Web Technologies: HTML, XML, CSS, XSLT, XHTML.

Web Servers: Apache Tomcat, JBoss.

J2EE Technologies: JDBC, AWS (S3, EC2, EMR).

Frameworks: Spring, MVC, Struts.

Tools: & IDEs: Eclipse, NetBeans, Maven, Toad, DB Visualizer.

Operating Systems: Windows, Linux (Cent OS, Ubuntu).

WORK EXPERIENCE:

Hadoop Developer/SPARK

Confidential - Beaverton, OR

Responsibilities:

• Installed and Configured Apache Hadoop clusters for application development and Hadoop tools like Hive, Pig, Oozie, Zookeeper, HBase, Flume and Sqoop.
• Implemented multiple Map Reduce Jobs in java for data cleaning and pre-processing.
• Worked in a team with 30 node cluster and increased cluster by adding Nodes, the configuration for additional data nodes was done by Commissioning process in Hadoop.
• Importing the data from the MySQL and Oracle into the HDFS using Sqoop.
• Worked with Nifi for managing the flow of data from source to HDFS.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Implemented Nifi flow topologies to perform cleansing operations before moving data into HDFS.
• Developed and implemented two Service Endpoints (end to end) in Java using Play framework, Akka server Hazelcast.
• Stored data in AWS S3 similar to HDFS. Also performed EMR programs on data stored in S3.
• Developed the Pig UDF'S to pre-process the data for analysis.
• Used Apache kafka to get the data from kafka producer which in turn pushes data to broker.
• Used Apache Nifi for ingestion of data from the Messages Queue’s.
• Experience with Test Driven Development (TDD) and acceptance- test using Behave.
• Implemented partitioning, bucketing in Hive for better organization of the data.
• Designed and built unit tests and executed operational queries on HBase.
• Built Apache Avro schemas for publishing messages to topics and enabled relevant serializing formats for message publishing and consumption.
• Implemented a script to transmit information from Oracle to HBase using Sqoop.
• Worked on migrating MapReduce programs into Spark transformations using Spark.
• Implemented SPARK batch jobs on AWS instances through Amazon Simple Storage Service (Amazon S3).
• Experience in working with NoSQL database HBase in getting real time data analytics using Apache Spark.
• Data ingesting from Teradata, Oracle, JSON files into AWS S3.
• Involved in migrating the map reduce jobs into Spark Jobs and Used Spark SQL and Dataframes API to load structured and semi structured data into Spark Clusters.
• Extensive experience in Spark Streaming (version 1.5.2) through core Spark API running Java & Scala Scripts to transform raw data from several data sources into forming baseline data.
• Used Spark API over Hadoop YARN to perform analytics on data in Hive.
• Involved in data processing using an ETL pipeline orchestrated by AWS Data Pipeline using PIG and Hive on Amazon EMR.
• Installed Oozie workflow engine to run multiple Map Reduce, HiveQL and Pig jobs.
• Triggering Spark jobs on the AWS - Elastic Map Reduce (EMR) cluster resources and perform fine tuning based on the cluster scalability.
• Used Apache Kafka and Apache Storm to gather log data and fed into HDFS.
• Developed Scala program for data extraction using Spark Streaming.
• Setting up and managing Kafka for Stream processing.
• Develop data ingestion jobs in Talend to acquire, stage, and aggregate data in technologies such as HAWQ, Hive, Spark, HDFS.
• Used Pig as a ETL tool to do Transformations, even joins and some pre-aggregations before storing data into HDFS.
• Extracted the data from Teradata into HDFS/Databases/Dashboards using SPARK STREAMING.
• Developed Scala scripts, UDFs using both Data frames/SQL and RDD/MapReduce in Spark 1.3+ for Data Aggregation, queries and writing data back to OLTP system directly or through Sqoop.
• Developed Oozie workflow for scheduling and orchestrating the ETL process.
• Created Produce, consumer and Zookeeper setup to Kafka replication.
• Experienced with batch processing of data source using Apache Spark and Elastic search.
• Experienced in implementing Spark RDD transformations, actions to implement business analysis.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Developed NoSQL database by using CRUD, Indexing, Replication and Sharing in MongoDB.
• Created and maintained Technical documentation for launching HADOOP Clusters and for executing Hive queries and Pig Scripts.

Environment: Hadoop, MapReduce, HDFS, Hive, Cloudera, Core Java, SQL, Flume, Spark, Pig, Sqoop, Apache Nifi, Oozie, impala, Scala, AWS, Hbase, Kafka, AVRO, Oracle, Unix.

Big Data Hadoop Developer

Confidential - Tampa, FL

Responsibilities:

• Responsible to manage data coming from different sources, loading of structured and unstructured data and involved in HDFS maintenance.
• Write Unix shell scripts in combination with the Talend data maps to process the source files and load into database.
• Involved in deploying the applications in AWS
• Responsible for building scalable distributed data solutions using Hadoop.
• Created Data Pipeline of Map Reduce programs using Chained Mappers.
• Implemented Hadoop YARN jobs to write data into Avro format.
• Processed Multiple Data sources input to same Reducer using Generic Writable and Multi Input format.
• Developed and executed hive queries for denormalizing the data.
• Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
• Implemented Optimized join base by joining different data sets to get top claims based on state using Map Reduce.
• Used Sqoop for various file transfers through the HBase tables for processing of data to several NoSQL DBs- Cassandra.
• Worked Big data processing of clinical and non-clinical data using Map Reduce.
• Performed data validation on the data ingested using Hadoop YARN by building a custom model to filter all the invalid data and cleanse the data.
• Extracted files from Cassandra through Sqoop and placed in HDFS and processed.
• Created Cassandra tables to load large sets of structured, semi-structured and unstructured data coming from a variety of portfolios.
• Used Flume for importing log files from various sources into HDFS.
• Load log data into HDFS using Flume, Kafka and performing ETL integrations.
• Created customized BI tool for manager team that perform Query analytics using HiveQL.
• Implemented Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive for optimized performance.
• Written Hive UDF to sort Structure fields and return complex data type.
• Worked on documentation of all Extract, Transform and Load: designed, developed, validated and deployed the Talend ETL processes for the data warehouse teams using PIG and HIVE on Hadoop.
• Involved in Installing and configuring Kerberos for the authentication of users and Hadoop daemons.
• Working on PIG Latin Scripts and UDF's while ingestion, querying, processing and analysis of Data.
• Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce Hive, Pig, and Sqoop.
• Designing data models in Cassandra and working with Cassandra Query Language (CQL).
• Implemented JMS for asynchronous auditing purposes.
• Develop data ingestion jobs in Talend to acquire, stage, and aggregate data in technologies such as HAWQ, Hive, Spark, HDFS.
• Modeled Hive partitions extensively for data separation and faster data processing and followed Pig and Hive best practices for tuning.
• Developed Hive queries to process the data and generate the data cubes for visualizing.
• Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSON and XML file formats.

Environment: Hadoop, Agile methodologies, Talend, HDFS, HBase, Cassandra, YARN, Java, Hive, Pig, Sqoop, Flume, Oozie, Hue, SQL, ETL, Hortonworks, Ambari, AVRO, Oracle, My SQL.

Big Data Analyst/Java Developer

Confidential - Kalamazoo, MI

Responsibilities:

• Installed and configured Apache Hadoop to test the maintenance of log files in Hadoop cluster.
• Installed and configured Hive, Pig, Sqoop, and Oozie on the Hadoop cluster.
• Installed and configured Hadoop MapReduce, HDFS and developed multiple MapReduce jobs in Java for data cleansing and preprocessing.
• Deployed the application jar files into AWS instances.
• Worked with Amazon Web Services (AWS) and involved in ETL, Data Integration and Migration.
• Extensively Involved in loading data from UNIX file system to HDFS.
• Involved in evaluating the business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Provided quick response to ad hoc internal and external client requests for data and experienced in creating ad hoc reports.
• Connected Tableau from client end with AWS ip addresses and view the end results.
• Responsible for building scalable distributed data solutions using Hadoop.
• Implemented Map Reduce jobs in HIVE by querying the available data.
• Used Amazon Redshift to Store and retrieve the data from data-warehouses.
• Loading data from diff servers to AWS S3 bucket and setting appropriate bucket permissions.
• Experience in using Hive and Pig as an ETL tool for event joins, filters, transformations and pre- aggregations.
• Developed PIG scripts to transform the raw data into intelligent data as specified by business users.
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig.
• Performed some unit testing for the development team within the sandbox environment.
• Used Hive and created Hive tables and also involved in writing Hive UDFs and data loading.
• Imported data into HDFS and Hive from other data systems by using Sqoop.
• Installed Oozie Workflow engine to run multiple Hive and Pig Jobs.
• Generated aggregations and groups and visualizations using Tableau.
• Developed Hive queries to process the data.
• Developed and maintain several batch jobs to run automatically depending on business requirements.

Environment: Apache Hadoop, Cloudera Manager, CDH2, CDH3 CentOS, Apache Hama, Eclipse Indigo, Java, MapReduce, Hive, Sqoop, Pig, AWS, Oozie and SQL, Struts, JUnit.

Java Developer

Confidential

Responsibilities:

• Involved in design and development phases of Software Development Life Cycle (SDLC).
• Involved in designing UML Use case diagrams, Class diagrams, and Sequence diagrams using Rational Rose.
• Followed agile methodology and SCRUM meetings to track, optimize and tailored features to customer needs.
• Developed user interface using JSP, JSP Tag libraries, and Java Script to simplify the complexities of the application.
• Implemented Model View Controller (MVC) architecture using Jakarta Struts frameworks at presentation tier.
• Developed a Dojo based front end including forms and controls and programmed event handling.
• Implemented SOA architecture with web services using JAX-RS (REST) and JAX-WS (SOAP).
• Developed various Enterprise Java Bean components to fulfill the business functionality.
• Created Action Classes which route submittals to appropriate EJB components and render retrieved information.
• Validated all forms using Struts validation framework and implemented Tiles framework in the presentation layer.
• Used Core java and object oriented concepts.
• Used Spring Framework for Dependency injection and integrated it with the Struts Framework.
• Used JDBC to connect to backend databases, Oracle and SQL Server 2005.
• Proficient in writing SQL queries, stored procedures for multiple databases, Oracle and SQL Server 2005.
• Wrote Stored Procedures using PL/SQL. Performed query optimization to achieve faster indexing and making the system more scalable.
• Deployed application on windows using IBM Web Sphere Application Server.
• Used Java Messaging Services (JMS) for reliable and asynchronous exchange of important information such as payment status report.
• Used Web Services - WSDL and REST for getting credit card information from third party and used SAX and DOM XML parsers for data retrieval.
• Implemented SOA architecture with web services using Web Services like JAX-WS.
• Used ANT scripts to build the application and deployed on Web Sphere Application Server.

Environment: Core Java, J2EE, Oracle, SQL Server, JSP, Struts, Spring, JDK, JavaScript, HTML, CSS, AJAX, JUnit, Log4j, Web Services, Windows.

Jr Java/J2EE Developer

Confidential

Responsibilities:

• Involved in specification analysis and identifying the requirements.
• Participated in design discussions for the methodology of requirement implementation
• Involved in preparation of the Code Review Document & Technical Design Document
• Designed the presentation layer by developing the jsp pages for the modules
• Developed controllers and JavaBeans encapsulating the business logic
• Developed classes to interface with underlying web services layer
• Used patterns including MVC, DAO, DTO, Front Controller, Service Locator and Business Delegate.
• Worked on Service Layer which provided business logic implementation.
• Involved in building PL\SQL queries and stored procedures for Database operations.
• Used Jasper Reports to provide print preview of Financial Reports and Monthly Statements.
• Carried out integration testing & acceptance testing
• Used JMeter to carry out performance tests on external web service calls, database connections and other dynamic resources.
• Participated in the team meetings and discussed enhancements, issues and proposed feasible solutions.

Environment: Java1.4, J2EE 1.4 Servlet, JSP, JDBC, XML, ANT, Apache Tomcat 5.0, Oracle 8i, JUnit, PL\SQL, UML, NetBeans.