PROFESSIONAL SUMMARY:

• Over 9 years of experience in IT industry with 4 years of experience in Big Data implementing complete Hadoop solutions along with Scala and Java/J2EE.
• Good working experience in using Apache Hadoop eco system components like Map Reduce, HDFS, Hive, Sqoop, Pig, Oozie, Flume, HBase and Zoo Keeper.
• Extensive experience working on spark in performing ETL using Spark Core, Spark - SQL and Real-time data processing using Spark Streaming.
• Expertise in working with RDD’s, Data Frames and Dataset API’s.
• Extensively worked with Kafka as middleware for real-time data pipelines.
• Writing UDFs and integrating with Hive and Pig using Java.
• Experience with Sequence files, AVRO and ORC file formats and compression.
• Experience in Hadoop Distributions: Cloudera and Hortonworks,
• Performed importing and exporting data into HDFS and Hive using Sqoop.
• Experience in job workflow scheduling tools like Oozie.
• Extensive knowledge in using SQL Queries for backend database analysis.
• Strong knowledge in NOSQL column oriented databases like HBase, Cassandra and its integration with Hadoop cluster.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems (RDBMS) and vice-versa.
• Led many Data Analysis & Integration efforts involving HADOOP along with ETL.
• Hands on experience on Enterprise Data Lake to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing, structured and unstructured data.
• Extensive experience with SQL, PL/SQL and database concepts.
• Transferred bulk data from RDBMS systems like Teradata into HDFS using Sqoop.
• Experience in analyzing data using Hive QL, Pig Latin, and custom MapReduce programs in Java.
• Well-versed in Agile, other SDLC methodologies and can coordinate with owners and SMEs.
• Worked on different operating systems like UNIX, Linux, and Windows
• Diverse experience utilizing Java tools in business, Web, and client-server environments including Java Platform, Enterprise Edition (Java EE), Enterprise Java Bean (EJB), JavaServer Pages (JSP), Java Servlets (including JNDI), Struts, and Java database Connectivity (JDBC) technologies.
• Fluid understanding of multiple programming languages, including C#, C, C++, JavaScript, HTML, and XML.
• Experience in web application design using open source MVC, Spring and Struts Frameworks.

TECHNICAL SKILLS:

Hadoop Core Services: HDFS, Map Reduce, Spark, YARN

Hadoop Distribution: Horton works, Cloudera, Apache

NO SQL Databases: HBase, Cassandra

Hadoop Data Services: Hive, Pig, Sqoop, Flume

Hadoop Operational Services: Zookeeper, Oozie

Monitoring Tools: Cloudera Manager

Cloud Computing Tools: Amazon AWS

Languages: C, Java/J2EE, Python, SQL, PL/SQL, Pig Latin, HiveQL, Unix Shell Scripting

Java & J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts

Application Servers: Web Logic, Web Sphere, JBoss, Tomcat

Databases: Oracle, MySQL, Postgress, Teradata

Operating Systems: UNIX, Windows, LINUX

Build Tools: Jenkins, Maven, ANT

Development Tools: Microsoft SQL Studio, Toad, Eclipse, NetBeans

Development methodologies: Agile/Scrum

Visualization and analytics tool: Tableau Software, Qlik View

PROFESSIONAL EXPERIENCE:

Confidential, Chicago, IL

Big data Developer

Responsibilities:

• Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO and Parquet to HDFS cluster with compressed for optimization.
• Worked on ingesting data from RDBMS sources like - Oracle, SQL Server and Teradata into HDFS using Sqoop.
• Loaded all data-sets into Hive from Source CSV files using spark and Cassandra from Source CSV files using Spark
• Created environment to access Loaded Data via spark SQL, through JDBC&ODBC (via Spark Thrift Server). Developed real time data ingestion/ analysis using Kafka / Spark-streaming.
• Configured Hive and written Hive UDF's and UDAF's Also, created Static and Dynamic with bucketing as required.
• Managing and scheduling Jobs on a Hadoop cluster using Oozie.
• Created Hive External tables and loaded the data in to tables and query data using HQL.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Developed Oozie workflow for scheduling and orchestrating the ETL process and worked on Oozie workflow engine for job scheduling.
• Managed and reviewed the Hadoop log files using Shell scripts.
• Migrated ETL jobs to Pig scripts to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Using Hive join queries to join multiple tables of a source system and load them to Elastic search tables.
• Experience in managing and reviewing huge Hadoop log files.
• Collected the logs data from web servers and integrated in to HDFS using Flume.
• Expertise in designing and creating various analytical reports and Automated Dashboards to help users to identify critical KPIs and facilitate strategic planning in the organization.
• Involved in Cluster maintenance, Cluster Monitoring and Troubleshooting
• Maintaining technical documentation for each and every step of development environment and launching Hadoop clusters.
• Worked on different file formats like Parquette, Orc, Avro, Sequence files using MapReduce/Hive/Impala.
• Worked with Avro Data Serialization system to work with JSON data formats.
• Used Amazon Web Services (AWS) S3 to store large amount of data in identical/similar repository.
• Worked with the Data Science team to gather requirements for various data mining projects.
• Wrote shell scripts for rolling day-to-day processes and it is automated.
• Involved in build applications using Maven and integrated with Continuous Integration servers like Jenkins to build jobs.
• Worked on BI tools as Tableau to create dashboards like weekly, monthly, daily reports using tableau desktop and publish them to HDFS cluster.

Environment: Spark, Spark SQL, Spark Streaming, Scala, Kafka, Hadoop, HDFS, Hive, Oozie, Pig, Sqoop, Nifi, AWS (EC2, S3, EMR), Shell Scripting, HBase, Jenkins, Tableau, Oracle, MySQL, Teradata and AWS.

Confidential, Florham Park, NJ

Big Data Developer

Responsibilities:

• Responsible for building scalable distributed data solutions using Hadoop
• Imported data using Sqoop to load data from MySQL to HDFS on regular basis from various sources.
• Written multiple MapReduce programs to power data for extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV & other compressed file formats.
• Reviewed the HDFS usage and system design for future scalability and fault-tolerance.
• Developed Hive Queries in Spark-SQL for analysis and processing the data. Used Scala programming to perform transformations and applying business logic.
• Involved in loading data from LINUX file system to HDFS.
• Loaded and transformed large sets of structured, semi structured and unstructured data in various formats like text, zip, XML and JSON.
• Defined job flows and developed simple to complex Map Reduce jobs as per the requirement.
• Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Developed PIG UDFs for manipulating the data according to Business Requirements and also worked on developing custom PIG Loaders.
• Hands on experience in setting up HBase Column based storage repository for archiving and retro data.
• Responsible for creating Hive tables based on business requirements.
• Used Enterprise data lake to provide support for various uses cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing, structured and unstructured data.
• Along with the Infrastructure team, involved in design and developed Kafka and Storm based data pipeline.
• Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Implemented Spark using python and Spark SQL for faster testing processing the data.
• Developed spark scripts by using python shell commands as per the requirement.
• Involved in data modeling and sharding and replication strategies in Cassandra.
• Load the data into Spark RDD and do in memory data Computation to generate the Output response.
• Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
• Exported the analyzed data into relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Spark, Spark SQL, Spark Streaming, Scala, Kafka, Apache Hadoop 2x, Cloudera, HDFS, MapReduce, Hortonworks, Hive, Pig, HBase, Spark, Scala, Sqoop, Kafka, FLUME, Cassandra, Rackspace, Oracle 11g/10g, Linux, XML,MYSQL.

Confidential, Bedford, MA

Hadoop Developer

Responsibilities:

• Installed & configured multi-node Hadoop Cluster and performed troubleshooting and monitoring of Hadoop Cluster.
• Installed and configured Flume, Oozie on the Hadoop cluster.
• Managing, defining and scheduling Jobs on a Hadoop cluster.
• Worked on installing cluster, commissioning & decommissioning of datanode, namenode recovery, capacity planning, and slots configuration.
• Resource management of Hadoop Cluster including adding/removing cluster nodes for maintenance and capacity needs.
• Wrote Hive queries for ad-hoc reporting and ETL.
• Worked with different file formats such as Text, Sequence files, Avro, ORC and Parquette.
• Pushing logs onto Hadoop cluster via SOAP UI calls.
• Involved in upgrading all the Hadoop components to the latest versions.
• Worked on NOSQL database Mongo DB.
• Experience in managing and reviewing Hadoop log files.
• Optimizing Hadoop MapReduce code, Hive and Pig scripts for better scalability, reliability and performance.
• Developed the OOZIE workflows for the Application execution.
• Performing data migration from Legacy Databases RDBMS to HDFS using Sqoop.
• Worked on Creation of Optimized Environment for testing and demonstrating Spark, Cassandra & Kafka Connectivity with objective of minimizing Spark SQL query times, Over JDBC/ODBC for databases
• Writing Pig scripts for data processing.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Implemented Hive tables and HQL Queries for the reports.
• Imported data from Cassandra into HDFS using Mongo export utility.
• Involved in developing shell scripts and automated data management from end to end integration work
• Experience in performing data validation using HIVE dynamic partitioning and bucketing.
• Written and used complex data type in storing and retrieved data using HQL in Hive.
• Developed Hive queries to analyze reducer output data.
• Implemented ETL code to load data from multiple sources into HDFS using pig scripts.
• Highly involved in designing the next generation data architecture for the Unstructured data.
• Developed PIG Latin scripts to extract data from source system.
• Created and maintained technical documentation for executing Hive queries and Pig scripts.
• Involved in Extracting, loading Data from Hive to Load an RDBMS using Sqoop

Environment: Spark, Scala, HDFS, Map Reduce, MySQL, Cassandra, Hive, HBase, Oozie, PIG, ETL, Hortonworks (HDP 2.0), Shell Scripting, Linux, Sqoop, Flume and Oracle 11g.

Confidential

Java/ Hadoop Developer

Responsibilities:

• Involved in importing data from Microsoft SQLServer, MySQL, Teradata into HDFS using Sqoop.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS.
• Used Hive to analyze the partitioned and bucked data to compute various metrics of reporting.
• Involved in creating Hive tables loading data, and writing queries that will run internally in MapReduce
• Involved in creating Hive External tables for HDFS data.
• Solved performance issues in Hive and Pig Scripts with understanding of Joins, Group and Aggregation and perform the MapReduce jobs.
• Used Spark for transformations, event joins and some aggregations before storing the data into HDFS.
• Troubleshoot and resolve data quality issues and maintain elevated level of data accuracy in the data being reported.
• Analyze the large amount of data sets to determine optimal way to aggregate.
• Worked on the Oozie workflow to run multiple Hive and Pig jobs.
• Involved in creating Hive UDF's.
• Developed automated shell script to execute Hive Queries.
• Involved in processing ingested raw data using Apache Pig.
• Monitored continuously and managed the Hadoop cluster using cloudera manager.
• Worked on different file formats like JSON, AVRO, ORC, Parquet and Compression like Snappy, zlib, ls4 etc.
• Executed HiveQL in Spark using SparkSQL.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, scala.
• Gained Knowledge in creating Tableau dashboard for reporting analyzed data.
• Expertise with NoSQL databases like HBase.
• Experienced in managing and reviewing the Hadoop log files.
• Used GitHub as repository for committing code and retrieving it and Jenkins for continuous integration.

Environment: HDFS, MapReduce, Sqoop, Hive, Pig, Shark, Spark, Oozie, MySQL, Eclipse, Git, GitHub, Jenkins.

Confidential

Java Developer

Responsibilities:

• Involved in various stages of Enhancements in the Application by doing the required analysis, development, and testing.
• Prepared the High and Low level design document and Generating Digital Signature
• For analysis and design of application created Use Cases, Class and Sequence Diagrams.
• For the registration and validation of the enrolling customer developed logic and code.
• Developed web-based user interfaces using struts frame work.
• Handled Client side Validations used JavaScript
• Wrote SQL queries, stored procedures and enhanced performance by running explain plans.
• Involved in integration of various Struts actions in the framework.
• Used Validation Framework for Server side Validations
• Created test cases for the Unit and Integration testing.
• Front-end was integrated with Oracle database using JDBC API through JDBC-ODBC Bridge driver at server side.
• Designed project related documents using MS Visio which includes Use case, Class and Sequence diagrams.
• Writing end-to-end flow i.e. controllers classes, service classes, DAOs classes as per the Spring MVC design and writing business logics using core java API and data structures
• Used Spring JMS related MDB to receive the messages from other team with IBM MQ for queuing
• Developed presentation layer code, using JSP, HTML, AJAX and JQuery
• Developed the Business layer using spring (IOC, AOP), DTO, and JTA
• Developed application service components and configured beans using Spring IOC. Implemented persistence layer and Configured EH Cache to load the static tables into secondary storage area.
• Involved in the development of the User Interfaces using HTML, JSP, JS, Dojo Tool Kit, CSS and AJAX
• Created tables, triggers, stored procedures, SQL queries, joins, integrity constraints and views for multiple databases, Oracle 11g using Toad tool.
• Developed the project using industry standard design patterns like Singleton, Business Delegate Factory Pattern for better maintenance of code and re-usability
• Developed unit test cases using Junit framework for testing accuracy of code and logging with SLF4j + Log4j
• Worked with defect tracking system Clear Quest
• Worked with IDE as Spring STS and deployed into spring tomcat server, WebSphere& used Maven as build tool
• Responsible for code sanity in integration stream used Clear Case as version control tool

Environment: Java, J2EE, Spring, Spring Batch, Spring JMS, MyBatis HTML, CSS, AJAX, JQuery, JavaScript, JSP, XML, UML, JUNIT, IBM WebSphere, Maven, Clear Case, SoapUI, Oracle 11g,, IBM MQ.

Confidential

Java Developer

Responsibilities:

• Worked with business analysts in gathering business rules and requirements.
• Helped in designing the User Interface
• Developed several JSPs and Servlets.
• Developed stateless Session Beans for handling the business process workflow.
• Developed DAO for database access to the Oracle database.
• Developed ANT scripts to create WAR file and deploy it on WebSphere Application Server.
• Developed Junits for unit testing

Environment: Java/J2EE, Struts, IBM WebSphere, Oracle 10g, Subversion, Eclipse