SUMMARY:

• Around 9+ years of programming experience involved in all phases of Software Development Life Cycle (SDLC).
• Around 5 Years of Big data related architecture experience developing Hadoop applications.
• Experienced in the Hadoop ecosystem components like Hadoop Map Reduce, Cloudera, Hortonworks, HBase, Oozie, Hive, Sqoop, Pig, Flume, Kafka, Storm, Spark, MongoDB, and Cassandra.
• Proven Expertise in performing analytics on Big Data using Map Reduce, Hive and Pig.
• Implemented POC to migrate map reduce jobs into Spark RDD transformations using Scala.
• Developed Apache Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.
• Hands on developing and debugging YARN (MR2) Jobs to process large Datasets.
• Data Processing: Processed data using MapReduce and Yarn. Worked on Kafka as a proof of concept for log processing
• Experienced with performing real time analytics on NoSQL databases like HBase and Cassandra.
• Good knowledge in working with Impala, Storm and Kafka.
• Experienced with Dimensional modeling, Data migration, Data cleansing, Data profiling, and ETL Processes features for data warehouses.
• Experience web services, data modeling, content processing, data replication.
• Worked with Oozie workflow engine to schedule time based jobs to perform multiple actions.
• Experienced in importing and exporting data from RDBMS into HDFS using Sqoop.
• Familiarity with distributed coordination system Zookeeper.
• Design and develop test plan for test cases based on functional and design.
• Analyzed large amounts of data sets using Pig scripts and Hive scripts.
• Experience in data warehousing with ETL tool Oracle Warehouse Builder (OWB).
• Hands on experience in working with database like Oracle, MySQL and PL/SQL.
• Good working knowledge on processing Batch applications.
• Experience with streaming data using IBM streams processing language.
• Experience in capturing and analyze the data in motion using info sphere stream.
• Experienced in writing MapReduce programs and UDFs for both Hive and Pig in Java.
• Experienced in developing Web Services with Python programming language.
• Involved in Design and Development of technical specifications using Hadoop Echo System tools
• Cutting edge experience on Splunk (Log based performance monitoring tool).
• Experience with configuration of Hadoop Ecosystem components: Hive, HBase, Pig, Sqoop and Flume.
• Good experience in Hive partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive SerDe like JSON a Avro.
• Experience in using different file formats - Avro, Sequence Files, ORC, JSON and Parquet.
• Experience in Performance Tuning, Optimization and Customization.
• Experience with Unit Testing Map Reduce programs using MRUnit, JUnit.
• Experience in Active Development as well as onsite coordination activities in web-based, client/server and distributed architecture using Java, J2EE which includes Web services, Spring, Struts, Hibernate and JSP/Servlets along with incorporating MVC architecture.
• Good working knowledge on servers like Tomcat, Web Logic 8.0.
• Ability to work in teams as well as an individual, quick learner and able to meet deadlines.

TECHNICAL SKILLS:

Big Data Eco System: HDFS, Map Reduce, Hive, Pig, HBase, Spark, Spark Streaming, Spark SQL, Kafka, Cloudera CDH4, CDH5, Hortonworks, Hadoop Streaming, Splunk, Zookeeper, Oozie, Sqoop, Flume, Impala, Solr, and Ranger.

No SQL: HBase, MongoDB, Couchbase, Neo4j, Cassandra

Languages: Java/ J2EE, SQL, Shell Scripting, C/C++, Python, Scala

Web Technologies: HTML, JavaScript, CSS, XML, Servlets, SOAP, Amazon AWS, Google App Engine

Web/ Application Server: Apache Tomcat Server, LDAP, JBOSS, IIS

Operating system: Windows, Macintosh, Linux and Unix

Frameworks: Springs, MVC, Hibernate, Swings

DBMS / RDBMS: Oracle 11g/10g/9i, SQL Server 2012/2008, MySQL

IDE: Eclipse, Microsoft Visual Studio (2008,2012), NetBeans, Spring Tool Suits

Version Control: SVN, CVS and Rational Clear Case Remote Client, GitHub, Visual Studio

Tools: FileZilla, Putty, TOAD SQL Client, MySQL Workbench, ETL, DWH, JUnit, SQL Oracle Developer, WinScp, Tahiti, Cygwin, pentaho

PROFESSIONAL EXPERIENCE:

Sr. Hadoop/Big data Developer

Confidential, TEXAS

Responsibilities:

• Experience in design and deployment of Hadoop cluster and different Big Data analytic tools including Hive, HBase, Oozie, Sqoop, Impala, Kafka and Spark.
• Performed Real-time streaming of data using Kafka and processing of data using SparkStreaming and loading of data into Kudu tables..
• Developed a solid solution for handling offsets for kafka topics in scala. Persist offsets in hbase for minimizing the data loss and reprocessing time.
• Implemented near real time aggregations by joining on multiple topic with intermediate storage (kudu). The aggregations are running in near real time with (10 mins) interval.
• Implemented a solution for data archiving process with configurable intervals.
• Good understanding with kudu partitions, primary keys.
• Configured workflows for scheduling spark jobs with oozie.
• Good understanding with integrating multiple ecosystems like spark-hbase and spark-kudu
• Experience in integrating Kafka with Spark Streaming for real time data processing.
• Extract streaming complex structured data from Kafka and process the files using Spark Streaming and load the data into Kudu tables.
• Worked on conversion of Hive/ SQL queries into Spark transformations using Spark RDDs and data frames (DF).
• Used Spark API over Hadoop YARN as execution engine for data analytics using Hive.
• Configured Zookeeper for coordinating the cluster to maintain data consistency.
• Involved in loading and transforming of large sets of structured, semi-structured and unstructured data into HDFS.
• Implemented dynamic partitioning and bucketing in Hive.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Worked on Cluster Monitoring, troubleshooting and Disk topology.
• Processed different file formats like AVRO, PARQUET, Sequence files and ORC.
• Involved in Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting.
• Worked with Data Scientists to design/develop solutions for Data Analysis
• Designed the ETL process from various sources into Hadoop/HDFS for analysis and further processing.

Confidential, Two-Destiny Way, TEXAS

Sr Hadoop Developer

Responsibilities:

• Assisted in the development of high quality analytical reports of weather and GIS data.
• Used Talend to generate optimized code to load, transform, enrich, and cleanse data inside Hadoop.
• Moved Relational Database data using Sqoop as ETL tool into the Data Lake,Hive Dynamic partition tables.
• Build Hadoop clickstream workflows using Apache Hive and Pig for extraction, transformation and loading of data.
• Imported unstructured data like logs from different web servers to HDFS using Flume and developed MapReduce jobs for log analysis, recommendations and analytics.
• Involved in real-time data processing using Storm.
• Expertise in real-time analytics, machine learning and continuous monitoring of operations using Storm.
• Converted and loaded local data files into HDFS through the UNIX shell.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Created HBase tables to store variable data formats coming from different portfolios.
• Worked with NoSQL database Hive, Hbase to create tables and store data.
• Worked with Apache Nifi to Develop Custom Processors for the purpose of processing and disturbing data among cloud systems
• Worked on Apache Nifi to Uncompress and move json files from local to HDFS
• Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
• Used Kafka along with HBase to render data streaming.
• Developed Cassandra data model to match the business requirements
• Involved in Administration of Cassandra cluster along with Hadoop, Pig and Hive.
• Analyzed the customer behavior by performing click stream analysis and to ingest the data used flume.
• Developed spark scripts by using scala shell as per requirements.
• Created Views from Hive Tables on top of data residing in Data Lake.
• Build advanced ETL logic on clickstream, log and tax data based on complex technical and business requirements.
• Developed Spark code using Scala and Spark-SQL for faster testing and data processing.
• In data exploration stage used hive and impala o get some insights about the customer data.
• Evaluated Spark's performance vs impala on transactional data.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD, scala.
• Involved in configuring batch job to perform ingestion of the source files in to the Data lake
• Wrote and executed various MYSQL database queries from Python using Python-MySQL connector and MySQL dB package.
• Worked on Github to check-in and checkout source code.
• Wrote scripts in Python for extracting data from HTML file.
• Worked with NoSQL Cassandra to store, retrieve, and update and manage all the details for Ethernet provisioning and customer order tracking.
• Implemented Spark for fast interactive data analysis of datasets loaded in RDD.
• Analyzed the data by performing Hive queries (HiveQL), ran Pig scripts, Spark SQL and Spark streaming.
• Developed tools using Python, Shell scripting, XML to automate some of the menial tasks
• Used Pig for aggregation, cleaning and incremental ETL functions and developing UDFs for filtering.
• Wrote scripts in Python for extracting data from HTML file.
• Working experience on maintaining MySQL databases creation and setting up the users and maintain the backup of cluster metadata databases with Cron jobs.
• Redshift data model and Tableau Server configurations to provide guaranteed response time for reports.
• Implemented AWS provides a variety of computing and networking services to meet the needs of applications.
• Migrated an existing on-premises application to AWS.
• Developed Pig UDFs to specifically preprocess and filter data sets for analysis.
• Used Teradata database management system to manage the warehousing operations and parallel processing.
• Validated data sets graphically with Excel and did touch ups in Photoshop.
• Designed & scheduled workflows for updating system reports using Oozie.
• Ability to working with team environment and solved problems.

Environment: Cloudera, Avro, HBase, HDFS, Hive, Pig, Java, SQL, Sqoop, Flume, Oozie, Java (jdk 1.7), Eclipse, Splunk, YARN, SQL Server, Spark, python, Hortonworks, Zookeeper, SVN, Talend

Confidential, Dallas, Texas

Hadoop Developer

Responsibilities:

• Worked on importing data from various sources and performed transformations using MapReduce, Hive to load data into HDFS.
• Configured Sqoop jobs to import data from RDBMS into HDFS using Oozie workflows.
• Worked on setting up Pig, Hive and HBase on multiple nodes and developed using Pig, Hive, HBase and Map Reduce.
• Involved in the process of data acquisition, data pre-processing and data exploration of telecommunication project in scala.
• Experience with Map Reduce coding.
• Solved small file problem using Sequence files processing in Map Reduce.
• Written various Hive and Pig scripts.
• Experience in Upgrading cluster, CDH and HDP Cluster.
• Involved in the process of data acquisition, data pre-processing and data exploration of telecommunication project in Scala.
• Used flume, sqoop, hadoop,spark and oozie for building data pipeline
• Created HBase tables to store variable data formats coming from different portfolios.
• Experience in upgrading Hadoop cluster hbase/zookeeper from CDH3 to CDH4.
• Performed real time analytics on HBase using Java API and Rest API.
• Developed MapReduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis
• Implemented complex MapReduce programs to perform joins on the Map side using distributed cache.
• Setup flume for different sources to bring the log messages from outside to HDFS.
• Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
• Worked on compression mechanisms to optimize MapReduce Jobs.
• Real time experience with analytics and BI.
• Wrote Python scripts to parse XML documents and load the data in database..
• Experienced with working on Avro Data files using Avro Serialization system.
• Implemented business logic by writing UDF's in Java and used various UDF's from Piggybanks and other sources.
• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
• Unit tested and tuned SQLs and ETL Code for better performance.
• Monitored the performance and identified performance bottlenecks in ETL code. .
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Map Reduce, HBase, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Zookeeper, YARN, Oozie, Java, Eclipse

Confidential, Wilmington, DE

Big Data/Hadoop Developer

Responsibilities:

• Created Hive tables and working on them using Hive QL.
• Involved in installing Hadoop Ecosystem components.
• Validated Name node, Data node status in a HDFS cluster.
• Importing and exporting data from HDFS to RDBMS and vice-versa using SQOOP.
• Experienced in developing HIVE Queries on different data formats like Text file, CSV file.
• Developed multiple MapReduce jobs in java for data cleaning and preprocessing.
• Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis
• Installed and configured Hadoop cluster in Test and Production environments
• Performed both major and minor upgrades to the existing CDH cluster
• Code review as per the customer coding standards.
• Testing and providing the valid test data to users as per requirement.
• Weekly meetings with technical collaborators and active participation in code review sessions with senior and junior developers.
• Responsible to manage data coming from different sources.
• Supporting Hbase Architecture Design with the Hadoop Architect team to develop a Database Design in HDFS.
• Involved in HDFS maintenance and loading of structured and unstructured data.
• Wrote Hive queries for data analysis to meet the business requirements.
• Installed and configured Pig and also written Pig Latin scripts.
• Developed UDFs for Pig Data Analysis.
• Involved in managing and reviewing Hadoop log files.
• Developed Scripts and Batch Job to schedule various Hadoop Program.
• Utilized Agile Scrum Methodology to help manage and organize a team of 4 developers with regular code review sessions.
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce.
• Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• Installed Oozie workflow engine to run multiple Hive and Pig jobs.
• Developed Hive queries to process the data and generate the data cubes for visualizing.

Environment: Java, Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, Zookeeper, Linux, XML, Eclipse, Cloudera

Confidential

Java Developer

Responsibilities:

• Performed requirements analysis and design and used of Rational Unified Process for analysis, design and documentation purposes.
• Interacted with the client to understand the requirements thoroughly in a short span of time.
• Developed Use cases, traceability matrices, design specification and test documents including UATs, training and implementation manuals.
• Involved in design and development of the architecture of the web applications in JSP.
• Developed various Java classes, SQL queries and procedures to retrieve and manipulate the data from backend Oracle database using JDBC.
• Involved in developing programs for parsing the XML documents using XML Parser.
• Involved in tuning and profiling the application for better data transactions and performance used JProbe for the same.
• Designed and developed Enterprise Stateless and Stateful Session beans to communicate with the Container Managed Entity Bean backend services.
• Involved in Unit Testing and Integration Testing.
• Designed and developed various modules of the application with J2EE design architecture and frameworks like Spring MVC architecture and Spring Bean Factory using IOC, AOP concept.
• Followed agile software development with Scrum methodology.
• Wrote application front end with HTML, JSP, JSF, Ajax/JQuery, Spring Web Flow and XHTML.
• Used J Query for UI centric Ajax behavior.
• Implemented JAVA/J2EE design patterns such as Factory, DAO, Session Façade and Singleton.
• Used Hibernate in persistence layer and developed POJO's, Data Access Object (DAO) to handle all Database operations.
• Implemented features like logging, user session validation using Spring-AOP module.
• Developed server-side services using Java, Spring, Web Services (SOAP, WSDL, JAXB, JAX-RPC)
• Worked on Oracle as the backend database.
• Used JMS for messaging.
• Used Log4j to assign, track, report and audit the issues in the application.
• Develop and execute Unit Test plans using J Unit, ensuring that results are documented and reviewed with Quality Assurance teams responsible for integrated testing.
• Worked in deadline driven environment with immediate feature release cycles.

Environment: - Java, Spring, Hibernate, JSP, HTML, CSS, XML, JavaScript, JQuery, JUnit, AJAX, Multi-Threading, Oracle, Web Service - SOAP, WebSphere, MYSQL.

Confidential

Java Developer

Responsibilities:

• Performed in various phases of the Software Development Life Cycle (SDLC)
• Developed user interfaces using JSP framework with AJAX, Java Script,HTML,XHTML,and
• CSS
• Performed the design and development of various modules using CBD Navigator Framework
• Deployed J2EE applications in Web sphere application server by building and deploying ear fileusing ANT script
• Created tables, stored procedures in SQL for data manipulation and retrieval.
• Used technologies like JSP, JavaScript and Tiles for Presentation tier.
• Performed the design and development of various modules using CBD Navigator Framework
• Deployed J2EE applications in Web sphere application server by building and deploying ear file using ANT script.
• Created tables, stored procedures in SQL for data manipulation and retrieval.
• Used technologies like JSP, JavaScript and Tiles for Presentation tier.
• CVS tool is used for version control of code and project documents.
• Application Server, UML, JUnit, JTest, Netbeans, Windows 2000.

Environment: JSP, Servlets, JDK, JDBC, XML, JavaScript, HTML, Spring MVC, JSF, Oracle 8i, Sun Application Server, UML, JUnit, JTest, Netbeans, Windows 2000.