SUMMARY:

• Around 8 years of experience in various IT sectors such as banking, health - care, and financial services, which includes hands-on experience in Big Data technologies.
• 3 years of experience as a Hadoop Developer in all phases of Hadoop and HDFS development.
• Hands on experience with HDFS, MapReduce and Hadoop Ecosystem (Pig, Hive, Oozie, Hbase, Zookeeper and Sqoop).
• Well versed with developing and implementing MapReduce jobs using Hadoop to work with Big Data.
• Have experience with Spark processing Framework such as Spark and Spark Sql.
• Experience in NoSQL databases like HBase, MongoDB
• Experience on Microservices, Spring boot and Spring cloud.
• Procedural knowledge in cleansing and analyzing data using HiveQL, Pig Latin, and custom MapReduce programs in Java.
• Experienced in writing custom UDFs and UDAFs for extending Hive and Pig core functionalities.
• Ability to develop Pig UDF'S to pre-process the data for analysis.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems (RDBMS), Teradata and vice versa.
• Skilled in creating workflows using Oozie for cron jobs.
• Strong experience in Hadoop Administration and Linux.
• Experienced with Java API and REST to access HBase data.
• Worked extensively with Dimensional modeling, Data migration, Data cleansing, Data profiling, and ETL Processes features for data warehouses.
• Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
• Hands on experience in PERL Scripting and Python.
• Experience on AWS EC2, EMR, LAMBDA and Cloud Watch
• Experience working with JAVA, J2EE, JDBC, ODBC, JSP, Java Eclipse.
• Extensive experience with SQL, PL/SQL and database concepts.
• Expertise in debugging and optimizing Oracle and java performance tuning with strong knowledge in Oracle 11g and SQL.
• Good experience working with Distributions such as MAPR, Horton works and Cloudera.
• Experience in all stages of SDLC (Agile, Waterfall), writing Technical Design document, Development, Testing and Implementation of Enterprise level Data mart and Data warehouses.
• Having good knowledge on Hadoop Administration like Cluster configuration, Single Node Configuration, Multi Node Configuration, Data Node Commissioning and Decommissioning, Name Node Backup and Recovery, HBase, HDFS and Hive Configuration, Monitoring clusters, Access control List.
• Good Inter personnel skills and ability to work as part of a team. Exceptional ability to learn, master new technologies and to deliver outputs in short deadlines. Delivery Assurance - Quality Focused & Process Oriented:
• Ability to work in high-pressure environments delivering to and managing stakeholder expectations.
• Application of structured methods to: Project Scoping and Planning, risks, issues, schedules and deliverables.
• Strong analytical and problem-solving skills.
• Good Inter personnel skills and ability to work as part of a team. Exceptional ability to learn and master new technologies and to deliver outputs in short deadlines

PROFESSIONAL EXPERIENCE:

Sr. Hadoop Developer

Confidential, San Jose, CA

Responsibilities:

• Worked as a Spark Expert and performance Optimizer.
• Member of Spark COE (Center Of Excellence) in Data Simplification project at Cisco.
• Experienced with Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Well experienced in handling Data Skewness in Spark-SQL.
• The main aim of the project is tuning the performance of the existing Hive Queries and preparing spark jobs that are scheduled daily in Tez.
• Implemented Spark using Scala, Java and utilizing Data frames and Spark SQL API for faster processing of data.
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
• Developed a data pipeline using Kafka, HBase, Spark and Hive to ingest, transform and analyzing customer behavioral data.
• Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and Spark.
• Developed Spark jobs and Hive Jobs to summarize and transform data.
• Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data.
• Worked under Mapr Distribution and familiar with HDFS.
• Designed and implemented Spark cluster to Analyse Data in Cassandra.
• Migrated tables from SQLServer to Cassandra, which are being used actively till date.
• ETL off-loading from Teradata to Hadoop.
• Real time streaming the data using Spark with Kafka.
• Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and then loading data into HDFS.
• Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.
• Designed and Maintained Tez workflows to manage the flow of jobs in the cluster.
• Worked with the testing teams to fix bugs and ensure smooth and error-free code.
• Involved in preparation of docs like Functional Specification document and Deployment Instruction documents.
• Fix defects as needed during the QA phase, support QA testing, troubleshoot defects and identify the source of defects.
• Involved in Agile methodologies, daily Scrum meetings, Sprint planning.

Environment: Hadoop, Spark Core, Spark-SQL, Spark-Streaming, MapReduce, HDFS, Hive, Java, Scala, Hue, SQL, Teradata, Pig, Sqoop, Tez, HBase, Cassandra, Zookeeper, PL/SQL, MySQL, DB2, Teradata.

Sr. Hadoop Developer

Confidential, Omaha, NE

Responsibilities:

• Used Sqoop and Java API's to import the data to Cassandra from different relational databases.
• Created tables in Cassandra and loaded large data sets of structured, semi-structured and unstructured data from various data sources.
• Developed Map reduce jobs in Java for cleaning and preprocessing data.
• Wrote Python scripts for wrapper and utility automation.
• Performed cleansing operations by using storm builder topologies before moving data in to Cassandra.
• Implemented Storm builder topologies to perform cleansing operations before moving data into Cassandra.
• Worked on configuring Hive, PIG, Impala, Sqoop, Flume and oozie in cloudera.
• Automated data movement between different Hadoop systems using Apache NiFi.
• Wrote Map reduce programs in python using Hadoop Streaming API.
• Wrote on creating Hive tables and loading them with data and writing Hive queries.
• Migration of ETL processes from SQL server to Hadoop using PIG for data manipulation.
• Developed spark jobs using Scala in test environment and Spark sql for querying.
• Worked on importing data from oracle tables to HDFS and Hbase tables using Sqoop.
• Wrote scripts to load data in to Spark RDDs and do in memory computations.
• Wrote Spark Streaming script which consumes topics from distributed messaging source Kafka and periodically pushes batch of data to Spark for real time processing.
• Involved in converting Cassandra/Hive/SQL queries into Spark transformations using Spark RDDs in Scala.
• Experience in Elastic search technologies in creating custom Solr Query components.
• Implemented Kafka Custom encoders for custom input format to load data into Kafka Partitions.
• Worked on different data sources such as Oracle, Netezza, MySQL, Flat files etc.
• Extensively used Sqoop to get data from RDBMS sources like Teradata and Netezza.
• Worked with Flume to load the log data from different sources into HDFS.
• Good knowledge in using apache NiFi to automate the data movement between different Hadoop systems.
• Developed Talend jobs to move inbound files to HDFS file location based on monthly, weekly, daily and hourly partitioning.

Environment: Cloudera, Map Reduce, SparkSQL, SparkStreaming, Pig, Hive, Flume, Hue, Oozie, Java, Eclipse, Zookeeper, Cassandra, Hbase, Talend, Github.

Hadoop Developer

Confidential, Sunnyvale, CA

Responsibilities:

• Worked closely with the business analysts to convert the Business Requirements into Technical Requirements and prepared low and high-level documentation.
• Played a key role in discussing about the requirements, analysis of the entire system along with estimation, development and testing accordingly keeping BI requirements as a note.
• Involved in developing and installations of Sqoop, Hive and FTP integrations to down systems.
• Performed importing data from various sources to the Cassandra cluster using Sqoop.
• Did a Sqoop job to help the import of data which is in different formats such as Avro, XML obtained from different vendors.
• Involved in writing MapReduce java scripts to process the data from HBase tables.
• Developed UNIX shell scripts for the business process and assimilation of data from different interfaces.
• Developed Sqoop scripts for writing the processed data into HBase tables which helps BI team for the data visualization.
• Established an Oozie component to implement a job scheduler which should occur on a daily basis.
• Also involved in developing a pipe line to load the data into tables using Spark streaming and Kafka which is integrated with ZooKeeper.
• Involved in developing Sqoop scripts that load the data from different interfaces to HDFS.
• Developed Scala code for reading multiple data formats on HDFS.
• Worked on debugging and performance tuning on Map Reduce, HIVE and Sqoop jobs.
• Involved in diagnosing different possible ways to optimize and improve the efficiency of the system.
• Developed multiple POC's using Scala which are deployed on the cluster in turn compared the performance of Spark with Map Reduce.
• Developed Spark code using Scala for generating Spark-RDD seeds for faster transformations.
• Involved in creating and maintaining of the technical documentation for the Map Reduce, Hive, Sqoop, UNIX jobs along with Hadoop clusters and also reviewing them to fix the post production issues.

Environment: Red Hat Enterprise Linux, HBase, Solr, Kafka, Map Reduce, Hive, Java SDK, Python, DB2, SQOOP, Spark, Scala, SBT, Akka, Maven, Solr, Github.

Hadoop Developer

Confidential, New York, NY

Responsibilities:

• Responsible for implementation and ongoing administration of Hadoop infrastructure and setting up infrastructure
• Cluster maintenance as well as creation and removal of nodes.
• Evaluation of Hadoop infrastructure requirements and design/deploy solutions (high availability, big data clusters.
• Cluster Monitoring and Troubleshooting Hadoop issues
• Manage and review Hadoop log files
• Works with application teams to install operating system and Hadoop updates, patches, version upgrades as required
• Created NRF documents which explains the flow of the architecture, which measure the performance, security, memory usage, dependency.
• Setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.
• Help maintain and troubleshoot UNIX and Linux environment.
• Experience analyzing and evaluating system security threats and safeguards.
• Experience in Importing and exporting data into HDFS and Hive using Sqoop.
• Developed Pig program for loading and filtering the streaming data into HDFS using Flume.
• Experienced in handling data from different data sets, join them and preprocess using Pig join operations.
• Developed Map-Reduce programs to clean and aggregate the data
• Developed HBase data model on top of HDFS data to perform real time analytics using Java API.
• Developed different kind of custom filters and handled pre-defined filters on HBase data using API.
• Imported and exported data from Teradata to HDFS and vice-versa.
• Strong understanding of Hadoop eco system such as HDFS, MapReduce, HBase, Zookeeper, Pig, Hadoop streaming, Sqoop, Oozie and Hive
• Implement counters on HBase data to count total records on different tables.
• Experienced in handling Avro data files by passing schema into HDFS using Avro tools and Map Reduce.
• Worked on custom Pig Loaders and Storage classes to work with a variety of data formats such as JSON, Compressed CSV, etc.
• We used Amazon Web Services to perform big data analytics.
• Implemented Secondary sorting to sort reducer output globally in map reduce.
• Implemented data pipeline by chaining multiple mappers by using Chained Mapper.
• Created Hive Dynamic partitions to load time series data
• Experienced in handling different types of joins in Hive like Map joins, bucker map joins, sorted bucket map joins.
• Created tables, partitions, buckets and perform analytics using Hive ad-hoc queries.
• Experienced import/export data into HDFS/Hive from relational data base and Tera data using Sqoop.
• Handling continuous streaming data comes from different sources using flume and set destination as HDFS.
• Integrated spring schedulers with Oozie client as beans to handle cron jobs.
• Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters
• Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews.
• Involved in story-driven agile development methodology and actively participated in daily scrum meetings.
• Worked on spring framework for multi-threading.

Environment: Hadoop, HDFS, Map Reduce, Hive, Pig,Sqoop, RDBMS/DB, Flat files, Teradata, Mysql, CSV, Avro data files. JAVA, J2EE.

Java/J2EE Developer

Confidential, Roseville, CA

Responsibilities:

• Write design document based on requirements from MMSEA user guide.
• Performed requirement gathering, design, coding, testing, implementation and deployment.
• Worked on modeling of Dialog process, Business Processes and coding Business Objects, QueryMapper and JUnit files.
• Involved in the design and creation of Class diagrams, Sequence diagrams and Activity Diagrams using UML models
• Created the Business Objects methods using Java and integrating the activity diagrams.
• Involved in developing JSP pages using Struts custom tags, jQuery and Tiles Framework.
• Used JavaScript to perform client side validations and Struts-Validator Framework for server-side validation
• Worked in web services using SOAP, WSDL.
• Wrote Query Mappers and MQ Experience in JUnit Test Cases.
• Developed the UI using XSL and JavaScript.
• Managed software configuration using ClearCase and SVN.
• Design, develop and test features and enhancements.
• Performed error rate analysis of production issues and technical errors.
• Developed test environment for testing all the Web Service exposed as part of the core module and their integration with partner services in Integration test.
• Analyze user requirement document and develop test plan, which includes test objectives, test strategies, test environment, and test priorities.
• Responsible for performing end-to-end system testing of application writing JUnit test cases
• Perform Functional testing, Performance testing, Integration testing, Regression testing, Smoke testing and User Acceptance Testing (UAT).
• Converted Complex SQL queries running at mainframes into pig and Hive as a part of a migration from mainframes into Hadoop cluster.

Environment: Shell Scripting, Java 6, JEE, Spring, Hibernate, Eclipse, Oracle 10g, JavaScript, Servlets, Nodejs, JMS, Ant, Log4j and Junit, Hadoop (Pig & Hive).