SUMMARY

• 8+ years of overall experience in data analysis, data modeling and implementation of enterprise class systems spanning Big Data, Data Integration, Object Oriented programming and Advanced Analytics.
• Excellent understanding of Hadoop Architecture and Deamons such as HDFS, Name Node, Data Node, Job Tracker, Task Tracker.
• Hands on experience in installing, configuring and using Hadoop ecosystem components like Hadoop, HDFS, MapReduce Programming, Hive, Pig, Sqoop, HBase, Impala, Solr, Elastic Search, Oozie, Zoo Keeper, Kafka, Spark, Cassandra with Cloudera and Hortonworks distribution.
• Created custom Solr Query segments to optimize ideal search matching.
• Used Solr Search &MongoDB for querying and storing data.
• Extracted files from Cassandra through Sqoop and placed in HDFS and processed.
• Involved in converting Cassandra/Hive/SQL queries into Spark transformations using RDDs, and Scala.
• Analyzed the Cassandra/SQL scripts and designed the solution to implement using Scala.
• Expertise in Big Data Technologies and Hadoop Ecosystem tools like Flume, Sqoop, Hbase, ZooKeeper, Oozie, MapReduce, Hive, PIG and YARN.
• Extracted and updated the data into MONGOD using MONGO import and export command line utility interface.
• Developed Collections in Mongo DB and performed aggregations on the collections.
• Hands on experience in installation, configuration, management and deployment of Big Data solutions and the underlying infrastructure of Hadoop Clusterusing Cloudera and Horton works distributions.
• In - depth Knowledge of Data Structures, Design and Analysis of Algorithms.
• Good understanding of Data Mining and Machine Learning techniques.
• Hands on experience in various Hadoop distributionsIBM Big Insights, Cloudera, Horton works and MapR.
• Scheduled various ETL process and Hive scripts by developing Oozie workflows.
• Experience in migrating ETL process into Hadoop, Designing Hive data model and wrote PigLatinscripts to load data into Hadoop.
• Experienced in using Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Expertise in using Custom loader functions in PiggyBank.
• Experience in handling various file formats like AVRO, Sequential, Parquet etc.
• Expertise in Implementing complex business logic by writing Generic UDF's and Hive UDF.
• Good understanding of MPP databases such Impala and Created tables and writing Queries in IMPALA and GreenPlum.
• Strong Knowledge in Informatica ETL Tool, Data warehousing and Business intelligence. knowledge on Web/Application Servers like Apache Tomcat, IBM WebSphere and Oracle WebLogic.
• Good Knowledge on Object Oriented Analysis and Design(OOAD) and Java Design patterns.
• In-depth understanding of Spark Architecture including Spark Core, Spark SQL, Data
• Frames, Spark Streaming, Spark MLlib.
• Expertise in writing SparkRDD transformations, actions, Data Frames, case classes for the required input data and performed the data transformations using Spark-Core.
• Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
• Expertise in developing Real-Time Streaming Solutions using Spark Streaming.
• Proficient in big data ingestion and streaming tools like Flume, Sqoop, Spark, Kafka and Storm.
• Hands on experience in developing Map Reduce programs using Apache Hadoop for analyzing the Big Data.
• Expertise in implementing ad-hoc Map Reduce programs using Pig Scripts.
• Experience in importing and exporting data from RDBMS to HDFS, Hive tables and HBase by using Sqoop.
• Experience in importing streaming data into HDFS using flume sources, and flume sinks and transforming the data using flume interceptors.
• Exposure on usage of Apache Kafka to develop data pipeline of logs as a stream of messages using producers and consumers.
• Experience in integrating Apache Kafka with Apache Storm and created Storm data pipelines for real time processing.
• Knowledge in handling Kafka cluster and created several topologies to support real-time processing requirements.
• Hands on Experience in writing SQL and PL/SQL queries.
• Ability to work with Onsite and Offshore Teams.
• Good understanding and experience with Software Development methodologies like Agile and Waterfall.
• Good understanding of all aspects of Testing such as Unit, Regression, Agile, White-box, Black-box.
• Hands on experience migrating complex map reduce programs into Apache Spark RDD transformations.
• Design and Programming experience in developing Internet Applications using Java, J2EE, JSP,MVC, Servlets, Struts, Hibernate, JDBC, JSF, EJB, XML, AJAX and web based development tools.
• Experience with Enterprise Java Beans (EJB) components Technical expertise & demonstrated high standards of skills in J2EE frameworks like Struts (MVC Framework).
• Proficient in using various IDEs like RAD, Eclipse, IntelliJ, SBTand JDeveloper.
• Extensive experience in programming, deploying, configuring middle-tier popular J2EE Application Servers like Bea WebLogic 8.1, IBM WebSphere 5.0, open source Apache Tomcat, and JBoss Servers.
• Good experience in client web technologies like HTML, CSS, JavaScript and AJAX, Servlets, JSP, JSON, XML, JSF,AWS.
• Experience in Software Development Life Cycle (SDLC), OOA, OOD and OOP through implementation and testing.
• Experience in working with Hadoop in Stand-alone, pseudo and distributed modes.
• Good at working on low-level design documents and System Specifications.

TECHNICAL SKILLS

Big Data Ecosystems: Hadoop, Map Reduce, HDFS, Zookeeper, Hive Pig, Sqoop, Oozie, Flume,Yarn,Spark

Database Languages: SQL,PL/SQL,Oracle

Programming Languages: Java, Scala

Frameworks: Spring, Hibernate, JMS

Scripting Languages: JSP, Servlets, JavaScript, XML, HTML

Web Services: RESTful web services

Databases: RDBMS, HBase, Cassandra

IDE: Eclipse, IntelliJ, SBT

Platforms: Windows, Linux, Unix

Application Servers: Apache Tomcat, Web Sphere, Web logic, JBoss

Methodologies: Agile, Waterfall

ETL Tools: Talend, Informatica

PROFESSIONAL EXPERIENCE

Hadoop/Spark Developer

Responsibilities:

• Experience in working with Hadoop clusters using Hortonworks distributions.
• Experienced in NOSQL databases like HBase, MongoDB and experienced with Hortonworks distribution of Hadoop.
• Responsible for building scalable distributed data solutions using Hadoop.
• Developed job processing scripts using Oozie workflow.
• Worked with Nifi for managing the flow of data from source to HDFS.
• Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster.
• Worked extensively with Sqoop for importing metadata from Oracle.
• Assisted in exporting analyzed data to relational databases using Sqoop.
• Configured Sparkstreaming to get ongoing information from the Kafka and stored the stream information to HDFS.
• Consumed JSON messages using Kafka and processed the JSON file using Spark Streaming to capture UI updates
• Experienced in writing live Real-time Processing and core jobsusing Spark Streaming withKafka as a data pipe-line system.
• Optimized Hive QL/ pig scripts by using execution engine like Tez, Spark.
• Tested Apache TEZ, an extensible framework for building high performance batch and interactive data processing applications, on Pig and Hive jobs
• Worked and learned a great deal from AWS Cloud services like EC2, S3, EBS, RDS and VPC.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2and S3for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Used DataStax Spark-Cassandra connector to load data into Cassandra and used CQL to analyze data from Cassandra tables for quick searching, sorting and grouping.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Experience in designing and developing POCs in Spark using Scala to compare the performance of spark with Hive and SQL/Oracle.
• Performed Scala, Data transformations in Scala, HIVE and used partitions, buckets for performance improvements.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data
• Imported data from AWS S3into Spark RDD, Performed transformations andactionson RDD's.
• Used Solr Search &MongoDB for querying and storing data.
• Implemented Elastic Search on Hive data warehouse platform.
• Practical experience in defining queries on JSON data using Query DSL provided by ElasticSearch.
• Experience in improving the search focus and quality in ElasticSearch by using aggregations.
• Load and transform large sets of structured, semi structured and unstructured data.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in Near real time and persist it to Cassandra.
• Designed, developed data integration programs in a Hadoop environment with NoSQL data store Cassandra for data access and analysis.
• Created Cassandra tables to store various data formats of data coming from different sources.
• Helped the team to increase the Cluster size from 100 to 150 Nodes.
• Involved in creating Hive tables, and loading and analyzing data using hive queries.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis. Experienced in running Hadoop streaming jobs to process terabytes of xml format data.
• Wrote Hive Queries and UDF's.
• Good in using version control like GITHUB and SVN.
• Hands-on building tool Maven and continuous integration like Jenkins.
• Strong knowledge in using different utility tools like Eclipse, Intellij, SBT.
• Developed Hive queries to process the data and generate the data cubes for visualizing.
• Gained experience in managing and reviewing Hadoop log files.

Environment: Hadoop, MapReduce, Sqoop, HDFS, HBase, Hive, Pig, Oozie, Spark, Kafka, Cassandra, AWS, Elasticsearch, Java, Oracle 10g, MySQL, Ubuntu, HDP.

Confidential, Pittsburgh, PA

Hadoop/Big Data Analyst

Responsibilities:

• Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, Hive, Hbase and Sqoop.
• Developed simple and complex MapReduce programs in Java for Data Analysis on different data formats
• Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Implemented different machine learning techniques using Weka machine learning library.
• Developed Spark programto analyze reports using Machine Learning models.
• Worked on Spark SQL and Data frames for faster execution of Hive queries using Spark Sql Context.
• Performed analysis on implementing Spark using Scalaandwrotespark sample programs using PySpark.
• Good exposure in development with HTML, Bootstrap,Scala
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala
• Worked in AWS environment for development and deployment of custom Hadoop applications.
• Strong experience in working with ELASTIC MAPREDUCE(EMR)and setting up environments on Amazon AWS EC2 instances.
• Experienced in implementing static and dynamic partitioning in hive.
• Experience in customizing map reduce framework at different levels like input formats, data types and partitioners.
• Was responsible for importing the data (mostly log files) from various sources into HDFS using Flume.
• Implemented API tool to handle streaming data using Flume.
• Created Oozie workflows to automate data ingestion using Sqoop and process incremental log data ingested by Flume using Pig.
• Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Involved in migrating hive queries and UDF’s in hive to Spark SQL.
• Extensively Used Sqoop to import/export data between RDBMS and hive tables, incremental imports and created Sqoop jobs for last saved value.
• Developed custom writable MapReduce JAVA programs to load web server logs into HBase using Flume.
• Responsible for Data Modeling as per our requirement in HBase and for managing and scheduling Jobs on a Hadoop cluster using Oozie jobs.
• Created customized BI tool for manager team that perform Query analytics using HiveQL.
• Involved in data migration from Oracle database to Mongo DB
• Involved in migrating tables from RDBMS into Hive tables using Sqoop and later generate visualizations using Tableau.
• Created Hive Generic UDF's to process business logic that varies based on policy.
• Experienced with using different kind of compression techniques to save data and optimize data transfer over network using Lzo, Snappyetc. in Hive tables.
• Creating HBase tables for random read/writes by the map reduce programs.
• Creating Hive tables, loading with data and writing Hive queries which will run internally in Map Reduceway
• Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSON and XML file formats
• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager
• Designed the ETL process and created the high-level design document including the logical data flows, source data extraction process, the database staging, job scheduling and Error Handling
• Developed and designed ETL Jobs usingTalend Integration Suite in Talend 5.2.2
• Created ETL Mapping with Talend Integration Suite to pull data from Source, apply transformations, and load data into target database.

Environment: Hadoop, Cloudera (CDH 4), HDFS, Hive, HBase, Flume, Sqoop, Pig, Kafka Java, Eclipse, Teradata, Tableau, Talend, MongoDB, Ubuntu, UNIX, and Maven.

Confidential, Houston, TX

Hadoop/ETL Developer

Responsibilities:

• Worked with terabytes of structured and unstructured data (240 TB with replication factor coming in from multiple web sources).
• Designed and developed entire pipeline from data ingestion to reporting tables.
• Scrutinized Hadoop Log Files, executed performance scripts.
• Used Cloudera Manager to monitor and manage Hadoop Cluster.
• Worked on streaming log data into HDFS from web servers using Flume.
• Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements. Extracted data from Oracle database, Teradata database into HDFSusing Sqoop.
• Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS
• Involved in loading data from LINUX file system to HDFS Importing and exporting data into HDFS and Hive using Sqoop Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Good understanding of Amazon web services like Elastic MapReduce (EMR), EC2.
• Working knowledge of MapR and Teradata unison to optimize high availability (HA).
• Installed, configured and participated in Hadoop MapReduce, Pig, Hive, Oozie, Sqoop environment.
• Extending HIVE and PIG core functionality by using custom User Defined Function's (UDF), User Defined Table- Generating Functions (UDTF) and User Defined Aggregating Functions (UDAF) for Hive and Pig Used
• Experienced in running Hadoop streaming jobs to process terabytes of JSON format data.
• Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs
• Performed data cleaning, integration, transformation, reduction by developing MapReduce jobs in javafor data mining
• Importing and exporting data into MapR-FS from various web servers.
• Creating Hive tables, loading data into it and customizing hive queries, internally operating in MapReduce way.
• Performed Map-Side joins and Reduce-Side joins for large tables.
• Defining a schema, creating new relations, performing Pig-Join, sorting and filtering using Pig-Group on large data sets.

Environment: Hadoop, MapReduce, HDFS, Pig, Hive, HBase, Oozie, Cloudera CDH4.5, Cloudera CDH5.1.3, SQL, LINUX, Java, J2EE, Web services, PostgreSQL, DB2.

Confidential

Java Developer

Responsibilities:

• Responsible for requirement gathering, requirement analysis, defining scope, and design.
• Developed the Use Cases, Class Diagrams and Sequence Diagrams using Rational Rose.
• Developed user Interface using JSP and HTML.
• Written Server-Side programs using Servlets.
• Used Java Script for client-side Validation.
• Used HTML, AWT with Java Applets to create web pages.
• Responsible for database designand developed stored procedures and triggers to improve the performance.
• Used Eclipse IDE for all coding in Java, Servlets and JSPs.
• Co-ordinate with the QA lead for development of test plan, test cases, test code and actual testing, responsible for defects allocation and ensuring that the defects are resolved.
• Used Flex Styles and CSS to manage the Look and Feel of the application.
• Deployed the application on Web SphereApplication server.

Environment: Java2.0, Eclipse, Apache Tomcat Web Server, JSP, JavaScript, AWT, Servlets, JDBC, HTML, Front Page 2000, Oracle, Win NT, CVS.

Confidential

Java Developer

Responsibilities:

• Setting up environment such as deploying application, maintaining Web server.
• Analyzing the Defects and providing fix.
• Developed Servlets and Java Server Pages (JSP), to route the submittals to the EJB components. Java Scripting handled the Front-end validations
• Wrote Korn Shell build scripts for configuring and deployment of GUI application on UNIX machines.
• Created Session Beans and controller Servlets for handling HTTP requests from JSP pages.
• Eclipse IDE used for the developing the application.
• Used Session Facade, Singleton design patterns.
• Development of XML files using XPATH, XSLT, XPointer, DTD’s, Schema’s and Parsing using both SAX and DOM parsers.
• Designed and developed XSL style sheets using XSLT to transform XML and display the Customer Information on the screen for the user and for processing.
• Extensively used Clear Case, the version control tool.

Environment: Java, J2EE, Oracle, RMI, ClearCase, JDBC, UNIX, Junit, Eclipse, WebSphere5.0, Struts, XML, XSLT, XPATH, XHTML, CSS, HTTP