SUMMARY:

• About 8+ years of professional IT experience which includes 4+ years of experience in Big data ecosystem related technologies like Hadoop HDFS, MapReduce (YARN), Apache Pig, Hive, Sqoop, Flume, Kafka, Oozie, Hbase, Spark and Scala.
• Excellent understanding and knowledge of Hadoop architecture and various daemons such as Name Node, Data Node, Job Tracker, Task Tracker, Resource Manager, Node Manager, and MapReduce programming paradigm.
• Hands on experience using Cloudera, MapR, Hortonworks and AWS Hadoop Distributions.
• Experienced in Amazon AWS like EMR, S3 and EC2 web services.
• Expertise in Data load management, importing & exporting data using Sqoop, Flume and Teradata Connector .
• Hands on experience in configuring and working with Flume to load the data from multiple sources directly into HDFS.
• Expertise in Creating Hive Internal/External Tables/Views using shared Meta store, writing scripts in HiveQL also data transformation & file processing using Pig Latin Scripts .
• Experienced in writing Map Reduce programs and UDFs for both Hive and Pig in Java.
• Expertise in NoSQL column oriented databases like HBase and its integration with Hive & Pig .
• Involved in importing the real - time data to Hadoop using Kafka and implemented the Oozie job for daily
• Involved in developing Hive DDLs to create, alter and drop Hive tables and storm, & Kafka.
• Experienced in transferring data from different data sources into HDFS systems using Kafka producers, consumers and Kafka brokers
• Having good knowledge in Benchmarking & Performance Tuning of cluster.
• Hands on experience in setting up workflow using Apache Oozie workflow engine for managing and scheduling Hadoop jobs using Oozie-Coordinator .
• Strong understanding in NoSQL databases like HBase & Cassandra .
• Worked on Kafka and Flume for handling megabytes of streaming data.
• Good experience in working with cloud environment like Amazon Web Services () EC2 and S3.
• Hands on experience on working with Amazon EMR framework transferring data to EC2 server.
• Handling different file formats on Parquet, ProtoBuff (Protocol Buffer), and Avro, Sequence, JSON, XML, CSV and Flat files.
• Hands on experience in dealing with Compression tools like Snappy, Gzip .
• Experience in implementing Spark using Scala and Spark SQL for faster analyzing and processing of data.
• Experience in Spark, and in-depth knowledge on Spark-SQL, RDD's, Lazy transformation and actions.
• Experience in writing database objects like Stored Procedures, Functions, Triggers, PL/SQL packages and Cursors for Oracle, SQL Server and MySQL .
• Detailed understanding of Software Development Life Cycle (SDLC) and sound knowledge of project implementation methodologies including Waterfall and Agile .

TECHNICAL SKILLS:

Hadoop/BigData Technologies: HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Oozie, Avro, Zookeeper, Kafka, Impala, Apache Spark, Spark Streaming, Spark-SQL, Hue, Ambari, Windows Azure, Solr

Programming Languages: Java, C, PYTHON, SCALA, SQL, PL/SQL, PIG-Latin, HQL

Databases: Oracle, MySQL, DB2, HBase, Cassandra, Teradata

IDE Tools: Eclipse 3.x, NetBeans

Web Technologies: HTML5, CSS3, JavaScript, JSP, Servlets, JNDI, JDBC, Java Beans

Web Services: SOAP, REST

Operating Systems: Windows (XP,7,8,10), UNIX, LINUX, Ubuntu, CentOS

Reporting Tools: ETL Tools Tableau, Microsoft Power BI

PROFESSIONAL EXPERIENCE:

Confidential, Austin, TX

Hadoop/Spark Developer

Responsibilities:

• Worked closely with the Source System Analysts and Architects in identifying the attributes and to convert the Business Requirements into Technical Requirements.
• Worked closely with SME to prepare a tool using Map Reduce to maintain versioning of the records and involved in setting up the standards for SCD2 Mapper.
• Created Hive, Phoenix, HBase tables and HBase integrated Hive tables as per the design using ORC file format and Snappy compression.
• Experience in integrating Cassandra with Elastic Search and Hadoop.
• Imported required tables from RDBMS to HDFS using Sqoop and also used Storm and Kafka to get real time streaming of data into HBase.
• Wrote different pig scripts to clean up the ingested data and created partitions for the daily data.
• Implemented Partitioning and bucketing in Hive based on the requirement.
• Wrote different UDF's to convert the date format and to create hash value using MD5 Algorithm in Java and used various UDF's from Piggybanks and other sources.
• Prepared pig scripts and Spark SQL/Spark streaming to handle all the transformations specified in the S2TM's and to handle SCD2 and SCD1 scenarios.
• Worked on a POC to compare processing time of Impala with Apache Hive for batch applications to implement the former in project
• Used Impala for aggregating jobs and runs an average of 100K aggregation queries per day.
• Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.
• Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Python and Scala.
• Hands on expertise in running the SPARK & SPARK SQL on AMAZON ELASTIC MAPREDUCE (EMR).
• Built-in Request builder, developed in Scala to facilitate running of scenarios, using JSON configuration files
• Migrated the code base from Cloudera Platform to Amazon EMR and evaluated Amazon eco systems components like RedShift, Mongo DB
• Worked on Apache spark writing Python applications to convert txt, xls files and parse data into JSON format.
• Experienced in implementing Spark RDD transformations, actions to implement business analysis and Worked with Spark accumulators and broadcast variables.
• Created shell scripts to parameterize the Pig, Hive actions in Oozie workflow.
• Utilize the give OLTP data models for Source Systems to design a Star Schema.
• Created tables in Teradata to export the data from HDFS using Sqoop after all the transformations and wrote Bteq scripts to handle updates and inserts of the records.
• Involved in Unit level and Integration level testing and prepared supporting documents for proper deployment.

Environment: CDH 5.5.1, Hadoop, Map Reduce, HDFS, Kafka, Hive, Pig, Sqoop, Ambari, Spark, Oozie, Impala, Shell Scripting, SQL, MongoDB, Java (jdk 1.6), Eclipse.

Confidential, Austin, TX

Hadoop/Spark Developer

Responsibilities:

• Migrating the needed data from Oracle, MySQL in to HDFS using Sqoop and importing various formats of flat files in to HDFS.
• Uploaded and processed more than 30 terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
• Designing and creating Hive external tables using shared meta-store instead of derby with partitioning, dynamic partitioning and buckets.
• Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
• Platform; configured the 10 nodes - 3 Web servers, 4 Kafka brokers and 3 Kafka consumers (Spark Strersming (DataFrames) with 3 Zookeeper nodes; Kafka brokers able to sustain 1 million wirtes per second.
• Analyzed the data by performing Hive queries and running Pig scripts to validate sales data.
• Used AWS services like EC2 and S3 for small data sets.
• Experience working on Solr to develop search engine on unstructured data in HDFS.
• Configured Spark streaming to receive real time data from the Kafka and store the stream data (after analysis is done) to No SQL Cassandra using Spark, Java and Scala Oozie workflows
• Written the Map Reduce programs, Hive UDFs & Pig UDFs in Java.
• Created external tables pointing to HBASE to access table with huge number of columns.
• Wrote Python code using happybase library of Python to connect to HBASE and use the HAWQ querying as well.
• Used Spark SQL to process the huge amount of structured data and Implemented Spark RDD transformations, actions to migrate Map reduce algorithms.
• Used Tableau for data visualization and generating reports.
• Created SSIS packages to extract data from OLTP and transformed to OLAP systems and Scheduled Jobs to call the packages and Stored Procedures. Created Alerts for successful or unsuccessful completion of Scheduled Jobs.
• Developed Python scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.
• Developed Spark code using Scala and Spark-SQL for faster testing and data processing.
• Worked on converting PL/SQL code into Scala code and also converted PL/SQL queries into HQL queries.
• Built-in Request builder, developed in Scala to facilitate running of scenarios, using JSON configuration files
• Involved in scheduling Oozie workflow engine to run multiple Hive and Pig jobs.

Environment: Cloudera Distribution CDH 5.5.1, Oracle 12c, HDFS, Map Reduce, Hive, HBase, Pig, Oozie, Sqoop, Flume, Hue, Tableau, Scala, Spark, Kafka Zookeeper, MongoDB, PL/SQL, UNIX shell scripts, Java, Python, AWS S3.

Confidential, Santa Clara, CA

Hadoop Developer

Responsibilities:

• Worked with a team to gather and analyze the client requirements
• Analyzed large data sets distributed across cluster of commodity hardware
• Connecting to Hadoop cluster and Cassandra ring and executing sample programs on servers
• Hadoop and Cassandra as part of Next generation platform implementation
• Developed several advanced MapReduce (YARN) programs to process received data files
• Responsible in building scalable distributed data solutions using Hadoop
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce YARN, loaded data into HDFS and Extracted the data from Oracle into HDFS using Sqoop
• Bulk loaded data into Cassandra using Stable loader
• Load the OLTP models and Perform ETL to load Dimension data for a Star Schema
• Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior
• Uploaded and processed more than 30 terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
• Used Apache, MongoDB (NoSQL) in AWS Linux instance to store and analyze data.
• Good experience with cloud platforms like Amazon AWS, Azure and Google App Engine.
• Involved in HDFS maintenance and loading of structured and unstructured data
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig
• Data was formatted using Hive queries and stored on HDFS
• Created complex schema and tables for analysis using Hive
• Worked on creating MapReduce programs to parse the data for claim report generation and running the Jars in Hadoop . Coordinated with Java team in creating MapReduce programs.
• Implemented the project by using Spring Web MVC module
• Responsible for managing and reviewing Hadoop log files. Designed and developed data management system using MySQL
• Cluster maintenance as well as creation and removal of nodes using tools like Cloud Era Manager Enterprise, and other tools.
• Followed Agile methodology, interacted directly with the client provided & receive feedback on the features, suggest/implement optimal solutions, and tailor application to customer needs.

Environment: Cloudera Distribution CDH 4.4.1, Hadoop HDFS, MapReduce, Hive, PIG, Hbase, Cassandra, Sqoop, Oozie, AWS, UNIX Shell Scripting, Linux, SQL, Web services, Micro services, IOC.

Confidential

Java Developer

Responsibilities:

• Gathered requirements, designed and implemented the application that utilizes Struts, Spring, JSP and Oracle database.
• Implemented J2EE design patterns like MVC and Front Controller.
• Implemented Static and Dynamic web pages using JSP, Java Script, CSS.
• Involved in Requirement analysis, design and provide the estimation.
• Responsibilities include designing and delivering web based J2EE solutions. Used JavaScript for Client Side validations.
• Involved in writing PL/SQL queries and stored procedures.
• Responsible for setup the environment and Production Environments in Server and Database level.
• Involved in developing portlets and deploying in Weblogic Portal Server.
• Involved in writing of release notes to deploy in various environments and production.
• Monitored the Server load average and prepare status report on it.
• Point of Contact to the client for all technical aspects.
• Prepared status reports.

Environment: JSP, JDBC, HTML, Java, Servlets, SQL Developer, Neat Beans IDE and Apache Tomcat.

Confidential

Jr. Java Developer

Responsibilities:

• Used Hibernate ORM tool as persistence Layer - using the database and configuration data to provide persistence services (and persistent objects) to the application.
• Implemented Oracle Advanced Queuing using JMS and Message driven beans.
• Responsible for developing DAO layer using Spring MVC and configuration XML's for Hibernate and to also manage CRUD operations (insert, update, and delete).
• Implemented Dependency injection of spring frame work.
• Developed and implemented the DAO and service classes.
• Developed reusable services using BPEL to transfer data.
• Participated in Analysis, interface design and development of JSP.
• Configured log4j to enable/disable logging in application.
• Wrote SPA (Single Page Web Applications) using RESTFUL web services plus Ajax and AngularJS.
• Developed Rich user interface using HTML, JSP, AJAX, JSTL, Java Script, JQuery and CSS.
• Implemented PL/SQL queries, Procedures to perform data base operations.
• Wrote UNIX Shell scripts and used UNIX environment to deploy the EAR and read the logs.
• Implemented Log4j for logging purpose in the application.
• Implemented agile development methodology.

Environment: Java, Spring, Hibernate, JMS, EJB, Web logic Server, JDeveloper, SQL Developer, Maven, XML, CSS, JavaScript, JSON.