SUMMARY

• Around 8 years of experience as Hadoop/Spark developer using Big data technologies like Hadoop Ecosystem, Spark Ecosystem.
• Extensive experience in working with structured/semi - structured and unstructured data by implementing complex map reduce programs using design patterns.
• Experience in Hadoop and its Eco System i.e., HDFS, MapReduce, Apache Pig, Hive, HBase. Spark, Scala and Linux scripting, Hortonworks.
• Experience with the Scala, Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Pair and YARN.
• Passionate towards working inHadoopand Big Data Technologies, data science, machine learning in Spark, Big Data Processing, Analytics and Visualization.
• Designed and developed RDD seeds using Scala, cascading and Streaming data.
• Good working experience in Scala Spark and SparkSQL.
• Hands on experience writing complex/custom data pipelines using scala, spark Hive and Pig.
• Experience in building highly scalable Big-data solutions using Hadoop distributed platforms like Cloudera/Hortonworks.
• Extensive experience in using eco systems components Map Reduce, Hive, HQL, Pig, Sqoop, Oozie, HBase and Apache Spark.
• Experience on the SparkCore, SparkSQL, Spark Streaming MLib, Graphx.
• Expertise on creating RDD's, Pair RDD's, Transformations and Actions in Spark.
• Familiarity with distributed coordination system Zookeeper.
• Having Experience in Loading Tuple shaped data into Pig and Generate Normal Data into Tuples. Ability to build User-Defined Functionalities(UDFs) not available in core Hadoop.
• Maintained, audited and built new clusters for testing purposes using the Hortonworks.
• Ability to move the data in and out of Hadoop RDBMS, No-SQL, UNIX and Mainframe from various systems using SQOOP and other traditional data movement technologies.
• Experience working in Scrum/Agile and waterfall model to successfully deliver on time projects.
• Experience in designing Map documents for Big-data applications
• Good understanding of NoSQL databases and hands on experience in writing applications on NoSQL databases such as HBASE.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Experience on SQL and No-SQL, Realational, Graph database systems.
• Having hands on experience in Data Warehousing to design and loading tables with large data and can develop the enterprise levels of data.
• Experience in data warehousing with ETL tool Oracle Warehouse builder
• Experience in domains like Railways, Insurance etc.

TECHNICAL SKILLS

Big Data Ecosystems: HDFS, YARN, Hive, Pig, HBase, Solr, Spark Core, Spark SQL, Spark Streaming.

BI tools: Ab Initio.

NoSQL: HBASE

Languages: SQL,JAVA, Python, Linux Shell Scripting, Scala, C, C++

Database: Oracle, SQL Server, MySQL.

Operating System: Windows, Linux, Ubuntu

SDLC: Agile/SCRUM, Waterfall

PROFESSIONAL EXPERIENCE

Confidential

Hadoop Developer

Responsibilities:

• Experience with the Hadoop ecosystem and NoSQL database
• Expertise in implementing Spark Scala application using higher order functions for both batch and interactive analysis requirement.
• Created Reports with different Selection Criteria from Hive Tables on the data residing in Data Lake.
• Deployed Hadoop components on the Cluster like Hive, HBase, Spark, Scalaand others with respect to the requirement.
• Used Spark UI to observe the running of a submitted SparkJob at the node level.
• Developed Spark jobs and Hive Jobs to summarize and transform data
• Developed code from scratch in Spark using SCALA according to the technical requirements.
• Involved in developing different components of system like Hadoop process involves Map Reduce & Hive.
• Involved in converting Hive/SQL queries into Spark transformations using Spark SQL and Scala.
• Used Scala Collection Framework to store and process the complex consumer informations.
• Worked in tuning Hive and Pig to improve performance and solved performance issues in Hiveand Pig scripts with understanding of Joins, Group and aggregation and how does it translate to Map Reduce jobs.
• Used both Hive context as well as SQL context of Spark to do the initial testing of the Spark job.
• Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and Spark.
• Involved in creating Hive tables, loading with data and writing hive queries.
• Developed Sqoop scripts for importing and exporting data into HDFS and Hive
• Exported the data into RDBMS using Sqoop for BI team to perform visualization and to generate reports
• Used Spark streaming to receive near real time data from the Kafka and store the stream data to HDFS using Scala and NoSQL database such as HDFS
• Involved in Analyzing data by writing queries using HiveQL for faster data processing
• Created customized BI tool for manager team that perform Query analytics using Hive QL.
• Developed Oozie workflows and they are scheduled through AutoSys on monthly basis
• Create a complete processing engine, based on Hortonwork's distribution, enhanced to performance.
• Utilized Agile and Scrum Methodology to help manage and organize a team of developers with regular code review session.

Environment: Hadoop 2.x, Spark Core, Spark SQL, Spark API Spark Streaming, Scala, Pyspark, Hive, Pig, kafka, Oozie, RDBMS, Hive, HDFS,YARN, JIRA, MapReduce.

Confidential

Hadoop Developer

Responsibilities:

• Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries and Pig scripts
• Implemented Hive complex UDF's to execute business logic with Hive Queries.
• Optimized MapReduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Worked with streaming for the proof of concept for carrying out log processing on a distributed system. Worked with NoSQL database HBase to create tables and store data.
• Used Oozie for deployment of the application and perforce as the secure versioning software.
• Developed Pig scripts to convert the data from Avro to text file format.
• Used Sqoop to import the data from RDBMS toHadoopDistributed File System (HDFS) and later analyzed the imported data usingHadoopComponents.
• Experienced in developing Spark scripts for data analysis in both python and scala.
• Responsible for building scalable distributed data solutions usingHadoop.
• Expertise in implementing Spark-Scala application using higher order functions for both batch and interactive analysis requirement.
• Involved in writing Hive scripts to extract, transform and load the data into Database
• Developed Pig scripts and UDF's as per the business rules.
• Performed code reviews, migrations from lower to higher environments
• Designed changes to transform currentETLjobs to Hadoop jobs with underlying database as HBase.
• Extracted large volumes of data feed on different data sources, performed transformations and loaded the data into various Targets.
• Involved in Agile SDLC during the development of project.

Environment: Spark, HDFS, Kafka, Sqoop, Scala,Hive, Oozie, NoSQL, Oracle, Zookeeper, Cassandra, Agile methodology, JIRA, AGILE(SCRUM).

Confidential

Abinitio Developer

Responsibilities:

• Understanding the business data model and customer requirements.
• Preparing high and low level designs of the system along with build strategy.
• Building various components i.e. Graphs, Scripts and Oracle Queries.
• Testing the changed and impacted components.
• Communicating timely statuses to the Client about the on-goings of the enhancement and code-reviews.
• Checking END-TO-END functionality of code by considering the performance and o/p relevance with the requirements.
• Unit testing for all the solutions to ensure minimum defects.
• Support for troubleshooting and fixing defects.
• Performing smooth cycle execution from source to target to provide test/production data.
• Production batch monitoring and fixing job failures.
• Resolving the incidents raised by users.
• Execution of IA cycle for every release.
• Responding to the defects raised by QA team and Supporting QA queries.
• Providing analysis report to BA queries for next releases.

Confidential

Application Developer

Responsibilities:

• Providing 24x7 supports to business users.
• Monitoring batch jobs.
• Solving production job failures and batch abends.
• Communicating with Front office and business users to resolve the business queries.
• Look into complex errors and production abends and makes changes in production server (Break fix).