PROFESSIONAL SUMMARY:

• 7+ years of IT experience which includes 4+ years of experience as Hadoop/Spark developer using Big data technologies like Hadoop Ecosystem , Spark Ecosystem and 3+ years of experience in application development using J2EE .
• Experience in working on various Hadoop data access components like MAPREDUCE, PIG, HIVE, HBASE, SPARK and SOLR.
• Having good knowledge on Hadoop data management components like HDFS and YARN .
• Hands on experience in using various Hadoop workflow compononets like SCOOP , FLUME a nd KAFKA .
• Worked on Hadoop data operation components like ZOOKEEPER and OOZIE .
• Working knowledge on AWS technologies like S3 and EMR for storage , big data processsing and analysis .
• Good understanding of Hadoop security components like RANGER and KNOX .
• Good experience working with Hadoop distributions such as HORTONWORKS and CLOUDERA .
• Excellent programming skills at higher level of abstraction using SCALA and JAVA .
• Experience in Java programming with skills in analysis, design, testing and deploying with various technologies like J2EE, JavaScript, JSP, JDBC, HTML, XML and JUNIT.
• Having good knowledge on Apache Spark components including SPARK CORE , SPARK SQL , SPARK STREAMING and SPARK MLLIB .
• Experience in performing transformations and actions on Spark RDDS using Spark Core .
• Experience in using Broadcast variables, Accumulator variables and RDD caching in Spark .
• Experience in troubleshooting Cluster jobs using Spark UI .
• Experience in using dependency management tools like SBT to create a self contained jar file.
• Created Data Frames and Data Sets using Spark SQL .
• Experience in creating DStreams using Spark Streaming .
• Experience in performing Extract - Transform-Load(ETL) operations on data pipelines using Pig .
• Simplified Complex tasks involving interrelated data transformations and encoded data flow sequences using Pig Latin .
• Expertise in creating Managed/External tables , Views , Partitions, Buckets and Analytical functions in Hive using HQL .
• Solved performance issues in Hive and Pig scripts with good understanding of Joins and Group .
• Extended Hive and Pig core functionality by writing custom User Defined Functions in Java .
• Experience in using Sequence files , ORC , AVRO and Parquet file formats .
• Good understanding of Hive extensions like HCatalog to allow other frameworks like Pig and MapReduce to share a common view on structured and unstructred data in HDFS using Hive metadata.
• Created very large tables for storing multi-structured/sparse data using Hbase .
• Imported data from ORACLE 11g to HDFS using Sqoop .
• Integrated Kafka and Apache Spark for real-time analysis and rendering of streaming data .
• Experience in implementing scheduler using Oozie .
• Experience in using security tools like Ranger and Knox to manage access control over Hadoop data access components like Hive and Hbase .
• Proficient in Shell Scripting .
• Used Project Management services like JIRA for handling service requests and tracking issues.
• Good experience with Software methodologies like Agile and Waterfall .

TECHNICAL SKILLS:

Data Access Tools: HDFS, YARN, Hive, Pig, HBase, Solr, Impala, Spark Core, Spark SQL, Spark Streaming

Data Management: HDFS, YARN

Data Workflow: Sqoop, Flume, Kafka

Data Operation: Zookeper, Oozie

Data Security: Ranger, Knox

BigData Distributions: Hortonworks, Cloudera

Cloud Technologies: AWS (Amazon Web Services) EC2, S3, IAM, CLOUD WATCH, DynamoDB, SNS, SQS, EMR, KINESIS

Programming and Scripting Languages: Java, Scala, Pig Latin, HQL, SQL, Shell Scripting, HTML, CSS, JavaScript

IDE/Build Tools: Eclipse, Intellij

Java/J2EE Technologies: XML, Junit, JDBC, AJAX, JSON, JSP

Operating Systems: Linux, Windows, Kali Linux

SDLC: Agile/SCRUM, Waterfall

PROFESSIONAL EXPERIENCE:

Confidential, Milwaukee, WI

Hadoop/spark Developer

Responsibilities :

• Worked on Big Data infrastructure for batch processing as well as real-time processing . Responsible for building scalable distributed data solutions using Hadoop .
• Experienced in working with spark eco system including Spark Core , Spark SQL and Spark Streaming .
• Imported data from AWS S3 into Spark RDD and performed transformations and actions on RDD’s .
• Converted existing MapReduce jobs into Spark transformations and actions using Spark RDDs , Data frames and Spark SQL APIs .
• Wrote new spark jobs in Scala to analyze the data of the customers and sales history.
• Involved in requirement analysis , design , coding and implementation phases of the project.
• Used Spark API over Hadoop YARN to perform analytics on data in Hive .
• Experience in both SQLContext and SparkSession .
• Developed Scala based Spark applications for performing data cleansing , data aggregation , de-normalization and data preparation needed for machine learning and reporting teams to consume.
• Worked on troubleshooting spark application to make them more error tolerant.
• Involved in HDFS maintenance and loading of structured and unstructured data and imported data from mainframe dataset to HDFS using Sqoop and written the PySpark Script to process the HDFS data.
• Experienced in handling large datasets using Spark in Memory capabilities , broadcasts variables , effective and efficient Joins, transformations and other capabilities.
• Involved in creating Hive tables , loading and analyzing data using Hive scripts .
• Implemented Static partitions , Dynamic partitions and Buckets in Hive .
• Implemented Spark using Scala and also used Pyspark using Python for faster testing and processing of data and developed Python code to gather the data from HBase and designs the solution to implement using PySpark.
• Developed Oozie workflows to automate and productionize the data pipelines .
• Used Reporting tools like Tableau to connect with Impala for generating daily reports of data.
• Designed, documented operational problems by following standards and procedures using JIRA .
• Collaborated with the infrastructure, network, database, application and BA teams to ensure data quality and availability.

Environment : Hadoop 2.x, Spark Core, Spark SQL, Spark Streaming, Pyspark, Hive, Oozie, Machine Learning, Amazon EMR, Tableau, Impala, JIRA.

Confidential, Charlotte, NC

Hadoop Developer

Responsibilities:

• Responsible for planning , organizing , and implementing complex business solutions , producing deliverables within stipulated time.
• Developed data pipelines using Pig and Java MapReduce to consume customer behavioral data and financial antiquities into HDFS for analysis.
• Created Partitioned tables in Hive and worked on them using HQL .
• Involved in using HCatalog to access Hive metadata from Map Reduce or Pig.
• Experience in creating, dropping and altering tables at run time without blocking updates and queries using HBase.
• Experience on working with Apache Solr for indexing and load balanced querying to search for specific data in larger datasets and implemented Near Real Time Solr index on Hbase and HDFS .
• Developed Spark jobs using Scala on top of Yarn for interactive and Batch Analysis.
• Developed Unix shell scripts to load large number of files into HDFS from Linux File System.
• Experience in querying data using Spark SQL for faster processing of the data sets.
• Offloaded data from EDW into Hadoop Cluster using Sqoop .
• Experience working with EMR cluster in AWS cloud and working with S3 .
• Hands on experience on Hortonworks tools like Ambari and Tez .
• Used Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Used Oozie workflow engine to create the workflows and automate the MapReduce , Hive and Pig jobs.
• Created Impala views for users to access history data.

Environment : Hadoop, MapReduce, Pig, Hive, Hbase, Sqoop, Oozie, Spark, Solr, Impala.

Confidential, St. Louis, MO

Hadoop Developer

Responsibilities :

• Involved in gathering business requirements , design (Low-level and High-level), development and testing .
• Used flume to collect the entire web log from the online ad-servers and pushed into HDFS .
• Written Map Reduce jobs in java to process the log data.
• Built complex data flows involving multiple inputs, transforms, and outputs using Pig .
• Written Apache Pig Scripts for loading, filtering and storing the data into HDFS , which later can be used by Hive or MapReduce jobs .
• Created Hive tables to store the processed results in a tabular format .
• Used Partitioning and Bucketing concepts for performance optimization in Hive .
• Developed Hive User Defined Functions (UDF’s) in Java , compiled them into jar’s and added them to the HDFS and executed them using Hive Queries .
• Used Apache HBase ’s low latency storage for scenarios that require real-time analysis and tabular data for end user applications.
• Implemented Publish/Subscribe model using Apache Kafka for real-time transactions to load into Hbase .
• Made use of Distributed cache to improve the performance of the jobs and reduce the load on cluster.
• Automated workflows using Oozie jobs.
• Involved in loading data from UNIX file system to HDFS and Created custom Solr Query components to enable optimum search matching .
• Worked on Impala to analyze stored data.
• Implemented POC to migrate MapReduce programs into Spark transformations using Spark and Scala .
• With deep and thorough understanding of ETL tools and how they can be applied in a Big Data environment , Invloved in managing Hadoop Clusters using Cloudera .
• Involved in securing Hadoop Clusters using security components of Hadoop like Ranger and Knox .

Environment : Hadoop, MapReduce, Pig, Hive, Hbase, Sqoop, Flume, Kafka, Oozie, Impala, Spark, Solr, Ranger, Knox.

Confidential

Java/J2EE Developer

Responsibilities:

• Involved in Analyzing the client requirements and functional areas of the existing system, client interactions, coordinating with the team members on critical problems and solving, technical guidance
• Involved in Agile Scrum methodology, agile meeting , and Sprint and Release Planning meetings .
• Involved in design development and testing of web application and integration projects using Object Oriented technologies such as Core Java, J2EE, Struts, JSP, JDBC, Spring Framework, hibernate, Java Beans, Web Services (REST/SOAP), XML, XSLT, XSL and ANT
• Skilled in using Rally Development software to meet the Sprint goals .
• Responsible for developing and enhancing the Client/Server application using J2EE standard open-source framework, Spring, REST, Hibernate, Core Java, Collections, Multithreading .
• Involved in implementation of enterprise integration with web services and legacy systems using REST API .
• Created Auto deploys job using Jenkins and Maven . Developed Spring DAO templates to all type of CRUD operations.
• Implemented Java persistence API (JPA) through Hibernate 3.0.
• Composed and executed MongoDB scripts to insert and update NoSQL database , automating data changes and reporting.
• Used JP-QL language as part of the Java PersistenceAPI (JPA) specification to perform transactions with MongoDB .
• Designed and Developed white-box test cases using JUnit, Git, JMeter, Mockito Framework and Eclipse .
• Created Maven pom.xml files and integrated with Jenkins .
• Configured Log4j mechanism for complete project using xml files.

Environment: Core Java, J2EE, Struts, JSP, JDBC, Spring Framework, hibernate, Java Beans, JPA, REST,SOAP, XML, XSLT, XSL and ANT, Jenkins and Maven, NoSQL, MongoDB, JUnit, Git JMeter Mockito Framework and Eclipse IDE .

Confidential

Software Engineer

Responsibilities:

• Responsible for the analysis, documenting the requirements and architecting the application based on J2EE standards.
• Attended Scrum meetings daily as a part of Agile Methodology .
• Involved in complete Software Development Life Cycle (SDLC) with Object Oriented Approach of client's business process and continuous client feedback.
• Implementing MVC Architecture using Spring Framework , customized user interfaces. Used Core Java , and Spring Aspect Oriented programming concepts for logging, security, error handling mechanism
• Developed application modules using Spring MVC, Spring Annotations, Spring Beans, Dependency Injection, with database interfaceusing Hibernate.
• Used the Java Collections API extensively in the application as security protection for XML, SOAP, REST and JSON to make a secure Web Deployment.
• Developed server-side services using Java, Spring, Web Services (SOAP, Restful, WSDL, JAXB, JAX-RPC)
• Built Web pages that are more user-interactive using JQuery plugins for Drag and Drop, AutoComplete, AJAX, JSON, Angular JS, JavaScript and Bootstrap.
• Used XSL to transform XML data structure into HTML pages.
• Used Struts as the framework in this project and developed struts action classes, form beans .
• Created dispatch Action classes, and Validation plug-in using Struts framework .
• DB2 was used as the database and wrote queries to extract data from the database.
• Developed SQL queries and stored procedures.
• Designed Developed white-box test cases using JUnit, Git, JMeter, Mockito Framework .

Environment: Java,J2EE, Spring MVC, Hibernate, SOAP, REST, WSDL, JAXB, JAX-RPC, Angular Js, JQuery, AJAX, JSON, JavaScript, Bootstrap, XSL, XML, Struts, DB2, JUnit, Log4j, NetBeans IDE.