PROFESSIONAL SUMMARY:

• Professional Software developer with 8+ years of technical expertise in all phases of Software development life cycle (SDLC), expertizing in Big Data Technologies like Hadoop and Spark Ecosystem and Java/J2EE.
• 3+ years of industrial experience in Big Data analytics, Data manipulation, using Hadoop Ecosystem MapReduce, HDFS, Yarn/MRv2, Pig, Hive, HBase, Spark, Kafka, Flume, Oozie, Sqoop, AWS, Cassandra, Solr, NiFi and Zookeeper.
• Extensive experience in developing applications that perform Data Processing tasks using Teradata, PostgreSQL, Oracle database.
• Hands on expertise in working and designing of Row keys & Schema Design with NoSQL databases like MongoDB, HBase, Cassandra and DynamoDB (AWS).
• Extensively worked on Spark using Scala on cluster for computational (analytics), On top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
• Excellent Programming skills at a higher level of abstraction using Scala, Java and Python.
• Hands on experience in developing SPARK applications using Spark libraries like Spark core, Spark MLlib, Spark Streaming and Spark SQL.
• Experience in D - Streams, Accumulator, Broadcast variables, RDD caching for Spark Streaming.
• Strong experience on real time data analytics using Spark Streaming, Kafka, Storm and NiFi.
• Working knowledge of Amazon’s Elastic Cloud Compute(EC2) infrastructure for computational tasks and Simple Storage Service(S3) as Storage mechanism.
• Experience working on Docker components like Docker Engine, Hub, Machine, creating Docker images, Compose Docker Registry and handling images primarily for middleware installation & domain configuration.
• Expertise in developing Pig Latin scripts and using Hive Query Language.
• Developed Customized UDFs in java to extend HIVE and Pig core functionality.
• Created Hive tables to store structured data into HDFS and processed it using HiveQL.
• Worked on GUI Based Hive Interaction tools like Hue, Karmasphere for querying data.
• Learnt and trained to work on Google cloud platform(GCP), Kubernetes, dataflow, Pub/Sub, Big query etc.,
• Expertise in software development using python and its packages like NumPy, SciPy, Pandas etc.,
• Implemented NiFi flow topologies to perform cleansing operations before moving data into HDFS.
• Experience in using NiFi in orchestrating the flow of data between different software systems.
• Experienced in migrating form on premise to AWS using AWS Data Pipeline and AWS Firehose.
• Running EMR step instances base on spot instance price with auto termination.
• Expertise in deploying lambda functions using java and python when we have a simple compute workload.
• Experience writing python script to spin up EMR cluster along with shell scripting.
• Experience in validating and cleansing data using PigLatin, experience in developing Pig MACROS.
• Working knowledge in installing and maintaining Cassandra by configuring the cassandra.yaml file as per the business requirement and performed reads/writes using Java JDBC connectivity.
• Experience in writing Complex SQL queries, PL/SQL, Views, Stored procedure, triggers.
• Experience in OLTP and OLAP design, development, testing and support of Data warehouses.
• Written multiple MapReduce Jobs using Java API, Pig and Hive for data extraction, transformation and aggregation from multiple file formats including Parquet, Avro, XML, JSON, CSV, ORCFILE and other compressed file formats Codecs like gzip, Snappy, LZO.
• Good experience in optimizing MapReduce algorithms using Mappers, Reducers, combiners and partitioners to deliver the best results for the large datasets.
• Good knowledge on build tools like Maven, Ant, sbt, Gradle.
• Experienced in migrating data from different sources into PUB-SUB model in Kafka using producers, consumers and pre-process data using Storm topologies.
• Had competency in using Chef, Puppet and Ansible configuration and automation tools. Configured and administered CI tools like Jenkins, Hudson Bambino for automated builds.
• Hands on experience in using various Hadoop distros Cloudera, Hortonworks and Amazon EMR Hadoop distributions.
• Knowledge on installation, configuration, supporting and managing Hadoop Clusters using Apache, Hortonworks and on Amazon web services (AWS).
• Experienced in writing Ad Hoc queries using Presto and Impala. Also used Impala analytical functions.
• In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, MapReduce Programming Paradigm, High Availability and YARN architecture.
• Proficient in developing, deploying and managing the Solr from development to production.
• Experience in importing data using Sqoop from various sources like RDBMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark.
• Used various Project Management services like JIRA for tracking issues, bugs related to code and GitHub for various code reviews and Worked on various version control tools like CVS, GIT, SVN.
• Experienced in checking status of cluster using Cloudera manager, Ambari, Ganglia and Nagios.
• Hands-on knowledge in Core Java concepts like Exceptions, Collections, Data-structures, I/O, Multithreading, Serialization and deserialization of streaming applications.
• Experience in Software Design, Development and Implementation of Client/Server Web based Applications using JSTL, jQuery, JavaScript, Java Beans, JDBC, Struts, PL/SQL, SQL, HTML, CSS, PHP, XML, AJAX, good understanding of React JavaScript Library.
• Experience in maintaining an Apache Tomcat MYSQL, LDAP, LAMP, Web service environment.
• Ability to work with Onsite and Offshore Teams.
• Designed ETL workflows on Tableau, Deployed data from various sources to HDFS.
• Generated various kinds of reports using Power BI and Qlik based on Business specification.
• Clustering, regression and Classification using Machine learning libraries Mahout, MLlib(Spark).
• Good experience with use-case development, with methodologies like Agile and Waterfall.
• Good understanding of all aspects of Unit, Regression, Agile, White & Black-box testing.
• Proven ability to manage all stages of project development Strong Problem Solving and Analytical skills and abilities to make Balanced & Independent Decisions.

TECHNICAL SKILLS:

Big Data Ecosystem: HDFS, MapReduce, Pig, Hive, Spark, YARN, Kafka, Flume, Sqoop, Solr, Impala, Presto, Oozie, ZooKeeper, Spark, Ambari, Mahout, NiFi.

Hadoop Distributions: Cloudera, Hortonworks, MapR and Apache

Languages: Java, Python, Scala, C/C++, go.

NoSQL Databases: Cassandra, MongoDB, HBase, DynamoDB, Bigtable.

Java Technologies: JSE, Servlets, JavaBeans, JSP, JDBC, JNDI, AJAX, EJB and struts

XML Technologies: XML, XSD, DTD, JAXP (SAX, DOM), JAXB

Web Design: HTML, DHTML, AJAX, JavaScript, JQuery and CSS, AngularJs, PHP

Development / Build Tools: Eclipse, Jenkins, Git, Ant, Maven, IntelliJ, JUNIT and log4J.

App/Web servers: WebSphere, WebLogic, JBoss and Tomcat

RDBMS: Teradata, Oracle 10g,11i, MS SQL Server, MySQL, DB2, PostgreSQL

Testing: MRUnit Testing, Quality Center (QC)

Virtualization: VMWare, Docker, AWS/EC2, Google Compute Engine, Vagrant.

PROFESSIONAL EXPERIENCE:

Confidential, Charlotte, NC

Hadoop/Spark Developer

Responsibilities:

• Developed Spark Applications by using Scala, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.
• Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD, Spark YARN.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data from Kafka in Near real time and persist it to Cassandra.
• Developed Kafka consumer API in Scala for consuming data from Kafka topics.
• Consumed XML messages using Kafka and process xml file using Spark Streaming to capture UI updates.
• Pre-processed jobs using Spark Data frames to flatten JSON documents to flat file.
• Load D-Stream data into Spark RDD and in memory data Computation to generate Output response.
• Experienced in Real-time Processing and core jobs using Spark Streaming with Kafka as a data pipeline.
• Worked and learned a great deal from AWS Cloud services like EC2, S3, EBS, RDS and VPC.
• Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Experience working on AWS lambda to create transformations on data between S3 buckets and EMR.
• Experience using AWS Data pipeline to migrate on premise data to S3 and DynamoDb.
• Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDDs.
• Experience in munging historical data and storing it in Amazon Redshift for data warehousing and use of Redshift apis to interact with data stored on Redshift.
• Experience in developing Docker images and deploying Docker containers in swarm.
• Implemented Elastic Search on Hive data warehouse platform.
• Worked with EMR and setup Hadoop environment in AWS EC2 Instance.
• Good understanding of Cassandra architecture, replication strategy, gossip, snitches etc.
• Designed columnar families in Cassandra and Ingested data from RDBMS, performed data transformations, and then exported the transformed data to Cassandra as per the requirement.
• Used the Spark DataStax Cassandra Connector to load data to and from Cassandra.
• Experienced in using Presto to query Cassandra and live topic data from Kafka.
• Experienced in Creating data-models for Client’s transactional logs, analysing data from Cassandra tables for quick searching, sorting and grouping using Cassandra Query Language(CQL).
• Tested cluster Performance using Cassandra-stress tool to measure and improve the Read/Writes.
• Used Hive QL to analyse the partitioned and bucketed data, Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business specification logic.
• Used Kafka functionalities like distribution, partition, commit log messaging systems by maintaining feeds.
• Used Apache Kafka to aggregate web log data from multiple servers and make them available in Downstream systems for Data analysis and engineering type of roles.
• Experience in using Avro, Parquet, RCFile and JSON file formats, developed UDFs in Hive and Pig.
• Worked with Log4j framework for logging debug, info & error data.
• Performed transformations like event joins, filter bot traffic and some pre-aggregations using PIG.
• Developed Custom Pig UDFs in Java and used UDFs from PiggyBank for sorting and preparing data.
• Developed Custom Loaders and Storage Classes in PIG to work on several data formats like JSON, XML, CSV and generated Bags for processing using pig etc.
• Used Amazon DynamoDB to gather and track the event-based metrics.
• Developed Sqoop and Kafka Jobs to load data from RDBMS, External Systems into HDFS and HIVE.
• Developed Oozie coordinators to schedule Pig and Hive scripts to create Data pipelines.
• Written several MapReduce Jobs using Java API, Jenkins for Continuous integration.
• Setting up and worked on Kerberos authentication principals to establish network communication on cluster and testing of HDFS, Hive, Pig and MapReduce to access cluster for new users.
• Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
• Modified Maven Scripts to build the JAR, Class files, WAR files and EAR files.
• Generated various kinds of reports using Power BI and Tableau based on Client specification.
• Used Jira for bug tracking and Bitbucket to check-in and checkout code changes.
• Worked with different teams to ensure data quality and availability.
• Responsible for generating actionable insights from complex data to drive real business results for various applications teams and worked in Agile Methodology projects extensively.
• Worked with SCRUM team in delivering agreed user stories on time for every Sprint.

Environment: Spark, Spark-Streaming, Spark SQL, AWS EMR, Redshift, DynamoDB, Lambda, HDFS, Hive, Pig, Apache Kafka, Sqoop, Java (JDK SE 6, 7), Scala, Shell scripting, Linux, Jenkins, Eclipse, Git, Oozie, Tableau, Soap, NIFI, Cassandra and Agile Methodology.

Confidential, Arlington, VA

Hadoop Developer

Responsibilities:

• Worked on migrating MapReduce programs into Spark transformations using Spark and Scala, initially done using python.
• Developed Spark jobs using Scala on top of Yarn/MRv2 for interactive and Batch Analysis.
• Experienced in querying data using Spark SQL on top of Spark engine for faster data sets processing.
• Worked on implementing Spark Framework a Java based Web Framework.
• Worked on Ad hoc queries, Indexing, Replication, Load balancing, Aggregation in MongoDB.
• Processed the Web server logs by developing Multi-hop flume agents by using Avro Sink and loaded into MongoDB for further analysis, Extracted files from MongoDB through Flume.
• Expert knowledge on MongoDB NoSQL data modelling, tuning, disaster recovery backup used it for distributed storage and processing using CRUD.
• Extracted and restructured the data into MongoDB using import and export command line utility tool.
• Experience in setting up Fan-out workflow in flume to design v shaped architecture to take data from many sources and ingest into single sink.
• Implemented Custom Sterilizer, interceptors to Mask, created confidential data and filter unwanted records from the event payload in flume.
• Experience in working with different join patterns and implemented both Map and Reduce Side Joins.
• Imported several transactional logs from web servers with Flume to ingest the data into HDFS. Using Flume and Spool directory for loading the data from local system(LFS) to HDFS.
• Installed and configured pig, written Pig Latin scripts to convert the data from text file to Avro format.
• Created Partitioned Hive tables and worked on them using HiveQL.
• Installed, Configured Talend ETL on single and multi-server environments.
• Experience in monitoring Hadoop cluster using Cloudera Manager, interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations.
• Experience in Cloudera Hadoop Upgrades and Patches and Installation of Ecosystem Products through Cloudera manager along with Cloudera Manager Upgrade.
• Worked on continuous Integration tools Jenkins and automated jar files at end of day.
• Worked with Tableau and Integrated Hive, Tableau Desktop reports and published to Tableau Server.
• Developed data pipeline expending Pig and Java MapReduce to consume customer behavioural data and financial antiquities into HDFS for analysis.
• Developed MapReduce programs in Java for parsing the raw data and populating staging Tables.
• Developed Unix shell scripts to load large number of files into HDFS from Linux File System.
• Experience in setting up app stack and debug Logstash to send Apache logs to AWS Elastic search.
• Monitored workload, job performance and capacity planning using Cloudera Manager.
• Used Impala connectivity from the User Interface(UI) and query the results using Impala SQL.
• Written and Implemented Teradata Fastload, Multiload and BTEQ scripts, DML and DDL.
• Used Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Experienced knowledge over designing Restful services using java-based API’s like JERSEY.
• Worked in Agile development environment having KANBAN methodology. Actively involved in daily Scrum and other design related meetings.
• Used OOZIE Operational Services for batch processing and scheduling workflows dynamically.
• Supported in setting up QA environment and updating for implementing scripts with Pig, Hive and Sqoop.

Environment: Hadoop, HDFS, Hive, MapReduce, AWS Ec2, Impala, Sqoop, Spark, SQL Talend, Python, PySpark, Yarn, Pig, Oozie, Linux-Ubuntu, Scala, Ab Initio, Tableau, Maven, Jenkins, Java (JDK 1.6), Cloudera, JUnit, agile methodology.

Confidential, Scottsdale, AZ

Big Data Hadoop Consultant

Responsibilities:

• Experienced in migrating and transforming of large sets of Structured, semi structured and Unstructured RAW data from HBase through Sqoop and placed in HDFS for processing.
• Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV.
• Worked with Solr to implement indexing, wrote Custom Solr query segments to optimize search.
• Written java code to format XML documents, uploaded them to Solr server for indexing.
• Experienced on Apache Solr for indexing and load balanced querying to search for specific data in larger datasets and implemented Near Real Time Solr index on Hbase and HDFS.
• Wrote Flume configuration files for importing streaming log data into HBase with Flume.
• Loading Data into HBase using Bulk Load and Non-bulk load.
• Written Java program to retrieve data from HDFS and providing it to REST Services.
• Implemented business logic by writing UDFs in Java and used various predefined UDFs.
• Sqoop for large data transfers from RDBMS to HDFS/HBase/Hive and vice-versa.
• Implemented partitioning, bucketing in Hive for better organization of the data.
• Involved in using HCatalog to access Hive table metadata from MapReduce or Pig code.
• Created HBase tables, HBase sinks and loaded data into them to perform analytics using Tableau.
• Installed, configured and maintained Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
• Created multiple Hive tables, running hive queries in those data, implemented Partitioning, Dynamic Partitioning and Buckets in Hive for efficient data access.
• Experience in creating tables, dropping and altering at run time without blocking using HBase and Hive.
• Experienced in running batch processes using Pig Latin Scripts and developed Pig UDFs for data manipulation according to Business Requirements
• Hands on experience in Developing optimal strategies for distributing the web log data over the cluster, importing and exporting of stored web log data into HDFS and Hive using Sqoop.
• Developed several REST web services which produces both XML and JSON to perform tasks, leveraged by both web and mobile applications.
• Developed Unit test cases for Hadoop M-R jobs and driver classes with MR Testing library.
• Continuously monitored and managed the Hadoop cluster using Cloudera manager and Web UI.
• Managed and scheduled several jobs to run over a time on Hadoop cluster using Oozie.
• Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box as well as system specific jobs.
• Used MAVEN for building jar files of MapReduce programs and deployed to cluster.
• Involved in final reporting data using Tableau for testing by connecting to the corresponding Hive tables using Hive ODBC connector.
• Performed Cluster tasks like adding, removing of nodes without any effect on running jobs.
• Installed Qlik Sense Desktop and developed applications for users and made reports using QlikView.
• Configured different Qlik Sense roles and attribute-based access control.
• Maintained System integrity of all sub-components (primarily HDFS, MR, HBase, and Hive).
• Helped in design of Scalable Big Data Clusters and solutions and involved in defect meetings.
• Followed Agile Methodology for entire project and supported testing teams.

Environment: Apache Hadoop, MapReduce, HDFS, HBase, CentOS 6.4, Unix, REST web Services, Hive, Pig, Oozie, Java (jdk 1.5), JSON, Eclipse, QlikView, Qlik Sense, Jenkins, Maven, Sqoop.

Confidential

Java Developer

Responsibilities:

• Involved in Requirements Analysis and design an Object-oriented domain model.
• Involvement in the detailed Documentation, written functional specifications of the module.
• Involved in development of Application with Java and J2EE technologies.
• Develop and maintain service-based architecture utilizing technologies like Hibernate, ORM and Spring.
• Designed and documented REST APIs, including JSON data formats and API versioning strategy.
• Developed server-side services using java multithreading, Struts, Spring, EJB, SOAP, WSDL, AXIS.
• Involvement in creating and configuring of build files using Ant.
• Development of Controller Servlet a Framework component for Presentation.
• MVC framework technologies like JSF based ICEfaces, RichFaces to implement the MVC architecture.
• Developed application using JSF, myFaces, and JDO technologies which communicate with Mainframe.
• Designing, Development and Implementation of JSPs in Presentation layer for Submission, Application, reference implementation.
• Developed JSP pages using Jdeveloper, such as HTML, Bean Tags, Logic Tags and Template Tags.
• Practiced and evangelized agile development approaches. Wrote ANT scripts and assisted with build and configuration management processes.
• Development of JavaScript for client end data entry validations and Front-End Validation.
• Deployed Web, presentation and business components on Apache Tomcat Application Server.
• Generating schema difference reports for database using toad.
• Developed PL/SQL procedures for different use case scenarios.
• Built the report module on reports based from Crystal reports.
• Involvement in post-production support, Testing and used JUNIT for unit testing of the module.

Environment: Java/J2EE, JSP, XML, Spring Framework, Hibernate, Eclipse(IDE), Micro Services, Java Script, Struts, Tiles, Ant, PL/SQL, Windows, UNIX, Soap, Jasper reports.

Confidential

Junior Java Developer

Responsibilities:

• Coded and Developed Multi-tier architecture in Java, J2EE, Servlets.
• Conducted analysis, requirements study and design according to various design patterns and developed rendering to the use cases, taking ownership of the features.
• Actively involved from fresh start of the project, requirement gathering to quality assurance testing.
• Used various design patterns such as Command, Abstract Factory, Factory, and Singleton to improve the system performance. Analysing the critical coding defects and developing solutions.
• Developed configurable front end using Struts technology. Also involved in component-based development of certain features which were reusable across modules.
• Designed, developed and maintained the data layer using the ORM framework called Hibernate.
• Used Hibernate framework for Persistence layer, involved in writing Stored Procedures for data retrieval and data storage and updates in Oracle database using Hibernate.
• Developed batch jobs which will run on specified time to implement certain logic in java platform.
• Developing & deploying Archive files (EAR, WAR, JAR) using ANT build tool.
• Used Software development best practices for Object Oriented Design and methodologies throughout Object oriented development cycle.
• Responsible for developing SQL Queries required for the JDBC.
• Designed the database and worked on DB2 and executed DDLS and DMLS.
• Active participation in architecture framework design and coding and test plan development.
• Strictly followed Waterfall development methodologies for implementing projects.
• Thoroughly documented the detailed process flow with UML diagrams and flow charts for distribution across various teams.

Environment: Java JDK (1.5), Java J2EE, Informatica, Oracle11g(TOAD and SQL developer) Servlets, Jboss application Server, Waterfall, JSPs, EJBs, DB2, RAD, XML, Web Server, JUNIT, Hibernate, MS ACCESS.