SUMMARY

• Experienced Hadoop developer wif over 9 years of experience in programming and hands on experience of 4 years in Big Data environment.
• In depth experience and good noledge in using Hadoop ecosystem tools like MapReduce, HDFS, Pig, Hive, Kafka, Yarn, Sqoop, Storm, Spark, Oozie, and Zookeeper.
• Excellent understanding and extensive noledge of Hadoop architecture and various ecosystem components such as HDFS, JobTracker, TaskTracker, NameNode, DataNode and MapReduce programming paradigm.
• Good usage of Apache Hadoop along enterprise version of Cloudera and Hortonworks. Good Knowledge on MAPR distribution & Amazon's EMR.
• Good noledge of Data modeling, use case design and Object - oriented concepts.
• Well versed in installation, configuration, supporting and managing of Big Data and underlying infrastructure of Hadoop Cluster.
• Good noledge on spark components like Spark SQL, MLib, Spark Streaming and GraphX,
• Extensively worked on Spark Streaming and Apache Kafka to fetch live stream data.
• Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
• Implemented Dynamic Partitions and Buckets in HIVE for efficient data access.
• Experience in data processing like collecting, aggregating, moving from various sources using Apache Flume and Kafka.
• Involved in integrating hive queries into spark environment using SparkSql.
• Hands on experience in performing real time analytics on big data using HBase and Cassandra in Kubernetes & Hadoop clusters.
• Experience in using Flume to stream data into HDFS.
• Good working experience using Sqoop to import data into HDFS from RDBMS and vice-versa.
• Good noledge in developing data pipeline using Flume, Sqoop, and Pig to extract teh data from weblogs and store in HDFS.
• Created User Defined Functions (UDFs), User Defined Aggregated Functions (UDAFs) in PIG and Hive.
• Good noledge in using job scheduling and monitoring tools like Oozie and ZooKeeper.
• Hands on experience working on NoSQL databases including Hbase, Cassandra, MongoDB and its integration wif Hadoop cluster & Kubernetes cluster.
• Proficient wif Cluster management and configuring Cassandra Database.
• Extensive experience in developing Pig Latin Scripts and using Hive Query Language for data analytics.
• Good working experience on different file formats (PARQUET, TEXTFILE, AVRO, ORC) and different compression codecs (GZIP, SNAPPY, LZO).
• Valuable experience on practical implementation of cloud-specific AWS technologies including IAM, Amazon Cloud Services like Elastic Compute Cloud (EC2), ElastiCache, Simple Storage Services (S3), Cloud Formation, Virtual Private Cloud (VPC), Route 53, Lambda, EBS.
• Build AWS secured solutions by creating VPC wif private and public subnets.
• Expertise in configuring Relational Database Service.
• Worked extensively in configuring Auto scaling for high Availability.
• Knowledge of data warehousing and ETL tools like Informatica, Talend and Pentaho.
• Experience working wif JAVA J2EE, JDBC, ODBC, JSP, Java Eclipse, Java Beans, EJB, Servlets.
• Expert in developing web page interfaces using JSP, Java Swings, and HTML scripting languages.
• Experience working wif spring and Hibernates frameworks for JAVA.
• Experience in using IDEs like Eclipse, NetBeans andIntellij.
• Proficient using version control tools like GIT, VSS, SVN and PVCS.
• Experience wif web-based UI development using jquery UI, jquery, CSS, HTML, HTML5, XHTML and JavaScript.
• Development experience in DBMS like Oracle, MS SQL Server, Teradata and MYSQL.
• Developed stored procedures and queries using PL/SQL.
• Hands on Experience wif best practices of Web services development and Integration (both REST and SOAP).
• Experience in working wif build tools like Ant, Maven, SBT, and Gradle to build and deploy applications into server.
• Expertise in Object Oriented Analysis and Design (OOAD) and noledge in Unified Modeling Language (UML).
• Expertise in complete Software Development Life Cycle (SDLC) in Waterfall and Agile, Scrum models.
• Excellent communication skills, interpersonal skills, problem solving skills and very good team player along wif a can do attitude and ability to TEMPeffectively communicate wif all levels of teh organization such as technical, management and customers.

PROFESSIONAL EXPERIENCE

Spark/Big Data Developer

Confidential - Raleigh, NC

Responsibilities:

• Experience wif Hortonworks distribution.
• Experienced in loading data from different relational databases to HDFS using Sqoop.
• Created Hive internal/external tables wif proper static and dynamic partitions and working on them using HQL.
• Deployed data from various sources into HDFS and building reports using Tableau.
• Written several Map reduce Jobs using Java API, also used Jenkins for Continuous integration.
• Collected teh logs data from Web Servers and integrated in to HDFS using Flume.
• Used teh Spark - Cassandra Connector to load data to and from Cassandra.
• Building, managing and scheduling Oozie workflows for end to end job processing.
• Experienced in extending Hive and Pig core functionality by writing custom UDFs using Java.
• Analyzing of large volumes of structured data using SparkSQL.
• Migrated HiveQL queries on structured into SparkSQL to improve performance.
• Created Hive tables, worked on loading data into hive tables and Experienced in loading and transforming heavy sets of structured, semi structured and unstructured data.
• Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load teh data into Cassandra.
• Used Spark streaming api for performing transformations and actions on teh fly for building common learner data model which gets data from Kafka in real time.
• Has developed custom components and multi-threaded configurations wif a flat file by writing JAVA code in Talend.
• Written Spark Streaming jobs in Scala Programming language.
• Extensive experience in creating Lambda functions manually using Java and Python.
• Implemented S3 by creating a bucket and passing teh event information to Lambda.
• Experienced in using Automation Scheduling tools like Autosys and Control-M.
• Knowledge on using spring and Rest services and connecting it to teh Kubernetes cluster.
• Extensively used Microservices and Postman for hitting teh Kubernetes DEV and Hadoop clusters.
• Deployed various Microservices like Spark, MongoDB, Cassandra in Kubernetes and Hadoop clusters using Docker.
• Launched Kinesis Streams services in new AWS regions and granting access to new users through IAM.
• Setting up and worked on Kerberos autantication TEMPprincipals to establish secure network communication on cluster and testing of HDFS, Hive, Pig and MapReduce to access cluster for new users.
• Used Amazon Kinesis as a messaging system for getting all teh weblogs, data from rest end points into teh Amazon Redshift.
• Used Hortonworks AMBARI for job browser, file browser, running hive and impala queries.
• Configured Spark streaming to receive real time data from teh Kafka and store teh stream data to HDFS and store in databases such as HBase.
• Worked on RapidMiner, a Data scientist tool and created several Hadoop operators like Hive Operator, Spark Operator, Mongo Operator.
• Designed ETL process using Talend Tool to load from Sources to Targets through data Transformations.
• Experienced in ETL methodology for performing Data Migration, Data Profiling, Extraction, Transformation and Loading using Talend and designed data conversions from large variety of source systems including Oracle DB2, SQL server, Teradata, Hive and non-relational sources like flat files, XML files.
• Implemented Security Side car and AAF as security autantications for Hadoop cluster.
• Used Python for pattern matching in build logs to format errors and warnings.
• Involved in creating UI using Node.js and called different microservices to setup teh frontend.
• Worked on CodeCloud, a git repo for continuous code checkins.
• Responsible for developing, support and maintenance for teh ETL (Extract, Transform and Load) processes using Talend.
• Developed teh Talend jobs and make sure to load teh data into HIVE tables & HDFS files and develop teh Talend jobs to integrate wif Teradata system from HIVE tables.
• Assisted wif testing and monitoring Control- M upgrade and active schedule functions, ensuring TEMPeffective issue resolution and continuous operations.
• Extensive experience in using Microservices, Kubernetes (prod, test, dev) environments and Docker.
• Good experience in developing several seed templates like Scala, MongoDB, Zeppelin and Spark.
• Experienced in configuring teh yaml files for Spark, MongoDB, and Cassandra and deployed in Docker for connecting to teh several Microservices.

Environment: Hadoop 2.7.0, YARN, HDFS, Spark 2.1, Sqoop 1.99.7, Hive 2.1.1, Flume 1.7.0, Oozie 4.2.0 HDP 2.5, Pig 0.16.0, Kafka 0.9.0, Hbase 1.1.2, Zookeeper 3.4.8, Talend Studio, Jenkins 2.0 MySQL 5.6.33, Java 8, Superputty, Scala IDE, Bit Bucket

Spark/Big Data Developer

Confidential - Bellevue, WA

Responsibilities:

• Evaluated business requirements and prepared detailed specifications that follow project guidelines required to develop written programs.
• Involved in teh process of data acquisition, data pre-processing and data exploration of telecommunication project in scala.
• Worked on Cloudera distribution and deployed on AWS EC2 Instances.
• Hands on experience on Cloudera Hue to import data on to teh graphical User Interface.
• Responsible for building scalable distributed data solutions using Hadoop.
• Involved in teh loading of structured and unstructured data into HDFS.
• Imported metadata from Relational Databases like Oracle, Mysql using Sqoop.
• Implemented Web Interfacing wif Hive and stored teh data in Hive tables.
• Loaded data from MySQL, a relational database to HDFS on regular basis using SqoopImport/Export.
• Responsible for implementing Map Reduce programs into Spark transformations using Spark and Scala.
• Developed Kafka consumer's API in Scala for consuming data from Kafka topics.
• Used Spark Streaming APIs to perform transformations and actions on teh fly for building common learner data model which gets teh data from Kafka in near real time and persist it to Cassandra.
• Worked on loading CSV/TXT/AVRO/PARQUET files using Scala/Java language in Spark Framework and process teh data by creating Spark Data frame and RDD and save teh file in parquet format in HDFS to load into fact table using ORC Reader.
• Developed and Scheduled Jobs in Talend Integration Suite.
• Involved in loading data to Kafka Producers from rest endpoints and transferring teh data to Kafka Brokers.
• Experienced in writing live Real-time Processing and core jobs using Spark Streaming wif Kafka as a data pipe-line system using Scala programming.
• Developed Amazon Kinesis streams for accessing teh real-time data from webservers.
• Loaded teh data into Spark RDD and do in memory data Computation to generate teh Output response.
• Ingested data in mini-batches and performed RDD transformations on those mini-batches of data.
• Good noledge in setting up batch intervals, split intervals and window intervals in Spark Streaming using Scala Programming language.
• Imported real time weblogs using Kafka as a messaging system and ingested teh data to Spark Streaming.
• Implemented data quality checks using Spark Streaming and arranged bad and passable flags on teh data.
• Implemented Spark-SQL wif various data sources like JSON, Parquet, ORC and Hive.
• Collected XML and JSON data from different Sources and developed Spark APIs that halps to do inserts and updates in Hive tables and made data available in Hive and Impala as per business requirement.
• Created separate branches wif in teh Talend repository for Development, Production and Deployment.
• Expertise in using Flume in Collecting, aggregating and loading log data from multiple sources into HDFS.
• Involved in Data Querying and Summarization using Pig and Hive and created UDF's, UDAF's and UDTF's.
• Worked on different file formats, Map side joins, bucketing, partitioning for hive performance enhancement and storage improvement.
• Implemented Sqoop jobs for large data exchanges between RDBMS and HBase/Hive/Cassandra clusters.
• Experienced in using Spark Core for joining teh data do deliver teh reports and for detecting teh fraudulent activities.
• Involved in a Huge Data Migration from 80+ MySQL Tables to JSON format using Talend.
• Extensively used Zookeeper as a backup server and job scheduled for Spark Jobs.
• Implemented Hive Partitioning and Bucketing on teh collected data in HDFS.
• Experienced wif Spark Context, Spark-SQL, Data Frame, Datasets, Spark YARN.
• Knowledge on MLLib(Machine Learning Library) framework for auto suggestions.
• Created executors for every created partition in Kafka Direct Stream as a receiverless approach.
• Developed business logic using Kafka Direct Stream in Spark Streaming and implemented business transformations.
• Implemented a distributed messaging queue to integrate wif Cassandra using Apache Kafka and Zookeeper.
• Involved in loading teh real-time data to NoSQL database like Cassandra.
• Experienced in using Data StaxSpark Connector which is used to store teh data in Cassandra database from Spark.
• Involved in NoSQL (Datastax Cassandra) database design, integration and implementation and written scripts and invoked them using CQLSH.
• Good noledge in using Data Manipulations, tombstones, Compactions in Cassandra.
• Designed Columnar families in Cassandra and Ingested data from RDBMS, performed data transformations, and tan exported teh transformed data to Cassandra as per teh business requirement.
• Experience in working on CQL (Cassandra Query Language), for retrieving teh data present in Cassandra cluster by running queries in CQL.
• Involved in maintaining teh Big Data servers using Gangila and Nagios.
• Worked on connecting Cassandra database to teh Amazon EMR File System for storing teh database in S3.
• Deployed teh project on Amazon EMR wif S3 connectivity for setting a backup storage.
• Implemented usage of Amazon EMR for processing Big Data across a Hadoop cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3).
• Worked on Production Server's on Amazon Cloud (EC2, EBS, S3, Lambda and Route53).
• Loaded teh data into Simple Storage Service (S3) in teh AWS Cloud.
• Worked wif Parallel connectors for Parallel Processing to improve job performance while working wif bulk data sources in Talend.
• Good noledge in using of Elastic Load Balancer for Autoscalingin EC2 servers.
• Defined Security groups for Amazon EC2 servers in Virtual Private Cloud (VPC).
• Experienced in configuring work flows that involves Hadoop actions using Oozie client.
• Experienced in using ETL tools like Informatica and Talendand involved in transferring teh workflows from Informatica to Talend.
• Coordinated wif admins and Sr. Technical staff for migrating Terradata to Hadoop and AbInitio to Hadoop as well.
• Worked on Cluster size of nodes.
• Experienced wif Full Text Search and Faceted Reader search usingSolr.
• Wrote Java code to format XML documents; upload them to Solr server for indexing.
• Experienced wif reporting tools like Tableau to generate teh reports.
• Developed Power enter mappings to extract data from various databases, Flat files and load into DataMart using teh Informatica.
• Worked wif SCRUM team in delivering agreed user stories on time for every sprint.

Environment: Hadoop YARN, Spark-Core, Spark-Streaming, AWS S3, AWS EMR, Spark-SQL, GraphX, Scala, Python, Kafka, Zeppelin, Jenkins, Docker, Microservices, Hive, Pig, Sqoop, Solr, Impala, Cassandra, Informatica, Cloudera, Oracle 10g, Linux.

Hadoop Developer

Confidential - San Jose, CA

Responsibilities:

• Experience on working wif Hortonworks distribution.
• Involved in review of functional and non-functional requirements (NFR's).
• Responsible to manage data coming from different sources.
• Responsible for coding MapReduce program, Hive queries, testing and debugging teh Map Reduce programs.
• Worked on migrating MapReduce programs into Spark transformations using Spark and Scala, initially done using python (PySpark).
• Designed, developed and maintained data integration programs in Hadoop and RDBMS environment wif both RDBMS and NoSQL data stores for data access and analysis.
• Implemented Map Reduce programs to handle semi/unstructured data like xml, Json, Avro data files and sequence files for log files.
• Developed Simple to complex Map/reduce Jobs using Hive, Pig and Python.
• Used Spark-SQL to Load data into Hive tables and written queries to fetch data from these tables.
• Experienced in querying data using SparkSQLon top of Spark engine for faster data sets processing.
• Imported semi-structured data from Avro files using Pig to make serialization faster.
• Managing and scheduling Jobs on a Hadoop cluster using Ooziework flows and java schedulers.
• Indexed documents using Elastic Search.
• Developed various Python scripts to find vulnerabilities wif SQL Queries by doing SQL injection, permission checks and performance analysis.
• Experienced wif Kerberos autantication to establish a more secure network communication on teh cluster.
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.
• Extraction, transformation and loading of data from various file formats like .csv, .xls, .txt and various delimited formats using Talend Open Studio.
• Experienced in writing Spark Applications in Scala and Python (Pyspark).
• Imported data from AWS S3 and into spark RDD and performed transformations and actions on RDD's.
• Processed teh web server logs by developing Multi-hop flume agents by using Avro Sink and loaded into MongoDB for further analysis.
• Experienced in converting Hive/SQL queries into Spark transformations using Spark RDD, Scala and Python.
• Imported weblogs & unstructured data using teh Apache Flume and stores teh data in Flume channel.
• Exported event weblogs to HDFS by creating a HDFS sink which directly deposits teh weblogs in HDFS
• Experienced in connecting Avro Sink ports directly to Spark Streaming for analyzation of weblogs.
• Implementing data quality checks and transformations using Flume Interceptor.
• Implemented teh business logic in Flume Interceptor in Java.
• Worked on MongoDB for distributed storage and processing.
• Responsible for using Flume sink to remove teh data from Flume channel and to deposit in No-SQL database like MongoDB.
• Implemented collections & Aggregation Frameworks in MongoDB.
• Implemented B Tree Indexing on teh data files which are stored in MongoDB.
• Implemented Flume NG MongoDB sink to load teh JSON - styled data into MongoDB.
• Good noledge in using MongoDB CRUD (Create, Read, Update and Delete) operations.
• Used codec's like snappy and LZO to store data into HDFS to improve performance.
• Designed and implemented map reduce jobs to support distributed processing using java, Hive and Apache Pig.
• Involved in maintaining Hadoop clusters using teh Nagios server.
• Developed Pig scripts and UDF's as per teh Business logic.
• Installed Oozie workflow engine to automate Map/Reduce jobs.
• Executed Hive queries on Parquet tables stored in Hive to perform data analysis s to meet teh business requirements.
• Experienced in working wif Network, database, application and BI teams to ensure data quality and availability.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Good experience in handling data manipulation using python Scripts.
• Created Pig Latin scripts to sort, group, join and filter teh enterprise wise data.
• Loaded JSON-Styled documents in NoSQL database like MongoDB and deployed teh data in cloud storage service, Amazon Redshift.
• Responsible for developing data pipeline wif Amazon AWS to extract teh data from weblogs and store in Amazon EMR.
• Worked on customizing Map Reduce code in Amazon EMR using Hive, Pig, Impala frameworks.
• Implemented test scripts to support test driven development and continuous integration.
• Used Reporting tools like Tableau to connect wif Hive for generating daily reports of data.
• Transformed teh reports from Informatica toTalend.
• Configured Hadoop clusters and coordinated wif BigData Admins for cluster maintenance.
• Experienced in using agile approaches, including Extreme Programming, Test-Driven Development and Agile Scrum.
• Worked in Agile development environment having KANBAN methodology. Actively involved in daily Scrum and other design related meetings.

Environment: Hortonworks HDP, Hadoop, Spark, Flume, Elastic Search, AWS, EC2, S3, Pig, Hive, Python, MapReduce, HDFS, Tableau, Informatica, VPC.

Hadoop Developer

Confidential - Springfield, IL

Responsibilities:

• Analyzed Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
• Installed Hadoop, MapReduce, HDFS, and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
• Importing and exporting data into HDFS and Hive using Sqoop.
• Imported teh web logs using Flume.
• Used Hive to analyze teh partitioned and bucketed data to compute various metrics for reporting.
• Coordinated wif business customers to gather business requirements, also interacted wif other technical peers to derive technical requirements and delivered teh BRD and TDD documents.
• Written Hive jobs to parse teh logs and structure them in tabular format to facilitate TEMPeffective querying on teh log data.
• Involved in creating Hive tables loading data and writing queries that will run internally in MapReduce way.
• Created Hive tables, loaded teh data and Performed data manipulations using Hive queries in MapReduce Execution Mode.
• Involved in processing ingested raw data using MapReduce, Apache Pig and HBase.
• Worked in developing Pig Scripts for data capture change and delta record processing between newly arrived data and already existing data in HDFS.
• Involved in scheduling Oozie workflow engine to run multiple Hive and Pig jobs.
• Involved in loading teh created HFiles into HBase for faster access of large customer base wifout taking Performance hit.
• Created HBase tables to store various data formats of PII data coming from different portfolios.
• Extensively involved in Design phase and delivered Design documents.
• Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
• Implemented test scripts to support test driven development and continuous integration.
• Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing teh data on to HDFS.
• Developed Hadoop streaming Map/Reduce works using Python.
• Used Reporting tools like Talend to connect wif Hive for generating daily reports of data.
• Set up Solr for distributing indexing and search.

Environment: CDH 3.x and 4.x, Hadoop, Hive, MapReduce, Pig, Oozie, Sqoop, Cloudera, HDFS, Solr, Zookeeper, HBase.

Java Developer

Confidential

Responsibilities:

• Responsible for teh support and maintenance of teh application.
• Developed, enhanced and fixed bugs in user interfaces using JSFs, JSTL, HTML, CSS.
• Experienced in creating various UI component using HTML, CSS, JavaScript, JSP, Servlets and Struts.
• Used Springs framework to implement teh MVC model and Hibernate to connect to teh Database.
• Implemented different modules of Spring Framework such as IOC, AOP creating transactions.
• Extensively developed Servlets and JDBC calls for accessing data from database.
• Used Ajax and JavaScript to perform client-side validations.
• Used RESTful web services wif MVC for parsing and processing XML data.
• Utilized XML and XSL Transformation for dynamic web-content and database connectivity.
• Involved in teh implementation of teh Design patterns such as Singleton and MVC.
• Experience wif SOAP Web Services and WSDL.
• Used ANT for building, creating and deploying teh war files and SVN for version control.
• Used Test First Development for development of teh project.
• Used Spring ORM module for integration wif Hibernate for persistence layer.
• Involved in writing stored procedures, triggers and creating table in Oracle database.
• Performed code refactoring for readability, simplify code structure and improve maintainability.
• Assisted QA team in Test cases preparation, execution and fixing of bugs.

Environment: J2SE 1.5, Servlets, WebLogic, Spring, Hibernate, JDBC, Oracle 9i, SOAP, WSDL, REST, XML, XSLT, Eclipse, HTML, CSS, JavaScript, JSF, ANT, SVN, Log4J, JUnit

Java Developer

Confidential

Responsibilities:

• Involved in all teh phases of teh life cycle of teh project from requirements gathering to quality assurance testing.
• Developed Class diagrams, Sequence diagrams using Rational Rose.
• Responsible in developing Rich Web Interface modules wif Struts tags, JSP, JSTL, CSS, JavaScript, Ajax, GWT.
• Developed presentation layer using Struts framework, and performed validations using Struts Validator plugin.
• Created SQL script for teh Oracle database.
• Implemented teh Business logic using Java Spring Transaction Spring AOP.
• Implemented persistence layer using Spring JDBC to store and update data in database.
• Produced web service using WSDL/SOAP standard.
• Implemented J2EE design patterns like Singleton Pattern wif Factory Pattern.
• Extensively involved in teh creation of teh Session Beans and MDB, using EJB 3.0.
• Used Hibernate framework for Persistence layer.
• Extensively involved in writing Stored Procedures for data retrieval and data storage and updates in Oracle database using Hibernate.
• Deployed and built teh application using Maven.
• Performed testing using JUnit.
• Used JIRA to track bugs.
• Extensively used Log4j for logging throughout teh application.
• Produced a Web service using REST wif Jersey implementation for providing customer information.
• Used SVN for source code versioning and code repository.

Environment: Java (JDK1.5), J2EE, Eclipse, JSP, JavaScript, JSTL, Ajax, GWT, Log4j, CSS, XML, Spring, EJB, MDB, Hibernate, Web Logic, REST, Rational Rose, Junit, Maven, JIRA,SVN.