SUMMARY:

• Over 12 +years of professional IT experience in analysis, architectural design, prototyping, development, Integration and testing of applications using Java/J2EE Technologies and 3+ years of experience in Big Data Analytics as Hadoop Developer with good knowledge in Hadoop ecosystem technologies.
• Designed and Modeled Hive database using Partitioned and Bucketing tables with storing data in various file systems like Parquet, Avro, RC, ORC and Text File
• Delivery experience on major Hadoop ecosystem Components such as Pig, Hive, Spark Kafka, Elastic Search &Hbase and monitoring with Cloudera Manager.
• Extensive working experience using Sqoop to import data into HDFS from RDBMS and vice - versa.
• Experience in doing bash, writing shell scripting.
• Experience spark Framework and Map Reduce for Hadoop to debug code.
• J2EE technologies such as spring, Spring MVC, Spring JDBC, Web services, SOAP, WSDL.
• Having good experience in Bigdata related technologies like Hadoop frameworks, Map Reduce, Hive, HBase, PIG, Sqoop, Spark, Kafka, Flume, ZooKeeper, Oozie, EMR Clusters and Storm.
• Experienced in using Integrated Development environments like Eclipse, Intellij IDEA and Version Control tools like SVN, GIT, TEAMCITY and Build tools like Maven and SBT
• Worked and migrated RDBMS databases into different NoSQL database
• Experience in Cluster maintenance processes which includes both HDFS, tasks like commissioning and decommissioning of datanode, File system check, balancing the cluster and Map Reduce tasks including commissioning and decommissioning a Task Tracker, killing of both Task Tracker and Job Tracker and dealing with blacklisted Task Tracker
• Analyzing the clients existing Hadoop infrastructure and understand the performance bottlenecks and provide the performance tuning accordingly
• Experience working with Cognos Framework Manager, Reporting Studio, Analysis Studio, Query Studio and Cognos Connections
• Performed Infrastructure capacity planning considering various workload patterns and actively participated in hardware design for Edge, Master and DataNodes
• Experience in installing and configuring Hadoop cluster using different distributions of Apache Hadoop like Cloudera distribution and EMR
• Hands on experience in Big data development using AWS EMR, Redshift, RDS, Hive, Presto, Hadoop cluster set up, performance fine-tuning, monitoring and administration
• Good knowledge of Hadoop Architecture and various Hadoop 1.0 and Hadoop 2.0 components such as HDFS,
• MapReduce, NameNode, Secondary NameNode, Standby NameNode, DataNode, Job Tracker, Task Tracker, Resource Manager, Application Master, Node Manager and YARN
• In depth exposure to HDFS concepts, blocks, master nodes, slave nodes, HDFS high-availability including both manual and automatic failover, command line interface, file reading and writing, copying a file from one cluster to another
• Experience in analyzing data using Hive QL, Pig Latin and custom MapReduce programs in Java
• Experience in monitoring infrastructure for Hadoop cluster using Nagios and Ganglia
• Good working experience in writing spark application in Scala using Spark Context, Spark SQL and Spark Streaming
• Experienced as Hadoop, expertise in providing end to end solutions for real time big data problems by implementing distributed processing concepts such as map reduce on Hadoop frameworks such as HDFS and Hadoop Ecosystem components.
• Experience in deploying Hadoop cluster on Public and Private Cloud Environment like Amazon AWS
• Experience in integration of various data sources like Oracle9i/10g, DB2, Teradata, SQL Server, MS Access, XML and Flat files
• Extensively worked on Data Warehouse design methodologies (Top-Down, Bottom-Up)
• Solid experience in Design and Development of Extraction, Transformation, Load and Business Intelligent processes
• Excellent technical, communication, analytical and problem-solving skills and ability to get on well with people including cross-cultural backgrounds.
• Extensively worked in leading ETL design, development, testing and implementation phases. Created logical and physical mapping documents and low level/high level ETL design documents using Info Sphere Information server business suites.
• Experience in working on large scale big data implementations and in production environment.
• Hands on experience on Data Migration from Relational Database to Hadoop Platform\ SQOOP.
• Developed analytical components using Scala, Spark, Spark Stream and Cloudera.
• Experience processing Avro data files using Avro tools and MapReduce programs.
• Implemented pre-defined operators in spark such as map, flat Map, filter, reduceByKey, groupByKey, aggregateByKey and combineByKey etc.

TECHNICAL SKILLS:

Hadoop/BigData Technologies: HDFS, MapReduce, Spark, Hive, Pig, Sqoop, Flume, HBase, Cassandra, Oozie, Zookeeper, YARN, Hue, Ambari, Kafka, Elastic Search, Cloudera, Horton works, Tez, Apache Parquet

Programming Languages: Java JDK1.4/1.5/1.6 (JDK 5/JDK 6), C/C++, Unix Shell Scripting, Python

Web /Applications Servers: Apache Tomcat, Weblogic, WebSphere and Bastion.

Operating Systems: UNIX, Windows, LINUX

Databases: HBase, Oracle 8i/9i/10g, Microsoft SQL Server 2008/2012, DB2 & MySQL 4.x/5.x, Teradata, Mongo DB

Java IDE: Eclipse 3.x, IBM Web Sphere Application Developer, IBM RAD 7.0

Tools: TOAD, SQL Developer, SOAP UI, ANT, Maven

PROFESSIONAL EXPERIENCE:

Confidential, Boston MA

Sr. Hadoop Developer

Responsibilities:

• Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
• Involved in creating Hive tables, loading data and running hive queries in those data.
• Extensive Working knowledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
• Involved in writing optimized Pig Script along with involved in developing and testing Pig Latin Scripts
• Working knowledge in writing Pig’s Load and Store functions.
• Installed Zookeepers, brokers, schema registry, control Center on multiple machine.
• Worked on installing Kafka on Virtual Machine.
• Configuring hosts as edge nodes with the desired file systems and adding them to the cluster.
• Involving in the weekly releases and maintaining the sync between different environments such as Production, Disaster Recovery and Pre-Production
• Developed Java MapReduce programs on log data to transform into structured way to find user location, age group, spending time.
• Developed optimal strategies for distributing the web log data over the cluster, importing and exporting the stored web log data into HDFS and Hive using Scoop.
• Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis
• Monitored multiple Hadoop clusters environments using Ganglia.
• Developed PIG scripts for the analysis of semi structured data.
• Developed and involved in the industry specific UDF (user defined functions).
• Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
• Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most purchased product on website.
• Deleting the users or adding the users as of the request in the Hue, DataRobot and Trifacta.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Played a key role in installation and configuration of the various Hadoop ecosystem tools such as Solr, Kafka, Pig, HBase and Cassandra.
• Implemented various hive optimization techniques like Dynamic Partitions, Buckets, Map Joins, Parallel executions in Hive.
• Involved in scheduling Airflow workflow engine to run multiple Hive and pig jobs using python.
• Used Flume to collect the logs data with error messages across the cluster.
• Extracted meaningful data from dealer csv files, text files, and mainframe files and generated Pythonpanda's reports for data analysis.
• Utilized Python to run scripts, generate tables, and reports.
• Set up Hortonworks Infrastructure from configuring clusters to Node
• Installed Ambari server on the clouds
• Actively updated the upper management with daily updates on the progress of project that include the classification levels that were achieved on the data.

Environment: Hadoop, HDFS, Pig, Hive, MapReduce, Sqoop, Kafka, CDH3, Cassandra, Python, Oozie, Java collection, Scala, AWS cloud, SQL, NoSQL, Bitbucket, Jenkins, HBase, Flume, spark, Solr, Zookeeper, ETL, Centos, Eclipse.

Confidential, Pittsburgh, PA

Sr. Hadoop Developer

Responsibilities:

• Provided use case for Centralize Staging Area on Hadoop cluster
• Setup ACL/SSL security for different users and assign users to multiple topics
• Assign access to users by multiple user's login.
• Played a key-role is setting up a 50 node Hadoop cluster utilizing Apache Spark by working closely with the Hadoop Administration team.
• Created Hive tables to store data into HDFS, loading data and writing hive queries that will run internally in map-reduce way.
• Uploaded and processed terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
• Involved in Cluster coordination services through Zookeeper.
• Provided proof of concept on virtual single node Hadoop instance
• Performed capacity planning for Hadoop cluster for all environments (Dev, Test, Prod)
• Planned the Hadoop cluster selecting hardware and operating system, Configuring disk and designing network
• Participated in installing and configuring Cloudera (CDH4) SuSE package on 40 node cluster
• Configured Rack awareness and Namenode high availability and federation
• Automated the setup of Hadoop Clusters and creation of Nodes
• Responsible for building scalable distributed data solutions using Hadoop.
• Extensively worked on HiveQL to create and modify databases, create external and managed partitioned tables, loading data into tables, Exporting data.
• Created Hive views to reduce query complexity and restrict access to data based on conditions
• Created Hive indexes on columns to speed some operations
• Used Impala to run some ad-hoc queries
• Wrote MapReduce programs in Java to process semi-structured xml and unstructured text files
• Prepared technical specifications and project estimates.
• Designed and created DataStage jobs to Extract, transform and load data from source into data warehouse and data marts.
• Worked on pipeline and partitioning parallelism techniques and ensured load balancing of data.
• Deployed different partitioning methods like Hash by field, Round Robin, Entire, Modulus, and Range for bulk data loading.
• Implemented Slowly Changing Dimension Type- 2 in DataStage.
• Analyze latest Big Data Analytic technologies and their innovative applications in both business intelligence analysis and new service offerings.
• Designed and Developed Shell Scripts and Sqoop Scripts to migrate data in and out of HDFS
• Designed and Developed Oozie workflows to execute MapReduce jobs, Hive scripts, shell scripts and sending email notifications
• Worked on pipeline and partitioning parallelism techniques and ensured load balancing of data
• Deployed different partitioning methods like Hash by field, Round Robin, Entire, Modulus, and Range for bulk data loading
• Implemented Slowly Changing Dimension Type- 2 in Datastage
• Created UNIX shell scripts for File Transfer and File validation during parallel job execution
• Tuned SQL queries for better performance for processing business logic in the database.

Environment: Hadoop, Scala Spark, Spark-SQL, Spark Sqoop, HBase/MapR DB, Apache Drill, Hive, Map Reduce, HDFS, ETL, Sqoop, Maven, Jenkins, Java (JDK 1.7), Java 7, Eclipse, Oracle 10g, PL/SQL, Linux, Teradata.

Confidential, Pleasanton, CA

Sr.Hadoop Developer

Responsibilities:

• Extensively used DataStage Designer stages such as Aggregator, Transformer, Join, Dataset, Lookup, Funnel, Peek, Pivot
• Implemented various shared containers for Re-using the business functionality
• Followed Datastage coding standards and best practices
• Performed Unit Testing in Data Stage and tracked results in unit Test cases (UTC)
• Performed extensive performance tuning in Datastage, to handle big volume of data
• Lead the other team members and guided the testing team members in executing the jobs, which helped them to develop their knowledge on the tool
• Performed code-fix in Datastage for the defects rose from system testing and the Business System Analyst (BSA's)
• Addressed defects on Datastage code added in Quality Center
• Responsible for data analysis, requirements gathering, source-to-target mapping, process flow diagrams and documentation
• Created UNIX shell scripts for File Transfer and File validation during parallel job execution.
• Analyzed the Data Acquisition, Data Integration, Data Transformation, and Data Delivery processes for Performance Assessment.
• Modified existing ETL and BI architecture to achieve maximum parallel processing and reducing overhead on Database servers.
• Lead the implementation of performance improvement recommendations.
• Use quality stages for data.

Environment: Hadoop 1.2.1, Map Reduce, HDFS, Pig, Hive, Java (J2EE), XML, Microsoft (Word & excel), Linux.

Confidential, Charlotte, NC

ETL Developer

Responsibilities:

• Developed Relational model for OLTP system, Demoralized model for Data Warehouse and Dimensional model (Star and Snow Flake) for different data marts using Erwin.
• Identified and documented data sources and transformation rules required to populate and maintain data warehouse content and created source to target Logical Mapping Document.
• Converted Logical Mapping Document into Source to target Physical Mapping Document for ETL.
• Imported Metadata from various Application Sources (Database tables, flat files, XML files) into DataStage Repository.
• Designed and developed DataStage jobs to extract data from heterogeneous sources. Applied business logic to extracted data in transformation and loaded into Data Warehouse and Data mart.
• Worked in Develop processes for cleansing, transforming, extracting, integrating and loading data with use of DataStage.
• Created DataStage jobs using different stages like Transformer, Aggregator, Sort, Join, Merge, Lookup, Data Set, Funnel,
• Remove Duplicates, Copy, Modify, Filter, Change Data Capture, Change Apply, Sample and Surrogate Key.
• Extensively used DataStage stages like Row Generator, Column Generator, Head, and Peek for development and de-bugging purposes.
• Extensively used CDC (Change Data Capture) stage to implement the slowly changing Dimensional and Fact tables.
• Extensively worked with Job sequences using Job Activity, Email Notification, Sequencer, Wait for File activities to control and execute the Data stage Parallel jobs.
• Created multiple configuration files and defined logical nodes, scratch disk, Resource scratch disk and pools.
• Developed complex store procedures using input/output parameters, cursors, views, triggers and complex queries using temp tables and joins.

Environment: Hive, SQL, Pig, Flume, Kafka, Map reduce, Sqoop, Python, Tableau, Java, Shell Scripting, Unix Scripting, Spark, Teradata, Oracle, Oozie, Java, Cassandra.

Confidential, Malvern, PA

DataStage ETL Developer, Data Modeling

Responsibilities:

• Managed servers on the Amazon Web Services (AWS) platform instances using Puppet, Chef Configuration management.
• Developed PIG scripts to transform the raw data into intelligent data as specified by business users.
• Worked in AWS environment for development and deployment of Custom Hadoop Applications.
• Defined the Primary keys and foreign keys for the Entities.
• Defined the query view, index options and relationships.
• Created logical schema using ERWIN 4.0 also created Dimension Modeling for building Cubes.
• Designed staging and Error handling tables keeping in view the overall ETL strategy.
• Assisted in creating the physical database by forward engineering.
• Extracted data from source systems transformed and loaded into Oracle database according to the required provision.
• Primary on-site technical lead for data quality projects using Integrity (now known as QualityStage).
• Created objects like tables, views, Materialized views procedures, packages using Oracle tools like PL/SQL, SQL*Plus, SQL*Loader and Handled Exceptions.
• Involved in database development by creating Oracle PL/SQL Functions, Procedures, Triggers, Packages, Records and Collections.
• Created views for hiding actual tables and to eliminate the complexity of the large queries.
• Created various indexes on tables to improve the performance by eliminating the full table scans.
• Used the DataStage Designer to develop processes for extracting, cleansing, transforming, integrating and loading data into Data Marts.
• Created source table definitions in the DataStage Repository.
• Identified source systems, their connectivity, related tables and fields and ensure data suitability for mapping.
• Generated Surrogate ID’s for the dimensions in the fact table for indexed and faster access of data.
• Used Aggregator stages to sum the key performance indicators in decision support systems and for granularity required in data warehouse.
• Created complicated reports using reporting tool Cognos.

Environment: JBoss, XMLSOAP, RESTful, Java EE 6, IBM WebSphere Application Server 7, Apache-Struts 2.0, EJB 3, Spring 3.2, JSP 2.0, Web Services, JQuery 1.7, Servlet 3.0, Struts-Validator, Struts-Tiles, Tag Libraries, ANT 1.5, JDBC, JMS, Service Bus.

Confidential, West Milford, NJ

Data Stage Developer

Responsibilities:

• Developed a process for Scooping data from multiple sources like SQL Server, Oracle.
• Responsible for creation of mapping document from source fields to destination fields mapping.
• Developed a shell script to create staging, landing tables with the same schema like the source
• Developed Oozie workflow’s for executing Sqoop and Hive actions.
• Performance optimizations on Spark/Scala. Diagnose and resolve performance issues.
• Worked on Parquet File format to get a better storage and performance for publish tables.
• Involved in loading transactional data into HDFS using Flume for Fraud Analytics.
• Developed Python utility to validate HDFS tables with source tables.
• Designed and developed UDF’S to extend the functionality in both PIG and HIVE.
• Import and Export of data using Sqoop between MySQL to HDFS on regular basis.
• Responsible for developing multiple Kafka Producers and Consumers.
• Involved in using CA7 tool to setup dependencies at each level (Table Data, File and Time).
• Automated all the jobs for pulling data from FTP server to load data into Hive tables using Oozie
• Involved in developing Spark code using Scala and Spark-SQL for faster testing.

Environment: Hadoop, HDFS, Map Reduce, Hive, HBase, Kafka, Zookeeper, Oozie, Impala, Java(jdk1.6), Cloudera, Oracle, Teradata SQL Server, UNIX Shell Scripting, Flume, Scala, Spark, Sqoop, Python.

Confidential, Richmond, VA

Datastage Developer

Responsibilities:

• Involved in almost all the phases of SDLC.
• Executed test cases manually and logged defects using Clear Quest
• Automated the functionality and interface testing of the application using Quick Test Professional (QTP)
• Design, Develop and maintain automation framework (Hybrid Framework).
• Analyze the requirements and prepare automation scripts scenario
• Develop test data for Regression testing using QTP
• Wrote Test cases on IBM rational Manual Tester
• Conducted Cross Browser testing on Different Platform
• Client Application Testing, Web based Application Performance, Stress, Volume and Load testing of the system using Load Runner 9.5.
• Analyzed performance of the application program itself under various test loads of many simultaneous Vusers.
• Analyzed the impact on server performance CPU usage, server memory usage for the applications of varied numbers of multiple, simultaneous users.
• Inserted Transactions and Rendezvous points into Web Vusers
• Created Vuser Scripts using VuGen and used Controller to generate and executed Load Runner Scenarios
• Complete involvement in Requirement Analysis and documentation on Requirement Specification.
• Prepared use-case diagrams, class diagrams and sequence diagrams as part of requirement specification documentation.

Environment: IBM WebSphere Datastage 8.1, Oracle 9i, DB2, SQL Server 2005, Flat Files, UNIX scripting, Windows 2003, and Autosys, Quality stage, Profile stage.