SUMMARY

• Over 7 years of professional IT experience that includes 3+ years of Hadoop related experience developing Big Data/Hadoop applications.
• Hands on experience with the Hadoop stack (MapReduce, HDFS, Sqoop, Pig, Hive, HBase, Flume, Oozie and ZooKeeper).
• Hands on experience in installation, configuration, management and deployment of Big Data solutions and the underlying infrastructure of Hadoop Cluster using Cloudera, Hortonworks distributions and MapR.
• Proven Expertise in performing analytics on Big Data using Map Reduce, Hive and Pig.
• Experienced with performing real time analytics on NoSQL databases like HBase, MongoDB and Cassandra.
• Experienced with ETL to load data into Hadoop/NoSQL.
• Experienced with Dimensional modeling, Data migration, Data Masking, Data cleansing, Data profiling, and ETL Processes features for data warehouses.
• Experience in creating pre - session and post-session scripts to ensure timely, accurate processing and ensuring balancing of job runs.
• Experience in integration of various data sources like SQL Server, Oracle, Teradata, Netezza, DB2, PostGreSQL, Flat files.
• Strong experience in complex PL/SQLpackages, functions, cursors, triggers, views, materialized views, T-SQL.
• Extensive knowledge of Teradata utilities(BTEQ, FastLoad, FastExport, MultiLoad Update/Insert/Delete/Upsert)
• Knowledge in Installation and configuration of Informatica server with sql server, oracle and able to handle the Informatica administrator tasks like Configuring DSN, creating Connection Strings, copying & moving mappings, workflows, creating folders etc.
• Experience working on Talend ETL performing data migration and data synchronization processes on the data warehouse.
• Worked with Oozie workflow engine to schedule time based jobs to perform multiple actions.
• Experienced in importing and exportingdatabetween RDBMS andTeradatainto HDFS using Sqoop.
• Experience in designing and scheduling custom Talend jobs to acquire, ingest, enrich and distribute the data in MapR Hadoop Ecosystem.
• Analyzed large amounts of data writing Pig scripts and Hive queries.
• Logical implementation and interaction with HBase.
• Experienced in writing MapReduce programs &UDFs for both Hive&Pig in Java.
• Used Flume, Kafka to channel data from different sources to HDFS.
• Very good understanding of Hadoop ecosystems like Sqoop2, Spark and YARN.
• Experience with configuration of Hadoop Ecosystem components: Hive, HBase, Pig, Sqoop, Mahout, Zookeeper and Flume.
• Experience using Talend to design, deploy and schedule jobs based on CRON and file-based triggers.
• Good experience in Hive partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive SerDe like JSON and Avro.
• Supported MapReduce Programs running on the cluster and wrote custom MapReduce Scripts for Data Processing in Java.
• Experience in Amazon AWS cloud services (EC2, EBS, S3).
• Experience in working with various versions of Informatica PowerCenter 9.6.1/9.1.0/9.0.1/8.6/8.5/8.1 - Client and Server tools.
• Expertise in tuning the performance of mappings and sessions in Informatica and determining the performance bottlenecks.
• Experience in creating pre-session and post-session scripts to ensure timely, accurate processing and ensuring balancing of job runs.
• Knowledge in Installation and configuration of Informatica server with sql server, oracle and able to handle the Informatica administrator tasks like Configuring DSN, creating Connection Strings, copying & moving mappings, workflows, creating folders etc.
• Involved in implementing Web based, Enterprise level applications using J2EE frameworks like spring, JMS, JSF and Java.
• Experienced with continuous integrations like Jenkins.
• Hands on experience in VPN, Putty, WinSCP, VNCviewer, etc.
• Experienced in all facets of Software Development Life Cycle (Analysis, Design, Development, Testing and maintenance) using Waterfall and Agile methodologies.
• Experience with various version control systems such as Clear Case, CVS, SVN.
• Motivated team player with excellent communication, interpersonal, analytical, problem solving skills and zeal to learn new technologies.
• Highly adept at promptly and thoroughly mastering new technologies with a keen awareness of new industry developments and the evolution of next generation programming solutions.

TECHNICAL SKILLS

Hadoop Ecosystem: HDFS, MapReduce, Hive, Pig, HBase, Sqoop, Flume, Oozie, ZooKeeper, Kafka, Talend

Data Warehousing: Informatica PowerCenter 9.6.1/9.0.1/8.1/7.1, ILM TDM 9.0.3, Informatica Cloud, Data cleansing, OLAP, OLTP, Star & Snowflake Schema, FACT & Dimension Tables, Physical & Logical Data Modeling, Erwin 9.0

Databases: Oracle 11g/10g/9i/8i/8/7.3, Teradata 15/13, SQL Server 2008/2005,Netezza, PostGreSQL 8.4, My SQL, MS-Access, DB2 8.0/7.0

Languages: C, C++, Java, SQL, PL/SQL, PIG Latin, HiveQL, Unix shell scripting

No SQL Databases: HBase, Cassandra and MongoDB

DB Tools: SQL*Plus, SQL*Loader, BTEQ, FastLoad, FastExport, Export/Import, TOAD, Aginity Workbench, Teradata SQL Assistant, SQL Navigator, SQL Trace

Methodologies: Scrum, Agile, Waterfall

PROFESSIONAL EXPERIENCE

Confidential - Minnetonka, MN

Big Data Developer

Responsibilities:

• Responsible to manage data coming from different source systems and involved in maintaining the structured data within the HDFS.
• Experience writing Sqoop jobs to move data from various RDBMS into HDFS and vice versa.
• Analyzed and transformed stored data by writing MapReduce jobs (using Pig and Hive) based on business requirements.
• Designed and scheduled custom Talend jobs to acquire, ingest, enrich and distribute the data in MapR Hadoop Ecosystem.
• Experience creating Pig UDF’s to perform enrichment transformationsand applying Data Quality checks over the data.
• Developed several MapReduce programs to parse the raw data, populate into HDFS, and store the data in partitioned tables in Hive.
• Created several UNIX shell scripts to perform file level operations and wrap around Talend jobs.
• Experience using Talend to design, deploy and schedule jobs based on CRON and file-based triggers.
• Processed multiple data sources using various input formats.
• Visualize the HDFS data to customer using clienttools with the help of Hive ODBC Driver.
• Worked on designing HBase tables to store metadata about various data sources coming from a variety of portfolios.
• Worked on custom Pig Loaders and storage classes to work with variety of data formats.
• Developed Pig scripts to perform Joins operations across various tables and perform complex enrichments.
• Created Hive tables on top of HBase to visually analyze the data.
• Experience in designing and inserting data into HBase tables with versioning. Perform scan, get operations to populate data into Talend jobs and display them to the end user.
• Experience working with Talend on designing jobs for the Big Data platform using components, context variables, joblets etc.
• Worked on developing re-usable joblets in talend for error handling and logging purposes.

Environment: Hadoop, HDFS, HBase, MapReduce, MapR, Java, Hive, Pig, Sqoop, Talend, Splunk, SQL, SVN, Jenkins.

Confidential - El Segundo, CA

Sr Hadoop Developer

Responsibilities:

• Involved in Installing, Configuring Hadoop Eco System, Cloudera Manager using CDH4 Distribution.
• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data.
• Imported data into HDFS from Teradatadatabase using Sqoop and Teradata connector.
• Integrated Quartz scheduler withOozie work flows to get data from multiple data sources parallel using fork.
• Processed Multiple Data sources input to same Reducer using Generic Writable and Multi Input format.
• Created Data Pipeline of Map Reduce programs using Chained Mappers.
• Exported the patterns analyzed back to Teradata using Sqoop.
• Visualize the HDFS data to customer using BI tool with the help of Hive ODBC Driver.
• Worked with Kafka streaming tool to load the data into HDFS and exported it into MongoDB database.
• Implemented Optimized Join base by joining different data sets to get top claims based on state using MapReduce.
• Implemented complex MapReduce programs to perform joins on the Map side using Distributed Cache in Java.
• Responsible for importing log files from various sources into HDFS using Flume.
• Created customized BI tool for manager team that perform Query analytics using HiveQL.
• Used Hive and Pig to generate BI reports.
• Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
• Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
• Created Hive Generic UDF's to process business logic that varies based on policy.
• Moved Relational Database data using Sqoop into Hive Dynamic partition tables using staging tables.
• Optimizing the Hive queries using Partitioning and Bucketing techniques, for controlling the data.
• Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSON and XML file formats.
• Experienced with different kind of compression techniques like LZO, GZip, Snappy.
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce Hive, Pig, and Sqoop.
• Developed Unit test cases using Junit, Easy Mock and MRUnit testing frameworks.
• Experienced in Monitoring Cluster using Cloudera Manager.

Environment: Hadoop, HDFS, HBase, MongoDb, Spark, MapReduce, Kafka, Teradata, Java, Hive, Pig, Sqoop, Flume, Oozie, Hue, SQL, ETL, Cloudera Manager, MySQL.

Confidential - Glendale, AZ

Sr Hadoop Developer

Responsibilities:

• Involved in complete implementation lifecycle, specialized in writing custom MapReduce, Pig and Hive programs.
• Handled importing of data from RDBMS into HDFS using Sqoop.
• Managing data flow into Pivotal HAWQ (Internal / External tables).
• Experienced in data cleansing processing using Pig Latin operations and UDFs.
• Experienced in writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL).
• Involved in creating Hive tables, loading with data and writing hive queries to process the data.
• Created scripts to automate the process of Data Ingestion.
• Developed PIG scripts for source data validation and transformation.
• Performed various performance optimizations like using distributed cache for small datasets, Partitioning, Bucketing in Hive and Map Side Joins.
• Managing and scheduling jobs to remove the duplicate log data files in HDFS using Oozie.
• Extensively used Hive/HQL or Hive queries to query or search for particular string in Hive tables in HDFS.
• Experience in developing customized UDF’s in java to extend Hive and Pig Latin functionality.
• Created HBase tables to store various data formats for data coming from different portfolios.
• Used File System Check (FSCK) to check the health of files in HDFS.
• Developed the UNIX shells scripts for creating the reports from Hive data.
• Optimized BigData performance in the cloud using Talend.
• Experience on loading and transforming of large sets of structured, semi structured and unstructured data.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Worked on Talend to migrate the data and perform extract, transform and load operations.
• All the tables were staged according to the EDW ETL standards. Made use of mapping variables, mapping parameters, mapplets.
• Implemented various Performance Tuning techniques for removing bottlenecks on Sources, Targets, Mappings, Workflows and database tuning.
• Experience in handling different file formats like Text file, Avro data files, Sequence files, XML and JSON files.
• Experienced in configuring workflows, submitting jobs, implementing schedulers using Cisco Tidal.
• Implemented test scripts to support test driven development and continuous integration.
• Wrote Java programs to generate reports to meet the business requirement using JAVA API’s and data from Pivotal HAWQ tables.

Environment: Pivotal HD 2.0, HDFS, HAWQ, Sqoop, Hive, Pig, MapReduce, HBase, Flume, Oozie, Talend, SQL, PostgreSQL, pgadmin, Cisco tidal, shell scripting, Python, Java.

Confidential - Austin, TX

Hadoop Developer

Responsibilities:

• Worked on importing data from various sources and performed transformations using MapReduce, Hive to load data into HDFS.
• Responsible for building scalable distributed data solutions using HadoopWritten various Hive and Pig scripts.
• Created HBase tables to store variable data formats coming from different portfolios.
• Performed real time analytics on HBase using Java API and Rest API.
• Worked with BI teams in generating the reports and designing ETL workflows on Tableau.
• Experience working on Talend for data migration and data synchronization processes.
• Created DDL, DML for all the GDW Staging tables.
• Developed mapplets to decrease redundancy and improve efficiency in development.
• Extensively used LTRIM, RTRIM, ISNULL, ISDATE, SUBSTR, INSTR functions in Expression transformation.
• Written complex SQL queries against the database to generate reports based on ad-hoc requests while the project was still in development phase.
• Involved in debugging issues in mappings/stored procedures when running the loads parallel in both the warehouses one source & Insight.
• Made use of mapping variables, mapping parameters, mapplets in mappings.
• Used Talend for BigDataIntegration to generate the native code to work with Hadoop and Spark.
• Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
• Configured Sqoop jobs to import data from RDBMS into HDFS using Oozie workflows.
• Worked on setting up Pig, Hive and HBase on multiple nodes and developed using Pig, Hive, HBase and MapReduce.
• Worked on compression mechanisms to optimize MapReduce Jobs.
• Analyzed the customer behavior by performing click stream analysis and to ingest the data used flume.
• Performed DataSerialization using Talend.
• Experienced with working on Avro Data files using Avro Serialization system.
• Solved small file problem using Sequence files processing in Map Reduce.
• Implemented business logic by writing UDF's in Java and used various UDF's from Piggybanks and other sources.
• Continuous monitoring and managing the Hadoop cluster using Cloudera Manager and also worked on Oozie workflow to run multiple jobs.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.

Environment: Horton works, Map Reduce, HBase, HDFS, Hive, Pig, Java, Talend, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Java (jdk 1.6), Eclipse.

Confidential - San Ramon, CA

Informatica Developer

Responsibilities:

• Developed mappings to extract data from Oracle, Teradata, Flat files, XML files, Excel and load into NIDW using the PowerCenter.
• Developed common routine mappings. Made use of mapping variables, mapping parameters and variable functions.
• Used Informatica Designer to create complex mappings using different transformations like Filter, Router, Connected & Unconnected lookups, Stored Procedure, Joiner, Update Strategy, Union, Expressions and Aggregator transformations.
• Extensively loaded data into Teradata from flat files using Teradata MultiLoad & FastLoad.
• Developed Slowly Changing Dimension for Type 2 SCD
• Used mapplets for use in mappings thereby saving valuable design time and effort
• Used Informatica Workflow Manager to create, schedule, execute and monitor sessions, worklets and workflows.
• Written procedures, Queries to retrieve data from EIDW and implemented in NIDW.
• Worked with Session Logs and Workflow Logs for Error handling and Troubleshooting in all environment.
• Responsible for Unit testing and Integration testing of mappings and workflows.
• Created Jobs and Job streams in Control-M scheduling tool to schedule Informatica, SQL script and shell script jobs.
• Extensively use LTRIM, RTRIM, ISNULL, ISDATE functions in Expression Transformation for Data quality in the staging area.
• Written SQL Queries, Triggers, and Stored Procedures to apply and maintain the Business Rules.
• Troubleshooting database, workflows, mappings, source, and target to find out the bottlenecks and improved the performance.
• Created, optimized, reviewed, and executed Teradata SQL test queries to validate transformation rules used in source to target mappings/source views, and to verify data in target tables
• Created/Enhanced Teradata Stored Procedures to generate automated testing SQLs
• Written Indexes, primary keys and checked other performance tuning at data base level.
• Conducted Database testing to check Constrains, field size, Indexes, Stored Procedures, etc.
• Implemented various Performance Tuning techniques on Sources, Targets, Mappings, Workflows and database tuning.
• Created ETLDesign documents, Test Results documents for all the workflows.
• Involved in generating reports from DM layer using Business Objects.
• Defects were tracked, reviewed and analysed.
• Used Source Analyzer and Warehouse designer to import the source and target database schemas, and the Mapping Designer to map the sources to the target.
• Performed Configuration Management to Migrate Informatica mappings/sessions /workflows from Development to Test to production environment.

Environment: Informatica PowerCenter 8.6.1, Oracle 10g, Teradata SQL Assistant, BTEQ, MultiLoad, FastLoad, SQL, IBM AIX, UNIX Shell Scripts, Business objects 6.5, Remedy, Control-M Enterprise Manager 6.3.01, Toad 10.1, MS Visual Source safe 6.0.

Confidential

ETL/SQL Developer

Responsibilities:

• Used Informatica Workflow Manager to create, schedule, execute and monitor sessions, worklets and workflows.
• Created new mappings according to business rules to extract data from different sources, transform and load target databases.
• Debugged the failed mappings and fixed them.
• Conducted Database testing to check Constrains, field size, Indexes, Stored Procedures, etc.
• Defects were tracked, reviewed and analyzed.
• Modified the mappings according to the new changes and implemented the persistent cache in several mappings for better performance.
• Implemented various Performance Tuning techniques on Sources, Targets, Mappings, and Workflows.
• Involved in writing stored procedures and shell scripts for automating the execution of jobs in pre and post sessions to modify parameter files, prepare data sources.
• Identified the issues in sources, targets, mappings and sessions and tuned them to improve performance.
• Created and used reusable mapplets and worklets to reduce the redundancy.
• Developed robust Informatica mappings and fine-tuned them to process lot of input records.
• Developing lightweight business component and integrated applications using struts.
• Developed automated test methods and documentations for these scripts.
• Worked together with the team in helping transition from Oracle to DB2.
• Developed the global logging module, which was used across all the modules using Log4Jcomponents.
• Developed the presentation layer for the credit enhancement module in JSP.
• Struts were used to implement the Model View Layer (MVC) architecture. Validations were done on the client side as well as the server side.
• Involved in the configuration management using ClearCase.
• Detecting and resolving errors/defects in the quality control environment.
• Using Ibatis for mapping Java classes with database.
• Involved in Code review and integration testing.
• Used Debugging tools such as PMD, Find Bugs and checkstyle.

Environment: Informatica Power Center, Oracle 9i, SQL, PL/SQL, Solaris, MS Vision, MS-Access, Java v1.6, J2EE 6, Struts 1.2, iBatis, XML, JSP, HTML, DB2, Unix, RAD, ClearCase, WebSphere V8.0 (beta).