SUMMARY:

• 12 plus years of total IT experience in Analysis, Design, Development, and implementation in the areas of Bigdata and Data warehousing technologies using Big Data Hadoop, Spark,Python,Java,Informatica, Oracle SQL,PL/SQL, Teradata.
• Hadoop Spark Developer with 6+ years of experience building scalable, distributed data solutions.
• Good work exposure on Cloudera and Hortonworks distribution platforms.
• Solid hands on development in Hadoop technologies including Spark using Python and Java, Hive,Impala, Sqoop, AWS EMR
• Extensive experience working with Spark distributed Framework involving Resilient Distributed Data sets(RDD) and Data Frames(Spark SQL) using Python, Java8, Scala.
• Good Experience in using Pandas and Numpy libraries in Python.
• Used Java8 to Create Spark RDD and Spark Datasets
• Used Spark SQL to read and write with external RDBMS like MySQL and Oracle.
• Good Exposure on Big data Spark Development using Scala
• Stored the data in compressed formats such as ORC, Parquet, Avro in HDFS using Spark.
• Pushed the documents using into Elastic Search using Pyspark.
• Good Knowledge on Kafka Producer, Broker and Consumer API s.
• Experienced loading data to Hive partitions and creating buckets in Hive.
• Load log data into HDFS using Kafka.
• Good Experience on Java Spring Core, Java Spring MVC.
• Worked on importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.
• Experienced in working with monitoring tools to check status of the cluster using Cloudera manager, Ambari
• Worked on both batch and streaming data processing using Spark
• Extensively used the basic constructs of python such as map, reduce, Filter and lambda functions
• Good Knowledge on Legacy Spark Streaming(D-Streams) and Spark Structured Streaming.
• Implemented Spark using Amzaon EMR for processing Big Data in AWS ecosystem.
• Created Data Ingestion Pipelines using AWS Lambda functions.
• Worked on Scheduling Lambda Functions using AWS Event Bridge.
• Created Crawlers on S3 data to form Glue Catalog tables
• Created and Ran the Glue jobs for the data transformation from one Schema to other.
• Used Python Boto3 to manage Glue Catalogs, S3 Buckets.
• Access Glue Catalog Databases and tables using AWS Athena.
• Built Streaming Pipeline using Kinesis Firehouse Delivery Stream.
• Experienced in working with Teradata, Oracle9i/10g, MySQL
• Strong knowledge in OLAP Systems, Kimball and Inmon methodology models, Dimensional data modelling using Star schema and Snowflake schema.
• Extensively used the Data Warehousing concepts like Slowly Changing Dimensions, Facts, De-Generated Dimensionsand Conformed Dimensions.
• Experience in Extraction Transformation and Loading (ETL) data from various data sources into Data Marts and Data Warehouse using Informatica PowerCenter components (Repository Manager, Designer, Workflow Manager and Workflow Monitor).
• Experienced in writing Oracle SQL queries using Sub queries, Joins, Inline views, Materialized views, and various Oracle functions.
• Experienced in writing PLSQL blocks using Cursors, stored Procedures, Functions, Packages, triggers and PLSQL Collections
• Extensively worked on Teradata, Primary, Secondary, Partition Primary and Join Indexes.
• In-depth expertise in the Teradata cost-based query optimizer, identified potential bottlenecks with queries from the aspects of query writing, skewed redistributions, join strategies, optimizer statistics, physical design considerations (UPI, NUPI, USI, NUSI, PPI and JI).
• Experienced in preparing the various documents like Requirement document, high level and low-level design documents, Business Mapping documents, Change request documents and Implementation documents.
• Good Experience in Performance tuning and error handling.

TECHNICAL SKILLS:

Languages: Python, Java, SQL, PL/SQL, T-SQL, UnixShellScripting, Java, Hadoop, MapReduce, Hive, Sqoop, Spark Core and SparkSQL using Python,Java8,Scala, AmazonEMR, SparkStreaming, Kibana, SOLR, Apache Elastic, Java Spring Core, Java Spring MVC.

RDBMS: Oracle 8i/9i/10g, Teradata V2R5/V2R6/12, MS SQL server 2008, DB2, MySQL

Tools: Apache Elasticsearch, ApacheSOLR, LireSOLR, SVN, Rally, DBVisualizer, Teradata SQL Assistant, TOAD, SQL Plus, SQL Developer, MSVisio, JIRA, HP Quality Center, Citrix, Mercury Quality Center, Erwin

SCM Tools: Harvest, Kintana, Informatica Version Control, PVCS

Data Warehousing: Informatica Power Center 9.6(Designer, Repository Manager, workflowmanager, Workflow monitor), Informatica DT Studio

Distributions: HortonworksDataPlatform, Cloudera

PROFESSIONAL EXPEREINCE:

Confidential, VA

Bigdata Hadoop Spark Developer

Responsibilities:

• Design Develop and Implement Big Data analytic solutions on a Hadoop - based platform.
• Refine a data processing pipeline focused on unstructured and semi-structured data refinement
• Create custom analytic and data mining algorithms to help extract knowledge and meaning from vast stores of data.
• Support quick turn and rapid implementations and larger scale and longer duration analytic capability implementations.
• Create Spark SQL and Spark RDD s using Java8 as part of Horton Works HDP 3.1 migration.
• Configure the Data flows from different sources (relational databases, XML, JSON) and orchestrate them using Nifi.
• Developed Spark Frameworks using PySpark and Java to build Raw/Analytical Layers in Bigdata.
• Developed utilities in Python and Core java.
• Write the Data Extraction, Processing and Transformation scripts using HIVE and SPARK wherever is needed.
• Used Jenkinsfor Continuous Integration and Git for Version Control.
• Write the shell scripts and job management scripts to invoke and manage the Data Ingestion steps.
• Design the HIVE tables for better performance as the data volume would be very high. Apply partitions wherever is needed.
• Design and develop the SPARK programs that processes the high volumes of data with higher processing speeds.
• Write the programs that generates the intermediator data sets in ORC or PARQUET format for better performance
• Write the programs that pushes the refined dataset into Elastic Search.
• Review the HDFS data organization and provide better mechanism to support Multi-Tenant features on Hortonworks.
• Work on AWS Services like S3, EMR, Lambda as part of Open data initiative.
• Design and develop the mechanism to segregate PUBLIC and PRIVATE datasets
• Implement the Data archiving and retention mechanisms.
• Provide solutions for Metadata Management.
• Loading from disparate data sets.
• Translate complex functional and technical requirements into detailed design.
• Perform analysis of vast data stores and uncover insights.
• Maintain security and data privacy.
• Create scalable and high-performance web services for data tracking.

Environment: Java8, Spark 2.3, Python with Spark, with Spark, Hadoop, Hive, HDP 2.6, Amazon EMR, Amazon S3, AWS Lamba, AWS Glue, AWS Athena, Boto3, MySQL, Oracle 10g, ElasticSearch, ApacheSOLR, lire SOLR, DB Visualizer, SharePoint 2010, Rally, SVN, Maven, GIT,Jenkins.

Confidential, Desmoines, IA

Data Lead

Responsibilities:

• Involved in full life cycle of the project from Design, Analysis, logical and physical architecture modeling, development, Implementation, testing.
• Responsible for design and creation of Hive tables, partitioning, bucketing, loading data and writing hive queries. Implemented, migrated existing Hive Script in SparkSQL for better performance.
• Developed Spark streaming jobs in JAVA 8 to receive real time data from Kafka, process and store the data to HDFS.
• Act as a single POC between business and technical teams for data and reports.
• Report to internal and external stakeholders on the progress of the Project(s).
• Responsible for technical support and expertise for the technical area represented.
• Involved in Designing a system that would process 1B records a day and stores 7 years of such data that would involve Hive, Sqoop, Spark, Kafka, Java UI, Presto, Oracle. This system would involve massive sized and number of Tar files and binary files.
• Understand the projects objectives, goals, scope and ensure technical changes are made andissues are resolved.
• Gather and create business requirement document and involve in framing the logic for extraction,transformation, and loading (ETL) processes.
• Translate technical specifications to system architecture.
• Create technical/design documentation of system and technical process changes per Businessrequirements.
• Create, design, develop and update system code changes using Informatica Power Center (9.6,10.2) or SQL Stored Procedures and provide technical expertise on solutions.
• Develop code, unit test cases and perform unit testing of jobs in development, support jobs andresolve any defects during UAT (User Acceptance Testing)
• Promote code and provide warranty in the higher pre - production and production environments.
• Work with onsite and offshore teams to guide the development of various solutions.
• Prepare Korn shell (Unix/Linux) scripts and integrate with the scheduler for automating theInformatica jobs.
• Ensures the integration of software packages, programs, and reusable solutions on multipleplatforms
• Coordinate back-out plans for test and production environments
• Performance tuning of the Teradata queries and help with data modelling in the EDW (EnterpriseData Warehouse) stream.
• Follow all steps of system life cycle and project SDLC phases in all technical work
• Determine the best logical and physical architecture for the project and maintain architecturalintegrity of software.
• Identify Root cause analysis and resolve testing defects. Assist with finding new test data andcreating extracts for test data as needed.
• Escalate issues, risks, and changes to project manager as they arise and communicate task/projectstatus periodically.

Environment: Hadoop, Sqoop, Hive, Cloudera Distribution platform, Spark, Python, Informatica PowerCenter 9.6, Informatica Power Exchange, Mainframes, DB2, Oracle 10g, SQL Developer, SharePoint 2010, Autosys, Jira

Confidential, Cedar Rapids, IA

Bigdata Hadoop Technical Lead

Responsibilities:

• Use Spark to land data in Hadoop systems using Python and Scala.
• Extensively used Hbase(Mapr - DB) Java APIs - PUT, GET, SCAN and Filter APIs.
• Developed UDFs, UADFs to prepare the data that will be feed to JAVA Mapreduce programs.
• Involved in REST API Development using JAVA.
• Developed Java code that APIs will use and executes HIVE, PIG scripts as a part of Java Code.
• Worked on the Continues Integration of BigData build process.
• Moving the data to Hadoop will help streamline Confidential s business by automating many processes which are currently manual
• Pulling data from mainframe policy administration systems and landing the data in Hadoop
• Worked on importing and exporting data, into & out of HDFS and Hive using Sqoop
• Worked on creating Hive tables and wrote Hive queries for data analysis to meet business requirements.
• Extensively used Spark stack to develop preprocessing job which includes RDD, Datasets and Data framesApi's to transform the data for upstream consumption.
• Analyzing the requirement specifications provided by the client and translating them to Technical Impacts in the system
• Participating and performing tuning of the ETL processes
• Create High Level design, Detailed Level design, design specifications, test plan and test scripts
• Resolve complex issues during integration and system testing
• Involve in code development and code review during development and Integration testing.

Environment: Hadoop, Sqoop, Hive, Cloudera Distribution platform, Spark, Python, Informatica PowerCenter 9.6, Informatica Power Exchange,Mainframes, DB2, Oracle 10g, SQL Developer, SharePoint 2010, Autosys, Jira

Confidential, Madison, WI

BigData/Informatica Technical Lead

Responsibilities:

• Analysis of the requirement specifications provided by the clients and translating them to technical impacts in the system.
• Utilized in - depth knowledge of functional and Technical experience in Data Warehousing and other leading-edge products and technology in conjunction with industry and business skills to deliver solutions to customer.
• Experienced in writing Sqoop scripts to import data into Hive/HDFS from RDBMS.
• Developed Hive Queries to analyze the data in HDFS to identify issues and behavioral patterns.
• Involved in writing optimized Pig Scripts
• Able to use, Spark, Python Pandas, Numpy modules for Data analysis, Data scraping and parsing
• Excellent experience in Informatica Designer to create complex mappings using different transformations like Filter, Router, Connected & Unconnected lookups, Stored Procedure, Joiner, Update Strategy, Expressions and Aggregator transformations to pipeline data to the target.
• Implemented Concurrent execution of workflows and Session Partitioning techniques as part of Performance Tuning.
• Worked on different tasks in Workflow Manager like Sessions, Events raise, Event wait, Decision, E-mail, Command, Worklets, Assignment, Timer and Scheduling of the workflow.
• Worked on Informatica DT Studio to parse input XML and Json files
• Created Informatica Web service provider using XML Generator, XML Parser and SQL Transform
• Handled Pushdown Optimization technique in order to tune the mappings and sessions when working on bulk loads or huge volumes of data.
• Used Debugger in the mappings, identified bugs in existing mappings by analyzing the data flow and evaluating transformations.
• Cleaned up mappings to not create lengthy log files by turning off verbose logging and getting rid of warning messages.
• Designed and developed stored procedures using PL/SQL and tuned SQL queries for better performance.
• Implemented slowly changing dimension (SCD Type 1 & 2) in various mappings.
• Created and used reusable Transformations using Informatica PowerCenter.
• Worked on AutoSys to automate Execution of Informatica jobs
• Created High Level Design, Detailed Level Design, Design Specifications, test plan and test scripts.
• Responsible for overseeing the Quality procedures related to the project.
• Complex Issue resolution during the integration testing and system testing phases.
• Involved in code development and reviewing the code developed by peer group.
• Integration testing of the code.
• Responsible for writing/documenting the Unit Test Cases with different testing scenarios to meet business rules implemented in ETL mappings.

Environment: Hadoop, HDFS, Hortonworks distribution platform PIG, Hive, Python, Spark, SparkSQL, Informatica PowerCenter 9.1, Informatica DT Studio, Oracle 10g, Greenplum, SQL Developer, Winsql, SharePoint 2010, Autosys, Jira, Harvest, XML Spy Editor

Confidential, Boston, MA

Sr. Informatica Developer

Responsibilities:

• Interacted with Business Analysts to identify the process and requirements gathering, implementing design process, development and preparing business rules.
• Worked closely with project managers, business analysts, DBA's to achieve business and functional requirements.
• Worked in Software Development Life Cycle (SDLC) like in Agile Scrum methodologies.
• Used Informatica Power Center 9.1 for extraction, loading and transformation (ETL) of data in the data warehouse.
• Built efficient ETL Informatica packages for processing fact and dimension tables with complex transformations using type 1 and type 2 changes.
• Designed and developed complex mappings from varied transformation logic like Unconnected and Connected lookups, Router, Filter, Expression, Aggregator, Joiner, Update Strategy and more.
• Worked on Informatica Power Center tool - Source Analyzer, Target Designer, Mapping Designer and Transformation Developer.
• Assisted in the design and Maintenance of the Metadata environment.
• Created Workflows and Sessions to load data from the SQL Server, Oracle, flat file, and XML file sources that exist on servers located at various locations.
• Responsible for creating business solutions for Incremental and full loads.
• Created different parameter files and changed Session parameters, mapping parameters, and variables at run time.
• Created high-level design documents for extracting data from complex relational database tables, data conversions, transformation and loading into specific formats
• Architected and Designed the ETL solution that included designing Mappings and workflows, deciding load strategies, implementing appropriate error handling and error notification processes, scheduling and designing re-usable ETL pieces through parameterization.
• Developed Mapping using parameters, Session parameters, Mapping variable/parameters and created Parameter files and runs of workflows based on changing variable values.
• Checked Sessions and error logs to troubleshoot problems and used debugger.
• Created Unix Shell Scripts to automate pre-session and post-session processes.
• Involved in creating new table structures and modifying existing tables discussing with Data Modeler.
• Updated the tables as per the requirements by generating queries using SQL to check the data consistency in the table.
• Created shortcuts for reusable transformation, source/target definitions, Mapplets in Shared folder.
• Performed unit testing, Integration testing, Performance and Functional testing of the mappings.
• Involved in High level and low-level design, analyzing Source to Target Mapping (STM's), Test Cases and Code migration report.
• Worked effectively in a distributed global team environment across US, India.
• Worked with Functional Designers and IT Planners to meet Business level requirements.
• Conduct code walkthroughs and review peer code and documentation.
• Train and Manage developers advising other groups in organization on Informatica development, data warehouse development and ETL development best practices.

Environment: InformaticaPowerCenter 9.6, MS SQL server 2008, SQL Developer, Toad, T-SQL, Windows Server

Confidential, Detroit, MI

Sr. Informatica Developer

Responsibilities:

• Identify all OMD/CRM Tables in PRD1 Shared with Other Systems
• Identify all CRM/OMD Tables that Provide Data to Other Systems
• Identify all Interfaces with CRM/OMD Data
• Identify all Tables Referenced by Interfaces
• Catalog all Dependencies on PRD1 CRM/OMD Data
• Formulate Oracle SQL - Queries on Informatica Metadata database repository in order to identify Informatica jobs which source through/from OMD/CRM systems to other systems.
• Involved in loading analysis tables to SDR-SAMBA s data repository.
• Involved in the design of SDR-SAMBA S data repository.
• Analyze Views, Materializedviews, and Stored procedures
• Analyze DB audit and Trace to check the Table access, patterns, and frequencies
• Involved in basic profiling on DB schema to Identify the table data usage and volume
• Analyze Informatica Jobs using Informatica Meta data to identify the tables involved
• Analyze Perl, Unix scripts to identify for the tables used
• Create the Interface dependencies and hierarchies.

Environment: Informatica PowerCenter, Toad, SQL developer, Oracle Metadata, Erwin, Visio, Windows Server

Confidential, SanJose, CA

Informatica Developer

Responsibilities:

• Responsible for designing and development, testing of processes necessary to extract data from operational databases, Transform and Load into data warehouse using Informatica Power center.
• Responsible for Modeling, Design, Development, and Integration Testing for BIDS Platform.
• Make sure that all the Dev and Stage Environments are sanitized within given SLA.
• Develop the ETL s based on the BRD s and ETL specification documents provided by the client using Informatica.
• Created complex mappings in Power Center Designer using Expression, Filter, Sequence Generator, Update Strategy, Joiner and Stored procedure transformations.
• Worked Extensively on Informatica tools like Designer, Workflow monitor and Workflow Manager.
• Worked on all the Transformations like Lookup, Aggregator, Expression, Router, Filter, Update Strategy, Stored procedure, Sequence Generator.
• Created connected and unconnected Lookup transformations to look up the data from the source to ETL target tables.
• Wrote SQL, PL/SQL, stored procedures for implementing business rules and transformations.
• Used the update strategy to effectively migrate data from source to target.
• Created test cases and completed unit, integration, and system tests for Data warehouse.
• Involved in debugging and validating the mappings and Code Review to rectify the issue.
• Extensively used various Performance tuning Techniques to improve the session performance.
• Developed, implemented, and enforced ETL best practices standards.
• Worked on all the Transformations like Lookup, Aggregator, Expression, Router, Filter, Update Strategy, Stored procedure, Sequence Generator.
• Debugging and validating and executing mappings and Involved in Code Review and rectify the error.
• Created Scheduled Sessions and Batch Process based on demand, run on time, run only once using Informatica Server Manager.
• The load ready files are loaded into Teradata Tables using Teradata ODBC/Fastload, Multiload connection.
• Responsible for fixing the issues on priority to meet SLA.
• Get the business requirements from various sources like business docs, Functional teams, Business analysts, SME s, and report users.
• Analyzed on the enhancements required on the source systems.
• Worked in Iterative based as well as agile environment.
• Re - designed multiple existing Power Center mappings to implement change request (CR) representing the updated business logic.
• Developed CDC and SCD Type 1,2,3 mappings in order to meet the business requirements.
• Created reusable transformations to increase the re-usability during the development life cycle.
• Closely worked with the reporting team to ensure that correct data is presented in the reports.
• Responsible for creation of ETL technical Specification documents based on BRD.

Environment: Informatica PowerCenter 9.1, Teradata V2R5, Oracle 10g, TOAD, SQL Developer HP Quality Center, Dollar Universe, Kintana, PVCS

Confidential, Columbus, OH

Oracle SQL, PLSQL, Forms10g Developer

Responsibilities:

• Involved in developing the technical documents from functional specs.
• Performed data quality analysis to validate the input data based on the cleansing rules.
• Back end Programming for Oracle Forms10g using PL/SQL Cursors, Stored Procedures, Functions, Exceptions, Packages, Dynamic SQL, and Triggers to in corporate the business logic into the application.
• Extensively used PLSQL Collections like NESTED Tables and VARRAYS.
• Ensure Quality and On - time delivery.
• Actively participated in gathering Business Requirements and System Specification from system users.
• Analyze the current data management procedures in practice and suggest ways for automating the process or improving the existing system.
• Improved performance and tuning of SQL queries and fixed the slow running queries in production by using utilities like Explain plan and tk prof.
• Involved in Developing and handling the PL/SQL Packages, Procedures using Bulk collect, for all and designed Database Triggers.
• Involved in unit testing and integrated testing.
• Involved in preparing the User Test plans.
• Tuned complex stored procedures for faster execution and developed database structures according to the requirements.
• Experience in developing various PL/SQL objects like procedures, functions, packages, and triggers in accordance with the business needs.
• Involved in interacting with the end-user (client) to gather business requirements and modeling the business requirements.
• Involved in performance tuning of a complex queries and used bulk collect feature to improve the performance.
• Participated in analyzing the requirements for the logical and physical design of the database.
• Designed, developed, and customized forms by using Oracle forms 9i.
• Implemented complex views, procedures, functions, packages, and triggers in PL/SQL.
• Performed exception handling for data that did not conform to the required format.
• Implemented PL/SQL scripts in accordance with the necessary Business rules and procedures.
• Developed PL/SQL packages and procedures for the back-end processing of the proposed database design.

Environment: Oracle 11g/10g, SQL, PL/SQL, SQL*Loader, Oracle Designer, Oracle Forms 9i, Mercury Quality Center.