We provide IT Staff Augmentation Services!

Project Lead / Sr. Etl/hadoop Developer Resume

Plymouth Meeting, PA

SUMMARY

  • Around 11 years of experience in software engineering profession in all phases of SDLC including application design, development, production support & maintenance projects.
  • Extensive experience in the field of Data Analysis, Data Conversion and integration, Data Cleansing, actively participated to gather Legacy systems/Source systems requirement information, Production Support (M&P & ACC) Designing and coding, scripting, development, project deployment by using DWH tools.
  • 2+ years of experience inHadoopand its ecosystems such as HDFS, MapReduce, Yarn, Hive, Pig, HBase, Sqoop, Python, Spark, Scala.
  • 8 years of experience in Data warehousing ETL Tools such as Datastage 11.5/9.1/8.5/8.1/7.5 , Informatica, Abnitio, IBM Cast Iron, Contextia.
  • Good knowledge of Data Warehousing concepts, methodologies and Banking processes.
  • Worked on IBM BDW (Banking Data Warehouse) model.
  • 7+ years on Datastage 11.5 / 9.1 / 8.1/ 7.5.1 including all stages, Job Sequence, Routines, Containers.
  • 2+ years on Informatica Workflows, sessions.
  • Have knowledge on Teradata.
  • Worked in Banking Process for calculating Credit Risk for Basel Accord II
  • Extensive experience on Banking, Insurance, Healthcare and Social Media domains.
  • Expertise in Designing and Architecting Hadoop Applications and recommending the right solutions and technologies for the applications.
  • 5+ years hands - on expertise with UNIX shell scripts and commands.
  • 6+ years hands-on expertise with DB2/SQL/Netezza writing queries.
  • 3+ years on Agile/Scrum methodology
  • Having Good knowledge on Performance Tuning based on SQL Queries, Pig scripting and Datastage and ETL tools functionalities.
  • Highly motivated team player with excellent communication, presentation and interpersonal skills, always willing to work in challenging and cross-platform environment.
  • In-depth knowledge of creating Project architecture, use cases, functional design specifications, activity diagrams, logical, component and deployment views to extract business and technical process flow.
  • Knowledge in the ETL (Extract, Transform and Load) of data into a data ware house/date mart and Business Intelligence (BI) tools like Cognos Report net.
  • Performed and supported UTC (Unit test cases), (SIT) System Integration testing, UAT and exposure to User Certification Testing (UCT) and Operational Readiness Testing (ORT)
  • Expertise in understanding and supporting the client with Project Planning, Project Definition, Requirements Definition, Analysis, Design, Testing, System documentation and user training.
  • Well versed in writing queries/Scripts for Data Analysis and QA reporting and testing.
  • Good Experience as a Tech / Project Lead.
  • Actively Working for Production Support.

TECHNICAL SKILLS

  • Big Data Hadoop
  • MapReduce
  • HDFS
  • YARN
  • Hive
  • Pig
  • Sqoop
  • Python.
  • Datastage 11.5/9.1/8.0/7.5.1
  • Informatica
  • Contextia
  • BO Reporting.
  • Data Warehousing Concepts
  • LINUX
  • UNIX
  • Windows
  • ORACLE
  • MySQL
  • DB2
  • Netezza
  • Waterfall
  • Agile
  • Control M Scheduler Tool

PROFESSIONAL EXPERIENCE

Confidential, Plymouth meeting, PA

Project Lead / Sr. ETL/Hadoop Developer

Environment: Hadoop, Spark, Hive, Python, Unix, Shell Scripting, Amazon Redshift, SVN, S3, Oracle

Responsibilities:

  • Provide design recommendations and thought leadership to sponsors /stakeholders that improved review processes and resolved technical problems.
  • Day-to-day operation and execution of a OMOP conversion
  • Is the decision maker for technical decisions of a single OMOP conversion
  • Provide solution to technical challenges by working with others, such as the senior developers
  • Work with the Data Analyst to have a strong understanding of the business rules for a OMOP conversion
  • Design/Develop the ETL code to ensure the completion of OMOP conversions on time and according to IMS standards
  • Working on Bigdata to store the conversion data on Analytics cluster of Hive.
  • Using Sqoop to import data from Hive to other Downstream projects.
  • Using Spark engine on Cloudera Environment to process the Hive load scripts.
  • Reading the XML files using Python scripting.
  • Configured project with Jenkins to build on continuous integration.
  • Working on AWS Services such as EC2, Redshift, S3 for ETL conversions.
  • Working with svn commands for the code check in/out from AWS EC2.
  • Using Purple and ant process to execute scripts on AWS.
  • Using Jenkins to schedule or trigger the project batch run.
  • Implemented continuous Integration on Jenkins to Build the Scripts/projects.
  • Responsible for creating the Refresh documentations for the datasets they own
  • Works with QA team to ensure quality of OMOP conversion
  • Attend requirements meeting to ensure full understanding of data and requirements
  • Ensure code review is done for all development.

Confidential, San Antonio, TX

Project Lead / Sr. ETL/Hadoop Developer

Environment: Hadoop, Hive, Pig Scripting, Python, Unix, Shell Scripting, Datastage 11.5, DB2, ControlM, Core Java

Responsibilities:

  • Provide design recommendations and thought leadership to sponsors /stakeholders that improved review processes and resolved technical problems.
  • Co-coordinate between the Business and the Off-shore/Near Shore team.
  • Requirement gathering and prepare the Design.
  • Make Request and Download API calls by using Hierarchical stage in Datastage 11.5.
  • Done json parsing and Error handling for API calls in Datastage.
  • Implemented CURL script to make API calls to Download csv Reports from Lithium.
  • Implemented Unix shell, Pig, Hive, Python scripting.
  • Invoked Pig, Python, Jar in Unix shell script.
  • Implemented SFTP script by using inbuilt Charon utility.
  • Implemented DB Export/Import scripts by using Thoosa utility.
  • Involve in creating Hive tables, loading with data and writing Hive queries.
  • Implement solutions using Hadoop, Hive, Sqoop, Java API, etc.
  • Code Reviews and Pair programming.
  • Load data into Hive partitioned tables.
  • Implemented Parameterized Shared container.
  • Updating Wiki for Source Information, Datastage and Hadoop design details, Requirement doc, Architecture and the same will be approved by the Client Business.
  • Following Agile methodology by 2 weeks Iteration.
  • Attend daily stand-up huddle to provide daily base status.
  • Supporting M&P (Production Support & ACC) Project.
  • Used control M tool to schedule scripts and Batch runs
  • Worked with Stakeholder and Product owners to provide adhoc reporting for the business users.
  • After the development used to have Unit test cases (UTC), SIT and UAT.
  • Co-ordination with Off Shore and Nearshore.
  • Preparing Demo’s to get sign off from Client/Business for the Iteration tasks.
  • Migrating Existing Social Datastage projects (Social Harvesting, SocialFinraUserInfo, Social Insights, Military Affairs) to New version of Datastage 11.5

Confidential, San Antonio, TX

Project Lead / Sr. ETL/Hadoop Developer

Environment: - Hadoop, Hive, Pig Scripting, Python, Spark, Scala, Unix, Shell Scripting, Netezza, Squirrel, DB2, Datastage, Control M, Java

Responsibilities:

  • Interacted with Client for the requirement. Gather Source information from legacy systems like Social Listening, ECIF OptIn, Harvesting.
  • Updating Wiki for Source Information, Hadoop design details, Requirement doc, Architecture and the same will be approved by the Client Business.
  • Demo preparation to get sign off from Client/Business for the Iteration tasks.
  • Following Agile methodology by 2 weeks Iteration.
  • Implemented Unix shell, Pig, Hive, Python scripting.
  • Invoked Pig, Python, Jar in Unix shell script.
  • Implemented Scripts for BO reporting.
  • Generate XML files for the member data.
  • Implemented SFTP script by using inbuilt MapReduce Charon utility.
  • Attend daily stand-up huddle to provide daily base status.
  • Supporting M&P (Production Support, ACC) Project.
  • Implemented DB Export/Import scripts by using Sqoop Thoosa utility.
  • Write Map Reduce Jobs, HIVEQL.
  • Implemented HBase tables on Spark.
  • Support code/design analysis, strategy development and project planning.
  • Create reports for the BI team using Sqoop to export data into HDFS and Hive.
  • Involve in creating Hive tables, loading with data and writing Hive queries which will run internally in MapReduce way.
  • Used control M tool to schedule scripts and Batch runs.
  • Co-ordination with Off Shore and Nearshore.
  • Load and transform large sets of structured data.
  • Load data into Hive partitioned tables.
  • Create UDF and UDAF using core java to have standard classes as per business needs.

Confidential

ETL Developer / Associate Projects

Environment: Datastage 8.5, MS SQL server 2008, Oracle PLSQL, Unix, Hadoop, Hive, PIG

Responsibilities:

  • Supported code/design analysis, strategy development and project planning.
  • Interacted with users as part of Business Analysis to collect requirements from Legacy systems like Galaxy, Tridion, Agent Customer etc., analyze and transform them into Functional and technical specifications via documentation.
  • As this is POC project, prepared the Architecture of the project which makes customers to understand the process much easier.
  • Defined data model requirements to meet reporting, analysis & specific Business Intelligence needs. Gathered high level requirements.
  • Documented Functional and Non-functional requirements.
  • Work closely with Project Manager in developing project plans and managed changes to the scope of the project.
  • Once the requirement is gathered from Legacy systems, Implement, Design and Develop Datastage jobs to extract info from Legacy systems and load to Data warehousing and DataMart’s.
  • Used Hive to store Analytics Data.
  • Worked with PIG for the validations on Analytics Data before Loading to Hive.
  • Worked with Quality stage to standardize the rules for Location and address match etc.
  • Worked with IBM Information server to analyze the mappings and Integrity.
  • Built the SQL queries and Stored procedure to extract and analyze the data from Legacy systems
  • Worked in various stages of Datastage to match requirement analysis for project needs.
  • Created business use case, functional FRS and technical specifications for trading system.
  • Extensively involved in the modelling and development of Reporting Data Warehousing System.
  • Prepared Business Requirement Documents (BRD’s) after the collection of Functional Requirements from System Users that provided appropriate scope of work for technical team to develop prototype, Use Cases for overall system.
  • Developed business requirement specification documents as well as high-level project plan.
  • Created and maintained source to target mapping document to meet new data requirements.
  • Developed & executed several Optimized queries in SQL on this data.
  • Wrote PL/SQL statement and stored procedures in Oracle for extracting as well as writing data.
  • Worked closely with developer and Architects to conducted data Analysis using SQL, Oracle database
  • Worked with Stakeholder and Product owners to provide adhoc reporting for the business users.
  • After the development used to have Unit test cases (UTC), SIT and UAT.

Confidential

Associate Projects

Environment: Informatica 9.5.1, Oracle PLSQL, Control M

Responsibilities:

  • Interacted with users to collect the requirement from source systems of Trail site metrics and clinical site metrics and convert them into Functional and technical specifications via documentation.
  • As part of this migration project of Tapas from Basel server to US server, prepared the Impact Analysis and by using PLSQL Stored procedures and Packages migration has been done for the same.
  • Defined data model requirements to design and enhance the Informatica mappings and reporting, analysis & specific Business Intelligence needs. Gathered high level requirements.
  • Documented Functional and Non-functional requirements.
  • Work closely with Project Manager in developing project plans and managed changes to the scope of the project.
  • Created Informatica mappings, Sessions and Workflows to convert Legacy systems data to Data warehousing and DataMart’s.
  • Built the SQL queries and Stored procedure to extract and analyze the data from Legacy systems
  • Created business use case, functional FRS and technical specifications for trading system.
  • Extensively involved in the modelling and development of Reporting Data Warehousing System.
  • Prepared Business Requirement Documents (BRD’s) after the collection of Functional Requirements from System Users that provided appropriate scope of work for technical team to develop prototype, Use Cases for overall system.
  • Developed business requirement specification documents as well as high-level project plan.
  • Created and maintained source to target mapping document to meet new data requirements.
  • Developed & executed several Optimized queries in SQL on this data.
  • Wrote PL/SQL statement and stored procedures in Oracle for extracting as well as writing data.
  • Worked closely with developer and Architects to conducted data Analysis using SQL, Oracle database
  • Worked with Stakeholder and Product owners to provide adhoc reporting for the business users.
  • After the development used to have Unit test cases (UTC), SIT and UAT.

Confidential

Associate Projects / Sr. ETL Developer

Responsibilities:

  • Participated in the requirements gathering, analysis & design of the application.
  • Interacting with the Business Analysts to get the User specifications.
  • Preparation of Functional & Program Specifications for the offshore team.
  • Working as Associate of the project and responsible to understand source systems, development, deployment of the project, batch runs, issue analysis and Handling KT for new joiners.
  • Communicating with Client users in the requirement gathering and issue analysis process of the project.
  • Actively participated in forming the Contextia Coding, SQL Queries and Run checklist.
  • Used to have Pre and Post conversion reports by using SQL Queries and Contextia code.
  • Involved in the creation of new Migration Project, which is being used for all the Insurance policy applications.
  • Involved in preparation of test cases and creation of technical specification documents for the Legacy systems.
  • Worked on Nexsure front end tool to retrieve Acords and Policy info for issue analysis.
  • Understand the Technical specification and different logging specifications.

Confidential

Product Analyst / ETL developer

Environment: IBM DB2, Ascential Datastage 7.5.1 Server and Parallel, Unix, Vertis, Ilog, Cognos Reportnet

Responsibilities:

  • Worked as a Product analyst to understand the Source systems functionality to analyze and generate reports by using SQL complex queries
  • Was responsible to prepare functional specification, development of the project.
  • Worked in Issue analysis and fix in Production.
  • Handled Batch runs of Basel.
  • Had been to onsite Malaysia to give training on Basel, Batch runs and Issues to be handled in Production data.
  • Participated in the requirement gathering and feasibility analysis process of the project.
  • Actively participated in forming the Data Stage Coding, DS Objects naming convention standards, DS Category/ Unix Folder Creation standards and DataStage Code Review checklist and Performance Tuning.
  • Involved in the creation of jobs, which is being used for all the Countries/source systems and involved in preparation of test cases and creation of technical specification documents for the source systems.
  • Engaged in Designing, Developing and Testing Datastage PX jobs for the different source systems.
  • Design and Development of ETL modules using Data Stage.
  • Working with Complex SQL and Dynamic SQL queries to extract the data in the Data stage jobs. Source File Validation and Transformation.
  • Reconciliation
  • Exception and Rejection Handling
  • Handling of Incremental Loading.
  • Handling Rollback in case offailure in loading to Warehouse
  • Involved in Unit Testing and UAT (User Acceptance Test).

Hire Now