Hadoop & Etl Developer Resume
Plano, TX
PROFESSIONAL SUMMARY:
- Result driven professional with over 10 years of experience in all phases of Software Development Life Cycle which includes extensive experience in BIG DATA, ORACLE, DB2, ETL & JAVA applications spanning across technologies and business domains.
- Good exposure on Hadoop ecosystem core components as HDFS,YARN,MapReduce and other major components like Hive, Pig, HBase, Sqoop, Oozie, Flume, Zookeeper, Spark and Scala programming.
- Experience in writing complex HIVE SQL for Data Warehousing and Ad - hoc analysis.
- Experience in SCALA programming to perform data loads from semantic zone to workzone for downstream applications consumption.
- Experience in importing and exporting data between RDBMS and HDFS using Sqoop.
- Good knowledge in using job scheduling and monitoring tools like Oozie and Zookeeper
- Experience in working with Spark echo system using Spark SQL and Scala on different formats like text file,Avro,Parquet files
- Good knowledge in NOSQL column store DB.
- Extensively worked on Oracle SQL & PL/SQL programming
- Strong experience in Oracle (9i, 10g, 11g&12 c) SQL & PL/SQL Programming.
- Strong experience as a PL/SQL Developer in writing complex scripts using
- Subqueries, joins, aggregate functions, dynamic queries, Functions, Stored Procedures,
- Packages, Triggers, PL/SQL Collections using associate arrays, PL/SQL table, varrays&bulk collection and bind variables.
- Experience in creating Tables,Views,Materialized Views(MView),Constraints
- Indexes, Synonyms & Table Partitions (Range, Range Interval, Hash, List)
- Experience in SQL Performance Tuning using Explain Plan and Query Optimization that includes Index Optimization, table partitioning, collect stats and other optimization techniques.
- Experience in using Oracle DB Links to connect multiple external Databases
- Experience in importing data from SharePoint site into oracle tables using SQLLDR
- Experience in uploading table data into SharePoint site using SFTP or CURL process.
- Extensively worked on Oracle Dynamic SQLs & Package Utility such as FILE UTL, DBMS OUTPUT and UTL SMTP.
- Developed JAVA application based on ITIL Framework using advanced concepts likeServlets, JSP, JSTL, JDBC, JavaScript, JQuery
- Experience in UNIX shell script. Developed UNIX shell scripts to call DB Objects from Control M scheduling tool.
- Experience in ETL tool Informatica for data processing and loading into stage, integration and semantic tables.
- Experience in ETL tool Ab initio for data processing and ingestion into HDFS file system from semantic zone to work zone and also to write into Oracle tables.
- Certified in Oracle and ITIL V3 Foundation
- Extensively worked on implementation of Incident Management, Problem Management, Change Management, Capacity Management, Request Fulfillment and KEDB.
- Conducted Various on ITIL Framework & Implementation
- Experience in support project as L2/L3 support
- Experience in both Waterfall & Agile Methodologies.
- Experience in Version Control System SVN (Tortoise Subversion Control), BIT Bucket & GIT Hub.
- Ability to blend technical expertise with strong Conceptual, Business and Analytical skills to provide quality solutions and result oriented problem solving techniques and leadership skills
TECHNICAL SKILLS:
Big Data Eco System: Hadoop,HDFS,YARN,MapReduce,Sqoop,Hive,Pig,Flume Oozie,Zookeeper,HBase,Spark,Scala
Programming Skills: Java,PL/SQL,Shell Scripting,Scala
Web Technologies: HTML, CSS, XML, JDBC, JSP, JSTL.
Operating System: Windows,Linux,Unix
Tools: Eclipse, TOAD for Oracle 12.8,, SQLDeveloper,GIT,FileZilla,WinScp,Service Manager, Ab Initio GDE 3.3.2 (Wester European) sqlplus, sqldrBMC Control M 9.0.00 Scheduling Tool,Tera Data SQL assistant,PuTTY for UNIX,ITSM Code Promotion Regulatory Tool,EPV AIM Account Management Security Tool,MS Office Suite of Tools,Issue Tracking Tool
Database: Oracle (9i,10g,11g,12c),DB2,SQL Server
Methodologies: Waterfall, Agile
PROFESSIONAL EXPERIENCE:
Confidential, Plano, TX
Hadoop & ETL Developer
Responsibilities:
- Analyzed functional specs provided by business, data architect and created technical spec documents.
- Converted complex Oracle procedure/packages to Ab-Initio components as part of Info1 Oracle exit program. During this process, worked on most of the Ab-initio components.
- Developed Generic graphs for data loads, data cleansing, data validation and data transformation.
- Responsible for cleansing the data from source systems usingAb Initio componentssuch as Join, Dedup Sorted, Denormalize, Normalize, Reformat, Filter-by-Expression, Rollup
- Worked with Departition Components like Concatenate, Gather, Interleave and
- Merge in order to departition and repartition data from Multifiles accordingly.
- Worked with Partition Components like Partition-by-key, Partition-by-expression Partition-by-Round Robin to partition the data from serial file.
- Extracted data from HDFS file system and loaded into Oracle tables using Ab-Initio tool.
- Developing the ETL framework code using Spark and Scala to follow the Data Vault approach.
- Developing the CDC utility code using Spark, Hbase and Scala to capture the changed records and to insert new records as well.
- Developing the partition utility code that helps to run both the History Load and Daily load.
- Prepare data mapping of source and target entities using Trifacta based upon the metadata sheet.
- Writing Spark code using Scala and performing the data loads from semantic zone to workzone. Develop Scala UDFs that are common to each entity and invoking through Spark Dataframes.
- Handling Audit part to store the metadata information into Hbase
- Scheduling the jobs through Control M scheduler.
- Developed Hive scripts for end user / analyst requirements to perform ad hoc analysis
- Very good understanding of Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance
- Developed PL/SQL packages, Stored Procedures, Views, Functions and Triggers.
- Extensively worked on Oracle table partition management & Index management
- Involved in Unit Testing and Code reviews of the components to ensure defect free code delivery.
- Hands on control-M for scheduling the jobs, running and monitoring the jobs.
- Quick error handling in the code development.
- Assisted the end users in making them to understand the technical/business logics during the UAT testing.
- Raised ITSMs to migrate the code to UAT, production and supported activator teams during implementation.
- Worked on project RUNBOOKs and KT for production support team. Monitored production jobs until initial validation is completed and signoff provided by end users.
- Attended Agile stand up meeting for day to day status update
Environment: Ab-Initio, Oracle 11g, UNIX, Hortonworks Hadoop,HDFS,YARN,MapReduce,Sqoop,Hive,Pig, Oozie,Zookeeper,Flume,HBase,Spark,Scala,Control M, ITSM, SVN, GIT, Flat Files.
Confidential, Lewisville, TX
ETL & PL/SQL Developer
Responsibilities:
- Monitor Control M jobs (rerun, kill, order etc.)
- Act on Peregrine Tickets
- Act on Assigned story points
- Act on On Demand/ad hoc requests
- Respond to emails
- Act on JIRAs created by users
- Attending various calls (status call, MIS call, offshore handover call, ad hoc calls)
- JPMC s
- Analyse and identify the root causes on control M job failures
- Coordinate with offshore teams on daily activities and challenges.
- ITSM Task Implementation & Validation
- Planning month end & year end tasks
- Coordinate with upstream & downstream teams on failures and fixes
- Update Business Users on failures & delays
- Attend weekly status call with business users
- Conduct Knowledge Transfer sessions with end users on new reports built
- Work on weekly Incident analysis and Problem Management
- Work on improvements and automation of manual tasks
- Work on RCAs and Fixes
Environment: Oracle 11g, UNIX, Control M Scheduling Tool.
Confidential
PL/SQL Developer
Responsibilities:
- Preparing remediation steps and implementation for current Schema violations against Risk and Governance for JPMC-MIS
- Analysis and Development of current Static/Dynamic/Black-duck Scan breaks’ Remediation in MIS Schema code-base
- Analysis and Implementation of Control-M, UNIX, PL SQL migration of current Manual/Windows Business Reports generation.
- SQL Server to Oracle PL SQL migration - Project analysis, plan and development.
- Aligning the MIS-Business to IT practices like security, Account on boarding to EPV, implementing SVN and ITSM for regulatory code promotions, etc.
- Analyzing Requirement and preparing high/low level design documents and functional diagrams
- Creating tasks for above different project works as per timeline and distributing to team including myself.
- Develop automation scripts/programs using PL SQL, UNIX programming, wherever possible for manual reports and tasks.
- Ensure development of jobs’, programs is as per JPMC IT standards, and guidelines.
- Assigning tasks to offshore team and coordinate with them in successful completion of deliverables.
- Conducting Knowledge Transition sessions on the applications overview
- Track project status and enhancement requests, ensure process and product quality using TCS tool IPMS.
- TCS KMS tool used as a knowledge repository for the best practices.
- Attending the status meetings with Client and TCS managers to report the status on daily and weekly basis
Environment: Oracle 11g, UNIX, Control M Scheduling Tool
Confidential
Informatica Developer
Responsibilities:
- Requirement gathering and understanding the requirements.
- Keen attention to detail in the identification of potential glitches and threats to performance and security for new applications.
- Working with SOR files to dump the source data (.xml or .dat files) into the staging table (Oracle) by ensuring the successful intermediate jobs completion with required control and audit information.
- Mapping source data to target in informatica and using transformations like filter, expression, rank, router, aggregator, joiner, lookup and update strategy to get data organized as per client requirements.
- Developing workflows and tasks, monitoring sessions, exporting and importing mapping and workflow, taking backups of the code developed.
- For dumping the source data into staging tables jobs need to be created with the help of control-m, involved in creating and modifying the jobs and scheduling as per client requirements.
- Perform unit testing on the mappings and Control-M jobs.
- Analyse and prepare use cases to implement suitable partitions on stage and integration tables in Oracle.
- Create Oracle package to maintain table partitions include drop, truncate & exchange partitions
- Create and maintain all the stage and integration DDLs and check in the changes in SVN
- Participated in Daily Status meetings and providing detailed Status Reports
Environment: Informatica Power center 9.0, Teradata, UNIX, Control-M
Confidential
PL/SQL &Informatica Developer
Responsibilities:
- Identify client needs and develop customized solutions; this involves understanding primary objectives and implementing solutions.
- Requirement gathering and understanding the requirements.
- Analysis of Informatica Mappings to determine the transformation logic and source of the data into the tables of the affected DB Server.
- Analyzed the lineage for attributes both in informatica as well as in PL/SQL.
- Acknowledge analysis and resolving the issues.
- Analyzing Informatica mappings and PL/SQL codes and CAPTURING the end to end lineage of report attributes in a standard TEMPLATE. THIS IS THEN USED AS a for developing new code.
- Prepare use cases in order to develop a PLSQL parser in JAVA which is used to get lineage of attributes in expected format and coordinate with JAVA developer.
- Participated in Daily Status meetings and providing detailed Status Reports.
Environment: Informatica Powercenter 9.5, Oracle 11g.
Confidential
PL/SQL & Java Developer
Responsibilities:
- Requirement gathering and direct interaction with customer and Senior Management.
- Analysis, design, effort estimation and Implementation of Incident, Problem, Access & Capacity Management modules.
- Develop various Standard, Customized& Management reports as per ITIL-V3 standards.
- Conducting online demos with end users, customers and Production Support Groups.
- Production support for the modules developed and deployed so far.
- Integration of tool with various third party tools like IDM OEM, SCCM, KIWIKAT, SAP to extract user’s data and configuration of data.
- Develop E-mail functionality in all ITIL process areas.
- Code review.
- Sharing ad hoc reports as per management requirement
- Attending DCAB meetings.
Environment: Oracle11g Struts 1.1 Frameworks
Confidential
PL/SQL & Java Developer
Responsibilities:
- Requirement gathering and direct interaction with customer and Senior Management.
- Analysis, design, effort estimation and Implementation of business level and service level metrics.
- Develop various reports and dashboards.
- Conducting online demos with end users, customers and Production Support Groups.
- Production support for the modules developed and deployed so far.
- Develop escalation mail functionality.
- Code review.
- Sharing ad hoc reports as per management requirement.
- Attending client meetings.
Environment: Struts 1.1 Framework, Oracle 11g