Etl-hadoop Architect/developer Resume
Wilmington, DE
SUMMARY
- Fourteen years of software life cycle experience in ETL, Data Warehouse, Business Intelligence and Client Server applications
- Experience in Big Data, Hadoop technologies
- Worked on CDH, CDM, Hadoop environment (HDFS) setup, Map Reduce Jobs, HIVE, Hbase, PIG and NoSQL and MongoDB.
- Worked on Multi Clustered environment and setting up Cloudera Hadoop echo - System
TECHNICAL SKILLS
Data Warehousing: Informatica PowerMart 4.7/5.0/6.2/7.0, Informatica Power Center 4.1/5.0/5.1/6.2/7.0/8.1.6/9.01, ETL Informatica PowerConnect.
Hadoop Ecosystem Development Hadoop: HDFS, Sun Grid Engine Administration, HIVE, PIG, Flume, Oozie, Zookeeper, HBASE and Sqoop.
Databases: Oracle 7.x/8.x/9i, MS SQL Server 6.5/7.0/2000, MS Access, DB2 UDB, SybaseOLAP Tools: Business Objects Designer 5.1, BO Supervisor, BO Reports, COGNOS Powerplay, Powerplay Transformer, Impromptu administrator and Impromptu.
Data Modeling: ER/Studio 3.5/4.0, Erwin 3.5/4.0.
Languages Visual Basic: 6.0, Visual Studio .NET, ASP, HTML, XML, Java Script, PL/SQL.
CRM Siebel Tools: eScript/Siebel VB, eBusiness Application Integration (EAI), Actuate Reports, Workflow Manager, Assignment Manager, EIM and Smart Scripts.
Other Software: VB.Net, ASP.Net, Perl, Microsoft Internet Information Server (IIS) 5.0, Microsoft Transaction Server (MTS), TOAD, SQL Station, SQL Navigator, Relational clearest, Crystal Reports 6.x/7.x., MS Windows NT/2000/98/95/XP, MS Office Professional, Solaris, Unix, Linux, MS - DOS.
PROFESSIONAL EXPERIENCE
ETL-Hadoop Architect/Developer
Confidential, Wilmington, DE
Responsibilities:
- Created Java classes for AVRO file formats.
- Worked with Maven for code migration.
- Worked with GitHub to perform the version control.
- Worked with crucible for code reviews.
- Worked implementing third party libraries like Dozer, UtilBeans and Reflection.
- Used Eclipse to write Java and Mapreduce programs.
- Worked as ETL Architect to make sure all the applications are migrated (along with server) smoothly.
- Deep understanding and related experience with Hadoop stack - internals, HBase, Hive, Pig and Map/Reduce
- Deep understanding of schedulers, workload management, availability, scalability and distributed data platforms
- Expert knowledge developing and debugging in Java/J2EE
- Wrote Hive Queries and UDF s.
- Wrote MapReduce jobs.
- Implemented Fair schedulers on the Job tracker to share the resources of the Cluster for the Map Reduce jobs given by the users.
- Upgrading the Hadoop Cluster from CDH3 to CDH4 and setup High availability Cluster Integrate the HIVE with existing applications
- Automated all the jobs starting from pulling the Data from different Data Sources like MySQL to pushing the result set Data to Hadoop Distributed File System.
- Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
- Specifying the Cluster size, allocating Resource pool, Distribution of Hadoop by writing the specification texts in JSON File format.
- Configured Ethernet bonding for all Nodes to double the network bandwidth
- Automated all the jobs starting from pulling the Data from different Data Sources like MySQL to pushing the result set Data to Hadoop Distributed File System.
- Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
- Exported the result set from HIVE to MySQL using Shell scripts.
- Develop HIVE queries for the analysts.
- Helped the team to increase Cluster from 25 Nodes to 40 Nodes.
- Wrote Nagios plugins to monitor Hadoop Name Node Health status, number of Task trackers running, number of Data Nodes running.
- Maintain System integrity of all sub-components (primarily HDFS, MR, HBase, and Flume).
- Monitor System health and logs and respond accordingly to any warning or failure conditions.
Environment: Hadoop, HDFS, MapReduce, Hive, Pig, Eclipse, Java, AVRO, GitHub, Maven, Scoop, Oracle 9i/10g, SQL Server. MySQL, ER/Studio 3.5/4.0, UNIX Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0, SQL Sever Management Studio, MySQL WorkBench. Informatica PowerCenter 9.0.1, Data Quality 9.0.1, Netezza 4.x
Informatica Architect/Lead
Confidential, Chesterbrook, PA
Major Responsibilities:
- Prepared the check list to upgrade the existing informatica 8.1.6 to informatica 9.0.1.
- Upgrade the Informatica server and client tools.
- Manages and mentors a team of five ETL developers.
- Involves in making Architecture decisions and technical road maps.
- Establish coding standards, migration methodology and naming conventions.
- Communicates with business users to gather requirements.
- Preparing high and low level ETL design documents.
- Created NZLoad and NZSQL frameworks for Netezza.
- Work with the team to set up their goals and do the performance reviews.
- Talk to the external and internal vendors about the data feeds.
- Delivered project from beginning to end.
- Built ETL for calculate the TOT for Veeva and for other call data.
- Involve in decision making of the technical tools.
- Prepare project plan and assign tasks to the direct reports. Work with them to accomplish their tasks on time.
- Promoting informatica processes from Dev to QA and from QA to Production.
- Worked on identifying and setting up proper distribution keys.
- Good knowledge on Netezza architecture like Zone maps, distribution keys etc
- Created design documents for ETL mappings.
- Involved in gathering requirements.
- Coordinate with Oracle DBA and Unix Admin to achieve better dB performance and identify performance bottlenecks.
Environment: Informatica PowerCenter 9.0.1, Data Quality 9.0.1, Netezza 4.x, Oracle 9i/10g, SQL Server. MySQL, ER/Studio 3.5/4.0, UNIX Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0, SQL Sever Management Studio, MySQL WorkBench.
Informatica-Netezza ETL Architect/Lead
Confidential, Wilmington, DE
Responsibilities:
- Review the existing informatica mappings and create design documents to migrate them to Netezza.
- Converted the existing XPONENT process into ELT to achieve good performance. Due to converting to ETL the load time was reduced to 4 hours from 70 hours.
- Converted Oracle Materialized views into Netezza views.
- Worked on performance and tuning netezza queries.
- Worked on identifying and setting up proper distribution keys.
- Good knowledge on Netezza architecture like Zone maps, distribution keys etc
- Created design approach to lift and shift the existing mappings to Netezza.
- Created design documents to convert the existing mappings to use informaitca pushdown optimization.
- Analyze the impact on the downstream systems and recommend the solutions to keep them intact.
- Planning the Dev, SIT and QA environments.
- Involved in designing the D/W using Star Schema. Identifying the Fact, Dimension and slowly changing dimension tables.
- Taking ETL architecture decisions.
- Created mappings, WorkFlows/Worklets and scheduled them using workflow manager and UNIX.
- Create stored procedures in Netezza.
- Converted the existing Oracle materialized and relational views into Netezza views.
- Identify issues, debug and resolving issues.
- Developed reusable frameworks for DB constraints and NZLoad.
- Coordinate with Oracle DBA and Unix Admin to achieve better dB performance and identify performance bottlenecks.
- NZ Administrator Tasks:
- Creating and maintaining databases.
- Create users, user groups and assigning permissions.
- Implemented workload management on NZ.
- Assisting development team with performance and tuning.
- Use NZMigrate to load data from netezza server to the other server.
- Extensively used PG.log to see the activity on Netezza.
Environment: Informatica PowerCenter 8.1, ETL, MicroStrategy, Netezza 4.x, Oracle 9i/10g, ER/Studio 3.5/4.0, Unix Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0.
ETL Architect/Lead
Confidential, Baltimore, MD
Responsibilities:
- Involved in re designing and re writing the exiting War Room Reporting project.
- Managed a team of ETL developers and Database Administrators.
- Established the QA and DEV environments.
- Worked with Siebel databases and Netezza.
- Managed projects to move from BODI to Informatica.
- Hands - On experience with both BODI and Informatica.
- Installed and configured informatica 8.1 on Unix Servers.
- Scheduling work for project team activities and gathering requirements from the business users.
- Creates and maintains the overall and detailed project plan(s) and supervise the D/W ETL processes.
- Involved in designing the D/W using Star Schema. Identifying the Fact, Dimension and slowly changing dimension tables.
- Taking ETL architecture decisions.
- Involved in designing the ETL processes and writing he design documents.
- Created WorkFlows/Worklets and scheduled them using workflow manager and Unix.
- Wrote numerous Pre and Post session SQL and Unix Scripts.
- Coordinate with Oracle DBA and Unix Admin to achieve better dB performance and identify performance bottlenecks.
Environment: Informatica PowerCenter 8.1, ETL, Business Objects, Business Objects Data Integrator 11.5, Netezza 3.0, Oracle 8i/9i, ER/Studio 3.5/4.0, Unix Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0.
Technical/Team Lead
Confidential, Wilmington, DE
Major Responsibilities:
- Involved in creating design documents, process flow diagrams and technical specs.
- Managed a team of ETL developers and Database Administrators.
- Established the ETL infrastructure, ETL/DW standards and ETL naming conventions.
- Evaluating the latest versions and educating other developers with new changes.
- Scheduling work for project team activities and gathering requirements from the business users.
- Creates and maintains the overall and detailed project plan(s) and supervise the D/W ETL processes.
- Involved in designing the D/W using Star Schema. Identifying the Fact, Dimension and slowly changing dimension tables.
- Involved in the upgrade of information 5.x to 6.x
- Taking ETL architecture decisions.
- Involved in performing informatica Administrative tasks like creating login/passwords, taking backups, Registering repositories and restoring the repository.
- Involved in designing the ETL processes and writing he design documents.
- Created WorkFlows/Worklets and scheduled them using workflow manager and Unix.
- Wrote numerous Pre and Post session SQL and Unix Scripts.
- Coordinate with Oracle DBA and Unix Admin to achieve better dB performance and identify performance bottlenecks.
- Worked on Power Mart/PowerCenter client tools like Source Analyzer, Warehouse Designer, Mapping Designer, Repository Manager, Mapplet Designer and Transformations Developer.
Environment: Informatica PowerCenter 5.x/6.2/7.x/8.1, ETL, Cognos Impromtu administrator, Cognos Powerplay Transformer, Cognos Powerplay, Oracle 8i/9i, ER/Studio 3.5/4.0, Unix Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0
ETL Tech Lead
Confidential, West Point, PA
Responsibilities:
- Involved in designing Logical and physical databases for the staging and D/W using ERWIN.
- Involved in design, analysis, implementation and support of ETL processes.
- Prepared ETL standards, Naming conventions and wrote ETL flow documentation.
- Responsible for the code drops and migrating processes into the production.
- Involved in working with WorkFlow Manager, WorkFlow Monitor, Designer, Repository Manager, Source Analyzer, Warehouse Designer, Mapping Designer, Mapplet Designer and Transformations Developer.
- Involved in working with Business objects Supervisor, Report designer and Universe.
- Used Striva/Powerconnect to access DB2 tables and data.
- Created WorkFlows/Worklets and scheduled them using workflow manager and Unix.
- Created various Reusable and Non - Reusable tasks like Session, Assignment, Command, Control, Decision, Event and Timer.
- Wrote numerous Pre and Post session SQL and Unix Commands.
- Create workflows to partition data using Pipeline, Round-Robin, Hash, Key Range and Pass-through partitions.
- Integrated third party tools like Postal Certification (FirstLogic) and Vality (Integrity) with Informatica to validate addresses, zip codes and to dedup the addresses.
- Worked on PowerMart/PowerCenter client tools like Designed and developed various kind of maps using transformations like Expression, Aggregator, External Procedure, Stored Procedure, Look up, Filter, Joiner, Rank, Router, Update Strategy and XML.
- Wrote store procedures, functions, database triggers and Unix shell scripts to support and Automate the ETL process.
- Developing new Business Objects Universes and deploying to users worldwide.
- Identifying the objects and classes need to be created in the universe.
- Involved in describing, creating, building and maintaining the universe.
- Involved in resolving loops, fan traps and chasm traps by creating Aliases and contexts.
- Identifying and creating joins and cardinalities between tables.
- Developing new Business Objects Universes to users worldwide.
- Involved in performance and tuning the SQL queries using Explain plan.
Environment: Informatica Powermart 5.1/6.2, Informatica PowerCenter 5.x/6.2, ETL, Business Objects 5.1, Oracle 8i/9i, ER/Studio 3.5/4.0, Unix Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Windows NT 4.0.
Data Warehouse Consultant
Confidential
Major Responsibilities:
- Involved in Installation and Configuration of Informatica PowerCenter, PowerMart, Informatica Client, Informatica Server.
- Used Striva/Powerconnect to access DB2 tables and data.
- Installed informatica server on both windows and Unix platforms.
- Upgrading Informatica Power Center 1.7 to 5.0/5.1.
- Trouble shooted connectivity problems. Looked up and read session, event and error logs for troubleshooting.
- Taking Back ups and Restoring the Informatica Repository.
- Worked on PowerMart/PowerCenter client tools like Source Analyzer, Warehouse Designer, Mapping Designer, Mapplet Designer and Transformations Developer.
- Creating and managing Informatica Local and Global Repository.
- Identifying and loading sources and targets metadata into repository.
- Created, launched & scheduled sessions. Configured email notification.
- Setting up Batches and sessions to schedule the loads at required frequency using Power Center Server manager. Generated completion messages and status reports using Server manager.
- Extensively worked in the Performance Tuning of the programs, ETL Procedures and processes.
- Creating User Groups, Users, folders and assigning permissions.
- Extensively used ETL to load data from Oracle 8i, MS SQL Server 7.0 and flat files to Oracle 8i.
- Used most of the transformations such as the Source qualifier, Aggregators, lookups, Filters, Sequence and Update strategy.
- Extensively involved in UNIX shell scripting.
- Created tables, Functions, Packages, Triggers, User defined data types.
- Wrote PL/SQL Packages, Stored procedures, created triggers to implement business rules and validations.
- Developing new Business Objects Universes and deploying to users worldwide.
- Identifying Fact and Dimension tables to build indexes to improve performance.
- Performance monitoring and tuning of the queries.
- Created Universes for report generation using Designer module. As a part of Universe development, created Classes, different kind of objects, cardinalities.
- Resolved the loops using Aliases and contexts.
- Created Adhoc Reports using Business Objects. As a part of report development created the reports using universes as a main data providers and using the Powerful business objects functionality's like @ functions, user response and formulae's.
- Export the Universe and documents to the Repository.
Environment: Informatica Powermart 4.7/5.0, Informatica PowerCenter 1.7/5.x, ETL, Business Objects, Oracle 8i, Sybase 12.0, SQL Server 7.0, MS SQL Server OLAP Services, ER/Studio 3.5/4.0, Unix Shell Scripting, SQL, PL/SQL, TOAD, Transact-SQL, Crystal Reports 7/8.5, Windows NT 4.0.