Big Data Engineering Architect Resume
Irving, TX
SUMMARY:
- 13 years of rich experience in IT Industry in Leading, Architecting, Integrating and driving proof of concepts on upcoming cutting - edge technology in big data space.
- Establish advanced application monitoring solutions for the ecosystem.
- Operationalize developer diagnostic tools to provide deeper insights into application code execution and impact to platform resources
- Operationalize Hadoop cluster virtualization and containerization.
- Operationalize hybrid cloud integration - Q3/Q4 Initiatives
- Extensive Experience in leading and managing Application Support and Maintenance Projects in Distributed computing world
- Excellent Client interaction & relationship management skills, Strong communication and interpersonal skills
- Experience in ability to build flexibility in team by cross skilling, rotation and knowledge retention and motivating team in the long haul
- Performed a good blend of offshore and onsite based customer facing roles in a similar or equivalent capacity
- 9 Years of extensive experience of working based on Onsite/Offshore Model at Client Place
- Conduct phase wise evaluation of the projects; report the progress to senior Directors and update organization database/knowledge repository
- 4 years of experience as Hadoop Admin and Architecture
- 4 years of experience in Cloudera Distribution and framework for Data ingestion
- 13 years of extensive experience in Ab-Initio Data Warehousing ETL Tool
- 13 years of extensive experience in AbInitio GDE 3.0.3, 1.15, 1.14, 1.13 and Co>Operating System 3.0.4, 2.15, 2.14
- 13 years of experience in Batch Data Processing and 2 years on Continuous flows and 3 years on AbInitio Admin activities
- 13 years of experience in UNIX Shell Scripting
- Experience in Conduct>IT to Schedule AbInitio Plans
- Experience in SSIS, PLSQL, Talend - Support Activities
- Experience in databases - MS SQL Server, Oracle, DB2
- Experience in HP Quality Center, IBM ClearQuest for defect management
- Experience in Service Now - expert in Change Management Process
- Experience in Agile Methodology of project implementation
TECHNICAL SKILLS:
Big Data Tool Stack: Cloudera, Spark, Hive, YARN, Hadoop, Oozie Kafka, Confluent
Cloud Infrastructure: Amazon Cloud (AWS), Google Cloud Platform
Docker/Containerization: Bluedata
Big Data Enhancing Tools: Pepperdata, Unravel
Scripting Languages: Shell scripting
Databases : DB2 7.2.0, Oracle 11g, MS-SQL Server
Operating Systems : MS-DOS, IBM AIX, Windows
Problem Management: ManageNow, Unicenter ServicePlus, HP Service Management, Service Now
Scheduling tools: Control-M, IBM Tivoli Scheduler, Conduct>IT
ETL Tools : Ab Initio (GDE 3 .2.2 Co>OS 3.2.5 ), SSIS, Talend, Informatica, Datastage
PROFESSIONAL EXPERIENCE:
Big Data Engineering Architect, Irving, TX
Confidential
Technologies/Tools Used: Cloudera, Spark, Hive, AbInitio, Kafka, YARN, Oozie, Hadoop, Unix Shell Scripts, Autosys, Service Now, HP Quality Center, Bluedata, Pepperdata, Unravel, Amazon Cloud (AWS), Google Cloud Platform
Responsibilities:
- Research and development of the new products in the space of Big data performance management (Example: Pepperdata and Unravel Data) and derive the pros and cons on a Big data cluster of such products in Lab environment. Providing periodic updates to Confidential Directors on the Proof of concepts and helping with decision making on value add that it would bring to Confidential enterprise.
- Work on proof of concept with Hybrid Cloud Integration - AWS and Google being the prime vendors, very early stage of building a strategy for cloud migration.
- Hadoop cluster virtualization and containerization using Bluedata - Proof of concept from past 8 months, been driving this effort extensively to reduce the Big Data Infra costs by using this product and implement the concept of BigData as a Service
- Facilitating the Hadoop Infrastructure for various internal clients within Confidential from North America, Asia, Europe, Mid East Asia and Latin America
- Analyze the requirements for new Hadoop cluster demand for North America region from various applications area’s
- Brainstorm and develop solutions for new cluster demand in consultation with Confidential Architects and Enterprise Platform Support and Sys Admin teams
- Work with Data Scientists and help them with providing a required infrastructure or a sandbox environment to run their high compute bound algorithms which were mostly based our H20, Arcadia or R
- Automate AD integration and suggest idea’s on security access.
- Proposed strategy on Sentry Roles and defined architecture on how the access should be defined for various DB’s in different layers of HDFS
- Collaborate with various application teams within Confidential as Ab Initio, MR ad Spark SME
- Provide Infrastructure to the Ab Initio teams to integrate with Cloudera and help in addressing the integration issues
- Working with various applications teams on addressing their issues with the data ingestion needs through POC’s on lab environments
- Work with client architecture team to architect and build scalable functioning prototypes/tools involving Ab Initio, Cloudera Distribution, Hive and Spark
- Secure necessary hardware from procurement and Unix teams and work on setting up the environments for various SDLC like Development, SIT, UAT and Production
- Address integration issues that may come up while running the applications on newly set Hadoop environment like Kerberos Authentication, beeline issues, hue access etc..
- Create Change orders using Service Now Full Suite on a every weekly basis to support various scheduled implementations
- Bi-weekly meeting with the Cloudera and Ab Initio vendor teams to address the ongoing issues with the Confidential Integration
- Facilitate Hadoop Infrastructure for various application area’s within Citigroup and providing continuous support across the SDLC
- Suggest idea and brainstorm with the team on keeping the Hadoop Cluster Healthy and high availability is maintained throughout serving all the customers in an efficient manner
- Pre and Post implementation support for the newly designed Hadoop clusters
- Assist Middleware team with issues that arise during the Cloudera (CDH) version upgrade
Confidential
ETL Lead Consultant, Chicago, IL
Technologies/Tools Used: AbInitio, SSIS, Unix Shell Scripts, Oracle, TWS, Service Now, HP Quality Center, Hadoop, Cloudera
Responsibilities:
- Assist the team in design, integration methodology/tool selection, migration activities and address data quality issues in AbInitio world.
- Meet with Architect group on a weekly basis to review the proposed architecture for various projects in pipeline and suggest changes wherever required.
- Assist Production Support in identifying the issues and helping fix them during Rapid Response Call.
- Track the progress with the Design, development, testing and deployment of ETL packages and coordinate with debugging issues if any during the project lifecycle.
- Cloudera Distribution Administration activities and upgrade.
- POC of migrating AbInitio code base to Hadoop eco system.
- Documenting process and procedures and provide training if any required.
- Subject Matter and technical expert in ETL space.
- Managing and leading a team of 10 resources, 2 at onshore and 8 from offshore.
- Working with Confidential consulting team on providing RFE’s for new proposal on BI/DW area across multiple horizon within Confidential .
- Handled UNIX to LINUX migration and Abinitio 3.0.3 upgrade from Abinitio 1.13 and 2.14 version. Almost 1800 graphs and 3000 plus scripts are being upgraded with the entire project implementation planned for May 2017. Tasks include setting up AbInitio environment on new Linux server and assist various teams in fixing the issues they face through service now tickets. Also involve in setting up TWS Jobs on Linux server through a Scheduling requests on Service Now.
- Trained in Agile Methodology of project implementation and working on a Pilot project of Agile Project development where a current Sprint run for 2~3 weeks duration.
Confidential
AbInitio Technical Lead
Technologies Used: AbInitio, Informatica, Talend, Unix Shell Scripts, PLSQL, Oracle, Conduct>IT, Hadoop
Responsibilities:
- Worked on the Knowledge Transition Plan and setting up meetings with various stake holders/vendor teams and grooming an In-house ETL team
- Forecast the pipeline projects and align TCS resources for timely delivery
- Involve various stake holders during the weekly calls and assist manger with resolving any conflicts between teams
- Involved in designing, developing and implementing AbInitio graphs for Breakfix items through standard SDLC process
- Ensure SDLC process and Change Management Process is followed for all Deployments
- Align Vendor resources for various tracks of the projects and forecast the effort.
- Worked with architecture team on early stages of understanding Hadoop and its implementation strategy at an enterprise scale
Confidential
AbInitio Technical Lead, Chicago
Technologies Used: AbInitio, SSIS, Oracle Database, Unix Scripting
Responsibilities:
- Manage expectations with internal and external stakeholders of the project
- Gather requirements by interacting with various business areas for an enterprise driven project
- Create sample set of data in Preprod environment and provide the file to Business Users and finalize the requirements based on the preprod file
- Work closely with the Architects in creating the conceptual design and reviewing it with the internal AbInitio team
- Create the design and UTP document and review it with the AbInitio team
- Coordinate with offshore on the Build work and Unit testing and reviews.
- Provide QA test support to Testers
- Create all documentation to fulfill mandatory SOX compliance process before moving the changes to production
- Create HPSM Change order and follow the process of moving it to Coordinate and Implement phase
- Work with the Build Operation Team to migrate the code to all the environments
- Work with Scheduling team in setting up the jobs in Tivoli Workload Scheduler (TWS - IBM product)
- Provide Support for first 3 months, before handing over the project to Production Support
- Involved in providing 24/7 production support to the clients.
- To accept the job abend tickets on time and resolving it within the specified time so as to keep the successor jobs unaffected and stick to the SLAs successfully.
- Co mmunication with Business to resolve the tickets raised by the Business
- Worked on enhancements of systems and change of code for the changing requirements from the client side.
- Developing the project plan, ensuring review and sign-offs with the required internal and stakeholders
- Work allocation across the different modules of the project and monitoring and tracking progress of the project
- Managing project risks including mitigation and contingency plans
- Adopting technical and quality strategy
- Quality of deliverables and for adopting and following defined project gating process and any defined assurance roles
- Maintenance and tracking of the various project metrics
- Report on a regular basis the project status to internal and external stakeholders
- Conduct Phase wise evaluation of projects and at the end of the project report to Senior Leaders
- Engage defined escalation and communication layers to ensure conflicts, risks and challenges are handled appropriately
- Establish and provide an conducive atmosphere for the project team to engage and deliver
