Devops Hadoop Engineer Resume
Redwood City, CA
PROFESSIONAL SUMMARY:
- 5+ Years of Extensive experience in Unix/Linux Systems Administration and configuration Management.
- 3+ years experience in Hadoop eco system with the focus on Hadoop, Yarn, Hive, Sqoop
- Experienced in automating Hadoop Installation, Configuration, Monitoring, and Alerting.
- Experienced in performance tuning and debugging of Hadoop cluster.
- Experienced with Cloudera Manager and Hortonworks Ambari.
- Experienced in configuration and management of security for Hadoop cluster using Kerberos and integration with LDAP/AD at an Enterprise level.
- Troubleshooting and performance tuning of Hadoop clusters of clusters ranging upto 4000+ nodes.
- Strong understanding of Hadoop eco system such as HDFS, MapReduce, Zookeeper, Pig, Hadoop streaming, Sqoop, oozie and Hive.
- Experience in installing, administering, and supporting Linux operating systems and hardware in an enterprise environment
- Expertise in typical system administration and programming skills such as storage capacity management, performance tuning.
- Proficient in shell scripting and Python.
- Experienced with SAP Business Objects Enterprise XI, XIR2, XIR3.1 (Designer, Web Intelligence, InfoView, Desktop Intelligence, Import Wizard, Report Conversion Tool, Life Cycle Manager, CCM, CMS, Central Management Console, Live Office, QAAWS, Dashboard Manger, Xcelsius)
- Experiences in working with BI4 IDT (information Design Tool).
- Highly experienced in developing static and dynamic dashboards using Xcelsius 8.0 via the web service connectors like QAAWS, Live Office, XML.
- Highly experienced in Architecting and development of executive and regional level project performance dashboards.
TECHNICAL SKILLS:
Big Data / Hadoop: Hadoop, HDFS, YARN, MapReduce, Hive, sqoop, Pig, Tableau, oozie, ZooKeeper, Netezza
Business Intelligence Software: Business Objects XIR3.1/XIR2 Enterprise Suite (Central Management Console (CMC), Auditor, Import Wizard, Repository Migration Wizard, Universe Designer, Dashboard Manager, Desktop Intelligence, Report Conversion Tool, Live Office, QAWS), Business Objects 6.5.2 Suite (Supervisor, Designer, Business Objects, Web Intelligence), IBM Rational Clear - Case, Clear-Quest BMC Remedy Crystal Reports(9,10,XI,2008), Business Views, Xcelsius (4.5,2008)
Languages: Python, Shell Scripting, Java Scripting, php, C++
Config Management: Puppet, Ansible
Version Control Systems: git, svn
Database(s): Netezza, DB2, Oracle 10g/9i/8i, MS Access, SQL server 2000,2005
Data modeling: Erwin, Star Schema, Snowflakes Modeling
Operating System: UNIX/Linux, Windows Serves, VM Ware
WORK EXPERIENCE:
Confidential, Redwood City, CA
Devops Hadoop Engineer
Environment: CDH 5.4.10
Responsibilities:
- Automation of Cloudera Manager Installation using Puppet, Built 4 clusters with the power of automation.
- Provisioning, installing, configuring, monitoring, and maintaining HDFS, Yarn, Sqoop, Oozie, Pig, Hive
- Coded Tcollector / Python scripts to fetch timeseries metrics from Cloudera Manger and report them on WaveFront
- Providing architectural guidance, planning and estimating cluster capacity, and creating roadmaps for Hadoop cluster deployment and management
- Hardening and securing Hadoop clusters with Kerberos
- Configuration Management via puppet, using Cloudera Manager API to eliminate vulnerability.
- Performed automated installation through puppet for additional data nodes.
- Cluster Auditing system using cloudera Manager API integrated with Puppet
- Integrated PagerDuty with Cloudera Manager for Alerting purpose
Confidential, Cupertino CA
Big Data / Hadoop Admin
Environment: HDP 1.0, HDP 2.0, HDP 2.1, HDP 2.2 MapReduce, Yarn, oozie, Zookeeper, Hive, Sqoop, SAS, Tableau, R and Revolution R.
Responsibilities:
- Responsible for installation of Hadoop clusters on Linux servers and configurations of namenode, datanodes, appnodes, HDFS, MapReduce, YARN and networking components.
- Responsible for providing a Direct Priority 1 Critical support to Application team Users.
- Responsible for HDFS administration and cluster maintenance tasks such as creation and removal of nodes, cluster monitoring and troubleshooting.
- Responsible for installation of various hadoop ecosystem tools, such as Hive, YARN and Oozie.
- Write Unix Shell scripts for validation and cluster wide enhancements.
Confidential, Hartford, CT
Big Data / Hadoop Admin / BI Application Developer
Environment: HDP 1.0, HDP 2.0, HDP 2.1, MapReduce, Yarn, oozie, Zookeeper, Pig, Hive, Sqoop, Apache Flume, Hbase, SAS, Tableau, R and Revolution R.
Responsibilities:
- Built Hadoop clusters using Hadoop design principals, configured cluster connectivity and data security.
- Sqooped large scale of data from multiple different source systems such as zOS,, DB2, EDW, Oracle, SQL Server, and Netezza.
- Custom coded multiple shell scripts to automate bulk loading data from multiple sources into HDFS.
- Wrote shell scripts to automate the process to build multiple hive tables loaded into HDFS.
- Built in the shell script code to read hash portioning on db2 tables to partition the data in HDFS.
- Built Lab Hadoop cluster based on multiple virtual machines for multiple testing reasons.
- Provided day to day user support, data support, cluster and security support in Hadoop cluster.
- Provided some basic linux user training to Data Science and Machine Learning teams.
- Configured Business Objects Application server with Kerberos to connect securely to Hive metastore and read data from HDFS.
- Installed and configured Pentaho community edition from a potential open source BI reporting tool perspective in the future.
- Provide 24x7 on call support to Hadoop cluster.
- Installed and configured MarkLogic and integrated it with Hadoop cluster to source data and built reports.
- Multiple Business Objects full lifecycle implementations.
- Designed and developed multiple universes with various different types of data sources.
- SME on multiple universe hosted on both Shared-Reporting-Services and stand-alone environments
- Estimated cost for Designing, Developing, Unit-testing, QA/UAT Support, Deployment, training and post-production support efforts.
- Worked closer to Business team and BAs to gather Business requirements, Designing universe tech spec documentations during JAD sessions.
- Created Application Design Documents, Component Design Documents, Test Strategies, Unit and Integration Test Plans, Test Scripts and Technical Specification Documents, Implementation plans.
- Presented these documents to project stakeholders during WPI & WPR sessions (Work Product Inspections and Work Product Reviews).
- Implemented Single sign on (SSO) with Windows Authentication and Kerberos delegation.
- Worked with IBM Rational Clear-Case and Clear-Quest for document sharing and WPR review sessions
- Worked with BMC Remedy to create change requests for each deployment efforts
- Assisted in L1 meeting and provided Project Manager estimated completion dates on the whole or staggered business objects universe.
- Developed Universes’ development strategies based on their dependencies.
- Developed universe and Performed Unit and Integration tastings
Confidential, Cranford, New Jersey
Business Objects Administrator/Architect
Environment: Business Objects XIR2/XIR3.0 (Designer, Reporter, Central Management Console (CMC), Desktop Intelligence, Life Cycle Manager, Universe Designer), Web Intelligence, QAWS, Xcelsius 2008, Data Integrator XI, Oracle9i, PL/SQL, SQL Tools
Responsibilities:
- Reviewed the documentation provided by the client for the current Business Objects XIR2 environment.
- Created a feasibility reports address concerns and issues that might occur during the migration process.
- Deployed Business Objects XI 3.1 on 7 different environments (Development, System Test, Performance Test, UAT, Production, SLA, and Disaster Recovery).
- Configured Active Directory as the default authentication type for the environment.
- Deployed SSO to assist Windows-AD authentication for Infoview and CMC.
- With migration tool created BIAR file and migrated content to Development environment.
- Contacted stakeholders on individual basis (owner of the Universes and Reports) to discuss the security features they require for their Universes, Reports and Folders.
- Created various custom (user defined) security level BO XIR3.1 to meet the client security requirements.
- Created Users, Profiles, User groups and implemented inheritance based security for all users, groups, folders, categories, reports, Universes and for Publication.
Confidential, New York, NY
Business Objects Architect
Environment: Business Objects XIR2, Business Objects XIR2 (Designer, Reporter, Central Management Console (CMC), Desktop Intelligence, Import Wizard, Repository Migration Wizard, Universe Designer), Web Intelligence, Oracle 10.2g
Responsibilities:
- Helped client to migrate from legacy Business Objects 6.5 environment to Business Objects XIR2.
- Successfully deployed five Business Objects environment with LDAP with SSO.
- Achieved load balancing by deploying Business Objects servers on different physical machines and clustered them.
- Customized Business Objects as of the client requirement (Infoview pages, CMC pages etc).
- Provided a complete end to end solution right from gathering requirements to designing universes and creating the dashboards
- Participated in the design of data warehouse (fact tables and dimension tables)
- Developed new universes for reporting purposes in Designer XIR2.
- Created classes, filters, objects and subclasses in the universe.
- Defined profiles and access restriction at the universe level.
- Created Hierarchal Drill down Reports.
- Using Data Sources (Universes and Stored Procedures) for reports
- Developed reports with drillable Bar charts, Pie Charts, Line Charts to display differnt views of a data.
- Implemented cascading prompts feature in reports to control the data retirved by the report to control unneceaary overhead on the reporting system.
- Responsible to migarate Objects (Universe, Reports) to various cycles of the Enviroment (Dev, Test and Production)
Confidential, New York, NY
Business Objects Developer
Environment: Business Objects XI R2 (Designer, Reporter, Central Management Console (CMC), Desktop Intelligence, Import Wizard, Repository Migration Wizard, Universe Designer), Web Intelligence, QAWS, Xcelsius 2008, Oracle9i, PL/SQL, SQL Tools, Apache Tomcat, Windows 2003 Server
Responsibilities:
- Successfully migrated client’s current Business Objects 6.5 environment to Business Objects XIR2 environment.
- Developed complex Webi reports in XIR2 with combination of query level filters, report level filters and smart measures to have optimum performance of the report.
- Troubleshooter data issues by understanding the data model in Oracle 9i
- Developed and scheduled reports in Web Intelligence on recurring basis.
- Wrote a full functional specification document for the security implemented in Business Objects.
- Created complex Universes and report for GL, AP and AR.
- Customized Rapid Marts of GL, AP and AR.
- Resolved Loops using Aliases and Context, Addressed chasm trap and Fan Trap using Multi Parse SQL.
- Created Custom Hierarchies for several groups.
- Used Toad to verify data pulled by the report is correct compared to the database data and also to verify the joins used in designing the universe are correct.