- Over 8+ years of administration experience including 4+years of Hadoop Ecosystem in installation and configuration of different Hadoop eco - system components in existing cluster.
- Strong Knowledge in Hadoop HDFS architecture and Map Reduce framework.
- Extensive hands on administration with Cloudera Hadoop.
- Extensive hands on administration with Hortonworks.
- Hands on experiences and thorough understanding of UNIX/Linux operating systems.
- Experienced in deploying and managing the multi-node development, testing and production Hadoop cluster with different Hadoop components (HIVE, PIG, SQOOP, OOZIE, HBASE, ZOO KEEPER).
- Hands on experience with production Hadoop applications such as administration, configuration management, performance tuning and debugging.
- Experience in understanding the security requirements for Hadoop and integrating with Kerberos authentication infrastructure.
- Experience in managing and scheduling batch jobs on Hadoop cluster.
- Worked with application team via scrum to provide operational support, install Hadoop updates, patches and version upgrades as required.
- Experienced in monitoring Hadoop cluster using tools like Nagios, Ganglia.
- Experienced in importing and exporting data from different databases like Mysql, Oracle into HDFS and Hive using sqoop.
- Experience in benchmarking, performing backup and recovery of Namenode metadata.
- Strong Knowledge in Name Node High availability and Namenode Federation.
- Experience in commissioning and decommissioning of data nodes on Hadoop Cluster.
- Built ingestion framework using flume for streaming logs and aggregating the data into HDFS.
- Installed, configured and maintained HBASE using Hbasemaster and Hbase regional server.
- Experience in configuring Zookeeper to provide high availability and Cluster services coordination.
- Experienced in Linux administration tasks like IP management (IP addressing, Sub netting, Ethernet Bonding and static IP)
- Experienced in Software Development Life Cycle(SDLC), application design, functional and technical Specs, and use case development using UML.
- Experience in installing and maintaining all server hardware and software systems and administer all server performance and ensure availability.
- Experience in designing and maintaining all system tools for all the scripts and automation processes and monitor all capacity planning.
Hadoop Ecosystem: HDFS, MapReduce, Yarn, zoo keeper, Sqoop, Flume, Pig, Hive, Oozie. Kafka, Ranger, Cloudera Manager, Hortonworks Ambari, Apache Hadoop
Programming Languages: Sql, PL/SQL, Unix Shell Scripting
Web Servers: Apache Tomcat, Apache http Web Server
Database Technologies: MySQL, SQL server, Oracle
Monitoring Tools: Ganglia, Nagios
Operating systems: CentOs, Red hat Linux, Debian, Fedora, Windows
Confidential, Phoenix, AZ
- Responsible for implementation and ongoing administration of Big Data platforms on Hortonworks
- Installed and configured HDP 2.2 with ambari manager.
- Responsible for architecting Hadoop clusters translation of functional and technical requirements into detailed architecture and design.
- Installed and configured Hadoop components on multi-nodes fully distributed Hadoop clusters of large number of nodes.
- Involved in cluster capacity planning, Hardware planning, Installation, Performance Tuning of the Hadoop cluster.
- Responsible for addressing and troubleshooting issues on daily basis.
- Responsible for managing and scheduling jobs on a Hadoop Cluster.
- Implemented commissioning and decommissioning of datanodes, killing the unresponsive tasks tracker and handled with blacklist task trackers.
- Worked with data delivery teams to setup new Hadoop users and responsible for setting up Linux users, setting up Kerberos principles and testing HDFS, Hive.
- Worked on importing and exporting data into HDFS and HIVE using Sqoop.
- Used Ganglia and Nagios to monitor the cluster all the time.
- Created a local YUM repository for installing and updating packages.
- Monitored Hadoop cluster connectivity and security.
- Involved in regular checkup of the system using Hadoop metrics.
- Involved in moving the data from one cluster to another cluster by using DISTCP and automated the dumping procedure using shell scripts.
- Worked with Hortonworks support to resolve the issues
- Responsible for assigning archival storage to Datanodes and worked on sstorage policies to assign files.
- Designed and allocated HDFS quotas for multiple groups.
- Performed various configurations like networking, resolving hostnames, user accounts, FTP, SSH keyless login.
Confidential, Miami, FL
- Installed, Configured and maintained Apache Hadoop clusters for application development.
- Worked on setting up high availability for production cluster and designed automatic failover control using Zookeeper.
- Developed shell scripts to monitor health check up of Hadoop daemon services and respond accordingly.
- Implemented rack aware topology on Hadoop Cluster.
- Involved in database performance tuning and data modeling.
- Responsible for managing file system check for blocks & directories on Hadoop cluster to ensure no block are missing.
- Responsible for cluster maintenance, adding and removal of cluster nodes, cluster monitoring and troubleshooting.
- Configured the cluster properties to gain the high cluster performance by taking cluster hardware configuration as criteria
- Monitored and analyzed the map reduce job executions on the cluster.
- Analyzed the inputs to the development regarding the efficient utilization of resources like memory and CPU utilization.
- Handled the upgrades and patch updates.
- Involved in loading data from UNIX file system to HDFS.
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Monitored and controlled local file system disk space usage, local log files, cleaning log files with automated scripts.
- Involved in creating HIVE tables, loading with data and writing HIVE queries.
- Work with Hadoop developers, designers in troubleshooting map reduce job failures and issues and helping to developers.
- Provide in-depth technical and business knowledge to ensure efficient design, programming, implementation and on-going support for the application.
Confidential, Rockville, MD
- Experienced in administration, installing, configuring and managing distributions of Hadoop cluster with cloudera manager CDH3 and CDH4.
- Involved in hardware planning and performance tuning.
- Continuous monitoring and managing the HADOOP cluster through cloudera manager.
- Good experience in troubleshooting issues in cluster and its functionality.
- Involved in collecting and aggregating large amounts of data and every node logs into HDFS using Flume.
- Supporting Hadoop developers and assisting in optimizations of Map-reduce jobs and Hive scripts.
- Managing nodes on Hadoop cluster connectivity and security using Kerberos.
- Responsible for the data coming from multiple sources and backing up on daily basis to another cluster using distcp.
- Experience in upgrading Hadoop cluster from current version to latest version upgrades .
- Demonstrated and understanding the concepts, best practices and functions to implement a Big Data solution.
- Experienced in writing the automated scripts and setting up the cron jobs to maintain cluster stability.
- Worked with network and Linux system engineers to define network configurations, server hardware and operating system.
- Responsible for cluster maintenance and on call support on weekly rotation.
- Applied patches and bug fixes on Hadoop Clusters.
- Extensive hands on experience in Hadoop file system commands for file handling operations.
- Worked on Providing User support and application support on Hadoop Infrastructure.
- Contributed to the creation and maintenance of system documentation
- Designed and implemented custom writable, custom input formats, custom partitions and custom comparators.
- Responsible for regular maintenance of Commissioned/decommission nodes as disk failures occur using Cloudera Manager.
- Worked on file system management and monitoring and Capacity planning.
- Experience in setting up and managing the batch scheduler Oozie.
Confidential, Jacksonville, FL
- Administration of RHEL which includes installation, testing, tuning, upgrading and loading patches, troubleshooting server issues.
- Implemented and administered VMware ESX 5.5 for windows servers.
- Coordinated with technical teams for installation of Hadoop and third party related applications.
- Developed strategies for regular operations inorder to improve availability and proper utilization.
- Involved in troubleshooting problems and issues related to the efficient, secure operations of the Linux operating system.
- Managed patching, monitoring system performance and network communication, backups, risk mitigation, troubleshooting, software upgrades and modifications of the Linux servers
- Perform backup, file replications and script management servers.
- Resolved DNS related issues.
- Troubleshooting PC hardware and software.
- Enabled remote access by constructing and managing Cisco VPN client.
- Solving network issues with network applications.
- Used Performance Scripts to troubleshoot the issues that arose during the backend disk failures.
- Good Understanding of TCP/IP networking and its security considerations.
- Responsible for scheduling the Cron Jobs.
- Analyze the Performance of the Linux System to identify Memory, disk I/O and network problem.
- Responsible for logs and Resource Monitoring.
- Responsible for patch installations and updates on the servers.
Confidential, San Diego, CA
- Administered Linux servers including Websphere, Apache Tomcat server, mail server, MySQL database in both development and production.
- Involved in server optimizations and identifying, recommending optimal server hardware based on user requirements.
- Managing system routine backups, scheduling jobs, enabling system logs, network logging of servers for maintenance, performance tuning, testing.
- Troubleshooting security relates issues, IP tables, firewalls, TCP wrappers.
- Installation of packages, patch management and volume management.
- Remote system administration like SSH, Telnet and Rlogin.
- Assisted in configuring the application server with other components such as database, LDAP directory, Active directory.
- Adding servers to the domain and managing the groups and user in AD, installing and configuring send mail.
- Worked with other teams in helping build and implement systems and standard.
- Involved in development, user acceptance, and performance testing and disaster recovery server.
- Performed capacity analysis, monitored and controlled disk space usage on system.
- Organized various meetings with different teams and designed the plan and took user approvals.
- Responsible for repairs and recoveries from hardware or software failures. Coordinated and communicated with impacted application team.
- Performed regular file archival and purge as necessary.
- Performed daily system monitoring, verifying the integrity and availability of all hardware, server resources, systems and key processes, reviewing system and application logs, and verifying completion of scheduled jobs such as backups.
- Build, configure and install Red Hat 5/6, OS on physical and virtual servers.
- Troubleshoot, install, configuration, performance, monitoring and other process issues of the servers.
- Implemented and administered VMware ESX 5.5 for running the windows servers.
- Performed configured of standard Linux and network protocols such as SMTP, SNMP, HTTP, DHCP, DNS, LDAP, NFS.
- Maintained software packages on servers using repository servers.
- Monitored hardware consoles for any hardware issues and troubleshoot any operating system issues.
- Managed Job schedules, Cron jobs, and batch processing.
- Responsible for monitoring system activities like CPU, Memory, Disk and swap space usage to avoid performance issues.
- Experience in troubleshooting and resolving network issues related to server.
- Experience in creating VM for Linux windows and Solaris.
- Managed and troubleshoot AD user login problems.
- Involved in creating change controls, resolving incident tickets and ability to multitask.
- Monitor daily backup activity and generate reports to avoid issues.
- Improve automation and productivity of backups through shell scripting.
- Performed periodic performance reporting to support capacity planning.
- Responsible for providing system health check and trending reports.