We provide IT Staff Augmentation Services!

Sr.hadoop Administrator Resume

5.00/5 (Submit Your Rating)

SUMMARY:

  • 8 years of professional IT experience which includes about 4 years of proven experience in Hadoop Administration in deploying, maintaining, monitoring and upgrading Hadoop Clusters using Apache, Cloudera (CDH), Hortonworks (HDP) Distributions.
  • Hands on experience on configuring a Hadoop cluster for a production environment, Amazon Web Services (AWS) using an EC2 instance.
  • Experience in using Ambari, Cloudera Manager for installation and management of Hadoop Cluster.
  • Excellent understanding ofHadoopEcosystem including HDFS, MapReduce, Hive, Pig, Kafka, YARN, HBase, Oozie, ZooKeeper, Flume and Sqoop based Big Data Platforms.
  • Strong Knowledge in configuring NameNode High Availability.
  • Experience in managing Hadoop infrastructure like commissioning, log rotation, and rack topology implementations.
  • Experience working with, extending, and enhancing monitoring systems like Nagios.
  • Experience with setting up and deploying NoSQL database like HBase.
  • Experience in dealing with structured, semi - structured and unstructured data in Hadoop eco system.
  • Experience in a variety of Relational Databases like MySQL, MSSQL, and Oracle.
  • Expertise in Hive Query Language (HiveQL), Hive Security and debugging Hive issues.
  • Experience in using Chefas the Software Configuration Management tool for automating Hadoop infrastructure changes.
  • Adept with Linux environment setup, maintenance and troubleshooting
  • Experience in upgrading the existingHadoopcluster to latest releases.
  • Experience in configuring Zookeeper to provide Cluster coordination services.
  • Experience in developing Shell Scripts for system management.
  • Experience in providing security forHadoopCluster with Kerberos.
  • Working experience with Jira and fast-paced Agile / Scrum environment.
  • Effective problem solving skills and outstanding interpersonal skills. Ability to work independently as well as within a team environment. Driven to meet deadlines.
  • Motivated to produce robust, high-performance software.
  • Ability to learn and use new technologies quickly.

TECHNICAL SKILLS:

Methodologies: Agile, Rapid Application Development, Waterfall Model, And Iterative Model

Operating Systems: Windows 98/NT/XP/Vista/7, Linux (Ubuntu, CentOS, RedHat), MacOSX.

Hadoop Platforms: Hortonworks, Cloudera, Amazon AWS, Apache

Hadoop Ecosystem: HDFS, MapReduce, Pig, Hive, Sqoop, Oozie, Flume, Storm, Knox, HBase, Ambari, CDH, HDP

Languages: Java, C, Pig Latin, PHP.

Databases (RDBMS and NoSQL): MySQL, Oracle, Hbase, DB2.

Development Tools & IDEs: Maven, Composer, Git, Eclipse, Sublime Text, NetBeans

Web Technologies: JSP, REST API, HTML5, CSS, JavaScript, JQuery

JEE Technologies Servlets, Web Services,: Apache Tomcat

Frameworks: Hadoop, Laravel.

PROFESSIONAL EXPERIENCE

Confidential

Sr.Hadoop Administrator

Responsibilities:

  • Responsible for Administration, installing, managing and upgrading distributions of Hadoopwith Apache & Hortonworks cluster in different environments such as development, stage and production (Operational & Analytics) environments.
  • Responsible for cluster maintenance, commissioning and decommissioning of nodes cluster monitoring and troubleshooting.
  • Performed ambari and HDP upgrades
  • Install and setup Azkaban for developers to schedule jobs.
  • Implemented Capacity Schedulers to share the resources of the cluster for the MapReduce jobs given by the users
  • Set up Kafka for the proof of concept for carrying out log processing
  • Worked with Hortonworks support to resolve issues with services upgrade
  • Implement HiveServer and Metastore HA and migrated jobs to use zookeeper discovery
  • Enabled PAM authentication for Hive and migrated all jobs without downtime
  • Performed hive tuning for faster results and efficient usage of resources
  • Work with users (analysts and developers) to resolve issues related to jobs
  • Work with Linux team to carry out hardware maintenance
  • Set up elastic pipelines using beats, logstash, Kafka and elastic
  • Created dashboards using kibana
  • Installed, maintained and upgraded elasticsearch
  • Worked on setting up high availability for production cluster and designed automatic failover control using Zookeeper.
  • Responsible for managing file system check for blocks & directories onHadoopcluster to ensure no block are missing.
  • Monitored and analyzed the map reduces job executions on the cluster.
  • Analyzed the inputs to the development regarding the efficient utilization of resources like memory and CPU utilization.
  • Handled the upgrades and patch updates.
  • Monitored and controlled local file system disk space usage, local log files, cleaning log files with automated scripts.
  • Worked withHadoopdevelopers, designers in troubleshooting map reduce job failures and issues and helping to developers.
  • Provide in-depth technical and business knowledge to ensure efficient design, programming, implementation and on-going support for the application.

Confidential, Cincinnati, Ohio

Sr.Hadoop Administrator

Responsibilities:

  • Responsiblefor Administration, installing, managing and upgrading distributions of Hadoopwith Apache & Hortonworks (HDP)on aCluster of 400+ Nodes in different environments such as Development, Test and Production (Operational & Analytics) environments.
  • Worked with network and Linux system engineers/admin to define optimum network configurations, server hardware and operating system.
  • Administer and support distribution of Hortonworks.
  • Assisted in design of scalable Big Data clusters and solutions.
  • Design and Configure the Cluster with the services required (Hive server2, Kerberos, HDFS, Hue, Hive, Hbase, Zookeeper).
  • Deployed high availability on the Hadoop cluster using quorum journal nodes.
  • Troubleshoot the Hadoop cluster by reviewing Hadoop service log files to find the root cause.
  • Implemented automatic failover using zookeeper and zookeeper failover controller.
  • Implemented Rack Awareness for data locality optimization.
  • Involved in collecting and aggregating large amounts of streaming data and every node logs into HDFS using Flume.
  • Monitoring and controlling local file system disk space usage, log files, cleaning log files with automated scripts.
  • Implemented Capacity Scheduler to share the resources of the cluster for the MapReduce jobs given by the users.
  • Used Chef to manage OS dependencies and deploy third party packages.
  • Experience in setting up Sqoop workflows to migrate data to and fro from HDFS and RDBMS (MySQL, Teradata) to assist further OLAP analytics on the retrieved data via Hive.
  • Experience in setting up perimeter level security forall components of Hadoop Cluster using Kerberos with ADintegration.
  • Worked with Kafka for the proof of concept for carrying out log processing on distributed system.
  • Installed and configured Spark ecosystem components (SparkSQL, SparkStreaming, MLlib)
  • Involved in defining job flows using Oozie for scheduling jobs.
  • Conducting root cause analysis and resolve production problems and data issues.

Confidential, Houston, TX

Hadoop Administrator

Responsibilities:

  • Support and enhance the capabilities of an existing HDP cluster by adding nodes.
  • Worked with Unix and Networking team for racking and provisioning of bare metal servers.
  • Deployed Hortonworks Hadoop Ecosystem components such as Sqoop, Hbase and Mapreduce. Deployed 300+ node Hortonworks Hadoop Cluster (HDP 2.1) using Ambari 1.5.1.
  • Involved in Hadoop cluster environment administration that includes De-commissioning and commissioning nodes, balancing and cluster capacity planning.
  • Worked on High Availability for Name Node to avoid single point of failure.
  • Enabled High Availability for Resource Manager, Oozie, Hive Server 2, Hive Metastore on production clusters with minimal to zero impact on existing jobs and scheduled workflows
  • Participated in development and execution of system and disaster recovery process.
  • Performed an Ambari upgrades and HDP Stack upgrades in production environment from HDP 1.3 through HDP 2.1
  • Monitor workload, job performance, and capacity planning.
  • Monitor Hadoop cluster environments through Nagios.
  • Build and manage development and testing environments, assisting developers in debugging application issues.
  • Deployed and supported Hortonworks Hadoop cluster in AWS EC2 environment.
  • Configured Fair Scheduler to provide service-level agreements for various teams.
  • Enabled job queues and integrated it with Active Directory to control access on per user basis.
  • Working with Hortonworks support to resolve the issues.
  • Integrated Hadoop cluster with LDAP and active directory for security.
  • Implemented and automated HDFS snapshots for critical data.
  • Defining and documenting operational processes and monitoring and maintenance procedures
  • Monitoring the cluster on a daily basis, check the error logs and debug issues.
Confidential, Marysville, OH

Hadoop Administrator

Responsibilities:

  • Installed, Configured and maintained Apache Hadoop clusters for application development.
  • Worked on setting up high availability for production cluster and designed automatic failover control using Zookeeper.
  • Developed shell scripts to monitor health check up of Hadoop daemon services and respond accordingly.
  • Implemented rack aware topology on Hadoop Cluster.
  • Involved in database performance tuning and data modeling.
  • Responsible for managing file system check for blocks & directories onHadoopcluster to ensure no block are missing.
  • Responsible for cluster maintenance, commissioning and decommissioning of nodes cluster monitoring and troubleshooting.
  • Monitored and analyzed the map reduce job executions on the cluster.
  • Analyzed the inputs to the development regarding the efficient utilization of resources like memory and CPU utilization.
  • Handled the upgrades and patch updates.
  • Involved in loading data from UNIX file system to HDFS.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
  • Monitored and controlled local file system disk space usage, local log files, cleaning log files with automated scripts.
  • Involved in creating HIVE tables, loading with data and writing HIVE queries.
  • Worked withHadoopdevelopers, designers in troubleshooting map reduce job failures and issues and helping to developers.
  • Provide in-depth technical and business knowledge to ensure efficient design, programming, implementation and on-going support for the application.

Confidential

Linux System Administrator

Responsibilities:

  • Administration of RHEL, which includes installation, testing, tuning, upgrading and loading patches, troubleshooting server issues.
  • Coordinated with technical teams for installation of Hadoop and third party related applications.
  • Developed strategies for regular operations in order to improve availability and proper utilization.
  • Involved in troubleshooting problems and issues related to the efficient, secure operations of the Linux operating system.
  • Managed patching, monitoring system performance and network communication, backups, risk mitigation, troubleshooting, software upgrades and modifications of the Linux servers
  • Perform backup, file replications and script management servers.
  • Resolved DNS related issues.
  • Troubleshooting PC hardware and software.
  • Enabled remote access by constructing and managing Cisco VPN client.
  • Solving network issues with network applications.
  • Used Performance Scripts to troubleshoot the issues that arose during the backend disk failures.
  • Good Understanding of TCP/IP networking and its security considerations.
  • Responsible for scheduling the Cron Jobs.
  • Analyze the Performance of the Linux System to identify Memory, disk I/O and network problem.
  • Responsible for logs and Resource Monitoring.
  • Responsible for patch installations and updates on the servers

Confidential

Linux System Administrator

Responsibilities:

  • Provide system administration services like backup, monitoring, installation, configuration and user permissions.
  • Installed Cent OS using Pre-Execution environment boot and Kick-start automation method on multiple servers.
  • Schedule backup jobs and object level recovery using maxima tool.
  • Dealing with hardware failures by working with vendors.
  • Keeping systems up to date with latest stable releases of OS.
  • Creating new users, Resetting user passwords, Lock/Unlock user accounts
  • Installed and maintained software packages (.rpm) as required.
  • Monitoring System Metrics and logs for any problems.
  • Running crontab to back up data.
  • Involved in Adding, removing, or updatinguser accountinformation, resettingpasswords etc.
  • Took Backup at regular intervals and planned with a good disaster recovery plan.
  • Documented and maintain server, network, and support documentation including application diagrams.

We'd love your feedback!