We provide IT Staff Augmentation Services!

Hadoop System Administrator Resume

Hire Now

SUMMARY

  • Around 8 years of professional experience, 4+ years of Hadoop Administration and 4+ years as System Admin.
  • Experienced in installation, configuration, supporting and monitoring 100+ node Hadoop cluster using Cloudera manager and Horton works distributions.
  • Experience in performing various major and minor Hadoop upgraded on large environments.
  • As an admin involved in Cluster maintenance, trouble shooting, Monitoring and followed proper backup& Recovery strategies.
  • Experience in HDFS data storage and support for running map - reduce jobs.
  • Installing and configuring Kafka cluster and monitoring the cluster using Nagios and Ganglia.
  • Load log data into HDFS using Flume, Kafka and performing ETL integrations Experience with ingesting data from RDBMS sources like - Oracle, SQL and Teradata into HDFS using Sqoop.
  • Experience in big data technologies: Hadoop HDFS, Map-reduce, Pig, Hive, Oozie, Sqoop, Zookeeper and NoSQL.
  • Experience in benchmarking, performing backup and disaster recovery of Name Nodemetadata and important sensitive data residing on cluster.
  • Experience in designing and implementing HDFS access controls, directory and file permissions user authorization that facilitates stable, secure access for multiple users in a large multi-tenant cluster
  • Experience in using Cloudera Manager for Installation and management of Hadoop clusters.
  • Monitoring and support through Nagios and Ganglia
  • Experience in Chef, Puppet or related tools for configuration management.
  • Experience in working large environments and leading the infrastructure support and operations.
  • Migrating applications from existing systems like MySQL, oracle, db2 and Teradata to Hadoop.
  • Expertise with Hadoop, Mapreduces, Pig, Sqoop, Oozie, andHive.
  • Benchmarking Hadoopclusters to validate the hardware before and after installation to tweak the configurations to obtain better performance.
  • Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster.
  • Experience on Commissioning, Decommissioning, Balancing, and Managing Nodes and tuning server for optimal performance of the cluster.

TECHNICAL SKILLS

Big Data Technologies: HDFS, Hive, Map Reduce, Cassandra, Pig, Hcatalog, Phoenix, Falcon, Scoop, Flume, Zookeeper, Mahout, Oozie, Avro, HBase, Kafka, MapReduce, HDFS, Storm, CDH 5.3, CDH 5.4, 5.5

Monitoring Tools: Cloudera Manager, Ambari, Nagios, Ganglia.

Scripting Languages: Shell Scripting, Puppet, Scripting, Python, Bash, CSH, Ruby, PHP

Programming Languages: C, Java, SQL, and PL/SQL.

Front End Technologies: HTML, XHTML, XML.

Application Servers: Apache Tomcat, WebLogic Server, Web sphere

Databases: Oracle 11g, MySQL, MS SQL Server, IBM DB2.

NoSQL Databases: HBase, Cassandra, MongoDB

Operating Systems: Linux, UNIX, MAC, Windows NT / 98 /2000/ XP / Vista, Windows 7, Windows 8.

Networks: HTTP, HTTPS, FTP, UDP, TCP/TP, SNMP, SMTP.

PROFESSIONAL EXPERIENCE

HADOOP SYSTEM ADMINISTRATOR

Confidential

Responsibilities:

  • Working as Hadoop Admin and responsible for taking care of everything related to the clusters total of 90 nodes ranges from POC (Proof-of-Concept) to PROD clusters.
  • Working as admin on Cloudera (CDH 5.5.2) distribution for clusters ranges from POC to PROD.
  • Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files.
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues.
  • Adding/installation of new components and removal of them through Cloudera Manager.
  • Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades.
  • Monitored workload, job performance and capacity planning using Cloudera Manager.
  • Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
  • Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations.
  • Used Kafka to load the data on Hadoop File system & move it to NoSQL
  • Exploring with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's, Spark YARN.
  • Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
  • Imported logs from web servers with Flume to ingest the data into HDFS.
  • Installing and configuring Kafka cluster and monitoring the cluster using Nagios and Ganglia.
  • Using Flume and Spool directory loading the data from local system to HDFS.
  • Experience in Chef, Puppet or related tools for configuration management.
  • Retrieved data from HDFS into relational databases with Sqoop.
  • Involved in extracting the data from various sources into Hadoop HDFS for processing.
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.
  • Creating collections and configurations, Register a Lily HBase Indexer configuration with the Lily HBase Indexer Service.
  • Creating and truncating HBase tables in hue and taking backup of submitter ID(s).
  • Configuring, Managing permissions for the users in hue.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: HDFS, Map Reduce, Hive 1.1.0, Hue 3.9.0, Pig, Flume, Oozie, Sqoop, CDH5, Apache Hadoop 2.6, Spark, SOLR, Storm, Knox, Zeppelin, Kafka, Cloudera Manager, Red Hat, MySQL and Oracle.

HADOOP ADMINISTRATOR

Confidential - PLEASANTON, CA

Responsibilities:

  • Installed and configured a Horton Works HDP 2.2 and Hadoop 2.5 using AMBARI.
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Worked on installing cluster, commissioning & decommissioning of DataNode, NameNode recovery, capacity planning, and slots configuration.
  • Installed, Configured, Tested Datastax Enterprise Cassandra multi-node cluster which has 4 Datacenters and 5 nodes each.
  • Installed and configured Cassandra cluster and CQL on the cluster.
  • Loaded log data into HDFS using Flume, Kafka and performing ETL integrations.
  • Created HBase tables to store variable data formats of PII data coming from different portfolios.
  • Managing and reviewing Hadoop log files and debugging failed jobs.
  • Implemented Kerberos Security Authentication protocol for production cluster.
  • Implemented a script to transmit sysprin information from Oracle to HBase using Sqoop.
  • Implemented test scripts to support test driven development and continuous integration.
  • Worked on tuning the performancePig queries.
  • Responsible for adding new eco system components, like spark, storm, flume, Knox with required custom configurations based on the requirements.
  • Managed the design and implementation of data quality assurance and data governance processes.
  • Worked with Infrastructure teams to install operating system, Hadoop updates, patches, version upgrades as required.
  • Implemented Fair scheduler to allocate fair amount of resources to small jobs.
  • Assisted the BI team by Partitioning and querying the data in Hive.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.

Environment: Hadoop HDFS, MapReduce, Hortonworks, Hive, Pig Hive, Kafka, Oozie, Zeppelin, Flume Sqoop, HBase.

HADOOP ADMINISTRATOR

Confidential

Responsibilities:

  • Capturing data from existing databases that provide SQL interfaces using Sqoop.
  • Processed information from HadoopHDFS. This information will comprise of various useful insights that can be used in the decision making process. All these insights will be presented to the users in the form of Charts.
  • Working on different Big Data technologies, good knowledge of Hadoop, Map-Reduce, Hive.
  • Developed various POCs over Hadoop, Big data.
  • Worked on deployments and automation task.
  • Installed and configured Hadoop cluster in pseudo and fully distributed mode environments.
  • Involved in developing the data loading and extraction processes for big data analysis.
  • Worked on professional services engagements to help customers design, build clusters, applications, troubleshoot network, disk and operating system related issues.
  • Worked with Puppet for automated deployments.
  • Installed and configured local Hadoop Cluster with 3 nodes and set up 4 nodes cluster on EC2 cloud.
  • Written MapReduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-HiveIntegration.
  • Work with HBase and Hive scripts to extract, transform and load the data into HBase and Hive.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports.

Environment: Hadoop2.4, 2.5.2, HDFS, Map Reduce, Hive, Flume, Sqoop, Cloudera CDH4, HBase, Oozie, Pig, AWS EC2 cloud.

Hire Now
Client Services
Job Seekers
Visa Sponsorship