We provide IT Staff Augmentation Services!

Hadoop System Administrator Resume

Charlotte, NC

SUMMARY:

  • Around 8 years of professional experience, 4+ years of Hadoop Administration and 4+ years as System Admin.
  • Experience in performing various major and minor Hadoop upgraded on large environments.
  • As an admin involved in Cluster maintenance, trouble shooting, Monitoring and followed proper backup& Recovery strategies.
  • Experienced in installation, configuration, supporting and monitoring 100+ node Hadoop cluster using Confidential manager and Horton works distributions.
  • Experience in HDFS data storage and support for running map - reduce jobs.
  • Experience in designing and implementing HDFS access controls, directory and file permissions user authorization that facilitates stable, secure access for multiple users in a large multi-tenant cluster
  • Experience in using Confidential Manager for Installation and management of Hadoop clusters.
  • Monitoring and support through Nagios and Ganglia
  • Experience in Chef, Puppet or related tools for configuration management.
  • Experience in working large environments and leading the infrastructure support and operations.
  • Installing and configuring Kafka cluster and monitoring the cluster using Nagios and Ganglia.
  • Load log data into HDFS using Flume, Kafka and performing ETL integrations Experience with ingesting data from RDBMS sources like - Oracle, SQL and Teradata into HDFS using Sqoop.
  • Experience in big data technologies: Hadoop HDFS, Map-reduce, Pig, Hive, Oozie, Sqoop, Zookeeper and NoSQL.
  • Experience in benchmarking, performing backup and disaster recovery of Name Nodemetadata and important sensitive data residing on cluster.
  • Migrating applications from existing systems like MySQL, oracle, db2 and Teradata to Hadoop.
  • Expertise with Hadoop, Mapreduces, Pig, Sqoop, Oozie, andHive.
  • Benchmarking Hadoopclusters to validate the hardware before and after installation to tweak the configurations to obtain better performance.
  • Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster.
  • Experience on Commissioning, Decommissioning, Balancing, and Managing Nodes and tuning server for optimal performance of the cluster.

TECHNICAL SKILLS:

Big Data Technologies: HDFS, Hive, Map Reduce, Cassandra, Pig, Hcatalog, Phoenix, Falcon, Scoop, Flume, Zookeeper, Mahout, Oozie, Avro, HBase, Kafka, MapReduce, HDFS, Storm, CDH 5.3, CDH 5.4, 5.5

Monitoring Tools: Confidential Manager, Ambari, Nagios, Ganglia.

Scripting Languages: Shell Scripting, Puppet, Scripting, Python, Bash, CSH, Ruby, PHP

Programming Languages: C, Java, SQL, and PL/SQL.

Front End Technologies: HTML, XHTML, XML.

Application Servers: Apache Tomcat, WebLogic Server, Web sphere

Databases: Oracle 11g, MySQL, MS SQL Server, IBM DB2.

NoSQL Databases: HBase, Cassandra, MongoDB

Operating Systems: Linux, UNIX, MAC, Windows NT / 98 /2000/ XP / Vista, Windows 7, Windows 8.

Networks: HTTP, HTTPS, FTP, UDP, TCP/TP, SNMP, SMTP.

PROFESSIONAL EXPERIENCE:

HADOOP SYSTEM ADMINISTRATOR

Confidential, Charlotte, NC

Responsibilities:

  • Working as Hadoop Admin and responsible for taking care of everything related to the clusters total of 90 nodes ranges from POC (Proof-of-Concept) to PROD clusters.
  • Working as admin on Confidential (CDH 5.5.2) distribution for clusters ranges from POC to PROD.
  • Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files.
  • Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues.
  • Adding/installation of new components and removal of them through Confidential Manager.
  • Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades.
  • Monitored workload, job performance and capacity planning using Confidential Manager.
  • Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
  • Interacting with Confidential support and log the issues in Confidential portal and fixing them as per the recommendations.
  • Used Kafka to load the data on Hadoop File system & move it to NoSQL
  • Exploring with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's, Spark YARN.
  • Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
  • Imported logs from web servers with Flume to ingest the data into HDFS.
  • Installing and configuring Kafka cluster and monitoring the cluster using Nagios and Ganglia.
  • Using Flume and Spool directory loading the data from local system to HDFS.
  • Experience in Chef, Puppet or related tools for configuration management.
  • Retrieved data from HDFS into relational databases with Sqoop.
  • Involved in extracting the data from various sources into Hadoop HDFS for processing.
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.
  • Creating collections and configurations, Register a Lily HBase Indexer configuration with the Lily HBase Indexer Service.
  • Creating and truncating HBase tables in hue and taking backup of submitter ID(s).
  • Configuring, Managing permissions for the users in hue.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: HDFS, Map Reduce, Hive 1.1.0, Hue 3.9.0, Pig, Flume, Oozie, Sqoop, CDH5, Apache Hadoop 2.6, Spark, SOLR, Storm, Knox, Zeppelin, Kafka, Confidential Manager, Red Hat, MySQL and Oracle.

HADOOP ADMINISTRATOR

Confidential, MN

Responsibilities:

  • Installed and configured a Horton Works HDP 2.2 and Hadoop 2.5 using AMBARI.
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Worked on installing cluster, commissioning & decommissioning of DataNode, NameNode recovery, capacity planning, and slots configuration.
  • Installed, Configured, Tested Datastax Enterprise Cassandra multi-node cluster which has 4 Datacenters and 5 nodes each.
  • Installed and configured Cassandra cluster and CQL on the cluster.
  • Loaded log data into HDFS using Flume, Kafka and performing ETL integrations.
  • Created HBase tables to store variable data formats of PII data coming from different portfolios.
  • Managing and reviewing Hadoop log files and debugging failed jobs.
  • Implemented Kerberos Security Authentication protocol for production cluster.
  • Implemented a script to transmit sysprin information from Oracle to HBase using Sqoop.
  • Implemented test scripts to support test driven development and continuous integration.
  • Worked on tuning the performancePig queries.
  • Responsible for adding new eco system components, like spark, storm, flume, Knox with required custom configurations based on the requirements.
  • Managed the design and implementation of data quality assurance and data governance processes.
  • Worked with Infrastructure teams to install operating system, Hadoop updates, patches, version upgrades as required.
  • Implemented Fair scheduler to allocate fair amount of resources to small jobs.
  • Assisted the BI team by Partitioning and querying the data in Hive.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.

Environment: Hadoop HDFS, MapReduce, Hortonworks, Hive, Pig Hive, Kafka, Oozie, Zeppelin, Flume Sqoop, HBase.

HADOOP ADMINISTRATOR

Confidential, Sacramento, CA

Responsibilities:

  • Capturing data from existing databases that provide SQL interfaces using Sqoop.
  • Processed information from HadoopHDFS. This information will comprise of various useful insights that can be used in the decision making process. All these insights will be presented to the users in the form of Charts.
  • Working on different Big Data technologies, good knowledge of Hadoop, Map-Reduce, Hive.
  • Developed various POCs over Hadoop, Big data.
  • Worked on deployments and automation task.
  • Installed and configured Hadoop cluster in pseudo and fully distributed mode environments.
  • Involved in developing the data loading and extraction processes for big data analysis.
  • Worked on professional services engagements to help customers design, build clusters, applications, troubleshoot network, disk and operating system related issues.
  • Worked with Puppet for automated deployments.
  • Installed and configured local Hadoop Cluster with 3 nodes and set up 4 nodes cluster on EC2 cloud.
  • Written MapReduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-HiveIntegration.
  • Work with HBase and Hive scripts to extract, transform and load the data into HBase and Hive.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports.

Environment: Hadoop2.4, 2.5.2, HDFS, Map Reduce, Hive, Flume, Sqoop, Confidential CDH4, HBase, Oozie, Pig, AWS EC2 cloud.

HADOOP SYSTEM ADMINISTRATOR

Confidential, CHICAGO, IL

Responsibilities:

  • Installed/Configured/Maintained Apache/ Confidential Hadoop clusters for application development and Hadoop tools like MapReduce, Hive, Pig, HBase, Zookeeper and Sqoop.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
  • Managing and Scheduling Jobs on a Hadoop cluster.
  • Implemented NameNode backup using NFS. This was done for High availability.
  • Involved in taking up the Backup, Recovery and Maintenance.
  • Shuffle algorithm, direct access to the disk, built-in compression, and code written in Java.
  • Worked on importing and exporting data from Oracle into HDFS and HIVE using Sqoop.
  • Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
  • Involved in commissioning and decommissioning at the time of node failure.
  • Implemented Partitioning and Bucketing concepts using Hive.
  • Involved in upgrading clusters to Confidential Distributed Clusters and deployed into CDH4.
  • Taken ad-hoc requests to get the non PII (Personally Identifiable Information) from the production cluster to non-production cluster to test out various scenarios

Environment: Hadoop, MapReduce, Hive, Pig, HBase, Sqoop, Flume, Zookeeper, Confidential Distributed Hadoop (CDH4), Confidential Manager.

LINUX/UNIX SYSTEM ADMINISTRATOR

Confidential

Responsibilities:

  • Installation, Configuration and Administration of Solaris 8/9/10, HP-UX 11.x, AIX 5.3, and Linux (RHEL 4/5, SLES 10)
  • Installed, Configured and Maintained Debian/RedHat Servers at multiple Data Centers.
  • Configured RedHat Kickstart server for installing multiple production servers.
  • Configuration and administration of DNS, LDAP, NFS, NIS and NISand Send mail on RedHat Linux/Debian Servers.
  • Hands on experience working with production servers at multiple data centers.
  • Involved in writing scripts to migrate consumer data from one production server to another production server over the network with the help of Bash and Perl scripting.
  • Installed and configured monitoring tools Munin and NagiOS for monitoring the network bandwidth and the hard drives status.
  • Automated server building using System Imager, PXE, Kickstart and Jumpstart.
  • Installed and configured DCHP server to give IP leases to production servers.
  • Password-less setup and agent-forwarding done for SSHS login using ssh-keygen tool.
  • Established and maintained network users, user environment, directories, and security.
  • Documented strongly the steps involved for data migration on production servers and also testing procedures before the migration.
  • Provided 24/7 on call support on Linux Production Servers. Responsible for maintaining security on Red Hat Linux.

Environment: RHEL 5.x/4.x, Solaris 8/9/10, Sun Fire, IBM blade servers, Web sphere 5.x/6.x, Apache 1.2/1.3/2.x, iPlanet, Oracle 11g/10g/9i, Logical Volume Manager, Veritas net backup 5.x/6.0, SAN Multipathing (MPIO, HDLM, Power path), VM ESX 3.x/2.x.

SYSTEM ADMINISTRATOR

Confidential

Responsibilities:

  • Involved in 24X7 Production support, Build and Migration Assignments.
  • Worked Primarily on RHEL 4/5, HPUX, and Solaris Operating Systems.
  • Text processing, also in network programming with Perl Scripting.
  • Involved in migration activities using Red Hat LVM, Solaris LVM, Veritas and EMC Open Migratory.
  • Installation of OAS (Oracle Application Server) on Solaris 9 and its configuration with oracle database.
  • Writing Shell and Perl Scripting for automation of job.
  • File system tuning, growing, and shrinking with Veritas File system 3.5/4.x.
  • Installed and configured GFS cluster for holding databases.
  • Manage user accounts for the team access for Red Hat Satellite Server.
  • Build channels and pull the packages from master Red Hat Satellite Server.
  • Troubleshooting hardware, software and configuration problems for various protocols and topologies.
  • Configured open LDAP Red Hat Linux systems.
  • Setup optimal RAID levels (fault tolerance) for protected data storage in NAS environments.
  • Install and configure DHCP, DNS (BIND, MS), web (Apache, IIS), mail (SMTP IMAP and POP3) and file servers.
  • Created new slices, mounted new file systems and uncounted file systems.

Environment: Red Hat Linux (RHEL 3/4/5), Solaris, Logical Volume Manager, Sun & Veritas Cluster Server, Global File System, Red Hat Cluster Servers.

Hire Now