We provide IT Staff Augmentation Services!

Hadoop Administrator Resume

Charlotte, NC

SUMMARY

  • Over 9 Years of extensive IT experience with 5+ years of experience as a Hadoop Administrator and around 4 years of experience in UNIX/Linux Administrator along with SQL developer in designing and implementing Relational Database model as per business needs in different domains.
  • Experience in installation, management, and monitoring of the Hadoop cluster using Cloudera Manager.
  • Optimized the configurations of Map Reduce, Pig and hive jobs for better performance.
  • Backup configuration and Recovery from a Namenode failure.
  • Expert level skills in Managing and Scheduling Jobs on a Hadoop cluster.
  • Ability to think creatively to help design innovative solutions to complex analytical questions.
  • Extensive experience in installation, configuration, management and deployment of Big Data components and the underlying infrastructure of Hadoop Cluster.
  • Good working knowledge on importing and exporting data from different databases namely MySQL, PostgreSQL, Oracle into HDFS and Hive using Sqoop.
  • Extensive experience in NoSQL and real - time analytics.
  • Strong knowledge of yarn terminology and the High-Availability Hadoop Clusters.
  • Hands-on experience in analyzing Log files for Hadoop and ecosystem services and finding the root cause.
  • Experience in Chef, Puppet or related tools for configuration management.
  • As an admin involved in balancing the loads on server and tuning of server for optimal performance of the cluster.
  • Expertise in Installing, Configuration and Managing Red hat Linux 5, 6.
  • Good experience in scheduling cron jobs in Linux.
  • Worked with PLSQL stored procedures in creating reports, which required modified data input from the source.
  • Experience in the encryption security layer in the Hadoop environment.
  • Proactively maintain and develop all Linux infrastructure technology to maintain a 24x7x365 uptime service.
  • Maintain best practices on managing systems and services across all environments.
  • Fault finding, analysis, and logging information for reporting of performance exceptions.
  • Proactively monitoring system performance and capacity planning.
  • Manage, coordinate, and implement software upgrades, patches, hotfixes on servers, workstations, and network hardware.
  • Create and modify scripts or applications to perform tasks.
  • Provide input on ways to improve the stability, security, efficiency, and scalability of the environment.
  • Install and maintain all server hardware and software systems and administer all server performance and ensure availability for the same.
  • Maintain all system backup and assist to restore all procedures and provide required training to all users.
  • Design and maintain all system tools for all scripts and automation processes and monitor all capacity planning.
  • Integrate all required software and resolve all issues across various technologies and design require enterprise servers and provide back up support.
  • Evaluate all documents according to system requirements and evaluate all design and perform tests on all development activities and administer all complex methodologies.
  • Experience with cloud: Hadoop-on-Azure, AWS/EMR, Cloudera Manager.
  • Develop an infrastructure to provide support to all business requirements and perform regular troubleshoot on a system to resolve all issues.
  • Monitor everyday systems and evaluate the availability of all server resources and perform all activities for Linux servers.
  • Assist to configure and deploy all virtual machines and install and provide backup to all configuration procedures.
  • Maintain and monitor all patch releases and design various patch installation strategies and maintain all systems according to NIST standardization.
  • Administer all performance for various resources and ensure optimization for the same and provide support to all applications and ensure the optimal level of customer services.
  • Maintain and monitor all system frameworks and provide after call support to all systems and maintain optimal Linux knowledge.

TECHNICAL SKILLS

Hadoop Framework: HDFS, Map Reduce, Hive, Pig, Zookeeper, Sqoop, Hbase, Flume

OS: RedHat Linux, UNIX, Windows 2000/NT/XP

Languages: C, C++, SAS,Azure CLI, PL/SQL

Scripting Languages: Unix Shell scripting

Database: Oracle 10g/11g,Azure Data Factory, Azure Bricks, SQLServer, Teradata

Database Tools: Oracle SQL Developer, SQL Plus

Version Control: CVS, SVN

PROFESSIONAL EXPERIENCE

Confidential, Charlotte, NC

Hadoop Administrator

Responsibilities:

  • Involved in start to end process of the Hadoop cluster setup was in installation, configuration and monitoring the Hadoop Cluster.
  • Responsible for Cluster maintenance, commissioning and decommissioning Data nodes, Cluster Monitoring, Troubleshooting, Manage and review data backups, Manage & review Hadoop log files.
  • Monitoring systems and services, architecture design and implementation of Hadoop deployment, configuration management, backup, and disaster recovery systems and procedures.
  • Installation of various Hadoop Ecosystems and Hadoop Daemons.
  • Responsible for Installation and configuration of Hive, Pig, HBase, and sqoop on the Hadoop cluster.
  • Configured various property files like core-site.xml, hdfs-site.xml, mapred-site.xml based upon the job requirement.
  • Involved in loading data from the UNIX file system to HDFS.
  • Provisioning, installing, configuring, monitoring, and maintaining HDFS, Yarn, HBase, Flume, Sqoop, Pig, Hive.
  • Expertise in recommending hardware configuration for the Hadoop cluster.
  • Installing, Upgrading and Managing Hadoop Cluster on Cloudera distribution.
  • Troubleshooting many cloud-related issues such as Data Node down, Network failure and data block missing.
  • Managing and reviewing Hadoop and HBase log files.
  • Experience with Unix or Linux, including shell scripting.
  • Strong problem-solving skills.
  • Working as Cloud Administrator on Microsoft Azure, involved in configuring virtual machines, storage accounts, resource groups.
  • Using Power shell to write, debug scripts to automate the processes and do migration of VM, involving copying and creating of VHDS.
  • Worked on the IBM Jupyter conductor tool, which helps users to run the codes.
  • Loading the data from the different Data sources like (Teradata and DB2) into HDFS using sqoop and load into Hive tables, which are partitioned.
  • Troubleshoot and identify performance, connectivity and other issues for the applications hosted in Azure platform.
  • Developed Hive UDF’s to bring all the customer’s information into a structured format.
  • Developed bash scripts to bring the Tlog files from the FTP server and then processing it to load into hive tables.
  • Built automated set up for cluster monitoring and issue escalation process.
  • Administration, installing, upgrading and managing distributions of Hadoop (CDH3, CDH4, Cloudera manager), Hive, HBase.

Environment: Hadoop, HDFS, Map Reduce, Shell Scripting, spark, Pig, Hive, HBase, Sqoop, Flume, Oozie, Zookeeper, cluster health, monitoring security, Red hat Linux, IBM jupyter conductor, impala, Cloudera Manager.

Confidential, Atlanta, GA

Hadoop Administrator

Responsibilities:

  • Involved in the design and planning phases of Hadoop Cluster planning.
  • Responsible for Regular health checkups of the Hadoop cluster using custom scripts.
  • Installed and configured multi-node fully-distributed Hadoop cluster of a large number of nodes.
  • Provided Hadoop, OS, and hardware optimizations.
  • Installed and configured Cloudera Manager for easy management of existing Hadoop cluster.
  • Monthly Linux server maintenance, shutting down essential Hadoop name node and data node.
  • Collaborated with the infrastructure, network, database, application, and BI teams to ensure data quality and availability.
  • Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Experienced in managing and reviewing the Hadoop log files.
  • Balancing the Hadoop cluster using balancer utilities to spread data across the cluster equally.
  • Implemented data ingestion techniques like Pig and Hive on the production environment.
  • Routine cluster maintenance on every weekend to make required configuration changes, installation, etc.
  • Implemented Kerberos Security Authentication protocol for an existing cluster.
  • Managing day to day activity of the cloud environment, supporting development teams with their requirements.
  • Creating Labs, Virtual Machines along with setting up policies and using Formulas and Custom Images to deploy the network.
  • Designing and creating ETL jobs through Talend to load huge volumes of data into Cassandra, Hadoop Ecosystems and relational databases.
  • Worked extensively with sqoop for importing metadata from Oracle. Used Sqoop to import data from SQL server to Cassandra.
  • Implement Flume, Spark, Spark Stream framework for real-time data processing. Developed analytical components using Scala, Spark, and Spark Stream. Implemented Proofs of Concept on Hadoop and Spark stack and different big data analytic tools, using Spark SQL as an alternative to Impala.
  • Implemented Apache Spark data processing project to handle data from RDBMS and streaming sources.
  • Monitoring and Debugging Hadoop jobs/Applications running in production.
  • Worked on Providing User support and application support on Hadoop Infrastructure.
  • Kerberos keytabs creation for ETL application use cases before onboarding to Hadoop.
  • Responsible for adding User to Hadoop cluster.
  • Worked on Evaluating, comparing different tools for test data management with Hadoop.
  • Helped and directed the testing team to get up to speed on Hadoop Application testing.

Environment: Java, RedHat Linux, HDFS, Map-Reduce, Hive, Pig, Sqoop, Spark, Scala, Flume, Zookeeper, Oozie, DB2, HBase and Pentaho.

Confidential, Chicago, IL

Hadoop Administrator

Responsibilities:

  • Collaborating with application teams to install the operating system and Hadoop updates, patches, version upgrades when required.
  • Installed and implemented the monitoring tools like ganglia and Nagios on both the clusters.
  • Build a new sandbox cluster for the testing purpose and move data from a secure cluster to an insecure sandbox cluster by using distcp.
  • Performance tuning of Hadoop clusters and Hadoop MapReduce routines.
  • Screen Hadoop cluster job performances and capacity planning.
  • Monitor Hadoop cluster connectivity and security.
  • Manage and review Hadoop log files.
  • HDFS support and maintenance.
  • Diligently teaming with the infrastructure, network, database, application, and business intelligence teams to guarantee high data quality and availability.
  • Installed and configured various components of the Hadoop ecosystem and maintained their integrity.
  • Planning for production cluster hardware and software installation on the production clusters and communicating with multiple teams to get it done.
  • Designed, configured and managed the backup and disaster recovery for HDFS data.
  • Experience with Unix or Linux, including shell scripting.
  • Installing, Upgrading and Managing Hadoop Cluster on Cloudera distribution.
  • Commissioned Data Nodes when data grew and decommissioned when the hardware degraded.
  • Worked with the systems engineering team to plan and deploy new Hadoop environments and expand existing Hadoop clusters.
  • Worked with application teams to install Hadoop updates, patches, version upgrades as required.
  • Involved in implementing High Availability and automatic failover infrastructure to overcome a single point of failure for Name node utilizing zookeeper services.
  • Implemented the HDFS snapshot feature.
  • Involved in start to end process of the Hadoop cluster setup was in installation, configuration and monitoring the Hadoop Cluster.
  • Ran monthly security checks through the UNIX and Linux environment and installed security patches required to maintain the high-security level for our clients.

Environment: Hadoop, Map Reduce, Shell Scripting, spark, Pig, Hive, Cloudera Manager, CDH 5.4.3, HDFS, Yarn, Hue, Sentry, Oozie, Zookeeper, Impala, Kerberos, cluster health, Puppet, Ganglia, Nagios, Flume, Sqoop, Storm, Kafka, KMS

Confidential, Peachtree City, GA

UNIX Administrator

Responsibilities:

  • Implemented and setup Virtualization environments for AIX LPARs, HP Integrity VMs; and Solaris Zones and Logical Domains
  • Updated and created provisioning scripts to setup new operating systems and software for supported platforms
  • Setup and maintained NFS/iSCSI, NetApp and Oracle ZFS Appliances
  • Consolidated servers at numerous smaller remote data centers to three central data centers
  • Managed Backups for critical data which was stored on more than 70 servers
  • Stabilized system by disk replacement, firmware upgrade in SAN storage, Solaris Volume Management, clustering environment on scheduled maintenance hours.
  • Enhanced business continuity procedure by adding critical middleware server and identified through power-down test activity.
  • Resolved issues, planned requests as point-of-contact to vendors, oversaw developers, business users, following change control procedure, reported result Monitor everyday systems and evaluate the availability of all server resources and perform all activities for Linux servers.
  • Assist to configure and deploy all virtual machines and install and provide backup to all configuration procedures.
  • Maintain and monitor all patch releases and design various patch installation strategies and maintain all systems according to NIST standardization.
  • Administer all performance for various resources and ensure optimization for the same and provide support to all applications and ensure the optimal level of customer services.
  • Maintain and monitor all system frameworks and provide after call support to all systems and maintain optimal Linux knowledge.
  • Perform troubleshoot on all tools and maintain multiple servers and provide back up for all files and script management servers.
  • Wrote and maintained shell scripts using Perl and Bash.
  • Monitored, troubleshot, and resolved issues involving operating systems.
  • Applied ITIL approach to incident and problem management.
  • Developed and maintained a troubleshooting journal for the incident management team.
  • Participated in on-call rotation to provide 24×7 technical support.
  • Tested numerous software and hardware configurations during the development stages to recreate the operating environments utilized by customers in an effort to avoid the distribution of releases with bugs and/or erroneous documentation.
  • Wrote utility scripts using BASH and KORN shell.
  • Configured UNIX systems to use Active directory, KERBEROS, NTPD, XDMCP, LDAP, SSH, FTP, TFTP, and DNS.
  • Configured SUN Workstation and SUN Server.
  • Performed problem diagnosis, corrected discrepancies, developed user and maintenance documentation, provided user assistance and evaluated system performance.
  • Installed, configured third party applications, hardened new and existing servers and desktops.
  • Converted all systems into trusted systems to comply with STIG.
  • Assisted DISA's system security office with the security testing and evaluation process.

Environment: Solaris, HP UX with Oracle, Sybase, KERBEROS, monitoring tool HP OpenView ITO (OVO), Redhat Linux, Windows, FTP

Hire Now