We provide IT Staff Augmentation Services!

Hadoop Administrator Resume

Raleigh, NC

PROFESSIONAL SUMMARY:

  • 14 Years of diverse experience in Software Engineering and Administration.
  • 3 years of Experience in Hadoop Administration.
  • 3 Years of Experience as Linux Support Engineer .
  • 3 Year of Automation Experience in Perl, Shell,Unix.
  • 7 Years of Software development experience in Java,C,C++ and Linux
  • Experience in installation, configuration and management of Hadoop Clusters
  • Experience with Hortonworks HDP 2.3.4, HDP 2.6.1 and Cloudera CDH4, CDH5 distributions
  • Experience in using Cloudera Manager for tracking cluster utilization and Cloudera navigator for defining data lifecycle rules
  • Extensive experience on configuration of cluster using Ambari Server for HDP
  • Good Knowledge of deploying Hadoop2 cluster on EC2 cloud service by AWS.
  • In depth knowledge on functionalities of every Hadoop daemon, interaction between them, resource utilizations and dynamic tuning to make cluster available and efficient
  • Experience in providing security for Hadoop Cluster with Kerberos, Ranger, Knox
  • Experience in creating job pools, assigning users to pools and restricting production job submissions based on pool
  • Experience in setting up the monitoring tools such as Nagios and Ganglia to monitor and analyze the functioning of cluster.
  • Experience in setting up and managing data gathering tools such as Sqoop for batch ingest
  • Good understanding of NoSQL databases such as Hbase and Cassandra
  • Experience in analyzing data on HDFS through MapReduce, Hive and Pig
  • Extensive experience with ETL and Query big data tools like Pig Latin and Hive QL
  • Experience in setting up workflows and scheduling the workflows using Oozie
  • Experience on UNIX commands and Shell Scripting
  • Excellent interpersonal, communication, documentation and presentation skills
  • Strong experience in interacting with business analysts and developers to analyze the user requirements, functional specifications and system specifications.
  • Working Knowledge on Configuration tools such as Chef and Puppet.

TECHNICAL SKILLS:

Hadoop/Big Data platform: HDFS, MapReduce, Hbase, Cassandra, Hive, Pig, Oozie, Zookeeper, Flume, Sqoop

Hadoop distribution: Horton Works,Cloudera

Admin operations: Access control, Cluster maintenance, Performance tuning, Storage capacity management

Programming Languages: C,C++, Java, Pig Latin

Web Development Tools: VB Script

Operating Systems: Windows Series, HP Unix, Linux

Databases: MYSQL, Hbase, Cassandra

Scripting Languages: Perl, Shell, Python

PROFESSIONAL EXPERIENCE:

Confidential, Raleigh, NC

Hadoop Administrator

Responsibilities:

  • Responsible for building scalable distributed data solutions using HDP Hortonworks
  • Maintenance of the HDP Hortonworks Cluster of 280 Nodes.
  • Configured the Clusters for various environments like PROD, SIT, CAT and DEV
  • Interacting with Dev team for day to day support.
  • Working with Unix/Linux Admin team in the administering of hardware and OS requirements for the Hortonworks Cluster.
  • Performed HDP 2.6.1 upgrades in PROD, SIT, CAT and DEV environments.
  • Enabled HA for Namenode, Resource Manager,Yarn Configuration, Hive Metastore and HBase
  • Worked on Kafka for data streaming.
  • Monitoring the Kafka data synchronization using Commands and Zookeeper administration
  • Configuration of Flume and Sqoop in the environments
  • Configuring and Moving of Journal Nodes during the expansion of the cluster.
  • Performed Ambari server upgrades
  • Adding and Decommissioning of nodes as part of maintenance of the cluster
  • Good Knowledge on Cassandra
  • Worked on Spark issues and Configuration
  • Monitored Hadoop cluster job performance and capacity planning.
  • HDFS, Hive and HBase Performance Tuning
  • Performed Cluster to Cluster Copy using distcp
  • Worked on implementation of SSL /TLS implementation.
  • Taking blueprints and the snapshots of the clusters during any major changes to the cluster.
  • Taking backup of Critical data, Hive data and creating snapshots.
  • Fixing Issues related to the cluster configuration and NameNodes
  • Monitoring and troubleshooting, and review Hadoop log files.
  • Working using Curl Commands to fix any issues
  • Configuration of SSL and trouble shooting in Hue.
  • Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms
  • Enabled Kerberos for authorization and authentication.
  • Experience on Ranger, Knox Configuration with LDAP
  • Enabled HA for Namenode, Resource Manager,Yarn Configuration and Hive Metastore.
  • Configured Journal nodes and Zookeeper Services for the cluster using Cloudera.
  • Monitored Hadoop cluster job performance and capacity planning.
  • Monitored and reviewed Hadoop log files.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Responsible for cluster maintenance, adding and removing cluster nodes, cluster
  • Handled importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS. Extraction data using Flume. Import/Export to HDFS/RDMS using Sqoop
  • Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.
  • Good Knowledge of NoSQL database like HBase
  • Performance tuning of Impala jobs and resource management in cluster.
  • Hands on Experience on Puppet in pushing/deploying the configurations to the cluster.

Environment: MapReduce, HDFS, Hive, SQL, Oozie, Sqoop, UNIX Shell Scripting, Yarn.

Confidential, Sugarland, TX

Linux Support and Hadoop Administrator

Responsibilities:

  • Worked on Data migration from hive warehouse to Anchor points.
  • Worked on implementation of SSL /TLS implementation.
  • Configuration of SSL and trouble shooting in Hue.
  • Responsible for building scalable distributed data solutions using HadoopCloudera works
  • Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms
  • Enabled Kerberos for authorization and authentication.
  • Enabled HA for Namenode, Resource Manager,Yarn Configuration and Hive Metastore.
  • Configured Journal nodes and Zookeeper Services for the cluster using Cloudera.
  • Monitored Hadoop cluster job performance and capacity planning.
  • Monitored and reviewed Hadoop log files.
  • Performed ClouderaManager and CDH upgrades
  • Taking backup of Critical data, Hive data and creating snapshots.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Responsible for cluster maintenance, adding and removing cluster nodes, cluster
  • Monitoring and troubleshooting, and review Hadoop log files.
  • Experience in Configuration Changes using Chef
  • Handled importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS. Extraction data using Flume. Import/Export to HDFS/RDMS using Sqoop
  • Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.
  • Good Knowledge of NoSQL database like HBase
  • Continuous monitoring and managing the Hadoop cluster through Cloudera Manager.
  • Installed Oozie workflow engine to run multiple Hive.
  • Integration of Impala and hive with Tableau .
  • Performance tuning of Impala jobs and resource management in cluster.

Environment: MapReduce, HDFS, Hive, SQL, Oozie, Sqoop, UNIX Shell Scripting, Yarn.

Confidential, Warrenville, IL

Linux Support Engineer/ Automation

Responsibilities:

  • As a Linux Support Engineer, understand the various business processes implemented via TMS6 - loading processes, End of Day and End of Month activities, Testing and Deployment process.
  • Involved in setting up the lab for the test environment.
  • Setting up of Virtual Machines using VMWare
  • Installation of Linux and configuring the system.
  • Configured Field Devices (Acculoads, PLC) to TAS Servers
  • Install and maintain all server hardware and software systems and administer all server performance and ensure availability for same.
  • Monitor everyday systems and evaluate availability of all server resources and perform all activities for Linux servers.
  • Maintain and monitor all system frameworks and provide after call support to all systems and maintain optimal Linux knowledge.
  • Perform tests on all new software and maintain patches for management services and perform audit on all security processes.
  • Gathered test data requirements for data conditioning from Business Units to test total application functionality.
  • Developed Automation Scripts using shell scripts to check the log files size, and report the application.
  • Responsible in writing the cron jobs to start the processes at regular intervals.
  • Involved in Database Testing Using SQL to pull data from database and check whether it matches with GUI.
  • Responsible in supporting the Deployment activities, tracking the data flow from TMS6 application to Tophat application and other external systems and Bubble support activities.

Environment: RedHat Linux 6.3, MySQL, VMware, Shell, Perl

Confidential

Systems Analyst

Responsibilities:

  • Involved in developing detailed test strategy, test plan, test cases and test scripts for Automation Testing.
  • Set up the test environment, defining detailed Test Requirements, converting them into Test Cases and collected Test Metrics for analyzing the Testing Effort.
  • Develop, maintain and conduct smoke test cases for QA environments.
  • Development of library functions for the Automation Test cases.
  • Coordinate and work along with the development and business teams. Controlled testing projects at every step of the quality cycle from test planning through execution of defect management.
  • Involved as part of automation team for the development of Perl and Shell Scripts to automate the DPG Product.
  • DPG product involves snap mirroring among volumes and aggregates, taking backup from disk to tape and other operations on filers.
  • Debugging the logs when the problem occurs during execution.
  • Unit testing and System testing of the scripts.

Environment: QTP,Linux,Filers,Perl

Confidential, Sandiego, CA

Automation Engineer

Responsibilities:

  • Coordinate and work along with the onsite coordinator
  • Automation of scripts in Perl.
  • Debugging the logs when the problem occurs during execution.
  • Unit testing and System testing and UAT of the scripts.
  • Performed regression, functional, system, UAT testing on main application
  • Developing and maintaining test scripts, analyzing bugs and interacting with development team members in fixing the defects.
  • Responsible for writing simple to complex SQL queries to verify the data in database.
  • Responsible for analysis, reports and defect tracking.

Environment: Linux, Perl,Java

Confidential

Sr Software Engineer/ Module Lead

Responsibilities:

  • Solving the bugs raised by customers as problems reports,
  • Software maintenance and Enhancements: Fixing bugs, releasing Change/technical notes.
  • Responsible to lead the module related activities
  • Developed Perl scripts to verify the output of the Network Elements.
  • Query the data using SQL*Plus
  • Responsible for extracting and loading data into database for report generation and other functionalities.

Environment: GUI based application, C++, HP UNIX, Perl,Shell, and Oracle 7.3

Confidential

Sr. Software Engineer

Responsibilities:

  • Solving the bugs raised by customers as problems reports.
  • Software maintenance and Enhancements: Fixing bugs, releasing Change/technical notes.
  • Responsible to lead the module related activities
  • Developed Perl scripts to verify the output of the Network Elements.
  • Query the data using SQL*Plus
  • Responsible for extracting and loading data into database for report generation and other functionalities.
  • Worked on fixing critical bugs found in customer labs. Analyzed bugs prepared Implementation analysis reports and then once approval was got from product management, implemented the fix.

Environment: GUI based application, C++, HP UNIX, Perl,Shell, and Oracle 7.3

Confidential

Software Engineer

Responsibilities:

  • Handled Collection and distribution modules in C++ and Linux
  • Requirements gathering for the project.
  • Involved in development of Collection and distribution modules in C++ contains CMIP protocol.
  • Processing of CDR (Call Detail Record) information in the Raw CDR files.
  • Unit testing of the developed modules.
  • Involved in the module of putting the processed information into the database.
  • Responsible for unit Testing and UAT testing of the product.
  • Collection of Data from the switches using RS232.

Environment: C++,Linux, MySQL

Confidential

Software Engineer

Responsibilities:

  • Responsible for attending release progress meetings
  • Normalization of the Raw CDR files
  • Requirements Gathering for the project
  • Processing of Raw CDRs files of Alcatel OCB 283 switch
  • Processing of Raw CDR files of EWSD Siemens switch
  • Responsible for the unit testing and UAT testing of the project.
  • Collection of data from the switches using RS232.

Environment: C++,Linux, MySQL

Hire Now