SUMMARY:

• Around 8 years of professional experience including 3+ years of Hadoop Administration and 4+ years as Linux Admin.
• Experienced in installation, configuration, supporting and monitoring 100+ node Hadoop cluster using Cloudera manager and Hortonworks distributions.
• Experience in performing various major and minor Hadoop upgraded on large environments.
• As an admin involved in Cluster maintenance, trouble shooting, Monitoring and followed proper backup& Recovery strategies.
• Experience in HDFS data storage and support for running map - reduce jobs.
• Involved in Infrastructure set up and installation of Cloudera stack on Amazon Cloud.
• Experience with ingesting data from RDBMS sources like - Oracle, SQL and Teradata into HDFS using Sqoop.
• Experience in big data technologies: Hadoop HDFS, Map-reduce, Pig, Hive, Oozie, Sqoop, Zookeeper and NoSQL.
• Experience in benchmarking, performing backup and disaster recovery of Name Node metadata and important sensitive data residing on cluster.
• Experience in designing and implementing HDFS access controls, directory and file permissions user authorization that facilitates stable, secure access for multiple users in a large multi-tenant cluster
• Experience in using Cloudera Manager for Installation and management of Hadoop clusters.
• Monitoring and support through Nagios and Ganglia
• Expertise with NoSQL databases like Hbase, Cassandra and MongoDB.
• Experience in Chef, Puppet or related tools for configuration management.
• Experience in working large environments and leading the infrastructure support and operations.
• Migrating applications from existing systems like MySQL, oracle, db2 and Teradata to Hadoop.
• Expertise with Hadoop, Map reduces, Pig, Sqoop, Oozie, and Hive.
• Benchmarking Hadoop clusters to validate the hardware before and after installation to tweak the configurations to obtain better performance.
• Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster.
• Experience on Commissioning, Decommissioning, Balancing, and Managing Nodes and tuning server for optimal performance of the cluster.
• Experience in Log file management where the logs greater than 7 days old were removed from log folder and loaded into HDFS and stored for 3 months.
• Planned, documented and supported high availability, data replication, business persistent, fail-over and fallback Solutions.
• Knowledge of NoSQL databases such as HBase, Cassandra and MongoDB.
• Strong analytical, diagnostics, troubleshooting skills to consistently deliver productive technological solutions.
• Provided 24/7 technical support to Production and development environments.
• Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations.
• Major strengths are familiarity with multiple software systems, ability to learn quickly new technologies, adapt to new environments, focused adaptive and quick learner with excellent interpersonal, technical and communication skills.

TECHNICAL SKILLS:

Big Data Technologies: HDFS, Hive, Map Reduce, Cassandra, Pig, Hcatalog, Phoenix, Falcon, Scoop, Flume, Zookeeper, Mahout, Oozie, Avro, HBase, MapReduce, HDFS, Storm, CDH 5.3, CDH 5.4

Monitoring Tools: Cloudera Manager, Ambari, Nagios, Ganglia

Scripting Languages: Shell Scripting, Puppet, Scripting, Python, Bash, CSH, Ruby, PHP

Programming Languages: C, Java, SQL, and PL/SQL.

Front End Technologies: HTML, XHTML, XML.

Application Servers: Apache Tomcat, WebLogic Server, Web sphere

Databases: Oracle 11g, MySQL, MS SQL Server, IBM DB2.

NoSQL Databases: HBase, Cassandra, MongoDB

Operating Systems: Linux, UNIX, MAC, Windows NT / 98 /2000/ XP / Vista, Windows 7, Windows 8.

Networks: HTTP, HTTPS, FTP, UDP, TCP/TP, SNMP, SMTP.

Security: Kerberos

WORK EXPERIENCE:

Confidential, El Segundo, CA

HADOOP SYSTEM ADMINISTRATOR

Responsibilities:

• Working as Hadoop Admin and responsible for taking care of everything related to the clusters total of 90 nodes ranges from POC (Proof-of-Concept) to PROD clusters.
• Working as admin on Cloudera (CDH 5.5.2) distribution for clusters ranges from POC to PROD.
• Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files.
• Day to day responsibilities includes solving developer issues, deployments moving code from one environment to other environment, providing access to new users and providing instant solutions to reduce the impact and documenting the same and preventing future issues.
• Adding/installation of new components and removal of them through Cloudera Manager.
• Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades.
• Monitored workload, job performance and capacity planning using Cloudera Manager .
• Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
• Interacting with Cloudera support and log the issues in Cloudera portal and fixing them as per the recommendations.
• Exploring with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's, Spark YARN .
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
• Imported logs from web servers with Flume to ingest the data into HDFS.
• Using Flume and Spool directory loading the data from local system to HDFS .
• Experience in Chef, Puppet or related tools for configuration management.
• Retrieved data from HDFS into relational databases with Sqoop .
• Parsed cleansed and mined useful and meaningful data in HDFS using Map-Reduce for further analysis Fine tuning hive jobs for optimized performance.
• Implemented custom interceptors for flume to filter data and defined channel selectors to multiplex the data into different sinks.
• Partitioned and queried the data in Hive for further analysis by the BI team.
• Worked on installing cluster, commissioning & decommissioning of Data Nodes, Name Node recovery, capacity planning, Cassandra and slots configuration.
• Involved in extracting the data from various sources into Hadoop HDFS for processing.
• Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.
• Creating collections and configurations, Register a Lily HBase Indexer configuration with the Lily HBase Indexer Service.
• Creating and truncating HBase tables in hue and taking backup of submitter ID(s).
• Configuring, Managing permissions for the users in hue .
• Troubleshooting, debugging & fixing Talend specific issues, while maintaining the health and performance of the ETL environment
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: HDFS, Map Reduce, Hive 1.1.0, Hue 3.9.0, Pig, Flume, Oozie, Sqoop, CDH5, Apache Hadoop 2.6, Spark, SOLR, Storm, Knox, Cloudera Manager, Red Hat, MySQL and Oracle.

HADOOP ADMINISTRATOR

Responsibilities:

• Installed and configured a Horton Works HDP 2.2 and Hadoop 2.6 using AMBARI .
• Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop .
• Responsible for building scalable distributed data solutions using Hadoop.
• Worked on installing cluster, commissioning & decommissioning of DataNode, NameNode recovery, capacity planning, and slots configuration.
• Installed, Configured, Tested Datastax Enterprise Cassandra multi-node cluster which has 4 Datacenters and 5 nodes each.
• Installed and configured Cassandra cluster and CQL on the cluster.
• Created HBase tables to store variable data formats of PII data coming from different portfolios.
• Managing and reviewing Hadoop log files and debugging failed jobs.
• Implemented Kerberos Security Authentication protocol for production cluster.
• Implemented a script to transmit sysprin information from Oracle to HBase using Sqoop.
• Implemented test scripts to support test driven development and continuous integration.
• Worked on tuning the performance Pig queries.
• Responsible for adding new eco system components, like spark, storm, flume, Knox with required custom configurations based on the requirements.
• Managed the design and implementation of data quality assurance and data governance processes.
• Worked with Infrastructure teams to install operating system, Hadoop updates, patches, version upgrades as required.
• Backed up data on regular basis to a remote cluster using distcp.
• Responsible to manage data coming from different sources.
• Involved in data analysis projects using Elastic Map Reduce on the Amazon Web Services (AWS) cloud.
• Cluster coordination services through Zookeeper .
• Loaded the dataset into Hive for ETL Operation .
• Automated all the jobs for pulling data from FTP server to load data into Hive tables, using Oozie workflows.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Implemented Fair scheduler to allocate fair amount of resources to small jobs.
• Assisted the BI team by Partitioning and querying the data in Hive .
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.

Environment: Hadoop HDFS, MapReduce, Hortonworks, Hive, Pig Hive, Oozie, Flume Sqoop, HBase.

HADOOP ADMINISTRATOR

Responsibilities:

• Capturing data from existing databases that provide SQL interfaces using Sqoop.
• Processed information from Hadoop HDFS. This information will comprise of various useful insights that can be used in the decision making process. All these insights will be presented to the users in the form of Charts.
• Working on different Big Data technologies, good knowledge of Hadoop, Map-Reduce, Hive.
• Developed various POCs over Hadoop, Big data.
• Worked on deployments and automation task.
• Installed and configured Hadoop cluster in pseudo and fully distributed mode environments.
• Involved in developing the data loading and extraction processes for big data analysis.
• Worked on professional services engagements to help customers design, build clusters, applications, troubleshoot network, disk and operating system related issues.
• Worked with Puppet for automated deployments.
• Installed and configured local Hadoop Cluster with 3 nodes and set up 4 nodes cluster on EC2 cloud.
• Written MapReduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration.
• Work with HBase and Hive scripts to extract, transform and load the data into HBase and Hive.
• Continuous monitoring and managing of the Hadoop cluster.
• Analyzed the data by performing Hive queries and running Pig scripts to know user behavior.
• Installed Oozie workflow engine to run multiple Hive and Pig jobs.
• Developing scripts and batch job to schedule a bundle (group of coordinators) which consists of various Hadoop programs using Oozie.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports.

Environment: Hadoop2.4, 2.5.2, HDFS, Map Reduce, Hive, Flume, Sqoop, Cloudera CDH4, HBase, Oozie, Pig, AWS EC2 cloud.

HADOOP SYSTEM ADMINISTRATOR

Responsibilities:

• Installed/Configured/Maintained Apache/Cloudera Hadoop clusters for application development and Hadoop tools like MapReduce, Hive, Pig, HBase, Zookeeper and Sqoop.
• Responsible for building scalable distributed data solutions using Hadoop.
• Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
• Managing and Scheduling Jobs on a Hadoop cluster.
• Implemented NameNode backup using NFS. This was done for High availability.
• Involved in taking up the Backup, Recovery and Maintenance.
• Shuffle algorithm, direct access to the disk, built-in compression, and code written in Java.
• Worked on importing and exporting data from Oracle into HDFS and HIVE using Sqoop.
• Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
• Created Hive External tables and loaded the data in to tables and query data using HQL.
• Wrote shell scripts for rolling day-to-day processes and it is automated.
• Collected the logs data from web servers and integrated in to HDFS using Flume.
• Implemented Fair schedulers on the Job tracker to share the resources of the Cluster for the Map Reduce jobs given by the users.
• Involved in commissioning and decommissioning at the time of node failure.
• Implemented Partitioning and Bucketing concepts using Hive.
• Involved in upgrading clusters to Cloudera Distributed Clusters and deployed into CDH4.
• Taken ad-hoc requests to get the non PII (Personally Identifiable Information) from the production cluster to non-production cluster to test out various scenarios

Environment: Hadoop, MapReduce, Hive, Pig, HBase, Sqoop, Flume, Zookeeper, Cloudera Distributed Hadoop (CDH4), Cloudera Manager.

LINUX/UNIX SYSTEM ADMINISTRATOR

Responsibilities:

• Installation, Configuration and Administration of Solaris 8/9/10, HP-UX 11.x, AIX 5.3, and Linux (RHEL 4/5, SLES 10)
• Installed, Configured and Maintained Debian/RedHat Servers at multiple Data Centers.
• Configured RedHat Kickstart server for installing multiple production servers.
• Configuration and administration of DNS, LDAP, NFS, NIS and NISand Send mail on RedHat Linux/Debian Servers.
• Hands on experience working with production servers at multiple data centers.
• Involved in writing scripts to migrate consumer data from one production server to another production server over the network with the help of Bash and Perl scripting.
• Installed and configured monitoring tools Munin and NagiOS for monitoring the network bandwidth and the hard drives status.
• Automated server building using System Imager, PXE, Kickstart and Jumpstart.
• Planning, documenting and supporting high availability, data replication, business persistence, and fail-over, fail-back using Veritas Cluster Server in Solaris, RedHat Cluster Server in Linux and HP Service Guard in HP environment.
• Automated tasks using shell scripting for doing diagnostics on failed disk drives.
• Configured Global File System (GFS) and Zetta byte File System (ZFS)
• Troubleshooting production servers with IPMI tool to connect over SOL.
• Configured system imaging tools Clonezilla and System Imager for data center migration.
• Configured yum repository server for installing packages from a centralized server.
• Installed Fuse to mount the keys on every Production server for password-less authentication on Debian servers.
• Installed and configured DCHP server to give IP leases to production servers.
• Management of RedHat Linux user accounts, groups, directories and file permissions.
• Implemented the Clustering Topology that meets High Availability and Failover requirement for performance and functionality.
• Performed Kernel tuning with the sysctl and installed packages with yum and rpm.
• Installed and configured PostgresSQL database on RedHat/Debian Servers.
• Performed Disk management with the help of LVM (Logical Volume Manager)
• Configuration and Administration of Apache Web Server and SSL.
• Backup management Recovery through Veritas Net Backup (VNB)
• Password-less setup and agent-forwarding done for SSHS login using ssh-keygen tool.
• Established and maintained network users, user environment, directories, and security.
• Documented strongly the steps involved for data migration on production servers and also testing procedures before the migration.
• Provided 24/7 on call support on Linux Production Servers. Responsible for maintaining security on Red Hat Linux.

Environment: RHEL 5.x/4.x, Solaris 8/9/10, Sun Fire, IBM blade servers, Web sphere 5.x/6.x, Apache 1.2/1.3/2.x, iPlanet, Oracle 11g/10g/9i, Logical Volume Manager, Veritas net backup 5.x/6.0, SAN Multipathing (MPIO, HDLM, Power path), VM ESX 3.x/2.x.