SUMMARY:

• 8 years of IT experience including 4+ years of experience in installation and configuration of different Hadoop eco - system components in the existing cluster.
• Experience in deploying and managing the multi-node development, testing and production Hadoop cluster with different Hadoop components ( HIVE , PIG , Spark , SQOOP, OOZIE , FLUME, RANGER , KNOX, HBASE , ZOOKEEPER ) using Apache Ambari .
• Experience on Horton works and Cloudera manager Strong knowledge on Hadoop HDFS architecture and Map-Reduce framework .
• Experience in implementing new cluster all together from scratch and done live data migration from the old cluster to the newly built one without affecting any running production jobs.
• Implemented Cloudera Backup and Disaster Recovery ( BDR ) for enabling data protection across datacenters for Disaster Recovery scenarios
• Excellent understanding of Hadoop Cluster security and implemented secure Hadoop cluster using Kerberos & TLS.
• Linux KVM installation and management VM's with RHEL 7.2
• Experience in improving the Hadoop cluster performance by considering the OS kernel , Storage, Networking, Hadoop HDFS and Map-Reduce by setting appropriate configuration parameters.
• Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster using Ambari.
• Experience in upgrading Hadoop cluster from current version to minor version upgrade as well as to major versions.
• Experience in using Zookeeper for coordinating the distributed applications.
• Experience in managing Hadoop infrastructure like commissioning, decommissioning, log rotation, rack topology implementation.
• Experienced with the Apache Nifi flow to perform the conversion of Raw XML data into JSON , AVRO .
• Experience in Chef , Puppet or related tools for configuration management.
• Experience in HDFS data storage and support for running map-reduce jobs.
• Expertise in implementation and designing of disaster recovery plan for Hadoop Cluster.
• Optimizing performance of Hbase/Hive/Pig jobs.
• Experience in managing the cluster resources by implementing fair and capacity scheduler .
• Experience in scheduling jobs using OOZIE workflow.
• Scheduling jobs using Crontab.
• Experience in benchmarking, performing backup and disaster recovery of Name Node metadata and important sensitive data residing on cluster.
• Strong knowledge in configuring Name Node High Availability.
• Experience in configuring Hadoop Security ( Ranger and Knox gateway).
• Experience in handling multiple relational databases: MySQL, SQL Server.
• Global Service Delivery experience by bringing together resources to accomplish organizational goals using ITIL framework.
• Effective problem solving skills and outstanding interpersonal skills.
• Ability to work independently as well as within a team environment.
• Worked on setting up Name Node high availability for major production cluster and designed Automatic failover control using zookeeper and quorum journal nodes.
• Setting up automated 24x7 monitoring and escalation infrastructure for Hadoop cluster using Ambari
• Experienced in Linux Administration and TSM Administration.

TECHNICAL SKILLS:

Big Data Technologies: HDFS, Hive, Map Reduce, Cassandra, Pig, Hcatalog, Phoenix, Falcon, Scoop, Flume, Zookeeper, Mahout, Oozie, Avro, HBase, MapReduce, HDFS, Storm, CDH 5.3, CDH 5.5

Monitoring Tools: Cloudera Manager, Ambari, Nagios, Ganglia

Scripting Languages: Shell Scripting, Puppet, Scripting, Python, Bash, CSH.

Programming Languages: C, Java, SQL, and PL/SQL.

Front End Technologies: HTML, XHTML, XML.

Application Servers: Apache Tomcat, WebLogic Server, Web sphere

Databases: Oracle 11g, MySQL, MS SQL Server, IBM DB2.

NoSQL Databases: HBase, Cassandra, MongoDB

Operating Systems: Linux, UNIX, MAC, Windows NT / 98 /2000/ XP / Vista, Windows 7, Windows 8.

Networks: HTTP, HTTPS, FTP, UDP, TCP/TP, SNMP, SMTP

Security: Kerberos, Knox, Ranger.Current Project

PROFESSIONAL EXPERIENCE:

Confidential - Chicago, IL

Hadoop System Administrator

Responsibilities:

• Involved in setup and designing the Big Data cluster HDP 2.5 on AWS and documented Ambari Technical User Guide includes Hadoop component basics, Step by Step instructions to Install and manage a Hadoop cluster using Ambari, Troubleshooting Ambari Deployments, and specific issues like HDFS Smoke Test fails, Cluster Install failure, Hcatalog Daemon Metastore Smoke Test Fails.
• Hadoop sizing estimate, node calculator, Cluster build on multiple hosts/virtual hosts and Cluster evaluation and standard Big Data Best Practices
• Active participant in defining User standards includes Best Practices includes coding schema, Objects, System level schemas, dynamic tables in the functions, leveraging truncate instead of delete, compression level, Partitions, table transformations and subscription backups, etc
• Involved in various phases of Software Development Life Cycle (SDLC) as Requirement gathering, Data Modelling, Analysis, Architecture Design and Development
• Setup Java Garbage Collector monitoring alert when memory usage of process exceeds 85% of allocated memory, so that we can fix it to improve applications performance and throughput.
• Big Data Lunch and Learn Demos on Big Data administration includes Hadoop Ecosystem deep dive includes Hadoop Overview, Ambari, Hadoop features and components, Yarn Architecture, MapReduce, Hive and HBase Architecture, Hive, Impala, Spark, Hue, Flume, Cluster, Sqoop, Kafka, Oozie, Zoo Keeper, Pig, Hue, Hadoop Encryption and Pepper data.
• General guidelines to users for long running Impala Queries, and Impala Compute Stats failing issues, Issue when using LIMIT clause and Setup load balancing and VIP
• Experience with AWS instances spanning across Dev, Test and Pre-Production and Cloud Automation through Open Source DevOps tools like Ansible, Jenkins & Docker.
• Worked hands-on to create automated, containerized cloud application platform (PaaS), and design and implement DevOps processes that use those platforms.
• Build AWS Cloud formation templates to create custom sized VPC, Subnets, NAT to ensure successful deployment of Web applications and database templates.
• Ingest 4 TB of Data from different mining sites to MS AZURE Data Lake
• Develop AZURE scalable cloud solution to Integrate Financial and Operational Cost data
• Designed roles and groups for users and resources using AWS Identity Access Management (IAM)
• Integrated AWS Dynamo DB using AWS Lambda to store the values items and backup the DynamoDB streams
• Documented Master Document for Developers includes Spark, Hive, Impala, Ambari, Authentication and Authorization and Proxy Server usage and also documented on failure scenarios and remedies includes hardware and software failures like NN/DN failure, logs filling up disk space, etc. Created Best Practices document for Hive Spark Sqoop File formats and compression and Query Optimization and troubleshooting Spark Applications.

Environment: RHEL 6.5, HDP 1.3.1/2.1/2.5, Ambari, Azure, Yarn, Impala, Hive, HUE, Kafka, Sqoop, Storm, Python, Hbase, Teradata Query Grid, Zookeeper, Oozie, Kerberos, Knox, Ranger, Pig, HBase, Avro.

Confidential, Richmond, VA

Hadoop Infrastructure Administrator

Responsibilities:

• Worked on analyzing Hortonworks Hadoop cluster and different big data analytic tools including Pig, HBase Database and Sqoop.
• Responsible for architecting Hadoop clusters Translation of functional and technical requirements into detailed architecture and design.
• Installed and configured multi-nodes fully distributed Hadoop cluster of large number of nodes.
• Addressing and Troubleshooting issues on a daily basis.
• File system management and monitoring.
• Coordinated with Hortonworks support team through support portal to sort out the critical issues during upgrades.
• Provided Hadoop, OS, Hardware optimizations.
• Installed and configured Hadoop ecosystem components like Map Reduce, Hive, Pig, Sqoop, HBase, Zookeeper and Oozie.
• Involved in testing HDFS, Hive, Pig and Map Reduce access for the new users.
• Cluster maintenance as well as creation and removal of nodes using Apache Ambari
• Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes.
• Set up Apache NiFi to transfer structured and streaming data into HDFS
• Ingested streaming data with Apache NiFi into Kafka
• Monitored systems and services, architecture design and implementation of Hadoop deployment, configuration management, backup, and disaster recovery systems and procedures Implemented capacity scheduler to allocate fair amount of resources to small jobs.
• Configured Oozie for workflow automation and coordination.
• Implemented rack aware topology on the Hadoop cluster.
• Worked on setting up Apache NiFi and used NiFi in orchestrating data pipeline.
• Importing and exporting structured data from different relational databases into HDFS and Hive using Sqoop.
• Configured Zookeeper to implement node coordination, in clustering support.
• Rebalancing the Hadoop Cluster.
• Allocating the name and space Quotas to the users in case of space problems.
• Installed and configured Hadoop security tools Knox, Ranger and enabled Kerberos
• Managing cluster performance issues.
• Creating snapshots and restoring snapshots.
• Good experience in troubleshoot production level issues in the cluster and its functionality.
• Backed up data on regular basis to a remote cluster using Distcp.
• Regular Commissioning and Decommissioning of nodes depending upon the amount of data.
• Maintaining Cluster in order to remain healthy and in optimal working condition.
• Handle the upgrades and Patch updates.

Environment: Hortonworks, Ambari, HDFS, Java, Shell Scripting, Python, Hive, Spark, Sqoop, Linux, SQL, Cloudera, Zookeeper, AWS, NiFi, HBase, Oozie, Kerberos, Ranger.

Confidential - San Jose, CA

Hadoop Administrator

Responsibilities:

• Installed and configured Hadoop MapReduce, HDFS and developed multiple MapReduce jobs.
• Deployed a Hadoop cluster and integrated with Nagios and Ganglia.
• Extensively involved in cluster capacity planning, Hardware planning, Installation, Performance tuning of the Hadoop cluster.
• Worked on installing cluster, commissioning & decommissioning of Data Nodes, Name node recovery, Capacity planning, Cassandra and slots configuration.
• Hands on experience in provisioning and managing multi-node Hadoop Clusters on public cloud environment Amazon Web Services (AWS) - EC2 and on private cloud infrastructure.
• Monitored multiple clusters environments using Metrics and Nagios.
• Experienced in providing security for Hadoop Cluster with Kerberos.
• Dumped the data from MYSQL database to HDFS and vice-versa using SQOOP.
• Used Ganglia and Nagios to monitor the cluster around the clock.
• Dumped the data from one cluster to other cluster by using DISTCP, and automated the dumping procedure using shell scripts.
• Involved in moving all log files generated from various sources to HDFS for further processing through Flume.
• Worked on analyzing Data with HIVE and PIG.
• Implemented Kerberos for authenticating all the services in Hadoop Cluster.
• Configured Zoo keeper to implement node coordination, in clustering support.
• Configured Flume for efficiently collecting, aggregating and moving large amounts of log data from many different sources to HDFS.

Environment: - HDFS, Map Reduce, Hive, Sqoop, PIG, Cloudera, Flume, SQL Server, UNIX, RedHat and CentOS.

Confidential

Linux System Administrator

Responsibilities:

• Installation, Configuration & Migration of UNIX and Linux operating systems
• Manage, maintain and fine tune clustered Apache Tomcat server configuration.
• Installation Packages and patches.
• Installed and configured Ubuntu troubleshooting hardware, operating system, applications & network problems and performance issues
• Worked closely with the development and operations organizations to implement the necessary tools and process to support the automation of builds, deployments, testing of infrastructure.
• Installed and configured various flavors of Linux like Red hat, SUSE and Ubuntu.
• Monitored trouble ticket queue to attend user and system calls, participated in team meetings, change control meetings to update installation progress, and for upcoming changes.
• Diagnosing and resolving systems related tasks in accordance with priorities setup in dealing with trouble tickets.
• Deployed patches for Linux and application servers, Red Hat Linux Kernel Tuning.
• Network trouble shooting using 'netstat', 'ifconfig', 'tcpdump', 'vmstat', 'iostat'.
• Managed cron jobs, batch processing and job scheduling.
• Monitored the servers and Linux scripts regularly and performed troubleshooting steps tested and installed the latest software on server for end-users.
• Troubleshooting application issues on Apache web servers and database servers running on Linux and solaris.
• Performed the manual backups of Database, software and OS using tar, cpio, mksysb.
• Manage file system utilization using script scheduled as a cron job
• Performed automation with simple shell scripting
• Monitoring backup using Backup Exac, regularly monitored Alert-log Files and trace files on the day-to-day basis.
• Monitoring system performance, Server load and bandwidth issues.
• Regularly manage backup process for Server and Client data.

Environment: LINUX/UNIX, ORACLE DB2, SQL SERVER, Windows 2008/2007 server, Unix Shell Scripting, SQL Manager Studio, Red Hat Linux, Microsoft SQL Server 2000/2005/2008, MS Access, NoSQL, Linux/Unix, Putty Connection Manager, Putty, SSH.