TECHNICAL SKILLS:

Programming Languages: Java, C/C++ Hadoop Ecosystem Components HDFS, Map Reduce Yarn, Hbase, Sqoop, Pig, Hive, Flume, Oozie, Strom, Zookeeper, spark and Cloudera Manager

Scripting Languages: Java scripting, SQL, PL/SQL

Databases: Oracle, MYSQL Database Languages Confidential - SQL, PL/SQL

Methodologies: Agile Operating Systems Windows & Linux (CentOS 5, 6 RHEL 6), CentOS, Unix, Red Hat Linux.

Hadoop Ecosystem: Hive, Pig, Sqoop, Flume, Impala, Oozie, Sentry, Spark, Zookeeper, Apache Kafka, Storm Hadoop Management Ambari, Cloudera Manager Hadoop Paradigms Map Reduce, Yarn, High Availability

Other Relevant Tools: SVN, Tableau, JIRA, QC, MS Office Suite RDBMS Oracle 10g/11g/12c, MS SQL Server 2000/2003/2008 R2/2012, DB2, Teradata, Netezza, MySQL., Mongo DB

Programming Languages: Linux, Unix Shell scripting, JAVA, SQL Monitoring and Alerting Nagios, Ganglia, Ambari Metrics Operating Systems Centos 5,6, Red hat 6, Ubuntu Server 14.04(Trusty), Windows Server 20 Bigdata Hadoop (MapReduce, HBASE, HDFS, Sqoop, Hive, Oozie, flume, PIG, Zookeeper)

Other Big data tools: Storm, Solr, NIFI, Kafka, ActiveMQ Distributions Cloudera, Hortonworks(HDP), Apache Hadoop. AWS SQS, SNS, RedShift, Lambda. NoSQL Data Stores HBASE Scripts Pig and bash.

Query Languages: Hive, Sql, PIG, PL/SQL.

WORK EXPERIENCE:

Hadoop Admin

Confidential, Providence, RI

Responsibilities:

• Extensively involved in Installation and configuration of Cloudera distribution Hadoop Name Node, Secondary Name Node, Resource Manager, Node Manager and Data Nodes. Done stress and performance testing, benchmark for the cluster.
• Installing Patches and packages on Unix/Linux Servers.
• Worked with development in design and ongoing operation of several clusters utilizing Cloudera's Distribution including Apache Hadoop.
• Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and Spark
• Responsible to migrate from Hadoop to Spark frameworks, in - memory distributed computing for real time fraud detection.
• Provided System support/maintenance for 24x7 for Customer Experience Business Services
• Supported Data Analysts in running Map Reduce Programs.
• Implemented Fair scheduler on the job tracker to allocate the fair amount of resources to small jobs.
• Involved in running Hadoop jobs for processing millions of records of text data. Troubleshoot the build issue during the Jenkins build process.
• Implement Docker to create containers for Tomcat Servers, Jenkins.
• Responsible for scheduling jobs in Hadoop using FIFO, Fair scheduler and Capacity scheduler
• Expertise in Hadoop Cluster capacity planning, performance tuning, cluster Monitoring, Troubleshooting.
• Worked on a live Big Data Hadoop production environment with 220 nodes.
• HA implementation of Name Node to avoid single point of failure.
• Experience working on LDAP user accounts and configuring ldap on client machines.
• Automated day to day activities using shell scripting and used Cloudera Manager to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions.
• Responsible for Cluster maintenance, Adding and removing cluster nodes, Cluster Monitoring and Troubleshooting, Manage and review data backups, Manage and review Hadoop log files.
• Involved in planning the Hadoop cluster infrastructure, resources capacity and build plan for Hadoop cluster installations.
• Resolving tickets submitted by users, P1 issues, troubleshoot the error documenting, resolving the errors.
• Installed and configured Hive in Hadoop cluster and help business users/application teams fine tune their HIVE QL for optimizing performance and efficient use of resources in cluster.
• Installed and configured Ganglia Monitoring system to get metrics and monitoring the Hadoop cluster. Also configured Hadoop logs rotation and monitoring them frequently.
• We do performance tuning of the Hadoop Cluster and map reduce jobs.
• Also the real-time applications with best practices to fix the design flaws.
• Implemented Oozie work-flow for ETL Process for critical data feeds across the platform.
• Configured Ethernet bonding for all Nodes to double the network bandwidth
• Implementing Kerberos Security Authentication protocol for existing cluster.
• Built high availability for major production cluster and designed automatic failover control using Zookeeper Failover Controller (ZKFC) and Quorum Journal nodes.
• Worked on Hive for exposing data for further analysis and for generating transforming files from different analytical formats to parquet files.
• Worked closely with Business stake holders, BI analysts, developers, and SAS users to establish SLAs and acceptable performance metrics for the Hadoop as a service offering.

Environment: Hadoop, Apache Pig, Hive, OOZIE, SQOOP, Spark, Hbase, Pig, LDAP, CDH5, Unravel, Splunk, Tomcat, and Java.

Hadoop Admin

Confidential, Detroit, MI

Responsibilities:

• Installed and configured Hadoop, Map Reduce, HDFS (Hadoop Distributed File System), developed multiple Map Reduce jobs for data cleaning in Hadoop cluster of 70 nodes.
• Implemented Spark RDD transformations, actions to migrate Map reduce algorithm using Scala and Spark SQL for faster testing and processing of data.
• Upgraded Production Hadoop cluster From CDH3U2 to CDH4U1 and CM3.7.5 to CM4.1.4 alongside of OS upgrade from RHEL5 to RHEL6 with security (Kerberos) Enabled and AD integrated
• Enabled HA for Name Node using Cloudera Manager to avoid single point of failure
• Installation of hue for GUI access for hive, pig and OOZIE and also resolve the issues reported by users in Hue
• Installation, configuration, deployment, maintenance, monitoring and troubleshooting Ambari clusters in environments such as Development and Production.
• Responsible for managing and scheduling jobs on a Hadoop Cluster.
• Monitoring all daemons, cluster health status on daily basis and tuning system performance related configuration parameters, backing up configuration xml files.
• Worked for Name node recovery and Balancing Hadoop Cluster. Worked on installing cluster, adding and removing of Data Nodes
• Providing immediate support to users for various Hadoop related issues.
• Flume configuration for data transfer from Webservers to Hadoop cluster.
• Administrating and optimizing the Hadoop clusters, monitoring Map Reduce jobs and worked with development team to fix the issues.
• Experience in Implementing Rack Topology scripts to the Hadoop Cluster and involved in capacity planning.
• Load data from various data sources into HDFS using Flume.
• Retrieved data from HDFS into relational databases with Sqoop.
• Parsed cleansed and mined useful and meaningful data in HDFS using Map - Reduce for further analysis
• Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java Map-Reduce, Hive and Sqoop as well as system specific jobs.
• Installed Ganglia to monitor Hadoop daemons and Implemented the changes in configuration parameters and in parallel monitored the changes in Ganglia
• Automated Deployments of ETL applications that reside on top of the Hadoop Cluster.
• Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
• Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.
• Formulated procedures for installation of Hadoop patches, updates and version upgrades.

Environment: Hadoop, Map Reduce, HDFS, Hive, Oracle 11g, Flume, SQOOP, pig, Kerberos, LDAP, JUnit, Tomcat 6, CDH 4 and CDH 5.

Hadoop Admin

Confidential, San Jose, CA

Responsibilities:

• Built multiple clusters running Cloudera as per the business requirements.
• Managing and scheduling Jobs on a Hadoop cluster.
• Deployed Network file system for Name Node Metadata backup.
• Used Ganglia and Nagios to monitor the cluster around the clock.
• Worked with the Linux administration team to prepare and configure the systems to support Hadoop deployment
• Import data using Sqoop to load data from Oracle Server/MySQL Server to HDFS on regular basis.
• Load and transform large sets of structured, semi structured and unstructured data
• Importing And Exporting Data from MySQL/Oracle to HiveQL using SQOOP.
• Involved in making very important and major enhancements to the already existing Map Reduce programs and Pig scripts
• User management, involving user creation, granting permission for the user to various tables and database, giving group permissions.
• Performed daily maintenance of servers and tuned system for optimum performance by turning off unwanted peripheral and vulnerable service.
• Created Users, Groups, Roles, Profiles and assigned users to groups and granted privileges and permissions to appropriate groups.
• Highly involved in operations and troubleshooting Hadoop clusters
• Monitored cluster job performance and capacity planning
• Worked on analyzing Data with HIVE and PIG and optimizing the jobs to improve runtimes.
• Managing and scheduling Jobs on a Hadoop cluster.
• Expertise in database performance tuning & data modeling
• Developed PIG Latin scripts to extract the data from the web server output files to load into HDF
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop
• Created Hive External tables and loaded the data in to tables and query data using HQL.
• Wrote shell scripts for rolling day - to-day processes and it is automated.
• Worked on performing minor upgrades
• Implemented Fair scheduler on the job tracker to allocate the fair amount of resources to small jobs.

Environment: Hadoop, Apache Pig, Hive, OOZIE, Hbase, Ganglia, Nagioa, SQOOP, CDH 4.1.x - 4.5x, Mysql 5.x

Hadoop Administrator

Confidential, Dallas, TX

Responsibilities:

• Experience in managing scalable Hadoop cluster environments.
• Involved in managing, administering and monitoring clusters in Hadoop Infrastructure.
• Diligently teaming with the infrastructure, network, database, application and business intelligence teams to guarantee high data quality and availability.
• Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required.
• Experience in HDFS maintenance and administration.
• Managing nodes on Hadoop cluster connectivity and security.
• Experience in commissioning and decommissioning of nodes from cluster.
• Experience in Name Node HA impl mentation.
• Working on architected solutions that process massive amounts of data on corporate and AWS cloud - based servers.
• Working with data delivery teams to setup new Hadoop users.
• Installed Oozie workflow engine to run multiple Map Reduce, Hive and pig jobs.
• Configured Megastore for Hadoop ecosystem and management tools.
• Hands-on experience in Nagios and Ganglia monitoring tools.
• Experience in HDFS data storage and support for running Map Reduce jobs.
• Performing tuning and troubleshooting of MR jobs by analyzing and reviewing Hadoop log files.
• Installing and configuring Hadoop eco system like Sqoop, Pig, Flume, and Hive.
• Maintaining and monitoring clusters.
• Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.
• Experience in using distcp to migrate data between and across the clusters.
• Installed and configured Zookeeper. Hands on experience in analyzing Log files for Hadoop eco system services.
• Coordinate root cause analysis efforts to minimize future system issues.
• Troubleshooting of hardware issues and closely worked with various vendors for Hardware/OS and Hadoop issues.

Environment: Cloudera4.2, HDFS, Hive, Pig, Sqoop, HBase, Chef, Rhel, Mahout, Tableau, Micro strategy, Shell Scripting, Red Hat Linux.

Linux Administrator

Confidential

Responsibilities:

• Installation, configuration and troubleshooting of Red hat 4.x, 5.x, Ubuntu 12.x and HP - UX 11.x on various hardware platforms
• Installed and configured JBoss application server on various Linux servers
• Configured Kick-start for RHEL (4, and 5), Jumpstart for Solaris and NIM for AIX to perform image installation through network
• Involved in configuration and troubleshooting of multipathing Solaris and bonding on Linux Servers
• Configured DNS Servers and Clients, and involved in troubleshooting DNS issues
• Worked with Red Hat Linux tools like RPM to install packages and patches for Red Hat Linux server
• Developed scripts for automating administration tasks like customizing user environment, and performance monitoring and tuning with nfsstat, netstat, iostat and vmstat
• Performed network and system troubleshooting activities under day to day responsibilities.
• Created Virtual server on VMware ESX/ESXi based host and installed operating system on Guest Servers
• Installed/Configured/Maintained/Administrated the network servers DNS, NIS, NFS, SENDMAIL and application servers Apache, JBoss on Linux.
• Extensive use of Logical Volume Manager (LVM), creating Volume Groups, Logical volumes and disk mirroring in HP-UX, AIX and Linux.
• Working with Clusters; adding multiple IP addresses to a Servers via virtual network interface in order to minimize network traffic (load-balancing and failover clusters)
• User and Security administration; addition of User into the System and Password management.
• System Monitoring and log management on UNIX and Linux Servers; including, crash and swap management, with password recovery and performance tuning.

Environment: Red hat, Ubuntu, JBoss, RPM, VMware, ESXi, LVM, DNS, NIS, HP-UX. Big Data Tools: HDFS, MapReduce, YARN, Hive, Pig, Sqoop, Flume, Oozie, Kafka, Horton work, Ambari, Knox, Phoenix, Impala, Storm.