We provide IT Staff Augmentation Services!

Hadoop Admin Resume

Irvine, CA

SUMMARY

  • 8 plus years of IT industrial experience in Administrating Linux, Database management, developing Map - reduce applications, designing, building and administrating large scale Hadoop production Clusters
  • 2.5 years of experience in big data technologies: Hadoop HDFS, Map-reduce, Pig, Hive, Oozie, Flume, Sqoop, Zookeeper, And NoSQL: Cassandra and Hbase.
  • Experience in deploying and managing the multi-node development, testing and production Hadoop cluster with different Hadoop components (HIVE, PIG, SQOOP, OOZIE, FLUME, HCATALOG, HBASE, ZOOKEEPER) using Hortonworks Ambari.
  • Strong knowledge on Hadoop HDFS architecture and Map-Reduce framework.
  • Strong knowledge of Apache Hive data warehouse, data cubes, Hive server, partitioning, bucketing, clustering and writing UDFS, UDAFS, and UDTFS in Java for hive.
  • Solid experience in Pig administration and development and writing PIG UDFS (Eval, Filter, Load and Store) and macros.
  • Experience in administering the Linux systems to deploy Hadoop cluster and monitoring the cluster using Nagios and Ganglia.
  • Experience in benchmarking, performing backup and disaster recovery of Name Node metadata and important sensitive data residing on cluster.
  • Experience in performing minor and major upgrades, commissioning and decommissioning of data nodes on Hadoop cluster.
  • Strong knowledge in configuring Name Node High Availability and Name Node Federation.
  • Familiar with writing Oozie workflows and Job Controllers for job automation - shell, hive, scoop automation.
  • Familiar with importing and exporting data using Sqoop from RDBMS MySQL, Oracle, Teradata and also using fast loaders and connectors Experience.
  • Experience in using Flume to stream data into HDFS - from various sources.
  • Hands on experience in provisioning and managing multi-tenant Hadoop clusters on public cloud environment - Amazon Web Services (AWS)-EC2.
  • Experience in installing and administering PXE Server with kick start, setting up FTP, DHCP, DNS servers and Logical Volume Management.
  • Experience in configuring and managing storage devices NAS (file level access - NFS) and SAN (block level access-iSCSI)
  • Experience in Storage management including JBOD, RAID Levels 1 5 6 10, Logical Volumes, Volume Groups and Partitioning
  • Exposure to Maven/Ant, GIT along with Shell Scripting for Build & Deployment Process.
  • Experience in understanding the security requirements for Hadoop and integrating with Kerberos authentication infrastructure- KDC server setup, crating realm /domain, managing principles, generation key tab file each service and managing key tab using key tab tools.
  • Experience in handling multiple relational databases: MySQL, SQL Server.
  • Familiar with Agile Methodology (SCRUM) and Software Testing.
  • Effective problem solving skills and outstanding interpersonal skills. Ability to work independently as well as within a team environment. Driven to meet deadlines. Ability to learn and use new technologies quickly.

TECHNICAL SKILLS

Hadoop Ecosystem: HDFS, SQOOP, FLUME, MAP-REDUCE, HIVE, PIG, OOZIE, ZOOKEEPER

NoSQL Database: Hbase, Cassandra

Security: Kerberos

Database: MySQL, SQL Server

Cluster management Tools: Cloudera Manager, Ambari

Os: LINUX (Centos, RHEL), windows, mac

PROFESSIONAL EXPERIENCE

Confidential, Irvine, CA

Hadoop Admin

Responsibilities:

  • Managing 5 Hortonworks cluster size of 1500 nodes altogether (Development, R&D, Discovery PROD, MARS HBase and MARS PROD)
  • Designed and Architected R&D cluster with HDP 2.3.2 andAmbari2.2.0
  • Worked on 4 different versions of HDP (1.3.2, 2.1.5, 2.2.6, 2.3.2 Latest Enterprise Release )
  • Upgraded HDP 1.3.2 and 2.1.5 to 2.2.6 using Blueprints. 2.2.6 to 2.3.2 using Rolling upgrade with no downtime to PROD Cluster
  • Configured Hadoop High Availability on Namenode, HBase, Hive, Yarn and Storm (Nimbus)
  • Configured Hadoop security Kerberos. Ranger and Knox for secured cluster
  • Configured HDFS data at rest Encryption using Ranger KMS
  • Configured Storm HA
  • Installed and configured Spark
  • Created kafka topics, produced and consumed messages
  • Cluster performance tuning
  • Setup 3 instance of zookeeper dedicated for HBase, Storm and kafka. 1st instance managed byAmbariand other 2 are out ofAmbari
  • Configured Apache Ranger centralized security and auditing for HDFS, YARN, HIVE, HBase, Storm and Kafka.
  • Installed and configured Informatica 9.6.1 HF1 Big Data Edition for Hadoop ETL
  • Commissioning and decommissioning of datanodes
  • Troubleshoot the issues reported by Nagios
  • Built and configured log data loading into HDFS using Flume.
  • Wrote shell script to monitor few components out ofAmbari
  • Performed Importing and exporting data into HDFS and Hive using Sqoop.
  • Provisioning, installing, configuring, monitoring, and maintaining HDFS, Yarn, HBase, Flume, Sqoop, Oozie, Pig, Hive
  • Recovering from node failures and troubleshooting common Hadoop cluster issues
  • Supporting Hadoop developers and assisting in optimization of map reduce jobs, Pig Latin scripts, Hive Scripts, and HBase ingest Required

Confidential, San Francisco, CA

Hadoop Admin

Responsibilities:

  • Designed and developed data solutions to help business and product teams make data driven decisions
  • Worked closely with data analysts to construct creative solutions for their analysis tasks
  • Lead end-to-end efforts to design, develop, and implement data warehousing and business intelligence solutions
  • Worked on performing major upgrade of cluster from CDH3u6 to CDH4.4.0
  • Developed Puppet modules to automated the installation, configuration and deployment of software, OS's and network infrastructure at a cluster level
  • Implemented Namenode HA and automatic failover infrastructure to overcome single point of failure for Namenode utilizing Zookeeper services
  • ImplementedClouderaManager on existing cluster
  • Optimized our Hadoop infrastructure at both the software and hardware level
  • Ensured our Hadoop clusters are built and tuned in the most optimal way to support the activities of our Big Data teams
  • Developed MapReduce programs to extract and transform the data sets and results were exported back to RDBMS using Sqoop
  • Installed, Configured and managed Flume Infrastructure
  • Was responsible for importing the data (mostly log files) from various sources into HDFS using Flume
  • Created tables in Hive and loaded the structured (resulted from MapReduce jobs) data
  • Configured Hive metastore to use MySQL Database, to make available all the tables created in Hive different users simultaneously.
  • Using HiveQL developed many queries and extracted the business required information.
  • Exported the business required information to RDBMS using Sqoop to make the data available for BI team to generate reports based on data.

Confidential, Jersey City, NJ

Hadoop Admin

Responsibilities:

  • Worked with the Linux administration team to prepare and configure the systems to support Hadoop deployment.
  • Performed various configurations, which includes, networking and IPTable, resolving hostnames, user accounts and file permissions, http, ftp, SSH key less login.
  • Implemented authentication service using Kerberos authentication protocol.
  • Created volume groups, logical volumes and partitions on the Linux servers and mounted file systems on the created partitions.
  • Master nodes disks are configured with RAID 1+0
  • Performed benchmarking on the Hadoop cluster using different benchmarking mechanisms.
  • Tuned the cluster by Commissioning and decommissioning the Data Nodes.
  • Upgraded the Hadoop cluster.
  • Implemented Fair scheduler on the job tracker to allocate the fair amount of resources to small jobs.
  • Deployed high availability on the Hadoop cluster quorum journal nodes.
  • Implemented automatic failover zookeeper and zookeeper failover controller.
  • Configured Ganglia which include installing GMOND and GMETAD daemons which collects all the metrics running on the distributed cluster and presents them in real-time dynamic web pages which would further help in debugging and maintenance.
  • Implemented Kerberos for authenticating all the services in Hadoop Cluster.
  • Deployed Network file system for Name Node Metadata backup.
  • Performed cluster back using DISTCP, Cloudera manager BDR and parallel ingestion.
  • Designed and allocated HDFS quotas for multiple groups.
  • Configured and deployed hive metastore using MySQL and thrift server.
  • Used hive schema to create relations in pig using Hcatalog.
  • Development of Pig scripts for handling the raw data for analysis.
  • Deployed Sqoop server to perform imports from heterogeneous data sources to HDFS.
  • Deployed and configured flume agents to stream log events into HDFS for analysis.
  • Performed deploying yarn, which facilitate multiple applications to run on the cluster.
  • Configured Oozie for workflow automation and coordination.
  • Custom monitoring scripts for Nagios to monitor the daemons and the cluster status.
  • Custom shell scripts for automating redundant tasks on the cluster.
  • Worked with BI teams in generating the reports and designing ETL workflows on Pentaho.

Environment: LINUX, HDFS, SQOOP, FLUME, MAP-REDUCE, HIVE, PIG, OOZIE, ZOOKEEPER

Confidential

Java Developer

Responsibilities:

  • Involved in the design and followed Agile Software Development Methodology throughout the software development lifecycle.
  • Designed Use Cases, Class Diagrams, and Sequence Diagrams using Visual Paradigm to model the detail design of the application.
  • Developed User Interface using JSP standard tags and Java script, HTML, CSS for Presentation layer.
  • Used the spring validation for Web Form Validation by implementing the Validator interface.
  • Application was built on Spring MVC framework and Hibernate as ORM
  • Used Spring-Core module for Dependency Injection and integrated view using Apace Tiles.
  • Consumed Web Services (WSDL, SOAP, UDDI) from third party for authorizing payments to/from customers using CXF Framework
  • Used JMS Queue communication in authorization module.
  • Mapped (one-to-many, one-to-one, many-to-one relations) DTOs to Oracle Database tables and Java data types to SQL data types by creating Hibernate mapping XML files
  • Oracle database was used, wrote stored procedures for common SQL queries
  • Used ANT for building the enterprise application modules, Used CVS for Version control, Log4J to monitor the error logs and performed unit testing using J Unit.
  • Deployed the applications on IBM Web Sphere Application Server 5.0.

Hire Now