We provide IT Staff Augmentation Services!

Hadoop Engineer Resume

0/5 (Submit Your Rating)

DE

SUMMARY

  • Over 7 years of IT Experience in working on Hadoop Cluster including cluster planning, designing, implementing, benchmarking, performance tuning and monitoring.
  • Experience in developing applications using core Java, Web Technologies along with data structures, collections, JDBC, Servlets, JSP, XML, HTML and LIFERAY portals.
  • Hands - on experience in Linux administration - user management, storage management, network management and shell scripting.
  • Strong knowledge on Hadoop HDFS architecture and Map Reduce Framework.
  • Experience in working with Hadoop ecosystem components: Hive, PIG, Sqoop, Flume, Zookeeper and Oozie.
  • Strong knowledge on NOSQL Databases: Cassandra 1.2.
  • Working knowledge in using Hortonworks Data Platform (HDP) for provisioning, Managing and operating Hadoop Cluster.
  • Strong understanding and working experience on injecting and accessing data to and from a Cassandra cluster.
  • Worked with different file formats: Text, Sequence and Avro file formats.
  • Experience in working with large volumes of streaming data using Flume.
  • Working knowledge in implementing business logic and optimize the queries using HiveQL by implementing partitioning and bucketing techniques.
  • Knowledge in handling custom user defined scripts (UDF’s) for non-regular data structure using Pig Latin.
  • Experience in automating Hadoop jobs using Oozie workflow engine.
  • Advanced Hands on Experience in troubleshooting operation issues and identifying root causes of a Hadoop cluster.
  • Exposure to handling Linux/Unix Shell Scripting for automation process.
  • Experience in Working on Cloud Environments like Amazon EC2, Microsoft Azure

TECHNICAL SKILLS:

Hadoop Technologies and Distributions: - Apache Hadoop, HDFS and MapReduce (CDH3 and CDH4).

Hadoop EcoSystem: - Hive, Pig, Sqoop, Flume, Oozie and ZooKeeper.

Languages & API's: - Java, J2EE, Servlets, Ruby, JSP, JSTL, Struts, Spring MVC, Web Services, SQL, Ajax, XML, HTML, JavaScript, Cascading Style Sheets, jQuery etc.

Programming Technologies: - Object Oriented Programming(OOP) .

Portal Server: - Liferay portal Server 5.x, 6.x

CMS: - Liferay CMS .

DBMS: - Oracle, MySQL, JDBC, Hibernate .

NoSQL Databases: - Apache Cassandra-1.2.

Application Server: - Tomcat 6.X

Source Repository: - SVN, GIT

Development IDE s: - Eclipse, NetBeans

Operating System: - Windows, Linux-CentOs and Ubuntu, Mac OS X.

Cloud Platforms: - Amazon AWS, Microsoft Azure

Virtual Machines: VMware Playe, VM WorkStation, Oracle VM Virtual Box.

PROFESSIONAL EXPERIENCE

Confidential, DE

Hadoop Engineer

Environment: - CDH5, Flume, Hive, Sqoop, PIG, Oozie, Cloudera Manager, JAVA, LINUX, CENTOS

Responsibilities:

  • Proactively monitored systems and services, architecture design and implementation of Hadoop deployment, configuration management and backup procedures.
  • Involved in Analyzing system failures to identify the root causes, and recommended course of actions.
  • Worked on expanding the cluster along with the Engineering team and deployed new Hadoop environments.
  • Performed operating system installation, Hadoop updates, patches and version upgrades as per requirement using automated tool.
  • Monitored workload, job performance and capacity planning using Cloudera Manager.
  • Worked on building a disaster recovery systems and procedures.
  • Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
  • Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
  • Performed Java: Map Reduce programs on log data to transform into a structured way to find user data.
  • Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most visited service on website.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and generated reports along with our BI team.
  • Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop)
  • Worked on system specific jobs to automate a process like shell scripts.
  • Documented the systems processes and procedures for future s.

Confidential, Richfield MN

Hadoop Engineer

Environment: - Flume, Oozie, Pig, Hive, MapReduce, MySQL, Cassandra, Cloudera Manager.

Responsibilities:

  • Designed, built and implemented a Hadoop cluster.
  • Configured remote hive metastore in a MySQL database to establish multiple connections from various clients simultaneously.
  • Implemented NFS for backing up the Name node metadata.
  • Designed and implemented security for Hadoop cluster with Kerberos security protocol.
  • Wrote shell scripts to monitor the health of Hadoop daemon services and act accordingly in case of a Warn/Fatal condition.
  • Responsible for Cluster maintenance, commissioning and decommissioning cluster nodes, cluster monitoring and troubleshooting, managing and reviewing data backups and Hadoop log files.
  • Worked on Map Reduce scripts to do the transformation on the extracted data for further analysis.
  • Synchronized the workflows for Flume, Hive, Pig and MapReduce jobs with the scheduler Oozie.
  • Worked on designing and implementing a Cassandra cluster.
  • Created the necessary keyspaces and modeled column families based on the queries.
  • Used DataStax Java driver to connect to the Cassandra cluster, load data in it and perform the required queries to extract data from the cluster.
  • Configured and managed backup and disaster recovery for Hadoop data using Amazon s3.

Confidential, Minneapolis MN

Associate Engineer - Hadoop

Environment: - Hadoop, MapReduce, Hive, Pig, Oozie, Sqoop, Linux.

Responsibilities:

  • Responsible for building scalable distributed data solutions using Hadoop.
  • Responsible for Cluster maintenance, Commissioning and Decommissioning cluster nodes, Cluster Monitoring and Troubleshooting, Manage and review data backups, Manage and review Hadoop log files.
  • Worked hands on with ETL process. Handled importing data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.
  • Extracted the data from Teradata into HDFS using the Sqoop.
  • Analyzed the data by performing Hive queries and running Pig scripts to know User behavior like Shopping Enthusiasts, Travelers, Auto Intenders, Music Lovers.
  • Exported the patterns analyzed back to Teradata using Sqoop.
  • Monitored and managed the Hadoop cluster using Nagios and Ganglia.
  • Installed Oozie workflow engine to run multiple Hive and Pig jobs that run independently with time and data availability.
  • Developed Hive queries to process the data and generate the data cubes for visualizing.

Confidential, St. Louis MO

Associate Engineer - Hadoop

Environment: - Hadoop, Hive, Pig, Flume, MapReduce, Sqoop, Maven

Responsibilities:

  • To extract Big data from various data sources into Hadoop HDFS. This included data from Excel, ERP systems, databases, CSV and also log data from sensors/meters.
  • Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers/sensors
  • Developed MapReduce programs to cleanse and parse data in HDFS obtained from various data sources.
  • Used Hive data warehouse tool to analyze the unified historic data in HDFS to identify issues and behavioral patterns.
  • The Hive tables created as per requirement were internal or external tables defined with proper static and dynamic partitions, intended for efficiency.
  • Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data.
  • Implemented Hive custom UDF’s to integrate the Weather and geographical data with business data to achieve comprehensive data analysis.
  • Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce, Hive and Sqoop as well as system specific jobs.
  • Worked along with the Hadoop Operations team in Hadoop cluster planning, installation, maintenance, monitoring and upgrades.

Software Engineer

Confidential

Responsibilities:

  • Worked in JavaScript and JSP’s for the client side as well as server side validation.
  • Created Data access objects to access the data from the database.
  • Worked in JavaScript and AlloyUI to make an AJAX calls.
  • Wrote SQL queries and stored procedures.
  • Created Custom Tags to separate business logic from UI.
  • Created and Worked on CSS classes to change front end.
  • Maintained the version control in repository.
  • Mentoring and helping members in the Project-team’.
  • Contributed in all phases of the development lifecycle.
  • Project: Matrimonial (Dating) Portal.
  • Architected, designed and Developed Portlets
  • Worked in JavaScript and AlloyUI to make an AJAX calls.
  • Customized the default theme according to Matrimonial theme.
  • Building, testing and deploying the application on Apache Tomcat server.
  • Created Data access objects to access the data from the MySQL database.
  • Wrote SQL queries and stored procedures
  • Maintaining Version Control of code.
  • Created Custom Tags to separate business logic from UI
  • Project: Online Examination Portal.
  • Primarily focused on design and development of Portlets
  • Worked in JavaScript and JSP’s for the client side as well as server side validation.
  • Building, testing and deploying the application on Apache Tomcat server.
  • Contributed in all phases of the development lifecycle.
  • Worked on Themes to modify the look and feel.
  • Wrote SQL queries and stored procedures

We'd love your feedback!