Hadoop Engineer Resume
Sunnyvale, CA
SUMMARY
- Over 6+ years of experience in software development, 2+ years of experience in all phases of Hadoop and HDFS development.
- Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement.
- Experienced on major Hadoop ecosystem’s projects such as PIG, HIVE and HBASE.
- Good working experience using Sqoop to import data into HDFS from RDBMS and vice - versa
- Good knowledge in using job scheduling and monitoring tools like Oozie and ZooKeeper
- Experience in Hadoop administration activities such as installation and configuration of clusters using Apache and Cloudera
- Experience in developing solutions to analyze large data sets efficiently
- Knowledge of administrative tasks such as installing Hadoop and its ecosystem components such as Hive and Pig
- Knowledge of NoSQL databases such as HBase, and MongoDB
- In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, JobTracker, TaskTracker, NameNode, Data Node and MapReduce concepts
- Extending Hive and Pig core functionality by writing customUDFs
- Good understanding of Data Mining and Machine Learning techniques
- Experience in analyzing data using HiveQL, Pig Latin, and custom MapReduce programs in Java.
- Strong work ethic with desire to succeed and make significant contributions to the organization
- Strong problem solving skills, good communication, interpersonal skills and a good team player
- Have the motivation to take independent responsibility as well as ability to contribute and be a productive team member
TECHNICAL SKILLS
Hadoop/Big Data Technologies: HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Hbase, Oozie, Zookeeper, Kerberos
Programming Languages: Java JDK1.4/1.5/1.6 (JDK 5/JDK 6), C/C++, HTML, SQL, PL/SQL, AVS & JVS
Frameworks: Hibernate 2.x/3.x, Spring 2.x/3.x,Struts 1.x/2.x
Web Services: WSDL, SOAP, Apache CXF/XFire, Apache Axis, REST, Jersey
Client Technologies: JQUERY, Java Script, AJAX, CSS, HTML 5, XHTML
Operating Systems: UNIX, Windows, LINUX
Application Servers: IBM Web sphere, Tomcat, Web Logic, Web Sphere
Web technologies: JSP, Servlets, JNDI, JDBC, Java Beans, JavaScript, Web Services(JAX-WS)
Databases: Oracle 8i/9i/10g & MySQL 4.x/5.x
Java IDE: Eclipse 3.x, IBM Web Sphere Application Developer, IBM RAD 7.0
Tools: TOAD, SQL Developer, SOAP UI, ANT, Maven, Visio, Rational Rose, Endur 8.x/10.x/11.x
PROFESSIONAL EXPERIENCE
Confidential, Sunnyvale CA
Hadoop Engineer
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Java (jdk 1.6), Eclipse
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop
- Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster
- Setup and benchmarked Hadoop/HBase clusters for internal use
- Developed Simple to complex Map/reduce Jobs using Hive and Pig
- Worked with Python and Python web framework
- Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop
- Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior
- Used UDF’s to implement business logic in Hadoop
- Implemented business logic by writing UDFs in Java and used various UDFs from Piggybanks and other sources.
- Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required
- Installed Oozie workflow engine to run multiple Hive and Pig jobs
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
Confidential, Dallas, TX
Hadoop Developer
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Hue, Oozie, Core Java, Eclipse, Hbase, Flume, Cloudera Manager, Oracle 10g, DB2, IDMS, VSAM, SQL*PLUS, Toad, Putty, Windows NT, UNIX Shell Scripting, Pentaho Kettle, Pentaho Bigdata, YARN
Responsibilities:
- Involved in writing Mapreduce programs and tested using MRUnit.
- Managing and scheduling Jobs on a Hadoop cluster using Oozie.
- Involved in moving all log files generated from various sources to HDFS for further processing through Flume.
- Involved in loading data from UNIX file system to HDFS.
- Worked on Hue interface for querying the data.
- Created Hive tables to store the processed results in a tabular format.
- Created HBase tables to store variable data formats of data coming from different portfolios.
- Involved in transforming data from Mainframe tables to HDFS, and HBASE tables using Sqoop and Pentaho Kettle.
- Implemented best income logic using Pig scripts.
- Implemented test scripts to support test driven development and continuous integration.
- Responsible to manage data coming from different sources.
- Experienced on loading and transforming of large sets of structured, semi structured and unstructured data.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Participate in requirement gathering and analysis phase of the project in documenting the business requirements by conducting workshops/meetings with various business users.
- Have deep and thorough understanding of ETL tools and how they can be applied in a Big Data environment.
Confidential
Java/J2EE Developer
Environment: Java, JDK 1.5, Servlets, Hibernate, Ajax, Oracle 10g, Eclipse, Apache Ant, Web Services (SOAP), Apache Axis, Apache Ant, Web Logic Server, JavaScript, HTML, CSS, XML
Responsibilities:
- Responsible for gathering and analyzing requirements and converting them into technical specifications
- Used Rational Rose for creating sequence and class diagrams
- Developed presentation layer using JSP, Java, HTML and JavaScript
- Used Spring Core Annotations for Dependency Injection
- Designed and developed a ‘Convention Based Coding’ utilizing Hibernate’s persistence framework and O-R mapping capability to enable dynamic fetching and displaying of various table data with JSF tag libraries
- Designed and developed Hibernate configuration and session-per-request design pattern for making database connectivity and accessing the session for database transactions respectively. Used HQL and SQL for fetching and storing data in databases
- Participated in the design and development of database schema and Entity-Relationship diagrams of the backend Oracle database tables for the application
- Implemented web services with Apache Axis
- Designed and Developed Stored Procedures, Triggers in Oracle to cater the needs for the entire application. Developed complex SQL queries for extracting data from the database
- Designed and built SOAP web service interfaces implemented in Java
- Used Apache Ant for the build process
- Used ClearCase for version control and ClearQuest for bug tracking
Confidential
Java/ J2EE Developer
Environment: Java, J2EE, XML, XML Schemas, JSP, HTML, CSS, PL/SQL, Junit, Log4j, IBM Web sphere Application Server.
Responsibilities:
- Involved in creation of UML diagrams like Class, Activity, and Sequence Diagrams using modeling tools of IBM Rational Rose
- Involved in the development of JSPs and Servlets for different User Interfaces
- Used Struts action forms and developed Action Classes, which act as the navigation controller in Struts framework
- Implemented the template-based categorization of presentation content using Struts-Tiles. MVC implementation using Struts framework
- Involved in Unit Testing of Various Modules based on the Test Cases
- Involved in Bug fixing of various modules that were raised by the Testing teams in the application during the Integration testing phase
- Involved and participated in Code reviews
- Used Log4J logging framework for logging messages
- Used Rational ClearCase for version control
- Used Rational Clear Quest for bug tracking
- Involved in deployment of application on IBM Websphere Application Server
