Hadoop Engineer Resume
DE
SUMMARY
- Over 7 years of IT Experience in working on Hadoop Cluster including cluster planning, designing, implementing, benchmarking, performance tuning and monitoring.
- Experience in developing applications using core Java, Web Technologies along with data structures, collections, JDBC, Servlets, JSP, XML, HTML and LIFERAY portals.
- Hands - on experience in Linux administration - user management, storage management, network management and shell scripting.
- Strong knowledge on Hadoop HDFS architecture and Map Reduce Framework.
- Experience in working with Hadoop ecosystem components: Hive, PIG, Sqoop, Flume, Zookeeper and Oozie.
- Strong knowledge on NOSQL Databases: Cassandra 1.2.
- Working knowledge in using Hortonworks Data Platform (HDP) for provisioning, Managing and operating Hadoop Cluster.
- Strong understanding and working experience on injecting and accessing data to and from a Cassandra cluster.
- Worked with different file formats: Text, Sequence and Avro file formats.
- Experience in working with large volumes of streaming data using Flume.
- Working knowledge in implementing business logic and optimize the queries using HiveQL by implementing partitioning and bucketing techniques.
- Knowledge in handling custom user defined scripts (UDF’s) for non-regular data structure using Pig Latin.
- Experience in automating Hadoop jobs using Oozie workflow engine.
- Advanced Hands on Experience in troubleshooting operation issues and identifying root causes of a Hadoop cluster.
- Exposure to handling Linux/Unix Shell Scripting for automation process.
- Experience in Working on Cloud Environments like Amazon EC2, Microsoft Azure
TECHNICAL SKILLS:
Hadoop Technologies and Distributions: - Apache Hadoop, HDFS and MapReduce (CDH3 and CDH4).
Hadoop EcoSystem: - Hive, Pig, Sqoop, Flume, Oozie and ZooKeeper.
Languages & API's: - Java, J2EE, Servlets, Ruby, JSP, JSTL, Struts, Spring MVC, Web Services, SQL, Ajax, XML, HTML, JavaScript, Cascading Style Sheets, jQuery etc.
Programming Technologies: - Object Oriented Programming(OOP) .
Portal Server: - Liferay portal Server 5.x, 6.x
CMS: - Liferay CMS .
DBMS: - Oracle, MySQL, JDBC, Hibernate .
NoSQL Databases: - Apache Cassandra-1.2.
Application Server: - Tomcat 6.X
Source Repository: - SVN, GIT
Development IDE s: - Eclipse, NetBeans
Operating System: - Windows, Linux-CentOs and Ubuntu, Mac OS X.
Cloud Platforms: - Amazon AWS, Microsoft Azure
Virtual Machines: VMware Playe, VM WorkStation, Oracle VM Virtual Box.
PROFESSIONAL EXPERIENCE
Confidential, DE
Hadoop Engineer
Environment: - CDH5, Flume, Hive, Sqoop, PIG, Oozie, Cloudera Manager, JAVA, LINUX, CENTOS
Responsibilities:
- Proactively monitored systems and services, architecture design and implementation of Hadoop deployment, configuration management and backup procedures.
- Involved in Analyzing system failures to identify the root causes, and recommended course of actions.
- Worked on expanding the cluster along with the Engineering team and deployed new Hadoop environments.
- Performed operating system installation, Hadoop updates, patches and version upgrades as per requirement using automated tool.
- Monitored workload, job performance and capacity planning using Cloudera Manager.
- Worked on building a disaster recovery systems and procedures.
- Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster.
- Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
- Performed Java: Map Reduce programs on log data to transform into a structured way to find user data.
- Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most visited service on website.
- Exported the analyzed data to the relational databases using Sqoop for visualization and generated reports along with our BI team.
- Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop)
- Worked on system specific jobs to automate a process like shell scripts.
- Documented the systems processes and procedures for future s.
Confidential, Richfield MN
Hadoop Engineer
Environment: - Flume, Oozie, Pig, Hive, MapReduce, MySQL, Cassandra, Cloudera Manager.
Responsibilities:
- Designed, built and implemented a Hadoop cluster.
- Configured remote hive metastore in a MySQL database to establish multiple connections from various clients simultaneously.
- Implemented NFS for backing up the Name node metadata.
- Designed and implemented security for Hadoop cluster with Kerberos security protocol.
- Wrote shell scripts to monitor the health of Hadoop daemon services and act accordingly in case of a Warn/Fatal condition.
- Responsible for Cluster maintenance, commissioning and decommissioning cluster nodes, cluster monitoring and troubleshooting, managing and reviewing data backups and Hadoop log files.
- Worked on Map Reduce scripts to do the transformation on the extracted data for further analysis.
- Synchronized the workflows for Flume, Hive, Pig and MapReduce jobs with the scheduler Oozie.
- Worked on designing and implementing a Cassandra cluster.
- Created the necessary keyspaces and modeled column families based on the queries.
- Used DataStax Java driver to connect to the Cassandra cluster, load data in it and perform the required queries to extract data from the cluster.
- Configured and managed backup and disaster recovery for Hadoop data using Amazon s3.
Confidential, Minneapolis MN
Associate Engineer - Hadoop
Environment: - Hadoop, MapReduce, Hive, Pig, Oozie, Sqoop, Linux.
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop.
- Responsible for Cluster maintenance, Commissioning and Decommissioning cluster nodes, Cluster Monitoring and Troubleshooting, Manage and review data backups, Manage and review Hadoop log files.
- Worked hands on with ETL process. Handled importing data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.
- Extracted the data from Teradata into HDFS using the Sqoop.
- Analyzed the data by performing Hive queries and running Pig scripts to know User behavior like Shopping Enthusiasts, Travelers, Auto Intenders, Music Lovers.
- Exported the patterns analyzed back to Teradata using Sqoop.
- Monitored and managed the Hadoop cluster using Nagios and Ganglia.
- Installed Oozie workflow engine to run multiple Hive and Pig jobs that run independently with time and data availability.
- Developed Hive queries to process the data and generate the data cubes for visualizing.
Confidential, St. Louis MO
Associate Engineer - Hadoop
Environment: - Hadoop, Hive, Pig, Flume, MapReduce, Sqoop, Maven
Responsibilities:
- To extract Big data from various data sources into Hadoop HDFS. This included data from Excel, ERP systems, databases, CSV and also log data from sensors/meters.
- Used Sqoop to efficiently transfer data between databases and HDFS and used Flume to stream the log data from servers/sensors
- Developed MapReduce programs to cleanse and parse data in HDFS obtained from various data sources.
- Used Hive data warehouse tool to analyze the unified historic data in HDFS to identify issues and behavioral patterns.
- The Hive tables created as per requirement were internal or external tables defined with proper static and dynamic partitions, intended for efficiency.
- Used the RegEx, JSON and Avro SerDe’s for serialization and de-serialization packaged with Hive to parse the contents of streamed log data.
- Implemented Hive custom UDF’s to integrate the Weather and geographical data with business data to achieve comprehensive data analysis.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types of Hadoop jobs such as Java map-reduce, Hive and Sqoop as well as system specific jobs.
- Worked along with the Hadoop Operations team in Hadoop cluster planning, installation, maintenance, monitoring and upgrades.
Software Engineer
Confidential
Responsibilities:
- Worked in JavaScript and JSP’s for the client side as well as server side validation.
- Created Data access objects to access the data from the database.
- Worked in JavaScript and AlloyUI to make an AJAX calls.
- Wrote SQL queries and stored procedures.
- Created Custom Tags to separate business logic from UI.
- Created and Worked on CSS classes to change front end.
- Maintained the version control in repository.
- Mentoring and helping members in the Project-team’.
- Contributed in all phases of the development lifecycle.
- Project: Matrimonial (Dating) Portal.
- Architected, designed and Developed Portlets
- Worked in JavaScript and AlloyUI to make an AJAX calls.
- Customized the default theme according to Matrimonial theme.
- Building, testing and deploying the application on Apache Tomcat server.
- Created Data access objects to access the data from the MySQL database.
- Wrote SQL queries and stored procedures
- Maintaining Version Control of code.
- Created Custom Tags to separate business logic from UI
- Project: Online Examination Portal.
- Primarily focused on design and development of Portlets
- Worked in JavaScript and JSP’s for the client side as well as server side validation.
- Building, testing and deploying the application on Apache Tomcat server.
- Contributed in all phases of the development lifecycle.
- Worked on Themes to modify the look and feel.
- Wrote SQL queries and stored procedures
