We provide IT Staff Augmentation Services!

Hadoop Developer Resume

0/5 (Submit Your Rating)

Orlando, FL

SUMMARY

  • Having 8 years of IT experience, including 3+ years hands on experience in Hadoop/Big Data technologies and 4+ years of extensive experience in JAVA, J2EE Technologies and Database development
  • Expertise in all components of Hadoop Ecosystem - Hive, Hue, Pig, Sqoop, HBase, Impala, Flume, Zookeeper, Oozie, Apache Nifi and Apache Spark.
  • Good Experience in Linux Bash scripting and following PEP Guidelines inpython.
  • Excellent understanding of Hadoop architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, YARN and MapReduce programming paradigm.
  • Experience in managing scalable Hadoop clusters including cluster designing, custom configurations, monitoring and maintaining of different Hadoop distributions like Cloudera CDH, Apache Hadoop, Hortonworks and Map Reduce.
  • Experience in creatingNifi flow to streaming data between Kafka, PulsarDB, ElasticSearch, and FTP. And also deploying those Nififlows both into staging and production.
  • Experience in writing Pig Latin, HiveQL scripts and extended their functionality using User Defined Functions (UDF’s).
  • Experience in analyzing data usingHiveQL, Pig Latin, HBase and custom MapReduce programs in Java.
  • Experience in handling structured arrangement of data within certain limits (Data Layout’s) using Partitions and Bucketing in Hive.
  • Experience in Hadoop Shell commands, writing Map Reduceprograms, verifying, managing and reviewing Hadoop log files.
  • Knowledge on NOSQL databases like MongoDB, HBase, Cassandra and its Integration with Hive.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
  • Experience in collecting the log data from different sources like (webservers and social media) using Flume and storing in HDFS to perform MapReduce jobs.
  • Integrated BI tool with Impala and analyzed the data.
  • Experience in installation, configuration, supporting and managing- Cloudera'sHadoop platform along with CDH3 & CDH4 clusters.
  • Experience with Cloudera Impala for ad-hoc querying and Apache sentry for role based authentication.
  • Strong team player with good communication, analytical, presentation and inter-personal skills.

TECHNICAL SKILLS

Programming Languages: SQL, C, C++, Java, Core Java, J2EE, Python, Pig Latin, HiveQL and Unix shell scripting

Big Data Technologies: Hadoop, HDFS, Map Reduce, Hive, Pig, HBase, Impala, Hue, Sqoop, Storm, Kafka, Oozie, Flume, Zookeeper, Nifi, Cassandra, Spark 1.4, Cloudera CDH4, and Horton works.

Databases: Oracle, MySQL, SQL Server, DB2, Familiar with NoSQL (HBase, Cassandra, MongoDB)

Scripting and Query Languages: UNIX Shell scripting, SQL and PL/SQL.

Web Technologies: JSP, Servlets, JavaBeans, JDBC, AWT, Swing, JSF, XML, CSS, HTML, XHTML, JavaScript, AJAX.

Hadoop Paradigms: MapReduce, YARN, In-memory computing, High Availability, Real-time Streaming.

Operating Systems: Windows,UNIX, Linuxdistributions (Centos, Ubuntu), Mac OS

Other Tools: Eclipse, Informatica, Tableau 9.0, JUnit, QTP, JIRA, QC (Quality Center)

PROFESSIONAL EXPERIENCE

Confidential, Orlando, FL

Hadoop Developer

Responsibilities:

  • Experience in developing solutions to analyze large data sets efficiently
  • Developed Map Reduce application to find out the useful metrics from the data.
  • Did a thorough testing in local mode and distributed mode found bugs with the code and ensured 100% issue free delivery to production.
  • Experienced in query tuning.
  • Implemented Kafka to handle real time data.
  • Worked on spark sql and spark streaming.
  • Implemented multiple sqoop jobs.
  • Written Ozzie work flows to automate jobs.
  • Written multiple python scripts for data visualizations and ETL automations
  • Expert level understanding of Map Reduce internals, including shuffling and partitioning.
  • Created Hive external tables and managed tables, designed data models in hive.
  • Implemented business logic using Pig and hive udf’s.
  • Finding the right joins logics and creates valuable data sets for further data analysis.
  • Responsible to develop custom udf’s in pig, hive.
  • Developed multiple pig and hive jobs in Java for data cleaning and processing.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Worked hands on with ETL process.
  • Handled importing of data from various data sources, performed transformations using Hive,
  • MapReduce, and loaded data into HDFS.
  • Extracted the data from multiple data sources into HDFS using Sqoop.
  • Developed Hive queries to process the data and generate the data cubes for visualizing.

Environment: Linux, MapReduce, HDFS, Hive, Ooozie, Java (jdk1.6), eclipse, HBase (NoSQL), Sqoop, pig.

Confidential - San Jose, CA

Big Data - Hadoop Developer

Responsibilities:

  • Involved in Installing, Configuring Hadoop components using CDH 5.2 Distribution.
  • Responsible for analyzing large data sets and derive customer usage patterns by developing new MapReduce programs.
  • Written MapReduce code to parse the data from various sources and storing parsed data into Hbase and Hive.
  • Worked on creating combiners, partitions, and distributed cache to improve the performance of MapReduce jobs.
  • Developed Shell Script to perform data profiling on the ingested data with the help of HIVE Bucketing.
  • Migrate infrastructure services to new data center, e.g. Elastic search, Hbase, Hive.
  • Migrated Existing MapReduce programs to Spark Models using python.
  • Leveraged spark(Pyspark) to manipulate unstructured data and apply text mining on user's table utilization data.
  • Use to collect, conduct and curate real time data by moving it from source to destination by ApacheNifi.
  • Experience in sequence data pre-processing, extraction, model fitting and validation using ML pipelines.
  • Designed Data Quality Framework to perform schema validation and data profiling onspark(Pyspark).
  • Responsible for debug, optimization of Hive scripts and also implementing DE duplication logic in Hive using a rank key function (UDF).
  • Experienced in writing Hive validation scripts that are used in validation framework (for daily analysis through graphs and presented to business users).
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Developed workflow in Oozie to automate the tasks of loading data into HDFS and pre-processing with Pig and Hive.
  • Used Pig as ETL tool to do transformations, joins and some pre-aggregations before storing the data into HDFS.
  • Imported all the customer specific personal data to Hadoop using Sqoop component from various relational databases like Netezza and Oracle.
  • Used Impala to read, write and query the Hadoop data in HDFS and Hbase.
  • Experienced in running query using Impala and used BI tools to run ad-hoc queries directly on Hadoop.
  • Worked with BI teams in generating the reports and designing ETL workflows on Tableau.
  • Set up Logstash ELK stack and created dashboards to monitor marketing emails status.
  • Develop testing scripts in Python and prepare test procedures, analyze test results data and suggest improvements of the system and software.
  • Experience in streaming log data using Flume and data analytics using Hive.
  • Developed a data pipeline using Kafka and Storm to store data into HDFS.
  • Extracted the data from RDBMS (Oracle, MySQL & Teradata) to HDFS using Sqoop.

Environment: Hadoop, MapReduce, HDFS, Pig, Hive QL, HBase, Zookeeper, Oozie, Flume, Impala, Cloudera 5.2, Storm, MySQL, UNIX Shell Scripting, Nifi, Tableau, Python, Spark.

Confidential, Austin, TX

Hadoop Developer

Responsibilities:

  • Responsible for loading the customer’s data and event logs from Oracle database, Teradata into HDFS.
  • Developed MapReduce jobs for Log Analysis, Analytics and to generate reports for the number of activities created on particular day.
  • Involved in implementing complex MapReduce programs to perform joins on the Map side using distributed cache in Java.
  • Used Spark DataFrame API over Cloudera platform to perform analytics on hive data.
  • Implemented Apache Spark data processing project to handle data from RDBMS and streaming sources.
  • Involved in creating Hive tables, loading with data and writing Hive queries, which will run internally in MapReduce way.
  • Complete end to end design and development of Apache Nifi flow which acts as the agent between middleware team and EBI team and executes all the actions mentioned above.
  • Implemented different analytical algorithms using MapReduce programs to apply on top of HDFS data.
  • Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access.
  • Utilized Apache Hadoop ecosystem tools like HDFS, Hive and Pig for large datasets analysis.
  • Developed Pig and Hive UDF's to analyze the complex data to find specific user behavior.
  • Designed a data warehouse using Hive and created Partitioned tables in Hive.
  • Experienced in using Pig for data cleansing and developed Pig Latin scripts to extract the data from web server output files to load into HDFS.
  • Load and transform large data sets of structured, semi structured and unstructured data.
  • Designed and modified database tables and used HBASE Queries to insert and fetch data from tables.
  • Developed code in Scala & Python to use Map Reduce framework by Hadoop streaming.
  • Worked on Hive by creating external and internal tables, loading it with data and written Hive queries.
  • Created HBase tables to store data from different sources.
  • Worked with various Hadoop file formats, including Text Files, Sequence File, RCFile.
  • Create/Modify shell scripts for scheduling various data cleansing scripts and ETL load process.
  • Configured Zookeeper for Cluster co-ordination services.

Environment: Hadoop, MapReduce, HDFS, Pig, Hive, HBase, Zookeeper, Nifi, Oozie, Horton works, Python, Scala, Oracle 11g/10g, Windows NT,UNIX Shell Scripting.

Confidential, Santa Clara, CA

Sr. Java/J2EE Developer

Responsibilities:

  • Involved in Full Life Cycle Development in Distributed Environment using Java and J2EE framework.
  • Designed the application by implementing Struts Framework based on MVC Architecture.
  • Designed and developed the front end using JSP, HTML and JavaScript and JQuery.
  • Implemented the Web Service client for the login authentication, credit reports and applicant information Apache Axis 2 Web Service.
  • Extensively worked on User Interface for few modules using JSPs, JavaScript and Ajax.
  • Developed framework for data processing using Design patterns, Java, XML.
  • Used the lightweight container of the Spring Framework to provide architectural flexibility for Inversion of Controller(IOC).
  • Used Hibernate ORM framework with Spring framework for data persistence and transaction management.
  • Designed and developed Session beans to implement the Business logic.
  • Developed EJB components that are deployed on Web logic Application Server.
  • Written unit tests using JUnit Framework and Logging is done using Log4J Framework.
  • Designed and developed various configuration files for Hibernate mappings.
  • Designed and documented REST/HTTP APIs, including JSON data formats and API versioning strategy.
  • Developed Web Services for sending and getting data from different applications using SOAP messages.
  • Actively involved in code reviews and bug fixing.
  • Applied CSS(Cascading style Sheets) for entire site for standardization of the site.
  • Assisted QA Team in defining and implementing a defect resolution process including defect priority, and severity.

Environment: Java 5.0, Struts, Spring 2.0, Hibernate 3.2, WebLogic7.0, Eclipse 3.3, Oracle 10g, JUnit 4.2,Maven, Windows XP, HTML, CSS, JavaScript, and XML.

Confidential

Java Developer

Responsibilities:

  • Involved in the analysis & design of the application using Rational Rose.
  • Developed and implemented the MVC Architectural pattern using Struts Framework including JSP, Servlets, EJB, From Bean and Action classes.
  • Object Oriented Analysis and Design using UML include development of class diagrams, Sequence diagrams and State diagrams and implemented these diagrams in Microsoft Visio.
  • Designed and created Java Objects, JSP pages, JSF, JavaBeans and Servlets to achieve various business functionalities.
  • Created validation methods using JavaScript and Backing Beans.
  • Involved in writing client side validations using JavaScript, CSS.
  • Involved in the design of the Referential Data Service module to interface with various databases using JDBC.
  • Designed and developed the UI using Struts view components JSP, HTML, CSS and JavaScript.
  • Implemented Java/J2EE Design Patterns like Business Delegate and Data Transfer Object (DTO), Data Access Object and service locator.
  • Developed JMS API using J2EE package.
  • Used Oracle as Database and used Toad for queries execution and also involved in writing SQL scripts, PL/SQL code for procedures and functions.
  • Involved in designing test plans, test cases and overall Unit testing of the system.
  • Prepared documentation and participated in preparing user's manual for the application.

Environment: Java, Rational Rose, Spring 2.0, Hibernate 3.2, WebLogic, Eclipse, JSP, Windows XP, HTML, CSS, JavaScript, and XML.

Confidential

Programmer Analyst/ SQL Developer

Responsibilities:

  • Gathered requirements from client, analyzed and prepared the requirement specification document.
  • Working Knowledge of relational database concepts like entities, attributes, relationships, data integrity.
  • Involved in writing SQL queries.
  • Generated database SQL Scripts and deployed databases including installation and configuration.
  • Developed SQL Scripts to Insert/Update and Delete data in MS SQL database tables.
  • Experience in writing PL/SQL and in developing and implementing Stored Procedures, Packages and Triggers.
  • Experience on modeling principles, database design and programming, creating E-R diagrams and data relationships to design a database.
  • Build data connection to the database using MS SQL Server.
  • Used different joins, sub queries and nested querying.
  • Worked with different sources such as Oracle, SQL and Flat files.
  • Worked on project to extract data from xml file to SQL table and generate data file reporting using SQL Server 2008.

Environment: My SQL, SQL Server 2008(SSRS & SSIS), Visual studio 2000/2005, MS Excel.

We'd love your feedback!