We provide IT Staff Augmentation Services!

Hadoop Developer Resume

0/5 (Submit Your Rating)

Midland, MI

SUMMARY

  • Over 8 Years of experience in the field of Information Technology which includes a major concentration on Big Data Tools and Technologies, various Relational Databases and NoSQL Databases, Java Programming language and J2EE technologies with highly recommended software practices.
  • Experience with Cloudera Manager Administration and MonitorHadoopcluster using Cloudera Manager and Apache Ambari.
  • Expertise in installing, designing, sizing, configuring, provisioning and upgrading Hadoop environments.
  • Excellent understanding ofHadooparchitecture and core components such as Master Node (Name Node), Secondary Node (Data Node).
  • Good experience in both MapReduce MRv1 and MapReduce MRv2 (YARN).
  • Extensive experience in developing application using Scala, Python, Java and android.
  • Experience in differentHadoop distributions like Cloudera (Cloudera distribution CDH 4 and 5) and knowlede of Horton Works Distributions (HDP).
  • Hands on experience in developing application using Hadoop ecosystem like Spark, Hadoop MapReduce, HDFS, Yarn, Pig, Hive, Sqoop, Oozie, Avro, HBase, Zookeeper, Flume, Hue, Kafka and Storm.
  • Extensive experience in testing, debugging and deploying MapReduce Hadoop platforms.
  • Worked with Spark engine to process large scale data and experience to create Spark RDD.
  • Expert in creating Hive tables and write Hive queries to do analysis of HDFS data.
  • Good experiences in creating Pig and Hive UDFs using Java in order to analyze the data efficiently.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational Database systems (RDBMS) and vice - versa.
  • Successfully loaded files to Hive and HDFS from Oracle and SQL Server using SQOOP.
  • Hands on NoSQL database experience with MongoDb, HBase and Cassandra.
  • Excellent relational database (RDBMS) experience in Oracle, MySQL, SQL Server.
  • Extensive experience in SQL (Structure Query Language) and PL/SQL - Stored Procedure, Trigger, Sequence, Index.
  • Experience in creating Map Reduce codes in Java as per the business requirements.
  • Extensive experience in developing Java application using Spring MVC, Spring Restful web service, Struts2, JSP (java server pages), Servlet, ORM(Object Relational mapping), Hibernate, Core java and Swing.
  • Strong experience in IO, Bean, String, JDBC,JSTL,HTML, AngularJS, Multithreading, JavaScript, Ajax, CSS, jQuery, Collections, JSON, XML and auto building tool Jenkins.
  • Excellent experience in developing web-base and desktop report using Jasper Report tool.
  • Extensively worked on Amazon web service (AWS) using difference services like EC2, S3, Relational Database Service (RDS), DynamoDB, Elastic load balancing (ELB), Auto scaling, Elastic Block Store (EBS), Elastic MapReduce (EMR).
  • Good working knowledge on Eclipse IDE for developing and debugging Java applications.
  • Experience in using version control tools like Subversion (SVN), GIT
  • Experience in working with software methodologies like Agile and Waterfall.
  • Thorough knowledge of Software Development Life Cycle (SDLC) with deep understanding of various phases like Requirements gathering, Analysis, Design, Development and Testing.

TECHNICAL SKILLS

Big Data Ecosystem: HDFS, Sqoop, Flume, Hive, Pig, MapReduce, YARN, Oozie, Airflow, Kafka, Spark, Impala, Storm, Ambari, Zookeeper, Parquet, Java, SQL.

Hadoop Distributions: Hortonworks, Cloudera, Amazon

Databases & NOSQL Databases: Oracle, MYSQL, Microsoft SQL Server, HBase

RDBMS: Teradata, Oracle Pl/SQL, MS SQL Server, MySQL and DB2

Operating Systems: Linux, UNIX, Windows.

Development Methodologies: Agile/Scrum, Waterfall.

IDE's: Eclipse, Net Beans, GitHub, Jenkins, Maven, IntelliJ.

Programming Languages: C, C++, XML, HTML, JavaScript, Web services, Python, Scala, PL/SQL & Shell Scripting.

Reporting tools: Tableau

PROFESSIONAL EXPERIENCE

Confidential, Midland, MI

Hadoop developer

Responsibilities:

  • Creating end to end Spark applications using python to perform various data cleansing, validation, transformation and summarization activities on user behavioral data.
  • Developed custom Input Adaptor utilizing the HDFS File system API to ingest click stream log files from FTP server to HDFS.
  • Developed end-to-end data pipeline using FTP Adaptor, Spark, Hive.
  • Used python to write code for all Spark use cases.
  • Implemented design patterns in python for the application.
  • Implemented Spark using python utilized Spark SQL heavily for faster development, and processing of data.
  • Exploring with Spark for improving the performance and optimization of the existing algorithms inHadoopusing Spark Context, Spark-SQL, Data Frame, Pair RDD's, YARN.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Java and Scala.
  • Used Scala collection framework to store and process the complex consumer information.
  • Implemented a prototype to perform Real time streaming the data using Spark Streaming with Kafka.
  • Handled importing other enterprise data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and then loading data into HBase tables.
  • Exported the analyzed data to the relational databases using Sqoop, to further visualize and generate reports for the BI team.
  • Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysisAnalyzed the data by performing Hive queries (Hive QL) to study customer behavior.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Created components like Hive UDFs for missing functionality in HIVE for analytics.
  • Worked on various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in Hive and Map Side joins.
  • Created validate and maintain scripts to load data using Sqoop manually.
  • Created Oozie workflows and coordinators to automate Sqoop jobs weekly and monthly.
  • Implemented POC Spark Cluster on AWS.
  • Uploaded and processed more than 30 terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
  • Used Reporting tools like to connect with Hive for generating daily reports of data.
  • Created reports for the BI team using Sqoop to export data into HDFS and Hive.
  • Used Oozie and Oozie coordinators to deploy end-to-end data processing pipelines and scheduling the workflows.
  • Continuous monitoring and managing theHadoopcluster.
  • Developed interactive shell scripts for scheduling various data cleansing and data loading process.
  • Performed data validation on the data ingested using Spark by building a custom model to filter all the invalid data and cleanse the data.
  • Experience with data wrangling and creating workable datasets.

Environment: HDFS, Pig, Hive, Sqoop, Flume, Spark, MapReduce, Scala, Tableau, Oozie, Oracle 11g, YARN, UNIX Shell Scripting, Agile Methodology

Confidential, Nashville, TN

Hadoop Developer

Responsibilities:

  • Created staging tables and ingested data as dynamic partitions in Hive.
  • Imported data using Sqoop from the MySQL tables. Developed MapReduce jobs in Java API to parse the raw data and store the refined data.
  • Developed custom Pig & Hive UDFs in Java for transforming the data as per the requirement for analysis.
  • Supported Map Reduce Programs that are running on the cluster.
  • Worked on Tableau for exposing data for further analysis and for generating transforming files from different analytical formats to text files.
  • Integrated the Hive warehouse with HBase.
  • Cluster monitoring, maintenance and troubleshooting
  • Processing this data using Spark Streaming API withScala.
  • Involved in loading data from UNIX file system to HDFS.
  • Developed Spark scripts using Python as per the requirement.
  • Used Spark API over ClouderaHadoopYARN to perform analytics on data in Hive.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
  • Worked with Avro Data Serialization system to work with JSON data formats.
  • Performed data validation on the data ingested using MapReduce by building a custom model to filter all the invalid data and cleansing the data.
  • Created HBase tables of large data sets of structured and unstructured data.
  • Used Flume to collect, aggregate, and store the log data from different web servers.
  • Worked collaboratively with all levels of business stakeholders to architect, implement and test Big Data based analytical solution from disparate sources.
  • Responsible to manage data coming from different sources.
  • Managed and reviewedHadooplog files.
  • Mentored analyst team in writing required Hive Queries.

Environment: OEM, CDH 5.0.6, HDFS, MapReduce, HBase, Tableaus, Java, Python Hive, Flume, Pig, Spark, Scala, SQL and Sqoop.

Confidential, Pleasanton, CA

Hadoop Developer

Responsibilities:

  • Involved in creating Hive tables, loading with data and writing hive queries which will run internally in map reduce way.
  • Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
  • Involved in Installing, Configuring HadoopEco System, and Cloudera Manager using CDH5 Distribution.
  • Good understanding and related experience withHadoopstack - internals, Hive, Pig and MapReduce.
  • Wrote MapReduce jobs in python to discover trends in data usage by users.
  • Involved in defining job flows.
  • Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
  • Involved in managing and reviewingHadooplog files.
  • Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
  • Involved in runningHadoopstreaming jobs to process terabytes of text data.
  • Load and transform large sets of structured, semi structured and unstructured data.
  • Responsible to manage data coming from different sources.
  • Installed and configured Hive and written Hive QL scripts.
  • Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
  • Experienced with working on Avro Data files using Avro Serialization system.
  • Solved small file problem using Sequence files processing in Map Reduce.
  • Monitor System health and logs and respond accordingly to any warning or failure conditions.
  • Performed cluster co-ordination through Zookeeper.
  • Used ClearCase for version control.
  • Used JUnit for unit testing and Continuum for integration testing.

Environment: Hadoop, MapReduce, HDFS, Hive, PIG, Linux/Ubuntu, Java (jdk1.6), Datamining,Hadoop distribution of Cloudera, HBase, Map R, UNIX Shell Scripting, Clear case, JUnit.

Confidential

Big Data Developer

Responsibilities:

  • Involved in collecting, aggregating and moving data from servers to HDFS using Apache Flume.
  • Performed Hive jobs to parse logs and structure them in tables to get effective querying on the log data.
  • Implemented the workflows using Apache Oozie framework to automate tasks.
  • Worked in Loading and transforming large sets of structured, semi structured and unstructured data.
  • Experienced in managing and reviewingHadooplog files.
  • Imported Data from Different Relational Data Sources like RDBMS, Teradata to HDFS using Sqoop.
  • Worked with Avro Data Serialization system to work with JSON data formats.
  • Worked on different file formats like Sequence files, XML files and Map files using MapReduce programs.
  • Involved in Unit testing and delivering Unit test plans &results documentation using Junit and MR-Unit.
  • Exported data from HDFS to RDBMS using Sqoop for report generation and data visualization purpose.
  • Developed scripts and automated data management from end to end and sync up between all the clusters.
  • Created and maintained technical documentation for launchingHadoopCluster.

Environment: Hadoop, HDFS, MapReduce, Hive, Oozie, Sqoop, Pig, MySQL, Java, Rest API, Maven, MRUnit, Junit

We'd love your feedback!