We provide IT Staff Augmentation Services!

Hadoop Developer Resume

5.00/5 (Submit Your Rating)

Plano, TX

SUMMARY:

  • Overall 6+ years of experience in Software applications development including Analysis, Design, Development, Integration, Testing and Maintenance of various applications using JAVA/J2EE technologies
  • Experienced developing big data applications in cloud.
  • Configured Spark Streaming to receive real time data from the Apache Kafka and by using Scala stored the stream data to HDFS.
  • Experienced in building highly scalable Big - data solutions using Hadoop and multiple distributions i.e., Cloudera, Hortonworks and NoSQL platforms (Hbase& Cassandra).
  • Experience in writing custom MapReduce programs in Java and Python, also used HiveQL, Pig Latin for analyzing the data.
  • Expertise in Big data architecture with Hadoop File system and its eco system tools MapReduce, HBase, Hive, Agile, Pig, Zookeeper, Oozie, Flume, Avro, Impala, Apache spark and Spark Streaming and Spark SQL.
  • Hands on experience in Apache Sqoop, Apache Storm and Apache Hive integration.
  • Experienced in using different compression techniques while storing data into Hive tables for performance improvement.
  • Hands on experience working with different File Formats like TEXTFILE, JSON, AVROFILE, ORC for Hive Querying and Processing.
  • Experience on Apache Kafka, used for Messaging broker, Log Aggregation and Stream processing.
  • Expertise in migration data from different databases (i.e. Oracle, DB2, Teradata) to HDFS.
  • Developed Spark applications using Scala for easy Hadoop transitions.
  • Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala.
  • Used Flume extensively in gathering and moving log data files from Application Servers to a central location in Hadoop File System (HDFS).
  • Experience in designing and coding web applications using Core Java & Web Technologies- JSP, Servlets and JDBC.
  • Experience in designing the User Interfaces using HTML, CSS, JavaScript and JSP.
  • Experience in version control tools like Git.
  • Using Apache Flume collected logs data from web servers and integrated into HDFS.

TECHNICAL SKILLS:

Big Data Technologies: HDFS, MapReduce, Hive, Pig, Sqoop, Flume, Oozie, Storm, Zookeeper, Kafka, Impala, HCatalog, Apache Spark, Spark Streaming, Spark SQL, Hbase and Cassandra

Hadoop Distributions: Cloudera (CDH3/CDH4/CDH5), Hortonworks (HDP/HDF).

Databases: DB Oracle, MySQL, Teradata, HBase and Cassandra

Programming Languages: SQL, Scala, Java, Python, Unix Shell Scripting

Java Technologies: JDBC, Servlets, JSP, Spring and Hibernate

Operating System: Windows XP/7/8, Linux Distro(Ubuntu), Cent OS.

Tools: & Utilities: HP Quality Center, Git, Maven.

PROFESSIONAL EXPERIENCE:

Confidential, Plano, TX

Hadoop Developer

Responsibilities/Roles:

  • Worked on designing multiple NIFI flows to ingest data from different sources like Cloudant (CouchDB), Sql Server to Data Lake with best practices.
  • Responsible for creating document for NiFi Flow for data ingestion into Data Lake.
  • Experienced in using compression techniques like Avro, Z-Zip, Sequence, Parquet, ORC and Text file and implementation of various Hadoop file-formats.
  • Setting up Dev/QA/Prod Hadoop distributed environment and cluster management with open source platforms like HDP (Hortonworks Data Platform) and HDF (Hortonworks Data Flow).
  • Experienced in designing HDP and HDF cluster.
  • Experienced in using Kerberos and secured both HDP and HDF clusters.
  • Analyzed the partitioned and bucketed data to compute various metrics for reporting by using Hive.
  • Involved in processing ingested raw data using MapReduce, Apache Hive, HBase, TEZ and NiFi.
  • Experienced working on Hive for creating schemas, databases and tables for Data Ingestion.
  • Responsible for creating multiple Ranger policies for authorization Hortonworks components like Hive, HDFS.
  • Worked on swapping old master nodes with new master nodes with minimal down time. (Hardware Change)
  • Made configuration changes to HDP and HDF cluster as required with best practices.
  • Upgraded Ambari from 2.6.1.0 to 2.6.2.2 and created and maintained a document.
  • Experienced in creating Oozie work flows to schedule the Hive and spark jobs using Work Flow Manager from Ambari.
  • Responsible for Cluster maintenance, commissioning and decommissioning Data nodes, Cluster Monitoring, Troubleshooting, Manage & review Hadoop log files.
  • Created and maintained Technical documentation (Run Book) for setting up the all the clusters QA/UAT/Prod.

Environment: MapReduce, HDFS, Hive, Pig, Spark, Spark-Streaming, Spark SQL, Apache Kafka, Sqoop, Java, Scala, HDP, HDF, AWS, Eclipse, Cloudant, Git, Shell Scripting, Nifi, Cloudant, Nifi Registry.

Confidential, Chicago, IL

Spark/Hadoop Developer

Responsibilities/Roles:

  • Prepared detailed specifications that follow project guidelines required to develop written programs by evaluating the Business requirements.
  • Worked on different big data analytical and processing tools including Spark, Spark Streaming, Pig, Hive and analyzed different Hadoop Clusters.
  • Experienced in Managing scalable Hadoop clusters including Cluster designing, provisioning, custom configurations, monitoring and maintaining using different Hadoop distributions: Cloudera CDH, Hortonworks HDP.
  • Analyzed the partitioned and bucketed data using Hive and compute various metrics for reporting.
  • Analyzed and determined optimal way to aggregate large amounts of data sets and created report on it.
  • Migrated various Hive UDF's and queries into Spark SQL for faster requests.
  • Experienced in importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and exported the data from HDFS to MYSQL using Sqoop.
  • Diverse set of responsibilities from maintaining JAVA based data management tools to implementing and automating ETL work-flows.
  • Hands on experience in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions.
  • Configured Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala.
  • Managed vast amounts of User data, eventually used as input to analytics processes, to optimize user experience in Mobile applications.
  • Created and automated test cases to run periodically, ensuring consistency in the process output.
  • Experience in using Apache Kafka for log aggregations.
  • Experience on working with different data types like AVRO, JSON, FLATFILES, and ORC.
  • Involved in implementing Cluster for Cassandra to address HBase limitations.
  • Provided design recommendations and thought leadership to sponsors/stakeholders that improved review processes and resolved technical problems and suggested some solution.

Environment: MapReduce, HDFS, Hive, Pig, Spark, Spark-Streaming, Spark SQL, Apache Kafka, Sqoop, Java, Scala, CDH4, CDH5, AWS, Eclipse, Oracle, Git, Shell Scripting and Cassandra.

Confidential, Baltimore, MD

Big Data Engineer

Responsibilities/Roles:

  • Worked on installing cluster, commissioning & decommissioning of Datanode, Namenode high Availability, capacity planning, and slots configuration.
  • Used CLOUDERA data platform.
  • Responsible for Cluster maintenance, Monitoring, commissioning and decommissioning Data nodes, Troubleshooting, Manage and review data backups, Manage & review log files
  • Monitored workload, job performance and capacity planning using Cloudera Manager.
  • Worked on analyzing, writing Hadoop MapReduce jobs using Java API, Pig and Hive
  • Very good understanding of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive for optimized performance
  • Developing scripts and batch job to schedule a bundle (group of coordinators), which consists of various Hadoop programs using Oozie.
  • Works with ETL workflow, analysis of big data and loaded them into Hadoop cluster
  • Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior.
  • Developed Spark Application by using Scala
  • Load and transform large sets of structured, semi structured and unstructured data
  • Configured MySQL Database to store Hive metadata.
  • Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of spark using Scala.
  • Developed Spark code using Scala and spark-SQL/Streaming for faster testing and processing of data.
  • Responsible for loading unstructured data into Hadoop File System (HDFS).
  • Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
  • Created tables, stored procedures in SQL for data manipulation and retrieval, Database Modification using SQL, PL/SQL, Stored procedures, and triggers.
  • Created reports for the BI team using Sqoop to export data into HDFS and Hive.
  • Involved in collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
  • Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
  • Worked on Map reduce Joins in querying multiple semi-structured data as per analytic needs.
  • Automated the process for extraction of data from warehouses and weblogs by developing work-flows and coordinator jobs in OOZIE.

Environment: Hadoop, Pig, Hive, Java, SQOOP, Kafka, Hbase, NoSQL, Oracle, SPARK, SCALA, STORM, ELASTIC SEARCH, ZOOKEEPER, OOZIE, RED HAT LINUX, AGILE

Confidential, Tampa, FL

Hadoop Developer

Responsibilities/Roles

  • Understand the exact requirement of report from the Business groups and users.
  • Imported trading and derivatives data in Hadoop Distributed File System using Eco System components MapReduce, Pig, Hive, Sqoop.
  • Was part of a team which was have activity to setup Hadoop ecosystem at development & QA Environment.
  • Reviewed and Managed Hadoop Log files.
  • Responsible writing Hive queries and PIG scripts for data processing.
  • Running Sqoop for importing data from Oracle and another Database.
  • Created of shell scripts to collect raw logs from different machines.
  • Created Hive as static and dynamic partitions.
  • Implemented Pig Latin scripts using operators such as LOAD, STORE, GENERATE, GROUP, COGROUP, DUMP, FILTER, DISTINCT, FOREACH ORDER, LIMIT and UNION.
  • Optimized script using illustrate and explain and used parameterize Pig Script.
  • Defined some PIG UDFs for some functions such as swap, hedging, Speculation and arbitrage.
  • Unstructured logs files are coded using MapReduce program.
  • Imported and exported data into HDFS and Hive using Sqoop.
  • Involved in the process of configuring HA, Kerberos security issues and name node failure restoration activity time to time as a part of zero downtime.
  • Implemented FAIR Scheduler as well.
  • Developed JUNIT test cases for application unit testing.
  • Used SVN as version control to check in the code, created branches and tagged the code in SVN.

Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Shell Scripting, Sqoop, Eclipse, SVN, SQL, Hive, Kerberos.

Confidential

Java/J2EE Developer

Roles & Responsibilities:

  • Involved in the complete SDLC software development life cycle of the application from requirement analysis to testing.
  • Developed the modules based on Struts MVC Architecture.
  • Followed AGILE methodology (Scrum Stand-ups, Sprint Planning, Sprint Review, Sprint Showcase and Sprint Retrospective meetings).
  • Developed business components using Core Java concepts and classes like Inheritance, Polymorphism, Collections, Serialization and Multithreading etc.
  • Developed the Web Interface using Servlets, Java Server Pages, HTML and CSS.
  • Developed the DAO objects using JDBC.
  • Business Services using the Servlets and Java.
  • Used Spring Framework for Dependency injection and integrated with the Struts Framework and Hibernate.
  • Developed JUnit test cases for all the developed modules.
  • Used Log4j to capture the log that includes runtime exceptions, monitored error logs and fixed the problems.
  • Performed Unit Testing, System Testing and Integration Testing.
  • Provided technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects. Resolved more priority defects as per the schedule.

Environment: Java, JDBC, Spring, Hibernate, HTML, CSS, Java Script, Log4j, Oracle, Struts and Eclipse.

We'd love your feedback!