We provide IT Staff Augmentation Services!

Sr. Hadoop Developer Resume

0/5 (Submit Your Rating)

Concord, CA

SUMMARY

  • 9+ years of experience as a software professional that starts from requirement gathering, analysis, design, implementation & testing software products using Java/ J2EE technologies and also in Big data technologies using Hadoop ecosystem.
  • 4 years of experience in working with different Hadoop ecosystem components such as HDFS, MapReduce, HBase, Spark, Yarn, Kafka, Zookeeper, PIG, HIVE, Sqoop, Storm, Oozie, and Flume.
  • Around 5 years of experience in Object Oriented Programming, using Java & J2EE (Servlets, JSP, JSF, Spring, Hibernate, Java Beans, EJB, JDBC, RMI, XML, JMS, Web Services, AJAX).
  • Very good experience in large scale data processing by using Apache Spark processing components Spark SQL, Spark Streaming, MLlib using Scala/python as programming languages.
  • Strong experience in Developing Simple to complex Map/reduce streaming jobs using Java language.
  • Experience in analyzing data using Pig Latin, Hive QL and custom MapReduce programs in Java.
  • Expertise in writing UDF’s in java/python to extend Pig and Hive core functionality.
  • Expertise in optimizing traffic across network using different concepts like Combiners, Joins, Partitions, Bucketing.
  • Hands on experience in Apache Storm for processing real time streaming data.
  • Experience with Kafka in understanding and performing thousands of megabytes of reads and writes per second on streaming data.
  • Experience in working with Apache Sqoop to load data from Relational Database systems to HDFS and vice - versa.
  • Hands on experience in working with Apache Oozie to define Job workflows, scheduling and monitoring them.
  • Experience with Apache Zookeeper to provide coordination services to the cluster.
  • Experience with Apache Flume for Collecting and aggregating a large amount of Log Data from application servers and storing data in HDFS for further analysis.
  • Experience in working with various Hadoop distributions like Cloudera, Hortonworks.
  • In depth understanding of Hadoop Architecture and its components such as Name Node, Data Node, Application Master, Resource Manager and Map Reduce concepts.
  • Worked in complete Software Development Life Cycle (analysis, design, development, testing, implementation and support) using Agile Methodologies.
  • Good experience in working with different NOSQL Data bases such as HBase, Cassandra, MongoDB and also in RDBMS technologies like MySQL, Oracle 12c, SQL server, PostgreSQL.
  • Proficient in programming languages like Java, Python, Scala.
  • Experience in using Tableau to create statistics dashboards for Analytics.
  • Experience in software component design technologies like UML Design, Use case and requirement Components diagrams.
  • Good working experience in working different IDE's like Eclipse, Net Beans, IntelliJ.
  • Very good knowledge on different concepts of Java like oops concepts, Design patterns, multithreading, serialization.
  • Active team player with ability organize and prioritize multiple tasks.
  • Excellent communication and inter-personal skills with technical competency and ability to quickly learn new technologies as required.
  • Ability to blend technical expertise with strong Conceptual, Business and Analytical skills to provide quality solutions and result-oriented problem solving technique and leadership skills

TECHNICAL SKILLS

Hadoop/Big Data: MapReduce, HDFS, Hive, Pig, HBase, Sqoop, Spark, Spark SQL, RDD, Spark Streaming, GraphX, Kafka, Storm, Flume, Zookeeper, Oozie, Impala, Hue.

Distributed plat forms: Hortonworks, Cloudera.

Programming Languages: C, C#, Java, Scala, Python, R.

Java/J2EE Technologies: Servlets, JSP, JSF, JDBC, Java Beans, RMI & Web services (SOAP, RESTFul).

Frameworks: Struts, Hibernate and Spring MVC.

Development Tools: Eclipse, Net Beans, SBT, SVN, Git, ANT, Maven, Jenkins, Bamboo, SOAP UI, QC, Selenium WebDriver, Jira, Bugzilla, SQL Developer.

Methodologies: Agile/Scrum, UML, and Waterfall

NoSQL Technologies: Cassandra, MongoDB, Neo4j, HBase

Databases: Oracle 12c, MySQL, MS-SQL Server, SQL Server, PostgreSQL.

Web/ Application Servers: Apache Tomcat, WebLogic, WebSphere.

Visualization: Tableau and Qlikview.

Web Technologies: HTML, CSS, XML, JavaScript, jQuery, AngularJS, AJAX, SOAP, and REST.

Scripting Languages: Unix Shell Scripting, Perl.

Operating Systems: Windows XP/Vista/7/8,10, UNIX, LINUX.

PROFESSIONAL EXPERIENCE

Confidential, Concord, CA

Sr. Hadoop Developer

Responsibilities:

  • Creating end to end Spark applications using Scala to perform various data cleansing, validation, transformation and summarization activities on user behavioral data.
  • Created custom FTP adaptors to pull the clickstream data from FTP servers to HDFS directly using HDFS File System API.
  • Used Spark SQL and dataframe API extensively to build spark applications.
  • Worked closely with Admin and Devops team to help in setup high availability, capacity planning, authentication and authorization roles etc
  • Created HBase tables to store variable data formats of data coming from different portfolios
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Used Sqoop job to import the data from RDBMS using Incremental Import.
  • Customized Avro tools used in MapReduce, Pig and Hive for deserialization and to work with Avro ingestion framework
  • Worked with using different kind of compression techniques to save data and optimize data transfer over network using Lzo, Snappy, etc.
  • Analyze large and critical datasets using Hbase, MapReduce, Hive, Hive UDF, Pig, Sqoop, Zookeeper, & Spark.
  • Customized Flume interceptors to encrypt and mask customer sensitive data as per requirement.
  • Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
  • Worked with NoSQL database Hbase to create tables and store data.
  • Created end to end data processing pipelines to manage the entire life cycle of the User behavioral data in the new platform.
  • Developed custom aggregate functions in Spark and performed interactive querying using Impala
  • Creating Hive tables, dynamic partitions, buckets for sampling, and working on them using Hive QL
  • Stored the data in tabular formats using Hive tables and Hive SerDe's.
  • Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
  • Implemented test scripts to support test driven development and continuous integration.
  • Worked on tuning the performance Pig queries.
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
  • Involved in writing the shell scripts for exporting log files to Hadoop cluster through automated process.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
  • Implemented MapReduce programs to handle semi/ unstructured data like Parquet, XML, JSON, Avro data files and sequence files for log files.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: Hadoop, HDFS, Pig, Sqoop, Spark, MapReduce, Cloudera, Snappy, Zookeeper, NoSQL, HBase, Shell Scripting, Ubuntu, Linux Ubuntu.

Confidential, Dallas, Texas

Hadoop Developer

Responsibilities:

  • Created various spark applications using Scala to perform various enrichment of these click stream data with enterprise data of the users
  • Performed data ingestion using kafka to the spark distribution environment.
  • Implemented batch processing of jobs using spark Scala API.
  • Used spark engine Spark SQL for data analysis and given to the data scientists for further analysis.
  • Also worked with data science team in buidling Spark MLlib applications to build various predictive models.
  • Developed multiple Map Reduce jobs in Java for complex business requirements including data cleansing and preprocessing.
  • Developed Sqoop scripts to import/export data from Oracle to HDFS and into Hive tables.
  • Worked on analyzing Hadoop clusters using Big Data Analytic tools including Map Reduce, Pig and Hive.
  • Involved in developing Pig scripts and to store unstructured data into HDFS.
  • Involved in creating tables in Hive and writing scripts and queries to load data into Hive tables from HDFS.
  • Scripted complex Hive QL queries on Hive tables for analytical functions.
  • Optimized the Hive tables using optimization techniques like partitions and bucketing to provide better performance with Hive QL queries.
  • Developed Java custom record reader, partition and serialization techniques.
  • Used different data formats (Text format and Avro format) while loading the data into HDFS.
  • Implemented MapReduce programs to handle semi/ unstructured data like Parquet, XML, JSON files and sequence files for log files.
  • Loaded the final processed data to HBase tables to allow downstream application team to build rich and data driven applications.
  • Involved in writing the shell scripts for exporting log files to Hadoop cluster through automated process.
  • Created custom UDF’s in Pig and Hive.
  • Created partitioned tables and loaded data using both static partition and dynamic partition methods.
  • Used oozie for automating the end to end data pipelines and oozie coordinators for scheduling the work flows.
  • Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
  • Analyzed the Hadoop log files using Pig scripts to oversee the errors.

Environment: HDFS, Map Reduce, Hive, Spark, Kafka, Sqoop, Pig, HBase, Oozie, Hortonworks distribution, Java, Eclipse, Shell Scripts, Tableau, Oracle, Windows, Linux.

Confidential, St Louis, MO

Hadoop Developer

Responsibilities:

  • Developed simple to complex Map Reduce streaming jobs using Java language for processing and validating the data.
  • Worked with highlyunstructured and semi structured data of 90 TBin size with replication factor of 3.
  • Developed data pipeline using Map Reduce, Flume, Sqoop and Pig to ingest customer behavioral data into HDFS for analysis.
  • Extensive experience in writingPig scripts to transform raw datafrom several data sources into forming baseline data.
  • Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW tables and historical metrics.
  • Enabled speedy reviews and first-mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System and PIG to pre-process the data.
  • Handled importing data from different data sources into HDFS using Sqoop and also performing transformations using Hive, Map Reduce and then loading data into HDFS.
  • Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Solved performance issuesin Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
  • Developed product profiles using Pig and commodity UDFs.
  • Identify concurrent job workloads that may impact or be impacted by failures or bottlenecks.
  • Developed some utility helper classes to get data from HBase tables.
  • Scheduled and executed workflows in Oozie to run Hive and Pig jobs.
  • Professional experience with NoSQL Hbase solutions to solve real world scaling problems.
  • Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
  • Attending daily status calls to follow scrum process to complete each user story within the timeline.
  • Also be the part of triage call to handle defect reported by tester team or QA team.
  • Viewing various aspect of a cluster using Cloudera Manager.

Environment: Hadoop, Linux, CDH, MapReduce, HDFS, Hive, Pig, Shell Scripting, Sqoop, Java 7, MySQL, NoSQL, Eclipse, Oracle 11g, Maven, Log4j, Git.

Confidential, Attleboro, MA

Sr. Java Developer

Responsibilities:

  • Involved in various SDLC phases like Design, Development and Testing.
  • Involved in developing JSP pages using Struts custom tags, jQuery and Tiles Framework.
  • Developed web pages using HTML,JavaScript, JQuery and CSS.
  • Used various CoreJavaconcepts such as Exception Handling, Collection APIs to implement various features and enhancements.
  • Developed server side components servlets for the application.
  • Involved in coding, maintaining, and administering Servlets and JSP components to be deployed on a Web Sphere application server.
  • Used automated test scripts and tools to test the application in various phases. Coordinated with Quality Control teams to fix issues that were identified
  • Implemented Hibernate ORM to Map relational data directly tojavaobjects
  • Worked with Complex SQL queries, Functions and Stored Procedures.
  • Involved in developing spring web MVC framework for portals application.
  • Implemented the logging mechanism using log4j framework.
  • Developed REST API, Web Services.
  • Wrote test cases in JUnit for unit testing of classes.
  • Used Maven to build the J2EE application.
  • Used SVN to track and maintain the different version of the application.
  • Involved in maintenance of different applications with onshore team.

Environment: Java, Spring Framework, Struts, Hibernate, RAD, SVN, Maven, Web Sphere Application Server, Web Services, Oracle Database 11g, IBM MQ, JMS, HTML, Java script, XML, CSS, REST API.

Confidential

Java / J2EE Developer

Responsibilities:

  • Involved in design and development of a front-end application using JavaScript, CSS, HTML and JSPs andspring
  • MVC for registering a new entry and management and configured it to connect to database using Hibernate.
  • Developed and tested the Efficiency Management module using EJB, Servlets, and JSP & Core Java components in WebLogic Application Server.
  • UsedSpring Framework as middle tier application framework, persistence strategy usingspring support for Hibernate for integrating with database.
  • Implemented Hibernate in the data access object layer to access and update information in the Oracle Database.
  • Configured the deployment descriptors in Hibernate to achieve object relational mapping.
  • Involved in developing Stored Procedures, Queries and Functions.
  • Write SQL queries to pull some information from the Backend.
  • Compiling and running the applications.
  • Writing the test plans and test cases for the developed screens.
  • Executing test cases and fixing the bugs through unit testing.

Environment: Java, J2EE (Servlets, JDBC, EJB, JSP, JMS), HTML, CSS, Struts 1.2, Hibernate, Spring, XML, CVS, Eclipse, Oracle 8i, PL/SQL, Windows, UNIX

Confidential

Java / J2EE Developer

Responsibilities:

  • Participated in requirement gathering and converting the requirements into technical specifications.
  • Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components using Business Objects, XML, and JDBC.
  • Designed user-interface and checking validations using JavaScript.
  • Involved in design of JSP’s and Servlets for navigation among the modules.
  • Developed various EJBs for handling business logic and data manipulations from database.
  • Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
  • Developed SQL queries and Stored Procedures using PL/SQL to retrieve and insert into multiple database schemas.
  • Developed the XML Schema and Web services for the data maintenance and structures Wrote test cases in JUnit for unit testing of classes.
  • Provided Technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
  • Built and deployed Java applications into multiple UNIX based environments and produced both unit and functional test results along with release notes.
  • Developed the presentation layer using CSS and HTML taken from bootstrap to develop for browsers.

Environment: Java, Spring, Jsp, Hibernate, XML, HTML, JavaScript, JDBC, CSS, SOAP Web services.

We'd love your feedback!