We provide IT Staff Augmentation Services!

Sr. Hadoop Developer Resume

0/5 (Submit Your Rating)

Reston, VA

SUMMARY

  • 8+ years of experience in software design, Requirement Analysis, development, maintenance, testing, Administration and troubleshooting of enterprise applications using Java/J2EE Technologies and Big data Hadoop technologies.
  • Proficiency in Java,HadoopMap Reduce, Pig, Hive, Oozie, Sqoop, Flume, Hbase, Spark, Impala and NoSQL Databases.
  • Solid expertise in the workings of Hadoop internals, architecture and supporting ecosystem components like Hive, Sqoop, Pig, Flume and Oozie.
  • Experience on Hadoop CDH3 and CDH4
  • Extensive experience in writing custom Map Reduce programs for data processing and UDFs for both Hive and Pig in Java.
  • Written multiple MapReduce programs in java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other file formats.
  • Experienced in developing and implementing Map Reduce jobs using java to process and perform various analytics on large datasets
  • Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice - versa. Experienced in loading the data from different Data sources like (Teradata and DB2) into HDFS using sqoop and load into partitioned Hive tables.
  • Hands-on experience withHadoopapplications (such as administration, configuration management, monitoring, debugging, and performance tuning).
  • Expertise in writing Shell scripting in UNIX using ksh and bash
  • Good exposure on usage of NoSQL databases column-oriented HBase and SQL Development. Expertise in developing NoSQL database by using CRUD, Sharding, Indexing and Replication.
  • Good knowledge on Amazon AWS concepts like EMR and EC2 web services which provides fast and efficient processing of Big Data and Machine Learning Concepts.
  • Sound knowledge of Business Intelligence and Reporting. Preparation of Dashboards using Tableau.
  • Skilled in programming with Java/J2EE using JSP, JSF, JavaScript, Servlets and EJB
  • Worked on Query Tuning, Application Server performance tuning, Server performance improvement settings, Database tuning, Network performance analysis
  • Experience in database design, entity relationships, database analysis, programming SQL, stored procedure’s PL/ SQL, packages and triggers in Oracle and MongoDB on Unix/Linux.
  • Good experience in implementing service oriented architecture using Web Services
  • Experience in Build Process and Deployments of bug fixes and customizations from the development team to the Application Server(IBM Websphere, Tomcat)
  • Experienced in all facets of Software Development Life Cycle (Analysis, Design, Development, Testing and maintenance) using Waterfall and Agile methodologies.
  • Experience in preparing deployment packages and deploying to Dev and QA environments and prepare deployment instructions to Production Deployment Team
  • Motivated team player with excellent communication, interpersonal, analytical and problem solving skills.
  • Highly adept at promptly and thoroughly mastering new technologies with a keen awareness of new industry developments and the evolution of next generation programming solutions.

TECHNICAL SKILLS

Hadoop Core Services: HDFS, Map Reduce, Spark, YARN.

Hadoop Distribution: Horton works, Cloudera, Apache.

NO SQL Databases: HBase, Cassandra.

Hadoop Data Services: Hive, Pig, Impala, Sqoop, Flume, Spark, Kafka (beginner).

Hadoop Operational Services: Zookeeper, Oozie.

Cloud Computing Tools: Amazon AWS.

Languages: C, Java, Scala, Python, SQL, PL/SQL, Pig Latin, HiveQL, Unix, Java Script, Shell Scripting.

Java & J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts, JMS, EJB.

Application Servers: Web Logic, Web Sphere, JBoss, Tomcat.

Databases: Microsoft SQL Server; MySQL; Oracle

Operating Systems: UNIX, Windows, LINUX.

Build Tools: Jenkins, Maven, ANT.

Business Intelligence Tools: Tableau, Splunk

Development Tools: Microsoft SQL Studio, Toad, Eclipse, NetBeans.

Development Methodologies: Agile/Scrum, Waterfall.

PROFESSIONAL EXPERIENCE

Confidential, Reston, VA

Sr. Hadoop developer

Responsibilities:

  • Involved in architecture development of Hadoop Clusters with CDH4 on Linux, using Cloudera Manager
  • Installed, configured, and maintained Apache Hadoop clusters for application development and major components of Hadoop Ecosystem: Hive, Pig, HBase, Sqoop, Flume, Oozie and Zookeeper.
  • Developed simple and complex Map Reduce programs in Java for Data Analysis on different data formats.
  • Developed Map Reduce programs that filter bad and un-necessary records and find out unique records based on different criteria.
  • Developed Secondary sorting implementations to get sorted values at reduce side to improve Map Reduce performance.
  • Implemented custom Data Types, Input Format, Record Reader, Output Format, Record Writer for Map Reduce computationsto handle custom business requirements.
  • Implemented Map Reduce programs to classified data organizations into different classifieds based on different type of records.
  • Worked on Sequence files, RC files, Map side joins, bucketing, partitioning for hive performance enhancement and storage improvement.
  • Implemented Daily Cron jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs.
  • Importing and exporting large data set into HDFS and Hive from different RDBMS using Sqoop.
  • Experienced in defining job flows to run multiple Map Reduce and Pig jobs using Oozie.
  • Importing log files using Flume into HDFS and load into Hive tables to query data.
  • Monitoring the runningMap Reduceprograms on the cluster.
  • Used HBase-Hive integration, written multiple Hive UDFs for complex queries.
  • Involved in writing APIs to ReadHBasetables cleanse data and write to anotherHBasetable.
  • Created multiple Hive tables implemented Partitioning, Dynamic Partitioning and Buckets in Hive for efficient data access.
  • Written multiple Map Reduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
  • Experienced in running batch processes using Pig Scripts and developed Pig UDFs to pre-process the data and data manipulation according to Business Requirements.
  • Experienced in writing programs using HBase Client AP.
  • Involved in loading data into HBase using HBase Shell, HBase Client API, Pig and Sqoop.
  • Experienced in design, development, tuning and maintenance of NoSQL database.
  • Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
  • Developed unit test cases for Hadoop Map Reduce jobs with MRUnit.
  • Excellent experience in ETL analysis, designing, developing, testing and implementing ETL processes including performance tuning and query optimizing of database.
  • Continuously monitored and managed the Hadoop cluster using Cloudera manager and Web UI.
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
  • Used Maven as the build tool and SVN for code management.
  • Worked on writing Restful web services for the application.
  • Implemented testing scripts to support test driven development and continuous integration.

Environment: CDH4 Cloudera Distribution, Sqoop, Pig Latin, Hive, Flume, HDFS, MapReduce, Eclipse IDE, UNIX Shell Scripting

Confidential, MN

Sr. Hadoop developer

Responsibilities:

  • Created design specifications to get rid of system limitations and create platform convergence by migrating data from various sources to a Hadoop cluster.
  • Involved in various phases of software development phases such as modeling, system analysis and design, programming and testing using AGILE methodology
  • Developed Java code components for connecting and querying the Cassandra database and designed the control table schema which is the driving force for the whole application.
  • Developed Java Map Reduce programs to transform the log data into structured way to find user location, age group, spending time.
  • Used Pig to process logs and semi structured content and imported the processed data into Hive warehouse which enabled the Business Analysts to perform analytics.
  • Developed Hive queries to pre-process the data for analysis by imposing read only structure on the stream data.
  • Tuned the cluster for optimal performance to process these large data sets
  • Analyzed the web log data using HQL and exported the analyzed data into relational Database using Sqoop for Visualization and to generate reports by our BI team.
  • Developed data pipeline using Pig and Hive from various data sources. These pipelines had customized UDF’S to extend the ETL functionality.
  • Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive querying.
  • Configured big data workflows to run on the top of Hadoop using Autosys and these workflows comprises of heterogeneous jobs like Pig, Hive, Sqoop and Map-Reduce
  • Worked with Avro Data Serialization system to work with JSON data formats.
  • Worked on different file formats like Sequence files, XML files and Map files using Map Reduce Programs.
  • Performed data analytics in Hive and then exported this metrics back to Oracle Database using Sqoop.
  • Developed workflow in Autosys to automate tasks of loading data into HDFS and preprocessing with PIG.
  • Developed suit of Unit Test Cases for Mapper, Reducer and Driver classes using MR Testing library.
  • Developed automated processes for archiving, Merging and data conversions.
  • Used Maven extensively for building jar files of Map-Reduce programs and deployed to Cluster
  • Involved in code reviews and verifying bug analysis reports.
  • Create support documentation and co-ordinate to form a COE (Centre of Excellence) for the Hadoop framework.

Environment: Hadoop (CDH3U4), HDFS, Hive, Cassandra, Java, Tableau, Eclipse IDE, Tomcat Server

Confidential, Nashville, TN

Hadoop developer

Responsibilities:

  • Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
  • Installed Hadoop, MapReduce, HDFS, and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
  • Coordinated with business customers to gather business requirements, also interacted with other technical peers to derive technical requirements and delivered the BRD and TDD documents.
  • Extensively involved in Design phase and delivered Design documents.
  • Involved in Testing and coordination with business in User testing.
  • Importing and exporting data into HDFS and Hive using Sqoop.
  • Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
  • Involved in creating Hive tables loading data and writing hive queries that will run internally in MapReduce way.
  • Experienced in defining job flows.
  • Used Hive to analyze the partitioned and bucketed data to compute various metrics for reporting.
  • Experienced in managing and reviewing the Hadoop log files.
  • Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data on to HDFS.
  • Load and transform large sets of structured and semi structured data.
  • Responsible to manage data coming from different sources.
  • Involved in creating Hive Tables, loading data and writing hive queries.
  • Utilized Apache Hadoop environment by Cloudera.
  • Created Data model for Hive tables.
  • Involved in Unit testing and delivered Unit test plans and results documents.
  • Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.

Confidential

Team Lead

Responsibilities:

  • Involved in various phases of software development phases such as modeling, system analysis and design, programming and testing using AGILE methodology
  • Involved in the application development based on the business requirements using Eclipse and Tomcat application server
  • Created shell scripts for scheduling various data cleansing and data transformation and ETL loading process
  • Handled structured and unstructured data and applied ETL process
  • Development process also including bug fixing, unit testing, code reviews
  • As an active build manager responsible for weekly builds and deploying the ear to the Dev and QA application servers for testing
  • Developed SQL queries, Stored Procedures and used JDBC to interact with database.
  • Used AABS Issue Tracking System for project management

Environment: Java 1.5, Struts, Hibernate 3.0, Oracle 10g, JSP, IBM Rational Application Developer, AJAX, Jquery, JMS, Log4j, JUNIT, Linux, WebSphere, Shell script

Confidential

Lead Java Developer

Responsibilities:

  • Application Packaging & Virtualization consultant. Discuss Requirements with E2E designers/Process designers and prepare HLD for the User stories scoped.
  • Reviewing and sign off of HLD and preparing the Component Design.
  • Developed technical designs for Confidential billing Systems Developed an application for managing end-to-end inventory lifecycle and also a Geo-spatial Analytics tool to analyze fault pattern and to visualize impact of fault in Telecom networks on the users.
  • Interact/communicate with Key stake holder (SME’s, CE’s Product ops team) to understand business problems and define the analytical approach to resolve problems
  • Worked on MLPD (Mid-Level Platform Design) and POC (Proof of Concepts) with JMS
  • Programmed in J2EE, PL/SQL and SQL on Unix Platform
  • Reviewed and edited data forms using Microsoft Excel
  • Project delivery in agile model, and tracking project’s progress through daily scrum regarding design defects.
  • Involved in problem solving, data analysis, and complex report generations
  • Worked as Business Analyst for OM1b Systems Project and helped Clients in taking Key decisions

Confidential

Associate JAVA Developer

Responsibilities:

  • Involved in the complete SDLC software development life cycle of the application from requirement analysis to testing.
  • Developed the modules based on struts MVC Architecture.
  • Developed The UI using JavaScript, JSP, HTML, and CSS for interactive cross browser functionality and complex user interface.
  • Created Business Logic using Servlets, Session beans and deployed them on Web logic server.
  • Used MVC struts framework for application design.
  • Created complex SQL Queries, PL/SQL Stored procedures, Functions for back end.
  • Prepared the Functional, Design and Test case specifications.
  • Involved in writing Stored Procedures in Oracle to do some database side validations.
  • Performed unit testing, system testing and integration testing
  • Developed Unit Test Cases. Used JUNIT for unit testing of the application.
  • Provided Technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
  • Resolved more priority defects as per the schedule.

Environment: Java, JSP, Servlets, Weblogic, Oracle, JUnit, SQL, XML, Toad

We'd love your feedback!