Hadoop Developer/ Architect Resume
Austin, TX
SUMMARY
- 8+ years of experience in software development, building scalable and high performance Big Data applications with specialization in Hadoop Stack, NoSQL Databases, Distributed computing and Java/ J2EE technologies.
- Good experience in using data processing tools Mapreduce, Pig and Hive for performing business transformations, data validations, meta data driven data parsing routines etc.
- Working experience in creating complex data ingestion pipelines, data transformations, data management and data governance in a centralized Enterprise Data Hub.
- Extensive hand on experience in writing complex Mapreduce jobs, Pig Scripts and Hive data modeling.
- Excellent understanding and knowledge of Hadoop Distributed system architecture and design principles.
- Good experience converting business ideas into workable solutions and suggesting the right tools/ solutions for the right problems by working closely with the business teams.
- Experience in working with BI team and transform big data requirements into Hadoop centric technologies.
- Experience in performance tuning the Hadoop cluster by gathering and analyzing the existing infrastructure.
- Good working experience using Sqoop import to import data into HDFS from RDBMS and vice - versa.
- Have good experience creating real time data streaming solutions using Apache Spark/ Spark Streaming/ Apache Storm, Kafka and Flume.
- Extending Hive and Pig core functionality by writing custom UDFs.
- Good understanding of Data Mining and Machine Learning techniques
- Experience in fine-tuning MapReduce jobs for better scalability and performance.
- Developed various Map Reduce applications to perform ETL workloads on terabytes of data.
- Strong work ethic with desire to succeed and make significant contributions to the organization
- Strong problem solving skills, good communication, interpersonal skills and a good team player
- Have the motivation to take independent responsibility as well as ability to contribute and be a productive team member
- Experienced in Java Application Development, Client/Server Applications, Internet/Intranet based applications using Core Java, J2EE patterns, Web Services, Oracle, SQL Server and other relational databases.
PROFESSIONAL EXPERIENCE
Confidential, Austin, TX
Hadoop Developer/ Architect
Responsibilities:
- Worked with the Data Science team to gather requirements for various data mining projects
- Responsible for loading the raw data sets from file dumps and databases to HDFS.
- Extracted insights from the datasets in the format expected by the data science team.
- Written many independent Pig scripts and Java Mapreduce jobs for extracting analytical information.
- Worked with Python to create UDF’s and used them as part of Pig Scripts, and used python for creating graphs for data analysis.
- Involved in creating Hive tables, and loading and analyzing data using hive queries
- Implemented Spark applications from existing MapReduce framework for better performance
- Developed Simple to complex MapReduce Jobs using Hive and Pig.
- Exported the business required information to RDBMS using Sqoop to make the data available for BI team to generate reports based on data.
- Involved in running Hadoop jobs for processing millions of records of text data.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Responsible for loading unstructured and semi-structured data into Hadoop cluster from different data sources using Flume.
- Developed multiple MapReduce jobs in java for data cleaning and preprocessing
- Responsible for managing data from multiple sources.
- Experienced in runningHadoopstreaming jobs to process terabytes of xml format data.
- Load and transform large sets of structured, semi structured and unstructured data.
- Responsible to manage data coming from different sources.
- Assisted in exporting analyzed data to relational databases using Sqoop.
- Created and maintained Technical documentation for launching HADOOP Clusters and for executing Hive queries and Pig Scripts.
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Java (jdk 1.6), Eclipse
Confidential, Atlanta, GA
Hadoop Developer
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop
- Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster
- Setup and benchmarked Hadoop/HBase clusters for internal use
- Developed Simple to complex Map/reduce Jobs using Hive and Pig
- Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop
- Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior
- Used UDF’s to implement business logic in Hadoop
- Implemented business logic by writing UDFs in Java and used various UDFs from Piggybanks and other sources.
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required
- Installed Oozie workflow engine to run multiple Hive and Pig jobs
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Java (jdk 1.6), Eclipse
Confidential, San Francisco, CA
Hadoop Developer
Responsibilities:
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Implemented best income logic using Pig scripts.
- Participate in requirement gathering and analysis phase of the project in documenting the business requirements by conducting workshops/meetings with various business users.
- Managing and scheduling Jobs on a Hadoop cluster using Oozie.
- Involved in moving all log files generated from various sources to HDFS for further processing through Flume.
- Involved in loading data from UNIX file system to HDFS.
- Worked on Hue interface for querying the data.
- Created Hive tables to store the processed results in a tabular format.
- Created HBase tables to store variable data formats of data coming from different portfolios.
- Involved in transforming data from Mainframe tables to HDFS, and HBASE tables using Sqoop and Pentaho Kettle.
- Implemented test scripts to support test driven development and continuous integration.
- Responsible to manage data coming from different sources.
- Experienced on loading and transforming of large sets of structured, semi structured and unstructured data.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Have deep and thorough understanding of ETL tools and how they can be applied in a Big Data environment.
Environment: Hadoop, MapReduce, HDFS, Hive, Pig, Hue, Oozie, Core Java, Eclipse, Hbase, Flume, Cloudera Manager, Oracle 10g, DB2, IDMS, VSAM, SQL*PLUS, Toad, Putty, Windows NT, UNIX Shell Scripting, Pentaho Kettle, Pentaho Bigdata, YARN
Confidential, MD
Java/ J2EE Developer
Responsibilities:
- Responsible for gathering and analyzing requirements and converting them into technical specifications
- Used Rational Rose for creating sequence and class diagrams
- Developed presentation layer using JSP, Java, HTML and JavaScript
- Used Spring Core Annotations for Dependency Injection
- Designed and developed a ‘Convention Based Coding’ utilizing Hibernate’s persistence framework and O-R mapping capability to enable dynamic fetching and displaying of various table data with JSF tag libraries
- Designed and developed Hibernate configuration and session-per-request design pattern for making database connectivity and accessing the session for database transactions respectively.
- Used HQL and SQL for fetching and storing data in databases
- Participated in the design and development of database schema and Entity-Relationship diagrams of the backend Oracle database tables for the application
- Implemented web services with Apache Axis
- Designed and Developed Stored Procedures, Triggers in Oracle to cater the needs for the entire application.
- Developed complex SQL queries for extracting data from the database
- Designed and built SOAP web service interfaces implemented in Java
- Used Apache Ant for the build process
- Used ClearCase for version control and ClearQuest for bug tracking
Environment: Java, JDK 1.5, Servlets, Hibernate, Ajax, Oracle 10g, Eclipse, Apache Ant, Web Services (SOAP), Apache Axis, Apache Ant, Web Logic Server, JavaScript, HTML, CSS, XML
Confidential, NC
JAVA/ J2EE Application Developer
Responsibilities:
- Responsible for design, document, implement, and unit test robust webservices framework to support templated payments and group payments in Java.
- Worked in an onsite-offshore model. Lead offshore developers, assisted them in understanding requirements and provided code reviews.
- Extensively used Spring Framework for Business Layer - accessing enterprise services like JNDI, JMS, and Job Scheduling.
- Also used Spring for transaction management and dependency injection.
- Created Database model, domain objects and DAO’s that interact with the database and store the template related data and events.
- Used JAX-WS and Spring webservices to create and consume SOAP based webservices.
- Used Hibernate as an ORM tool for database operations.
- Created MDB’s to consume messages on various user events.
- Worked in Agile software methodology with Scrum type development.
Environment: Java 1.6, Spring, Hibernate, Maven, Apache MQ, JUnit, JAXB, Oracle 10g, Oracle Coherence, Log4j, Shell Script, Soap UI, TOAD, SQL Developer, Quality Center, Linux, Windows
Confidential, OH
Junior JAVA Developer
Responsibilities:
- Responsible for understanding the business requirement.
- Worked with Business Analyst and helped representing the business domain details in technical specifications.
- Was also actively involved in setting coding standards and writing related documentation.
- Developed the Java Code using Eclipse as IDE.
- Developed JSPs and Servlets to dynamically generate HTML and display the data to the client side
- Developed application on Struts MVC architecture utilizing Action Classes, Action Forms and validations.
- Tiles were used as an implementation of Composite View pattern
- Was responsible in implementing various J2EE Design Patterns like Service Locator, Business Delegate, Session Facade and Factory Pattern.
- Code Review & Debugging using Eclipse Debugger.
- Was responsible for developing and deploying the EJB (Session & MDB).
- Configured Queues in WebLogic server where the messages, using JMS API, were published.
- Consumed Web Services (WSDL, SOAP, UDDI) from third party for authorizing payments to/from customers.
- Writing/Manipulating the database queries.
- Build web application using MAVEN as build tool.
- Used CVS for Version control,
- Performed unit testing using JUnit Testing Framework and Log4J to monitor the error log.
Environment: Java,/J2EE, Eclipse, Web Logic Application Server, Oracle, JSP, HTML, JavaScript, JMS, Servlets, UML, XML, Eclipse, Struts, Web Services, WSDL, SOAP, UDDI
