Sr. Hadoop Developer Resume
Reston, VA
SUMMARY
- 8+ years of experience in software design, Requirement Analysis, development, maintenance, testing, Administration and troubleshooting of enterprise applications using Java/J2EE Technologies and Big data Hadoop technologies.
- Proficiency in Java,HadoopMap Reduce, Pig, Hive, Oozie, Sqoop, Flume, Hbase, Spark, Impala and NoSQL Databases.
- Solid expertise in the workings of Hadoop internals, architecture and supporting ecosystem components like Hive, Sqoop, Pig, Flume and Oozie.
- Experience on Hadoop CDH3 and CDH4
- Extensive experience in writing custom Map Reduce programs for data processing and UDFs for both Hive and Pig in Java.
- Written multiple MapReduce programs in java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other file formats.
- Experienced in developing and implementing Map Reduce jobs using java to process and perform various analytics on large datasets
- Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice - versa. Experienced in loading the data from different Data sources like (Teradata and DB2) into HDFS using sqoop and load into partitioned Hive tables.
- Hands-on experience withHadoopapplications (such as administration, configuration management, monitoring, debugging, and performance tuning).
- Expertise in writing Shell scripting in UNIX using ksh and bash
- Good exposure on usage of NoSQL databases column-oriented HBase and SQL Development. Expertise in developing NoSQL database by using CRUD, Sharding, Indexing and Replication.
- Good knowledge on Amazon AWS concepts like EMR and EC2 web services which provides fast and efficient processing of Big Data and Machine Learning Concepts.
- Sound knowledge of Business Intelligence and Reporting. Preparation of Dashboards using Tableau.
- Skilled in programming with Java/J2EE using JSP, JSF, JavaScript, Servlets and EJB
- Worked on Query Tuning, Application Server performance tuning, Server performance improvement settings, Database tuning, Network performance analysis
- Experience in database design, entity relationships, database analysis, programming SQL, stored procedure’s PL/ SQL, packages and triggers in Oracle and MongoDB on Unix/Linux.
- Good experience in implementing service oriented architecture using Web Services
- Experience in Build Process and Deployments of bug fixes and customizations from the development team to the Application Server(IBM Websphere, Tomcat)
- Experienced in all facets of Software Development Life Cycle (Analysis, Design, Development, Testing and maintenance) using Waterfall and Agile methodologies.
- Experience in preparing deployment packages and deploying to Dev and QA environments and prepare deployment instructions to Production Deployment Team
- Motivated team player with excellent communication, interpersonal, analytical and problem solving skills.
- Highly adept at promptly and thoroughly mastering new technologies with a keen awareness of new industry developments and the evolution of next generation programming solutions.
TECHNICAL SKILLS
Hadoop Core Services: HDFS, Map Reduce, Spark, YARN.
Hadoop Distribution: Horton works, Cloudera, Apache.
NO SQL Databases: HBase, Cassandra.
Hadoop Data Services: Hive, Pig, Impala, Sqoop, Flume, Spark, Kafka (beginner).
Hadoop Operational Services: Zookeeper, Oozie.
Cloud Computing Tools: Amazon AWS.
Languages: C, Java, Scala, Python, SQL, PL/SQL, Pig Latin, HiveQL, Unix, Java Script, Shell Scripting.
Java & J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts, JMS, EJB.
Application Servers: Web Logic, Web Sphere, JBoss, Tomcat.
Databases: Microsoft SQL Server; MySQL; Oracle
Operating Systems: UNIX, Windows, LINUX.
Build Tools: Jenkins, Maven, ANT.
Business Intelligence Tools: Tableau, Splunk
Development Tools: Microsoft SQL Studio, Toad, Eclipse, NetBeans.
Development Methodologies: Agile/Scrum, Waterfall.
PROFESSIONAL EXPERIENCE
Confidential, Reston, VA
Sr. Hadoop developer
Responsibilities:
- Involved in architecture development of Hadoop Clusters with CDH4 on Linux, using Cloudera Manager
- Installed, configured, and maintained Apache Hadoop clusters for application development and major components of Hadoop Ecosystem: Hive, Pig, HBase, Sqoop, Flume, Oozie and Zookeeper.
- Developed simple and complex Map Reduce programs in Java for Data Analysis on different data formats.
- Developed Map Reduce programs that filter bad and un-necessary records and find out unique records based on different criteria.
- Developed Secondary sorting implementations to get sorted values at reduce side to improve Map Reduce performance.
- Implemented custom Data Types, Input Format, Record Reader, Output Format, Record Writer for Map Reduce computationsto handle custom business requirements.
- Implemented Map Reduce programs to classified data organizations into different classifieds based on different type of records.
- Worked on Sequence files, RC files, Map side joins, bucketing, partitioning for hive performance enhancement and storage improvement.
- Implemented Daily Cron jobs that automate parallel tasks of loading the data into HDFS and pre-processing with Pig using Oozie co-coordinator jobs.
- Importing and exporting large data set into HDFS and Hive from different RDBMS using Sqoop.
- Experienced in defining job flows to run multiple Map Reduce and Pig jobs using Oozie.
- Importing log files using Flume into HDFS and load into Hive tables to query data.
- Monitoring the runningMap Reduceprograms on the cluster.
- Used HBase-Hive integration, written multiple Hive UDFs for complex queries.
- Involved in writing APIs to ReadHBasetables cleanse data and write to anotherHBasetable.
- Created multiple Hive tables implemented Partitioning, Dynamic Partitioning and Buckets in Hive for efficient data access.
- Written multiple Map Reduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
- Experienced in running batch processes using Pig Scripts and developed Pig UDFs to pre-process the data and data manipulation according to Business Requirements.
- Experienced in writing programs using HBase Client AP.
- Involved in loading data into HBase using HBase Shell, HBase Client API, Pig and Sqoop.
- Experienced in design, development, tuning and maintenance of NoSQL database.
- Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
- Developed unit test cases for Hadoop Map Reduce jobs with MRUnit.
- Excellent experience in ETL analysis, designing, developing, testing and implementing ETL processes including performance tuning and query optimizing of database.
- Continuously monitored and managed the Hadoop cluster using Cloudera manager and Web UI.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Used Maven as the build tool and SVN for code management.
- Worked on writing Restful web services for the application.
- Implemented testing scripts to support test driven development and continuous integration.
Environment: CDH4 Cloudera Distribution, Sqoop, Pig Latin, Hive, Flume, HDFS, MapReduce, Eclipse IDE, UNIX Shell Scripting
Confidential, MN
Sr. Hadoop developer
Responsibilities:
- Created design specifications to get rid of system limitations and create platform convergence by migrating data from various sources to a Hadoop cluster.
- Involved in various phases of software development phases such as modeling, system analysis and design, programming and testing using AGILE methodology
- Developed Java code components for connecting and querying the Cassandra database and designed the control table schema which is the driving force for the whole application.
- Developed Java Map Reduce programs to transform the log data into structured way to find user location, age group, spending time.
- Used Pig to process logs and semi structured content and imported the processed data into Hive warehouse which enabled the Business Analysts to perform analytics.
- Developed Hive queries to pre-process the data for analysis by imposing read only structure on the stream data.
- Tuned the cluster for optimal performance to process these large data sets
- Analyzed the web log data using HQL and exported the analyzed data into relational Database using Sqoop for Visualization and to generate reports by our BI team.
- Developed data pipeline using Pig and Hive from various data sources. These pipelines had customized UDF’S to extend the ETL functionality.
- Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive querying.
- Configured big data workflows to run on the top of Hadoop using Autosys and these workflows comprises of heterogeneous jobs like Pig, Hive, Sqoop and Map-Reduce
- Worked with Avro Data Serialization system to work with JSON data formats.
- Worked on different file formats like Sequence files, XML files and Map files using Map Reduce Programs.
- Performed data analytics in Hive and then exported this metrics back to Oracle Database using Sqoop.
- Developed workflow in Autosys to automate tasks of loading data into HDFS and preprocessing with PIG.
- Developed suit of Unit Test Cases for Mapper, Reducer and Driver classes using MR Testing library.
- Developed automated processes for archiving, Merging and data conversions.
- Used Maven extensively for building jar files of Map-Reduce programs and deployed to Cluster
- Involved in code reviews and verifying bug analysis reports.
- Create support documentation and co-ordinate to form a COE (Centre of Excellence) for the Hadoop framework.
Environment: Hadoop (CDH3U4), HDFS, Hive, Cassandra, Java, Tableau, Eclipse IDE, Tomcat Server
Confidential, Nashville, TN
Hadoop developer
Responsibilities:
- Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
- Installed Hadoop, MapReduce, HDFS, and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
- Coordinated with business customers to gather business requirements, also interacted with other technical peers to derive technical requirements and delivered the BRD and TDD documents.
- Extensively involved in Design phase and delivered Design documents.
- Involved in Testing and coordination with business in User testing.
- Importing and exporting data into HDFS and Hive using Sqoop.
- Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
- Involved in creating Hive tables loading data and writing hive queries that will run internally in MapReduce way.
- Experienced in defining job flows.
- Used Hive to analyze the partitioned and bucketed data to compute various metrics for reporting.
- Experienced in managing and reviewing the Hadoop log files.
- Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data on to HDFS.
- Load and transform large sets of structured and semi structured data.
- Responsible to manage data coming from different sources.
- Involved in creating Hive Tables, loading data and writing hive queries.
- Utilized Apache Hadoop environment by Cloudera.
- Created Data model for Hive tables.
- Involved in Unit testing and delivered Unit test plans and results documents.
- Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
Confidential
Team Lead
Responsibilities:
- Involved in various phases of software development phases such as modeling, system analysis and design, programming and testing using AGILE methodology
- Involved in the application development based on the business requirements using Eclipse and Tomcat application server
- Created shell scripts for scheduling various data cleansing and data transformation and ETL loading process
- Handled structured and unstructured data and applied ETL process
- Development process also including bug fixing, unit testing, code reviews
- As an active build manager responsible for weekly builds and deploying the ear to the Dev and QA application servers for testing
- Developed SQL queries, Stored Procedures and used JDBC to interact with database.
- Used AABS Issue Tracking System for project management
Environment: Java 1.5, Struts, Hibernate 3.0, Oracle 10g, JSP, IBM Rational Application Developer, AJAX, Jquery, JMS, Log4j, JUNIT, Linux, WebSphere, Shell script
Confidential
Lead Java Developer
Responsibilities:
- Application Packaging & Virtualization consultant. Discuss Requirements with E2E designers/Process designers and prepare HLD for the User stories scoped.
- Reviewing and sign off of HLD and preparing the Component Design.
- Developed technical designs for Confidential billing Systems Developed an application for managing end-to-end inventory lifecycle and also a Geo-spatial Analytics tool to analyze fault pattern and to visualize impact of fault in Telecom networks on the users.
- Interact/communicate with Key stake holder (SME’s, CE’s Product ops team) to understand business problems and define the analytical approach to resolve problems
- Worked on MLPD (Mid-Level Platform Design) and POC (Proof of Concepts) with JMS
- Programmed in J2EE, PL/SQL and SQL on Unix Platform
- Reviewed and edited data forms using Microsoft Excel
- Project delivery in agile model, and tracking project’s progress through daily scrum regarding design defects.
- Involved in problem solving, data analysis, and complex report generations
- Worked as Business Analyst for OM1b Systems Project and helped Clients in taking Key decisions
Confidential
Associate JAVA Developer
Responsibilities:
- Involved in the complete SDLC software development life cycle of the application from requirement analysis to testing.
- Developed the modules based on struts MVC Architecture.
- Developed The UI using JavaScript, JSP, HTML, and CSS for interactive cross browser functionality and complex user interface.
- Created Business Logic using Servlets, Session beans and deployed them on Web logic server.
- Used MVC struts framework for application design.
- Created complex SQL Queries, PL/SQL Stored procedures, Functions for back end.
- Prepared the Functional, Design and Test case specifications.
- Involved in writing Stored Procedures in Oracle to do some database side validations.
- Performed unit testing, system testing and integration testing
- Developed Unit Test Cases. Used JUNIT for unit testing of the application.
- Provided Technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
- Resolved more priority defects as per the schedule.
Environment: Java, JSP, Servlets, Weblogic, Oracle, JUnit, SQL, XML, Toad
