Hadoop Developer Resume
Reston, VA
PROFESSIONAL SUMMARY:
- 6 +years of experience in Hadoop expertise in the various product application development, Agile, systems programming, requirements gathering, technical documentation writing, Big Data solutions design and development of a different enterprise application systems, extensive design and implementation exposure practices across various domains.
- Firm experience inHadoop /Big Data related technologies in Storage, Querying, Processing and analysis of data.
- Well versed and experienced in data architecture including data ingestion pipeline design,Hadoop information architecture.
- Having experience on spark performance tuning Options.
- Hands on experience in data cleaning, transformation using Informatica Developer.
- Hands on experience in pushing data as delimited files into HDFS using Informatica Developer.
- Load and transform data into HDFS from large set of structured data from /Oracle/SQL.
- Well versed in installation, configuration, supporting and managing of Big Data and underlying infrastructure of Hadoop Cluster.
- Good knowledge of Hadoop Development and various components such as HDFS, Job Tracker, Task Tracker, Data Node, Name Node and Map - Reduce concepts.
- Hands on experience on major components in Hadoop Ecosystem like Hadoop Map Reduce, HDFS, HIVE, PIG, H Base, Zookeeper, Sqoop, Oozie, Cassandra, Flume and Avro.
- Experience in installation, configuration, Management, supporting and monitoring Hadoop cluster using various distributions such as Apache and Cloudera.
- Experience in analyzing data using Hive QL, Pig Latin, H Base and custom Map Reduce programs in Java.
- Involved in project planning, setting up standards for implementation and design of Hadoop based applications.
TECHNICAL SKILLS:
HadoopEcosystems: CDH3, CDH4, CDH5, HDFS, MapReduce.
Data Discovery Applications: Elastic Search and Kibana NoSQL Databases.
HadoopDistributions: CDH3, CDH4, CDH5.
Build Management Tools: Maven, Apache Ant, Git Hub and Version.
Programming Languages: Java, C, C++, Data Structures, Python Java.
Web User Interfaces: HTML, DHTML, JavaSript, and XML.
Databases: Oracle 11g,, MySQL, IBM DB2, PL/SQL, SQL.
Network Protocols: TCP/IP, UDP, HTTP, DHCP, DNSTCP/IP.
PROFESSIONAL EXPERIENCE:
Hadoop Developer
Confidential, Reston, VA
Responsibilities:
- Good Experience in concurrency programming in Spark using Scala with Message passing and Actor concepts using scala.
- Implemented distributed collection of objects that can be cached in memory across 80 node cluster.
- Implemented pre-defined operators in spark such as map, reduce, sample, filter, count, co group, group By, sort, reduce By Key, take, group By Key, union, left OuterJoin, right Outer Join, and etc.
- Responsible for writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL). Involved in defining job flows using Oozie for scheduling jobs to manage apache Hadoop jobs by directed. Directed Acyclic graph (DAG) of actions with control flows.
- Good Knowledge in loading data into cassandra NoSQL Database.
- Implemented record level atomicity on writes using cassandra.
- Developed a data pipeline using Kafka and Storm to store data into HDFS.
- Developing Hive User Defined Functions (UDFs) in java, compiling them into jars and adding them to the HDFS. and executing them with Hive Queries.
- Experienced in managing and reviewing Hadoop log files.
- Responsible to manage data coming from different sources.
- Assisted in monitoring the Hadoop cluster using Ganglia tool.
- Dealing with high volume of data in the cluster.
- Tested and reported defects in an Agile Methodology perspective.
- Consolidate all defects, report it to PM/Leads for prompt fixes by development teams and drive it to closure.
- Importing data from Oracle to HDFS & Hive for analytical purpose.
- Handling imported data in HDFS & Hive using Hive QL and custom Map Reduce programs in Java.
Environment: CDH4, HDFS, Map Reduce, Hive, Oozie, Java, PIG, Shell Scripting, Linux, HUE, SQOOP, Flume, Kafka, DB2, and Oracle.
Hadoop Developer
Confidential, Plano, TX
Responsibilities:
- Installed and configured Apache Hadoop, Hive and Pig environment on the prototype server.
- Good knowledge in Configured MySql Database to store Hive metadata.
- Responsible for loading unstructured data into Hadoop File System (HDFS).
- Created POC to store Server Log data in Mongo DB to identify System Alert Metrics.
- Created Reports and Dashboards of Server Alert Data.
- Created Map Reduce Jobs using Pig Latin and Hive Queries.
- Data is collected from Tera data and pushing into Hadoop using SQOOP.
- Used SQOOP tool to load data from RDBMS into HDFS.
- Using Apache Kafka for Integration and data processing pipelines.
- Cluster coordination services through Zoo Keeper.
- Automated all the jobs for pulling data from FTP server to load data into Hive tables, using Oozie workflows.
- Created Reports and Dashboards using structured and unstructured data.
- Maintained documentation for corporate Data Dictionary with attributes, table names and constraints. Extensively worked with SQL scripts to validate the pre and post data load.
- Created unit test plans, test cases and reports on various test cases for testing the data loads.
- Worked on integration testing to verify load order, time window.
- Performed the Unit Testing which validate the data is processed correctly which provides a qualitative check of overall data flow up and deposited correctly into targets.
- Responsible for post production support and SME to the project.
- Involved in the System and User Acceptance Testing.
- Performing data migration from Legacy Databases RDBMS to HDFS using SQOOP.
- Writing Pig scripts for data processing.
- Responsible for understanding the scope of the project and requirement gathering.
- Involved in creation of Test Cases for JUnit Testing.
Environment: Hadoop, Pig, Hive, Java, SQOOP, Kafka, H Base, no SQL, Oracle 10g, PL/SQL, SQL Server, SQLDeveloperToad, Windows NT, Stored Procedures.
Hadoop Developer
Confidential, Bangalore, KA
Responsibilities:
- Good experience in all phases of development activities from requirements collection to production support.
- Detailed understanding of current system and find out the different sources of data.
- Involved in Cluster Troubleshooting and upgrades.
- Implemented Map reduce jobs for Vehicle Build Summary and populate which parts are short and in turn which Trucks will not be able to build in stipulated time.
- Analysis of various parts included in manufacturing are available and various tests during manufacturing phase.
- Design and implement map reduce jobs to support distributed processing using Java, Hbase.
- Creating Hive external tables on the map reduce output before partitioning, bucketing is applied on top of it.
- Providing pivotal graphs in order to show the trends.
- Experience in written Oozie work flow and scheduling for the jobs automation.
- Developed and maintain several batch jobs to run automatically depending on business requirements.
- Import and export data between the environments like DB2 and HDFS/H Base using sqoop.
- Worked on Storm Topology integrated with Kafka in order to achieve near to real time processing of various Tests involved in Manufacturing of Trucks.
- Performed unit testing and internal deployments for all the deliverables to the Business.
- Written Java classes to test UI and Web services through JUnit.
- Used spring, Hibernate module as an Object Relational mapping tool for back end operations over SQL database.
- Developed the business components using EJB Session Beans.
- Experience on database design for new modules and developed the persistence layer based on Hibernate.
Environment: Java Map-Reduce, Core Java, Python, UNIX, MYSQL, Eclipse, Oozie, Sqoop, Flume and, Oracle, Tera data and Mysql.
Hadoop Developer
Confidential, Hyderabad, TS
Responsibilities:
- Good experience in Business Requirements and identified mapping documents required for system.
- Performed requirement gathering, analyzing and negotiating customer requirements.
- Developed User Interface using HTML, CSS, Java script and JSP for various functionalities.
- Developed Stored Procedures and Triggers using SQL, PL/SQL in order to store and retrieve data and implemented the J2EE architecture using struts based on MVC pattern.
- Experience on IDE tools like RAD (Rational Application Developer) and Eclipse.
- Struts Action followed by design patterns such as session, facade, singleton were used for business flow.
- Responsible for creating, sending and receiving messages by using SOAP protocols.
- Handled Configured Hibernate and spring through required configuration and XML files.
- Extensively used Spring Inversion-of-Control for Dependency Injection.
- Experience with WebSphere, Apache and IBM HTTP, used Log4j for logging and JUnit for Unit Testing.
- Good experience with writing SQL queries using Joins and Stored Procedures.
- Experience in Waterfall Software Development Lifecycle Model.
- Exported data from DB2 to HDFS using Sqoop.
- Developed Map Reduce jobs using Java API.
- Installed and configured Pig and also wrote Pig Latin scripts.
- Wrote Map Reduce jobs using Pig Latin.
- Developed workflow using Oozie for running MapReduce jobs and Hive Queries.
- Worked on Cluster coordination services through Zookeeper.
- Worked on loading log data directly into HDFS using Flume.
- Compressing data using LZO
Environment: HADOOP, HDFS, Map Reduce, UNIX., Mongo DB, Flume Mahout, Oozie.
