Hadoop Developer Resume
Deerfield, IL
SUMMARY
- Hadoop Developer wif around 8 years of experience in Information Technology & 4+ years in Hadoop Ecosystem.
- Expertise in Hadoop Ecosystem components HDFS, Map Reduce, Hive, Pig, Sqoop, Hbase and Flume for Data Analytics.
- Have a hands - on experience on fetching teh live stream data from DB2 to Hbase table using Spark Streaming and Apache Kafka.
- Capable of processing large sets of structured, semi-structured and unstructured data sets.
- Experience in job workflow scheduling and monitoring tools like Oozie and Zookeeper.
- Expertise in writing Map-Reduce Jobs in Java for processing large sets of structured, semi-structured and unstructured data sets and store them in HDFS.
- Experience in developing Custom UDFs for datasets in Pig and Hive.
- Proficient in designing and querying teh NoSQL databases like HBase.
- Experience in designing and developing tables in HBase and storing aggregated data from Hive Table.
- Knowledge on integrating different eco-systems like HBase - Hive, HBase - Pig
- Experience on streaming data using Apache Flume.
- Good Knowledge in Apache Spark and SparkSQL.
- Skilled on migrating teh data from different databases to Hadoop HDFS and Hive using Sqoop.
- Deep Knowledge in teh core concepts of MapReduce Framework and Hadoop ecosystem
- Analyzed large structured datasets using Hive's data warehousing infrastructure
- Extensive knowledge of creating manage tables and external tables in Hive Eco system.
- Worked extensively in design and development of business process using SQOOP, PIG, HIVE, HBASE
- Knowledge on Spark framework for batch and real time data processing.
- Knowledge on Scala Programming Language.
- Hands on experience in database design using PL/SQL to write Stored Procedures, Functions, Triggers and strong experience in writing complex queries, using Oracle, DB2, and MySQL.
- Hands on Experience in Designing and Executing of Test Cases to ensure application functionality using Selenium IDE, Selenium Web driver, Firebug and TestNG.
- Good knowledge in Software Development Life Cycle (SDLC) and Software Testing Life Cycle (STLC).
TECHNICAL SKILLS
Big Data Frameworks: Hadoop, Spark, Scala, Hive, Kafka, AWS, HBase, Flume, Pig, Sqoop, Map Reduce, Cloudera.
Big data distribution: Cloudera, Amazon EMR
Programming languages: Core Java, Scala, SQL, Shell Scripting
Operating Systems: Windows, Linux (Ubuntu)
Databases: Oracle, SQL Server
Designing Tools: Eclipse
Java Technologies: JSP, Servlets, Junit, Spring, Hibernate
Web Technologies: XML, HTML, JavaScript, JVM, JQuery, JSON
Linux Experience: System Administration Tools, Puppet, Apache
Web Services: Web Service (RESTfuland SOAP)
Frame Works: Jakarta Struts 1.x, Spring 2.x
Development methodologies: Agile, Waterfall
Logging Tools: Log4j
Application / Web Servers: Apache Tomcat, WebSphere, Weblogic
Messaging Services: Kafka
Version Tools: Git, SVN and CVS
Analytics: Tableau
PROFESSIONAL EXPERIENCE
Confidential, Deerfield, IL
Hadoop Developer
Responsibilities:
- Maintained System integrity of all sub-components (primarily HDFS, MR, HBase, and Hive).
- Integrated teh hive warehouse wif HBase
- Migrating teh needed data from MySQL into HDFS using Sqoop and importing various formats of flat files into HDFS.
- Load teh data into HBase tables for UI web application.
- Written customized Hive UDFs in Java where teh functionality is too complex.
- Maintain System integrity of all sub-components related to Hadoop.
- Designed and created Hive external tables using shared meta-store instead of derby wif partitioning, dynamic partitioning and buckets.
- HiveQL scripts to create, load, and query tables in aHive.
- Supported Map Reduce Programs those are running on teh cluster
- Monitored System health and logs and respond accordingly to any warning or failure conditions.
- Performed advanced procedures like text analytics and processing, using teh in-memory computing capabilities of Spark using Scala.
- Worked on Big Data Integration and Analytics based on Hadoop, SOLR, Spark, Kafka, Storm and web Methods technologies.
- Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
- Real time streaming teh data using Spark wif Kafka.
- Worked on migrating MapReduce programs into Spark transformations using Spark and Scala
- Generate final reporting data using Tableau for testing by connecting to teh corresponding Hive tables using Hive ODBC connector.
- Strongly recommended to bring in Elastic Search and was responsible for installing, configuring and administration.
- Developing and maintaining efficient ETL Talend jobs for Data Ingest.
- Worked on Talend RTX ETL tool, develop jobs and scheduled jobs in Talend integration suite.
- Modified reports and Talend ETL jobs based on teh feedback from QA testers and Users in development and staging environments.
- Involved in migration Hadoop jobs into higher environments like SIT, UAT and Prod.
Environment: Cloudera, HDFS, Hive, Scala, Map Reduce, Storm, Java, HBase, Pig, Sqoop, Shell Scripts, Oozie, Coordinator, MySQL, Tableau, Elastic search, Talend and SFTP.
Confidential, Chicago IL
Hadoop Developer
Responsibilities:
- Experience in supporting and managing HadoopClusters using Hortonworks distributions by deploying it on AWS cloud.
- Collected aggregated large amount of web log data from different sources such as web servers, mobile and network devices using Apache Kafka.
- Ingestion framework was developed in python.
- Creating teh RDD’s, Data frames for faster execution and performing data transformations and actions using Spark.
- Developed optimal strategies for distributing teh web log data over teh cluster.
- Implemented Hive Generic UDF's to incorporate business logic into Hive Queries.
- Configuring Spark Streaming to receive real time data from teh Kafka for high speed data processing and Store teh stream data to HDFS.
- Used Scala to read text data, CSV data, image data from HDFS, S3 and Hive
- Worked on Spark SQL for faster execution of Hive queries using Spark SQL Context.
- Invoking HQL from Spark SQL and storing it in parquet file as teh storage format.
- Creating Hive tables and working on them using HQL for evaluation, filtering, loading and storing of data.
- Design and implemented Incremental Imports using Sqoop into Hive tables.
- Analyzed teh web log data using teh HQL to extract number of unique visitors per day, page views, visit duration, most visited page on website.
- Developed ETL jobs and scheduling it in Oozie and performing ingestion into HDFS.
- Cluster authentication(security) is maintained by using Kerberos.
- Good experiences in designing and developing various kinds of data visualizations, dashboards, stories, and workbooks in Tableau.
- Involved in Agile methodologies, daily scrum meetings, sprint planning.
Environment: Hortonworks, HDFS, Hive, Sqoop, Oozie, Storm, Scala 2.11.8, Spark 2.0, Spark SQL, Spark streaming, Python, Kafka, GitHub, Kerberos, AWS, Amazon S3, Amazon EC2, Amazon EBS, Tableau.
Confidential, Salt Lake City, Utah
Hadoop Developer
Responsibilities:
- Developed Big Data Solutions that enabled teh business and technology teams to make data-driven decisions on teh best ways to acquire customers and provide them business solutions.
- Installed and configured Apache Hadoop, Hive, and HBase.
- Worked on Hortonworks cluster, which was used to process teh big data.
- Developed multiple map reduce jobs in java for data cleaning and pre-processing.
- Sqoop was used to pull data into Hadoop distributed file system from RDBMS and vice versa
- Defined workflows using Oozie.
- Used Hive to create partitions on hive tables and analyzes dis data to compute various metrics for reporting.
- Created Data model for Hive tables
- Good Experience in managing and reviewing Hadoop log files
- Used Pig as ETL tool to do transformations, joins and pre-aggregations before loading data onto HDFS.
- Worked on large sets of structured, semi structured and unstructured data
- Responsible to manage data coming from different source
- Installed and configured Hive and developed Hive UDFs to extend core functionality of hive
- Responsible for loading data from UNIX file systems to HDFS.
- Arrange for on-site job coaching or assistive devices, such as specially equipped wheelchairs, to halp clients adapt to work or school environments.
- Locate barriers to client employment, such as inaccessible work sites, inflexible schedules, and transportation problems, and work wif clients to develop strategies for overcoming these barriers.
Environment: Cloudera, Apache Hadoop 2.x, MapReduce, HDFS, Hive, HBase, Pig, Oozie, Linux, Java 7, Eclipse Mars wif physicians, psychologists, occupational therapists, and other professionals to develop and implement client rehabilitation programs.
Confidential, Eden Prairie, MN
Hadoop Developer-Health Care
Responsibilities: -
- Implemented Big Data platforms using Cloudera CDH4 as data storage, retrieval and processing systems.
- Involved in installing, configuring andmanaging HadoopEcosystem components like Hive, Pig, Sqoop and Flume.
- Responsible for loading data from Teradata database into a Hadoop Hive data warehousing layer, and performing data transformations using Hive.
- Partitioned teh collected people's data by disease type and medication prescribed to teh patient to improve teh query performance.
- Used Hive Query language (HQL) to further analyze teh data to identify issues and behavioral patterns.
- Used Apache Spark and Scala language to find patients wif similar symptoms in teh past and medications used for them to achieve best results.
- Configured Apache Sentry server to provide authentication to various users and provide authorization for accessing services that they were configured to use.
- Worked on Kafka while dealing wif raw data, by transforming into new Kafka topics for further consumption.
- Installed and configured Hortonworks Sandbox as part of POC involving Kafka-Storm-HDFS data flow.
- Log data from webservers across teh environments is pushed into associated Kafka topic partitions, SparkSQL is used to calculate teh most prevalent diseases in each city from dis data.
- Implemented Zookeeper for job synchronization
- Deployed NoSQL database HBase to store teh outputs of teh jobs.
- Supported operations team in Hadoop cluster maintenance activities including commissioning and decommissioning nodes and upgrades.
- Extended teh core functionality of Hive language by writing UDF's and UDAF'S
- Worked on Oozie workflow engine for job scheduling.
Confidential, Basking Ridge, NJ
Hadoop Developer
Responsibilities:
- Developed multiple Map Reduce jobs in Java for data cleaning and preprocessing
- Responsible for design and creation of Hive tables, partitioning, bucketing, loading data and writing hive queries.
- Created HBase tables to store various data formats of personally identifiable information (PII) data coming from different portfolios.
- Wrote Pig scripts to transform teh data in HDFS
- Responsible to manage data coming from different sources.
- Involved in managing and reviewing Hadoop log files.
- Installed Oozie workflow engine to run multiple Hive and Pig jobs.
Environment:Cloudera, Apache Hadoop 2.x, MapReduce, HDFS, Hive, HBase, Pig, Oozie, Sqoop, Linux, Oracle 11g, Java 7, Eclipse
Confidential, Bridgewater, NJ
Sr. Java Developer
Responsibilities:
- Full life cycle experience including requirements analysis, high level design, detailed design, UMLs, data model design, coding, testing and creation of functional and technical design documentation.
- Used Spring Framework for MVC architecture wif Hibernate to implement DAO code and also used Web Services to interact other modules and integration testing.
- Developed and implemented GUI functionality using JSP, JSTL, Tiles and AJAX.
- Designed database and involved in developing SQL Scripts.
- Used SQL navigator as a and involved in testing teh application.
- Implementing teh Design Patterns like MVC-2, Front Controller, Composite view and all Struts framework design patterns to improve teh performance.
- Used Clear case, and also subversion for maintaining teh source version control.
- Wrote Ant scripts to automate teh builds and installation of modules.
- Involved in writing Test plans and conducted Unit Tests using JUnit.
- Used Log4j for logging statements during development.
- Design and implementation of log data indexing and search module, and optimization for performance and accuracy. To provide a full text search capability for archived log data, utilizing Apache Lucene library.
- Involved in teh testing and integrating of teh program at teh module level.
- Worked wif production support team in debugging and fixing various production issues.
Environments:Java1.5,AJAX,XML,Spring3.0,Hibernate2.0,Struts1.2,Webservices,Websphere7.0,Junit,Oracle10g,SQL, PL/SQL, log4j, RAD 7.0/7.5, Clear case, Unix, HTML, CSS, Java script.
Confidential
Java Developer
Responsibilities:
- Coordinated wif teh business analysts, project managers to analyze new propose Ideas/Requirements, designed teh integrated tool, developed and implemented all teh modules.
- Designed database and involved in developing SQL Scripts.
- Used Case Studio for developing teh DB Design and generating SQL file for various databases.
- Contributed significantly in designing teh Object Model for teh project as senior developer and Architect.
- Responsible for development of Business Services.
- Developed Business Rules for teh project using Java.
- Developed portal screens using JSP, Servlets, and Struts framework.
- Developed teh test plans and involved in testing teh application.
- Implementing teh Design Patterns like MVC-2, Front Controller, Composite view and all Struts framework design patterns to improve teh performance.
- Re-engineered OMT Wholesale Internet Service Engine (WISE) using an “n” tiered architecture involving latest technologies like EJB, XML and JAVA.
- Used CVS for maintaining teh source version control.
- Used Log4j for logging statements during development.
- Design and implementation of log data indexing and search module, and optimization for performance and accuracy. To provide a full text search capability for archived log data, utilizing Apache Lucene library.
- Used Spring Framework for MVC architecture wif Hibernate to implement DAO code and also used Web Services to interact other modules and integration testing.
- Developed and implemented GUI functionality using JSP, JSTL, Tiles and AJAX.
- Designed database and involved in developing SQL Scripts.
- Used SQL navigator as a tool to interact wif DB Oracle 10g.
- Developed portal screens using JSP, Servlets, and Struts framework
- Involved in writing Test plans and conducted Unit Tests using JUnit.
- Used Log4j for logging statements during development
Environments:Java, J2EE, Struts 1.2/2.0, JDK, JSP, Servlets, EJB 3.0, Java Beans, JavaScript, HTML XML, Eclipse, CORBA SSL, JUnit, Log4j, CVS, Deployment in Web Logic.
