We provide IT Staff Augmentation Services!

Etl Hadoop Developer Resume

0/5 (Submit Your Rating)

GA

SUMMARY

  • Over 8 years of experience spread across Hadoop, Java and ETL, that includes extensive experience into Big Data Technologies and in development of standalone and web applications in multi - tiered environment using Java, Hadoop, Hive, HBase, Impala, Pig, Sqoop, J2EE Technologies (Spring, Hibernate), Oracle, HTML, Java Script.
  • Extensive experience on BigData Analytics with hands on experience in writing MapReduce jobs on Hadoop Ecosystem including Hive and Pig
  • Excellent knowledge on Hadoop architecture; as in HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce programming paradigm
  • The system consists of several applications, highly distributive, scalable and large in nature using Cloudera Hadoop
  • BigData frame work and Eco system with evolving technologies
  • Experience with distributed systems, large scale non-relational data stores, MapReduce
  • Systems, data modeling, and big data systems, Tera Data.
  • Experience on Apache, Cloudera, Hortonworks Hadoop distributions
  • Involved in developing solutions to analyze large data sets efficiently
  • Excellent hands on with importing and exporting data from different Relational Database Systems like Mysql and Oracle into HDFS and Hive and vice-versa, using Sqoop.
  • Knowledge in Kafka installation & integration with Spark Streaming
  • Hands-on experience in writing Pig Latin scripts, working with grunt shells and job scheduling with Oozie
  • Install Kafka on Hadoop cluster and configure producer and consumer coding part in java to establish connection from twitter source to HDFS with popular hash tags.
  • Experience in analyzing data using Hive QL, Pig Latin, and custom Map Reduce programs in Java
  • Expertize in Marathon for long-running services, using it to start Job Tracker or Spark schedule
  • Launching the actual batch jobs like Hadoop/Spark tasks on a batch framework like Chronos
  • Restarting tasks using Marathon during the complete/fail scenario and setting up scheduled jobs and complex workflows
  • Experience with web-based UI development using jQuery, ExtJS, CSS, HTML, HTML5, XHTML and Java script
  • Knowledge of job workflow scheduling and monitoring tools like Oozie and Zookeeper
  • Experience with databases like DB2, Oracle 9i, Oracle 10g, MySQL,Couchbase, SQL Server, lambda and MS Access
  • Expertize in Elastic Search in Hadoop that bridges that gap, letting us the leverage for the best of Hadoop's big data analytics and the real-time analytics
  • Good Understanding in Couch base Server cluster, document based database system that makes use of a caching layer to provide very fast access to data by storing the majority of data in RAM
  • Experience in creating complex SQL Queries and SQL tuning, writing PL/SQL blocks like stored procedures, Functions, Cursors, Index, triggers and packages
  • Very good understanding on NOSQL databases like MongoDB and HBase
  • Have good Knowledge in ETL and hands on experience in Informatica ETL
  • Experience in Docker Container Execution which runs in non-secure mode of HDFS and YARN
  • Extensive experience in creating Class Diagrams, Activity Diagrams, Sequence Diagrams using Unified Modeling Language (UML)
  • Experienced in SDLC, Agile (SCRUM) Methodology, Iterative Waterfall
  • Experience in developing test cases, performing Unit Testing, Integration Testing, experience in QA with test methodologies and skills for manual/automated testing using tools like WinRunner, JUnit
  • Experience with various version control systems Clear Case, CVS, SVN
  • Expertise in extending Hive and Pig core functionality by writing custom UDFs
  • Development Experience with all aspects of software engineering and the development life cycle
  • Strong desire to work for a fast-paced, flexible environment
  • Hands on experience in Spark 1.2 /1.3 on Cloudera 5.3 distribution, expertise in creating Spark RDD (Resilient Distributed Dataset), performing transformations actions.
  • Proactive problem solving mentality that thrives in an agile work environment
  • Expertize in running Hadoop on Mesos to add the Hadoop Mesos on to the Hadoop distribution and setting new configuration properties
  • Good Experience on SDLC (Software Development Life cycle)
  • Exceptional ability to learn new technologies and to deliver outputs in short deadlines
  • Worked with developers, DBAs, and systems support personnel in elevating and automating successful code to production
  • Monitoring Map Reduce Jobs and Yarn Applications
  • Possess strong Communication skills of written, oral, interpersonal and presentation
  • Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities

TECHNICAL SKILLS

Languages: Hadoop, MapReduce, Pig, Hive, Sqoop, Zookeeper, Oozie, Python, Impala

Functional: Big Data/ DW/ Business Intelligence/ Mainframes/Project Management

Distribution: Cloudera/Hortonworks

ETL Tools: Informatica & Datastage

Database: HDFS, No SQL (HBASE, MongoDB), Oracle, DB2, IMS DB, Couchbase, Teradata

Build Tools: Eclipse, Maven

Learning Skills: Apache Spark, Storm, Kafka Professional Experience

Management Tool: Accurev

Automation Tool: Control-M

PROFESSIONAL EXPERIENCE

Confidential, GA

ETL Hadoop Developer

RESPONSIBILITIES:

  • Involved in design and development phases of Software Development Life Cycle (SDLC) using Scrum methodology
  • Developed data pipeline using Flume, Sqoop, Pig and MapReduce to ingest customer behavioral data and purchase histories into HDFS for analysis
  • Developed job flows in Oozie to automate the workflow for extraction of data from warehouses and weblogs
  • Used Pig as ETL tool to do transformations, event joins, filter both traffic and some pre-aggregations before storing the data onto HDFS
  • Written Hive queries to parse the logs and structure them in tabular format for log data
  • Installation, configuration and administration experience in Big Data platforms, Hortonworks, Apache Hadoop on Redhat, and Centos as a data storage, retrieval, and processing systems
  • Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way
  • Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way
  • Used hive optimization techniques during joins and best practices in writing hive scripts.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting
  • Experienced in managing and reviewing the Hadoop log files
  • Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS
  • Worked on Oozie workflow engine for job scheduling
  • Developed custom implementation for Partioner, Input / Output Formats, Record Reader and Writers
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard
  • Loaded the aggregated data onto DB2 for reporting on the dashboard
  • Project also involved building analytical reports using Sql server and Excel
  • Monitoring and Debugging Hadoop jobs/Applications running in production
  • Worked on Providing User support and application support on Hadoop Infrastructure
  • Reviewing ETL application use cases before on boarding to Hadoop
  • Worked on Evaluating, comparing different tools for test data management with Hadoop
  • Helped and directed testing team to get up to speed on Hadoop Application testing
  • Worked on Installing 20 node UAT Hadoop cluster
  • Worked on the Analytics Infrastructure team to develop a stream filtering system on top of Apache Kafka
  • Used Scala and Java to dynamically introspect thrift messages
  • Install Kafka on Hadoop cluster and configure producer and consumer coding part in java to establish connection from twitter source to HDFS with popular hash tags.
  • Perform ETL on data from Dev and QA box for staging it for micro batching to Spark
  • Design a real time computation engine using Flume, Kafka, a Complex Event Processing engine to provide increase customer base

ENVIRONMENT: Horton works Data Platform, Hadoop 2.2, HDFS, Hive, Pig, Linux, Sqoop, Flume, oozie, Yarn, MR, HBase

Confidential, NY

ETL Hadoop Developer

RESPONSIBILITIES:

  • Developed multiple MapReduce programs for analyzing the insurance data of the customer and produce summary results from Hadoop to downstream systems
  • Worked on importing and exporting data from Oracle and DB2 into HDFS using Sqoop
  • Developed data pipeline using Flume to ingest customer behavioral data and financial histories into HDFS for analysis
  • Prepared the best practices in writing map reduce programs and hive scripts
  • Scheduled a workflow to import the weekly transactions in the revenue department from RDBMS database using Oozie, Couchbase.
  • Built wrapper shell scripts to hold these Oozie workflow
  • Developed PIG Latin scripts to transform the log data files and load into HDFS
  • Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS, Tera data
  • Hands on experience with NoSQL databases like Cassandra for POC (proof of concept) in storing URL's and images
  • Developed hive UDF for functions that were not preexisting in Hive like the rank etc.
  • Created External Hive tables and involved in data loading and writing Hive UDFs
  • Experienced in implementing POC's to migrate iterative map reduce programs into Spark transformations using Scala
  • Created concurrent access for hive tables with shared and exclusive locking that can be enabled in hive with the help of Zookeeper implementation in the cluster
  • Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions
  • Developed Unit test cases using MR unit for map reduce code
  • Involved in creating Hadoop streaming jobs
  • Involved in Installing, Configuring Hadoop Eco System, Cloudera Manager using CDH4 Distribution
  • Used the most popular streaming tool Kafka to load the data on Hadoop File system and move the same data to MongoDB -NoSQL, Couchbase, database.
  • Using Kafka we stream the data from source to Hadoop.

ENVIRONMENT: Hadoop, MapReduce, Hive, HDFS, PIG, Impala, Sqoop, Kafka, Spark, Oozie, lambda, Cloudera, Flume, HBase, ZooKeeper, CDH3, Cassandra, Oracle, NoSQL and Unix/Linux.

Confidential, MN

ETL Hadoop Developer

RESPONSIBILITIES:

  • Provide technical designs, architecture, Support automation, installation and configuration tasks and upgrades and planning system upgrades of Hadoop cluster
  • Design development and architecture of the Hadoop cluster, map reduce processes, Hbase system
  • Design and develop process framework and support data migration in Hadoop system
  • Experience in working with NoSQL databases like HBase and Cassandra
  • Experience in the analysis, design, development and MR Unit testing of Hadoop Cluster Structureusing Java
  • Developed Map reduce program using Java and deployed the Jobs at the customer environment.
  • Designed the web-based system for business analytics and data visualization in Hadoop ecosystem integrated Tableau on Hadoop frame work to visualize and analyze data.
  • Hive is used for data delivery
  • Cluster Manager for Apache Hadoop that enables Bright Computing’s leading cluster management software to build, manage, and maintain big data clusters using Apache Hadoop
  • Worked on several apaches Hadoop projects. Maps reduce programs were developed using Hadoop java API and also using hives and pig
  • Designing and implementing solutions for Hadoop AFMS system validation for all components of data analytics stack
  • Worked with scoop to import/export data from relational database to Hadoop and flume to collect data and populate in Hadoop
  • Implemented and integrated Hadoop based business intelligence and Data Warehouse system including implementations of searching, filtering, indexing, aggregation for reporting and report generation and general information retrieval
  • Maintained Hadoop clusters for dev/staging/production. Trained the development, administration, testing and analysis teams on Hadoop framework and Hadoop eco system
  • Give extensive presentations about the Hadoop ecosystem, best practices, data architecture in Hadoop
  • Data warehouse, Business Intelligence architecture design and develop. Designed the ETL process from various sources in to Hadoop/HDFS for analysis and further processing
  • Provide review and feedback for existing physical architecture, data architecture, analysis, designs and code. Designed the next generation architecture for unstructured data
  • Debug and solve issues as the subject matter expert focusing issues around data sciences and processing
  • Define business and technical requirements, design Proof of Concept for evaluating afms agencies data evaluation criteria and scoring and select data integration and information management
  • Collaborate with infrastructure and security architects to integrate enterprise information architecture into overall enterprise architecture
  • Integrating Big data technologies and analysis tools into the overall architecture

ENVIRONMENT: Hadoop, Cassandra, Hbase, HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Oozie, Java, JSP, RMI, JNDI, JDBC, Tomcat, Apache, Shell Scripting.

Confidential

Java Developer

RESPONSIBILITIES:

  • Used Microsoft Visio for designing the Use Case Diagrams, Class model, Sequence diagrams, and Activity diagrams for SDLC process of the application
  • Implemented GUI pages by using JSP, JSTL, HTML, DHTML, XHTML, CSS, JavaScript, AJAX
  • Configured the project on WebSphere 6.1 application servers
  • Implemented the online application by using Core Java, Jdbc, JSP, Servlets and EJB 1.1, Web Services, SOAP, WSDL
  • Communicated with other Health Care info by using Web Services with the help of SOAP, WSDL JAX-RPC
  • Implemented Singleton, factory design pattern, DAO Design Patterns based on the application requirements
  • Used SAX and DOM parsers to parse the raw XML documents
  • Used RAD as Development IDE for web applications
  • Used Clear Case for Version Control tool and Clear Quest for bug tracking tool
  • Configured job scheduling in Linux using shell scripts and Crontab
  • Developed test plan documents for all back end database modules
  • Deployed the project in Linux environment

ENVIRONMENT: JDK 1.5, JSP, WebSphere, JDBC,EJB2.0, XML, DOM, SAX, XSLT, Apache Commons Digester, Apache Commons, CSS, HTML, JNDI, Web Services, WSDL, SOAP, RAD, SQL, PL/SQL, JavaScript, DHTML, XHTML, Oracle10g, Java Mail, PL/SQL Developer, Toad, POI Reports, Log4j, ANT, Clear Case, Windows XP, Red Hat Linux.

Confidential

Java Developer

RESPONSIBILITIES:

  • Developed the application based on MVC (Model-View-Controller) Architecture using Spring Web MVC
  • Used Microsoft Visio for designing the Use Case Diagrams, Class model, Sequence diagrams, and Activity diagrams for SDLC process of the application
  • Implemented GUI pages by using JSP, JSTL, HTML, DHTML, XHTML, CSS, JavaScript, AJAX
  • Extensively used Java multi-threading to implement batch Jobs with JDK 1.5 features
  • Configured the project on Web Logic 10.3 application servers
  • Implemented the online application using Core Java, JDBC, JSP, Servlets, spring, Hibernate, Web Services, SOAP, and WSDL
  • Communicated with other Health Care info using Web Services with the help of SOAP, WSDL
  • Tested the web services with SOAP UI tool
  • Implemented Singleton, Factory Design Pattern, DAO Design Patterns based on the application requirements
  • Used SAX and DOM parsers to parse the raw XML documents
  • Used RAD as Development IDE for web applications
  • Tuning complex database queries and joining the tables to improve the performance of the application
  • Designed and developed base classes, framework classes and common re-usable components
  • Used SVN for Version Control tool and Silk Central for bug tracking tool

ENVIRONMENT: JDK 1.5, JSP, web logic 10.3, JDBC, XML, DOM, SAX, CSS, HTML, JNDI, Web Services, WSDL, SOAP, Eclipse 3.0, SQL, PL/SQL, JavaScript, HTML, Oracle11g, PL/SQL Developer, Log4j, ANT, SVN, Silk Central.

We'd love your feedback!