Etl Hadoop Developer Resume
GA
SUMMARY
- Over 8 years of experience spread across Hadoop, Java and ETL, that includes extensive experience into Big Data Technologies and in development of standalone and web applications in multi - tiered environment using Java, Hadoop, Hive, HBase, Impala, Pig, Sqoop, J2EE Technologies (Spring, Hibernate), Oracle, HTML, Java Script.
- Extensive experience on BigData Analytics with hands on experience in writing MapReduce jobs on Hadoop Ecosystem including Hive and Pig
- Excellent knowledge on Hadoop architecture; as in HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce programming paradigm
- The system consists of several applications, highly distributive, scalable and large in nature using Cloudera Hadoop
- BigData frame work and Eco system with evolving technologies
- Experience with distributed systems, large scale non-relational data stores, MapReduce
- Systems, data modeling, and big data systems, Tera Data.
- Experience on Apache, Cloudera, Hortonworks Hadoop distributions
- Involved in developing solutions to analyze large data sets efficiently
- Excellent hands on with importing and exporting data from different Relational Database Systems like Mysql and Oracle into HDFS and Hive and vice-versa, using Sqoop.
- Knowledge in Kafka installation & integration with Spark Streaming
- Hands-on experience in writing Pig Latin scripts, working with grunt shells and job scheduling with Oozie
- Install Kafka on Hadoop cluster and configure producer and consumer coding part in java to establish connection from twitter source to HDFS with popular hash tags.
- Experience in analyzing data using Hive QL, Pig Latin, and custom Map Reduce programs in Java
- Expertize in Marathon for long-running services, using it to start Job Tracker or Spark schedule
- Launching the actual batch jobs like Hadoop/Spark tasks on a batch framework like Chronos
- Restarting tasks using Marathon during the complete/fail scenario and setting up scheduled jobs and complex workflows
- Experience with web-based UI development using jQuery, ExtJS, CSS, HTML, HTML5, XHTML and Java script
- Knowledge of job workflow scheduling and monitoring tools like Oozie and Zookeeper
- Experience with databases like DB2, Oracle 9i, Oracle 10g, MySQL,Couchbase, SQL Server, lambda and MS Access
- Expertize in Elastic Search in Hadoop that bridges that gap, letting us the leverage for the best of Hadoop's big data analytics and the real-time analytics
- Good Understanding in Couch base Server cluster, document based database system that makes use of a caching layer to provide very fast access to data by storing the majority of data in RAM
- Experience in creating complex SQL Queries and SQL tuning, writing PL/SQL blocks like stored procedures, Functions, Cursors, Index, triggers and packages
- Very good understanding on NOSQL databases like MongoDB and HBase
- Have good Knowledge in ETL and hands on experience in Informatica ETL
- Experience in Docker Container Execution which runs in non-secure mode of HDFS and YARN
- Extensive experience in creating Class Diagrams, Activity Diagrams, Sequence Diagrams using Unified Modeling Language (UML)
- Experienced in SDLC, Agile (SCRUM) Methodology, Iterative Waterfall
- Experience in developing test cases, performing Unit Testing, Integration Testing, experience in QA with test methodologies and skills for manual/automated testing using tools like WinRunner, JUnit
- Experience with various version control systems Clear Case, CVS, SVN
- Expertise in extending Hive and Pig core functionality by writing custom UDFs
- Development Experience with all aspects of software engineering and the development life cycle
- Strong desire to work for a fast-paced, flexible environment
- Hands on experience in Spark 1.2 /1.3 on Cloudera 5.3 distribution, expertise in creating Spark RDD (Resilient Distributed Dataset), performing transformations actions.
- Proactive problem solving mentality that thrives in an agile work environment
- Expertize in running Hadoop on Mesos to add the Hadoop Mesos on to the Hadoop distribution and setting new configuration properties
- Good Experience on SDLC (Software Development Life cycle)
- Exceptional ability to learn new technologies and to deliver outputs in short deadlines
- Worked with developers, DBAs, and systems support personnel in elevating and automating successful code to production
- Monitoring Map Reduce Jobs and Yarn Applications
- Possess strong Communication skills of written, oral, interpersonal and presentation
- Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities
TECHNICAL SKILLS
Languages: Hadoop, MapReduce, Pig, Hive, Sqoop, Zookeeper, Oozie, Python, Impala
Functional: Big Data/ DW/ Business Intelligence/ Mainframes/Project Management
Distribution: Cloudera/Hortonworks
ETL Tools: Informatica & Datastage
Database: HDFS, No SQL (HBASE, MongoDB), Oracle, DB2, IMS DB, Couchbase, Teradata
Build Tools: Eclipse, Maven
Learning Skills: Apache Spark, Storm, Kafka Professional Experience
Management Tool: Accurev
Automation Tool: Control-M
PROFESSIONAL EXPERIENCE
Confidential, GA
ETL Hadoop Developer
RESPONSIBILITIES:
- Involved in design and development phases of Software Development Life Cycle (SDLC) using Scrum methodology
- Developed data pipeline using Flume, Sqoop, Pig and MapReduce to ingest customer behavioral data and purchase histories into HDFS for analysis
- Developed job flows in Oozie to automate the workflow for extraction of data from warehouses and weblogs
- Used Pig as ETL tool to do transformations, event joins, filter both traffic and some pre-aggregations before storing the data onto HDFS
- Written Hive queries to parse the logs and structure them in tabular format for log data
- Installation, configuration and administration experience in Big Data platforms, Hortonworks, Apache Hadoop on Redhat, and Centos as a data storage, retrieval, and processing systems
- Involved in creating Hive tables, loading with data and writing hive queries that will run internally in map reduce way
- Involved in creating Hive tables, loading with data and writing hive queries that will run internally in MapReduce way
- Used hive optimization techniques during joins and best practices in writing hive scripts.
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting
- Experienced in managing and reviewing the Hadoop log files
- Used Pig as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS
- Worked on Oozie workflow engine for job scheduling
- Developed custom implementation for Partioner, Input / Output Formats, Record Reader and Writers
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard
- Loaded the aggregated data onto DB2 for reporting on the dashboard
- Project also involved building analytical reports using Sql server and Excel
- Monitoring and Debugging Hadoop jobs/Applications running in production
- Worked on Providing User support and application support on Hadoop Infrastructure
- Reviewing ETL application use cases before on boarding to Hadoop
- Worked on Evaluating, comparing different tools for test data management with Hadoop
- Helped and directed testing team to get up to speed on Hadoop Application testing
- Worked on Installing 20 node UAT Hadoop cluster
- Worked on the Analytics Infrastructure team to develop a stream filtering system on top of Apache Kafka
- Used Scala and Java to dynamically introspect thrift messages
- Install Kafka on Hadoop cluster and configure producer and consumer coding part in java to establish connection from twitter source to HDFS with popular hash tags.
- Perform ETL on data from Dev and QA box for staging it for micro batching to Spark
- Design a real time computation engine using Flume, Kafka, a Complex Event Processing engine to provide increase customer base
ENVIRONMENT: Horton works Data Platform, Hadoop 2.2, HDFS, Hive, Pig, Linux, Sqoop, Flume, oozie, Yarn, MR, HBase
Confidential, NY
ETL Hadoop Developer
RESPONSIBILITIES:
- Developed multiple MapReduce programs for analyzing the insurance data of the customer and produce summary results from Hadoop to downstream systems
- Worked on importing and exporting data from Oracle and DB2 into HDFS using Sqoop
- Developed data pipeline using Flume to ingest customer behavioral data and financial histories into HDFS for analysis
- Prepared the best practices in writing map reduce programs and hive scripts
- Scheduled a workflow to import the weekly transactions in the revenue department from RDBMS database using Oozie, Couchbase.
- Built wrapper shell scripts to hold these Oozie workflow
- Developed PIG Latin scripts to transform the log data files and load into HDFS
- Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS, Tera data
- Hands on experience with NoSQL databases like Cassandra for POC (proof of concept) in storing URL's and images
- Developed hive UDF for functions that were not preexisting in Hive like the rank etc.
- Created External Hive tables and involved in data loading and writing Hive UDFs
- Experienced in implementing POC's to migrate iterative map reduce programs into Spark transformations using Scala
- Created concurrent access for hive tables with shared and exclusive locking that can be enabled in hive with the help of Zookeeper implementation in the cluster
- Wrote the shell scripts to monitor the health check of Hadoop daemon services and respond accordingly to any warning or failure conditions
- Developed Unit test cases using MR unit for map reduce code
- Involved in creating Hadoop streaming jobs
- Involved in Installing, Configuring Hadoop Eco System, Cloudera Manager using CDH4 Distribution
- Used the most popular streaming tool Kafka to load the data on Hadoop File system and move the same data to MongoDB -NoSQL, Couchbase, database.
- Using Kafka we stream the data from source to Hadoop.
ENVIRONMENT: Hadoop, MapReduce, Hive, HDFS, PIG, Impala, Sqoop, Kafka, Spark, Oozie, lambda, Cloudera, Flume, HBase, ZooKeeper, CDH3, Cassandra, Oracle, NoSQL and Unix/Linux.
Confidential, MN
ETL Hadoop Developer
RESPONSIBILITIES:
- Provide technical designs, architecture, Support automation, installation and configuration tasks and upgrades and planning system upgrades of Hadoop cluster
- Design development and architecture of the Hadoop cluster, map reduce processes, Hbase system
- Design and develop process framework and support data migration in Hadoop system
- Experience in working with NoSQL databases like HBase and Cassandra
- Experience in the analysis, design, development and MR Unit testing of Hadoop Cluster Structureusing Java
- Developed Map reduce program using Java and deployed the Jobs at the customer environment.
- Designed the web-based system for business analytics and data visualization in Hadoop ecosystem integrated Tableau on Hadoop frame work to visualize and analyze data.
- Hive is used for data delivery
- Cluster Manager for Apache Hadoop that enables Bright Computing’s leading cluster management software to build, manage, and maintain big data clusters using Apache Hadoop
- Worked on several apaches Hadoop projects. Maps reduce programs were developed using Hadoop java API and also using hives and pig
- Designing and implementing solutions for Hadoop AFMS system validation for all components of data analytics stack
- Worked with scoop to import/export data from relational database to Hadoop and flume to collect data and populate in Hadoop
- Implemented and integrated Hadoop based business intelligence and Data Warehouse system including implementations of searching, filtering, indexing, aggregation for reporting and report generation and general information retrieval
- Maintained Hadoop clusters for dev/staging/production. Trained the development, administration, testing and analysis teams on Hadoop framework and Hadoop eco system
- Give extensive presentations about the Hadoop ecosystem, best practices, data architecture in Hadoop
- Data warehouse, Business Intelligence architecture design and develop. Designed the ETL process from various sources in to Hadoop/HDFS for analysis and further processing
- Provide review and feedback for existing physical architecture, data architecture, analysis, designs and code. Designed the next generation architecture for unstructured data
- Debug and solve issues as the subject matter expert focusing issues around data sciences and processing
- Define business and technical requirements, design Proof of Concept for evaluating afms agencies data evaluation criteria and scoring and select data integration and information management
- Collaborate with infrastructure and security architects to integrate enterprise information architecture into overall enterprise architecture
- Integrating Big data technologies and analysis tools into the overall architecture
ENVIRONMENT: Hadoop, Cassandra, Hbase, HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Oozie, Java, JSP, RMI, JNDI, JDBC, Tomcat, Apache, Shell Scripting.
Confidential
Java Developer
RESPONSIBILITIES:
- Used Microsoft Visio for designing the Use Case Diagrams, Class model, Sequence diagrams, and Activity diagrams for SDLC process of the application
- Implemented GUI pages by using JSP, JSTL, HTML, DHTML, XHTML, CSS, JavaScript, AJAX
- Configured the project on WebSphere 6.1 application servers
- Implemented the online application by using Core Java, Jdbc, JSP, Servlets and EJB 1.1, Web Services, SOAP, WSDL
- Communicated with other Health Care info by using Web Services with the help of SOAP, WSDL JAX-RPC
- Implemented Singleton, factory design pattern, DAO Design Patterns based on the application requirements
- Used SAX and DOM parsers to parse the raw XML documents
- Used RAD as Development IDE for web applications
- Used Clear Case for Version Control tool and Clear Quest for bug tracking tool
- Configured job scheduling in Linux using shell scripts and Crontab
- Developed test plan documents for all back end database modules
- Deployed the project in Linux environment
ENVIRONMENT: JDK 1.5, JSP, WebSphere, JDBC,EJB2.0, XML, DOM, SAX, XSLT, Apache Commons Digester, Apache Commons, CSS, HTML, JNDI, Web Services, WSDL, SOAP, RAD, SQL, PL/SQL, JavaScript, DHTML, XHTML, Oracle10g, Java Mail, PL/SQL Developer, Toad, POI Reports, Log4j, ANT, Clear Case, Windows XP, Red Hat Linux.
Confidential
Java Developer
RESPONSIBILITIES:
- Developed the application based on MVC (Model-View-Controller) Architecture using Spring Web MVC
- Used Microsoft Visio for designing the Use Case Diagrams, Class model, Sequence diagrams, and Activity diagrams for SDLC process of the application
- Implemented GUI pages by using JSP, JSTL, HTML, DHTML, XHTML, CSS, JavaScript, AJAX
- Extensively used Java multi-threading to implement batch Jobs with JDK 1.5 features
- Configured the project on Web Logic 10.3 application servers
- Implemented the online application using Core Java, JDBC, JSP, Servlets, spring, Hibernate, Web Services, SOAP, and WSDL
- Communicated with other Health Care info using Web Services with the help of SOAP, WSDL
- Tested the web services with SOAP UI tool
- Implemented Singleton, Factory Design Pattern, DAO Design Patterns based on the application requirements
- Used SAX and DOM parsers to parse the raw XML documents
- Used RAD as Development IDE for web applications
- Tuning complex database queries and joining the tables to improve the performance of the application
- Designed and developed base classes, framework classes and common re-usable components
- Used SVN for Version Control tool and Silk Central for bug tracking tool
ENVIRONMENT: JDK 1.5, JSP, web logic 10.3, JDBC, XML, DOM, SAX, CSS, HTML, JNDI, Web Services, WSDL, SOAP, Eclipse 3.0, SQL, PL/SQL, JavaScript, HTML, Oracle11g, PL/SQL Developer, Log4j, ANT, SVN, Silk Central.
