We provide IT Staff Augmentation Services!

Sr. Big Data Hadoop Developer Resume

Sacramento, CA

SUMMARY:

  • I have 9+ years of experience with designing, developing, deploying & supporting large scale distributed system and over 4+ years of experience in Hadoop Eco system, and Big - Data Analytics. I Have great experience in agile and Scrum software development methodologies. experience in Application analysis, Design, Development, Maintenance and Supporting web, Client-server based applications in Java/J2EE technologies which include 2+ years of experience with Big Data and Hadoop related components like HDFS, Map Reduce, Pig, Hive, YARN, Sqoop, Flume, Crunch, Spark, Strom, Scala, and Kafka.
  • Experience in multiple Hadoop distributions like Cloudera, MapR, and Horton works.
  • Excellent understanding of NoSQL databases like HBase, Cassandra and MongoDB.
  • Experience on working structured, unstructured data with various file formats such as Avro data files, xml files, JSON files, sequence files using Map Reduce programs.
  • Work experience with cloud configurations like Amazon web services (AWS).
  • Implemented custom business logic and performed join optimization, secondary sorting, custom sorting using Map Reduce programs.
  • Experienced testing and running of Map Reduce pipelines on Apache Crunch.
  • Expertise in Data ingestion using SQOOP, Apache Kafka, Spark Streaming and FLUME
  • Implemented business logic using Pig scripts. Wrote custom Pig UDF’s to analyze data
  • Performed different ETL operations using Pig for joining operations and transformations on data to join, clean, aggregate and analyze data.
  • Experience with Oozie Workflow Engine to automate and parallelize Hadoop, Map Reduce and Pig jobs
  • Extensive experience with wiring SQL queries using HiveQL to perform analytics on structured data.
  • Experience in performing data validation using HIVE dynamic partitioning and bucketing.
  • Experienced in importing and exporting data between RDBMS and Teradata intoHDFS using Sqoop.
  • Experienced in handling streaming data like web server log data using flume.
  • Worked on Cassandra database and related web services for storing unstructured data.
  • Good knowledge analyzing data usingPython development and scripting for Hadoop Streaming.
  • Experience in implementing algorithms for analyzing using spark.
  • Experience in implementing Spark using Scalaand SparkSQL for fasterprocessing of data.
  • Experience in getting data from various sources into HDFS and building reports using Tableau.
  • Experience in creating tables on top of data on AWSS3 obtained from different data sources and providing them to analytics team building reports using Tableau.
  • Proficient in managing complex Bigdata environments of Cloudera / Hortonworks Hadoop.
  • Extensive Hands on experience with Accessing and perform CURD operations against HBase data using Java APIand implementing time series data management.
  • Involved in various datamining tasks such as pattern mining, classification and clustering techniques.
  • Expert knowledge over J2EE Design Patterns like MVC Architecture, Singleton, Factory Pattern, Front Controller, Session Facade, Business Delegate and Data Access Object for building J2EE Applications.
  • Experienced in J2EE, Spring, Hibernate, SOAP/Rest web services, JMS, JNDI, EJB, JAX-WS .
  • Expertise with Application servers and web servers likeOracle WebLogic, IBM WebSphere, Apache Tomcat, JBOSS and VMware .
  • Proven expertise in implementing IOC/Dependency Injection features in various aspects of Spring Framework .
  • Experienced in developing the unit test cases usingMRUnit, JUnit and Easy Mock.
  • KnowledgeonSplunkfor logging mechanism.
  • Knowledge on Build tool Jenkins.
  • Experience in using Mavenand ANTfor build automation.
  • Experience in using version control and configuration management tools like SVN, CVS.
  • Experience working in environments using Agile (SCRUM) and Waterfall methodologies.
  • Experience in designing applications using UML Diagrams like Class Diagram, Component Diagram, Sequence Diagrams, and Deployment Diagram using MS Visio, Rational Rose.
  • Expertise in database modeling, administration and development using SQLand PL/SQL in Oracle (8i, 9i and 10g), MySQL, Teradata, DB2and SQL Server environments.

TECHNICAL SKILLS:

Hadoop/Big Data: HDFS, Map Reduce, Hive, Pig, YARN, Sqoop, Flume, Oozie, Crunch, Strom, Scala, Kafka, Spark, AWS, RHadoop

Database/No Sql: SQL.Pl/SQL,HBase,Cassandra

J2EE Frameworks: Hibernate, Spring, JMS, EJB, JSF

XML/Web Services: SOAP/ Rest

Methodologies: Agile, Waterfall

Build Tools: Maven, ANT, Log4j.

Scripting languages: JavaScript, HTML, HTML5, XML, Python, JSP and Servlets

Tools: Rational Rose, Microsoft Visio, CSS

Operating Systems: Linux/Unix,WINDOWS

PROFESSIONAL EXPERIENCE:

Confidential, Sacramento, CA

Sr. Big Data Hadoop Developer

Responsibilities:

  • Loading data from diff servers to AWS S3 bucket and setting appropriate bucket permissions.
  • Apache Kafka to transform live streaming with the batch processing to generate reports
  • Cassandra data modeling for storing and transformation in spark using datastax connector.
  • Imported data into HDFS and Hive using Sqoop and Kafka. Created Kafka topics and distributed to different consumer applications.
  • Worked on Spark SQL and Data frames for faster execution of Hive queries using Spark and AWS EMR
  • Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for increasing performance benefit and helping in organizing data in a logical fashion.
  • Experienced with Spark Context, Spark -SQL, Data Frame and Pair RDD's.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.
  • Developed Spark code using scala and Spark-SQL/Streaming for faster testing and processing of data.
  • Worked on a POC for streaming data using Kafka and Spark streaming.
  • Used Hive, spark SQL Connection to generate Tableau BI reports.
  • Imported data using Sqoop to load data from MySQL to HDFS on regular basis.
  • Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
  • Created Hive Generic UDF's to process business logic that varies based on policy.
  • Developed various data connections from data source to SSIS, Tableau Server for report and dashboard development
  • Worked with clients to better understand their reporting and dash boarding needs and present solutions using structured Waterfall and Agile project methodology approach
  • Developed metrics, attributes, filters, reports, dashboards and also created advanced chart types, visualizations and complex calculations to manipulate the data.

Environment: CDH4, Java, MapReduce, HDFS, Hive, Spark, Scala, Pig, Linux, XML, MySQL, MySQL Workbench, Cloudera, Maven, Java 6, Eclipse, PL/SQL, SQL connector, Sub Version.

Confidential, Lewisville, TX

Hadoop Developer

Responsibilities:

  • Worked with the various business teams to gather business requirements, designed prototype of the system and presented to the business users/managers.
  • Prepare user guide and training materials technical training and user support to business groups.
  • Understanding of rules, Properties, Embedded Pages and Correspondences.
  • Created agents which will run in the background to generate daily, weekly and monthly reports.
  • Followed the UML methodology to Create UML Diagrams including Use Cases Diagrams, Activity Diagrams, Sequence Diagrams, Collaboration Diagrams, Data Flow Diagrams (DFD's) and MS Visio and thus defining the Business Process Model and Data Process Model.
  • Created High Level Class structure, Workflows and Rule sets in PEGA PRPC.
  • Mentor and train client business stakeholders in key PRPC Functionality (Work list, Workbasket, Get Next Work, Access Group, Work Group, Portal)
  • Responsible for gathering, analysing, and documenting business requirements; writing use cases (UML), functional and non-functional specifications based on information gathering sessions with business unit stakeholders and technical team members.

Environment: CDH4, Java, MapReduce, HDFS, Hive, Spark, Scala, Pig, Linux, XML, MySQL, MySQL Workbench, Cloudera, Maven, Java 6, Eclipse, PL/SQL, SQL connector, Sub Version.

Confidential, Long beach, CA

Software Engineer

Responsibilities:

  • Developed front-end screens using JSP, HTML and CSS.
  • Developed server side code using Struts and Servlets.
  • Developed core java classes for exceptions, utility classes, business delegate, and test cases.
  • Developed SQL queries using MySQL and established connectivity.
  • Worked with Eclipse using Maven plugin for Eclipse IDE.
  • Designed the user interface of the application using HTML5, CSS3, Java Server Faces 2.0 (JSF 2.0), JSP and JavaScript.
  • Tested the application functionality with JUnit Test Cases.
  • Developed all the User Interfaces using JSP and Struts framework.
  • Writing Client Side validations using JavaScript.
  • Extensively used JQuery for developing interactive web pages.
  • Developed the DAO layer using hibernate and for real time performance used the caching system for hibernate.
  • Experience in developing web services for production systems using SOAP and WSDL.
  • Developed the user interface presentation screens using HTML, XML, and CSS.
  • Experience in working with spring using AOP, IOC and JDBC template.
  • Developed the Shell scripts to trigger the Java Batch job, Sending summary email for the batch job status and processing summary.
  • Co-ordinate with the QA lead for development of test plan, test cases, test code and actual testing responsible for defects allocation and those defects are resolved.
  • The application was developed in Eclipse IDE and was deployed on Tomcat server.
  • Involved in scrum methodology.
  • Supported for bug fixes and functionality change.

Environment: Java Struts 1.1, Servlets, JSP, HTML, CSS, JavaScript, Eclipse 3.2, Tomcat, Maven 2.x, MySQL, Windows and Linux, JUnit.

Confidential

Java developer

Responsibilities:

  • Developed data pipeline using Flume, Sqoop, Pig and Java map reduce to ingest customer behavioral data and financial histories into HDFS for analysis.
  • Involved in writing MapReduce jobs.
  • Experienced in Pig to do transformations, event joins, filter bot traffic and some pre-aggregations before storing the data onto HDFS.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Good experience in developing Hive DDLs to create, alter and drop Hive TABLES.
  • Involved in developing Hive UDFs for the needed functionality that is not out of the box available from Apache Hive.
  • Using HCATALOG to access Hive table metadata from Map Reduce or Pig code.
  • Computed various metrics using Java MapReduce to calculate metrics that define user experience, revenue etc.
  • Responsible for developing data pipeline using flume, sqoop and pig to extract the data from weblogs and store in HDFS Designed and implemented various metrics that can statistically signify the success of the experiment.
  • Worked on AWS to create EC2 instance and installed Java, Zookeeper and Kafka on those instances.
  • Worked on S3 buckets on AWS to store Cloud Formation Templates
  • Involved in using SQOOP for importing and exporting data into HDFS and Hive.
  • Responsible for processing ingested raw data using MapReduce, Apache Pig and Hive.
  • Developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
  • Involved in pivot the HDFS data from Rows to Columns and Columns to Rows.
  • Involved in emitting processed data from Hadoop to relational databases or external file systems using SQOOP, HDFS GET or Copy To Local.
  • Discussed the implementation level of concurring programing in spark using python with message passing.
  • Involved in discussing spark-SQL and spark MLib.

Environment: Java1.4, J2EE 1.4, JSP 2.0, Servlets2.4, XML, XSLT, Hibernate, Spring Framework, VSS Version controller, Ms Sql 5.0, UML, WebSphere portal server 5.0, Rational Application Developer 6.0

Confidential

Java developer

Responsibilities:

  • Perform detailed analysis of project requirements and design application architecture.
  • Design Business validation and implementing them in the Business Objects.
  • Perform reviews of modeling diagrams and source code.
  • Conduct code walk through sessions with the java development team.
  • Since the data that need to be passed between different tiers is huge, used Transfer.
  • Used Data Access Objects (DAO) to abstract and encapsulate all access to the data source.
  • Used Persistence - MYSQL service to call the custom selects and updates to the database.
  • Used Servlets, JSPs, and MVC framework.
  • Developing DB mapping XML files in Hibernate.
  • Designing the User interface using JSF, HTML and Java Script.
  • Used Rational Application Developer for Application Development and SVN for version control.
  • Perform Integration, Unit and Regression testing on the application components.
  • Research & Development activities to solve the technical challenges faced during the development phase of the project.

Environment: Java1.4, J2EE 1.4, JSP 2.0, Servlets2.4, XML, XSLT, Hibernate, Spring Framework, VSS Version controller, Ms Sql 5.0, UML, WebSphere portal server 5.0, Rational Application Developer 6.0

Hire Now