Sr. Hadoop Developer Resume
Concord, CA
SUMMARY
- 9+ years of experience as a software professional that starts from requirement gathering, analysis, design, implementation & testing software products using Java/ J2EE technologies and also in Big data technologies using Hadoop ecosystem.
- 4 years of experience in working with different Hadoop ecosystem components such as HDFS, MapReduce, HBase, Spark, Yarn, Kafka, Zookeeper, PIG, HIVE, Sqoop, Storm, Oozie, and Flume.
- Around 5 years of experience in Object Oriented Programming, using Java & J2EE (Servlets, JSP, JSF, Spring, Hibernate, Java Beans, EJB, JDBC, RMI, XML, JMS, Web Services, AJAX).
- Very good experience in large scale data processing by using Apache Spark processing components Spark SQL, Spark Streaming, MLlib using Scala/python as programming languages.
- Strong experience in Developing Simple to complex Map/reduce streaming jobs using Java language.
- Experience in analyzing data using Pig Latin, Hive QL and custom MapReduce programs in Java.
- Expertise in writing UDF’s in java/python to extend Pig and Hive core functionality.
- Expertise in optimizing traffic across network using different concepts like Combiners, Joins, Partitions, Bucketing.
- Hands on experience in Apache Storm for processing real time streaming data.
- Experience with Kafka in understanding and performing thousands of megabytes of reads and writes per second on streaming data.
- Experience in working with Apache Sqoop to load data from Relational Database systems to HDFS and vice - versa.
- Hands on experience in working with Apache Oozie to define Job workflows, scheduling and monitoring them.
- Experience with Apache Zookeeper to provide coordination services to the cluster.
- Experience with Apache Flume for Collecting and aggregating a large amount of Log Data from application servers and storing data in HDFS for further analysis.
- Experience in working with various Hadoop distributions like Cloudera, Hortonworks.
- In depth understanding of Hadoop Architecture and its components such as Name Node, Data Node, Application Master, Resource Manager and Map Reduce concepts.
- Worked in complete Software Development Life Cycle (analysis, design, development, testing, implementation and support) using Agile Methodologies.
- Good experience in working with different NOSQL Data bases such as HBase, Cassandra, MongoDB and also in RDBMS technologies like MySQL, Oracle 12c, SQL server, PostgreSQL.
- Proficient in programming languages like Java, Python, Scala.
- Experience in using Tableau to create statistics dashboards for Analytics.
- Experience in software component design technologies like UML Design, Use case and requirement Components diagrams.
- Good working experience in working different IDE's like Eclipse, Net Beans, IntelliJ.
- Very good knowledge on different concepts of Java like oops concepts, Design patterns, multithreading, serialization.
- Active team player with ability organize and prioritize multiple tasks.
- Excellent communication and inter-personal skills with technical competency and ability to quickly learn new technologies as required.
- Ability to blend technical expertise with strong Conceptual, Business and Analytical skills to provide quality solutions and result-oriented problem solving technique and leadership skills
TECHNICAL SKILLS
Hadoop/Big Data: MapReduce, HDFS, Hive, Pig, HBase, Sqoop, Spark, Spark SQL, RDD, Spark Streaming, GraphX, Kafka, Storm, Flume, Zookeeper, Oozie, Impala, Hue.
Distributed plat forms: Hortonworks, Cloudera.
Programming Languages: C, C#, Java, Scala, Python, R.
Java/J2EE Technologies: Servlets, JSP, JSF, JDBC, Java Beans, RMI & Web services (SOAP, RESTFul).
Frameworks: Struts, Hibernate and Spring MVC.
Development Tools: Eclipse, Net Beans, SBT, SVN, Git, ANT, Maven, Jenkins, Bamboo, SOAP UI, QC, Selenium WebDriver, Jira, Bugzilla, SQL Developer.
Methodologies: Agile/Scrum, UML, and Waterfall
NoSQL Technologies: Cassandra, MongoDB, Neo4j, HBase
Databases: Oracle 12c, MySQL, MS-SQL Server, SQL Server, PostgreSQL.
Web/ Application Servers: Apache Tomcat, WebLogic, WebSphere.
Visualization: Tableau and Qlikview.
Web Technologies: HTML, CSS, XML, JavaScript, jQuery, AngularJS, AJAX, SOAP, and REST.
Scripting Languages: Unix Shell Scripting, Perl.
Operating Systems: Windows XP/Vista/7/8,10, UNIX, LINUX.
PROFESSIONAL EXPERIENCE
Confidential, Concord, CA
Sr. Hadoop Developer
Responsibilities:
- Creating end to end Spark applications using Scala to perform various data cleansing, validation, transformation and summarization activities on user behavioral data.
- Created custom FTP adaptors to pull the clickstream data from FTP servers to HDFS directly using HDFS File System API.
- Used Spark SQL and dataframe API extensively to build spark applications.
- Worked closely with Admin and Devops team to help in setup high availability, capacity planning, authentication and authorization roles etc
- Created HBase tables to store variable data formats of data coming from different portfolios
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Used Sqoop job to import the data from RDBMS using Incremental Import.
- Customized Avro tools used in MapReduce, Pig and Hive for deserialization and to work with Avro ingestion framework
- Worked with using different kind of compression techniques to save data and optimize data transfer over network using Lzo, Snappy, etc.
- Analyze large and critical datasets using Hbase, MapReduce, Hive, Hive UDF, Pig, Sqoop, Zookeeper, & Spark.
- Customized Flume interceptors to encrypt and mask customer sensitive data as per requirement.
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
- Worked with NoSQL database Hbase to create tables and store data.
- Created end to end data processing pipelines to manage the entire life cycle of the User behavioral data in the new platform.
- Developed custom aggregate functions in Spark and performed interactive querying using Impala
- Creating Hive tables, dynamic partitions, buckets for sampling, and working on them using Hive QL
- Stored the data in tabular formats using Hive tables and Hive SerDe's.
- Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
- Implemented test scripts to support test driven development and continuous integration.
- Worked on tuning the performance Pig queries.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Involved in writing the shell scripts for exporting log files to Hadoop cluster through automated process.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team
- Implemented MapReduce programs to handle semi/ unstructured data like Parquet, XML, JSON, Avro data files and sequence files for log files.
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.
Environment: Hadoop, HDFS, Pig, Sqoop, Spark, MapReduce, Cloudera, Snappy, Zookeeper, NoSQL, HBase, Shell Scripting, Ubuntu, Linux Ubuntu.
Confidential, Dallas, Texas
Hadoop Developer
Responsibilities:
- Created various spark applications using Scala to perform various enrichment of these click stream data with enterprise data of the users
- Performed data ingestion using kafka to the spark distribution environment.
- Implemented batch processing of jobs using spark Scala API.
- Used spark engine Spark SQL for data analysis and given to the data scientists for further analysis.
- Also worked with data science team in buidling Spark MLlib applications to build various predictive models.
- Developed multiple Map Reduce jobs in Java for complex business requirements including data cleansing and preprocessing.
- Developed Sqoop scripts to import/export data from Oracle to HDFS and into Hive tables.
- Worked on analyzing Hadoop clusters using Big Data Analytic tools including Map Reduce, Pig and Hive.
- Involved in developing Pig scripts and to store unstructured data into HDFS.
- Involved in creating tables in Hive and writing scripts and queries to load data into Hive tables from HDFS.
- Scripted complex Hive QL queries on Hive tables for analytical functions.
- Optimized the Hive tables using optimization techniques like partitions and bucketing to provide better performance with Hive QL queries.
- Developed Java custom record reader, partition and serialization techniques.
- Used different data formats (Text format and Avro format) while loading the data into HDFS.
- Implemented MapReduce programs to handle semi/ unstructured data like Parquet, XML, JSON files and sequence files for log files.
- Loaded the final processed data to HBase tables to allow downstream application team to build rich and data driven applications.
- Involved in writing the shell scripts for exporting log files to Hadoop cluster through automated process.
- Created custom UDF’s in Pig and Hive.
- Created partitioned tables and loaded data using both static partition and dynamic partition methods.
- Used oozie for automating the end to end data pipelines and oozie coordinators for scheduling the work flows.
- Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
- Analyzed the Hadoop log files using Pig scripts to oversee the errors.
Environment: HDFS, Map Reduce, Hive, Spark, Kafka, Sqoop, Pig, HBase, Oozie, Hortonworks distribution, Java, Eclipse, Shell Scripts, Tableau, Oracle, Windows, Linux.
Confidential, St Louis, MO
Hadoop Developer
Responsibilities:
- Developed simple to complex Map Reduce streaming jobs using Java language for processing and validating the data.
- Worked with highlyunstructured and semi structured data of 90 TBin size with replication factor of 3.
- Developed data pipeline using Map Reduce, Flume, Sqoop and Pig to ingest customer behavioral data into HDFS for analysis.
- Extensive experience in writingPig scripts to transform raw datafrom several data sources into forming baseline data.
- Created Hive queries that helped market analysts spot emerging trends by comparing fresh data with EDW tables and historical metrics.
- Enabled speedy reviews and first-mover advantages by using Oozie to automate data loading into the Hadoop Distributed File System and PIG to pre-process the data.
- Handled importing data from different data sources into HDFS using Sqoop and also performing transformations using Hive, Map Reduce and then loading data into HDFS.
- Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.
- Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
- Solved performance issuesin Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
- Developed product profiles using Pig and commodity UDFs.
- Identify concurrent job workloads that may impact or be impacted by failures or bottlenecks.
- Developed some utility helper classes to get data from HBase tables.
- Scheduled and executed workflows in Oozie to run Hive and Pig jobs.
- Professional experience with NoSQL Hbase solutions to solve real world scaling problems.
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
- Attending daily status calls to follow scrum process to complete each user story within the timeline.
- Also be the part of triage call to handle defect reported by tester team or QA team.
- Viewing various aspect of a cluster using Cloudera Manager.
Environment: Hadoop, Linux, CDH, MapReduce, HDFS, Hive, Pig, Shell Scripting, Sqoop, Java 7, MySQL, NoSQL, Eclipse, Oracle 11g, Maven, Log4j, Git.
Confidential, Attleboro, MA
Sr. Java Developer
Responsibilities:
- Involved in various SDLC phases like Design, Development and Testing.
- Involved in developing JSP pages using Struts custom tags, jQuery and Tiles Framework.
- Developed web pages using HTML,JavaScript, JQuery and CSS.
- Used various CoreJavaconcepts such as Exception Handling, Collection APIs to implement various features and enhancements.
- Developed server side components servlets for the application.
- Involved in coding, maintaining, and administering Servlets and JSP components to be deployed on a Web Sphere application server.
- Used automated test scripts and tools to test the application in various phases. Coordinated with Quality Control teams to fix issues that were identified
- Implemented Hibernate ORM to Map relational data directly tojavaobjects
- Worked with Complex SQL queries, Functions and Stored Procedures.
- Involved in developing spring web MVC framework for portals application.
- Implemented the logging mechanism using log4j framework.
- Developed REST API, Web Services.
- Wrote test cases in JUnit for unit testing of classes.
- Used Maven to build the J2EE application.
- Used SVN to track and maintain the different version of the application.
- Involved in maintenance of different applications with onshore team.
Environment: Java, Spring Framework, Struts, Hibernate, RAD, SVN, Maven, Web Sphere Application Server, Web Services, Oracle Database 11g, IBM MQ, JMS, HTML, Java script, XML, CSS, REST API.
Confidential
Java / J2EE Developer
Responsibilities:
- Involved in design and development of a front-end application using JavaScript, CSS, HTML and JSPs andspring
- MVC for registering a new entry and management and configured it to connect to database using Hibernate.
- Developed and tested the Efficiency Management module using EJB, Servlets, and JSP & Core Java components in WebLogic Application Server.
- UsedSpring Framework as middle tier application framework, persistence strategy usingspring support for Hibernate for integrating with database.
- Implemented Hibernate in the data access object layer to access and update information in the Oracle Database.
- Configured the deployment descriptors in Hibernate to achieve object relational mapping.
- Involved in developing Stored Procedures, Queries and Functions.
- Write SQL queries to pull some information from the Backend.
- Compiling and running the applications.
- Writing the test plans and test cases for the developed screens.
- Executing test cases and fixing the bugs through unit testing.
Environment: Java, J2EE (Servlets, JDBC, EJB, JSP, JMS), HTML, CSS, Struts 1.2, Hibernate, Spring, XML, CVS, Eclipse, Oracle 8i, PL/SQL, Windows, UNIX
Confidential
Java / J2EE Developer
Responsibilities:
- Participated in requirement gathering and converting the requirements into technical specifications.
- Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components using Business Objects, XML, and JDBC.
- Designed user-interface and checking validations using JavaScript.
- Involved in design of JSP’s and Servlets for navigation among the modules.
- Developed various EJBs for handling business logic and data manipulations from database.
- Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
- Developed SQL queries and Stored Procedures using PL/SQL to retrieve and insert into multiple database schemas.
- Developed the XML Schema and Web services for the data maintenance and structures Wrote test cases in JUnit for unit testing of classes.
- Provided Technical support for production environments resolving the issues, analyzing the defects, providing and implementing the solution defects.
- Built and deployed Java applications into multiple UNIX based environments and produced both unit and functional test results along with release notes.
- Developed the presentation layer using CSS and HTML taken from bootstrap to develop for browsers.
Environment: Java, Spring, Jsp, Hibernate, XML, HTML, JavaScript, JDBC, CSS, SOAP Web services.
