Hadoop & Spark Consultant Resume
Sioux Falls, SD
SUMMARY
- 8 years of experienced specializing in Banking and Financial Services institutions with web based applications.
- Enjoys building relationships with stakeholders, understanding business issues, gathering and analyzing requirements, inventing creative solutions, documenting functional and technical design specifications, and working with Development, Testing, Production Support and Quality Assurance teams.
- 8 years of overall experience in IT industry, which includes hands on experience in Big data eco - system related technologies like Map Reduce, Hive, HBase, Cassandra, Pig, SQOOP, Kafka, Scala, Spark, OOZIE, flume, Zookeeper and HDFS.
- 3 years of comprehensive experience as a Big Data & Analytics (Hadoop).
- Very Strong Object-oriented concepts with complete software development life cycle experience - Requirements gathering, Conceptual Design, Analysis, Detail design, Development, System and User Acceptance Testing.
- Experience in working with MapReduce programs using Apache Hadoop for working with Big Data.
- Hands on experience in big data ingestion tools like Flume and SQOOP.
- Strong noledge of Pig and Hive's functions, extending Hive and Pig core functionality by writing UDFs.
- Proficiency in different Hadoop distributions Cloudera and Hortonworks Distributions.
- Experience in analyzing data using HiveQL, Pig Latin, Impala and custom Map Reduce programs in Java.
- Hands on NoSQL database experience with Apache HBase & Mongo DB.
- Knowledge in job workflow scheduling and coordinating tools like Oozie and Zookeeper.
- Experience with various scripting languages like Linux/Unix shell scripts, Python 2.7 and Scala.
- Good experience Knowledge on Apache Spark & Scala.
- Good experience on EMS, RV, JDK 5, JDBC, Servlets, JSP, Struts 2.1, Spring 2, AJAX, DOJO, JSON Hibernate 3, XML Processing (JAXB), JMS, and TIBCO
- Strong front-end UI development skills using scripting languages like JSP, HTML, JavaScript, JQuery and CSS.
- Good experience on application server WebSphere 6.0 and JBoss 4.x.
- Experience to build tool like Ant.
- Good noledge on Talend Integration for Big data Open Studio 5.6.
- Good Knowledge on life cycle of ETL (Extraction, Transformation and Loading) using Informatica Power Center (Repository Manager, Mapping Designer, Workflow Manager, Workflow monitor).
- Good database experience using SQL Server, Stored Procedures, Cursors, Constraints and Triggers.
- Highly motivated, adaptive and quick learner.
- Domain Knowledge: Credit Cards domain experience with expertise in Non-Monetary & Monetary process, Payments, Fees, Collections, Statements functionalities, Banking, SMS Alerts & Email gateway solutions, IVR, Cards Origination, GL Accounting and middleware integration.
- Strong Analytical and Problem Solving skills, Multi-Tasking abilities, with proven experience in utilizing people and process noledge to assist enterprises in making critical decision.
TECHNICAL SKILLS
Hadoop/Big Data Technologies: HDFS, Map Reduce Hive, Pig, Impala, SQOOP, Flume, OOZIE, Spark, SparkQL and Zookeeper.
Languages: Core Java, XML,HTML and HiveQL
J2EE Technologies: Servlets, JSP, JMS, JSTL, AJAX, DOJO, JSON and Blaze DS.
Frameworks: Spring 2, Struts 2 and Hibernate 3
XML Processing: JAXB
Reporting Tools: BIRT 2.2.
Application & Web Services: WebSphere 6.0, JBoss 4.X and Tomcat 5.
Scripting Languages: Java Script, Angular JS, Pig Latin, Python 2.7 and Scala.
Database(SQL/No SQL): Oracle 9i, SQL Server 2005,MySQL, HBase, Cassandra and Mongo DB 2.2
IDE: Eclipse and Edit plus
Tools: SVN, Apache Ant and Junit
ETL Tool: Talend Open Studio 5.6
Bug tracking/ Ticketing: Mercury Quality Center and Service Now
Operating System: Windows 98/2000 and Linux /Unix
PROFESSIONAL EXPERIENCE
Confidential, Sioux Falls, SD
Hadoop & Spark Consultant
Responsibilities:
- Worked on Distributed/Cloud Computing (Map Reduce/Hadoop, Hive, Pig, HBase, SQOOP, Flume, Spark, AVRO, Zookeeper etc.), Cloudera distributed Hadoop (CDH5.X).
- Handling raw data from various modules and Load teh data from different subsystems into to HDFS for further processing.
- Developed Apache Pig scripts and UDF’s extensively for data transformations and calculating Statement date formats and aggregates teh monitory transactions.
- Created Hive tables to store teh processed results in a tabular format.
- Developed teh SQOOP scripts in order to make teh interaction between Pig and DB2 Database.
- Involved in gathering teh requirements, designing, development and testing.
- Writing teh script files for processing data and loading to HDFS.
- Used SVC tool extensively for building jar files of MapReduce programs and deployed to Cluster.
- Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive querying.
- Implemented performance-tuning techniques along various stages of teh Migration process.
- Developed Hive Scripts for implementing dynamic Partitions.
- Experienced in handling different optimization join operations like Map join, Sorted Bucketed Map join etc.
- Expertise in performance tuning on Hive Queries, joins and different configuration parameters to improve query response time.
- Completely involved in teh requirement analysis phase.
- Import teh data from different sources like HDFS into Spark RDD.
- Developed Spark code using scala and Spark-SQL/Streaming for faster testing and processing of data.
- Load teh data into Spark RDD and do in memory data Computation to generate teh Output response
- Migrated Hive QL queries on structured into Spark QL to improve performance
- Implemented POC writing programs in Scala and data processed using SparkQL.
- Experienced in implementing Spark RDD transformations, actions to implement business analysis
- Conducted POC for Hadoop and Spark as part of NextGen platform implementation. Implemented recommendation engine using Scala.
- Experienced in implementing Spark RDD transformations, actions to implement business analysis.
- Migrated Hive QL queries on structured into Spark QL to improve performance.
- Responsible for building scalable distributed data solutions using Hadoop.
- Monitor System health and logs and respond accordingly to any warning or failure conditions.
- Configured Hive QL with data base MySQL used as a remote Metastore .
- Exported analyzed data to relational databases using SQOOP for visualization to generate reports for teh BI team.
- Implemented POC for using APACHE IMPALA for data processing on top of HIVE.
- Streamlined Hadoop jobs and workflow operations using Oozie workflow.
- Experienced in writing unit test cases, implement unit test cases using JUnit.
- Worked with testing teams and resolved defects.
Environment: Hadoop (CDH5.4), UNIX, Scala, Spark, SparkQL, Map Reduce, Apache Pig, Hive, Impala, SQOOP, Java, Eclipse, Python 2.7, DB2, MySQL and OOZIE.
Confidential, Charlotte, NC
Hadoop Developer
Responsibilities:
- Worked on Hortonworks (HDP 2.X) platform to perform Hadoop operations.
- Installed and configured Hive, Pig, SQOOP and flume on teh Hadoop cluster.
- Developed Simple to complex Mapreduce Jobs using Java, Hive and Pig.
- Teh logs and semi structured content dat are stored on HDFS were preprocessed using PIG and teh processed data is imported into Hive warehouse which enabled business analysts to write Hive queries.
- Importing and exporting data into HDFS using SQOOP and flume.
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, and loaded data into HDFS.
- Continuous monitoring and managing teh Hadoop cluster through Cloudera Manager.
- Installed and configured Hadoop MapReduce, HDFS, Developed multiple Map Reduce jobs in java for data cleansing and processing.
- Injected teh data from logs and relational databases using flume and SQOOP.
- Involved in gathering teh requirements, designing, development and testing.
- Worked on loading and transformation of large sets of structured, semi structured data into Hadoop system.
- Worked on Hue interface for querying teh data.
- Developed Pig scripts for data analysis and extended its functionality by developing custom UDF's.
- Extensive noledge on PIG scripts using bags and tuples.
- Experience in managing and reviewing Hadoop log files.
- Developed workflow in middleware to automate teh tasks of loading teh data into HDFS and pre-processing with Pig.
- Exported analyzed data to relational databases using SQOOP for visualization to generate reports for teh BI team.
Environment: Hadoop (HDP 2.X), UNIX, Eclipse, TIBCO, HDFS, Java, MapReduce, Apache Pig, Hive, flume, Python 2.7, SQOOP and MySQL.
Confidential, CA
Sr. Java Developer
Responsibilities:
- As part of teh lifecycle development prepared class model, sequence model and flow diagrams by analyzing Use cases using Rational Tools.
- Reviewing and analyzing data model for developing teh Presentation layer and Value Objects.
- Implemented log4j in teh portal application.
- Responsible for identifying and resolving project risks, issues and problems.
- Implemented teh front end UI for teh above component using JSP and Struts Framework
- Implemented module called PMS interface which enforces different property management system protocol to java objects to parallax system
- Had done extensive research on heterogeneous IP phones platforms.
- Setup and provided packaging teh product ion Linux environment.
- Played a crucial in developing teh business logic as per client requirements.
- Understanding functional specifications, designing, and creating custom solutions to match teh requirements.
- Designed and developed teh front end using HTML, CSS, JavaScript with DWR Manager Ajax and tag libraries.
- Handled events and runtime errors using event listeners and validations.
- Used Tomcat as application server and Jetty as Servlet containers
- Used Site Mesh for menu and page header decorations.
- Designed and developed a 'Convention Based Coding' utilizing Hibernate and JPA's persistence framework and O-R mapping capability to enable dynamic fetching and displaying of various table data with JSF tag libraries.
- Performance enhancement processes after every implementation cycle. This helped teh product to pass its bench mark of supporting 1500 guest rooms
- Designed and developed teh Validators, Controller Classes and Java bean components
- Designed dynamic and browser compatible pages using HTML5, CSS3, JQuery and JavaScript and Angular JS.
- Integrate teh dynamic pages with Angular JS and jquery to make teh pages dynamic.
- Participated in teh design and development of database schema and Entity-Relationship diagrams of teh backend Oracle database tables for teh application.
- Used Maven for teh build process.
- Used JUnit for unit testing.
- Involved in QA and UAT support.
- As per teh Iteration methodology teh Item/task was delivered.
Environment: JDK 1.5, J2EE, JSP, Struts, JavaScript, Ajax, XML, jUnit, Sybase, SQL Server 2005, WebSphere Application Server v6.0.1.
Confidential
Sr. Java Developer
Responsibilities:
- Involved in design and development phases of Software Development Life Cycle (SDLC)
- Involved in designing UML Use case diagrams, Class diagrams, and Sequence diagrams using Rational Rose.
- Followed agile methodology and SCRUM meetings to track, optimize and tailored features to customer needs.
- Developed user interface using JSP, JSP Tag libraries, and Java Script to simplify teh complexities of teh application.
- Implemented Model View Controller (MVC) architecture using Jakarta Struts frameworks at presentation tier.
- Developed a Dojo based front end including forms and controls and programmed event handling.
- Implemented SOA architecture with web services using JAX-RS (REST) and JAX-WS (SOAP)
- Developed various Enterprise Java Bean components to fulfill teh business functionality.
- Created Action Classes which route submittals to appropriate EJB components and render retrieved information.
- Validated all forms using Struts validation framework and implemented Tiles framework in teh presentation layer.
- Used Core java and object oriented concepts.
- Extensively used Hibernate in data access layer to access and update information in teh database.
- Used Spring Framework for Dependency injection and integrated it with teh Struts Framework and Hibernate.
- Used JDBC to connect to backend databases, Oracle and SQL Server 2005.
- Proficient in writing SQL queries, stored procedures for multiple databases, Oracle and SQL Server 2005.
- Wrote Stored Procedures using PL/SQL. Performed query optimization to achieve faster indexing and making teh system more scalable.
- Deployed application on windows using IBM Web Sphere Application Server.
- Used Java Messaging Services (JMS) for reliable and asynchronous exchange of important information such as payment status report.
- Used Web Services - WSDL and REST for getting credit card information from third party and used SAX and DOM XML parsers for data retrieval.
- Implemented SOA architecture with web services using Web Services like JAX-WS.
- Used ANT scripts to build teh application and deployed on Web Sphere Application Server
Environment: Core Java, J2EE, Oracle, SQL Server, JSP, Struts, Spring, JDK, Hibernate, JavaScript, HTML, CSS, AJAX, Junit, Log4j, Web Services, Windows.
Confidential
Java Developer
Responsibilities:
- Involved in design and development phases of Software Development Life Cycle (SDLC)
- Developed all parts of modules Artifacts, Reporting, Costing and Admin Modules.
- Designing and developing of teh User Interface for teh application using JSP and DOJO (for validation and AJAX requests).
- Developed teh runtime for teh business reports using BIRT runtime tool.
- Followed agile methodology and SCRUM meetings to track, optimize and tailored features to customer needs.
- Developed teh middle layer development and using spring framework
- Worked on developing teh ORM layer using Spring framework’s ORM module involved in making compatible for WAS
- Prepared teh document for unit test scripts and testing results.
- Worked on defect resolutions raised at QA and Production level issues.
Environment: Core Java, Spring, Log4j, JSP, CSS, JDBC, Oracle, SQL Server 2005, DB2, JMS, JAXB, DOJO, JBoss, Web sphere, Eclipse, XML, ANT and CVS.
Confidential
Java Developer
Responsibilities:
- Developed JavaScript behavior code for user interaction.
- Created database program in SQL server to manipulate data accumulated by internet transactions.
- Wrote Servlets class to generate dynamic HTML pages.
- Developed Servlets and back-end Java classes using Web Sphere application server.
- Developed an API to write XML documents from a database.
- Performed usability testing for teh application using JUnit Test.
- Maintenance of a Java GUI application using JFC/Swing.
- Created complex SQL and used JDBC connectivity to access teh database.
- Involved in teh design and coding of teh data capture templates, presentation and component templates.
- Part of teh team dat designed, customized and implemented metadata search and database synchronization.
- Used Oracle as Database and used Toad for queries execution and also Involved in writing SQL scripts, PL SQL code for procedures and functions
Environment: Java, Web Sphere 3.5, EJB, Servlets, JavaScript, JDBC, SQL, JUnit, Eclipse IDE and Apache Tomcat 6