Senior Hadoop Consultant Resume
Dublin, OH
SUMMARY:
- 11 years of Information Technology experience in application analysis, design, development and testing of web based and n - tier applications with 3+ years of Big Data Ecosystems experience in ingesting, storage, querying, processing and analysis of Big Data.
- Strong foundational knowledge and experience with Hadoop architecture and various components such as HDFS, YARN, Pig, Hive, Sqoop, Oozie, Flume, Spark, Storm, Solr, Kafka, Impala, HBase, HCatalog and MapReduce programming paradigm.
- Experience in importing and exporting data using Sqoop from relational databases to HDFS and vice-versa.
- Experience in analyzing data using SparkSQL, HIVEQL, PIG Latin, and custom MapReduce programs in JAVA and experience in developing custom UDFs using Pig and Hive
- Experience in ingesting large volumes data from various application servers/sources into HDFS, HIVE, HBase and S3.
- Experience in designing both time-driven and data-driven work flows with Oozie
- Experience in NoSQL databases like MongoDB, HBase and Cassandra
- Experience in AWS services like EC2 instances, S3 storage and RDS services.
- 8 years of extensive architecture/application development experience mainly utilizing Java and J2EE Technologies using GWT, JSP, Servlets, JDBC, Struts, Struts Validator framework, Tiles, Spring Framework and Hibernate.
- 1+ years of experience in developing web applications, windows services and web services in Microsoft Visual Studio .Net IDE using C#, VB, ASP.Net, ADO.Net,, AJAX, WCF, WPF, MVC
- Excellent grasp in Struts, Spring, JSF, Hibernate, EJB, JSP, Servlets, Axis, JDBC, and JMS.
- Good working experience in Unix commands, Shell scripting and Perl Scripting
- Experience with service oriented architecture (SOA), workflows and web service using XML, SOAP, and WSDL, HTML, AJAX, and Java script.
- Worked on various architectures like MVVM, MVC4 and Service Oriented Architecture (SOA).
- Good experience with RDBMS like Oracle, MSSQL and Teradata have extensively used DB tools like TOAD and SQL Developer.
- Good working experiences with Hibernate open source object/relational mapping framework.
- Experience in oracle business intelligence tool (OBIEE), JASPER for reports and knowledge in Tableau
- Experience in version control repositories like Git, SVN, Clearcase, Visual Source Safe and Team Foundation Server
- Experience of agile software development process and development best practices
- Ability to switch easily between different projects and flexibility to handle rapidly changing environment.
- Good understanding of high performance, scalable and distributed systems.
- Excellent communication skills with strong analytical problem-solving ability.
TECHNICAL SKILLS:
Big Data/Hadoop: HDFS, MapReduce, Spark, Hive, Pig, HBase, Sqoop, Flume, OozieZooKeeper, Spark, Tez, Apache Solr, Storm, Kafka, Impala
Languages: Java 1.4/1.5/1.6/1.7/8, C/C++, C#, VB, PL/SQL, Python, Groovy
Technologies: Java, J2EE, Servlets, JSP, EJB, JDBC, JMS, JNDI, AJAXVB.Net, C#.Net, ADO.Net, ASP.NET, WCF, LINQ, REST
Application Servers: Web Logic 8.0, JBoss, Tomcat 4.0/5.0/6.0,IIS6, IIS7
Frame Works: Struts, spring, Hibernate, MVC, JUnit, and Log4J
Databases: Teradata, Oracle8i/9i/11g, MySQL, SQL server, MS Access
Dev Tools: Eclipse, VS, SQL Developer, NetBeans, JBuilder and RAD6.0
Web Dev. Technologies: HTML, XML, XSL, XSLT, SAX/DOM, SOAP, WSDL
Scripting Languages: JavaScript, Perl, JSON, Shell scripting, Python, Node.js
Operating Systems: Windows 95/98/2000/XP/7, Windows Server 2003/2008 and Linux
Cloud Platforms: Microsoft Azure, AWS
NoSQL Databases: HBase, MongoDB, Cassandra
Others: Ant, Maven, Putty, WinSCP, Git
PROFESSIONAL EXPERIENCE:
Confidential, Dublin, OH
Senior Hadoop Consultant
Responsibilities:
- Understanding the existing framework for bibliographic records processing and suggesting the automated process using spark framework.
- Working with customers directly and proposing efficient big data solutions for their complex needs.
- Coordinating with cross-functional Agile team to create and enhance the software modules
- Implemented automated batch processing of correcting records using spark jobs.
- Enhancing the existing MapReduce programs for ingesting data
- Improvised the logging, activities and collection of metrics from various jobs and storing them HBase and sending them to Elastic Search.
- Contributed to technical architecture design, application design and development
- Generating various Json files with standardized format that will used by Elastic Search and Kibana
- Tuning the various spark job attributes to achieve the desirable results
- Utilizing development frameworks such as continuous integration to enable fast delivery of working code
- Updating existing shell scripts for invoking jobs through ControlM.
- Writing unit test cases with MRUnit and LocalMR jobs in Java and Groovy
- Debugging and troubleshooting the existing complex batch jobs when they fail.
Environment: CDH 5.4/5.5, Spark 1.5, MapReduce, HBase, Hive, ControlM, Java 1.8, Spring Batch, Groovy, MRUnit, SparkSql, Eclipse, IntelliJ, UNIX, JSON, Web Services, SVN, Git, JIRA, Maven, Gradle, ELK
Confidential, Phoenix, AZ
Senior Hadoop Consultant
Responsibilities:
- Involved in all phases of the project and interacting with various business units to understand requirements for project
- Implemented java based Quartz scheduler for ingesting data from various sources like relational databases, FTP locations and Event based messages
- Contributed to technical architecture design, platform selection, application design and development
- Creating the parquet file with standardized format using the schema.
- Ingesting large amounts of data and converting them to parquet format and storing them in S3 using spark data frames.
- Tuning the various spark job attributes to achieve the desirable results
- Providing the access to spark data frames from Tableau and Hive.
- Frequently using AWS to access S3 for Data Storage, EC2 instances and RDS service for meta store.
- Utilizing AWS SDK for java to access S3.
- Writing the technical design specs and creating data model diagrams and control flow diagrams.
- Writing the run books for deployment and maintenance teams hand off.
Environment: CDH 5.4/5.5, AWS, Spark 1.5, Hive, Oozie, Java 1.8, SparkSql, DataFrames, Eclipse, MySql, SQLServer, Oracle, JSON, Web Services, Git, Rally, Maven, ELK, Jenkins.
Confidential, Hoffman Estates, IL
Senior Hadoop Developer
Responsibilities:
- Decides technologies for integrating data discovery, collection, processing and applications.
- Developing Hadoop applications to process huge amounts of transactional data for the analysis
- Importing and exporting large volumes of data into HDFS and Hive from Teradata using Teradata ETL process.
- Developed MapReduce jobs in Java for data cleaning and preprocessing.
- Worked on data transformation and data processing using PigLatin and MapReduce.
- Reading data from Flume sink and process them using MapReduce Programs
- Provides detailed task plans to implement the projects
- Performs analysis into site incidents and presents key findings in Root Cause Analysis meetings
- Created tables/views in Hive, written custom UDFs in Java for Pig and Hive
- Used Kafka as messaging system to call third party web services and load the data into HDFS after processing with Pig scripts.
- Using Apache Solr for full text search and indexing the generated data collections.
- Written unit test cases to test various modules developed.
Environment: Hadoop 2, CDH 5.2, HDFS, Hive, Pig, MapReduce, Impala, Control M, Java 1.7, Eclipse, Flume, Kafka, Spark, Solr, Teradata, JSON, SQL, ETL, Web Services, LINQ, GitHub, Ice Scrum, Maven
Confidential, Raleigh, NC
Senior Hadoop Developer
Responsibilities:
- Identifying the core study data needed and define the required the canonical data models
- Extracting data from various sources and importing data into HDFS and Hive using Sqoop.
- Created tables/views in Hive, written custom UDFs and Hive based exception processing.
- Worked on implementing migrating SQL to HQL (Hive Query Language)
- Used Oozie to define various work flow scenarios and scheduling
- Experience in writing and deploying ETL packages with SSIS
- Mentoring and providing guidance to the team
- Prepared Technical Design Documents for the new development
- Identifying and Troubleshooting production issues quickly
Environment: Hadoop 2, HDFS, Hive, Oozie, HDP, Java 1.7, Eclipse, Shell Script, SQL server, Oracle, JSON, SQL, ETL, Web Services, SAS VA
Confidential, Phoenix, AZ
Hadoop Developer
Responsibilities:
- Involved in reviewing and developing specifications, develop and test various application components to meet the requirements of application development.
- Used Scrum methodology for development and providing tasks and time estimates for product backlog items.
- Importing and exporting data into HDFS and Hive using Sqoop.
- Developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
- Worked on data transformation and file processing using PigLatin
- Created tables/views in Hive, written custom UDFs and Hive based exception processing.
- Worked on implementing migrating SQL to HQL (Hive Query Language)
- Used Flume to collect, aggregate and store the log data from web servers and pushed to HDFS
- Used Oozie to define various work flow scenarios and scheduling
- Exposure to Ambari to monitor and install services on Hadoop cluster
- Experience in writing and deploying ETL packages with SSIS
- Prepared Technical Design Documents for the new development
- Extensively used various Java Script plug-ins for smooth navigation throughout the applications.
- Good at understanding and using various third party java script libraries.
- Designed and generated reports using java script based high charts
- Written unit test cases to test various modules developed.
Environment: Hadoop 2, HDFS, Hive, Pig, MapReduce, Ambari, Oozie, HDP, Java 1.7, Eclipse, Flume, SQL server, High Charts, JSP, Spring, HTML 5, CSS, JQuery, C#, Ajax, knockout, bootstrap, JavaScript, JSON,SQL, ETL, Web Services, LINQ
Confidential, Phoenix, AZ
Hadoop Developer
Responsibilities:
- Involved in reviewing and developing specifications, develop and test various application components to meet the requirements of application development.
- Worked in design and development of a 4 node Hadoop cluster for POC
- Importing and exporting data into HDFS and Hive using Sqoop.
- Used Oozie to define various work flow scenarios and scheduling
- Used MongoDB for storing unstructured data
- Extensively used Pig for data cleansing.
- Exposure to Apache Solr search capabilities
- Worked with business teams and created Hive queries for ad hoc access.
- Prepared Technical Design Documents for the new development
- Used SQL Developer to execute the query and analyzing the data.
- Experience in writing and deploying ETL packages with SSIS
- Involved in preparing Unit Test cases and Test Results
- Involved in documentation, code review, analysis and fixed post production issues.
Environment: Hadoop 2, HDFS, Pig, Hive, MongoDB, MapReduce, Sqoop, Apache Solr, Java 6, Eclipse, Zoo Keeper, Node.js, SQL developer, Apache Tomcat, Web Services, Java, SQL, JSP,JSON, JavaScript, HTML 5, CSS, JQuery, Ajax.
Confidential, Indianapolis, IN
Lead/Sr. Application Developer
Responsibilities:
- Developed rich front end user interface screens for various modules using html, CSS, JSP and Ajax based DOJO technologies.
- Consult with customers to help them analyze problems or bottleneck. Study their workflows to enhance with more accurate and efficient IT solutions.
- Define and achieve client needs through one-on-one sessions and on-going communication.
- Expert in understanding database model and extracting data to send it to CDC.
- Extracted Data from various sources using ETL packages and monitoring Production ETL issues.
- Review and develop specifications, develop and test various application components to meet the requirements of application development.
- Extensively used Object Oriented analysis and Design concepts and used design patterns like Singleton and Abstract Factory, spring framework
- Implemented Web services with Kerberos
- Used N-tier architecture for presentation layer, the business and Data access layers and coded using Java, J2EE, Hibernate as persistence framework
- Responsible for development of database tables, views, procedures, functions, indexing and triggers on Oracle 11g using PL/SQL, T-SQL using tools like Toad and SQL developer.
- Developed Cascading Style Sheets (CSS) for user interface uniformity throughout the application.
- Designed and generated reports using Jasper reports
Environment: JAVA 1.5 and 1.6, Servlets, JSP, Oracle10g, Eclipse, Clear Case, EJB2.0/3.0, HTML, XML, Ajax, Java Script, OBIEE, Struts 1.0/2.0, JBOSS, Junit, Toad, REST Web Services, JSON, JQuery, DOJO, CSS, HTML5, XML, PL/SQL
Confidential, Indianapolis, IN
Sr. Application Developer
Responsibilities:
- Define and achieve client needs through one-on-one sessions and on-going communication.
- Developed front end user interface screens using Html, JSP and Ajax technologies.
- As the lead developer, coordinate and negotiate timelines and priorities of the technical aspects of project management with other teams.
- Implemented web application security methodologies throughout the life cycle of the system.
- Manage project development through progress tracking and status reporting.
- Review and develop specifications, develop and test various application components to meet the requirements of application development.
- Developed the core business logic using EJB (2.0 Session and Entity Beans)
- Written stored procedures, triggers using SQL in Oracle
- Developed Cascading Style Sheets (CSS) for user interface uniformity throughout the application.
- Used various J2EE patterns business delegate, Service Locator, Singleton and Data Access Object.
- Designed and generated Reports using SQL OBIEE.
- Used Hibernate with Oracle10g dialect for database persistence
- Written test cases and doing unit and integration testing.
Environment: JAVA 1.5 and 1.6, Servlets, JSP, Oracle10g, Eclipse, Clear Case, EJB2.0/3.0, HTML, XML, Ajax, Java Script, OBIEE, Struts 1.0/2.0, JBOSS, Junit, Toad, REST Web Services
Confidential, Indianapolis, IN
Sr. Application Developer
Responsibilities:
- Had regular interactions with customers and gathered requirements effectively.
- Understood the existing system eShare and proposed modifications to improve user interaction and code maintenance.
- Developed the GUI screens using JSP, Java script and Ajax.
- Designed reports using iReport and implemented using Jasper reports
- Migrated the refugee data from old system to new database
- Done unit testing, system testing of modules to check the functionality.
Environment: Java, J2EE, Tomcat, XML, Oracle, SQL, Web Services, Jasper, VSS, Junit, Eclipse, iReport
Confidential, Salt Lake City, UT
Design Engineer
Responsibilities:
- Contributed significantly in requirement analysis, layered system design and data modeling. Built functional and technical specification, Junit and nunit test cases.
- Prepared sequence diagrams, class diagrams, collaboration diagram using rational rose
- Developed webservice using xfire, WSDL, SOAP.
- Workflow and POJO Design, Data Flow Mapping & Implementation of business requirements.
- Used Hibernate for relational persistence of the data.
- Prepared hibernate mapping file and persistence classes.
- Used IOC and AOP concepts of spring framework.
- Developed front end screens using asp.net, Ajax tool kit.
- Preventing the handout from being edited by two users at the same time. Involved lock, unlock and version control mechanisms.
- Implemented E-Signature Capture feature- to capture signature of the patient on the handout electronically once he is about to discharged
- Used Junit and Nunit for the functional and unit testing of the services
- Used Maven to build the project and generating the reports.
- Deployment of application in production, creation of deployment document and monitoring
Environment: Java, J2EE, JBoss, Hibernate 3.0, Webservice, XML, spring, xfire, Oracle, SQL, SOAP, Maven, WSDL, Rational Rose, Clearcase, Junit, Eclipse, Toad, Tandem, C#, ASP.Net, VS 2005,XAML, WCF,WPF, Web services, Windows Server 2003, .Net Frame work 2.0.
Confidential, NY
Application Developer
Responsibilities:
- Actively involved in design and developing of J2EE components.
- Worked on a multi-tier web based environment using J2EE technologies: JSP, EJB and Java Servlets.
- Involved in writing Design Approach Document.
- Developed the component multi bank payment approve/reject.
- Developed Web-Applications using Struts Framework
- Developed the Java Code using RAD as IDE.
- Involved in creating UI using Java Server Pages
- Developed Session beans and BMP Entity beans with EJB 2.0 standards.
- The business logic is developed using J2EE framework and deployed components on Application server where RAD was used for component building.
- Actively involved in deployment EJB service jars, Application war files in WebLogic 8.1 Application server.
- Incorporated JUnit to do the unit testing around the developed module.
Environment: Java1.4, Servlets, Struts1.1, JSP, EJB, JDBC, JavaScript, Rational Rose, Web Logic 8.1, Oracle 8i, RAD, JUnit
Confidential
Intern/Application Developer
Responsibilities:
- Requirement gathering, Low level design and test case preparation.
- Designed and developed the UI for the application.
- Unit testing and Integration Testing of the business modules using JUnit.
Environment: Java, J2EE, EJB, JDBC, Oracle, Junit, UML, Eclipse, Rational Rose, SQL