- 9+ years of IT experience in Analysis, Design, Development, Implementation, Integration and testing of Application Software in web - based environments, distributed n-tier products and Client/Server architectures .
- 5+ years of Big Data/Hadoop experience using Hadoop stack (MapReduce Programming, Pig, Hive, Oozie and Sqoop, Flume, Spark/Spark SQL, Strom, Kafka, Yarn/MRv2) and NoSQL technologies such as Cassandra, Hbase and MongoDB .
- Good knowledge of Hadoop Architecture and various components such as HDFS , Job Tracker, Task Tracker, Name Node, Data Node, Resource manager and YARN concepts .
- Sophisticated experience with Hadoop distributions like Apache, IBM Big Insights, Cloudera, Hortonworks & MapR.
- Expert in importing data using Sqoop into HDFS from various Relational Database Systems .
- Worked on standards and proof of concept in support of CDH4 and CDH5 implementation using AWS cloud infrastructure.
- Hands on experience in implementing complex business logic and optimizing the query using Hive QL and controlling the data distribution by partitioning and bucketing techniques to enhance performance.
- Good knowledge in Architecting, Designing, re-Engineering and Performance Optimization .
- Experience with Oozie Workflow Engine in running workflow jobs with actions that run Hadoop Map Reduce and Pig jobs .
- Used Talend Open Studio for Big Data Integration and ETL operations.
- Very Good Knowledge in Logical and Physical Data modeling, creating new data models, data flows and data dictionaries.
- Experience in understanding the security requirements for Hadoop and integrate with Kerberos authentication and authorization infrastructure .
- Experience in importing and exporting the data using Sqoop from HDFS to Relational Database systems/mainframe and vice-versa.
- Hands-on experience in using relational databases like Oracle, MySQL and MS-SQL Server .
- Experienced the integration of various data sources like Java, RDBMS, Shell Scripting, Spreadsheets , and Text files.
- Experience in Web Services using XML, HTML and SOAP .
- Excellent Java development skills using Java 6/7/8 , J2EE, J2SE, Servlets, Junit, JSP, JDBC .
- Experience using middleware architecture using Sun Java technologies like J2EE, JSP, Servlets , and application servers like Web Sphere and Web logic .
- Familiarity working with popular frameworks likes Struts, Hibernate, Spring MVC and AJAX .
- Exceptional ability to learn and master new technologies and to deliver outputs in short deadlines.
- Quick Learner , with high degree of passion and commitment in work.
- Proficiency with mentoring and on-boarding new engineers who are not proficient in Hadoop and getting them up to speed quickly.
Big Data Technology: HDFS, Mapreduce, HBase, Pig, Hive, SOLR, Sqoop, Flume, MongoDB, Cassandra, Puppet, Oozie, Zookeeper, Spark, Kafka, Talend
Java/J2EE Technology: JSP, JSF, Servlets, EJB, JDBC, Struts, Spring, Spring MVC, Spring Portlet, Spring Web Flow, Hibernate, iBATIS, JMS, MQ, JCA, JNDI, Java Beans, JAX-RPC, JAX-WS, RMI, RMI-IIOP, EAD4J, Axis, Castor, SOAP, WSDL, UDDI, JiBX, JAXB, DOM, SAX, MyFaces(Tomahawk), Facelets, JPA, Portal, Portlet, JSR 168/286, LifeRay, WebLogic Portal, LDAP, JUnit.NET
Hadoop Distribution: Cloudera, Hortonworks, IBM Big Insights
Cloud Computing Service: AWS (Amazon Web Services)
Development Framework/IDE: RAD 8.x/7.x/6.0, IBM WebSphere Integration Developer 6.1, WSAD 5.x, Eclipse Galileo/Europa/3.x/2.x, MyEclipse 3.x/2.x, NetBeans 7.x/6.x, IntelliJ 7.x, Workshop 8.1/6.1, Adobe Photoshop, Adobe Dreamweaver, Adobe Flash, Ant, Maven, Rational Rose, RSA, MS Visio, OpenMake Meister
Web/Application Servers: WebSphere Application Server 8.x/ 7.0/6.1/5.1/5.0, WebSphere Portal Server 7.0/6.1, WebSphere Process Server 6.1, WebLogic Application Server 8.1/6.1, JBoss 5.x/3.x, Apache 2.x, Tomcat 7.x/6.x/5.x/4.x, MS IIS, IBM HTTP Server
Databases: NoSQL, Oracle 11g/10g/9i/8i, DB2 9.x/8.x, MS SQL Server 2008/2005/2000, MySQL
NoSQL: HBase, Cassandra, MongoDB
Reporting Tools: Tableau, Datameer
Operating Systems: Windows XP, 2K, MS-DOS, Linux (Red Hat), Unix (Solaris), HP UX, IBM AIX
Version Control: CVS, SourceSafe, ClearCase, Subversion
Monitoring Tools: Embarcadero J Optimizer 2009, TPTP, IBM Heap Analyzer, Wily Introscope, JMeter
Other: JBoss Drools 4.x, REST, IBM Lotus WCM, MS ISA,CA SiteMinder, BMC WAM, Mingle
Confidential - Minneapolis, MN
- Built a suite of Linux scripts as a framework for easily streaming data feeds from various sources onto HDFS.
- Written multiple MapReduce programs in java 8 for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
- Wrote Interface specifications to ingest structured data into appropriate schemas and tables to support the rules and analytics.
- Involved in migration from Hadoop System to Spark System;
- Involved in start to end process of Hadoop cluster setup where in installation, configuration and monitoring the Hadoop Cluster.
- Responsible for Cluster maintenance, Cluster Monitoring, Troubleshooting, Manage & review Hadoop log files. Installation of various Hadoop Ecosystems.
- Responsible for Installation and configuration of Hive, Pig, Oozie, HBase and sqoop on the Hadoop cluster.
- Used Spark Streaming API with Kafka to build live dashboards; Worked on Transformations & actions in RDD and Spark Streaming, MapReduce, Pair RDD Operations, Partitioner, Check-pointing, and SBT.
- Refactored formal Hive queries to Spark SQL.
- Involved in moving all log files generated from various sources to HDFS for further processing through Flume.
- Extracted the data from Teradata into HDFS using Sqoop and exported the patterns analyzed back into Teradata using Sqoop.
- Processed HDFS data and created external tables using Hive and developed scripts to ingest and repair tables that can be reused across the project.
- Used Talend Open Studio to perform ETL aggregations in Hadoop HIVE & PIG .
- Wrote Pig Scripts to generate MapReduce jobs and performed ETL procedures on the data in HDFS.
- Developed Pig Scripts, Pig UDFs and Hive Scripts, Hive UDFs to load data files.
- Used hive schema to create relations in pig using HCatalog.
- Developed a Java MapReduce and pig cleansers for data cleansing.
- Implemented POC to migrate map reduce jobs into Spark RDD transformation using Scala IDE for Eclipse.
- Implemented Machine Learning Models like K-means clustering using PySpark.
- Used Spark to create reports for analysis of the data coming from various sources like transaction logs.
- Used Oozie Operational Services for batch processing and scheduling work flows dynamically.
- Used Maven extensively for building jar files of MapReduce programs and deployed to cluster.
- Managed Agile Software Practice using Rally by creating Product Backlog, Iterations and Sprints in collaboration with the Product Team.
Environment: Hadoop, HDFS, Hive, Pig, MapReduce, YARN, Datameer, Sqoop, Flume, Oozie, Linux, Teradata, HCatalog, Java 8, Eclipse IDE, GIT.
Confidential - Louisville, KY
- Designed, developed and tested Map Reduce programs on Mobile Offers Redemptions and Sent it to the downstream applications like HAVI. Scheduled this MapReduce job through Oozie workflow.
- Ingested huge amount of XML files into Hadoop by Utilizing DOM Parsers with in Map Reduce. Extracted Daily Sales, Hourly Sales and Product Mix of the items sold in Yum Brand Restaurant's and loaded them into Global Data Warehouse.
- Wrote and tested the MapReduce code to do aggregations on identified and validated data.
- Scheduled Multiple Map Reduce jobs in Oozie. Involved in extracting the promotions data for all stores within USA by writing the map reduce jobs and automating it with UNIX shell script.
- Gathered business requirements in meetings for successful implementation and POC and moving it to Production.
- Implemented POC to migrate map reduce jobs into Spark RDD transformations using Scala IDE for Eclipse.
- Implemented different machine learning techniques in Scala using Scala machine learning library.
- Developed Spark applications using Scala for easy Hadoop transitions.
- Created RDD's in Spark Scala and Python.
- Closely worked with Admin team to gather hardware for Data nodes, edge nodes, and Name nodes.
- Successfully loaded files to Hive and HDFS from Oracle, Netezza and SQL Server using SQOOP.
- Used Talend Open Studio to load files into Hadoop HIVE tables and performed ETL aggregations in Hadoop HIVE.
- Designed & Created ETL Jobs through Talend to load huge volumes of data into Cassandra.
- Used Sqoop to import data from SQL server to Cassandra.
- Worked on analyzing, writing Hadoop MapReduce jobs using Java API, Pig and Hive.
- Developed some machine learning algorithms using Mahout for data mining for the data stored in HDFS.
- Used Flume extensively in gathering and moving log data files from Application Servers to a central location in Hadoop Distributed File System (HDFS).
- Worked with Oozie Workflow manager to schedule Hadoop jobs and high intensive jobs.
- Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
- Loaded data into HIVE tables, and exten sively used Hive/HQL or Hive queries to query data in Hive Tables.
- Introduced Tableau Visualization to Hadoop to produce reports for Business and BI team.
- Creating UDF functions in Pig & Hive and applying partitioning and bucketing techniques in Hive for performance improvement.
- Creating indexes and tuning the SQL queries in Hive and Involved in database connection by using Sqoop.
- Involved in Hadoop Namenode metadata backups and load balancing as a part of Cluster Maintenance and Monitoring.
- Worked on Spark with Python and Scala.
- Serialized data in Hadoop using Avro serialization system.
- Used File System Check (FSCK) to check the health of files in HDFS.
- Monitored Nightly jobs to export data out of HDFS to be stored offsite as part of HDFS backup.
- Used Pig for analysis of large data sets and brought data back to Hbase by Pig.
- Scheduled, monitored and debugged various MapReduce, Pig, Hive jobs using Oozie Workflow.
- Design and deployment of Storm cluster integration with Kafka and HBase.
- Implemented authentication and authorization service using Kerberos authentication protocol.
Environment: Hadoop 1.2.1, MapReduce, Sqoop 1.4.4, Hive 0.10.0, Flume 1.4.0, Oozie 3.3.0, Pig 0.11.1, Hbase 0.94.11, Scala, Python, Zookeeper 3.4.3, Talend Open Studio, kafka, Storm, Oracle 11g/10g, Apache Cassandra, SQL Server 2008, MySQL 5.6.2, Java 7, SQL, PLSQL, Toad 9.7, Eclipse Kepler IDE, Microsoft Office 2007, MS Outlook 2007, SharePoint Teamsite.
Confidential - San Francisco, CA
- Analyzed, Designed and developed the system to meet the requirements of business users.
- Participated in the design review of the system to perform Object Analysis and provide best possible solutions for the application.
- Imported and exported terabytes of data using Sqoop from HDFS to Relational Database Systems.
- Developed MapReduce Jobs using Hive and Pig.
- Collected and aggregated large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
- Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in Java for data cleaning and preprocessing.
- Developed Map Reduce (YARN) jobs for accessing and validating the data.
- Involved in managing and reviewing Hadoop log files.
- Load and transform large sets of structured, semi structured and unstructured data.
- Responsible to manage data coming from different sources
- Involved in loading data from UNIX file system to HDFS.
- Installed and configured Hive and also written Hive QL scripts.
- Involved in creating Hive tables, loading with data and writing hive queries which run internally in map reduce way.
- Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
- Monitor System health and logs and respond accordingly to any warning or failure conditions.
- Used ClearCase for version control.
Environment: Hadoop, Map Reduce, HDFS, Hive, Java, Hadoop distribution of Hortonworks, Cloudera, Map, Flat files, Oracle 11g/10g, UNIX Shell Scripting, ClearCase, Junit
Confidential - Horsham, PA
- Involved in the analysis, Design, Coding, Modification and implementation of User Requirements in the Electronic Credit File Management system.
- Designed the application using Front Controller, Service Controller, MVC, Session Facade Design Patterns.
- The application is designed using MVC Architecture.
- Implemented the required functionality using Hibernate for persistence & Spring Frame work.
- Used Spring Framework for Dependency Injection.
- Designed and implemented the Hibernate Domain Model for the services.
- Involved in design of JSP's and Servlets for navigation among the modules.
- Developed various EJBs for handling business logic and data manipulations from database.
- Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
- Developed SQL queries and Stored Procedures using PL/SQL to retrieve and insert into multiple database schemas.
- Developed the XML Schema and Web services for the data maintenance and structures Wrote test cases in JUnit for unit testing of classes.
- Used DOM and DOM Functions using Firefox and IE Developer Tool bar for IE.
- Debugged the application using Firebug to traverse the documents.
- Involved in developing web pages using HTML and JSP.
- Provided Technical support for production environments resolving the issues, analysing the defects, providing and implementing the solution defects.
- Built and deployed Java applications into multiple UNIX based environments and produced both unit and functional test results along with release notes.
- Developed the presentation layer using CSS and HTML taken from bootstrap to develop for browsers.
- Involved in Development of master screens like Service Requests, Change Requests Screens.
- Design architecture following J2EE MVC framework.
- Developed interfaces using HTML, JSP pages and Struts -Presentation View.
- Developed Struts Framework and configuring web.xml and struts-config.xml according to the struts framework.
- Developed and implemented Servlets running under JBoss.
- Used J2EE design patterns and Data Access Object (DAO) for the business tier and integration Tier layer of the project; Developed Java UI using swing.
- Used Java Message Service (JMS) for reliable and asynchronous exchange of important information between the clients and the customer
- Designed and developed Message driven beans that consumed the messages from the Java message queue.
- Development of database interaction code to JDBCAPI making extensive use of SQL Query Statements and advanced prepared statement.
- Taken care of complete Java multi-threading part in back end components.
- Inspection/Review of quality deliverables such as Design Documents.
- Wrote SQL Scripts, Stored procedures and SQL Loader to load reference data.
- Used Web Services for interacting with a remote client to access data; Performed Unit Testing and Regression testing; Fixed the bugs identified in test phase; Used Junit for testing Java classes.
- Used Ant building tool to build the application.
Environment: J2EE (Java Servlets, JSP, Struts), MVC Framework, Apache Tomcat, Oracle8i, JMS, SQL, HTML, JDBC, EJB, ANT, Junit.