OBJECTIVE:

Software professional with over 20 years of experience and wants to be focused in the near future on BigData using the Hadoop ecosystem of products to solve real world problems.

SUMMARY:

• I have been working in the Big Data space using Hadoop technologies for the past 6+ years.
• I have expertise in the following areasArchitecting and delivering innovative solutions for large enterprises using the Hadoop ecosystem of products.
• The projects spanned the gamut, having to deal with all aspects of Big Data for Volume, Velocity and Variety.
• Responsible for at least three medium to large scale greenfield deployments to production of hadoop based apps and counting.
• Big Picture visualization with gound level realization and execution. Very hands - on Architect who gravitates towards mild depression: -) if I do not write code for a day.
• Specialties: - Taken several BigData projects from POC to Pilot and Production Phases.
• Technologies used were Hadoop, HBase, Hive, Spark, Flink, Kafka, Flume and ELK stack. Approach from business angle to do fitment analysis in the pre-sales phase with the goal of obtaining a PO and paid engagement.
• Also done small projects in the IoT space.
• Deploy said apps, clusters, workflows on bare-metal, in private cloud or in AWS public cloud.
• Expertise in Product Development including Architecture, Design, Development, Production Rollout, Performance Tuning for highly transactional and concurrent app usage.
• Build the solutions so that they scale linearly which involves extensive performance tuning of the various applications that run on YARN, Map/Reduce and Spark (YARN and standalone).
• Stand up multi-node infrastructures on-prem bare metal, private cloud or public cloud.
• Implement multi-stage lambda workflows onn these infrastructures. Performance tuning for linear scaling and asymmetrically where possible.
• Analytics in R for specific domains for regression, classification with integrations to Hadoop
• Technologies and Platforms (all current and using actively): CDH, Hortonworks, MapR, Flume, Kafka, Spark (Core, Streaming), Flink, Hive (M/R, Tez, LLAP), HBase, Presto, Phoenix, Druid, Elasticsearch, Kibana, R, H20, Predictive Analytics (linear regression, multi-variate regression, k-means, naïve bayes clustering, classification). Visualization of same with ggplot2, python libraries etc.

PROFESSIONAL EXPERIENCE:

Senior Solutions Architect

Confidential

Responsibilities:

• Confidential has built an in-memory computing engine that helps accelerate Big Data workflows. Lambda and Kappa workflows using Kafka, Flume, Spark, Hadoop Map/Reduce, HBase, Hive with Tez, LLAP, Flink, Presto, Druid, ElasticSearch, Kibana, Zeppelin.
• Work with engineering and Cancun’s clients to implement the solution. Also design a reference lambda architecture based on Spark and Flink streaming to insert Cancun at various legs of the workflow. Implemented using Cloudera CDH 5.x, Hortonworks HDP 2.6, Apache Hadoop 2.7, 2.8 Hadoop platforms.
• Setup Presto in a 25 node setup to run queries against hive using hive-hadoop2 connector.
• Ran tests with TPC-DS hive-testbench framework for all the queries and for production on the 25 node cluster with 10 concurrent users.

Principal Solutions Architect

Confidential

Responsibilities:

• Worked with two of Robin's largest customers. Saw company grow to the next level including the closing of the first million dollar plus PO from one of those clients that i worked very closely with, shepherding them from POC to Pilot to Production Deployment.
• Implementing Robin’s Container and Application Orchestration solution for Big Data Applications (Hadoop, Spark) for the world largest retailer and another top 10 retailer.
• This involved capacity planning for the hardware infrastructure to take into account auto scaling for peak usage.
• Cluster security with Kerberos, Ranger.
• Designing and implementing Lambda streaming and batch workflows with Flume, Kafka, Spark streaming, HBase and Zeppelin.

Sr. Big Data/Hadoop Architect

Confidential

Responsibilities:

• Apart from the Hadoop related engagement listed below, I have been involved in the following types of activities,
• Evolve a comprehensive Training program for Confidential for Hadoop, HDFS/mapreduce, HBase, Hive, Pig, Sqoop, Oozie.
• I have conducted training as well as train the trainer programs. Assigned $3m revenue target for FY2012-13.
• Always presented to audience at “C”, VP and Sr. Director levels. Closed several small to medium sized deals. Big Data being an emerging technology, everyone is taking baby steps towards adoption.
• All are in try and then implement mode. Start with several POCs and then grow into a large project.
• Have built teams in both onshore and offshore.
• Have evolved comprehensive Hadoop training programs and trained a lot of offshore resources.
• Responsible for interviewing and hiring onshore Hadoop Architects.
• One of the projects for a key BFSI major started in September 2011 and is not is Phase II of production deployment.
• Have prepared presentations and been involved in pre-sales for very senior level executives for various of Confidential ’s clients and prospects.
• Have presented at the Hadoop Strata conference held in San Jose in June 2012 on behalf of Confidential I have written several articles on Big Data and Hadoop for the Confidential website as well as for other websites such as cloudstory.in I am a self professed Hadoop and Big Data junkie.
• Not only do I try to bring new and innovative ideas and technologies into the projects I implement but I do have a fully functioning (pseudo distributed, almost complete Cloudera CDH4 stack) Hadoop cluster installed on my mac book pro and doing project related development activities and as well as R&D activities with a focus in the areas of Analytics.
• I feel that the true value of a Big Data cluster can only be realized through the extraction of those nuggets that are embedded in the ocean of data, akin to searching for the needle in the haystack, the detection of key signals from the noise and these can be achieved through tools like Hive, R, Mahout and technologies built around these concepts of Classification and Clustering.
• Have evangelized Hadoop within the organization by conducting training programs, brown bag sessions, webinars and writing white papers and articles.
• Have built teams that are working on very large accounts and involved in designing and delivering Hadoop based solutions.
• I have had to evaluate other related technologies such as columnar databases and appliances such as Neteeza, Vertica, Exadata.
• Products that work with the Hadoop ecosystem such as Pentaho, Talend, Tableau.
• Have evolved architectures and solutions where a mix of products have come into play with Hadoop being the Hub for all Big Data activity, supporting Transactional systems as an OLAP and archiving store.
• Evangelized Lambda Architecture and implementing it using Kafka, Storm, HBase, HDFS.
• Complex Event Processing using Storm and heterogenous message inputs including social media feeds, XML messages and raw text messages.
• Orchestration of these messages and processing them in a storm topology and disposition of messages to end systems. Also using spark/shark framework for consumption oriented, in-memory analytic use cases.
• Involved in a very diverse range of activities that include the followingShepherded a project from POC to Pilot and Production
• The key use case was to replatform an existing application to Hadoop.
• Transfer data from a Teradata/Memcache environment to Hadoop/HBase and ensure that the same functionality achieved with Hadoop.
• Devise ELTC and archiving strategies for Hadoop and HBase using Hadoop, HBase APIs, map/reduce and bulkload.
• Performed load tests for file and data uploads using multi-threaded ETL routines
• Used myriad patterns for ETL and query including HBase bulkupload, map/reduce, Get, Put, Range scans and Puts etc.
• Scripts performed well within set SLAs and realized a minimum of 3x improvement over earlier mechanisms for both ETL and Query. HBase query performance improved by more than 10x.
• Develop scripts, programs and instrumentation to run performance anlysis tests for ETL and Query
• Develop scripts for Querying and Extract of information from Hadoop and HBase using APIs. Perform a variety of different query operations on HBase using gets and scans.
• Devise overall monitoring and alerting strategy for Hadoop clusters. Built Ganglia and Nagios from source tarballs on RHEL Linux. Deployed on a 10 node test and 40 node production cluster.
• Wrote cluster management Tcl scripts to manage 40+ node clusters (distribute files, start processes etc remotely).
• Wrote custom PERL scripts to collect process level info from all nodes into Mysql DB instance
• Setup a 5 node test cluster with Hadoop, HBase and Hive (CDH3) based on Vmware vsphere and Oracle Linux VMs.
• Also doing a POC on Hive (0,70) with various ETL strategies including sqoop, loads, external tables, partitions etc. Also testing integration of Hive with HBase

Technologies: Hadoop and java API, HBase and java API, customer ETL and query scripts for Hadoop and Hive.

Confidential

Lead Architect

Responsibilities:

• As part of this effort designed and implemented a very unique metadata driven Process Orchestration Engine.
• All processes that have to be orchestrated are specified as metadata in HBase tables.
• Written custom data formats to process EBCDIC and XML files.
• Designed and built a unique application layer over Hadoop known as the Process Orchestrator Engine.
• The Orchestration Engine is coded in Java using HBase APIs, will read metadata information at runtime and execute the processes.
• All post-operational information related to the execution of a process is recorded in a checkpoint table for audit, tracking and reconciliation purposes.
• The processing steps within Hadoop are orchestrated using a metadata driven process orchestration engine. The metadata steps are defined in HBase during design time. At runtime detailed execution metrics are captured in a HBase checkpoint table
• A total of 800+ different feed types are being implemented currently
• The Hub addressed the two key aspects of Volume and Velocity as both batch and real-time feeds were processed
• Designed and implemented a Disaster Recovery mechanism for the cluster
• Currently implementing solutions in the statistical analysis space using Shiny R and R regression analysis
• Completed Phase II production deployment in Oct 2013.
• Implemented Kerberos and AD based security to establish data co-location in HBase.
• Implemented data pipelines to ingest and process large files and other internal data formats like EBCDIC.
• Ingestion of batch feeds with HBase sink
• Ingestion of realtime feeds with Storm topology consisting of JMS spout and HBase and HDFS sinks.
• Worked on POC to show usage of Kafka and Spark and it’s advantages of Storm streaming.

Technologies: Hadoop, HBase, Hive, Pig, Sqoop, Java map/reduce (secondary sort, map-side join and multipletextoutputformat patterns), Process Orchestrator Application deployed on J2EE application server, Pentaho DI, Shiny R framework.

Confidential

Responsibilities:

• Strategic engagement where I built the team including providing training in relevant Hadoop technologies.
• Was responsible for the overall architecture and initiation.
• The project is ongoing with implementation going on full swing.
• The overall goal of the project is to acquire as much Customer data (master and transactional) as possible.
• For master data it will be monthly data consisting of 750 attributes and 4 years worth of transactional data with more than 5 billion records.
• Master data also known as demographic data will be acquired from an external agency and transactional data also known as market basket data will be sourced from various internal systems mostly relational databases including Teradata.
• This effort will integrate the various brick and mortar and .com data contained in Relational data stores by moving them to Hadoop and make the data available for the purposes of creating the Unique customer record and performing advanced analytics for targeted advertising.

Technologies: Sqoop, Oozie, Hadoop java APIs

Confidential

Responsibilities:

• Stragtegic engagement where I was responsible for the overall architecture and initiation including Hadoop cluster configuration and sizing, installation and setup of Hadoop.
• Evolving best practices for Monitoring, Alerting. Design and initial POC for some of their initial ETL efforts and their metadata repository setup, synching and maintenance.
• The project is ongoing with implementation going on full swing.
• Was involved in the phases of evolving the overall Big Data strategy for the purposes of ELTL of data from disparate sources into Hadoop and make data available for targeted advertising and other clustering and grouping functions. During the course of this effort performedCluster Sizing for the Hadoop Cluster.
• Evolved and developed custom ELTL scrips to source data from upstream systems into Hadoop.
• Evolved a detailed metadata management strategy.
• Evolved a detailed cluster monitoring and management strategy.

Technologies: Hadoop Java APIs, Hive, Sqoop

Confidential

Responsibilities:

• Very strategic engagement that involved evolving a Big Data Strategy from ground up.
• This whole effort is driven by four key use cases that they want to vet out in a Hadoop Environment.
• Evolving all aspects of the strategy including user workshops, stakeholder meetings, building delivery teams, infrastructure, architecture, design, delivery and execution.
• Started on implementation of first three use cases.
• Evangelization of key technologies like Phoenix, Stinger through POCs and ensuring adoption of use cases for production use.
• One of the projects involved the processing of very large volumes of claims data using custom map/reduce code to process incoming XML feeds and store them in HBase.
• Conducted several POCs to convert several complex DB2 SQL queries into equivalent Hive QL queries.
• Used Phoenix to showcase ad-hoc querying against HBase. Currently in the process of moving all of these to production.

Confidential

Responsibilities:

• Started as a pre-sales engagement.
• Customer had already evaluated Big Data offerings from three large global vendors but shortlisted Confidential after attending our Workshop.
• Conducted workshop for key stakeholders and at end of workshop client agreed to work with us.
• Architecture and Design of first Big Data use case in progress.
• Additional use cases being identified and additional groups within banks being engaged to evolve unified strategy.
• The Architecture and Design for Phase I implementation is underway.

Confidential

Responsibilities:

• Strategic initiative that includes advice on long term roadmap for Hadoop Adoption.
• Key tasks include evolving roadmap for the setting up of a Data Lake over a period of time by sourcing data from various legacy systems, Data Pipeline/Ingestion Patterns, Consumption patterns including BI, ad-hoc queries and statistical analysis, H/W configuration for Clusters among other things.
• Currently in project initiation stage and evolving overall Solution Architecture, Implementation Roadmap and Project Plan.

Client Architect

Confidential

Responsibilities:

• Involved in a variety of efforts including practice lead for SOA, Cloud and BigData practices, pre-sales activities.
• Program Management and SOA Architect at Confidential ’s leading clients, Confidential Center of Excellence (COE) lead and key driver.
• Thought leader and author contributing to Confidential ’s website Perspectives section.
• Author of own blog.
• Author of article on ecommercetimes.com. Guru contributor to bigdatacloud.com.
• To sum it up, a very hands on technical leader who can drive strategy and ground level tactics.
• Responsible for overall strategy for the COE. These included the following tasksHardware and Vendor selection

SOA Architect

Confidential

Responsibilities:

• Stabilized critical application that was crashing every 3-4hrs.
• This application was processing roughly 100 JMS requests per hour and was servicing several critical departments of the city including the 311 program.
• Stabilization included several surgical procedures including the rewriting and regression testing of several modules.
• After rewrite and deploy in 10/4/07, application was running fine within all defined parameters until I last checked with the folks around 12/15/07.
• SOA enabled certain components of this application by migrating it to WLI 9.2 and ALSB 2.6. As part of SOA planning, conducted workshops and interviews and formulated an initial SOA Strategy aroundSOA deployment Roadmap
• SOA Governance Model and staffing
• Services design, development, deployment and maintenance.
• A pilot was conducted to prove out and confirm some of these using WLI 9.2 and ALSB 2.6

Confidential

Senior Developer/Tech Lead

Responsibilities:

• Architect, Design and Implement various SOA initiatives for various departments of the city using weblogic WLI and Aqualogic.
• Currently working in the DocGen project which is one of the projects in an effort termed as Confidential CORE.
• This project is an enterprise wide effort implemented employing SOA principles and technologies.
• The primary function of this application is to process Home Mortgage Applications. It is a highly integrated application with interfaces to various external systems such as document generation systems, document storage and retrieval systems, document processing (distribution, compliance and closing) systems.
• The interfaces to the various external and internal systems have been implemented using JMS, webservices, EJB and NDM file transfer technology.
• My personal role in this project has been in the capacity of a Team Lead and Architect.
• In this function, I have had to wear different hats and some of the tasks and responsibilities in the course of discharging the duties for this function has beenDesign and implement EAI applications for various internal mortgage fulfillment functions using Weblogic WLI 8.1.

Confidential

Solutions Architect

Responsibilities:

• Involved in the architecture, design and implementation of two major projects within the company.
• Provide technical assistance and mentorship to two other projects.
• Involved in the formulation of various SOA strategies for an enterprise wide effort for services enablement.
• Involved in various Strategic and Tactical efforts some of which are articulated below.

Confidential

Sr. Weblogic Developer/Consultant

Responsibilities:

• Design and implement EAI components using Weblogic WLI 8.1
• EAI effort to streamline loan handling process by tying together 4 internal application
• Process flow from loan origination to loan fulfillment

Confidential, San Jose, CA

Architect/Principal Consulting Engineer

Responsibilities:

• Involved in the Architecture, Design and Implementation of a commercial Returns Management Product currently deployed in 6 worldwide sites for a fortune 100 corporation.
• My Last project involved the design, development and deployment of a highly transactional returns management system for a major Fortune 100 client who has deployed it in a production environment of 100 concurrent users for 2 major modules.
• Responsibilities included Managing and leading a team of 5 engineers.
• Architected and designed a returns management system which is part of a reverse logistics product suite and comprised of several major modules.
• The system was implemented using standards based technology and deployed on the weblogic 8.1 application server.
• Implementation of the system involved extensive use of patterns for the various client and server components. The interaction between the client and server was based on the MVC design pattern built on the Struts application framework.
• Communication between client and server was established through EJBs by sending lightweight XML constructs between the JSP and the server.
• Tag libraries were used extensively to maintain a very thin client and all server objects were coded as Session, Entity and Message driven EJBs.
• Communication to the ERP (Oracle Applications) was established via Weblogic Integration and communication between the RMS system and WLI was done through JMS. Data transfer between the ERP and WLI was through RosettaNet 2.0 PIPs.
• All the messaging needs was addressed through JMS.
• A multithreaded event manager had to be built to monitor and trigger the various types of events that were identified in the returns system.
• The whole system was designed in such a fashion that new components could be dynamically added as classes that implemented the various interfaces without much impact or change to the core components. System state was achieved through a combination of server caches with unique identifiers.
• Implementing RosettaNet business processes using the Weblogic Integration Business Process Manager.
• Designing and implementing RosettaNet PIPs for RMAs. This effort was undertaken for a few select sub vendors who wanted to do bulk returns in an automated fashion.
• Designing and building a generic rules infrastructure using ILOG. This infrastructure was to enable business users design and implement their own rules without having to implement or change the underlying rules infrastructure Java code.
• Some of the submodules were webservices enabled using one of the early implementations in Weblogic 8.1 using JAX-RPC.