WORK HISTORY:

Confidential

Project Manager

Responsibilities:

• Successfully acted as an experienced & consultative voice to improve the working relationship in several key areas between US & Germany
• Develop roadmap for AWS & Azure hybrid cloud solutions for AI, ML, and CI between China, US, EU, including security, encryption, access and risk analysis and mitigations.
• Manage the project with HP/DXC to install and configure 3 petabyte MapR clusters (2 in Germany, 1 in US), work with global security team to develop and implement security & access controls. Successfully installed clusters, upgrades to networks, power, HVAC
• Manage the projects with ComputaCenter to install 80 gpu clusters (two US locations), resolving diversion of resources to other client projects by upper management. Install Mesos and configure for HPC scheduling (tensorflow).
• Successfully manage the deployment of a $2.8M MapR (6 Petabytes) installation with DXC/HP
• Turn around project to deploy in 3 months, a 1+year delayed effort project: global, distributed HPC clusters for ML, AD ground truth, etc. leveraging docker, Jenkins, for CI.

Technical Lead

Confidential

Responsibilities:

• 300% improvement in throughput of Hadoop clusters by changing management “requirements”
• 350% improvement to Postgres systems by moving management from a dedicated Indian DBA team to local resources
• Buildout of GFS (1 Pbyte) and Hortonworks Hadoop clusters (4 Pbytes), security enable Hadoop, enable HA (Namenode, NodeManager).
• Configure Apache Mesos for HPC scheduling (CPU, GPU, Docker containers, HPC on Hadoop); support existing Kubernetes
• Fix various Ansible scripts for AWS and on site provisioning of MapR, Apache Hadoop, Kubernetes, Docker
• Resolved major instabilities in network performance between multiple US, China, India, and German sites. Increased throughput by 300%, reduced network errors/timeouts to zero from 10% of traffic. Removed need for major network upgrades that were stated by senior management as the “solution”
• Resolved massive failures in global Git/Bitbucket, Confluence, Jira implementations by forcing 3rd party to permit me to see the real time monitoring of the VM’s, DB’s (managed by 3rd party offshore), and ESX hosts.

Confidential

Principal Solution Architect

Responsibilities:

• Azure based big data solution to support IoT scale monitoring, predictive maintenance (AI), and managed fleet operation of the world’s 2nd largest mining and port equipment handling manufacturer. Hadoop, Spark, Apache NiFi, moving from AWS, Microsoft PowerBI. POC for supply chain Block Chain (MultiChain). Azure Machine Learning, Azure Big Data, HortonWorks HWD, HWF, HW Storm. SDAF, MQTT, Kerberos, Satellite data (IoT) acquisition. ($5.2M)
• Take over existing project from another vendor after failing to meet performance and scale requirements.
• For a bank/issuer who has acquired 20+ financial firms in the past 4 years, architect and deliver a solution for very rapid onboarding of acquisitions to enable fintech analytics (BI, Actionable), including a BlockChain POC (HyperChain) for SOX driven subsidiary reporting. Leveraging Kafa, NiFi, Hadoop (full cluster & at rest encryption), Informatica into mixed data lake + data mart architecture.
• POC completed, $5.5M project has kicked off.
• For the health care management subsidiary of the largest drug distributor in the world, architect and lead the big data solution to allow predictive analytics across a range of IoT events and supply chain channel events (up & down stream), including obfuscated HIPA data. Client developed several POC’s: I am serving as the Chief Solution Architect (20 person team) to recommend and guide the execution of the recommended solution: Google Cloud based, Lambda architecture, HortonWorks HWD Hadoop/Hive, HWF (Kafka, NiFi, miNiFi), Apache Ignite, Alpine Data for AI, memSQL, etc. Phase 1 (infra & 4 Recommendation Engine re - platformings) complete.
• Develop a predictive service solution to leverage the client’s existing field support offering in a manner that both obfuscates PII data and allows relevant cell specific analytics, enabling predictive repair services.
• Ingest daily volumes from 50 countries of 2 billion events/day, able to scale 10x in case of solar flares and equivalent planet scale impacts: architect a small scale IoT Big Data solution capable of providing predictive analytics in real time. Legacy Oracle EBS, Exadata, multiple Oracle RAC’s, Informatica. Hortonworks Hadoop (HDP), NiFi+Kafka (HDN), Storm, Apache Ignite (HDP), Spark Scala (cached RDD’s), ANN ML. Pure Lambda Architecture. Minimally enriched data lakes, operational data marts, real time feeds for ML to Spark.
• Providing SRE level improvements to Hadoop and Cassandra implementations on a contractual basis (tech debt reduction). Core responsibility is Hadoop/Cassandra/Kafka operational stabilization and performance improvement.

Confidential

Responsibilities:

• Improve the stability and performance of existing Cassandra clusters.
• Determine that the problem is with data ETL and related ingest latencies rather than non-performant C* clusters, inability to test C* and ETL (IBM Streams) code at scale. Help CD team to enhance the CD/CI infrastructure to include near-production data volumes for CI testing (I added the C* clusters via Terraform, Chef, Jenkins/Maven).
• Work with IBM teams to get a production volume Streams cluster, etc. IOT (2.2 billion devices) events every 5 minutes. Support Watson Earth team’s weather monitoring, prediction, and Machine Learning optimization and scalability, tuning of Bluemix OpenWhisk (serverless apps), IoT Sensor flows (MQTT, SDAF framework data warehouses).
• Move historic data from MongoDB databases into Cassandra, stabilize the Cassandra cluster, etc.
• Cassandra (DataStax), AWS, S3, Chef, Docker, Terraform. SAP HANA, IBM Technologies (Watson (Hadoop), Guardium, BigInsights, DataEngine/Spark, etc.), Kafka. MongoDB, ELB/ALB, ElasticSearch, Redshift, Jenkins. Agile, Jira, Confluence, Slack, Streaming, Java, Scala, Splunk. Datadog, Graphana, Akamai.
• Retained to assess and improve long running MR jobs, latency issues around Facebook fire hose, missing resiliency in the Hadoop & Spark clusters. Also supported CD/CI team in leveraging containers for improved testing (Jenkins, Mavin)
• Memcached, Puppet, memSQL, Redis, Terraform, Confidential, AWS (Hadoop: multiple 2-3,000 nodes clusters, heavy S3, ELB, AMI, Route53), Pinball Workflows, Hadoop, Scalding, Cascading, Qubole, Kafka, Zookeeper, Datadog, Akamai, Splunk, ELK.
• Reduce growth rate of AWS instances by 3-5% (approx. 15% drop from growth expectations)
• Reduced operating costs by $85,000 per month through performance improvements. Improve Hive, MR, Spark, S3n utilization, work with dev. team to hack around S3n 1-300ms open delays
• Performance optimization of queries after client moved extensively to ORC, add cascading support, file Combiner, etc. for mapreduce throughput improvement (smallfiles problem in Hive input). Cost savings of $20-30,000 per month.
• Architect high availability solutions and lead the migration of 6,000 Hadoop/Spark/Hive nodes (4 clusters) to high availability (NN, NM, RM (Zookeeper, replicated journals) leveraging existing Terraform, puppet, AWS. Reduce several execution days (times 1-2000 nodes) outages due to growing AWS EC2 (S3.8xl) failures as the hardware ages, assist dev teams in implementing cascading, high input file count combiner, etc.
• Performance optimization: Annualized $250m AWS spend cut by over $2m, reduced Qubole spend by $200,000 annualized

Confidential

DevOps Team Lead

Responsibilities:

• Install and configure Redshift, EBS, Pentaho as persisted side of Lambda architecture. Data enrichment for multischema data lake.
• Document client specific AWS (Redshift, Lambda) best practices for big data lakes. Worked with client to make better use of their $100k spend on licensing of ETL tools
• Support the machine learning projects through data driven architecture optimizations, including proper data obfuscation to maintain confidentiality of PII. Assessments of 3rd party security risks, identify mitigations and get signoff of acceptable risks
• Results are tentative, due to the small timeline but: Improve classroom utilization by 5-15%. Improve cross-matriculation by 12-18%. Reduce dropout rate of first year students by 3-5% with improved prerequisite and remedial training.

Environment: Redshift, NgInx, MuleSoft, JasperSoft, Pentaho, PostgreSQL, AWS S3n, EC2

Confidential

Responsibilities:

• Stabilize and productionalize the CDH cluster, attempt to work with the Puppet team to reduce incidents of bad configuration pushes. Migrate to HA mode (NN, NM, Yarn RM, etc.). Upgrade Cloudera 5.4 clusters to 5.7, upgrade of Kafka to 0.9. Enhancements to Datadog, Grafina.
• Stabilize Platfora team’s cluster, bringing up production clusters, integrating non-prod cluster into Jenkins for CB/CI. Upgrades, integration to Hadoop (Cloudera), monitoring, lens build error debugging, etc. Integrate with LDAP
• Set up Hive and Spark clusters for Spark streaming (cached RDD) applications
• Set up Confidential ’s enhanced Azkaban/Voldemort for job scheduling
• Work with CB/CI teams to add Hadoop and Spark resources to Jenkins jobs

Environment: Cloudera Hadoop (CM, CDH), Flume, Kafka, Logstash, Spark, Scala/Cascading, Impala, Neo4j, Platfora, HP Vertex. Hosted (NetApp, VMWare), bare metal, and AWS environments (EC2, S3, Confidential, Terraform), Puppet. Production clusters ranging in size from 30-200 nodes: Regions/AZ’s in China, US, UK, Germany, etc. Jenkins, Mavin, Puppet

Confidential

Architect/team lead

Responsibilities:

• Benchmarking of various document and schema repositories for client’s migration of several key applications to client’s new private cloud: resolve admin and stability issues with several existing (Cassandra, Kafka, LogStash) components; analysis to identify business use cases for migrating 200 applications of 400+ nodes from (mostly) Amazon Web Services to client’s under construction private cloud with a focus on fast data (under 10msec ingest times).
• Performance improvements to Pentaho to handle data stream (ETL) enrichment at scale. 100* node DataStax Cassandra (extensive stabilization and performance cleanup), 100 node Kafka, ELK, Pentaho, Tableau, Informatica. Tensorflow
• For a new joint venture to consolidate and analyze financial instruments data, support Confidential ScaleIO infrastructure POC’s (Cassandra, Kafka/SmartStream). Support the client team in identifying key scenarios, architecting solutions to meet same around portfolio analytics, small loans, credit cards, a private block chain platform (Hyperchain, Multichain). Oracle EBS, purpose built systems, Confidential River, etc.
• AWS S3, ELB, EC2, Route53/AIM, Cassandra, Hadoop (Pivotal Greenplumb, HortonWorks, Apache), Terraform, Confidential, ScaleIO, Docker, Mesos/Mesosphere, Kafka, SmartStream, Spark/Storm, ScaleIO