SUMMARY:

• Expertise in C/C++/Assemblers, Embedded Systems RTOS Kernels, Schedulers, Device Drivers, ISRs, VMware/UNIX/Solaris/Linux 10Gbps Ethernet, TCP/Multicast UDP, Caching in real - time concurrent, multicore/multithreaded, applications. NVMe Kernel/User-mode SSD drivers for VMware and Linux, SPDK/DPDK zero-copy kernel bypass framework.
• Expert in Multicore/Multithreaded Lock-free/Wait-free data structures/algorithms of FIFO/Circular Queues, Stacks Linked Lists, and Multiple-Reader/Multiple-Writer Software Multicast Data Message queuing/distribution for Ultra-Low Latency, Market Data Feedhandlers, Order Book, Order Management and Execution, Dynamic Memory Management, Memory Pool utilities/applications.
• Expert in concurrency/multithreading/IPC: Atomics/CAS, Transactional Memory, spinlocks, mutexes, semaphores, shared memory, condition variables, barriers, sockets, thread pools, events, signals, message queues, pipes.
• Expert in high performance mass storage SSD caching, Cache Replacement Algorithms: LRU, CLOCK, NRU, CAR Compression/Deduplication, NVMe Kernel/User-mode SSD drivers for VMware/Linux OS, SPDK/DPDK.
• Extensive experience in project technical leadership, design, development and testing of software/firmware for time/mission-critical applications in SAN systems, SSD NVMe Drivers, High Frequency Trading, OS Kernel/Drivers, Networking, Defense and Aerospace, Imaging, Transaction/Signal Processing, FPGA interfacing.
• Investments/Securities Trading: Ultra-Low Latency High Frequency Trading Infrastructure for Equities, Futures, Options, FX, 10GbE Market Data Feeds, ITCH/OUCH, CME FIX/FAST, CBOE, ISE, TCP/UDP, Full-depth Order Book, Trading Algorithm scheduling/execution, Order Crossing, Risk/Order Execution, multiple exchanges/assets, accurate Time Stamping, Exchange co-location, Ultra-fast data retrieval/update.
• Experience with Asynchronous Processing and Event Driven Architecture, knowledge of IoT networking and major protocols: IPv6 Low Power Personal Area Network (6LoWPAN), Message Queue Telemetry Transport (MQTT) Machine-to-Machine/Internet of Things, CoAP - an Internet application protocol for constrained devices.
• Team player, able to help create a positive energy team environment, enthusiastic and hard worker with passion for mentoring, coaching and learning through subject-matter expert collaboration, flexible.
• Excellent analytical skills, attention to detail, utilizing principles of Object-Oriented Design and Hierarchical Finite State Machines, resulting in robust simple to understand, support and maintain systems.

SKILL:

Hardware: Intel x86 Broadwell/Haswell/IvyBridge/SandyBridge 8051, ARM Cortex, Tilera TilePro TileGx, Sun SPARC, ARC, PIC, Motorola MC68xxx, PowerPC, HP, MIPS, Cavium Octeon, NetroNome Network Packet Processors/Network Search Engines, NVidia GPU Tesla/Fermi/Cuda, Intel Phi, Multicast UDP/TCP network switches routers, Arista, NetLogic LSI, Endace DAG (Data Acquisition and Generation) Card, 10/40/100GbE NICs Solarflare, Mellanox, InfiniBand, PCIExpress, QuickPath, Xilinx Virtex, Altera FPGAs, Soc, ASIC, TCAM memories, Redline, Celoxica, DSP SHARC, NAND SSD.

Software: OS/Kernels/Internals/DeviceDrivers/ISRs, Symmetrical and Asymmetrical MultiCore Multithreaded software, NUMA, POSIX-threads, MPI, OpenMP, SPDK/DPDK, Multicast UDP/ and TCP/IP Network Protocols.

Languages: C/C++, Assemblers Fortran, Python, Java, Verilog, Pascal, Basic, COBOL, LISP, Prolog, Bourne/C/Korn/Bash shell scripts, PLC Ladder Logic.

O/S: VMware/Unix/Solaris/Linux Internals, CentOS, Red Hat, Ubuntu, Suse, IPC, Real-Time OS: CMX, C-exec, VxWorks, QNX, VDK+, Integrity, FreeRTOS, Deos, OS-9, pSOS, uCOS, eCOS, VRTX, self-developed proprietary RTOS RTKek.

Tools: git, Confluence, Jira, BitBucket, RedLine, MDE, PVCS, PerForce, DOORS, Wireshark tcpdump, Hierarchical State Machines, Purify, ClearCase, UML, TeamWare, CodeSonar Static Code Analysis, performance analysis: top/htop, Valgrind, perf, VTune

Investments/Securities: Ultra-Low/Deterministic Lock-Free/Wait-Free Tick-to-Trade Latency, Trading Algorithms, Full-depth Order Book retrieve/update algorithms, 10GbE Market Data Feed processing, ITCH/OUCH, PITCH, CME FIX/FAST, TCP/UDP market data/order execution and routing. Multiple exchanges/assets trading, Deep Packet Inspection, Network Search Engines, Accurate Time Stamping.

Other: Hard Real-Time Embedded systems, DO-178B, AR INC 429/664, CANbus, MISRA C, OOAD, Design by Contract (DbC), DbC real-time systems, Time-Domain Profiling for Real-Time systems, Multithreading, SMP, Digital Signal Processing (DSP), Time Division Multiplexing (TDM), client/server, CORBA, RPC, Ethernet, TCP/IP, RS232/422/485, MIL-STD-1553, SPI, I2C, ProfiNet, Modbus, CMMI3, SHA1, xxHash.

PROFESSIONAL EXPERIENCE:

Confidential, Phoenix Arizona

Senior Testing and Development Engineer

Responsibilities:

• Developed Linux/C PCIE Device Driver for custom SSD controller in an Embedded Data Acquisition System.
• Developed Linux/C software library used by SSD verification/testing C++ scripts.
• Developed Linux/C++ API application framework for Embedded Data Acquisition System verification/testing.
• Developed a simulator Linux Device Driver to simulate SSD devices for testing/debugging.
• Developed tests to benchmark SSD data throughput and latency response
• Developed high precision/low latency (< 1us precision) software performance testing using X86 RDTSCP
• Collaborated with other engineers on FPGA/zynq7000 development/testing.
• Developed/collaborated on Linux/C API for a third-party vendor providing environmental testing equipment.
• Conducted design/code reviews, developed driver and API documentation, provided schedule plans/estimates to the management, used git/confluence for software management/release.

Confidential, Milpitas, CA

Senior Firmware Engineer

Responsibilities:

• Firmware Development of Confidential Embedded NVMe SSD Controller based on FPGA SoC with 3 soft-core ARC CPU module under embedded MQX RTOS kernel and C/Assembler languages implementation.
• Development of NVMe Controller Device Drivers, memory management, NAND Flash Translation layer modules.
• Testing and validation of the NVMe SSD Controller performed on Windows platform using C++/Python scripts, in Agile environment utilizing git, Confluence, Jira and BitBucket tools for development process management.
• Developed test/validation design documents for Secure Firmware Update, Secure Boot and Power Management SSD Controller modules, based on the Controller Firmware Architecture documents and collaboration with Development Team engineers.
• Developed C++/Python test scripts for the above Controller modules.
• Performed testing and benchmarking of data throughput, IOPS, and latency response for a variety of inputs utilizing C++/python test scripts.
• Developed test/validation design documents for Secure Firmware Update, Secure Boot and Power Management.
• Run the test scripts to validate the Controller function, performed debugging to determine the sources of errors and provide feedback to the developers, and fixed errors found during the QA phase.

Confidential, San Jose, CA

Senior Firmware Engineer

Responsibilities:

• Development of NVMe SKHynix SSD controllers based on Xilinx SoC with 3-core ARM-r5 soft-core processor module running on embedded Linux kernel with C/Assembler languages and I2S, SPI, UART ports.
• Developed NVMe Controller Linux Device Drivers, Controller management, NAND Flash Translation layer modules and boot security features with authentication and encryption
• Implemented Secure Boot process based on the Public Key Infrastructure (PKI) to authenticate executable modules during the boot process.
• Developed testing/verification scripts on Linux platform using C/Python, in Agile environment utilizing Perforce, Jira and Confluence, tools for development process management.
• Tested the SSD Controller performance: data throughput, IOPS and latency for a matrix of input configurations.
• Developed library of Lock-free Data structures/Algorithms: Stack, FIFO, Circular Queues, Memory Pools, Cache Replacement Algorithms, and Concurrent Bit Maps.
• Debugged software/firmware errors with software dbg and JTAG debuggers.

Confidential, San Diego, CA

Senior Staff Software Engineer

Responsibilities:

• Development of high-performance mass storage caching software product called AutoCache for VMware systems, and NVMe kernel and user-mode drivers for Samsung SSDs for VMware and Linux operating systems.
• Custom-developed, ultra-low latency, high throughput, concurrent, multithreaded, multi-CPUcore, Lock-free/Wait-free, Multiple-Reader/Multiple-Writer shared data structures, algorithms and libraries, such as Stack, FIFO, Circular, Priority and Multicast Queues, Memory Pools, Cache Replacement Algorithms, Thread Pools, Hash Tables and Object Lifetime Management, all based on Atomic Operations, custom-developed Spinlocks, Transactional Memory, Thread Local Storage and Thread CPU Affinity, carefully following design principles of aligning global data structures on CPU Cache Line boundaries to avoid Cache False Sharing, preventing deadlocks and priority inversion.
• Fixed serious intractable errors in preexisting code, caused by race conditions from improper use of Atomic operations in the implementation of the Lock-free Circular Queue Confidential the core of the Cache Replacement Algorithm, preventing the product launch. The fix allowed the production testing to resume and AutoCache to be launched.
• Developed very fast DRAM Clock Cache Replacement Algorithm, based on Lock-free implementation of Circular Queue and use of other Atomic primitives, replacing previously used mutexes and spinlocks, and resulting in a major latency reduction by a factor of two.
• Developed extensive library of Lock-free Data structures/Algorithms: Stack, FIFO, Circular, Priority and Multicast Queues, Memory Pools, Spinlocks, Cache Replacement Algorithms, Thread Pools, Hash Tables, Bit Manipulation and Concurrent Bit Maps, Object Lifetime Management, for VMware and Linux operating systems.
• Significantly improved original VMware NVMe kernel driver performance by 40%, by rearranging global Data Structures to be aligned on Cache Line Boundaries, and by improving parallelism of IO job submission and completion kernel threads via implementing Lock-free Queues and two-stage, deferred processing.
• Developed a prototype of User-mode SSD NVMe Linux driver utilizing SPDK/DPDK zero-copy kernel bypass framework. Initial testing achieved in excess of 1million IOPS. Further improvements planned by custom-adding of Wait-Freedom and ABA prevention to original DPDK Lock-free Memory Pools and Circular Queues.
• Developed a powerful Latency Test Framework (LTF) using RDTSCP to precisely measure run-times of sections of executable code, with the precision of approx. 30ns and overhead of 40ns per trace. This has proven to be the most valuable tool for accurate latency/performance code profiling, and quick identification of bottlenecks.
• Run Controller/Device Driver benchmarking to measure performance improvement over the previous version.
• Mentored team members in the proper design and implementation of Lock-free Data Structures and Algorithms using advanced Atomic operations and correct use of memory barriers access/release operations.
• Followed Agile development management process using Confluence, Jira, git and Jenkins tools.

Confidential, Holmdel, NJ

Engineering Fellow, System Performance

Responsibilities:

• The company, a High Technology Startup develops high performance, small foot-print 2U Linux server, Storage Attached Network (SAN) systems, employing Deduplication algorithms to minimize physical storage utilization.
• Reported directly to CEO, responsible for improving the performance, in terms of both lowering system latency and increasing throughput, of the Storage Attached Network (SAN) product.
• Developed multitude of ultra-low latency high throughput, concurrent, multithreaded, Lock-free/Wait-free, Multiple-reader/Multiple-writer data structures/algorithms such as Multicast Message Queues, FIFOs and Memory Pools, based on Atomic Operations, custom-developed Spinlocks, Transactional Memory and Thread Local Storage.
• Developed a novel virtual/physical HDD block allocation framework, allowing concurrent allocation by multiple application threads and eliminating dedicated block recycling storage, resulting in simpler logic and improved performance. The implementation was based on custom developed Concurrent TRIE Bitmaps and Spinlock Bitmap Array, to allow simultaneous multiple block allocation/recycling by multiple, concurrent application threads, with minimum resource contention and context switching.
• The implementation replaced previous solution based on mutex locking, achieving a 3-fold reduction in the number of context switches and corresponding processing overhead and nondeterministic latency jitter.
• Developed a powerful Latency Test Framework (LTF) to precisely measure run-times of sections of executable code. Tracing can be performed on multiple sections of code in multiple threads, providing very high precision of 20-30 nanoseconds and negligible overhead of 40-50 nanoseconds per trace. This has proven to be very valuable tool for latency/performance code profiling, and quick identification of bottlenecks.
• Developed NUMA/CPU Core Isolation/Thread Affinity/Thread Real-Time FIFO Scheduling Framework to optimize and configure the system for maximum performance of reducing latency and increasing IO throughput.
• Run extensive testing to benchmark the performance improvement in IOPS, data throughput and latency jitter.

Confidential, Rochelle Park, NJ

Senior Software Consultant

Responsibilities:

• Member of a team developing high throughput, Big-Data system for Confidential &T, with the requirements to capture, process and store massive amounts of network data Confidential rates of tens of Gigabits/s (I cannot disclose the exact numbers due to the Non-Disclosure agreement).
• Developed following concurrent Lock-free/Wait-free, Multiple-reader/Multiple-writer data structures/algorithms for low latency, high throughput multithreaded processing:
• Circular Message Queue to distribute multicast Data messages to consumer threads,
• Message FIFO for queuing command and data messages,
• Memory Pools for very low latency, low jitter dynamic memory allocation avoiding nondeterministic locks,
• Custom low latency user spinlocks with CPU PAUSE feature, exponential back-off and Priority Inheritance.
• Developed application to filter out TCP/IP connection information from data stored in files using advanced string processing, hash tables and heap-based priority queues.
• Modified/added new features to HTTP header processing using hashing and regular expression processing. Increased throughput by 50% by replacing Linux Message Queues with Lock-free solution.
• Refactored/developed a library for string processing, packet tokenizing and math, eliminated hard to identify race conditions resulting in further increase in performance and reduction of latency and jitter.

Confidential, San Francisco, CA

Principal Architect/Developer

Responsibilities:

• Developed Low Latency High Frequency Trading Platform based on Intel Xeon E5 2687W v2 Linux server to initially trade EMini Futures on CME, followed by other products/instruments.
• Specified all trading infrastructure hardware components as follows:
• High Performance Supermicro trading and backup servers specially designed for High Frequency Trading, based on 8-core Intel Xeon E5 2687W v2 over-clocked processors running Confidential 3.6GHz, SSD and hard drives and ultra-high-performance RAID controller card with SAS3 12GB/s transfer rate and 1GB Cache Vault.
• SuperMicro storage server with 100TB storage capacity, with 10-core Intel Xeon E5 2690v2 processors, 24 SSD and hard drives, 3x high-performance RAID controllers with SAS3 12GB/s transfer rate and 1GB Cache Vault.
• Solarflare SFN 7322 dual 10GbE Network Interface Card with PTP (Precision Time Protocol for each server, to handle Multicast UDP and TCP network packets in the Exchange Computer network.
• Arista 7150S-52 switch with 52 10GbE ports for switching Multicast UDP Market Data packets incoming from the Exchange network, and incoming/outgoing TCP Trade Order messages generated by Order Gateway.
• Performed network packet capture and analysis using Endace DAG (Data Acquisition and Generation) Card and passive fiberoptic splitters, resulting in dramatic performance increase over software-only network sniffing tools.
• Specified and installed Redline inRush Market Data Feedhandler and Order Execution Gateways, providing Exchange Connectivity API functionality, with the goal of reducing the initial platform development cycle.
• Developed the entire software for the Trading Platform composed of the following major modules:
• Market Data Feedhandler: receives Market Data as Multicast UDP network messages, inserts preprocessed messages into proprietary Lock-free Circular Queue for an update of the Order Book.
• Trading Strategy/Algorithms: using latest Market Data and Order Execution messages/updates and the current state of the Order Book, perform trading algorithms processing, producing possible trading signals.
• Order Book trading data update and retrieval.
• Memory Management using proprietary Lock-free Memory Pools, with extremely fast, deterministic, dynamic allocation of memory buffers used throughout the system for passing market data, order, log and other messages.
• Order Execution Gateway Interface: sends order requests such as Order New, Cancel or Modify to CME Matching Engine, receives Matching Engine responses such as request Accepted/Rejected, Full or Partial Fill, using TCP network communication for reliable Order Data transmission.
• Debug/Compliance logging of the Storage Server.
• Communicate with Trading Desk to receive traders’ commands and provide real-time trading status and P&L.
• Implemented Lock-free/Wait-free Concurrent Data Structures/Algorithms, to improve latency of multithreaded Shared Memory Multicast Message Distribution, Queuing, Hash Tables, FIFOs, Link Lists, Stacks and Dynamic Memory Management.
• Successfully performed product testing in CME UAT (User Acceptance Testing) and limited production environments with the initial results of Tick-to-Trade trading latencies around 10microseconds, with the goal of achieving 5-6us with the use of custom, proprietary Market Data Feedhandlers and Order Execution Gateways.
• Tested various performance benchmarks of the X86 CPUs for different configurations of Market Data, Traded Instruments, Trading Algorithms.
• Run testing on nVidia Tesla GPU to assess latency performance for possible future Options trading.

Confidential, New York, NY

Director/Senior Software Developer

Responsibilities:

• Designed/developed System Architecture/Software for the 64-core Low Latency High Frequency Trading Platform featuring Market Data Feed/Order Book processing, parallel Strategy Algorithms scheduling and execution, Order Crossing, Risk Management/Order Execution, Compliance and Diagnostic Logging.
• Designed/developed proprietary Real-Time Embedded 64-core < 7 microsecond Tick-to-Trade latency Trading Platform (C/C++/Linux/Real-Time Linux).
• Developed custom Market Data Feedhandlers with ultra-low Tick-to-Strategy latencies: <2us for NASDAQ ITCH/BATS PITCH and <3us CME FIX-FAST Market Data Protocols, utilizing Multicast UDP network protocols.
• Developed FIX Gateways to interface with third-party trading platform Linux servers, NYSE Matching Engine for order submission and Floor Broker's hand-held devices, utilizing TCP for reliable trading orders delivery.
• Used Endace DAG (Data Acquisition and Generation) Card for 100% full line-rate network packet capture and analysis, delivering dramatic performance increase over software-only network sniffing tools, without affecting the network data flow by using passive fiberoptic splitters.
• Developed High Frequency trading strategy algorithms for Equities, Index Arbitration and Futures trading.
• Implemented Black-Scholes, Cubic Spline and Binomial algorithms for Options theoretical price calculations.
• Performed benchmark testing of Nvidia Tesla GPU and evaluated Intel Phi 1Teraflops floating point Co-processor for use as Options Theoretical Price Calculation module, to significantly reduce Options trading latency.
• Tested various performance benchmarks of the X86 CPUs for different configurations of Market Data, Traded Instruments and Trading Algorithms.
• Proposed and lead the development of the Proof of Concept 10GbE NIC hardware accelerator for Market Data and Order Execution modules, in collaboration with Tilera Corp, using 64-core TILEPro64 and 72-core TILE-Gx72 Processor SoC with 8x 10Gb Ethernet Ports, PCIe and Networking Offloads. The prototype reduced the average latency of Market Data and Order Execution modules by 40% and reduced the peak latency jitter by an order of magnitude .
• Developed Lock-free/Wait-free Data Structures/Algorithms for Multiple-Reader/Multiple-Writer Software Multicast Data Message queuing/distribution and Dynamic Memory Management utilities, significantly improving latency and determinism of key multithreaded applications in Market Data/Order Book processing, Trading Strategy scheduling/execution and Order Execution.
• Evaluated/tested various 10GbE Network Switches, Network Packet Processors, kernel-bypass Network Interface Cards and GPUs for adoption in the custom Ultra-Low Latency High Frequency Trading Platform.

Confidential, Chicago, IL

Principal Architect/Lead Developer

Responsibilities:

• Lead three-member development team.
• Designed System Architecture/Software for the 64-core Ultra High Frequency Trading Platform featuring.
• Market Data Feed processing, full-depth Order Book, parallel Strategy Algorithms scheduling and execution.
• Developed < 500ns latency Full-depth Order Book retrieval/update Data Structures and Software Algorithms.
• Designed/developed ultra-low latency, multi-core Market Data FeedHandlers/Order Execution Gateways for exchange connectivity (CME FIX/FAST, CBOE, and ISE) on top of Multicast UDP/TCP network protocols.
• Used Endace DAG (Data Acquisition and Generation) Card for 100% full line-rate network packet capture and analysis, delivering dramatic performance increase over software-only network sniffing tools, without affecting the network data flow by using passive fiberoptic splitters.
• Developed server based Theoretical Price interpolation framework for Options trading.
• Performed benchmark latency testing of Nvidia Tesla GPU for use as Options Theoretical Price Calculation module, to significantly reduce Options trading latency.
• Tested various performance benchmarks of the X86 CPUs for different configurations of Market Data, Traded Instruments and Trading Algorithms.
• Designed/developed ultra-low latency, lock-free/wait-free, deterministic Memory Management System allowing multiple program threads to simultaneously, in-parallel allocate dynamic memory buffers in constant time.
• Developed from scratch in collaboration with Tilera Corp a POC (Proof Of Concept) hardware acceleration module for Market Data and Order Execution, based on 64-core Tilera Corp. TILEPro64 CPU.

Confidential, Chicago, IL

Senior High Frequency Infrastructure Developer/Architect

Responsibilities:

• Designed/developed prototype of Ultra Low-Latency, High Frequency Embedded Trading Platform based on 64-core Tilera processor.
• Designed/developed ultrafast Full-depth Order Book retrieval/update, used Linux CentOS/C.
• Participated in the development of nVidia GPU based Theoretical Price Calculation software for Options trading.
• Evaluated wide array of chips/boards/platforms as potential components for a new UHFT platform.
• Evaluated many-core MPU/GPU processors, Network Packet Processors, Network Search Engines/TCAMs networking/interconnect technologies: 10GbE, InfiniBand, PCIExpress, Interlaken/LA-1, QuickPath and Network Interface Cards for handling of Multicast UDP/TCP 10GbE data transmission in Exchange Computers network.
• Used software-based Network Sniffer tool to troubleshoot the Market Data/Trade Order UDP/TCP network data packets between the Exchange Computer’s switches and the Trading server platform.
• Assembled and led a three-person development team.

Confidential, Huntsville, AL

Senior Software Consultant

Technical Environment: PC104/Linux/C++, ADSP SHARC/21xx/C/Assembler, Xilinx FPGA

Responsibilities:

• Development of Real-Time Embedded software/firmware for Army/Air Force multi-FPGA multi-processor/multi-DSP/Codec failsafe switch/router to connect/route military radios, analog/digital voice/data channels, RS422/485 2/4-wire serial and high speed Fibre channels, intercoms, modems and flat panel operator display/control devices for Patriot Missile Defense System multilevel battleground communications, command and control.
• The system uses dual Time Division Multiplex serial buses for physical transfer of all digitized data.
• Developed device drivers for Analog Devices SHARC/21xx DSPs to interface with TDM bus and FPGA.
• Developed Real-Time DSP signal processing software applications for tone detection, generation, audio channel mixing and control of remote military radios using GRA-39 and RF-3045B standards.
• Designed DSP Interrupt architecture to interface with FPGAs, TDM busses and A/D Codecs.
• Refactored legacy C/Assembler programs dramatically reducing code size, complexity and testing effort.

Confidential, Hauppauge, NY

Senior Software Consultant

Technical Environment: ARC SOC/CMX Real-Time OS/C/Assembler, 8051/C/Assembler.

Responsibilities:

• Performed thorough assessment of the company’s current software architecture and developed the transition plan to the new Real-Time Operating System based platform.
• Designed and developed a Device Driver infrastructure for CMX Real-Time OS which did not offer it.
• Lead the team, coached and mentored team members in development of event-driven Real-Time OS.
• Introduced/implemented DbC (Design by Contract) to preemptively find/fix errors, and self-document the code.
• Developed software analysis tools for profiling/tuning of Real-Time system behavior in time-domain.
• Discovered several bugs in the commercial Real-Time CMX RTOS kernel, designed/implemented workarounds.
• Saved the company approx. two man-years of development (four developers for six months).