Paper Title: Optimizing NVMe-over-Fabric Targets using Offload, Peer-2-Peer and NVMe CMBs

Paper Abstract: Now that NVMe SSDs with Controller Memory Buffers (CMBs) are coming to market we can leverage those along with the NVMe offload engine in modern Mellanox ConnectX adaptors to develop optimized NVMe-oF target systems. In this paper we look at the hardware and software frameworks that allow this to happen. We will review the NVMe offload engine in the Connect-X5 and present a summary of the NVMe CMB ecosystem. We will then review the Linux kernel code developed to control the NVMe offload engine and tie it into the NVMe p2pdma framework to provide access to NVMe CMBs. We will conclude with some performance results comparing a legacy NVMe-oF target with one enabled with the NVMe offload engine and NVMe CMBs. We will show how the later leads to a reducing in CPU utilization, memory utilization and PCIe utilization whilst maintaining throughput and reducing latency.

Paper Author: Stephen Bates, CTO, Eideticom

Author Bio: Stephen Bates is the CTO of Eideticom and is a renowned expert on topics like NVMe, RDMA, TCP/IP and NVM. He has worked on a range of complex storage and communication systems include the NVMe controllers and PCIe switches developed by his former employer Microsemi (formerly PMC-Sierra). He enjoys working at the interface between hardware and software and is an active contributor to the Linux kernel. Before Eideticom he worked in the CTO at PMC-Sierra and before that was an Assistant Professor in Computer Engineering at The University of Alberta. He holds a PhD from The University of Edinburgh, Scotland.