Paper Title: NVMe-oF Aware Filesystem Accelerates Machine Learning Workloads

Paper Abstract: Machine learning workflows require systems that can provide high throughput access to large numbers of files and large amounts of data storage. Legacy (HDD-based) filesystems can’t do the job, as they are not suitable for low-latency, small file I/O, and metadata heavy workloads. The result is that the GPUs and CPUs sit idle much of the time waiting for I/O (so-called I/O starvation). A new next-generation parallel shared filesystem solves the problem. It is NAND flash and NVMe optimized, so it takes advantage of the latest hardware. A case study shows how a large autonomous driving car manufacturer used the new file system to ensure that their GPUs are always saturated with data, hence working at their maximum potential. Such next-generation file systems utilize flash to turn the compute layer into a high-performance storage system which keeps hot data closer to the applications.

Paper Author: Liran Zvibel, CTO And Co-Founder, WekaIO

Author Bio: Liran Zvibel is Co-Founder/CEO at Weka.io, a startup that has developed the world’s fastest parallel I/O filesystem with an emphasis on AI and life sciences applications. He guides long-range technical strategy and key partnerships with HPE, Mellanox, and Amazon Web Services. Weka.io’s file system is cloud-native and scalable and delivers unprecedented performance. Liran was previously a principal architect at XIV, where he was responsible for the hardware platform, clustering infrastructure, and overall systems integration for the high-end, grid-scale XIV storage system (now an IBM product) that excels in tuning-free consistent performance, extreme ease of use, and exceptional data economics. Liran earned a BSc in mathematics and computer science from Tel Aviv University. He has given presentations for Weka.io at many events, including Supercomputing, Stanford HPC Conference, HPCast, Storage Field Day, and past Flash Memory Summits.