Paper Title: The Design of Key-Value Store for Data Acquisition Systems

Paper Abstract: Data acquisition (DAQ) systems read out and process data from thousands of sensors. Traditionally, an important stage in the data acquisition chain has always been event building: data fragments coming from different sensors are identified as belonging to the same physical event, and are assembled into a single file. The data is then moved to a compute farm for processing. A new approach involves building logical events using a large distributed key-value store. Fragments belonging to one event can be queried directly by the processes carrying out the data analysis and filtering when needed. The approach leads to optimized usage of network and storage resources, and greater CPU efficiency in the compute farm. The main challenge in the design of key-value store for DAQ is handling the bandwidth and capacity requirements of real applications. For example, physics experiments within the European Center for Nuclear Research (CERN) Large Hadron Collider (LHC) produce data with rates up to 6 TB/s. The storage required for buffering the data is many petabytes in size. Meeting such ultra-high bandwidth requirements depends on binding persistent memory, NVMe disks, and RDMA technology with open-source frameworks such as PMDK (Persistent Memory Development Kit), SPDK (Storage Performance Development Kit), and DPDK (Data Plane Development Kit).

Paper Author: Maciej Maciejewski, Software Architect, Intel
Jakub Radtke, , Intel

Author Bio: Maciej Maciejewski is a Software Architect at Intel Poland, where he works on nonvolatile memory projects in the Data Center Group. He has been the lead architect of a data acquisition database and the architect of a persistent memory storage engine for MongoDB, as well as having worked on non-volatile memory enabling software, RDMA solutions, and Windows kernel drivers. He has 15 years experience in the technology industry. He earned a PhD and MSc in optoelectronics at the Gdansk Institute of Technology (Poland) and holds a patent in memory systems.

Author 2 Bio: