Paper Title: A Scalable AI Data Pipeline for Storing and Processing Ingested Data

Paper Abstract: Artificial intelligence (AI) requires processing power and adequate storage while executing various deep learning (DL) frameworks. Unlike traditional programming, DL requires the computer to learn by itself. Different stages of training and deployment of a DL system have different data and processing needs. On the one hand, the large volumes of data involved in the training stage demands systems with support for massive storage capacity, multiple data formats and protocols for processing dispersed data sets, as well as sharing of data and models across users/applications. On the other hand, AI deployment for delivering inference on new incoming data requires fast access to the data to meet the demand for AI responsiveness for users/applications. The processing and storage needs vary for the different phases of an AI data pipeline comprising data ingestion, model training and model serving. Disaggregation of GPUs, flash and object storage can enable the delivery of rapid response times and scaling requirements of an AI data pipeline, without compromising on data persistence, data quality, durability and cost.

Paper Author: Sanhita Sarkar, Global Director, Western Digital

Author Bio: Sanhita Sarkar is a Global Director, Analytics Software Development at Western Digital, where she focuses on software design and development of analytical features and solutions spanning edge, data center, data lake, and cloud. She has expertise in key vertical markets such as the Industrial Internet of Things (IIoT), Defense and Intelligence, Financial Services, Genomics, and Healthcare. Sanhita previously worked at Teradata, SGI, Oracle, and a few startups. She was responsible for overseeing design, development, and delivery of optimized software and solutions involving large memory, scale-up, and scale-out systems. Sanhita has authored multiple patents, published several papers, and has spoken at several conferences and meetups. She received her Ph.D. in Electrical Engineering and Computer Science from the University of Minnesota, Minneapolis.