Paper Title: Using Flash Storage to Obtain Real-Time Insight from a Data Lake

Paper Abstract: A data lake provides a large repository of raw storage for data of various types. For example, a data lake for a legal firm could include documents (such as contracts), videos, audio files (such as telephone conversations), images, databases, and other types of information. However, processing such a complex structure to obtain analytic results can require tremendous amounts of computing power (such as large clusters) that are extremely expensive. The IoT is an emerging situation that involves many data providers with different types of information. A new unified architecture combining flash and object storage, as a component within a data lake, can ingest, process, and analyze real-time streaming data and persist and correlate with historical data. It thus allows for intuitive searches, queries, and visualizations for actionable insights. Designers can then leverage flash with existing object storage to meet response times and scaling requirements for IoT use cases, without compromising on long-term data persistence, data centralization, data quality, durability, high availability, and cost.

Paper Author: Sanhita Sarkar, Global Director, Analytics Software Development, Western Digital

Author Bio: Sanhita Sarkar is a Global Director, Analytics Advanced Development at Western Digital, where she focuses on software design and development of analytical features and solutions spanning edge, data center, data lake, and cloud. She has expertise in key vertical markets such as the Industrial Internet of Things (IIoT), defense and intelligence, financial services, genomics, and healthcare. Sanhita previously worked at Teradata, SGI, Oracle, and a few startups. She was responsible for overseeing design, development, and delivery of optimized software and solutions involving large memory, scale-up, and scale-out systems. Sanhita has authored multiple patents, published several papers, and has spoken at several conferences and meetups. She received her PhD in Electrical Engineering and Computer Science from the University of Minnesota, Minneapolis.