Paper Title: Accelerating Hadoop at Twitter with NVMe SSDs: A Hybrid Approach

Paper Abstract: Twitter found our Hadoop performance to be bottlenecked on HDD storage IOPS limits. The industry trend toward larger disks only made the problem worse. For Twitter's Hadoop capacity needs, a full flash solution remained too expensive compared to HDDs lower cost per unit of storage. Twitter's Hadoop and Hardware Engineering teams worked with Intel to experiment on approaches to solve this by firstly adding NVMe SSD caches, discovering that didn't work and then going deeper into storage profiling. The solution was a hybrid approach of adjusting workload and adding intelligent caching. This resulted in accelerated Hadoop compute, improved storage performance and reduced TCO. We will discuss the original premise of the investigation, the hardware and software tools used for benchmarking and telemetry, as well as how the test plan evolved (along with the results of each phase of testing). We will compare and contrast several approaches to managing YARN temporary data and multiple SSD technologies, and address the overall shift in the compute to storage ratios. We will summarize both the business case for this work and the economics of the resulting hybrid storage architecture.

Paper Author: Matthew Singer, Sr. Staff Hardware Engineer, Twitter

Author Bio: Matt Singer is Senior Staff Hardware Engineer and manages the System Engineering Team at Twitter. An expert in performance and server architecture, Matt is the technical leader for the hardware team that created one of the most resilient and cost-effective cloud infrastructures in the industry. He is also the architect of Twitter's 4th & 5th Generation Server Platforms, and continues to lead the company's performance analysis of new architectures and technologies. He also has extensive experience as a firmware engineer, contributing to desktop, mobile, server, and embedded system designs. He has almost 20 years experience in the technology industry, and earned a BS in computer engineering from Case Western Reserve University.