A Large-scale Analysis of Hundreds of In-memory Key-value Cache Clusters at Twitter

Analysis

Why This Paper Matters

This paper provides an unprecedented large-scale analysis of in-memory cache workloads from 153 production clusters at Twitter, processing over 80 TB of trace data. While prior studies have examined cache workloads in isolation, this work offers a comprehensive view across diverse business logic, revealing patterns that challenge long-held assumptions. For AI practitioners building latency-sensitive services, understanding real-world cache behavior is critical for optimizing performance and cost. The findings directly impact the design of caching layers in AI inference pipelines, recommendation systems, and real-time data processing.

Technical Contributions

• Scale and diversity: Analyzed 153 clusters covering a wide range of use cases, from user-facing services to internal data pipelines.
• Workload characterization: Detailed analysis of traffic patterns, TTL distributions, popularity skew, and size distributions, showing that many workloads are far more write-heavy and skewed than previously documented.
• Temporal patterns: Identified unique temporal behaviors, such as diurnal cycles and bursty access patterns, that affect cache performance.
• Replacement policy insights: Through simulations, the authors found that FIFO (First-In-First-Out) replacement works best for a large number of workloads, outperforming more complex policies like LRU and LFU.
• TTL as a defining parameter: Demonstrated that TTL is a critical factor in defining cache working sets, often more important than access frequency.

Results

The analysis reveals that many cache workloads exhibit write-heavy ratios exceeding 50%, and popularity distributions are highly skewed (e.g., 80% of requests target less than 20% of objects). Simulation results show that FIFO replacement achieves hit rates within 1-2% of the optimal policy for over 60% of workloads, while LRU and LFU can degrade performance by 5-10% in write-heavy scenarios. The study also quantifies the impact of TTL: short TTLs (minutes) dominate in real-time systems, while longer TTLs (hours to days) are common for content caching.

Significance

This paper provides a foundational dataset and analysis for cache system research, directly influencing the design of next-generation caching systems. For AI applications, where low latency and high throughput are paramount, the insights on replacement policies and TTL management can lead to more efficient resource utilization. The finding that simple FIFO often outperforms complex algorithms is particularly valuable for practitioners seeking to reduce engineering overhead while maintaining performance. This work also opens avenues for machine learning-based cache optimization, as the rich workload characteristics can serve as training data for predictive models.