💡 The paper addresses the problem of efficient memory management for large language models, which is crucial for high throughput serving. Existing systems struggle with managing the key-value cache memory, which is huge and dynamically grows and shrinks, resulting in significant waste due to fragmentation and redundant duplication. To solve this problem, the authors propose PagedAttention, an attention algorithm inspired by classical virtual memory and paging techniques in operating systems. They also build vLLM, a large language model serving system that achieves near-zero waste in key-value cache memory and flexible sharing of the cache within and across requests. The vLLM system is designed to reduce memory usage and improve throughput. The authors evaluate vLLM and show that it improves the throughput of popular large language models by 2-4 times with the same level of latency compared to state-of-the-art systems. The improvement is more significant with longer sequences, larger models, and more complex decoding algorithms. Overall, the paper contributes to the development of efficient memory management for large language models, enabling higher throughput and better performance.