To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.

As in OS’s virtual memory, vLLM does not require reserving the memory for the maximum possible generated sequence length initially.

The key idea behind vLLM’s memory manager is analogous to the virtual memory [25] in operating systems.

In this work, we develop a new attention algorithm, PagedAttention, and build an LLM serving engine, vLLM, to tackle the challenges outlined in §3

vLLM implements various decoding algorithms using three key methods: fork, append, and free.

vLLM is effective in distributed settings by supporting the widely used Megatron-LM style tensor model parallelism strategy on Transformers

In vLLM, we adopt the first-come-first-serve (FCFS) scheduling policy for all requests, ensuring fairness and preventing starvation.

We show the general applicability of vLLM on them in this section.

We evaluate the performance of vLLM with basic sampling (one sample per request) on three models and two datasets.

In this section, we evaluate the performance of vLLM under a variety of workloads.

To implement a chatbot, we let the model generate a response by concatenating the chatting history and the last user query into a prompt.

We explore the effectiveness of vLLM for the case a prefix is shared among different input prompts

This paper proposes PagedAttention, a new attention algorithm that allows attention keys and values to be stored in non-contiguous paged memory

Model serving has been an active area of research in recent years, with numerous systems proposed to tackle diverse aspects of deep learning model deployment.

The idea of virtual memory and paging is effective for managing the KV cache in LLM serving because the workload requires dynamic memory allocation

In this section, we study various aspects of vLLM and evaluate the design choices we make with ablation experiments.