By reducing the queueing delay and the inefficiencies from padding, the fine-grained batching mechanisms significantly increase the throughput of LLM serving.

Once trained, LLMs are often deployed as a conditional generation service (e.g., completion API [34] or chatbot.

In this section, we describe the generation and serving procedures of typical LLMs and the iteration-level scheduling used in LLM serving.

To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems.

As in OS’s virtual memory, vLLM does not require reserving the memory for the maximum possible generated sequence length initially.

The key idea behind vLLM’s memory manager is analogous to the virtual memory [25] in operating systems.

In this work, we develop a new attention algorithm, PagedAttention, and build an LLM serving engine, vLLM, to tackle the challenges outlined in §3

Due to the unpredictable output lengths from the LLM, they statically allocate a chunk of memory for a request based on the request’s maximum possible sequence

In vLLM, we adopt the first-come-first-serve (FCFS) scheduling policy for all requests, ensuring fairness and preventing starvation.

We evaluate the effectiveness of memory sharing in PagedAttention with two popular sampling methods: parallel sampling and beam search.

To implement a chatbot, we let the model generate a response by concatenating the chatting history and the last user query into a prompt.

This paper proposes PagedAttention, a new attention algorithm that allows attention keys and values to be stored in non-contiguous paged memory

In this section, we study various aspects of vLLM and evaluate the design choices we make with ablation experiments.