Before diving into vLLMs it is key to understand how LLM models and their memory system works as we know them. At the core of LLMs lies a Transformers model, which uses an auto regressive decoder to generate one token at a time based on the input (or the prompt) and the previous sequence of the output’s tokens it has generated so far. For each request token generation is performed until the model returns a termination token. The paper asserts that this sequence makes the process memory-bound, which underutilizes the computational capabilities of GPUs and limits the serving throughput. Above half of the memory is taken by model weights which are static and a smaller portion is taken up by dynamic states of the requests. In Transformers these dynamic states are called key and value tensors associated with the attention mechanism, commonly referred to as KV cache which represents the context from earlier tokens. KV cache is a structure that dynamically grows and shrinks over time based on the number of tokens generated. When not managed efficiently this leads to limited batch size and consequently lower throughput. Generally current systems are hindered by two major inefficiencies; first one being internal and external memory fragmentation, and the second being the inability to exploit opportunities for memory sharing.

vLLM addresses these bottlenecks by introducing a novel technique: PagedAttention. Inspired by the concept of virtual
memory with paging inoperating systems, PagedAttention differs from conventional attention mechanisms by enabling non-contiguous storage of key and value tensors. Instead of storing the KV cache as a single continuous chunk, it divides it into fixed-size blocks, each representing a set number of tokens. When performing attention, the PagedAttention kernel efficiently locates and retrieves only the relevant KV blocks needed for computation.

Moreover, vLLM supports popular LLMs such as GPT, OPT, LLaMA of varying sizes, and according to the evaluations done in the paper vLLM improves LLM serving throughput by 2-4× compared to cutting-edge LLMs with no negative affect to model accuracy.
vLLM provides a significant advantage in parallel sampling where multiple output sequences are generated from the same prompt, in which case computation and memory for the prompt can be shared between the output sequences. vLLM can realize this sharing easily and save memory. Similarly, vLLM has led to higher efficiency, throughput and lower latency in beam search through parallel beam processing.