Inside vLLM: KV Cache, Scheduling, and the Model Runner

vLLM is useful to study because it exposes the difference between owning model weights and operating an inference service. The model defines computation; the engine decides how concurrent requests share memory, time, accelerators, and distributed workers.

Architecture at a glance

The API boundary

An API server accepts requests, validates generation parameters, tokenizes inputs, and converts user-facing operations into engine requests. Compatibility at this layer does not make two serving systems operationally equivalent. Queueing, cancellation, streaming, metrics, and failure behavior remain implementation concerns.

The scheduler is a policy engine

Autoregressive generation alternates between prompt processing and repeated decode steps. The scheduler selects work for each engine iteration under token, sequence, memory, and parallel-execution constraints.

Scheduling policy determines who waits, who advances, and whether throughput gains create unacceptable tail latency. A representative load test must therefore include the real distribution of prompt lengths, output lengths, concurrency, cancellation, and service priorities.

KV-cache management is capacity management

Attention reuses keys and values computed for previous tokens. Without caching, decoding would repeatedly recompute the entire prefix. The cache accelerates generation but consumes accelerator memory as active sequences grow.

The original vLLM paper introduced PagedAttention and block-based cache management to reduce fragmentation and enable flexible sharing. That idea remains historically important, but current vLLM architecture should be read through its active KV-cache manager, scheduler, attention backends, and model-runner implementation rather than through the 2023 paper alone.

The model runner owns accelerator execution

The model runner prepares batch state, invokes model computation, coordinates attention metadata and cache tables, captures optimized execution paths where supported, and samples outputs. Current development has emphasized more modular model state, GPU-native bookkeeping, persistent batching, and asynchronous CPU/GPU coordination.

These changes reveal a general systems lesson: once accelerator kernels become fast, host-side orchestration and synchronization can dominate small-model or high-throughput workloads.

Distributed execution multiplies failure modes

Tensor, pipeline, data, and expert parallel strategies distribute different parts of work. They also introduce topology, communication, synchronization, placement, and partial-failure constraints. Scaling out is not a universal latency optimization; it is a capacity and model-fit decision that must include communication cost.

Control path for one request

1. Validate and tokenize the request.
2. Estimate or allocate cache and scheduling resources.
3. Schedule prompt tokens for prefill.
4. Execute model layers and populate cache blocks.
5. Sample a next token under the requested policy.
6. Reschedule active sequences for further decoding.
7. Stream tokens, handle cancellation, and release resources.
8. Emit metrics that distinguish queue, prefill, decode, and end-to-end latency.

What to measure

| Measurement | Decision it supports | |---|---| | Time to first token | Interactive responsiveness and queue health | | Inter-token latency | Perceived streaming smoothness | | Request throughput | Capacity and cost under defined workload | | Cache utilization | Memory pressure and concurrency headroom | | Prefix-cache hit behavior | Benefit of repeated stable prefixes | | Cancellation release time | Whether abandoned work returns capacity quickly | | Tail latency by prompt/output bucket | Fairness and service-objective risk |

Failure modes

• Benchmarking only short, uniform requests hides fragmentation and queue behavior.
• Maximizing aggregate throughput can starve latency-sensitive work.
• Treating an OpenAI-compatible endpoint as a complete compatibility guarantee misses semantics and operational controls.
• Upgrading without a pinned workload regression suite can change supported models, kernels, scheduling, or memory behavior.
• Exposing arbitrary model loading or unsafe repository code creates a supply-chain boundary, not merely a configuration option.

Architectural takeaway

An inference engine is a resource-allocation system wrapped around model computation. Select and operate it by pinning the version, defining representative workloads, tracing the complete control path, and measuring outcome-level service objectives rather than repeating headline throughput claims.