vLLM KV Offloading — The v0.19→v0.23 Rewrite

1. Why a KV-cache offloading subsystem, and why rebuild it in 2026

A serving engine keeps, for every running request, a key–value tensor for each transformer layer — the KV cache.

Why now

The bottleneck this subsystem targets is GPU high-bandwidth memory (HBM).

A concrete failure of GPU-only caching

As an illustration of that waste, consider a request whose 8,000-token prompt has just been processed when concurrency forces the scheduler to preempt it.

The baseline cost being paid

The cost of a cache miss is a full prefill of the missed prefix.

How offloading removes the waste

KV offloading addresses the problem directly: blocks are copied to CPU DRAM before eviction and loaded back instead of recomputed.

The obvious fix, and why it is not enough

The naive remedy is to copy evicted KV blocks to CPU memory and copy them back on a hit.

The rewritten design meets each of these constraints in turn.

Position relative to neighbouring systems

vLLM already had a connector framework for external KV stores; the native OffloadingConnector fills a different gap.

With the pressure, the strawman and its resolution, and the positioning all established, the quantitative stakes can be stated compactly.

2. The rewrite at a glance

The flat v0.16 offloading directory was reorganised over five releases into a layered package whose shape encodes the new capabilities.

Old layout (v0.16, flat) → new layout (current package)

Each row maps one concern to the file that held it in the flat v0.16 layout (middle) and the file or subpackage that owns it now (right).

Current package file tree

The diagram below groups the current vllm/v1/kv_offload package by responsibility before the class table narrows to individual APIs.

A class-to-file map of the subsystem

The map below ties each class to its file under vllm/v1/kv_offload/ and its purpose; these six classes recur throughout this page.

It was incremental, not a single drop

A common recollection is that the rewrite landed in v0.19.0. That description is incomplete.

3. The core API rewrite

The package is built on a few abstract classes in base.py; five changes distinguish the new design.

3.1 OffloadKey — making the cache hybrid-aware

A key identifies one offloaded block; the new OffloadKey packs the KV-cache group index into the key.

# vllm/v1/kv_offload/base.py:31-48 — key = block_hash + 4-byte big-endian group index
OffloadKey = NewType("OffloadKey", bytes)

def make_offload_key(block_hash: bytes, group_idx: int) -> OffloadKey:
    return OffloadKey(block_hash + group_idx.to_bytes(4, "big", signed=False))

def get_offload_block_hash(key: OffloadKey) -> bytes: return key[:-4]
def get_offload_group_idx(key: OffloadKey) -> int: return int.from_bytes(key[-4:], "big")

Encoding the group index as appended bytes avoids per-key tuple allocation on the scheduler hot path.

3.2 Single-block lookup returning a tri-state

Lookup now takes one key and returns True / False / None, where None means a transfer is in flight.

# vllm/v1/kv_offload/base.py:151 (abstract) · cpu/manager.py:115 (CPU impl)
def lookup(self, key: OffloadKey, req_context: ReqContext) -> bool | None:
    block = self._policy.get(key)
    if block is None:        return False     # absent
    if not block.is_ready: return None      # write in-flight; caller should retry
    return True

The retry state is intentionally conservative: returning None preserves the cache hit without letting the scheduler consume a CPU slot that is still being filled. It also prevents a secondary-tier promotion from racing a second miss path for the same key.

3.3 ReqContext and the per-request OffloadPolicy

Every manager method now receives a ReqContext, and on_new_request declares a per-request OffloadPolicy.

# vllm/v1/kv_offload/base.py:51-68
@dataclass
class ReqContext:
    req_id: str
    kv_transfer_params: dict | None = None

class OffloadPolicy(Enum):
    BLOCK_LEVEL   = "block_level"    # offload only newly-computed blocks
    REQUEST_LEVEL = "request_level"  # offload the request's complete context

3.4 CanonicalKVCaches — one layout abstraction instead of backend probing

CanonicalKVCaches presents every KV cache as int8 (num_blocks, page_size_bytes) tensors, so the worker is architecture-agnostic.

3.5 Metadata-driven metrics

A spec declares its metrics via build_metric_definitions; full Prometheus migration (RFC #44008) is the one part still finishing.

4. The CPU primary tier and pluggable eviction

The CPU tier's manager owns shared logic and delegates the eviction decision to a pluggable CachePolicy.

The reference-count lifecycle

A block's whole readiness/pinning protocol is one integer: -1 not-ready, 0 evictable, >0 pinned.

4.1 LRU — the baseline policy

LRU keeps blocks in one ordered dict and evicts the least-recently-used; its eviction is atomic.

# vllm/v1/kv_offload/cpu/policies/lru.py:42-57 (condensed)
def evict(self, n: int, protected: set) -> list | None:
    candidates = []
    for key, block in self.blocks.items():           # oldest first
        if block.ref_cnt == 0 and key not in protected:
            candidates.append((key, block))
            if len(candidates) == n: break
    if len(candidates) < n: return None           # atomic: nothing mutated
    for key, _ in candidates: del self.blocks[key]
    return candidates

The important edge case is failed eviction: the method returns None before mutating the ordered dictionary, so a partially available candidate set cannot corrupt policy state. Protected and pinned blocks therefore remain stable across the whole scheduler step.

4.2 ARC — adapting between recency and frequency

ARC tracks frequency as well as recency via T1/T2 lists and B1/B2 ghost lists, self-tuning a target split.

Concretely, the policy maintains a target size for T1 and adjusts it on every ghost-list hit:

Here $p$ is the target size of $T_1$ and $c$ the capacity; the step adapts to the relative ghost-list lengths.

The adaptation rule appears in the source as a small branch on each ghost-list hit; the excerpt below condenses the hit handler to that branch.

# vllm/v1/kv_offload/cpu/policies/arc.py:88-104 (condensed)
def _on_ghost_hit(self, key: OffloadKey) -> None:
    if key in self.b1:                          # recency under-weighted
        delta = max(1, len(self.b2) // max(1, len(self.b1)))
        self.p = min(self.p + delta, self.capacity)
    else:                                      # key in b2: frequency under-weighted
        delta = max(1, len(self.b1) // max(1, len(self.b2)))
        self.p = max(self.p - delta, 0)
    # ... omitted: move the key into T2 and trim a real list to honour p ...

The code makes the clamping explicit: p stays in [0, c] on every update, and the denominators are floored at one; eviction itself remains the atomic LRU-style scan, applied within whichever list p now favours.

The transfer engine

Transfers use swap_blocks_batch (cuMemcpyBatchAsync); GPU→CPU stores wait on the compute stream to avoid reading unfinished KV.

5. The tiering subsystem — offloading beyond CPU

Multi-tier offloading rests on one rule: a secondary tier may never touch GPU memory; all transfers stage through CPU.

That rule turns every store into a cascade and every load into a promotion, both staged through CPU:

5.1 The SecondaryTierManager contract

A secondary tier implements lookup / submit_store / submit_load / get_finished_jobs / on_new_request / drain_jobs, all non-blocking, all scheduler-side.

5.2 The three shipped tiers

Three concrete tiers are registered in tiering/factory.py.

Non-blocking lookups via a background thread

Real tiers compose an AsyncLookupManager and implement only batch_lookup; the non-blocking machinery is inherited.

6. PR and version archaeology

The table pins each headline change to a PR, author, and release; the rewrite runs v0.19.0→v0.23.0.

7. Open issues and limitations (June 2026)

The rewrite closed older data-safety bugs but opened higher-level state-machine and contract problems.

The columns are the issue number, a one-line problem statement, and the current status with a suitability judgement (marked by a coloured dot).

No issue carries a good-first-issue label, but #46027/#44888/#45405 are contributor-ready.

8. Contributing: extension points and two worked scaffolds

The rewrite established well-isolated extension points; this lab includes two scaffolds built against current main.

The columns name the extension being added, the interface to implement, and the registration hook that makes it discoverable.

8.1 A GPUDirect-Storage (cuFile) NVMe tier

The scaffold adds a gds tier mirroring the fs tier, routing block I/O through cuFile when enabled, with an O_DIRECT fallback.

8.2 A session-aware eviction policy

The scaffold is a SessionAwareLRUCachePolicy that protects blocks of recently-active sessions, keyed by session_id.

8.3 Contribution mechanics

AGENTS.md mandates gh duplicate-checks, human accountability, AI disclosure, uv+venv, pre-commit, 88-char lines; >500 LOC needs an RFC.