Continuum-D

1. Why agent workloads break KV offloading

Serving stacks built before 2024 assumed each request was a single forward pass: a prompt arrives, tokens stream out, the request completes, and its KV cache is freed. The agent workloads that dominate 2025 traffic violate that assumption structurally. A single agent job is not one request but a sequence of them — the model proposes an action, an external tool runs for several seconds, the tool result is appended, and the model is called again on the now-longer conversation. Across six such turns a job's prompt grows monotonically, and every turn re-reads the entire prefix built by the turns before it.

The gap between two turns of the same job is the crux. During the tool call the job holds no compute, yet its growing KV prefix still occupies scarce GPU memory. When many jobs are in flight and GPU memory is under pressure, the serving engine must reclaim that space, and the prefix of an idle job is the natural victim. If the prefix has been discarded by the time the tool returns, the next turn recomputes it from scratch — a full prefill over thousands of tokens — and the user waits through a time-to-first-token spike that had nothing to do with the model and everything to do with cache management.

A concrete instance makes the pressure legible. A data-analysis agent is asked to investigate a dataset; it runs six turns, each proposing a query, waiting on an external tool for about five seconds, and appending the result. Its prompt grows by roughly seventeen hundred tokens per turn and reaches about ten thousand by the last. Now place two dozen such agents on one GPU, arriving at about two per second — the mixed, multi-turn shape that agent traffic took on through 2025, where a single conversation spans many calls rather than one. Their combined KV working set is around thirty-one gigabytes, but under memory pressure only about fifteen gigabytes of GPU memory is available to hold it, so at any moment roughly half of the live prefixes cannot stay resident and something must give.

vLLM's native KV offloading, introduced across the v0.19–v0.23 releases, offers a place to put the prefix that is cheaper than recomputing it: a DRAM pool one tier below GPU memory. When GPU memory is reclaimed, the prefix is copied to DRAM rather than dropped; when the next turn arrives, it is copied back. The mechanism is sound, but the policy that decides which DRAM blocks to keep and which to discard when the DRAM pool itself fills is content-blind. The default is least-recently-used, occasionally adaptive replacement — both of which reason only about access recency, and neither of which knows that a block belongs to an agent job scheduled to wake up in two seconds rather than to a job that finished ten seconds ago and will never return.

1.1 Where this sits relative to prior work

Three lines of work touch this problem and each leaves a different piece on the table. Continuum makes GPU-memory retention job-aware via a KV-cache TTL (over 8× JCT) but stays within GPU memory. vLLM native offloading built the DRAM tier and made eviction pluggable, but ships only content-blind LRU/ARC. LMCache offloads at the connector layer, keyed by content hash, with no job notion. None routes agent job structure into the DRAM eviction decision. A wave of very recent concurrent systems attacks closely related pieces; §11 positions Continuum-D against them honestly.

2. The failure, concretely

The abstract claim that LRU is a poor fit becomes concrete once the access pattern is spelled out. Twenty-four agent jobs run concurrently, each growing its prompt over six turns. Turn two re-reads turn one's prefix; turn six re-reads everything from turns one through five. The blocks a job most needs are its oldest blocks — the shared prefix laid down early and referenced by every subsequent turn. This is the exact opposite of the recency assumption LRU is built on.

2.1 What LRU and ARC actually do

Both policies vLLM ships reason purely about access history. Least-recently-used keeps a queue ordered by last touch and, when it must free space, discards whichever block has gone longest without being read; its single assumption is that a block untouched for a while is unlikely to be touched soon. Adaptive replacement is a refinement that keeps two lists, one for blocks seen once and one for blocks seen repeatedly, shifting its budget between them to protect frequently-reused blocks — but it, too, decides entirely from the history of accesses. Neither has any notion of who owns a block or when that owner will next run. On workloads where recency predicts reuse, that is the right blindness; on the growing-prefix pattern above, where the most valuable block is the oldest one, it is precisely the wrong signal, which is why the next subsection's trace ends in a near-total miss.

2.2 A trace through an undersized DRAM pool

Suppose the GPU holds about half the combined working set, so blocks are continually demoted to DRAM, and the DRAM pool is itself smaller than the working set — eight gigabytes against a roughly thirty-one gigabyte KV working set. Each newly written block is most-recently-used, so LRU keeps the newest and evicts the oldest. But the oldest blocks are exactly the shared prefixes the next turn will ask for. When a job wakes and looks up its prefix, the blocks it needs have already been evicted to make room for other jobs' newer blocks, which are evicted before their own jobs wake. The pool churns continuously and satisfies almost no lookups.

2.3 The baseline pain, in numbers

At the tight eight-gigabyte pool, LRU achieved an external cache hit rate of zero. Not low — zero. Every lookup that reached the DRAM tier missed, so the machinery ran, copied blocks back and forth, and recovered nothing. Late-turn TTFT under LRU was 0.0965 s, marginally worse than the 0.0953 s measured with offloading disabled entirely. This reproduces an earlier SWE-smith finding on this project: an undersized DRAM pool is not merely unhelpful but actively harmful, adding transfer cost while returning no hits. The status quo, at this operating point, is worse than doing nothing.

3. Why the obvious fixes fail

A reader is right to ask why the DRAM pool is not simply made larger. The first strawman is exactly that: provision enough DRAM that the whole working set fits and no eviction occurs. The measurements confirm this makes the problem disappear — at a forty-gigabyte pool, larger than the working set, LRU and the job-aware policy perform identically, because with no eviction there is no decision to make. But DRAM is a shared, finite resource contended by co-located tenants; over-provisioning it for every instance is exactly the waste a memory hierarchy exists to avoid. And even an ample pool spends its capacity indiscriminately, holding finished jobs' KV alongside the prefixes of jobs about to wake. Making the pool large hides the policy question; it does not answer it.

The fundamental obstacle is an information gap, not a mechanism gap. The transfer machinery already exists and works; the eviction policy simply operates with strictly less information than the serving layer holds. The scheduler knows every request's job identity, knows from the agent framework whether a turn is the last, and can be told the expected tool-gap duration. The cache policy, one layer down, sees only an opaque stream of block accesses. The optimal eviction — keep the block whose owning job wakes soonest, discard the block whose job has finished — is trivial to state and impossible to execute without the job metadata. Closing the gap is a plumbing problem.

The three numbers below summarize the baseline pain that motivates the design: at the tight pool, the status-quo policy recovers nothing and the pool is a quarter of what the working set would need.

4. The Continuum-D design

Continuum-D closes the gap with three cooperating pieces: a metadata channel carrying job structure from client to cache policy, an eviction policy that spends the DRAM pool on blocks that will be reused, and an admission gate that refuses to let a burst of new stores evict the resident set of jobs about to wake. A fourth, optional piece runs client-side and promotes a job's prefix back to GPU memory shortly before its tool gap ends. Throughout, no core vLLM code is modified — every piece attaches to a seam the engine already exposes.

4.1 The metadata channel

vLLM already forwards an arbitrary per-request dictionary, kv_transfer_params, from the HTTP request body through to the KV connector. Continuum-D uses it to carry four fields describing where a request sits in its agent job. The client attaches them to every turn; nothing in the core engine needs to know they exist.

# attached to every turn's request body, under kv_transfer_params
{
  "job_id":         "job-17",   # which agent job this request belongs to
  "turn_idx":       3,          # position in the multi-turn sequence
  "expected_gap_ms": 5200,       # predicted tool-gap before the next turn
  "last_turn":      false       # true on the final turn; the job becomes finishable
}

These four fields are enough to reconstruct, inside the policy, a per-job record: the DRAM blocks the job owns, the wall-clock time it is expected to wake (now plus expected_gap_ms), and whether it has finished. Two edge cases matter. The first is stale prediction: expected_gap_ms is only a hint, so a job whose predicted wake time has already passed must age out of its protected class rather than pin DRAM indefinitely — this is what the overdue class in the eviction order handles. The second is the reclamation boundary carried by last_turn: once it reads true, every block the job owns becomes a first-choice eviction candidate even if its recency score is high, because a finished job will never read them again.

4.2 The eviction policy: four classes before recency

The core idea is to order eviction candidates by how likely their owning job is to read them again, and fall back to recency only when that signal is absent. When the policy must free blocks, it partitions candidates into four classes and empties them in order, so a block from a lower class is never evicted while a higher class still has blocks to give.

A block from a job waking in two hundred milliseconds sits in class 2 with a near-term wake time and is evicted last; a block from a job that finished ten seconds ago sits in class 0 and is evicted first. This is the exact inversion of LRU's behaviour on this workload, and it is what turns a zero-percent hit rate into a positive one.

4.3 Admission control: refuse thrash before it starts

Ordering evictions well is not enough alone. If a burst of new blocks arrives while the pool is already full of soon-to-wake prefixes, honouring every store would evict even class-2 blocks the policy would rather keep. Before a batch of stores is accepted, the admission gate checks whether admitting them would push the resident set of active jobs below what they need, and refuses the batch if so. A refused store means the block stays put and is recomputed later if needed — a bounded, local cost — whereas admitting it would inflict an unbounded, diffuse cost by evicting a prefix many future turns depend on.

4.4 Predictive warm promotion (client-side)

The policies above keep a job's prefix alive in DRAM, but a lookup that finds its blocks in DRAM still pays to copy them back to GPU memory before the turn runs. The warm-promotion variant hides that copy behind the tool gap. Knowing the expected gap, the client issues a tiny throwaway request — a single generated token over the full prefix — a fixed lead time before the real turn is due. That touches the prefix, triggering its promotion from DRAM back to GPU memory, so the real turn finds the blocks already resident. The promotion overlaps with the tail of the tool call rather than sitting on the next turn's critical path.

5. How it maps onto vLLM v0.23 seams

The design ships without touching core vLLM because the KV-offload rewrite made three things pluggable at exactly the points Continuum-D needs. A late-2024 pull request split the CPU offloading path into a package with a swappable cache-policy registry; later releases added a per-request context object carrying kv_transfer_params down to the policy, and a factory that lets an out-of-tree offloading spec be named at launch. Continuum-D is one subclass at each of these three seams.

Concretely, three small files hold the entire plugin, each subclassing an upstream base. The table below lists them alongside the unmodified vLLM files they attach to, with the local Continuum-D files in red and the upstream vLLM files in green — nothing in green is edited.

At launch the spec is selected purely through configuration — the connector's extra-config dictionary names the module and class to load — so the server starts an unmodified vLLM binary and injects the plugin by path. The engine log confirms the out-of-tree spec is constructed on both the scheduler and worker sides:

# --kv-transfer-config passed to `vllm serve` (unmodified binary)
{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "spec_name":        "JobAwareOffloadingSpec",
    "spec_module_path": "continuum_d.spec",
    "admission_control": true,
    "cpu_bytes_to_use":  8589934592
  }
}

# engine log — the plugin loads inside the engine process
INFO factory.py:51] Creating offloading spec with name: JobAwareOffloadingSpec

Nothing below is a fork of vLLM; every box labelled Continuum-D is a subclass loaded by configuration into an unmodified engine. That is what makes an eventual upstream contribution plausible: the request is not to change core scheduling but to make one more field of already-forwarded metadata visible to the cache policy.

6. Experimental setup

The measurements ran on a single PACE H200 under vLLM v0.23.0, serving Llama-3.1-8B-Instruct in bfloat16. The workload is a synthetic but structurally faithful agent trace: twenty-four jobs arrive as a Poisson process at two jobs per second, each running six turns. A job's prompt grows by roughly seventeen hundred tokens per turn, reaching about ten thousand tokens by the final turn, and every turn re-reads the full prefix. Between turns a job sleeps for a lognormal tool gap averaging about five seconds. The KV working set across all jobs is approximately thirty-one gigabytes.

The single most important knob is GPU memory utilization, set to 0.25, which caps usable KV space on the H200 at roughly fifteen gigabytes — about half the working set. This is the deliberate VRAM-pressure regime in which offloading is supposed to matter. The DRAM pool is swept across two points: an ample forty gigabytes, larger than the working set, and a tight eight gigabytes, about a quarter of it. Every condition served one hundred forty-four requests with zero errors.

7. Results

The table reports, per condition, average and ninetieth-percentile job-completion time, average time-to-first-token, the TTFT averaged over the late turns of each job (turns two onward, where prefix reuse actually matters), and the external DRAM cache hit rate. Bold rows are the tight eight-gigabyte pool, where the policy question is live; the forty-gigabyte rows are the ample-pool control.

↔ scroll the table horizontally on a narrow screen.

The warm-promotion hit rate reads lower than plain job_aware because each warm probe issues its own external query, diluting the ratio; the honest signal for the warm conditions is late-turn TTFT, not hit rate.

7.1 Reading the tight-pool rows

At the eight-gigabyte pool the three offloading conditions separate cleanly. LRU returns a zero-percent hit rate and a late-turn TTFT of 0.0965 s, marginally worse than the 0.0953 s of no offloading — the reproduction of the actively-harmful undersized-pool result. The job-aware policy, with the same eight gigabytes, keeps an 11.7 percent hit rate and cuts late-turn TTFT to 0.0802 s, a seventeen percent reduction relative to LRU. Adding warm promotion brings it to 0.0718 s, a twenty-six percent reduction relative to LRU, recovering most of the ample forty-gigabyte pool's benefit (0.0700 to 0.0748 s) using one-fifth of the DRAM. Job-awareness is the difference between a DRAM tier that helps and one that hurts.

7.2 Reading the ample-pool rows

At forty gigabytes the two policies are indistinguishable, both holding a 22.7 percent hit rate and late-turn times within a millisecond of each other. This is the expected control result: when the pool exceeds the working set no eviction occurs, the eviction policy is never consulted, and the two conditions execute identical code paths. The ample-pool rows anchor the tight-pool comparison — they show the achievable benefit when memory is not the constraint, which is the target the eight-gigabyte job-aware policy gets most of the way back to.

8. Hypothesis verdicts

The experiment was framed around three hypotheses set before the runs. Each is resolved by the table above.

9. Caveats and what this does not yet show

This is a first validating measurement, not a finished evaluation. Every number comes from a single seed, one GPU model, and one load point of two jobs per second. Job-completion time is essentially flat across conditions because at this arrival rate it is dominated by tool gaps rather than queueing, so the effect shows up in time-to-first-token; a heavier arrival rate, where requests actually contend, is where any completion-time benefit would appear and has not been measured. The absolute TTFT deltas are also modest at this scale — around twenty milliseconds — and would widen with longer contexts and harder pressure.

10. Reproduce

The full rig lives under the agent-kvcache project — the out-of-tree plugin, the benchmark driver, and a data-driven analyzer that globs result files so a partial run analyzes cleanly and a backfill is a zero-edit rerun. On PACE the matrix launches with the shared fan-out helper; a single condition is one line.

# one condition of the matrix on an H200 (tight 8 GB pool, job-aware)
./fanout.sh --sbatch continuum_d.sbatch --name cd-ja-d8 --gpus h200 \
  --exports "CONDITION=job_aware,DRAM_GB=8,JOBS=24,TURNS=6,SEG=512,GEN=64,JPS=2,GPU_UTIL=0.25,EXP_TAG=cd" \
  --mem 64G --time 0:50:00

# aggregate all result JSONs into the comparison table (partial-safe)
python3 benchmarks/continuum_d_analyze.py --root results-pace/continuum-d

Two properties of the harness make a stalled cluster queue survivable. The analyzer discovers its inputs by globbing the result directory rather than being handed a fixed list, so a run that produced only three of seven conditions analyzes into a clean partial table and the missing rows simply do not appear; when the remaining jobs land, rerunning the same command with no edits backfills them. The condition identity is also encoded in each result directory name, so a result is attributable even if its metadata file is lost, and two concurrent fan-out jobs never collide.

The design, plugin, benchmark, analyzer, and written report are committed locally on the continuum-d branch of the agent-kvcache repository; the CPU unit tests for the policy run without a GPU against any environment where vLLM 0.20 or newer imports.

11. Recent related work (2025–2026)

The distance between this design and the published literature is unusually short. Multi-turn agent KV management became an active subfield across 2025 and 2026, and by the time these measurements were taken several systems had already attacked pieces that overlap substantially with Continuum-D. Honesty requires stating that overlap plainly rather than claiming a clean gap, so this section positions the design against the closest concurrent work and then narrows the novelty claim to what the measurements here actually support. A literature check was run at the time of writing (2 July 2026); the systems below are the nearest neighbours it surfaced.

Read together, these systems establish that the ideas Continuum-D leans on — using the tool-call gap to move KV between GPU and DRAM, predicting that gap to prefetch before the next turn, and preferring future activation over recency in eviction — are shared with concurrent work and are not claimed here as novel. What remains specific to Continuum-D is narrower, and worth stating precisely. The first part is the eviction decision under an undersized DRAM pool: the four-class order of finished, overdue, far-wake-up, then recency, together with admission control, measured against the finding that plain LRU in that regime returns a zero-percent hit rate and underperforms disabling offloading altogether. The second part is the engineering result that this policy installs as a zero-core-change plugin on vLLM v0.23's pluggable CachePolicy and OffloadingSpec seams, which is the credible path to contributing it upstream. The measurements in this deep dive validate that narrow claim, not a broad one — and the existence of MORI and TokenCake in particular means the tool-gap-offload framing is now common ground, so the honest next step is a direct comparison against them on a shared workload rather than against LRU alone.