Tenure

1. Why agent sessions turned serving memory into a lifecycle problem

A chat request arrives, computes, and leaves. An agent session stays. The single change that reorganizes the whole memory story is that agents replaced the request with a long-lived loop: the model reads its entire accumulated context, emits a short action, waits out a tool call for seconds to minutes, appends the result, and repeats for tens of turns. This section walks a reader who has never studied KV-cache management into why that shift makes the serving stack's inherited memory hierarchy fail before introducing anything the paper builds.

1.1 What changed: the workload since 2024

Serving systems built before 2024 assumed each request was a single forward pass whose KV cache is allocated on arrival and freed on completion. The traffic that now dominates violates that assumption structurally, and the violation is measurable rather than rhetorical. Across 30 real SWE-agent sessions (609 consecutive turn pairs), 100% of turns extend the previous turn's token sequence exactly — the append-only fraction is 1.0 — and the corpus reads its context 230 times for every token it writes (median 30 output tokens against a 6,945-token input). A session therefore holds a monotone, will-be-re-read object: bytes written once stay valid for the session's lifetime, and the only question the serving system faces is whether those bytes are resident when the next turn arrives.

1.2 The bad scenario, at request granularity

Consider one SWE-agent session on a pressured GPU. Turn 1 builds a 7K-token prefix and emits a patch attempt; the agent then calls a build-and-test tool that runs for roughly six seconds. During that window the session holds no compute, yet its 1.3–1.7 GB of KV state still occupies scarce GPU memory. Because dozens of other sessions are live and arriving, the engine must reclaim space, and the idle session's prefix is the obvious victim. If it has been spilled to the DRAM tier and then evicted from that tier by the time the tool returns, turn 2 recomputes the entire 7K-token prefix from scratch. The user waits through a time-to-first-token spike that has nothing to do with the model and everything to do with which bytes the cache chose to keep.

1.3 The status-quo pain, in one number

The baseline is not merely imperfect; under this pressure it is inert. Replaying real agent trajectories against stock vLLM's CPU-offload path, LRU offloading returns an external hit rate of 0.1–0.3%: the DRAM tier almost never serves a block back before that block is needed again. Because contexts are append-only and re-read every turn, losing residency compounds — a session that re-prefills its full context every turn pays $11.3\times$ the prefill tokens of one whose residency is preserved (per-session median $10.6\times$, maximum $18.6\times$). That factor is the analytic ceiling on what any residency policy can recover, and stock offloading recovers essentially none of it.

1.4 The obvious fix, and why it fails

The natural response is to keep the tier but make eviction smarter: rank blocks by observed idleness, or by a utility score that trades recency against the compute a block would save, or pin each session's blocks for a hinted time-to-live. Each is a published mechanism and each was reimplemented faithfully here through the same serving path. None makes the tier useful. The structural reason is that all four are eviction-only, and eviction cannot manufacture the one thing agent working sets demand, which is capacity headroom: when the aggregate live working set exceeds the budget, a better ordering of which block to drop still drops a block the owner will re-read, because everything ages out together. The idle windows of 6–16 s exceed any recency-based horizon, heavy-tailed real gaps make observed idleness indistinguishable from death, and pins are force-evicted or expire the moment pressure rises.

1.5 The fundamental obstacle

Underneath the failing fixes is a single obstacle: at realistic arrival rates the aggregate working set is several times any single GPU's KV budget, so some sessions' state must be displaced at all times, and the tier's problem is not which resident block to rank highest but whether the resident set is protected from the incoming stream at all. An eviction policy answers a ranking question; the workload poses an admission question. Recasting the tier's decision from ‘evict best’ to ‘refuse worst’ is the pivot the rest of this page develops and measures.

1.6 Position relative to immediate prior work

Four lines of work touch this problem, and each leaves the admission question unanswered. The table states what each carried and what it left on the table; Tenure's delta is the mechanism the last column exposes.

1.7 What this deep dive shows

Only now, after the pain is established, does the solution appear. The claim is two-sided and the data separates the two sides. A capacity-aware admission gate is the necessary substrate: with every lifecycle signal blinded it already beats every baseline, while no eviction-only baseline separates from running no tier at all. On top of that substrate, one trivially-available bit — the session's own end declaration — converts the protected headroom into most of the remaining win, and the two compose superadditively. Gap prediction, which the natural design would reach for, turns out to be unnecessary: absent gap hints perform within noise of exact ones.

2. The agent-session memory problem, quantified

Before proposing a manager, the paper measures the workload it must serve, because the design choices only make sense against these numbers. Two properties are load-bearing, and a third turns them into a cost.

2.1 P1 — contexts are strictly append-only

Across 609 consecutive turn pairs, every turn's token sequence is exactly the previous turn's sequence plus new suffix, giving an append-only fraction of 1.0. A session's KV cache is therefore a monotone object rather than a mutable buffer: once written, a block remains a valid cache entry for as long as the session lives. This is why residency, not correctness, is the entire game — there is never a stale block, only a missing one.

2.2 P2 — turns are separated by idle windows, and the pressure that follows

Real tool gaps have a median of 2.4 s and a p99 of 343 s in a production Claude-Code corpus (2,817 steps); in the replay model the per-session cold window is p50 6.0 s and p99 15.8 s. During each window the session holds no compute but its working set — 1.3–1.7 GB per SWE session at Llama-3.1-8B/fp16, and up to about 28 GB at the p99 of real Claude-Code contexts (229K tokens) — either stays resident or must be rebuilt. Multiplied across concurrent sessions this produces the pressure that defines the regime: replaying the 100-session SWE corpus under Poisson arrivals, at 0.5 sessions/s the p95 live-session count is 53 and the p95 aggregate working set is 41.9 GB; at 1.0 s the corresponding figures are 90 sessions and 71.5 GB — three to five times the combined VRAM-KV and DRAM budget of the evaluation platform. Some sessions' state must be displaced at all times; the policy question is which, and in what representation.

2.3 The stakes: an 11× thrash tax

Because contexts are append-only (P1) and re-read across idle windows (P2), the cost of losing residency compounds rather than adding. Summed over the corpus, a session that re-prefills its full context on every turn processes $11.3\times$ the prefill tokens of one whose residency is preserved (per-session median $10.6\times$, maximum $18.6\times$). This factor is simultaneously the motivation and the measuring stick: it is the analytic ceiling on what residency management can recover, and §3–§5 report how much of it each mechanism actually captures.

3. The decorative tier: a seven-policy ladder

The central empirical result is a ladder in which four faithfully-reimplemented published mechanisms are lined up against no-tier and against Tenure through a byte-identical serving path, so the only difference between rows is policy. On real SWE-agent replay (H100, 8 GB DRAM tier, pooled over 3 seeds, 5,371–5,373 turns per cell), every eviction-only baseline lands within noise of running no DRAM tier at all.

Tenure is CI-separated from every baseline under both per-turn and session-clustered bootstrap. The tags-off ablation is separated from all baselines except Continuum-TTL, with which it overlaps marginally. On the heavier-pressure A100-80 the ordering is identical (Tenure 57.9 s vs 60.7–64.7 s, +10.1% vs LRU on p95, +12.0% on mean JCT).

3.1 Reading 1 — the tier is worthless under every baseline

Because paired conditions share each seed's arrival and gap schedule, the sharpest comparison is a paired-delta bootstrap of each baseline against the no-tier control, which cancels the shared arrival variance. Paired this way, LRU, Marconi, and MORI all straddle zero (−0.18 [−0.41, +0.11], −0.25 [−0.46, +0.02], and −0.08 [−0.32, +0.18] s), and TTL buys only −0.46 s [−0.70, −0.24] of Tenure's 2.71 s potential. The DRAM tier's value for agents is created almost entirely by the admission mechanism and its one-bit signal, not by the hardware.

3.2 Reading 2 — the substrate requires refusal, not signals

Continuum-TTL is the decisive control here because it receives byte-identical lifecycle hints to Tenure's and still loses 17.6% while hitting 1.0%. The mechanism, not the information, is the bottleneck: pins are force-evicted or expire under pressure, whereas refusal-based admission prevents the pressure from reaching the resident set in the first place. Signal value is conditional on the substrate; the substrate is not conditional on the signals.

3.3 The byte economy makes the mechanism visible

The policy-internal counters explain the ladder mechanically rather than by assertion. Over a 3-seed cell LRU writes 4,441 GB to the tier and reloads 17.6 GB — a 250:1 write-to-reuse ratio that is pure churn — while the gate stores 166 GB and reloads 819 GB, each admitted byte reused about five times; TTL matches LRU's churn at 4,450 GB written for 49 GB reloaded. The refusals cost nothing downstream: end-to-end throughput rises under the gate (3.71 vs 3.37–3.47 turns/s for the baselines and no-tier). Tenure's own eviction counters confirm the classes do the work — finished 16.2–20.8K, overdue 0.1–1.0K, far-wakeup 1.4–4.8K, plain-recency exactly zero — against 87.4–87.9K stores refused at the gate.

4. Tenure: one capacity gate, four currencies

The characterization reduces agent memory management to a single recurring decision: when a session's turn ends, what happens to its monotone, will-be-re-read state under a finite budget? Tenure answers with one capacity gate that owns four substitutable currencies rather than four separate subsystems. The name is literal: the gate grants a session's state tenure in the tier for exactly its useful lifetime.

4.1 The gate, concretely

The whole policy is a few lines of decision logic on the store, evict, and finish paths. On a store, the gate sums resident bytes over live sessions and refuses the incoming blocks if admitting them would exceed capacity; otherwise it admits and tags them with the session, its expected gap, and its last-turn bit. On an eviction it takes victims strictly in lifecycle-class order. On a finish it reclaims a session's state the instant the client declares the last turn.

on store(request r, blocks B):
  ws := sum over sessions s alive:      # active working sets
          resident_bytes(s)
  if ws + bytes(B) > capacity:
      refuse(B)                         # anti-thrash admission
  else: admit(B); tag(B, r.session, r.expected_gap, r.last_turn)

on evict(need n):
  order: finished(session ended)        # class 0: dead state first
       > overdue(now > wakeup+slack)    # class 1: most-overdue first
       > far-wakeup(largest gap left)   # class 2
  never: plain recency                  # measured: 0 such evictions

on request_finished(r):
  if r.last_turn: mark_finished(r.session)   # instant reclaim

Three properties follow directly from these lines. Stability: admitted working sets are never churned by arrivals, because refusals absorb overload, so a resident session survives the 6–16 s window that defeats recency. Aggression where it is free: dead sessions' state is reclaimed the moment the client declares its last turn, which accounts for the largest eviction class at zero cost. Graceful degradation: with the tags blinded the classes collapse to idle-order but the admission invariant still holds, which is exactly the measured tags-off ablation — still 4–6% better than every baseline even with no lifecycle information at all.

5. Attribution: what actually carries the win

An honest systems result must say which component earns the gain, so the win is decomposed by knocking each component out, three seeds per cell, in absolute p95 seconds against LRU. The total win is 2.71 s; the decomposition assigns it two ways and finds the two composing superadditively.

5.1 The superadditive split

The gate alone (every lifecycle signal blinded) recovers 0.88 s [0.67, 1.08], which is 33% of the total. The last_turn signal alone (LRU plus reclaim-on-finish, no gate) recovers 0.59 s, which is 22%. Applied together they recover 2.62 s, or 97% — strictly more than the 1.47 s their independent contributions would sum to. The two are complementary by construction: reclaim creates space that only the gate can defend, and the gate protects state that only reclaim can free fast enough. Exact expected-gap hints add the final 0.08 s, within noise. Under the heavier A100 pressure the split shifts to 58% gate / 42% signal, so neither term is second-order; which one dominates depends on how hard the GPU is pressed.

5.2 No oracle needed: the signal requirement is one error-tolerant bit

Because the replay's expected-gap hint equals the actual upcoming gap, the natural objection is that Tenure relies on an oracle; the paper attacks its own signal to eliminate that objection. Perturbing the gap hint by ±25% or ±50%, randomizing it uniformly, dropping half the hints, or removing gap hints entirely moves p95 only within 10.91–11.79 s against the 11.42 s exact point, with hit rates holding at 17.6–18.5% throughout; with no gap information at all (11.50 s) Tenure still beats every baseline by at least 17%. What matters instead is the last_turn bit: blinding it collapses the win to the gate-only level (12.94 s). Yet the bit itself tolerates abuse — if half the sessions never signal their end, p95 is unchanged (11.43 s), and injecting a spurious end signal mid-session costs nothing either (11.12 s). The real signal requirement is therefore a single, error-tolerant bit every agent client owns by construction — ‘this session is done’ — plus session identity.

5.3 Per-session fairness

A tail-latency win can hide a regressive redistribution, so each session is paired with itself under LRU at the same seed and arrival slot. On both GPUs 100% of sessions improve their JCT under Tenure (median +14.5% on H100, +11.2% on A100; worst decile +10.5% and +8.9%), and on the harsher TraceLab column 96.3% improve (median +3.1%, worst decile +1.5%). The gate does not purchase its tail win by starving any session class.

6. The operating envelope: where the win lives, and where it closes

A single operating point could be a knife-edge artifact, so the win is swept along the two axes that define the regime — arrival rate and DRAM budget — three seeds per point, every adjacent pair CI-separated. The sweeps also locate the boundary at which the win closes, which the paper reports as honestly as the win itself.

6.1 Arrival rate: value peaks with headroom, degrades gracefully into saturation

Sweeping arrivals from 0.25 to 1.0 sessions/s, Tenure improves p95 over LRU by 43.7 / 19.2 / 11.5 / 10.8%, with hit rates of 14–34% against LRU's 0.25–0.77% throughout. The mechanism's value is largest when there is per-session headroom to protect and narrows toward GPU saturation, where queueing dominates any residency policy — the win never inverts across the swept range, it degrades gracefully.

6.2 DRAM budget: capacity does not substitute for policy

Sweeping the DRAM budget from 4 to 32 GB at fixed arrivals, the gap widens monotonically — +12.5 / 19.2 / 35.1 / 57.8% — because within the pressured range LRU keeps churning the active set even at 0.76× the peak working set (33.8% hit) while the gate converts the same capacity at twice the rate (73.2%). More capacity does not rescue an eviction-only tier; it widens the distance between the policies.

6.3 The boundary is measured, not asserted

At 64 GB, which is at or above the whole working set, LRU achieves full residency: 1.84 s p95 at 90.6% hit, the residency upper bound any policy can reach. There the relationship inverts — the prototype's per-request bookkeeping (6.71 s at the same 90.6% hit) becomes pure overhead once no decision is needed, and a production build should simply bypass the gate with a one-line check when resident bytes fit capacity. The inversion sharpens the scope claim rather than weakening it: Tenure is for the pressured regime that realistic arrival rates produce, which is precisely where eviction-only tiers fail; at loose budgets the design space collapses to full-residency LRU, and the paper says so.

7. Honest negatives and a transfer test

Two of the gate's four currencies do not simply pay off, and the paper reports both as measured rather than dressing them up. A second, independently-collected trace family and a second model class then test whether the positive result is an artifact of one workload or one architecture.

7.1 The precision tier is a measured negative

The physical case for quantize-on-offload is genuine: host bytes fall to 0.35–0.51× fp16, reload PCIe is 0.86×, top-1 and top-5 outputs are preserved, and on a paired BFCL task-success floor fp8 costs only 1.25–1.75 pt (38.75/39.25% versus a 40.5% fp16 baseline, n=400, Wilson CIs overlapping) — a delta that sits inside the engine's own concurrency-nondeterminism floor. The end-to-end system result is nevertheless negative, and it is reported as measured: routing offloads through the compressed side-region raises p95 to 17.07 s at fp16 path-control and 21.71 s with quantization active, against 11.42 s for the plain gate. The failure is mechanistic, not noise — shrinking DRAM from 8 to 4 GB improves the codesign conditions (17.07→15.24, 21.71→19.09) because fewer admitted bytes traverse the codec, the opposite slope of every healthy condition, and it replicates on A100. The capacity currency is physically sound, but the synchronous placement of the codec on the store path spends more latency encoding than it recovers in hits. An asynchronous, DMA-overlapped codec is the obvious next step, and the paper claims nothing for it.

7.2 The gate transfers to a hybrid model, bounded by state granularity

Run unchanged over Falcon-H1-7B, a Mamba-plus-attention hybrid whose SSM states ride the same BlockPool as all-or-nothing blocks, the gate transfers the win: mean JCT 309.8 s [299.5, 319.7] versus 337.6 s [327.0, 347.7] under LRU, a CI-clean +8.2%, with external hit 6.3% against 0.2%. The p95 improvement is 8.4% but its marginal CIs overlap; the paired-delta bootstrap, which cancels the shared arrival-schedule variance, is clean at −3.25 s [−4.17, −2.21]. Two readings follow: the mechanism transfers unchanged across a model class it was not tuned on, and its effectiveness is bounded by state granularity — all-or-nothing SSM blocks cap the achievable hit rate at roughly a third of the transformer column's (6.3% vs 18.5%), which is itself a design datum for hybrid serving stacks.

7.3 A second trace family: real production sessions

Because the SWE column could in principle reflect benchmark-agent idiosyncrasies, the paper replays a second, independently-collected family: 100 real production Claude-Code sessions from the public TraceLab corpus (357K rounds, 43 developers), which are qualitatively harsher — 2–2.6× larger contexts, recorded rather than calibrated per-turn gaps (p50 0.3 s, p99 270 s), and native context compactions preserved by the derivation. The ladder replicates and sharpens: at 0.5 sessions/s Tenure reaches 125.1 s against LRU 128.3 / TTL 128.6 / no-tier 127.7, with hit rates 5.8% versus at most 0.31%. The sharpening is that under saturation the tier becomes a net negative for the baselines — paired against no-tier, LRU is +0.59 s [+0.34, +1.02] slower and TTL +0.92 s [+0.59, +1.26], the transfer cost of moving bytes for about 0.2% hits. Both scaling laws transfer: lightening arrivals grows the win +2.4→+7.5%, and scaling DRAM to this workload's session size grows it +3.1→+5.7%. The absolute win at the saturated default point is modest, but every mechanism-level conclusion reproduces on real production traces with real gaps.

8. How it maps onto stock vLLM: three seams, zero fork

Tenure is about 1.2 kLoC of Python deployed as an out-of-tree plugin against stock vLLM v0.23 — no fork and no engine patch — through three public seams. Deploying without an engine fork is what makes the ladder an apples-to-apples mechanism comparison: every baseline loads through the same seam, so the serving path is byte-identical and the only diff is policy.

8.1 The CachePolicy seam (~300 LoC of policy)

vLLM's CPU-offload manager delegates admission and eviction to a pluggable CachePolicy. Tenure implements the gate here: on store it sums resident bytes over live sessions and refuses blocks that would exceed capacity; on evict it takes victims in lifecycle-class order. The baselines — Marconi-utility, Continuum-TTL, and an idleness-rank proxy inspired by MORI — are implemented through the same seam with explicit, swept parameters, which is what makes the comparison fair rather than favorable.

8.2 The connector seam (lifecycle hints)

Lifecycle hints ride the OpenAI-compatible API's kv_transfer_params passthrough: clients annotate each request with session_id, expected_gap_ms, and last_turn, and the OffloadingConnector hands them to the policy without engine changes. Hints are advisory — sessions that omit them are simply untagged, which is the measured tags-off ablation and degrades gracefully rather than catastrophically.

8.3 The side-region codec (for the precision study)

A compressed side-region replaces the fixed-stride pinned pool: blocks are quantized (fp8/int4, scale-per-block) at store and dequantized at reload. The physical path works, but its synchronous placement on the store path is exactly what §7 prices as a measured negative. Hybrid models need no additional code — vLLM's offload path already moves Mamba state blocks through the same BlockPool, so the gate manages them as all-or-nothing units unchanged.

9. Status: an active project, iterating

This page documents an active, pre-acceptance project rather than a published result, and the review status is reported in that spirit. The evaluation core is complete — two GPUs, seven policies, three seeds, two independent trace families, a hybrid-model column, and the attribution ablations — and every headline separation survives per-turn, session-clustered, and paired-delta bootstrap treatments. What is still iterating is the reviewing.

A warm internal review placed the draft at 7/10. The binding gate for this line of work is a deliberately harsher cold ‘nightmare’ review, in which an adversarial reviewer is given only the files and the data with zero framing and asked to find every flaw and verify each headline number. Across three cold rounds that gate has scored 4 → 5 → 4, and each round has retired concrete blockers rather than cosmetic ones: round 1 reframed the mechanism attribution (from ‘tags second-order’ to the two-term gate-plus-signal decomposition this page presents) and added the hint-noise ablations; round 2 added an admission-capable TTL baseline, a residency upper bound, and more seeds; round 3 requested signal-matched baselines and per-class fairness, which are the wave-3 ablations now in flight. The reviewer has verified every headline number against the raw JSONs in each round; the open items are baseline breadth and statistical phrasing, not arithmetic.

10. Reproduce

Every table and CI on this page derives from one analyzer command over the raw per-turn JSON snapshots; there are no hand-transcribed numbers in the pipeline. The analyzer emits per-turn and session-clustered bootstrap CIs, the policy counters, and every table, into a single source-of-truth text file.

# regenerate every table, CI, and policy counter from raw per-turn JSON snapshots
python results/mainline/analyze_mainline.py --boot 10000
# -> results/mainline/ANALYSIS.txt   (single source of truth for every number above)

# the numbers manifest maps each paper element to its analyzer section + snapshot tree
results/mainline/MANIFEST.md