Large-DRAM Sweep: DRAM Size Determines LMCache Effectiveness

1. Experiment Context

1.1 Research Question

Under what conditions does LMCache's CPU KV offload provide net benefit? We test across 4 GPUs and 5 load levels with sufficient DRAM (400 GB on H200/H100, 200 GB on A100, 100 GB on L40S). The initial sweep (64 GB DRAM) showed LMCache consistently slower than FCFS across all tested GPUs and load levels. The large-DRAM sweep tests whether providing adequate DRAM capacity eliminates the bottleneck and reveals a regime where LMCache's offload tier delivers genuine reuse benefit.

1.2 Three Strategies Under Test

1.3 Workload Design

The benchmark simulates an 8-turn agent job: the client sends a prompt, the server generates a short response containing a tool call, the client sleeps for a fixed tool-execution time, then appends a (large) tool output back onto the context and starts the next turn. Each turn grows the prompt by ~2K tokens, so turn 8 sees a prompt of ~13K tokens. Jobs arrive on a Poisson process at rate JPS (jobs per second).

Context growth breakdown across 8 turns (measured)

Each turn's input prompt contains the full conversation so far: original user prompt + all previous turns' responses + all previous tool outputs. Three quantities determine how the context grows:

• Input prompt — cumulative; what vLLM receives as the prefill for that turn
• Completion — fixed 20 tokens/turn via min_tokens=max_tokens=20 in ab_benchmark.py:190. Not variable.
• Tool output — appended between turns by the client. Follows a fixed target length sequence [1640, 1510, 2455, 1335, 2730, 1930, 775] for the 7 tool outputs between turns 1–7. Content is per-job random (seeded by hashlib.md5(f'{job_id}:{turn_idx}') in ab_benchmark.py:78) so length is reproducible but tokens are unique per job → defeats prefix cache sharing.

, about 80 tokens per tool output. The relationship next_input = this_input + 20 + tool holds exactly across all 7 transitions." data-zh="實測 vs target：controlled-sweep H200 JSON 裡的 per_turn_avg_prompt_tokens 顯示 tool output 比 target 序列稍大（穩定多 65–101 tokens）。這是 _generate_random_tool_output 的 header — f\"[{{job_id}}:turn{{turn_idx}}:{{seed[:8]}}] \" — 的固定 overhead，每個 tool output 約 80 tokens。next_input = this_input + 20 + tool 的關係在所有 7 次 transition 都精確成立。">Measured vs target: the per_turn_avg_prompt_tokens field in the controlled-sweep H200 JSON shows slightly larger tool outputs than the target sequence (+65 to +101 tokens consistently). This is the fixed overhead of the _generate_random_tool_output header — f"[{{job_id}}:turn{{turn_idx}}:{{seed[:8]}}] ", about 80 tokens per tool output. The relationship next_input = this_input + 20 + tool holds exactly across all 7 transitions.

1.4 Why Random Prompts Are Load-Bearing

The fix (committed in ab_benchmark.py:70) uses per-job per-turn seeded randomness:

def _generate_random_tool_output(job_id: str, turn_idx: int,
                                 target_tokens: int) -> str:
    # Uses a seeded RNG per (job_id, turn_idx) for reproducibility.
    seed = hashlib.md5(f"{job_id}:{turn_idx}".encode()).hexdigest()
    rng = random.Random(seed)
    header = f"[{job_id}:turn{turn_idx}:{seed[:8]}] "
    ...  # random words from a ~80-word pool

Each (job, turn) combination gets a unique token sequence. vLLM's prefix cache can still match the first few tokens of system prompt but cannot share tool-output bodies across jobs. All results in Sections 2–6 use this randomized prompt.

1.5 Benchmark Configuration

All numbers in Sections 2–6 of this report come from the primary large-DRAM sweep. Historical experiments are documented separately.

1.6 KV Cache Budget per GPU

For Llama 3.1 8B bf16 (~15 GB weights) at 85% gpu-memory-utilization. Block size = 2 MiB × 16 tokens. These determine the saturation point for each GPU.

1.7 The DRAM Hypothesis

1.8 Data Provenance

Canonical documents: BOTTLENECK_REPORT.md (bottleneck ablation analysis), EXPERIMENTS.md (initial sweep, the "definitive experiment" per author note). The older experiments.md (lowercase) contains the retracted Exp 4 and should be treated as historical only.

2. Large-DRAM Sweep: Four-GPU JPS Comparison

The large-DRAM sweep covers JPS from 1 to 15 across four GPU types with large DRAM provisioned (400 GB on H200/H100, 200 GB on A100, 100 GB on L40S), measuring average job completion time (JCT). The key question is whether sufficient DRAM shifts the regime where LMCache is competitive. All data below is from our_experiments/results/20260412_large_dram_sweep/, seed=42.

H200 (141 GB VRAM, 400 GB DRAM)

H100 (80 GB VRAM, 400 GB DRAM)

A100 (40 GB VRAM, 200 GB DRAM) — partial data

L40S (48 GB VRAM, 100 GB DRAM) — partial data

Large-DRAM Sweep: Cross-GPU Summary

3. DRAM Impact Analysis

The large-DRAM sweep results reveal a VRAM-dependent benefit window for LMCache. This section explains the mechanism: why DRAM size matters, why it is not sufficient at very high concurrency, and how PCIe bandwidth creates a hard ceiling.

3.1 Why VRAM Pressure Is the Key Variable

LMCache offloads evicted KV blocks from GPU VRAM to CPU DRAM via PCIe. Whether this offload is beneficial depends on three interacting factors: (1) DRAM capacity, (2) DRAM LRU eviction under concurrency, and (3) PCIe bandwidth saturation.

3.2 PCIe Bandwidth as a Hard Ceiling

At very high JPS (≥15), PCIe becomes the binding constraint regardless of DRAM size. As the number of concurrent jobs grows, each vLLM step encounters more prefill requests in the batch, each triggering a separate wait_for_save DMA. The cumulative PCIe bandwidth consumed exceeds available PCIe throughput (H100: Gen5 x16 ≈ 128 GB/s; A100: Gen4 x16 ≈ 64 GB/s), causing DMA latency to spike. The measured consequence:

4. Per-Turn Latency

All data below use the large-DRAM sweep results. H200 and H100 are shown side by side for each load level. Breaking latency down by conversation turn reveals three structurally distinct failure modes — one per scheduling policy.

4.1 JPS=3 (below saturation)

At JPS=3 on H200, all three policies remain below 4 s per turn. Continuum is the fastest throughout (933 ms at T1, plateauing near 2.5 s), LMCache occupies the middle band, and FCFS is the slowest. VRAM pressure is minimal at this load, so LMCache's offload overhead is visible but small. On H100, the picture reverses: LMCache is the fastest policy, starting at 1420 ms and holding a flat ~3 s plateau from T3 onwards. FCFS grows linearly to 10.5 s by T7, revealing the queuing penalty. Continuum shows a spike at T1 (5952 ms) driven by pin-allocation overhead, then drops to 3–4 s — still above LMCache. H100's smaller VRAM (80 GB) creates enough eviction pressure at JPS=3 that LMCache's DRAM offload pays off, a dynamic absent on H200.

4.2 JPS=6 (saturation threshold)

At the saturation threshold the three shapes diverge structurally. On H200, LMCache (3.6–9.7 s range) significantly beats FCFS (6.8–34.3 s at T7). Continuum has its characteristic T1 spike (17.9 s) then drops to 4.6–7.7 s — the flattest tail. LMCache provides the lowest variance T1–T8 curve. On H100 the same pattern is amplified: FCFS reaches 45 s by T7 while LMCache stays 4.2–9.9 s. Continuum has a dramatic T2 spike (34.5 s — first full prefill competing for pinned VRAM slots) then drops to 5–18 s. The PCIe bus is not yet saturated at JPS=6 on either GPU, so LMCache's offload tier delivers meaningful savings.

4.3 JPS=10 (high load)

At JPS=10, Continuum's pinned-VRAM advantage fully materialises. T3–T8 on both GPUs hold at 5–12.5 s while FCFS climbs to 72–88 s. The T2 spike grows to 54.9 s (H200) and 85.9 s (H100) — reflecting high admission contention for a smaller pinned pool. LMCache (7.5–26.8 s on H200) sits between Continuum and FCFS but shows acceleration at later turns: the PCIe bus is beginning to back up, and each turn's KV reload competes with concurrent onloads from other jobs. FCFS and LMCache are nearly parallel lines growing at the same rate, confirming that LMCache's benefit at this load level is only at early turns.

4.4 JPS=15 (fully saturated)

At full saturation, LMCache collapses. On H200, LMCache T5–T8 (106–149 s) exceeds FCFS T5–T8 (104–137 s) — a qualitative reversal. PCIe bus saturation makes KV onload itself the new bottleneck: each turn's reload queues behind concurrent onloads from other jobs, so LMCache becomes strictly worse than doing nothing. H100 mirrors the H200 pattern exactly: LMCache T7 reaches 152.9 s while FCFS T7 is 150.8 s. Continuum's T2 reaches 110.6 s (H200) and 155.5 s (H100), but T3–T8 remain at 5–14 s across both GPUs. Continuum's pinned-KV guarantee means T3–T8 are entirely decoupled from system load; no other policy achieves this.

4.5 Failure Mode Summary

The per-turn latency data across all four JPS levels identifies three structurally distinct failure modes — each produced by a different mechanism and visible in a different part of the latency curve.

5. Intra-Job Latency Spread

The T8/T1 ratio measures how much a job's own later turns are penalised relative to its first turn — a proxy for intra-job starvation. Unlike inter-job starvation (early vs. late arrivals), intra-job spread directly determines whether a multi-turn user experience degrades over the course of a conversation.

6. Per-Job KV Cache Lifecycle (Scheduler Trace)

Using vLLM's per-step scheduler trace (SCHED_TRACE_PATH), we can track block-level KV cache behavior for individual jobs. The table below shows job_0020 from the H100 LMCache JPS=6 trace test (255 jobs, seed=42, 400 GB DRAM).

6.1 job_0020 KV Cache Trace (H100, JPS=6)

Theoretical best = (prev turn prompt + 20) / this turn prompt. All values use job_0020's actual counts.

6.2 Interpretation

6.3 KV Cache Operations per Scheduler Step

Each vLLM scheduler step processes a batch of requests. The KV cache operations differ by request phase and scheduling policy.

Block states

Cache hit detection — two-tier lookup

When a new request enters schedule(), the scheduler performs a two-tier cache lookup to determine how many tokens can be skipped.

During schedule() — block allocation

After forward pass — request completion

When a request finishes, _free_request() dispatches based on policy:

During tool-call gap

6.4 GPU Hardware: Why Async Offload Should Work (But Doesn't)

KV offload/onload uses PCIe DMA (Copy Engine), not GPU compute (SM). They can theoretically run with zero interference.

Current timeline (blocking)

LMCache v0.3.7 (current):

PR #37160 SimpleCPUOffloadConnector:

Full call chain with source locations

// vLLM Step Loop
│
├── schedule()                               // scheduler.py:280
│   ├── Tier 1: kv_cache_manager.get_computed_blocks(req)  // kv_cache_manager.py:153
│   │   └── coordinator.find_longest_cache_hit(block_hashes)
│   │       └── block_pool.get_cached_block(hash)        // single_type_kv_cache_manager.py:252
│   │           chain hash: hash(parent_hash, token_ids, extra_keys)  // kv_cache_utils.py:539
│   │
│   ├── Tier 2 (LMCache only):
│   │   └── connector.get_num_new_matched_tokens()      // lmcache_connector.py:158
│   │       └── lmcache_engine.get_num_new_matched_tokens  // vllm_v1_adapter.py:1123
│   │
│   ├── allocate_slots()                              // kv_cache_manager.py:196
│   └── _free_request() → _connector_finished()       // scheduler.py:1481, 1657
│       └── _free_blocks()                             // scheduler.py:1514
│           ├── FCFS:      kv_cache_manager.free(req)
│           ├── LMCache:   connector.request_finished() + free
│           └── Continuum: pin_request(req, ttl)         // scheduler.py:240
│
├── execute_model()                             // gpu_model_runner.py:2000
│   └── with maybe_get_kv_connector_output():    // kv_connector_model_runner_mixin.py:97
│       │
│       ├── ENTER: start_load_kv()                // onload from DRAM (async, load_stream)
│       │
│       ├── model(input_ids, positions, ...)     // forward pass (default stream, SM)
│       │   GPU simultaneously: SM=forward, CE=onload DMA
│       │
│       └── EXIT: wait_for_save()  ⚠ CPU BLOCKS HERE
│           └── lmcache_connector.wait_for_save()    // lmcache_connector.py:111
│               └── vllm_v1_adapter.wait_for_save()  // vllm_v1_adapter.py:1019
│                   │
│                   ├── slot_mapping.cuda()          // vllm_v1_adapter.py:1055
│                   │   → PyTorch allocator GC → cudaStreamSynchronize(default)
│                   │   Sync #1: waits for forward pass to finish
│                   │
│                   ├── lmcache_engine.store()       // cache_engine.py:277
│                   │   └── gpu_connector.batched_from_gpu()  // gpu_connector.py:339
│                   │       ├── with torch.cuda.stream(store_stream):
│                   │       │   lmc_ops.multi_layer_kv_transfer()  // gpu_connector.py:279
│                   │       │   GPU→CPU DMA on Copy Engine
│                   │       │
│                   │       └── store_stream.synchronize()  // gpu_connector.py:348
│                   │           Sync #2: waits for DMA to complete
│                   │
│                   └── CPU blocked → cannot launch next step
│
└── // back to step loop → next schedule() + execute_model()

Three synchronization points

Why async overlap fails

The fix: defer synchronization

Pass wait_for_save=False so execute_model returns immediately. Sync in get_finished() after next schedule. See upstream PR #37160.

7. Key Findings