Exp: Execution Note

1. Environment

2. Server Launch

# Activate environment
source ~/anaconda3/etc/profile.d/conda.sh && conda activate contextserve

# Start vLLM with Continuum scheduling
RUN_OUTPUT_DIR=./e2e_output \
vllm serve /home/hclin/shared_models/Meta-Llama-3.1-8B-Instruct \
  --scheduling-policy continuum \
  --port 8199 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85

Startup takes ~70 seconds (weight loading + torch.compile). Health check via:

curl http://localhost:8199/health

3. Test Execution

3.1 Test Design

A Python script sends multi-turn agentic requests via the OpenAI client. Each turn provides job_id and is_last_step through the extra_body parameter, which vLLM's protocol layer unpacks into SamplingParams.extra_args.

# Key: how Continuum metadata is passed
resp = client.chat.completions.create(
    model=MODEL,
    messages=[
        {"role": "system", "content": "Respond with ONLY a ```bash``` block."},
        {"role": "user",   "content": "List files with ls."},
    ],
    max_tokens=150,
    extra_body={
        "job_id":       "job_alpha",   # ← Identifies the agent job
        "is_last_step": False,          # ← Tells scheduler to pin after completion
    },
)

3.2 Prompts & LLM Output

2 sequential jobs × 5 turns each. Each turn carries the full conversation history (system prompt + all previous user/assistant/tool messages), simulating a real agent. Simulated tool outputs are injected between turns.

3.3 Server Log — Tool Detection & Pin Chain

Filtered server log showing the complete Continuum pipeline for job_alpha:

All 5 turns correctly parsed. Turns 1–4 triggered set_up_pin → return 2.0. Turn 5 (is_last_step=True) skipped pin and freed blocks.

4. Pin Verification via /metrics

A second test script polls vllm:kv_cache_usage_perc from the Prometheus /metrics endpoint in real-time, before and after each turn, during the simulated tool execution gap.

4.1 Raw Results (5 Turns, Accumulated Context)

# 5 turns with accumulated conversation history + simulated tool outputs
# Turn 1-4: is_last_step=False (should pin)
# Turn 5: is_last_step=True (should free)

[Baseline] KV cache usage: 0.000000

Turn 1/5  last=False  319ms  prompt=66  output=7
  Output: ```bash ls ```
  KV usage: 0.000000 → 0.000876  (delta: +0.000876)
  Simulating tool execution — KV should stay pinned...
    t+0.3s: KV usage = 0.000876
    t+0.6s: KV usage = 0.000876
    t+0.9s: KV usage = 0.000876

Turn 2/5  last=False  115ms  prompt=131 output=9
  Output: ```bash cat main.py ```
  KV usage: 0.000876 → 0.001577  (delta: +0.000701)  ← context grew
  Simulating tool execution — KV should stay pinned...
    t+0.3s: KV usage = 0.001577
    t+0.6s: KV usage = 0.001577
    t+0.9s: KV usage = 0.001577

Turn 3/5  last=False  154ms  prompt=279 output=12
  Output: ```bash grep -r 'TODO' . ```
  KV usage: 0.001577 → 0.003330  (delta: +0.001753)  ← big jump (grep results injected)
  Simulating tool execution — KV should stay pinned...
    t+0.3s: KV usage = 0.003330
    t+0.6s: KV usage = 0.003330
    t+0.9s: KV usage = 0.003330

Turn 4/5  last=False  106ms  prompt=397 output=8
  Output: ```bash git status ```
  KV usage: 0.003330 → 0.004557  (delta: +0.001227)  ← context grew again
  Simulating tool execution — KV should stay pinned...
    t+0.3s: KV usage = 0.004557
    t+0.6s: KV usage = 0.004557
    t+0.9s: KV usage = 0.004557

Turn 5/5  last=True    98ms  prompt=503 output=7
  Output: ```bash pytest ```
  KV usage: 0.004557 → 0.000000  (delta: -0.004557)  ← last step, all freed!

[Final] KV cache usage: 0.000000

4.2 Token Accounting & Block Calculation

Each turn's input = system + all previous messages + chat template tokens. Tool output from Turn N is included in Turn N+1's input.

4.3 KV Cache Usage Over Time

4.4 Interpretation

4.4 System Functions for Verification

5. Bugs Found

# Crash site:
request.last_func_call = self.job_to_history[job_id][-1]["func_call"]  # KeyError!
# history[-1] is {"arrival_time": ...}, not {"departure_time": ..., "func_call": ...}

6. vLLM Block Pool Design — Why Continuum's Pin Works

Understanding Continuum's pin mechanism requires understanding vLLM v1's underlying KV cache architecture. Here we document the key design decisions and community discussions.

6.1 Append-Only Block Table

vLLM v1's block table is append-only — once a block ID is written into a request's block table, it cannot be changed. New blocks can only be appended at the end. To understand why, consider this scenario:

Scenario: Two requests with the same system prompt

Here's where v0 and v1 diverge:

After Request A finishes

Why this matters for Continuum

Continuum pins blocks after a request finishes. Because block tables are append-only, the pinned blocks' IDs are guaranteed stable — no other process can rewrite them. When the next turn arrives, prefix cache finds the pinned blocks by hash and appends their IDs to the new block table.

Source: vLLM v1 Prefix Caching Design Doc

6.2 Null Block (block_id=0)

The BlockPool reserves one block as a permanent placeholder:

# vllm/v1/core/block_pool.py:66-70
# To represent a placeholder block with block_id=0.
# The ref_cnt of null_block is not maintained, needs special care to
# avoid freeing it.
self.null_block = self.free_block_queue.popleft()
self.null_block.is_null = True

Because block tables are append-only and fixed-length, positions that have no real KV data (e.g., prefix cache miss, or tokens outside the sliding window) must be filled with a placeholder rather than left empty:

# Example: sliding window attention, only last 3 blocks active
block_table = [null, null, block_8, block_3, block_12]
               ↑ outside window, filled with null_block

The null block is marked is_null=True so it is never returned to the free queue and never allocated to any request. This is also why get_usage() subtracts 1 from total blocks:

# vllm/v1/core/block_pool.py:314
total_gpu_blocks = self.num_gpu_blocks - 1  # Subtract 1 to account for null block

6.3 Reference Counting & Free Queue

Each KVCacheBlock has a ref_cnt. This is the mechanism Continuum exploits for pinning:

Freed blocks are added to the tail of the free queue in reverse order — the last block of a request hashes the most tokens and is least likely to be reused, so it should be evicted first.

6.4 How Continuum Exploits This Design

Continuum's pin is remarkably simple — it doesn't add any new locking or memory management. It just skips calling kv_cache_manager.free() when a request finishes:

# scheduler.py — _free_blocks()
if policy == CONTINUUM and not request.is_last_step:
    length_of_pin = tool_call_estimator.set_up_pin(request)
    if length_of_pin > 0.01:
        self.pin_request(request, length_of_pin)
        return   # ← kv_cache_manager.free() is SKIPPED
                 # ← ref_cnt stays > 0
                 # ← blocks stay out of free queue
                 # ← GPU memory remains occupied

# Normal path (not pinned):
self.kv_cache_manager.free(request)  # ← ref_cnt decremented → blocks return to free queue

When the same job's next turn arrives, get_computed_blocks() finds the still-allocated blocks via prefix cache hash lookup → hit_length > 0 → skip re-prefill.

6.5 Community Discussions & References

7. How Pin Works at the Memory Level

7.1 No CUDA Memory API Involved

vLLM's KV cache pin does not call cudaMalloc, cudaFree, cudaMemcpy, or any CUDA memory management API. The entire mechanism is pure Python bookkeeping on top of a pre-allocated GPU buffer.

7.2 One-Time GPU Allocation at Startup

The only CUDA memory allocation happens once at server startup:

# vllm/v1/worker/gpu_model_runner.py:3367 — called once at init
tensor = torch.zeros(kv_cache_tensor.size, dtype=torch.int8, device="cuda")

This allocates 10.55 GiB as a single contiguous tensor on GPU. This memory is never freed until server shutdown. It is logically partitioned into 5,402 blocks:

7.3 What's Inside a Block?

Each block stores 16 tokens' Key and Value tensors across all 32 layers. The GPU tensor shape per layer (FlashAttention backend):

# vllm/attention/backends/flash_attn.py:77
shape = (2, num_blocks, block_size, num_kv_heads, head_size)
        ↑      ↑          ↑          ↑            ↑
       K/V   5402        16          8           128

One block (block_id = N) in one layer:

FlashAttention kernel uses block_id to index directly into this tensor — no copying, no indirection beyond the block table lookup.

7.4 ref_cnt — Who Owns a Block?

Each KVCacheBlock has a ref_cnt (reference count) that tracks how many requests are using it. This is needed because prefix caching allows multiple requests to share the same block — if two requests have the same system prompt, their first blocks contain identical K/V data, so vLLM lets them share instead of duplicating.

Continuum's pin exploits this: by keeping ref_cnt = 1 (skipping free()), the block stays out of the free queue — exactly the same as if a request were still using it.

7.5 Why a Doubly-Linked List?

The FreeKVCacheBlockQueue is a doubly-linked list instead of Python's deque. The reason: prefix cache hit requires removing a block from the middle.

When a new request shares a prefix with a cached block, that block (which is free and evictable) must be pulled out of the free queue immediately:

# vllm/v1/core/block_pool.py:236-250 — touch() on prefix cache hit
def touch(self, blocks):
    for block in blocks:
        if block.ref_cnt == 0:                    # block is in free queue
            self.free_block_queue.remove(block)  # ← remove from MIDDLE! O(1)
        block.ref_cnt += 1

The remove() operation is just 2 pointer assignments:

# vllm/v1/core/kv_cache_utils.py:334-340
block.prev_free_block.next_free_block = block.next_free_block
block.next_free_block.prev_free_block = block.prev_free_block
block.prev_free_block = block.next_free_block = None

7.6 block_id → GPU VRAM Address Mapping

Each layer's KV cache is a contiguous torch.zeros tensor on GPU. block_id is simply an index into the second dimension — the GPU address is computed by a single multiplication, with no hash table or pointer chasing:

# Tensor shape per layer (FlashAttention)
kv_cache.shape = (2, 5402, 16, 8, 128)
                  ↑    ↑
                 K/V  block_id ← this dimension IS the block_id

# GPU address for block N:
base = kv_cache.data_ptr()                      # tensor 起始 GPU 位址
block_size_bytes = 16 × 8 × 128 × 2 = 32,768 bytes  # per block per K or V

key_block_N   = base + 0 × total_kv_size + N × 32,768
value_block_N = base + 1 × total_kv_size + N × 32,768

The CUDA kernel does exactly this pointer arithmetic (csrc/cache_kernels.cu:306):

key_dst = key_cache + block_idx * block_stride + block_offset * page_stride;
//                    ^^^^^^^^^^^^^^^^^^^^^^^^
//                    block_idx × 32,768 = jump directly to that block

7.7 Block Read/Write: The CUDA Kernels

Block allocation/free is pure Python, but the actual K/V data read/write happens in two CUDA kernels:

Write kernel: slot → block address

// csrc/cache_kernels.cu:291-309 — reshape_and_cache_flash_kernel

slot_idx     = slot_mapping[token_idx];         // e.g. 35
block_idx    = slot_idx / block_size;           // 35 / 16 = block 2
block_offset = slot_idx % block_size;           // 35 % 16 = position 3

// Pointer arithmetic → destination in GPU tensor
key_dst   = key_cache   + block_idx * block_stride + block_offset * page_stride;
value_dst = value_cache + block_idx * block_stride + block_offset * page_stride;

// Vectorized copy (8 bf16 elements per instruction)
vectorize_copy(key_src   → key_dst);
vectorize_copy(value_src → value_dst);

Read path: block_table → FlashAttention

# vllm/attention/backends/flash_attn.py:770 (prefill) / :805 (decode)

flash_attn_varlen_func(
    q=query,
    k=key_cache,                  # entire GPU tensor
    v=value_cache,                # entire GPU tensor
    block_table=[1, 2, 3, 4, 5],   # which blocks belong to this request
    ...
)
# FlashAttention: for each query → look up block_table
# → index into key_cache/value_cache → compute attention

Why pin preserves data

7.8 Block Lifecycle: Python Data Structures Only

7.9 The FreeKVCacheBlockQueue

The core data structure is a doubly-linked list of KVCacheBlock objects. All allocate/free operations are O(1) pointer swaps:

# vllm/v1/core/kv_cache_utils.py:158-175 — each block is a node
@dataclass
class KVCacheBlock:
    block_id: int           # index into the GPU tensor
    ref_cnt: int = 0       # > 0 = in use, 0 = free/evictable
    _block_hash: Optional    # for prefix cache lookup
    prev_free_block: Optional["KVCacheBlock"]  # doubly-linked list
    next_free_block: Optional["KVCacheBlock"]  # doubly-linked list
    is_null: bool = False   # placeholder block

# Allocate: pop from front of free list
block = free_block_queue.popleft()   # O(1) pointer swap
block.ref_cnt += 1

# Free: append to tail of free list
block.ref_cnt -= 1
free_block_queue.append(block)       # O(1) pointer swap

# Pin: do nothing (skip the free above)

7.10 Why This Design?

7.11 Concrete Example: Allocating 5 Blocks

Tracing Turn 1 (66 tokens → 5 blocks) step by step:

7.12 Relationship to PagedAttention

Everything described above is PagedAttention — the core contribution of the original vLLM paper. The naming maps directly to OS virtual memory concepts:

Just as OS paging lets processes use non-contiguous physical memory via a page table, PagedAttention lets requests use non-contiguous GPU memory via a block table:

Where Continuum Fits

Continuum operates entirely within PagedAttention's framework. It adds one behavior:

8. KV Cache Budget Estimation

How does vLLM decide it can allocate 5402 blocks? This section traces the math.

8.1 The Formula

# vllm/v1/worker/gpu_worker.py:176-177
requested_memory = init_snapshot.total_memory × gpu_memory_utilization

# vllm/v1/worker/gpu_worker.py:279-280 — after profile run
non_kv_cache_memory = weights + peak_activation + non_torch_memory
available_kv = requested_memory − non_kv_cache_memory
num_gpu_blocks = available_kv // cache_block_size

8.2 Per-Block Size Calculation

Each block stores KV cache for block_size tokens across all attention layers:

# vllm/worker/cache_engine.py:121-145 — get_cache_block_size()

cache_block_size = dtype_size × num_layers × block_size × (key_entry + value_entry)

# where:
key_entry   = num_kv_heads × head_size
value_entry = num_kv_heads × head_size   # same as key (except MLA)

Llama 3.1 8B concrete values:

key_entry   = 8 × 128 = 1024
value_entry = 8 × 128 = 1024

cache_block_size = 2 × 32 × 16 × (1024 + 1024)
                 = 2 × 32 × 16 × 2048
                 = 2,097,152 bytes
                 = 2 MB per block

8.3 Memory Budget Breakdown (vLLM Measured)

These are exact values reported by vLLM's startup log (vllm/v1/worker/gpu_worker.py:391), not estimates:

# vLLM startup log (vllm/v1/worker/gpu_worker.py:391)
Free memory on device (28.66/31.34 GiB) on startup.
Desired GPU memory utilization is (0.85, 26.64 GiB).
Actual usage is 14.99 GiB for weight,
                1.01 GiB for peak activation,
                0.09 GiB for non-torch memory,
            and -0.41 GiB for CUDAGraph memory.
Available KV cache memory: 10.55 GiB
num_gpu_blocks: 5402

Step 1: How much can vLLM use?

Step 2: Subtract non-KV usage → remaining = KV cache

vLLM runs a profile, measures all non-KV memory, and subtracts from requested_memory:

available_kv = requested_memory − non_kv_total
             = 26.64 − 16.09
             = 10.55 GiB

num_gpu_blocks = 10.55 GiB ÷ 2 MiB = 5,402 blocks  ✓

8.4 What Is Activation Memory?

Activations are the intermediate tensors produced during a forward pass. Unlike model weights (permanent) and KV cache (persistent across steps), activations are transient — created during computation and discarded immediately after.

The two largest activations in each Transformer layer:

8.5 GPU Memory Map

8.6 Why This Matters for Continuum

Continuum pins blocks to keep them allocated. Every pinned block is one fewer block available for new requests. With 5,402 total blocks:

This is why Continuum's pin TTL and eviction policy matter — aggressive pinning without eviction would exhaust the KV cache budget and stall all new requests.

9. Policy Benchmark: FCFS vs Continuum vs LMCache

9.1 Experiment Design

9.2 Low Load: JPS=2.0 (no significant difference)

9.3 High Load: JPS=8.0 (significant improvement)

Per-turn latency comparison

9.4 Why Load Matters

10. Code Architecture: FCFS / Continuum / LMCache

This section traces exactly how each scheduling strategy is implemented in the vllm-continuum codebase. All three share the same scheduler loop (vllm/v1/core/sched/scheduler.py), but differ in request ordering, preemption policy, and post-completion block handling.

10.1 Policy Enum & Queue Factory

The four supported policies and their queue implementations are registered in a single factory:

# vllm/v1/core/sched/request_queue.py:17-22
class SchedulingPolicy(Enum):
    FCFS          = "fcfs"
    PRIORITY      = "priority"
    CONTINUUM     = "continuum"
    CONTEXT_SERVE = "context_serve"

# vllm/v1/core/sched/request_queue.py:342-353
def create_request_queue(policy):
    if policy == SchedulingPolicy.FCFS:
        return FCFSRequestQueue()        # deque[Request], pure FIFO
    elif policy == SchedulingPolicy.CONTINUUM:
        return ContinuumRequestQueue()   # deque + job_id-level FCFS
    elif policy == SchedulingPolicy.CONTEXT_SERVE:
        return ContinuumRequestQueue()   # reuses Continuum queue logic

10.2 FCFS — First-Come-First-Served

The simplest policy — pure FIFO with no job awareness.

Request Queue

# vllm/v1/core/sched/request_queue.py:85-140
class FCFSRequestQueue(deque[Request], RequestQueue):
    def add_request(self, request):
        self.append(request)           # → right end (newest)

    def pop_request(self):
        return self.popleft()          # ← left end (oldest)

    def peek_request(self):
        return self[0]                # oldest, without removing

    def prepend_request(self, request):
        self.appendleft(request)       # preempted → back to head

Scheduling dispatch

# vllm/v1/core/sched/scheduler.py:470-472 — schedule() waiting loop
if self.policy == SchedulingPolicy.FCFS:
    request = self.waiting.peek_request()   # oldest arrival

Preemption

# vllm/v1/core/sched/scheduler.py:394-396
else:  # FCFS
    preempted_req = self.running.pop()  # evict last-added running request

Post-completion

# vllm/v1/core/sched/scheduler.py:1398-1399
self.kv_cache_manager.free(request)  # ref_cnt → 0, blocks → free queue
del self.requests[request.request_id]

10.3 Continuum — Pin-based KV Cache Reservation

Continuum adds three mechanisms on top of the base scheduler: (1) job-ID level FCFS ordering with pinned-job priority, (2) smart preemption that protects last-step requests, (3) TTL-based KV cache pinning after request completion.

① Request Queue: ContinuumRequestQueue

Two-tier priority: pinned jobs first, then job-ID level FCFS (not request-level).

# vllm/v1/core/sched/request_queue.py:221-274
class ContinuumRequestQueue(deque[Request], RequestQueue):
    def __init__(self):
        self.job_id_first_entry_time: dict[str, float] = {}  # job → first arrival

    def peek_request(self, pinned_requests, kv_cache_manager, connector):
        pinned_job_ids = {req.job_id for req, _ in pinned_requests}

        # Tier 1: find the waiting request whose job has pinned blocks
        #         (among pinned jobs, pick the one with earliest first-entry)
        earliest = None
        for request in self:
            if request.job_id in pinned_job_ids:
                entry = self.job_id_first_entry_time.get(request.job_id)
                if entry < earliest_time:
                    earliest = request

        if earliest:
            return earliest   # ← pinned-job request goes first!

        # Tier 2: job-ID level FCFS — the job that arrived earliest overall
        for request in self:
            entry = self.job_id_first_entry_time.get(request.job_id)
            if entry < earliest_time:
                earliest = request
        return earliest

② Preemption: Protect Last-Step Requests

# vllm/v1/core/sched/scheduler.py:197-236
def pop_running_request_based_on_last_step(self, request):
    # Priority 1: evict non-last-step request with latest job entry time
    for req in self.running:
        if not req.is_last_step and job_entry_time > latest:
            latest_request = req

    # Priority 2: if all are last-step, evict the latest entry regardless
    if latest_request is None:
        for req in self.running:
            if job_entry_time > latest:
                latest_request = req

    # Edge case: if running ≤ 1, steal from pinned_requests instead
    if len(self.running) <= 1:
        # Pop the pinned request with the largest end_time
        return latest_pinned, True  # is_unpin=True

③ Pin Lifecycle: TTL-based Reservation

The complete pin/unpin lifecycle is managed by three methods in the scheduler:

# vllm/v1/core/sched/scheduler.py:239-258

## 1. Pin: skip free(), store (request, end_time) tuple
def pin_request(self, request, length_of_pin):
    self.pinned_requests.append((request, time.time() + length_of_pin))

## 2. Unpin: remove from list, call free() to release blocks
def unpin_request(self, request, end_time):
    self.pinned_requests.remove((request, end_time))
    self.kv_cache_manager.free(request)  # ref_cnt → 0

## 3. Regular cleanup: called at the start of every schedule() step
def unpin_requests_regular(self):
    waiting_job_ids = [req.job_id for req in self.waiting]
    for request, end_time in self.pinned_requests:
        # Only unpin if: (a) TTL expired AND (b) no waiting request for this job
        if request.job_id not in waiting_job_ids and time.time() >= end_time:
            self.unpin_request(request, end_time)

④ TTL Decision: ToolCallEstimator

# vllm/v1/core/estimate_with_func.py:151-160
FIXED_THRESHOLD_CONTINUUM = 2.0   # seconds

def set_up_pin(self, request):
    if request.this_func_call is None:
        return 0                        # can't detect tool → don't pin

    exec_time = self.get_func_call_exec_time(request.this_func_call)

    if exec_time > FIXED_THRESHOLD_CONTINUUM:
        return 0                        # slow tool (>2s) → don't pin

    return FIXED_THRESHOLD_CONTINUUM    # fast tool → pin for 2.0s

⑤ Post-completion: _free_blocks() Dispatch

# vllm/v1/core/sched/scheduler.py:1356-1399
def _free_blocks(self, request):
    # Step 1: unpin any sibling pinned request with same job_id
    for req, end_time in self.pinned_requests:
        if req.job_id == request.job_id:
            self.unpin_request(req, end_time)

    # Step 2: Continuum decision
    if self.policy == SchedulingPolicy.CONTINUUM and not request.is_last_step:
        length_of_pin = self.tool_call_estimator.set_up_pin(request)
        if length_of_pin > 0.01:
            self.pin_request(request, length_of_pin)   # ← SKIP free()
            del self.requests[request.request_id]
            return

    # Default: free blocks (FCFS path, or is_last_step=True)
    self.kv_cache_manager.free(request)
    del self.requests[request.request_id]

10.4 ContextServe — Three-Way Dispatch Extension

ContextServe extends Continuum's binary pin/no-pin decision into a three-way dispatch: FAST (strict pin), SLOW (proactive free), UNCERTAIN (fallback to Continuum TTL). It shares the same ContinuumRequestQueue and preemption logic.

# vllm/v1/core/sched/scheduler.py:1364-1388
if self.policy == SchedulingPolicy.CONTEXT_SERVE and not request.is_last_step:
    speed, pin_duration = self.tool_call_estimator.set_up_pin_contextserve(request)

    if speed == ToolSpeed.FAST:           # ls, cat, pwd, git status ...
        self.pin_request(request, pin_duration)   # lock in VRAM

    elif speed == ToolSpeed.SLOW:         # pytest, pip install, docker build ...
        self.kv_cache_manager.free(request)     # proactive free at t=0

    else:                                # ToolSpeed.UNCERTAIN
        if pin_duration > 0.01:             # fallback to Continuum TTL
            self.pin_request(request, pin_duration)

Three-Layer Prediction Stack

The ContextAwarePredictor (vllm/v1/core/context_predictor.py) uses a three-layer classification strategy:

10.5 LMCache — CPU DRAM Offload

LMCache is an orthogonal strategy to Continuum. Instead of pinning blocks in GPU VRAM (which reduces capacity for other jobs), LMCache offloads evicted KV cache blocks to CPU DRAM or a remote server, then retrieves them when the same job returns. It integrates via the KVConnectorBase_V1 connector interface.

Connector Architecture

# vllm/distributed/kv_transfer/kv_connector/v1/lmcache_connector.py:23-27
class LMCacheConnectorV1(KVConnectorBase_V1):
    def __init__(self, vllm_config, role):
        self._lmcache_engine = LMCacheConnectorV1Impl(vllm_config, role, self)
        # Delegates to lmcache.integration.vllm.vllm_v1_adapter

Two operational roles, split across scheduler and worker processes:

Data Flow: Store & Retrieve

## Store (after each forward step)
Attention layer → save_kv_layer(layer_name, kv_tensor, attn_metadata)
    → LMCache chunks KV by 256 tokens (configurable)
    → Stores to CPU DRAM (LMCACHE_LOCAL_CPU=True)
       or remote server (LMCACHE_REMOTE_URL=lm://host:port)

## Retrieve (on new request for same prefix)
Scheduler → connector.get_num_new_matched_tokens(request)
    → LMCache matches prompt prefix hash against stored chunks
    → Returns count of externally cached tokens

Worker → connector.start_load_kv(forward_context)
    → Per-layer: wait_for_layer_load(layer_name)
    → Loaded KV appears in GPU paged buffer
    → Attention skips recomputation for loaded tokens

Configuration

# CLI: pass via JSON to --kv-transfer-config
vllm serve MODEL --kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

# Environment variables (set before launch)
export LMCACHE_USE_EXPERIMENTAL="True"
export LMCACHE_LOCAL_CPU="True"              # enable CPU DRAM backend
export LMCACHE_MAX_LOCAL_CPU_SIZE="5.0"     # 5 GB CPU memory limit
export LMCACHE_CHUNK_SIZE="256"             # tokens per chunk

10.6 Side-by-Side Comparison

10.7 Request Lifecycle per Strategy

How each strategy handles KV cache during tool-call gaps under high memory pressure (KV cache >90%). Toggle between short and long tool execution times:

request finished → _free_blocks() → kv_cache_manager.free() → blocks → free queue
  → blocks immediately available for any new request (no pin)

request finished → _free_blocks():
  if not is_last_step + fast tool → pin_request(2.0s) → blocks RESERVED (skip free)
  → next turn → pinned-job gets scheduling priority → cache hit

forward → per-layer: save_kv_layer() → async copy to CPU DRAM
  → wait_for_save() → free() → GPU blocks released
  → next turn → connector.get_num_new_matched_tokens() → skip prefill for cached portion
    → start_load_kv() → per-layer wait_for_layer_load()

request finished → _free_blocks() → kv_cache_manager.free() → blocks → free queue
  → long tool — blocks overwritten by others — full recompute on return

pin_request(2.0s) → TTL counting...
  → unpin_requests_regular() → time.time() >= end_time → unpin + free()
  → blocks evicted before tool returns — degrades to FCFS behavior

per-layer: save_kv_layer() → async GPU→CPU copy → free()
  → long tool — KV safe on CPU, GPU fully available for others
  → next turn → start_load_kv() → reload from DRAM (skip recompute)

10.8 Eviction Behavior Under Memory Pressure

When KV cache is nearly full (~95%), a new request arrives that needs blocks. How does each strategy decide what to evict? Below demonstrates with 3 concurrent jobs and only 8 free blocks.

10.9 Multi-GPU JPS Sweep Results

Three policies tested across three GPUs at varying load levels. Key findings:

10.10 Full Call Graph: Experiment → vLLM Internals → Response

End-to-end trace from ab_benchmark.py request through every vLLM layer to HTTP response. Shows how job_id / is_last_step propagate and where each strategy diverges.

Variable Propagation: extra_body → pin decision

10.11 Experiment Scripts — Source Code Walkthrough

Three key scripts drive all experiments: the policy benchmark, KV pin verifier, and JPS sweep plotter.

① ab_benchmark.py — Poisson-Arrival Policy Benchmark

547 lines. Simulates realistic multi-turn agentic workload with Poisson-distributed job arrivals. Runs one policy per invocation; compare by running multiple times with different --policy.

# CLI usage
python3 ab_benchmark.py --policy continuum --port 8199 --jps 8.0 --duration 90 --turns 8

# Core: Poisson arrival process (line 381)
def poisson_arrival(jps, duration_s, base_url, model_id, n_turns):
    gap = random.expovariate(jps)   # exponential inter-arrival time
    time.sleep(gap)
    # spawn job as thread with job_id = f"job_{count:04d}"

# Core: how Continuum metadata is passed (line 329)
resp = client.chat.completions.create(
    model=model_id,
    messages=messages,       # accumulated context (grows each turn)
    max_tokens=150,
    extra_body={
        "job_id": job_id,
        "is_last_step": (i == n_turns - 1),  # True only on final turn
    },
)

# Tool execution time per turn (line 276-286)
TOOL_TIME_DISTRIBUTIONS = [
    (0.05, 0.15),   # Turn 1: find/ls — fast
    (0.05, 0.10),   # Turn 2: cat — fast
    (0.05, 0.10),   # Turn 3: cat — fast
    (0.08, 0.20),   # Turn 4: grep — fast
    (2.0,  5.0),    # Turn 5: pytest — SLOW (long-tailed)
    (0.05, 0.10),   # Turn 6: cat — fast
    (0.10, 0.30),   # Turn 7: patch — fast
]

# Output JSON schema (line 516-538)
{
  "policy": "continuum",
  "jps": 8.0,
  "avg_duration_s": 12.465,
  "p90_duration_s": 14.371,
  "p95_duration_s": 14.629,
  "per_turn_avg_latency_ms": {"1": 1051, "2": 519, ...},
  "per_turn_avg_prompt_tokens": {"1": 92, "2": 469, ...},
  "job_durations": [6.185, 6.607, ...]
}

② test_pin_verify.py — KV Cache Pin Verification

102 lines. Verifies Continuum's pin mechanism by polling vllm:kv_cache_usage_perc from the Prometheus /metrics endpoint during tool execution gaps.

# Parse Prometheus metric (line 36-41)
def get_kv_usage():
    r = req.get(f"{BASE}/metrics", timeout=5)
    for line in r.text.split("\n"):
        if line.startswith("vllm:kv_cache_usage_perc{"):
            return float(line.split()[-1])

# Verification: poll every 0.3s during 1s tool gap (line 84-95)
if not is_last:
    print("  Simulating tool execution (1s) — KV should stay pinned...")
    for t in [0.3, 0.6, 0.9]:
        time.sleep(0.3)
        u = get_kv_usage()
        print(f"    t+{t:.1f}s: KV usage = {u:.6f}")
        # Expected: u ≈ usage_post (constant = pinned)
        # Failure:  u drops → blocks evicted, pin broken

③ plot_jps_sweep.py — Multi-GPU Comparison Plots

153 lines. Reads JSON results from 3 GPUs × 3 policies, generates publication-quality comparison figures.

# GPU configurations (line 13-17)
GPUS = {
    "rtx_6000": {"name": "RTX 6000 (24 GB)", "jps": [1.0, 2.0, 5.0, 8.0, 12.0]},
    "A100":     {"name": "A100 (40 GB)",      "jps": [2.0, 5.0, 10.0, 15.0, 25.0, 40.0]},
    "L40S":     {"name": "L40S (48 GB)",      "jps": [2.0, 5.0, 10.0, 15.0, 25.0]},
}

# Speedup formula (line 124)
speedup = (fcfs["avg_duration_s"] - other["avg_duration_s"]) / fcfs["avg_duration_s"] * 100

# Generates 3 figure types:
# 1. jps_sweep_{gpu}.png   — per-GPU: avg duration vs JPS for 3 policies
# 2. jps_sweep_combined.png — 3-subplot side-by-side comparison
# 3. jps_sweep_speedup.png  — % improvement over FCFS baseline

10.12 Key Files Reference

11. Observations & Takeaways