Does Bulk H2D Really Overlap with Decode?

1. Why we profiled

In production runs of the SimpleCPUOffloadConnector with FCFS scheduling on H100, decode-only steps slow down 2.8× (p50) and 6× (p99) when bulk H2D + D2H DMA fires concurrently versus when no DMA is active. The connector deep dive explains the design — multi-stream, async, ~5ms launch hidden under compute. The numbers say the design is not delivering. Either the design is wrong, or the design isn't running as written.

The slowdown numbers from sched_trace

Initial hypothesis: H100 has 3 TB/s HBM. Bulk H2D writes pinned-host → HBM at PCIe Gen5 speeds. FlashAttn reads KV from HBM at compute speeds. If both compete for the same memory subsystem at the same time, attention runs at degraded effective bandwidth → kernel duration grows → step elongates. This is the classic memory-contention story.

Profiling with nsys lets us check it directly: GPU SM/Tensor/DRAM utilization counters at 100 Hz, plus per-event CUPTI traces with stream attribution.

2. What the GPU was actually doing

The H100 trace contains 2.55 M GPU metric samples at 100 Hz across 850 s. Bucketing 68 k samples in the steady-state window (170 s … 850 s) by which activity is in flight at the sample timestamp:

DRAM Read Bandwidth, SM Active, Tensor Active per regime

3. The stream handle's full code path

If H2D really runs on a dedicated stream, GPU metrics should sometimes show H2D and FA co-active (concurrent kernels on different streams). Before drawing conclusions from the metrics, let's check what the code actually does. The path from torch.cuda.Stream to the CUDA driver call is short — four files, ~10 lines:

4. The stream attribution paradox

nsys's TARGET_INFO_CUDA_STREAM table confirms the CUDA context created 135 streams at runtime, including the priority-0 NON_BLOCKING streams that should hold load_stream/store_stream. But when we look at which streams actually carry traffic:

The 73 H2D events ≥ 100 MB totaling 9.5 GB in steady-state — exactly what we'd expect connector bulk loads to look like — all live on stream 7. The dedicated load_stream/store_stream the source code creates are not visibly active.

Three explanations, ranked by likelihood

5. Reconciling the two pieces of evidence

There is one critical piece of source code that makes everything consistent. From worker.py:213-217:

def start_load_kv(self) -> None:
    # NOTE: we defer launching both load and store to get_finished(),
    # which runs after model execution. This hides the CPU-side
    # block copy op overhead (~5ms) behind GPU compute.
    pass

The actual per-step pipeline (with deferred launch)

What we'd see if launch were moved before forward

6. Fix direction & ranking

Validation plan for the proposed fix

7. Open questions

8. Reproduction

Trace

# Source nsys-rep on PACE
/storage/scratch1/1/hlin464/swe_smith/results/nsys_swe_h100_jps10_dram64_fcfs_cpuoff/nsys_h100.nsys-rep

# SQLite export on lab (366 MB)
/tmp/nsys_analyze/nsys_h100.sqlite

# Bench: Llama-3.1-8B-Instruct, 100 jobs, jobs_100_pinjab.json
# Connector: SimpleCPUOffloadConnector, CPU pool = 64 GB
# Scheduling: FCFS

Analysis scripts

Source code

# PACE editable install
/storage/project/r-rs275-0/hlin464/vllm-agent-kvcache/vllm/v1/simple_kv_offload/
├── worker.py          # SimpleCPUOffloadWorker; lines 189-217 critical
├── copy_backend.py    # DmaCopyBackend, _copy_loop daemon thread
├── cuda_mem_ops.py    # cuMemcpyBatchAsync invocation, line 148
└── manager.py         # scheduler-side admission logic

# Findings doc on PACE
/storage/project/r-rs275-0/hlin464/agent-kvcache/nsys_h100_dma_findings.md