Checkpoint and front-end research: ByteCheckpoint, Checkmate, DistDPU檢查點與前端網路研究：ByteCheckpoint、Checkmate、DistDPU

Section 9.3 sized a checkpoint as a burst of bytes and the stall as bytes divided by a rate, and Section 9.1 kept that burst off the back end. Three recent papers extend this page. Two NSDI papers attack the GPU stall $C$ of Young–Daly and the age of the newest recovery point, from opposite ends. A SIGCOMM paper redraws the front-end side of the GPU server itself (figure below).

第 9.3 節把檢查點當成一波位元組突發，停頓是位元組數除以速率；第 9.1 節則把這波突發留在後端之外。三篇近期論文延伸了本頁。兩篇 NSDI 論文從相反的兩端處理 Young–Daly 中 GPU 的停頓 $C$，以及最新恢復點的「年齡」。一篇 SIGCOMM 論文則重新設計 GPU 伺服器本身的前端那一側（見下圖）。

ByteCheckpoint (NSDI 2025; The University of Hong Kong and ByteDance) keeps the copy-persist path of Section 9.3 and makes it cheap and portable. It stores shards in global coordinates, so that any later parallel layout can load them. It spreads writes over the data-parallel replicas and reads each byte once per data-parallel group, before an all-to-all on the GPU network. In production, a 405B model on 8,960 GPUs stalls 0.59 s per checkpoint, and the checkpoint is durable 51.06 s later. At that scale the limits were control paths, not link bandwidth.

ByteCheckpoint（NSDI 2025；香港大學與字節跳動）保留第 9.3 節的「複製後持久化」路徑，並讓它便宜又可攜。它以全域座標記錄分片，讓之後的任何平行配置都能載入。它把寫入分散到各資料平行副本，每個資料平行群組只讀每個位元組一次，再於 GPU 網路上做 all-to-all。在正式環境中，一個 405B 模型在 8,960 顆 GPU 上每次檢查點停頓 0.59 秒，檢查點在 51.06 秒後才持久。在這個規模下，極限來自控制路徑，不是鏈路頻寬。

Checkmate (NSDI 2026; Tufts University and MIT) removes the GPU from checkpointing. The reduced gradient crosses the top-of-rack switch in every iteration anyway, so a programmable switch copies it to CPU shadow nodes that run the same optimizer step, and every iteration becomes a checkpoint. This moves checkpoint capture onto the back end, against Section 9.1's rule; it adds no bytes to the training ports, because the copies leave on dedicated, PFC-protected ports. On a 12-GPU testbed at 100 Gb/s, throughput equalled that of no checkpointing; its 16,384-GPU savings come from a cost model.

Checkmate（NSDI 2026；塔夫茨大學與 MIT）把 GPU 從檢查點工作中拿掉。歸約後的梯度本來每次迭代都會經過機櫃頂端交換器，所以一台可程式化交換器把它複製給 CPU 影子節點，由它們執行同樣的優化器步驟，每次迭代都成為一個檢查點。這把擷取檢查點的工作搬到後端，違反第 9.1 節的規則；但它不在訓練用的埠上加任何位元組，因為副本從專用、受 PFC 保護的埠送出。在 100 Gb/s、12 顆 GPU 的測試平台上，吞吐與不存檢查點相同；16,384 顆 GPU 的節省則來自成本模型。

DistDPU (SIGCOMM 2026, Experience track; Fudan, Tencent and Michigan) works on the front-end and VPC side of the same server. Tencent's AI cloud must give each server 400 Gb/s to 3.2 Tb/s of RDMA with full VPC policy, even for bare-metal tenants with root access. Stacking one DPU per NIC repeats an SoC, memory and software stack nine times, although AI instances mostly create only a few percent of the flow-table sessions of general-compute instances. DistDPU keeps one orchestration module on the front-end NIC and turns the other NICs into lean execution modules, managed over a private PCIe channel: one logical DPU with up to 3.6 Tb/s of provisioned capacity. In production the orchestration module's CPU load averages below 15%; on a two-node H20 testbed, RDMA throughput is within 0.4–2.8% of BlueField-3 and ConnectX-7. Its cost and power ratios are analysis.

DistDPU（SIGCOMM 2026 Experience 議程；復旦、騰訊與密西根大學）處理同一台伺服器的前端與 VPC 那一側。騰訊的 AI 雲必須給每台伺服器 400 Gb/s 到 3.2 Tb/s 的 RDMA，並保有完整的 VPC 政策，即使租戶是擁有 root 權限的裸機租戶也一樣。每張 NIC 疊一顆 DPU，會把 SoC、記憶體與軟體堆疊重複九次，但 AI 執行個體產生的流表 session 大多只有一般運算執行個體的百分之幾。DistDPU 在前端 NIC 上保留一個編排模組（orchestration module），把其他 NIC 變成精簡的執行模組（execution module），透過私有的 PCIe 通道管理：成為一顆邏輯 DPU，提供最多 3.6 Tb/s 的配置容量。在正式環境中，編排模組的 CPU 負載平均低於 15%；在兩台 H20 節點的測試平台上，RDMA 吞吐與 BlueField-3 和 ConnectX-7 相差 0.4–2.8%。它的成本與功耗比值屬於分析。

What the checkpoint papers agree on, and where they differ. The stall and the time to durability are different numbers, and only the stall is Young–Daly's $C$: ByteCheckpoint reports both, and Checkmate drives the stall to zero. The interval is a cost trade-off; Checkmate's cost model is Young–Daly with $\mathrm{MTBF} = 1/(\lambda N)$ ($N$ GPUs failing at $\lambda$ per GPU-hour). The costs land in different places: storage metadata and host memory for ByteCheckpoint, which supports FSDP and runs in production; switch ports, NICs and CPU servers for Checkmate, which needs all-reduce data parallelism and lossless shadow ports.

兩篇檢查點論文的共識與差異。停頓與達到持久的時間是兩個不同的數字，只有停頓是 Young–Daly 的 $C$：ByteCheckpoint 兩者都報告，Checkmate 則把停頓壓到零。間隔是成本上的取捨；Checkmate 的成本模型就是 Young–Daly，其中 $\mathrm{MTBF} = 1/(\lambda N)$（$N$ 顆 GPU、每 GPU 小時故障 $\lambda$ 次）。成本落在不同的地方：ByteCheckpoint 落在儲存中繼資料與主機記憶體，它支援 FSDP，而且在正式環境執行；Checkmate 落在交換器埠、NIC 與 CPU 伺服器，它需要 all-reduce 的資料平行與無損的影子埠。

What all three share. Each crosses the line between the two networks of Section 9.1 in a controlled way. ByteCheckpoint puts recovery bytes on the back end only at load time. Checkmate copies back-end bytes onto dedicated ports. DistDPU keeps checkpoint and storage traffic on the front-end NIC but manages the NICs of both networks from one control module. None lets storage or management bursts compete with the collectives.

三篇的共同點。每一篇都以受控的方式跨越第 9.1 節兩個網路之間的界線。ByteCheckpoint 只在載入時把恢復用的位元組放上後端。Checkmate 把後端的位元組複製到專用的埠。DistDPU 把檢查點與儲存流量留在前端 NIC，卻由同一個控制模組管理兩個網路的 NIC。三者都不讓儲存或管理的突發與集體通訊競爭。

Each card below opens one digest of about ten minutes, with five questions. Read them in order: ByteCheckpoint, Checkmate, then DistDPU.

下面每張卡片打開一篇約十分鐘的導讀，附五題測驗。請依序閱讀：ByteCheckpoint、Checkmate，然後是 DistDPU。