Research digest 2 of 3 · Chapter 9: Front end and checkpoints

研究導讀 2 / 3 · 第 9 章：前端網路與檢查點

Checkmate: Zero Performance Overhead Model Checkpointing via Network Gradient ReplicationCheckmate：以網路梯度複製實現零效能開銷的模型檢查點

NSDI 2026Ankit Bhardwaj, Weiyang Wang, Jeremy Carin, Adam Belay, Manya Ghobadi (Tufts University, MIT) · paper論文

After reading, you can explain how Checkmate copies each iteration's reduced gradients in the switch to CPU shadow nodes, so that every iteration is a checkpoint and the GPUs never stall.讀完後，你能說明 Checkmate 如何在交換器裡把每次迭代歸約後的梯度複製給 CPU 影子節點，讓每次迭代都是一個檢查點，GPU 從不停頓。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 9 ↗第 9 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Copying model and optimizer state out of GPU memory slows training or limits how often it runs, so large jobs checkpoint every tens of minutes and repeat half an interval of work per failure. Checkmate removes the GPU from checkpointing. In data-parallel training, the next state follows from the current state and the reduced gradient, which crosses the network every iteration anyway. An NCCL plugin tags each final gradient chunk once during the ring all-gather, and a programmable switch copies the tagged packets to CPU shadow nodes that run the same AdamW step. On 12 single-A100 nodes at 100 Gb/s, throughput matched no checkpointing with a checkpoint every iteration: 5 to 34.5× more often than CheckFreq, with 80 to 97.1% less repeated work. The 16,384-GPU savings come from a cost model. The price is switch ports, CPU servers and a lossless shadow path; FSDP is not supported yet.

把模型與優化器狀態從 GPU 記憶體複製出來，會拖慢訓練或限制它能多常執行，所以大型任務每隔數十分鐘才存一次檢查點，每次故障重做半個間隔的工作。Checkmate 把 GPU 從檢查點工作中拿掉。在資料平行訓練中，下一個狀態由目前狀態與歸約後的梯度決定，而那個梯度本來每次迭代都會經過網路。一個 NCCL 外掛在 ring 的 all-gather 期間把每個最終梯度 chunk 標記一次，一台可程式化交換器再把標記的封包複製給 CPU 影子節點，由它們執行同樣的 AdamW 步驟。在 12 個各有一顆 A100、100 Gb/s 的節點上，每次迭代都存檢查點，吞吐卻與不存相同：頻率是 CheckFreq 的 5 到 34.5 倍，重做的工作少 80% 到 97.1%。16,384 顆 GPU 的節省來自成本模型。代價是交換器埠、CPU 伺服器與無損的影子路徑；目前還不支援 FSDP。

1. Where this paper sits1. 這篇論文的位置

The Chapter 9 keeps checkpoints on the front end. Its section 1 gives Meta's rule that checkpoints never touch the back end, because a storage incast on a lossless fabric would spread PFC pauses into the collectives. Its section 3 shrinks the stall with a two-stage copy and picks the interval with Young–Daly, $T \approx \sqrt{2C \cdot \mathrm{MTBF}}$ . Each failure still repeats about half an interval.

第 9 章把檢查點留在前端。它的第 1 節給出 Meta 的規則：檢查點永遠不碰後端，因為無損網路上的儲存 incast 會把 PFC 暫停擴散到集體通訊。它的第 3 節用兩階段複製縮短停頓，並用 Young–Daly 選擇間隔， $T \approx \sqrt{2C \cdot \mathrm{MTBF}}$ 。每次故障仍要重做約半個間隔。

Checkmate asks what happens when $C$ is zero for the GPU and the interval is one iteration. It moves checkpoint capture onto the back end (Figure 1), against the rule above, but it adds no bytes to the training ports: it copies bytes that the ring already sends, onto dedicated switch ports.

Checkmate 問的是：如果對 GPU 而言 $C$ 是零、間隔是一次迭代，會怎樣？它把擷取檢查點的工作搬到後端（圖 1），違反上面的規則，但它不在訓練用的埠上加任何位元組：它複製 ring 本來就會送的位元組，送到專用的交換器埠。

Figure 1. Top: Chapter 9's two-stage path; the GPUs wait for the PCIe copy, and the bytes reach storage over the lossy front end. Bottom: Checkmate; the GPUs send only their usual all-reduce traffic, and the top-of-rack switch copies tagged packets to a CPU shadow node on a dedicated, PFC-protected port.圖 1。上：第 9 章的兩階段路徑；GPU 等待 PCIe 複製，位元組經有損的前端到達儲存。下：Checkmate；GPU 只送原本的 all-reduce 流量，機櫃頂端交換器把標記的封包複製到一個專用、受 PFC 保護的埠上的 CPU 影子節點。Drawn for this primer from Bhardwaj et al., NSDI 2026, §2.2 and Figure 3.

2. The problem, measured2. 問題，用量測說話

Copy-persist has two costs (§2.2): the copy competes with training for the GPU, and the persist must end before the next checkpoint, which caps the frequency. On GPT3-XL (batch 16, 4 nodes) with a checkpoint every iteration (Figure 2), the 1.77 s iteration became 9.5× slower with synchronous checkpoints, 8.45× with asynchronous ones and 3.5× with PyTorch DCP sharded over the four nodes. Gemini, which keeps checkpoints in remote CPU memory over the training network, lost up to 3.47× in throughput.

複製後持久化有兩種成本（§2.2）：複製會和訓練搶 GPU，而持久化必須在下一個檢查點前結束，這限制了頻率。在 GPT3-XL（批次 16、4 個節點）上每次迭代都存檢查點（圖 2），1.77 秒的迭代在同步檢查點下慢 9.5 倍，非同步下慢 8.45 倍，分片到四個節點的 PyTorch DCP 慢 3.5 倍。Gemini 透過訓練網路把檢查點存在遠端 CPU 記憶體，吞吐最多損失 3.47 倍。

Figure 2. Slowdown when existing systems checkpoint every iteration, measured on the authors' testbed (§3.2, the paper's Figure 2). The bar lengths are the ratios stated in the text; 1× is the 1.77 s iteration without checkpoints.圖 2。現有系統每次迭代都存檢查點時的變慢倍數，在作者的測試平台上量得（§3.2，論文 Figure 2）。長條長度是正文給出的比值；1 倍是不存檢查點時 1.77 秒的迭代。Drawn for this primer from Bhardwaj et al., NSDI 2026, §3.2.

The paper motivates scale with Llama 3 405B on 16,384 GPUs (419 interruptions in 54 days), but it did not run that job. Section 3.4 treats those numbers as what they are: inputs to a cost model.

論文用 16,384 顆 GPU 上的 Llama 3 405B（54 天 419 次中斷）說明大規模的情況，但並沒有跑那個任務。第 3.4 節把那些數字當成它們實際的身分：成本模型的輸入。

3. The design3. 設計

Checkmate has four steps: ① the training nodes tag reduced gradients; ② the switch multicasts the tagged packets; ③ shadow nodes map them to parameters; ④ shadow nodes run the optimizer step.

Checkmate 有四個步驟：① 訓練節點標記歸約後的梯度；② 交換器多播標記的封包；③ 影子節點把它們對應到參數；④ 影子節點執行優化器步驟。

Diagram: a training cluster whose NCCL instances tag gradients (1), programmable switches that replicate them (2), and a shadow cluster whose NICs (3) and TCP stacks feed a model update (4). — Figure 3 of the paper: Checkmate's architecture (a design diagram). Follow the numbers: NCCL tags gradients during the normal synchronization (1), programmable switches copy them (2), the shadow cluster's NICs receive them (3), and the shadow nodes apply them to their model copy (4).論文 Figure 3：Checkmate 的架構（設計示意圖）。照編號看：NCCL 在一般的梯度同步中標記梯度 (1)，可程式化交換器複製它們 (2)，影子叢集的 NIC 接收 (3)，影子節點把它們套用到自己的模型副本 (4)。Screenshot of Bhardwaj et al., NSDI 2026, Figure 3 (USENIX PDF).

3.1 Every iteration is already on the wire3.1 每次迭代本來就在線路上

With weights $\theta_t$ , optimizer state $s_t$ (for example Adam's moments) and reduced gradient $g_t$ , one training step is a function:

以權重 $\theta_t$ 、優化器狀態 $s_t$ （例如 Adam 的動量）與歸約後的梯度 $g_t$ 表示，一個訓練步驟是一個函數：

(\theta_{t+1},\ s_{t+1}) = \mathrm{Opt}(\theta_t,\ s_t,\ g_t)

A replica that starts from the same $\theta_0, s_0$ and receives every $g_t$ stays identical to the GPUs, and in data parallelism $g_t$ crosses the network every iteration. SGD, Adam and AdamW update each parameter from its own value, gradient and state; the authors call such optimizers functional. The shadow work therefore splits over any number of CPU nodes by gradient bucket, with no coordination.

從同樣的 $\theta_0, s_0$ 出發、收到每一個 $g_t$ 的副本，會一直與 GPU 上的狀態相同，而在資料平行中 $g_t$ 每次迭代都經過網路。SGD、Adam 與 AdamW 只用每個參數自己的值、梯度與狀態更新它；作者稱這類優化器為函數式（functional）。因此影子節點的工作可以按梯度 bucket 分給任意多台 CPU 節點，不需協調。

3.2 Heartbeat tagging: each final chunk once3.2 Heartbeat 標記：每個最終 chunk 一次

Ring all-reduce is a reduce-scatter followed by an all-gather (Chapter 4, section 3). Only the all-gather carries final chunks, and it forwards each one $N-1$ times. Mirroring every all-gather packet would therefore copy each chunk $N-1$ times; tagging every chunk in round 0 would make all $N$ ranks send to the shadow at once, an $N$ -way incast. Heartbeat tagging spreads the tags (Figure 3): rank 0 tags its own chunk in round 0, and the last rank tags every chunk it sends. Each chunk is tagged once, only round 0 has two tagged senders (hence two NICs per shadow), and only two links per ring are mirrored.

Ring all-reduce 是 reduce-scatter 接著 all-gather（第 4.2 節）。只有 all-gather 承載最終 chunk，而且每個 chunk 被轉送 $N-1$ 次。因此鏡像每個 all-gather 封包會把每個 chunk 複製 $N-1$ 次；在第 0 輪標記所有 chunk，則會讓全部 $N$ 個 rank 同時送往影子節點，形成 $N$ 對一的 incast。Heartbeat 標記把標記分散到各輪（圖 3）：rank 0 在第 0 輪標記自己的 chunk，最後一個 rank 標記它送出的每個 chunk。每個 chunk 只被標記一次，只有第 0 輪有兩個標記的傳送者（所以每個影子節點有兩張 NIC），每個 ring 只有兩條鏈路被鏡像。

Figure 3. Heartbeat tagging on 4 GPUs, after the paper's Figure 4. Each cell is the chunk a GPU sends in that all-gather round; the highlighted cells are tagged. Every chunk reaches the shadow once, and at most two tagged packets arrive at the same time.圖 3。4 顆 GPU 的 heartbeat 標記，依論文 Figure 4 重畫。每一格是某顆 GPU 在該輪 all-gather 送出的 chunk；強調的格子被標記。每個 chunk 只到達影子節點一次，同一時間最多兩個標記的封包到達。Drawn for this primer from Bhardwaj et al., NSDI 2026, §4.1 and Figure 4.

The tag is one bit in the DSCP field of the IP header. Each NCCL channel counts its tagged chunks in a custom TCP option. On the copy, the switch writes this count into the TCP sequence field and rewrites the IP addresses, so the shadow sees one continuous TCP stream per channel. Panel 1 runs the tagging rules on rings of any size.

標記是 IP 標頭 DSCP 欄位中的一個位元。每個 NCCL channel 在自訂的 TCP 選項中計算它標記的 chunk 數。在副本上，交換器把這個計數寫進 TCP 序號欄位，並改寫 IP 位址，所以影子節點看到的是每個 channel 一條連續的 TCP 串流。面板 1 讓你在任意大小的 ring 上執行這些標記規則。

Panel 1面板 1

Tag a ring once把一個 ring 標記一次

Ranks in the DP ring

N

DP ring 中的 rank 數

N

Tagging rule標記規則

Model (FP32 gradients, Table 1)模型（FP32 梯度，Table 1）

Link rate鏈路速率

Tagged senders at once (max)

同時標記的傳送者（最多）

Peak rate into the shadow

進入影子節點的峰值速率

Copies of each final chunk

每個最終 chunk 的副本數

Bytes to the shadow per iteration

每次迭代送往影子節點的位元組

Mirrored links in the ring

ring 中被鏡像的鏈路

All-gather phase at line rate

以線速計的 all-gather 階段

Rows are the ranks of one DP ring and columns the $N-1$ all-gather rounds. After the reduce-scatter, rank $i$ holds final chunk $(i+1) \bmod N$ , and in round $r$ it sends chunk $(i+1-r) \bmod N$ ; for $N = 4$ this is the paper's Figure 4. Filled cells are tagged and copied by the switch. The gradient size is parameters × 4 bytes, because these Table 1 models train in FP32. The all-gather time is bytes ÷ link rate and leaves out latency. Defaults are the testbed: 12-way DP at 100 Gb/s. Switch to the other two rules and watch the senders per round and the copies.

每一列是一個 DP ring 的 rank，每一欄是 $N-1$ 輪 all-gather 中的一輪。reduce-scatter 之後，rank $i$ 持有最終 chunk $(i+1) \bmod N$ ，並在第 $r$ 輪送出 chunk $(i+1-r) \bmod N$ ； $N = 4$ 時就是論文的 Figure 4。實心格子被標記、由交換器複製。梯度大小是參數數 × 4 位元組，因為 Table 1 中這些模型以 FP32 訓練。all-gather 時間是位元組 ÷ 鏈路速率，不含延遲。預設值是測試平台：12 路 DP、100 Gb/s。切換到另外兩種規則，看每輪的傳送者與副本數如何變化。

3.3 The switch and the shadow nodes3.3 交換器與影子節點

For the first and the last rank of each DP group, the switch's control plane installs a multicast group that holds the next training rank and the shadow. A minimal TCP server on the switch plays a parameter server: shadows connect to it before training, and the switch drops their ACKs. The data plane keeps no per-flow state. A tagged packet gets a multicast group at ingress, the packet replication engine (PRE) copies it, and egress rewrites the shadow's copy (Figure 4).

對每個 DP 群組的第一個與最後一個 rank，交換器的控制平面安裝一個多播群組，成員是下一個訓練 rank 與影子節點。交換器上一個最小的 TCP 伺服器扮演參數伺服器：影子節點在訓練前就連上它，而交換器丟棄它們回的 ACK。資料平面不保存每條流的狀態。標記的封包在 ingress 被指定多播群組，由封包複製引擎（PRE）複製，egress 再改寫給影子節點的那一份（圖 4）。

Figure 4. One packet through the switch, after §4.3. Copy A continues to the next rank unchanged; copy B gets the channel counter as its TCP sequence number and the shadow's address.圖 4。一個封包通過交換器，依 §4.3 重畫。副本 A 原樣送往下一個 rank；副本 B 把 channel 計數器當成 TCP 序號，並換上影子節點的位址。Drawn for this primer from Bhardwaj et al., NSDI 2026, §4.1.2 and §4.3.

Copies have no retransmission path, and one lost gradient makes the shadow diverge. Checkmate prevents loss instead: dedicated switch ports for shadow traffic, copies at the rate the training link already runs, and PFC on the shadow connections, with pause thresholds tuned for both clusters.

副本沒有重傳路徑，而遺失一個梯度就會讓影子節點偏離。Checkmate 改用預防：影子流量使用專用的交換器埠、副本以訓練鏈路本來的速率傳送，並在影子連線上啟用 PFC，為兩個叢集調校暫停門檻。

A shadow binds one channel set for the first rank and one for the last to its two NICs; NCCL's fixed schedule tells it which chunk arrives where. It buckets gradients as PyTorch DDP does and loops: receive, optimizer step, repeat. After a failure, every shadow sends its shard back in parallel, and the job repeats half an iteration on average.

影子節點把一組給第一個 rank、一組給最後一個 rank 的 channel 綁到兩張 NIC 上；NCCL 固定的排程告訴它哪個 chunk 會到哪裡。它像 PyTorch DDP 一樣把梯度分成 bucket，然後循環：接收、優化器步驟、重複。故障後，每個影子節點平行地把自己的分片送回去，任務平均重做半次迭代。

3.4 The cost model is Young–Daly (analysis)3.4 成本模型就是 Young–Daly（分析）

For 16,384 GPUs, the paper builds a cost model (Appendix B) from Llama 3's published inputs; nothing ran at that scale. With $\lambda$ failures per GPU-hour, $N$ GPUs, a run of $D$ hours, a checkpoint every $f$ iterations of $t$ hours and a stall $\omega$ per checkpoint, a conventional system wastes

對 16,384 顆 GPU，論文用 Llama 3 公開的輸入建立成本模型（附錄 B）；沒有任何東西在那個規模上執行過。設每 GPU 小時 $\lambda$ 次故障、 $N$ 顆 GPU、執行 $D$ 小時、每 $f$ 次迭代存一次檢查點、每次迭代 $t$ 小時、每個檢查點停頓 $\omega$ ，傳統系統浪費的是

\mathrm{Wasted}(f) = ND\left(\tfrac{1}{2}\lambda N f t + \frac{\omega}{f t}\right)

Its minimum, $f^{\star} t = \sqrt{2\omega/(\lambda N)}$ , is Chapter 9's $T \approx \sqrt{2C \cdot \mathrm{MTBF}}$ with $\mathrm{MTBF} = 1/(\lambda N)$ and $C = \omega$ . Checkmate wastes $\tfrac{1}{2}\lambda N^2 D t$ , half an iteration per failure, plus the cost of the CPU nodes. With the paper's inputs and Chapter 9's Llama 3 MTBF:

它的最小值 $f^{\star} t = \sqrt{2\omega/(\lambda N)}$ ，就是第 9 章的 $T \approx \sqrt{2C \cdot \mathrm{MTBF}}$ ，其中 $\mathrm{MTBF} = 1/(\lambda N)$ 、 $C = \omega$ 。Checkmate 浪費的是 $\tfrac{1}{2}\lambda N^2 D t$ ，即每次故障半次迭代，再加上 CPU 節點的成本。代入論文的輸入與第 9 章的 Llama 3 MTBF：

Question問題	Calculation算式	Result結果
Checkpoint of the 405B model at the paper's 2 + 4 bytes per parameter (Chapter 9 counts 14)以論文的每參數 2 + 4 位元組計的 405B 模型檢查點（第 9 章算 14）	$405 \times 10^{9} \times 6$	$\approx 2.43$ TB
Stall $\omega$ at Tectonic's 2 TB/s sustained rate以 Tectonic 持續 2 TB/s 計的停頓 $\omega$	$2.43 \div 2$	$\approx 1.2$ s
MTBF from 419 interruptions in 54 days (Section 9.3)由 54 天 419 次中斷得到的 MTBF（第 9.3 節）	$54 \times 86{,}400 \div 419$	$\approx 11{,}100$ s
Best conventional interval傳統做法的最佳間隔	$\sqrt{2 \times 1.2 \times 11{,}100}$	$\approx 163$ sabout 36 iterations of 4.58 s約 36 次 4.58 秒的迭代
Work a failure repeats, conventional傳統做法每次故障重做的工作	$163 \div 2$	$\approx 82$ s
Work a failure repeats, CheckmateCheckmate 每次故障重做的工作	$4.58 \div 2$	$\approx 2.3$ s

The 4.58 s iteration is the paper's FLOP model; Chapter 9's $6N$ estimate gives about 6.2 s, with the same conclusion. The paper's model outputs, not measurements, are over 300,000 wasted GPU-hours at its best interval (32 iterations) and 4,367 with Checkmate. Its hardware estimate: 128 DP groups need 256 multicast streams, so 256 extra switch ports and NICs and up to 128 CPU servers, about 0.31% of the cluster's cost.

4.58 秒的迭代時間是論文的 FLOP 模型；第 9 章的 $6N$ 估計給出約 6.2 秒，結論相同。論文的模型輸出（不是量測）是：在它的最佳間隔（32 次迭代）下浪費超過 300,000 GPU 小時，Checkmate 則是 4,367。它的硬體估計：128 個 DP 群組需要 256 條多播串流，所以要 256 個額外的交換器埠與 NIC，以及最多 128 台 CPU 伺服器，約佔叢集成本的 0.31%。

4. Implementation4. 實作

Checkmate is about 10,000 lines of Rust, C, C++, Python and P4; the training application is unchanged, PyTorch has about 50 changed lines and NCCL 2.22.3 about 10. A 5,000-line Rust NCCL plugin adds the tags over libtpa, ByteDance's DPDK-based TCP stack, and reaches 98.4 Gb/s of all-reduce bus bandwidth on a 100 Gb/s NIC. The shadow saturates two 100 Gb/s NICs. The switch runs about 800 lines of P4 on a 32-port Tofino-1. The transport is TCP, not RoCE.

Checkmate 約 10,000 行，使用 Rust、C、C++、Python 與 P4；訓練應用程式不需修改，PyTorch 約改 50 行，NCCL 2.22.3 約改 10 行。一個 5,000 行的 Rust NCCL 外掛在 libtpa（字節跳動建在 DPDK 上的 TCP 堆疊）之上加入標記，在 100 Gb/s NIC 上達到 98.4 Gb/s 的 all-reduce bus 頻寬。影子節點能跑滿兩張 100 Gb/s NIC。交換器在 32 埠的 Tofino-1 上執行約 800 行 P4。傳輸層是 TCP，不是 RoCE。

5. What was measured5. 量到了什麼

Setup: 16 machines on one 32-port Tofino-1 at 100 Gb/s; 12 training nodes with one A100 80 GB each, and up to 4 shadow nodes with a 28-core Xeon and two 100 Gb/s NICs. Nine models (Table 1): three vision models in FP32 and three GPT models in BF16 with 12-way DP and one shadow, and three LLaMA models at 2 PP × 6 DP with two shadows. Runs are 1,000 AdamW iterations into an in-memory null file system. Checkmate runs over DPDK, the other systems over NCCL's RoCEv2.

設定：16 台機器接在一台 100 Gb/s、32 埠的 Tofino-1 上；12 個訓練節點各有一顆 A100 80 GB，最多 4 個影子節點，各有一顆 28 核心的 Xeon 與兩張 100 Gb/s NIC。九個模型（Table 1）：三個 FP32 的視覺模型與三個 BF16 的 GPT 模型，12 路 DP、一個影子節點；三個 LLaMA 模型，2 PP × 6 DP、兩個影子節點。每次執行 1,000 次 AdamW 迭代，寫到記憶體中的空檔案系統。Checkmate 走 DPDK，其他系統走 NCCL 的 RoCEv2。

Compared with比較對象	Setup設定	Measured實測結果
No checkpoints不存檢查點	All nine models全部九個模型	Same throughput, with a checkpoint every iteration (Figure 6)吞吐相同，且每次迭代都存檢查點（Figure 6）
Torch Async, every 20 iterationsTorch Async，每 20 次迭代	Average over models各模型平均	About 5% lower throughput than Checkmate吞吐比 Checkmate 低約 5%
Torch Async, every iterationTorch Async，每次迭代	Vision / language averages視覺 / 語言模型平均	Checkmate 2.2× / 6.5× the throughputCheckmate 吞吐為其 2.2 倍 / 6.5 倍
CheckFreqCheckFreq	ResNet152ResNet152	1.05× the throughput, 15.3× the checkpoints吞吐 1.05 倍、檢查點 15.3 倍
CheckFreqCheckFreq	LLaMA2-13B, 2 PP × 6 DPLLaMA2-13B，2 PP × 6 DP	1.1× the throughput, 34.5× the checkpoints吞吐 1.1 倍、檢查點 34.5 倍
CheckFreqCheckFreq	All models全部模型	5–34.5× the checkpoints檢查點 5–34.5 倍
Gemini, every iterationGemini，每次迭代	Vision and GPT models視覺與 GPT 模型	1.32–2.7× the throughput, 2.2× on average吞吐 1.32–2.7 倍，平均 2.2 倍

Nine scatter panels, one per model, of average throughput versus number of checkpoints for No Checkpoint, Torch Async, CheckFreq, Gemini and Checkmate; Checkmate at the top right of each. — Figure 6 of the paper: average throughput against the number of checkpoints in 1,000 iterations, for nine models, measured on the 12-node A100 testbed behind one Tofino switch. Up and to the right is better. Checkmate's star sits at 1,000 checkpoints and near the no-checkpoint throughput; the other systems trade throughput for frequency.論文 Figure 6：九個模型在 1,000 次迭代中的平均吞吐對檢查點次數，在一台 Tofino 交換器下 12 個 A100 節點的測試平台上實測。越往右上越好。Checkmate 的星號位在 1,000 次檢查點、接近不存檢查點的吞吐；其他系統則以吞吐換取頻率。Screenshot of Bhardwaj et al., NSDI 2026, Figure 6 (USENIX PDF).

The abstract's "1.3–6.5× throughput at equal frequency" joins two baselines: 1.32× is against Gemini, 6.5× against Torch Async on language models. Repeated work per failure is half the checkpoint interval, so it falls in proportion:

摘要的「相同頻率下吞吐 1.3–6.5 倍」合併了兩個基準：1.32 倍是對 Gemini，6.5 倍是對語言模型上的 Torch Async。每次故障重做的工作是檢查點間隔的一半，所以等比例下降：

Question問題	Calculation算式	Result結果
LLaMA2-13B, 34.5× the checkpoints of CheckFreqLLaMA2-13B，檢查點為 CheckFreq 的 34.5 倍	$1 - 1/34.5$	$\approx 97.1\%$ less更少
Low end of the range, 5× the checkpoints範圍下限，檢查點 5 倍	$1 - 1/5$	$80\%$ less更少
Torch Async every 20 iterations: 10 vs 0.5 iterations repeatedTorch Async 每 20 次迭代：重做 10 次對 0.5 次	$1 - 0.5/10$	$95\%$ less更少
ResNet152, 15.3× the checkpoints of CheckFreqResNet152，檢查點為 CheckFreq 的 15.3 倍	$1 - 1/15.3$	$\approx 93.5\%$ less更少the paper states 91.5%論文寫的是 91.5%

Shadow speed. One shadow took about 25 ms per checkpoint for ResNet50 and under 600 ms for GPT3-XL (Figure 7). AdamW for GPT3-6.7B took 500 ms on 28 cores and 180 ms on 112 (Figure 8). Correctness. ResNet152, restored from the shadow every other iteration for 500 iterations, matched to the eighth decimal place. Network overhead. With tagged packets of a 1 GB all-reduce copied to 16 extra ports, bus bandwidth did not change, and the switch sent only 1.9× the frames it received.

影子節點的速度。一個影子節點處理 ResNet50 每次檢查點約 25 毫秒，GPT3-XL 不到 600 毫秒（Figure 7）。GPT3-6.7B 的 AdamW 在 28 核心上 500 毫秒，112 核心上 180 毫秒（Figure 8）。正確性。ResNet152 在 500 次迭代中每隔一次就從影子節點恢復，一直到小數點後第八位都相同。網路開銷。把 1 GB all-reduce 中標記的封包複製到 16 個額外的埠，bus 頻寬不變，交換器送出的幀數只有收到的 1.9 倍。

6. Limits and open questions6. 限制與未解的問題

The paper's own (§8): FSDP and ZeRO replace all-reduce by a reduce-scatter and an all-gather, which does not fit the design; public-cloud tenants cannot program switches; the cost advantage shrinks as failures become rarer. A careful reader should add:

論文自己列出的（§8）：FSDP 與 ZeRO 把 all-reduce 換成 reduce-scatter 加 all-gather，與設計不合；公有雲租戶無法程式化交換器；故障越少，成本優勢越小。細心的讀者還應補上：

Scale. 12 GPUs, one per node, one switch, 100 Gb/s. Rings that run partly over NVLink, multi-tier fabrics and 400G links are untested, and Llama 3 itself trained with FSDP.規模。12 顆 GPU、每節點一顆、一台交換器、100 Gb/s。部分走 NVLink 的 ring、多層網路與 400G 鏈路都沒有測試，而 Llama 3 本身是用 FSDP 訓練的。
No repair path. A lost or corrupted copy silently invalidates the shadow; detection, repair and shadow crashes are not described. Bit-exact state was checked on FP32 ResNet152 only.沒有修復路徑。一個遺失或損壞的副本會無聲地讓影子節點失效；論文沒有描述偵測、修復與影子節點當機。逐位元一致只在 FP32 的 ResNet152 上驗證過。
PFC couples the shadow to training. A slow shadow pauses its upstream port, which also carries ring traffic; see Section 9.1 and the Chapter 7, section 3. The evaluation never injects a slow shadow.PFC 把影子節點和訓練綁在一起。慢的影子節點會暫停它的上游埠，而那個埠同時承載 ring 流量；參見第 9.1 節與第 7.3 節。評估中從未注入一個慢的影子節點。
Schedule and transport. Tagging relies on NCCL's fixed ring or tree order, which in-network reduction such as SHARP would change. With RoCE, the switch would also have to rewrite packet sequence numbers and recompute the invariant CRC; the paper shows TCP only.排程與傳輸層。標記依賴 NCCL 固定的 ring 或 tree 順序，而 SHARP 之類的網內歸約會改變它。若用 RoCE，交換器還得改寫封包序號並重算不變 CRC；論文只展示了 TCP。
The scale case is analysis. At $\omega = 1.2$ s the model breaks even near $1.1 \times 10^{-7}$ failures per GPU-hour, about 1/180 of Llama 3's $2.0 \times 10^{-5}$ . Section 6.7 calls $10^{-6}$ per GPU-hour "0.5% of Meta's rate"; it is 5%.大規模案例是分析。在 $\omega = 1.2$ 秒時，模型在每 GPU 小時約 $1.1 \times 10^{-7}$ 次故障處打平，約為 Llama 3 的 $2.0 \times 10^{-5}$ 的 1/180。§6.7 說每 GPU 小時 $10^{-6}$ 是「Meta 故障率的 0.5%」；其實是 5%。

7. How it connects7. 與其他內容的關聯

Chapter 9, section 1: the counterexample to "checkpoints never touch the back end", acceptable only because the copies use dedicated ports. Section 3 and the Chapter 10, section 3: the Young–Daly limit, with $C \to 0$ for the GPUs and an interval of one iteration.第 9.1 節：「檢查點永遠不碰後端」的反例，只因副本走專用埠才站得住。第 3 節與第 10.3 節：Young–Daly 的極限，GPU 的 $C \to 0$ 、間隔為一次迭代。
Chapter 4, section 3: the ring's index arithmetic is exactly what heartbeat tagging uses.第 4.2 節：ring 的索引算術正是 heartbeat 標記所用的。
Previous digest: ByteCheckpoint keeps copy-persist, makes it cheap, and supports FSDP.上一篇導讀：ByteCheckpoint 保留複製後持久化、讓它變便宜，而且支援 FSDP。
Guided readings: Meta RoCE (PFC alone at 16K GPUs), Alibaba HPN (storage kept off the back end), MegaScale (the copy removed here), Llama 3 (the model's inputs), Microsoft RoCE and IRN (losslessness without retransmission).論文導讀：Meta RoCE（16K 顆 GPU 只靠 PFC）、Alibaba HPN（儲存留在後端之外）、MegaScale（這裡拿掉的那次複製）、Llama 3（模型的輸入）、Microsoft RoCE 與 IRN（只靠無損、沒有重傳）。

Key numbers關鍵數字

Number	數字	Meaning and setup
9.5× / 8.45× / 3.5×	Slowdown with sync / async / DCP checkpoints every iteration; GPT3-XL, 4 nodes	每次迭代以同步 / 非同步 / DCP 存檢查點的變慢倍數；GPT3-XL、4 個節點
5–34.5× · 80–97.1%	More checkpoints than CheckFreq · less repeated work per failure; 12 A100s	檢查點為 CheckFreq 的倍數 · 每次故障少重做的工作；12 顆 A100
2.2× / 6.5× · 1.32–2.7×	Throughput vs per-iteration Torch Async (vision / language) · vs Gemini	相對每次迭代的 Torch Async（視覺 / 語言）· 相對 Gemini 的吞吐
25 ms · 1.9×	Shadow checkpoint, ResNet50 · switch TX/RX frames at 16 copies	影子節點的檢查點，ResNet50 · 複製 16 份時交換器送出 / 收到的幀數比
2	Mirrored links and shadow NICs per DP group, for any ring size (design)	每個 DP 群組被鏡像的鏈路與影子 NIC 數，不論 ring 大小（設計）
256 · ≈0.31%	Extra switch ports · share of cluster cost, Llama 3 405B (the paper's estimate)	額外的交換器埠 · 佔叢集成本比例，Llama 3 405B（論文的估計）

Terms used on this page本頁用到的術語

Copy-persist: Copy state out of GPU memory, then persist it to storage.; 把狀態從 GPU 記憶體複製出來，再持久化到儲存。
Shadow node: A CPU server that replays every iteration's optimizer step on a copy of the state.; 在狀態副本上重播每次迭代優化器步驟的 CPU 伺服器。
Heartbeat tagging: Tags spread over the all-gather rounds so that each final chunk is tagged once.; 分散在各輪 all-gather 的標記，讓每個最終 chunk 只被標記一次。
Packet replication engine (PRE): The Tofino block that copies a packet to every member of a multicast group.; 把封包複製給多播群組每個成員的 Tofino 區塊。
Functional optimizer: An optimizer that updates each parameter from its own value, gradient and state.; 只用每個參數自己的值、梯度與狀態更新它的優化器。
DSCP: A traffic-class field in the IP header; Checkmate uses one bit as the tag.; IP 標頭中的流量類別欄位；Checkmate 用其中一個位元當標記。