Research digest 8 of 16 · Chapter 4: Collective communication

研究導讀 8 / 16 · 第 4 章：集體通訊

PReCCL: Performant and Resilient Collective Communication via Integrated Inband Telemetry and Workload ReallocationPReCCL：結合頻內遙測與工作量重新分配的高效能、具韌性集體通訊

SIGCOMM 2026Zhiyong Chen, Kaihui Gao, Li Chen et al. (Tsinghua University, Zhongguancun Laboratory, Alibaba Cloud) · paper論文

After reading, you can explain how PReCCL measures each NCCL channel's progress from inside the library and moves bytes away from a slow or failed path at every collective boundary.讀完後，你能說明 PReCCL 如何在函式庫內部量測每個 NCCL channel 的進度，並在每次集體操作的邊界，把位元組從變慢或故障的路徑移走。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 4 ↗第 4 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

NCCL splits every collective into channels, here called virtual topologies (VTs), over different NICs and paths, and gives each the same bytes: one congested VT sets the completion time, and a dead NIC aborts the job. PReCCL, a drop-in NCCL replacement, counts how often each VT's sender waits for FIFO credit, shares the counts on the collective's own traffic, and at every collective boundary lets all ranks compute the same new byte split. A failed NIC becomes a VT with zero bytes. On a 32-GPU testbed shared by three jobs, it raised bus bandwidth 1.8× (Ring-AllReduce) and 2.1× (AlltoAll) above 64 MB and made training 1.18–1.21× faster; in a separate fault test it kept 78% of bus bandwidth after one of eight NICs failed, where NCCL aborted.

NCCL 把每次集體操作拆成多個經由不同 NIC 與路徑的 channel（本論文稱為虛擬拓樸，VT），並給每個一樣多的位元組：一個壅塞的 VT 就決定完成時間，一張壞掉的 NIC 會讓任務中止。PReCCL 可直接替換 NCCL：它計算每個 VT 的送端等待 FIFO credit 的次數，把計數搭在集體操作自己的流量上分享，並在每次集體操作的邊界讓所有 rank 算出相同的新位元組分配。故障的 NIC 變成分到零位元組的 VT。在三個任務共用的 32 GPU 測試平台上，它讓 64 MB 以上的 bus bandwidth 提高 1.8 倍（Ring-AllReduce）與 2.1 倍（AlltoAll），訓練快 1.18–1.21 倍；在另一個故障實驗中，它在八張 NIC 壞一張時保有 78% 的 bus bandwidth，而 NCCL 直接中止。

1. Where this paper sits1. 這篇論文的位置

Section 4.5 shows NCCL's channels: each is one CUDA block with its own staging FIFO, and different channels leave through different NICs. NCCL cuts the message into equal parts, one per channel. Section 8 defines bus bandwidth (busbw), and section 9 ends with a theme: decide at run time what the host cannot know in advance. PReCCL applies that theme to the channel split. A collective communication task (CCT) is one call of a collective; each sub-task (channel) follows a VT, an ordered graph of GPUs on particular NICs; in AlltoAll, a VT is one GPU pair. The Theseus digest swaps whole schedules at run time; PReCCL keeps the schedule and changes how many bytes each VT carries.

第 4.5 節畫出 NCCL 的 channel：每個是一個帶有自己暫存 FIFO 的 CUDA block，不同 channel 經由不同 NIC 出去。NCCL 把訊息切成相等的幾份，每個 channel 一份。第 8 節定義 bus bandwidth（busbw），第 9 節以一個主題作結：在執行時決定主機事先無法知道的事。PReCCL 把這個主題用在 channel 的分配上。集體通訊任務（CCT）是一次集體操作呼叫；每個子任務（channel）沿著一個 VT 前進，也就是一張在特定 NIC 上的有序 GPU 圖；在 AlltoAll 中，一個 VT 是一對 GPU。Theseus 導讀在執行時抽換整個排程；PReCCL 則保留排程，改變每個 VT 承載的位元組數。

2. The problem, measured2. 問題，用量測說話

A collective ends when its slowest VT ends. If VT $i$ carries $D_i$ of the $D$ bytes at bandwidth $B_i$ :

集體操作在最慢的 VT 結束時才結束。若 VT $i$ 以頻寬 $B_i$ 承載 $D$ 位元組中的 $D_i$ ：

\begin{gathered} T_{\mathrm{CCT}} = \max_i \frac{D_i}{B_i} \\ D_i^{*} = \frac{B_i}{\sum_j B_j}\,D, \qquad T^{*} = \frac{D}{\sum_j B_j} \end{gathered}

NCCL's equal split, $D_i = D/N$ , takes $D/(N \min_i B_i)$ ; the split $D_i^{*}$ makes every VT finish at the minimum $T^{*}$ (Appendix A). In the paper's example, two VTs at 200 and 50 Gb/s finish 2.5× sooner with a 4:1 split than with 1:1 (Figure 1).

NCCL 的均分 $D_i = D/N$ 需要 $D/(N \min_i B_i)$ ；分配 $D_i^{*}$ 讓每個 VT 都在最小值 $T^{*}$ 完成（附錄 A）。在論文的例子中，兩個 VT 分別為 200 與 50 Gb/s，4:1 分配比 1:1 快 2.5 倍完成（圖 1）。

Figure 1. The paper's two-VT example. With an equal split, the fast VT finishes at 0.25 and idles until the slow one ends at 1.0; a 4:1 split, as their bandwidths are, ends both at 0.4, 2.5× sooner on the same links.圖 1。論文中兩個 VT 的例子。均分時，快的 VT 在 0.25 完成，閒置到慢的在 1.0 結束；依頻寬按 4:1 分配，兩者都在 0.4 結束，同樣的鏈路快 2.5 倍。Drawn for this primer from the PReCCL paper (Chen et al., SIGCOMM 2026), §1 (worked example; analysis, not a measurement).

A measured trace shows the effect (Figure 2): with NIC1 congested, NIC2–NIC4 finish their shares quickly and wait. Two weeks of traces from a large public cloud's multi-tenant GPU clusters show many training jobs sharing a physical link with another job, and Alibaba HPN reported 1–2 job crashes per month from timeouts caused by faulty NICs.

一筆實測紀錄顯示了這個效應（圖 2）：NIC1 壅塞時，NIC2–NIC4 很快送完自己那份，然後等待。某大型公有雲多租戶 GPU 叢集兩週的紀錄顯示，許多訓練任務與另一個任務共用實體鏈路；Alibaba HPN 也報告，壞掉的 NIC 造成的逾時每月導致 1–2 次任務崩潰。

Figure 3 of the PReCCL paper: throughput of NIC1 to NIC4 over about 18 seconds; NIC1 runs long and low while NIC2 to NIC4 show short bursts separated by idle gaps — Figure 2. Figure 3 of the paper, a measured trace of four NICs during collectives. Congested NIC1 runs long and low; NIC2–NIC4 send their equal shares in short bursts and then idle. The paper does not state the setup.圖 2。論文的 Figure 3：集體操作期間四張 NIC 的實測紀錄。壅塞的 NIC1 又長又低；NIC2–NIC4 以短暫突發送完均分的那份，然後閒置。論文沒有說明環境。Screenshot from Chen et al., SIGCOMM 2026, Figure 3 (measured).

Existing answers do not see the VT: congestion control and load balancing act on one flow or packet, solvers such as SyCCL plan before the job, and MCCS and FuseLink adapt GPU order or NICs without moving bytes between VTs. The paper's 512- and 1,024-GPU simulations are not quoted here.

現有的做法都看不到 VT：壅塞控制與負載平衡作用在單一 flow 或封包，SyCCL 等求解器在任務開始前規劃，MCCS 與 FuseLink 調整 GPU 順序或 NIC，卻不在 VT 之間搬移位元組。論文 512 與 1,024 GPU 的模擬，這裡不引用。

3. The design3. 設計

PReCCL never creates or destroys connections or channels. It changes only byte ranges (Ring, Tree) or relay choices (AlltoAll).

PReCCL 從不建立或拆除連線與 channel。它只改變位元組範圍（Ring、Tree）或轉送的選擇（AlltoAll）。

3.1 One contiguous block per VT3.1 每個 VT 一段連續區塊

NCCL interleaves: each loop step is cut into one chunk per VT, so all VTs advance through one ring together and none can finish early (Figure 3a). PReCCL gives each VT one contiguous block and runs an independent ring on it (Figure 3b). With this path isolation, a VT finishes when its own block is done, bytes can move between VTs without loss (Figure 3c), and each progress signal belongs to one VT.

NCCL 採交錯配置：每個迴圈步驟切成每個 VT 一個 chunk，所以所有 VT 一起走同一個環，誰都不能提早結束（圖 3a）。PReCCL 給每個 VT 一段連續區塊，並在上面跑一個獨立的環（圖 3b）。有了這種路徑隔離，VT 在自己的區塊做完時就結束，位元組可以在 VT 之間無損地移動（圖 3c），每個進度訊號也只屬於一個 VT。

Figure 3. The paper's Figure 9, redrawn with two VTs: (a) NCCL interleaves one chunk per VT in each loop step; (b) PReCCL gives each VT one block; (c) after VT0 slows, the next CCT moves the boundary.圖 3。依論文 Figure 9 以兩個 VT 重畫：(a) NCCL 在每個迴圈步驟交錯地給每個 VT 一個 chunk；(b) PReCCL 給每個 VT 一段區塊；(c) VT0 變慢後，下一個 CCT 移動邊界。Drawn for this primer from the PReCCL paper (Chen et al., SIGCOMM 2026), §3.3.2 and Figure 9.

3.2 Measure each VT by its FIFO stalls3.2 以 FIFO 停頓量測每個 VT

The sender at the head of a VT advances a FIFO's tail pointer, and the receiver at the far end advances the head pointer: the slot and credit mechanism of Section 4.5. When any hop slows (NVLink, PCIe, NIC, switch), the FIFO fills and the sender waits; while waiting, its GPU threads increment a shared-memory counter (Figure 4). The count covers the whole path and costs almost nothing. Direct timing was rejected because SM clocks are not synchronized. PReCCL converts the count $c$ for $b$ bytes into a time:

VT 起點的送端推進 FIFO 的 tail 指標，遠端的收端推進 head 指標：這就是第 4.5 節的槽與 credit 機制。任何一跳變慢（NVLink、PCIe、NIC、交換器），FIFO 就會填滿，送端只能等待；等待時，它的 GPU 執行緒會遞增一個共用記憶體計數器（圖 4）。這個計數涵蓋整條路徑，幾乎不花成本。直接計時被捨棄，因為 SM 的時脈不同步。PReCCL 把 $b$ 位元組對應的計數 $c$ 轉成時間：

T = \alpha\,c + \beta\,b + \Delta

Here $\alpha$ prices a stall, $\beta$ a byte and $\Delta$ fixed overhead. Fitted on more than 30,000 samples, the model explains 94% of the variance in sub-task time, and it is refit online every 10 s. The counts ride on the collective's own data: $N_{\text{rank}} \times N_{\text{VT}}$ bytes per CCT, 4 KB for 256 GPUs with 16 VTs.

其中 $\alpha$ 是停頓的代價， $\beta$ 是每位元組的代價， $\Delta$ 是固定開銷。以超過 30,000 個樣本擬合，這個模型解釋了子任務耗時 94% 的變異，並每 10 s 線上重新擬合。計數搭在集體操作自己的資料上：每個 CCT $N_{\text{rank}} \times N_{\text{VT}}$ 位元組，256 顆 GPU、16 個 VT 時是 4 KB。

Figure 4. The control loop. A full FIFO makes the sender wait and count stalls; at the end of CCT

e

the counts travel on the last synchronization step, become time estimates, and feed the same deterministic update on every rank.圖 4。控制迴圈。FIFO 滿了，送端就等待並計算停頓；在 CCT

e

結束時，計數搭在最後一次同步步驟上傳送，轉成時間估計，再輸入每個 rank 上相同的確定性更新。Drawn for this primer from the PReCCL paper (Chen et al., SIGCOMM 2026), §3.2, §3.4, Figures 5–7 and Appendix E.

3.3 Rebalance at the CCT boundary3.3 在 CCT 邊界重新平衡

Each CCT has an epoch number $e$ and a fixed allocation $A_e$ . Its last synchronization step carries the epoch-tagged counts; every rank checks the epoch, VT count and checksum, then computes $A_{e+1}$ with a deterministic function (fixed VT order, ties to the lower ID, integer rounding). No extra round trip is needed. The update comes from TeXCP, a 2005 traffic-engineering controller. With shares $w_i = D_i / \sum_j D_j$ and mean time $\bar T$ :

每個 CCT 有一個 epoch 編號 $e$ 與固定的分配 $A_e$ 。它最後一次同步步驟帶著標上 epoch 的計數；每個 rank 核對 epoch、VT 數與校驗和，然後以確定性函式（固定 VT 順序、平手取較小 ID、整數捨入）算出 $A_{e+1}$ ，不需額外往返。更新規則來自 2005 年的流量工程控制器 TeXCP。設占比 $w_i = D_i / \sum_j D_j$ 、平均時間為 $\bar T$ ：

\begin{gathered} \Delta D_i = w_i\,(\bar T - T_i) \;\;(+\,\epsilon \ \text{if } T_i = T_{\max}) \\ D_i \leftarrow D_i + \Delta D_i \end{gathered}

\begin{gathered} \Delta D_i = w_i\,(\bar T - T_i) \;\;(+\,\epsilon \ \text{若 } T_i = T_{\max}) \\ D_i \leftarrow D_i + \Delta D_i \end{gathered}

Slower-than-mean VTs lose bytes and faster ones gain, in a coordinated way, so the moves do not oscillate as per-VT AIMD does. Appendix A proves convergence to $T^{*}$ for fixed $B_i$ , but the rule has four gaps:

比平均慢的 VT 失去位元組、較快的得到，而且彼此協調，所以不會像各 VT 獨立的 AIMD 那樣擺盪。附錄 A 證明在 $B_i$ 固定時會收斂到 $T^{*}$ ，但這條規則有四個缺口：

Units. $w_i$ has no unit and $\bar T - T_i$ is a time, so $\Delta D_i$ is in seconds, not bytes. The appendix's step size $\eta$ ("implicitly 1") must carry bandwidth units, and its real value is not stated.單位。 $w_i$ 沒有單位， $\bar T - T_i$ 是時間，所以 $\Delta D_i$ 的單位是秒而不是位元組。附錄的步長 $\eta$ （「隱含為 1」）必須帶有頻寬單位，實際數值沒有說明。
Mean. The main text uses the plain mean, the appendix the share-weighted one; only the weighted mean makes the moves sum to zero.平均。正文用單純平均，附錄用依占比加權的平均；只有加權平均能讓移動總和為零。
$\epsilon$ makes the slowest VT lose slightly fewer bytes; the paper gives no value. $\epsilon$ 讓最慢的 VT 少失去一點位元組；論文沒有給數值。
Rate. The appendix bounds the imbalance $\Phi = \sum_i (T_i - T^{*})^2$ to shrink by at least $1 - 1/(\kappa N_{\text{VT}})$ per step, $\kappa = \max B_i / \min B_i$ , and claims more than 90% within 5 steps for $\kappa \le 4$ , $N_{\text{VT}} \le 8$ . The bound gives only about 15% (table below).速率。附錄給出不平衡度 $\Phi = \sum_i (T_i - T^{*})^2$ 每步至少縮小為 $1 - 1/(\kappa N_{\text{VT}})$ 倍的上界， $\kappa = \max B_i / \min B_i$ ，並宣稱在 $\kappa \le 4$ 、 $N_{\text{VT}} \le 8$ 時 5 步內減少 90% 以上。這個上界只給出約 15%（見下表）。

Question問題	Calculation算式	Result結果
Guaranteed factor per step, $\kappa = 4$ , $N_{\text{VT}} = 8$ 每步保證的倍數， $\kappa = 4$ 、 $N_{\text{VT}} = 8$	$1 - 1 \div (4 \times 8)$	$31/32 \approx 0.969$
Imbalance left after 5 steps5 步後剩下的不平衡度	$(31/32)^5$	$\approx 0.853$ a 15% reduction, not 90%只減少 15%，不是 90%
Steps the bound needs for a 90% reduction這個上界要減少 90% 所需的步數	$\ln 0.1 \div \ln(31/32)$	$\approx 73$ steps步

The panel runs this rule with the missing scale made explicit.

面板以明確補上的比例執行這條規則。

Panel 1面板 1

Re-weight the VTs, one collective at a time逐次集體操作，重新分配各 VT 的位元組

VT0's rate (its NIC)VT0 的速率（它的 NIC）

VTs, one per NICVT 數，每張 NIC 一個

Step size

\eta

(not stated in the paper)步長

\eta

（論文未說明）

Run執行

NCCL, equal split

NCCL，均分

PReCCL now

PReCCL 目前

Optimum,

D_i \propto B_i

最佳，

D_i \propto B_i

VT0's bytes

VT0 的位元組

Imbalance

\Phi

left

剩下的不平衡度

\Phi

Model: a 256 MB collective over $N$ VTs, each on one 400 Gb/s (50 GB/s) NIC, as on the testbed's 8-NIC servers. VT $i$ needs $D_i/B_i$ , and the slowest VT ends the CCT. Each run is one CCT: telemetry gives each $T_i$ (exact here; PReCCL estimates it from stall counts), and every rank applies $\Delta D_i = \eta\,D_i(\bar T - T_i)/\bar T$ with the share-weighted mean $\bar T$ , then renormalizes. This is the paper's update with the missing scale written out; $\eta$ is our choice. At 0%, VT0's NIC is down: the next CCT times out, VT0 is masked and its bytes are spread (Figure 5); raise the rate again and the probe lets VT0 rejoin. The $\Phi$ tile compares the imbalance left with what the paper's bound guarantees. Defaults: 8 VTs, as in the paper's Figure 10, and 256 MB, as in its ablation tests; 32% is the rate at which this model's converged gain over NCCL equals the measured 2.9× under 1% packet loss. Measured: 2.9× under loss; 78.0% kept with the NIC down (this model's ideal is 87.5%); reconvergence within 5 round trips. Try $\eta = 0.8$ : the slow VT drops to zero bytes, and because its share $w_i$ is then 0, the rule never gives it bytes back.

模型：一次 256 MB 的集體操作分給 $N$ 個 VT，每個 VT 用一張 400 Gb/s（50 GB/s）的 NIC，與測試平台上每台 8 張 NIC 的伺服器相同。VT $i$ 需要 $D_i/B_i$ ，最慢的 VT 決定 CCT 何時結束。每次執行就是一個 CCT：遙測給出每個 $T_i$ （這裡是精確值；PReCCL 由停頓次數估計），每個 rank 以依占比加權的平均 $\bar T$ 套用 $\Delta D_i = \eta\,D_i(\bar T - T_i)/\bar T$ ，再重新正規化。這就是論文的更新規則，只是把缺少的比例寫出來； $\eta$ 是我們的選擇。設為 0% 時 VT0 的 NIC 斷線：下一個 CCT 逾時，VT0 被遮罩，它的位元組被分掉（圖 5）；再把速率調高，探測就讓 VT0 歸隊。 $\Phi$ 那一格比較剩下的不平衡度與論文上界所保證的值。預設值：8 個 VT（同論文 Figure 10）與 256 MB（同論文的消融實驗）；32% 是讓本模型收斂後相對 NCCL 的增益等於 1% 掉包時實測 2.9 倍的速率。實測：掉包時 2.9 倍；NIC 斷線時保有 78.0%（本模型的理想值是 87.5%）；5 個往返內重新收斂。試試 $\eta = 0.8$ ：慢的 VT 掉到零位元組，而且因為它的占比 $w_i$ 變成 0，這條規則再也不會分給它位元組。

3.4 AlltoAll relays and failures3.4 AlltoAll 的轉送與故障

AlltoAll. Each AlltoAll VT serves one source–destination pair, so bytes cannot simply move to another VT. PReCCL relays part of a slow pair's bytes through one other GPU in the same node, over NVLink, which leaves $7N^2$ candidates on 8-GPU nodes. It rejects a relay whose cost exceeds the direct cost; on the testbed, relay was active in 23% of CCTs during congestion. Unlike NVIDIA's static PXN proxy, it picks the relay per CCT.

AlltoAll。每個 AlltoAll VT 服務一對特定的來源與目的地，位元組不能直接移到別的 VT。PReCCL 把慢的那一對的部分位元組，經 NVLink 由同一節點的另一顆 GPU 轉送，在 8 GPU 節點上候選有 $7N^2$ 個。轉送成本超過直接傳送時就放棄；在測試平台上，壅塞期間 23% 的 CCT 啟用轉送。與 NVIDIA 靜態的 PXN 代理不同，它每個 CCT 重新選擇轉送對象。

Failures. Degradation, such as a weak fiber, only slows VTs. When a NIC, optical module or switch port dies, PReCCL catches NCCL's timeout before it reaches the framework, broadcasts the failure over a socket channel, gives the dead VT zero bytes, spreads its share and re-runs the CCT (Figure 5). A probe at the start of each later CCT detects recovery. Losing all VTs, such as a GPU failure, still means checkpoint restart.

故障。劣化，例如光纖訊號變弱，只會讓 VT 變慢。當 NIC、光模組或交換器埠壞掉時，PReCCL 在 NCCL 的逾時傳到框架之前攔下它，經 socket 通道廣播故障，不再分給壞掉的 VT 任何位元組，把它那份分掉，並重跑 CCT（圖 5）。之後每個 CCT 開始時的探測會發現它已恢復。所有 VT 都失效時，例如 GPU 故障，仍須從檢查點重啟。

Figure 5. The paper's Figure 10, redrawn: eight VTs, one per RDMA NIC (RNIC). When RNIC0 fails, VT0 gets nothing and its 70 MB are spread over the other seven; a later probe lets it rejoin.圖 5。依論文 Figure 10 重畫：八個 VT，每個對應一張 RDMA NIC（RNIC）。RNIC0 故障時 VT0 不再分到資料，它的 70 MB 分給其他七個；之後的探測讓它歸隊。Drawn for this primer from the PReCCL paper (Chen et al., SIGCOMM 2026), §4 and Figure 10.

4. Implementation4. 實作

PReCCL is about 3,000 lines of C++ on NCCL v2.29, loaded through an environment variable with no framework change. Reallocation is off for small collectives: startup nccl-tests pick the smallest message whose latency is within 2% of NCCL's, which for 32 GPUs is 32 MB (Ring) and 8 MB (Tree). Below that, PReCCL behaves exactly like NCCL.

PReCCL 是建在 NCCL v2.29 上、約 3,000 行的 C++，以環境變數載入，不必修改框架。小型集體操作不做重新分配：啟動時的 nccl-tests 選出延遲與 NCCL 相差不到 2% 的最小訊息，32 顆 GPU 時是 32 MB（Ring）與 8 MB（Tree）。低於門檻時，PReCCL 的行為與 NCCL 完全相同。

5. What was measured5. 量到了什麼

The testbed has 4 servers with 8 Hopper GPUs and 8 ConnectX-7 400 Gb/s NICs each. Each server has its own top-of-rack (ToR) switch and 2 spines join them, so every inter-server flow crosses a spine. Three jobs share it: Qwen-MoE on 4 GPUs per server, GPT and BERT on 2 each. Baselines are NCCL v2.29, SyCCL and MCCS (Figure 6).

測試平台有 4 台伺服器，每台有 8 顆 Hopper GPU 與 8 張 400 Gb/s 的 ConnectX-7 NIC。每台伺服器有自己的機櫃頂端（ToR）交換器，由 2 台 spine 連接，所以每條跨伺服器 flow 都經過 spine。三個任務共用平台：Qwen-MoE 每台用 4 顆 GPU，GPT 與 BERT 各用 2 顆。基準是 NCCL v2.29、SyCCL 與 MCCS（圖 6）。

Figure 12 of the PReCCL paper: bus bandwidth of NCCL, MCCS, SyCCL and PReCCL from 4 MB to 4096 MB for Ring-AllReduce and AlltoAll, with PReCCL's speedup over NCCL as a line — Figure 6. Figure 12 of the paper, measured on the testbed with three co-located jobs: busbw of (a) Ring-AllReduce and (b) AlltoAll from 4 MB to 4 GB, speedup over NCCL on the right axis. The gap is already open at 16 MB.圖 6。論文的 Figure 12，在三個任務共用的測試平台上實測：(a) Ring-AllReduce 與 (b) AlltoAll 從 4 MB 到 4 GB 的 busbw，右軸是相對 NCCL 的加速比。差距在 16 MB 就已拉開。Screenshot from Chen et al., SIGCOMM 2026, Figure 12 (testbed).

Test測試	Setup設定	Measured實測結果
Training訓練	3 jobs on the testbed測試平台上 3 個任務	1.19× GPT (median 2.0 → 1.68 s), 1.21× Qwen-MoE (1.27 → 1.05 s), 1.18× BERT (0.71 → 0.60 s)GPT 1.19 倍（中位數 2.0 → 1.68 s）、Qwen-MoE 1.21 倍（1.27 → 1.05 s）、BERT 1.18 倍（0.71 → 0.60 s）
Collectives集體操作	same, above 64 MB同上，64 MB 以上	busbw 1.8× NCCL (Ring-AllReduce), 2.1× (AlltoAll)busbw 為 NCCL 的 1.8 倍（Ring-AllReduce）、2.1 倍（AlltoAll）
Overhead開銷	no contention, 64 MB無競爭，64 MB	0.11% (AlltoAll), 0.97% (Ring/Tree)0.11%（AlltoAll）、0.97%（Ring/Tree）
Signal訊號	256 MB, 16 GPUs256 MB、16 顆 GPU	Direct timing: 53% (Ring) and 46% (AlltoAll) slower medians直接計時：中位數慢 53%（Ring）與 46%（AlltoAll）
TeXCP vs AIMDTeXCP 對 AIMD	congestion in CCTs 11–20 of 3030 個 CCT 中第 11–20 個壅塞	TeXCP 78.5 → 125.1 GB/s, then 167 GB/s within 3 CCTs; AIMD keeps oscillatingTeXCP 78.5 → 125.1 GB/s，之後 3 個 CCT 內到 167 GB/s；AIMD 持續擺盪
Faults故障	1% loss on one NIC, then that NIC down and up一張 NIC 1% 掉包，接著斷線再恢復	About 2.9× NCCL under loss; NIC down: NCCL aborts, PReCCL keeps 78.0% within 5 round trips掉包時約為 NCCL 的 2.9 倍；斷線：NCCL 中止，PReCCL 在 5 個往返內保有 78.0%
Trace replay紀錄重播	300 Crux-trace jobs, 1,024 H200300 個 Crux 任務，1,024 顆 H200	37.1% of jobs at risk; with Crux placement, 5.5% lower median and 22.1% lower P95 job time37.1% 的任務有風險；搭配 Crux 配置，任務時間中位數降低 5.5%、P95 降低 22.1%
Production生產環境	1,024 H200, 2 weeks, 146 jobs1,024 顆 H200，2 週、146 個任務	5.7% lower median and 51.2% lower P95 job time than the 2 weeks before; 2.7% of jobs survived optical link flaps與前 2 週相比，任務時間中位數降低 5.7%、P95 降低 51.2%；2.7% 的任務撐過光鏈路閃斷

The fault experiment is the clearest result (Figure 7). With one of eight equal VTs gone, the ideal is $7/8 = 87.5\%$ of the bandwidth before the failure; PReCCL kept 78.0%.

故障實驗是最清楚的結果（圖 7）。八個相同的 VT 少了一個時，理想值是故障前頻寬的 $7/8 = 87.5\%$ ；PReCCL 保有 78.0%。

Figure 18 of the PReCCL paper: Ring-AllReduce bus bandwidth over 24 collectives for PReCCL, PReCCL without reallocation, and NCCL, through a packet-loss period, an RNIC failure and its recovery — Figure 7. Figure 18 of the paper, measured with nccl-tests Ring-AllReduce: 1% loss on one NIC, then the NIC goes down and comes back. NCCL is held at the lossy VT's pace, then aborts; the curve without reallocation shows what masking alone gives.圖 7。論文的 Figure 18，以 nccl-tests Ring-AllReduce 實測：一張 NIC 1% 掉包，接著斷線再恢復。NCCL 被掉包的 VT 拖住，然後中止；不重新分配的曲線顯示只靠遮罩能得到多少。Screenshot from Chen et al., SIGCOMM 2026, Figure 18 (testbed).

A note on busbw. The paper defines Ring-AllReduce busbw with the factor $\tfrac{N-1}{N}$ ; nccl-tests and Section 4.8 use $\tfrac{2(N-1)}{N}$ for all-reduce. At a fixed $N$ the factor cancels in ratios, so 1.8×, 2.9× and 78% stand; but the paper's Figure 18 comes from nccl-tests, which prints its own factor, and Figure 22 names nccl-tests only for its background load, so it is unclear which one their GB/s use. Do not compare them with all-reduce busbw from other sources.

關於 busbw。論文定義 Ring-AllReduce 的 busbw 時用的係數是 $\tfrac{N-1}{N}$ ；nccl-tests 與第 4.8 節對 all-reduce 用 $\tfrac{2(N-1)}{N}$ 。在固定的 $N$ 下，係數在比值中會消掉，所以 1.8 倍、2.9 倍與 78% 都成立；但論文的 Figure 18 來自 nccl-tests，而它印出的是自己的係數，Figure 22 則只說背景負載用 nccl-tests，所以不清楚圖上的 GB/s 用哪一個。別拿它們和其他來源的 all-reduce busbw 比較。

6. Limits and open questions6. 限制與未解的問題

The paper's own limits: a first call or a new group starts from the equal split; broadcast, gather and single-shot collectives fall back to NCCL. There is no support for custom collectives or for all-to-allv with uneven per-GPU sizes, so the skewed MoE dispatch of the collectives research notes, 9.4 is out of scope; the paper does not say how the dynamic AlltoAll of its Qwen-MoE runs fits this limit. Partitions and silent corruption are not handled, inter-job stability is shown for only two jobs, and the production trial is observational. Besides the gaps of section 3.3, a careful reader adds:

論文自己的限制：第一次呼叫或新群組從均分開始；broadcast、gather 與只執行一次的集體操作退回 NCCL。不支援自訂集體操作，也不支援各 GPU 大小不同的 all-to-allv，所以第 4 章第 9.4 節那種偏斜的 MoE dispatch 不在範圍內；論文沒有說明它 Qwen-MoE 實驗中的動態 AlltoAll 如何符合這個限制。網路分割與靜默損毀無法處理，任務間的穩定性只展示了兩個任務，生產環境試驗也是觀察性的。除了第 3.3 節的缺口，細心的讀者還應補上：

The testbed favors re-weighting: one ToR per server and two spines push all inter-server traffic through the spine, where the jobs collide.測試平台有利於重新分配：每台伺服器一台 ToR、只有兩台 spine，所有跨伺服器流量都經過 spine，各任務就在那裡相撞。
A small inconsistency: the counter is a shared-memory counter in §3.2 but "a hardware counter" in Appendix N.一個小的不一致：計數器在 §3.2 是共用記憶體計數器，在附錄 N 卻成了「硬體計數器」。
Many adaptive jobs: the proof assumes one job with fixed $B_i$ ; when many jobs move bytes off shared links at once, fairness among them is untested.許多會調適的任務：證明假設只有一個任務、 $B_i$ 固定；許多任務同時把位元組從共用鏈路移走時，它們之間的公平性沒有測試。

7. How it connects7. 與其他內容的關聯

Deep dive: the stall count watches section 5's FIFO slots and section 6's credit running out; section 8 explains the busbw factor.第 4 章：停頓計數觀察的是第 5 節的 FIFO 槽與第 6 節 credit 用盡的情形；第 8 節說明 busbw 的係數。
Research digests: NCCLX shrinks a 4K-GPU replica group on failure and caps bytes in flight statically per distance; Theseus, with the same Alibaba co-authors, swaps whole schedules under user policies.研究導讀：NCCLX 在故障時縮減 4K GPU 的 replica group，並依距離靜態地限制在途位元組；Theseus（有相同的 Alibaba 共同作者）依使用者策略抽換整個排程。
Guided readings: Alibaba HPN survives a port failure in hardware with dual ToRs, PReCCL in software; IRN explains why 1% loss with Go-Back-N cripples NCCL's lossy VT in Figure 7. The failures and sharing chapters cover restarts and placement.論文導讀：Alibaba HPN 以雙 ToR 在硬體上撐過埠故障，PReCCL 則在軟體上；IRN 說明為何 1% 掉包加上 Go-Back-N 會拖垮圖 7 中 NCCL 掉包的那個 VT。故障與共用第 4 章涵蓋重啟與配置。

Key numbers關鍵數字

Number	數字	Meaning (32-GPU testbed unless noted)
1.8× · 2.1×	busbw over NCCL above 64 MB, Ring-AllReduce · AlltoAll	64 MB 以上相對 NCCL 的 busbw，Ring-AllReduce · AlltoAll
1.18–1.21×	Faster training iterations, three co-located jobs	三個共用任務的訓練迭代加速
78.0%	busbw kept after one of eight NICs failed (NCCL aborts; ideal 87.5%)	八張 NIC 壞一張後保有的 busbw（NCCL 中止；理想 87.5%）
0.11% · 0.97%	Latency overhead at 64 MB, AlltoAll · Ring/Tree	64 MB 時的延遲開銷，AlltoAll · Ring/Tree
94%	Variance in sub-task time explained by the stall-count model	停頓計數模型解釋的子任務耗時變異
51.2%	Lower P95 job time in a 2-week production trial on 1,024 H200 (observational)	1,024 顆 H200 兩週生產環境試驗中 P95 任務時間的降幅（觀察性）

Terms used on this page本頁用到的術語

CCT: Collective communication task: one call of a collective.; 集體通訊任務：一次集體操作呼叫。
Virtual topology (VT)虛擬拓樸（VT）: The ordered graph of GPUs, over particular NICs and paths, that one sub-task (NCCL channel) follows.; 一個子任務（NCCL channel）所沿的、經由特定 NIC 與路徑的有序 GPU 圖。
Stall count停頓次數: How often a VT's sender found its FIFO full while the CCT ran; PReCCL's health signal.; CCT 執行期間，VT 的送端發現 FIFO 已滿的次數；PReCCL 的健康訊號。
Path isolation路徑隔離: One contiguous block per VT, so each VT runs its own ring and finishes on its own.; 每個 VT 一段連續區塊，讓每個 VT 跑自己的環、自行完成。
Epoch: A CCT's sequence number; the allocation is fixed within an epoch and recomputed identically on all ranks after it.; CCT 的序號；一個 epoch 內分配固定，之後所有 rank 以相同方式重新計算。