Research digest 5 of 16 · Chapter 4: Collective communication

研究導讀 5 / 16 · 第 4 章：集體通訊

AutoCCL: Automated Collective Communication Tuning for Accelerating Distributed and Parallel DNN TrainingAutoCCL：自動調校集體通訊，加速分散式與平行 DNN 訓練

NSDI 2025Guanbin Xu, Zhihao Le, Yinhe Chen et al. (University of Science and Technology of China, Microsoft Research) · paper論文

After reading, you can explain which NCCL knobs set a collective's speed, why the best setting moves when a GEMM runs beside it, and how AutoCCL finds it during training.讀完後，你能說明哪些 NCCL 參數決定集體操作的速度、為何旁邊跑著 GEMM 時最佳設定會移動，以及 AutoCCL 如何在訓練中把它找出來。

≈ 10 min · 1 interactive panel · 5-question quiz≈ 10 分鐘 · 1 個互動面板 · 5 題測驗Chapter 4 ↗第 4 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

NCCL sets six knobs per collective from a fixed cost model, and on A40 servers the result was far from the best. A 64 MB AllReduce ran at 4.0 GB/s with NCCL's Ring and 8.9 GB/s with a tuned Tree. A GEMM beside an 80 MB AllGather cut NCCL's default from 30.08 to 18.26 GB/s, while a setting tuned under that GEMM kept 32.44 GB/s. AutoCCL tries the 12 combinations of the few-valued knobs and searches the three resource knobs in each by coordinate descent, with every trial on the job's own repeated collectives. On up to 32 A40 GPUs: 1.24–1.29× NCCL's bandwidth in microbenchmarks, and training iterations 1.07–1.32× faster for three LLMs, tuning included.

NCCL 依固定的成本模型為每次集體操作設定六個參數，在 A40 伺服器上結果離最佳很遠。64 MB 的 AllReduce 用 NCCL 的 Ring 只有 4.0 GB/s，用調校後的 Tree 達 8.9 GB/s。80 MB 的 AllGather 旁邊跑一個 GEMM，NCCL 的預設從 30.08 掉到 18.26 GB/s，而在同一 GEMM 下調校出的設定仍有 32.44 GB/s。AutoCCL 試遍選項少的三個參數的 12 種組合，在每種組合內用座標下降搜尋三個資源參數，每次試驗都跑在任務自己重複的集體操作上。在最多 32 顆 A40 GPU 上：微基準頻寬為 NCCL 的 1.24–1.29 倍，三個 LLM 的訓練迭代快 1.07–1.32 倍，已含調校時間。

1. Where this paper sits1. 這篇論文的位置

Section 4.5 describes what AutoCCL tunes: channels, each one thread block on its own SM, and protocols. NCCL picks algorithm and protocol from an $\alpha$ – $\beta$ cost model with built-in tables, and Chapter 4 warns that every threshold is wrong at once if the fabric differs from the tables.

第 4.5 節描述了 AutoCCL 調校的對象：channel（每個都是在自己 SM 上的 thread block）與協定。NCCL 用帶內建表格的 $\alpha$ – $\beta$ 成本模型挑選演算法與協定；第 4 章警告，若網路與表格不符，所有門檻會同時出錯。

The authors read NCCL's code. Of 158 parameters, 93 are undocumented and 28 change performance. They group the 28 into six knobs per collective (Figure 1): algorithm A, protocol P, transport T (P2P, a direct GPU-to-GPU copy, or SHM, staging through host memory), channels NC (1–128), threads per thread block NT (32–640) and chunk size C (256 B to about 2 MB). Until then, tuning meant environment variables set by hand, as at Meta and for Llama 3, or AFNFA, a random-forest tuner trained offline. Missing was a measured answer: how far NCCL's choice is from the best, why the best moves during a job, and how to find it quickly.

作者閱讀 NCCL 的程式碼：158 個參數中，93 個沒有文件，28 個會影響效能。他們把這 28 個歸成每次集體操作的六個參數（圖 1）：演算法 A、協定 P、傳輸方式 T（P2P 是 GPU 對 GPU 的直接複製，SHM 是經主機記憶體中轉）、channel 數 NC（1–128）、每個 thread block 的執行緒數 NT（32–640），以及 chunk 大小 C（256 B 到約 2 MB）。在此之前，調校靠的是手動設定環境變數（如 Meta 與 Llama 3 的做法），或 AFNFA：一個離線訓練的隨機森林調校器。缺少的是實測的答案：NCCL 的選擇離最佳多遠、最佳設定為何會在任務中移動、以及如何快速找到它。

Figure 1. The six knobs of one collective (§2.3, Table 2). Blue: implementation parameters, which change what runs. Orange: resource parameters, which set how much hardware runs; each channel is one thread block on an SM.圖 1。一次集體操作的六個參數（§2.3，Table 2）。藍色：實作參數，決定執行什麼。橘色：資源參數，決定用多少硬體；每個 channel 是一個占用 SM 的 thread block。Drawn for this primer from Xu et al., NSDI 2025, §2.3 and Table 2.

2. The problem, measured2. 問題，用量測說話

On servers with eight A40 GPUs, single collectives showed the gap (§3.1, Tables 3–5):

在每台有八顆 A40 GPU 的伺服器上，單一集體操作就顯示出差距（§3.1，Tables 3–5）：

80 MB AllGather over PCIe. The best of four settings was 1.23× NCCL's default and 2.69× the worst, with more and smaller pieces: 8 channels and 32 KB chunks, not 2 and 2 MB.PCIe 上 80 MB 的 AllGather。四種設定中最好的是 NCCL 預設的 1.23 倍、最差的 2.69 倍，切得更多、更小：8 個 channel、32 KB chunk，而非 2 個、2 MB。
The same AllGather over NVLink bridges. The best setting (SHM, 64 channels, 108 KB chunks) was 1.28× the default; with P2P, 64 channels fell below the default to about 23 GB/s, although the text calls the default the worst.同一個 AllGather 走 NVLink 橋接。最佳設定（SHM、64 個 channel、108 KB chunk）是預設的 1.28 倍；改用 P2P 時，64 個 channel 掉到約 23 GB/s，低於預設，雖然正文說預設最差。
64 MB AllReduce on 16 GPUs over PCIe. NCCL's Ring ran at 4.0 GB/s, Tree with the same resources at 5.4 GB/s, and Tree with 8 channels, 160 threads and 59 KB chunks at 8.9 GB/s. On a 15 MB AllReduce over 8 GPUs, that Tree setting fell below a Ring setting (8.1 against 8.8 GB/s): the best setting does not carry over.16 顆 GPU、經 PCIe 的 64 MB AllReduce。NCCL 的 Ring 為 4.0 GB/s，相同資源的 Tree 為 5.4 GB/s，Tree 加 8 個 channel、160 個執行緒、59 KB chunk 為 8.9 GB/s。在 8 顆 GPU 的 15 MB AllReduce 上，這個 Tree 設定反而低於某個 Ring 設定（8.1 對 8.8 GB/s）：最佳設定無法直接沿用。

Figure 1 of the AutoCCL paper: bandwidth of an 80 MB AllGather on 8 A40 GPUs under four NCCL configurations, PCIe on the left and NVLink on the right, with 1.23x and 1.28x gains over the default marked — Figure 1 of the paper: an 80 MB AllGather on 8 A40 GPUs under the four settings of Tables 3 and 4, measured; left PCIe, right NVLink. The dotted line is NCCL's default. On the right, $C_{N1}$ (P2P, 64 channels) is below the default.論文的 Figure 1：8 顆 A40 上 80 MB 的 AllGather，採用 Tables 3 與 4 的四種設定，實測；左為 PCIe，右為 NVLink。虛線是 NCCL 的預設。右圖中 $C_{N1}$ （P2P、64 個 channel）低於預設。Screenshot of Xu et al., NSDI 2025, Figure 1 (PDF p. 5).

Next, a GEMM (matrix multiply plus sigmoid) ran beside an 80 MB AllGather over NVLink (§3.2, Table 6); both compete for SMs, cache and memory bandwidth. The GEMM sizes are not given.

接著，在 NVLink 上 80 MB 的 AllGather 旁邊跑一個 GEMM（矩陣乘法加 sigmoid）（§3.2，Table 6）；兩者爭用 SM、快取與記憶體頻寬。論文沒有給出 GEMM 的大小。

Beside the AllGatherAllGather 旁邊	NCCL defaultNCCL 預設	Tuned for that load針對該負載調校
Nothing沒有其他工作	30.08 GB/s	38.62 GB/s
Light GEMM輕量 GEMM	26.21 GB/s	35.14 GB/s
Heavy GEMM重量 GEMM	18.26 GB/s	32.44 GB/s

What these numbers mean (our arithmetic; the paper states the 39.3%):

這些數字的意義（我們的計算；39.3% 是論文給的）：

Question問題	Calculation算式	Result結果
Default's loss under the heavy GEMM重量 GEMM 下預設的損失	$1 - 18.26 \div 30.08$	$\approx 39.3\%$
Tuned setting's loss under the heavy GEMM重量 GEMM 下調校後設定的損失	$1 - 32.44 \div 38.62$	$\approx 16.0\%$
Tuned under the heavy GEMM vs default with no GEMM重量 GEMM 下調校後對無 GEMM 時的預設	$32.44 \div 30.08$	$\approx 1.08\times$

A setting tuned under a GEMM even beats the default without one, by 7.8–16.8%: the best setting depends on what runs beside the collective (Figure 2). And the space is large: up to about 5 million combinations per primitive and size, and one exhaustive test took several hours.

在 GEMM 下調校出的設定，甚至比沒有 GEMM 時的預設快 7.8–16.8%：最佳設定取決於集體操作旁邊在跑什麼（圖 2）。而且空間很大：每種原語與大小最多約 500 萬種組合，一次窮舉測試就要好幾個小時。

Figure 2. Two measured gaps (Tables 5 and 6, A40 testbed). Left: NCCL picks Ring; Tree with more channels is 2.2× faster. Right: a GEMM costs NCCL's default 39%, while a setting tuned under the same GEMM loses 16%.圖 2。兩個實測的差距（Tables 5 與 6，A40 測試平台）。左：NCCL 選了 Ring；Tree 加更多 channel 快 2.2 倍。右：GEMM 讓 NCCL 預設損失 39%，在同一 GEMM 下調校出的設定只損失 16%。Drawn for this primer from Xu et al., NSDI 2025, Tables 5 and 6.

3. The design3. 設計

3.1 Two kinds of knobs3.1 兩類參數

AutoCCL splits the knobs (§4.1). The implementation parameters A, P and T have few values and change what runs; the resource parameters NC, NT and C have many values and set how much hardware runs. Each combination of A, P and T is a subspace, $2 \times 3 \times 2 = 12$ in all (our count). AutoCCL searches the resource knobs in each subspace and keeps the best (Algorithm 1). With 128 channel counts and about 8,192 chunk sizes, this needs a shape to exploit.

AutoCCL 把參數分成兩類（§4.1）。實作參數 A、P、T 選項少，決定執行什麼；資源參數 NC、NT、C 選項多，決定用多少硬體。A、P、T 的每一種組合是一個子空間，共 $2 \times 3 \times 2 = 12$ 個（我們的計算）。AutoCCL 在每個子空間內搜尋資源參數，保留最好的（Algorithm 1）。有 128 種 channel 數、約 8,192 種 chunk 大小，這需要一個可以利用的形狀。

3.2 A model with a sweet point3.2 有最佳點的模型

The model has two phases that run in turn on each piece of data (§4.2). Phase 0, the transport, copies a peer's data into a local buffer; its granularity is the data in flight, $NC \cdot C$ . Phase 1, the protocol, loads the data into the SMs, reduces it and stores it; its granularity is the threads in flight, $NC \cdot NT$ . The slower phase sets the bandwidth:

模型有兩個階段，對每一份資料依序執行（§4.2）。階段 0 是傳輸，把對端的資料複製到本地緩衝區，粒度是在途資料量 $NC \cdot C$ 。階段 1 是協定，把資料載入 SM、做歸約、再存回，粒度是在途執行緒數 $NC \cdot NT$ 。較慢的階段決定頻寬：

\beta = \min\big(\beta_0(NC,C),\ \beta_1(NC,NT)\big)

Each step of phase 0 pays a start-up cost $\alpha_0$ , then moves $NC \cdot C$ bytes at the peak bandwidth $\hat\beta_0$ times a congestion factor $\gamma$ . Phase 1 has the same form; the paper omits its derivation:

階段 0 的每一步先付啟動成本 $\alpha_0$ ，再以峰值頻寬 $\hat\beta_0$ 乘上壅塞係數 $\gamma$ 搬 $NC \cdot C$ 位元組。階段 1 形式相同；論文省略了它的推導：

\beta_0 = \frac{NC \cdot C}{\alpha_0 + \dfrac{NC \cdot C}{\hat\beta_0\,\gamma}}, \qquad \beta_1 = \frac{NC \cdot NT}{\alpha_1 + \dfrac{NC \cdot NT}{\hat\beta_1\,\gamma}}

Bigger pieces hide the start-up cost, so bandwidth rises; then contention for links, SMs and memory makes it level off or fall. Bandwidth is thus a unimodal function of each knob, with a sweet point that moves when the other knobs change. The paper says $\gamma$ grows with the knobs, which as printed would raise bandwidth; we read $\hat\beta\gamma$ as the bandwidth left under congestion.

較大的片段能藏住啟動成本，所以頻寬上升；接著鏈路、SM 與記憶體的爭用讓它持平或下降。因此頻寬是每個參數的單峰函數，其最佳點會隨其他參數移動。論文說 $\gamma$ 隨參數增加，照式子印出的寫法這會提高頻寬；我們把 $\hat\beta\gamma$ 解讀為壅塞下剩下的頻寬。

The paper's Figure 3 measured this shape for an 80 MB AllGather: the best channel count was 4 in one setting and 16 in another.

論文的 Figure 3 對 80 MB 的 AllGather 量到了這個形狀：最佳 channel 數在一種設定下是 4，另一種下是 16。

3.3 Coordinate descent inside each subspace3.3 在每個子空間內做座標下降

Coordinate descent exploits this shape (Algorithm 2, Figure 3). From a random setting, it raises one knob by a step and profiles the result. After a gain, the next step equals the relative gain; after a loss, it moves to the next knob and resets the step to 0.01. It stops when no knob helps. The paper reports no trial counts and no comparison with exhaustive search.

座標下降利用了這個形狀（Algorithm 2，圖 3）。它從一組隨機設定出發，把某個參數加一步並量測。有增益時，下一步等於相對增益；變差時，換下一個參數並把步長重設為 0.01。沒有參數能再改善時就停止。論文沒有回報試驗次數，也沒有與窮舉搜尋比較。

Figure 3. Algorithms 1 and 2, drawn from §4.1–§4.3. Left: the 12 subspaces. Right: bandwidth contours over two resource knobs; each dot is one profiled call, and each leg moves one knob until it stops helping (NT not drawn).圖 3。依 §4.1–§4.3 繪製的 Algorithms 1 與 2。左：12 個子空間。右：兩個資源參數上的頻寬等高線；每個點是一次量測的呼叫，每段路徑只移動一個參數，直到它不再有幫助（未畫出 NT）。Drawn for this primer from Xu et al., NSDI 2025, §4.1–§4.3.

The panel computes this model for two measured tasks. The paper gives no constants; ours are fitted so that NCCL's default and the best setting reproduce Tables 5 and 6. Values in between come from the model.

面板針對兩個有實測的任務計算這個模型。論文沒有給出常數；我們擬合常數，使 NCCL 預設與最佳設定重現 Tables 5 與 6。兩者之間的數值來自模型。

Panel 1面板 1

Tune one collective, with and without a GEMM beside it調校一次集體操作：旁邊有沒有 GEMM

Task (80 MB · 64 MB)任務（80 MB · 64 MB）

Beside it旁邊在跑

Algorithm A演算法 A

Protocol P協定 P

Transport T傳輸方式 T

Channels NCchannel 數 NC

Threads per block NT每個 block 的執行緒 NT

Chunk size Cchunk 大小 C

Bandwidth (model)

頻寬（模型）

Slower phase

較慢的階段

Best setting, this load

此負載下的最佳設定

Coordinate descent

座標下降

Curves: bandwidth against the number of channels for your protocol, transport, threads and chunk size, one curve per load (bold: the load you chose; dot: your setting; ring: each curve's peak). The model is the paper's Eqs. 1–5, with the congested peak shrinking as $NC \cdot C$ or $NC \cdot NT$ grows and, for P2P, as NC grows; the constants are ours, fitted so that the measured points printed above come out right. In the model a GEMM lowers the protocol phase's peak and moves both contention knees to smaller settings, because it takes SMs, cache and memory bandwidth (§3.2); the paper does not say in which direction the best setting moves, so read the direction as the model's. The search is a simplified coordinate descent in every subspace: step one knob up or down while bandwidth improves, then the next knob. The paper starts from a random point; we start from the middle of each range or from NCCL's default. From the default (2 MB chunks) it stalls below the best: phase 0 depends only on $NC \cdot C$ , so trading chunk size for channels is a diagonal move that one-knob steps cannot make. Try: press "Table's best row", then switch to the heavy GEMM. The peak moves from 32 channels to 16, and the row's 64 channels fall to 23.4 GB/s.

曲線：在你選的協定、傳輸方式、執行緒數與 chunk 大小下，頻寬對 channel 數的關係，每種負載一條（粗線：你選的負載；實心點：你的設定；圓圈：各曲線的峰值）。模型是論文的 Eqs. 1–5，壅塞下的峰值隨 $NC \cdot C$ 或 $NC \cdot NT$ 變大而縮小，P2P 還會隨 NC 變大而縮小；常數是我們擬合的，使上方列出的實測點都能重現。模型中 GEMM 會降低協定階段的峰值，並把兩個爭用拐點移到更小的設定，因為它占用 SM、快取與記憶體頻寬（§3.2）；論文沒有說最佳設定往哪個方向移動，所以方向請視為模型的結果。搜尋是在每個子空間內做的簡化座標下降：頻寬變好時把一個參數往上或往下走一步，再換下一個參數。論文從隨機點出發；我們從各範圍的中點或 NCCL 預設出發。從預設（2 MB chunk）出發時，它會停在最佳之下：階段 0 只取決於 $NC \cdot C$ ，用 chunk 大小換 channel 數是一個對角方向的移動，一次只動一個參數做不到。試試看：按「表中最佳列」，再切到重量 GEMM。峰值從 32 個 channel 移到 16 個，而該列的 64 個 channel 掉到 23.4 GB/s。

3.4 Tuning online, one trial per call3.4 線上調校：每次呼叫一次試驗

Interference depends on operators, partitioning, inputs and scheduling, so the paper rejects offline prediction and uses repetition instead (Table 1, 32 GPUs). Per iteration, Phi-2-2B runs an 80 MB AllGather 3,120 times and Yi-1.5-34B a 56 MB AllGather 61,440 times; VGG-19 runs 6 AllReduces of 15–392 MB. A task is keyed by primitive, message size and group, and each call runs one search step. In this online tuning, every trial runs beside the real GEMMs.

干擾取決於運算子、切分、輸入與排程，所以論文不採用離線預測，改為利用重複（Table 1，32 顆 GPU）。每次迭代中，Phi-2-2B 執行 80 MB 的 AllGather 3,120 次，Yi-1.5-34B 執行 56 MB 的 AllGather 61,440 次；VGG-19 執行 6 次 15–392 MB 的 AllReduce。任務以原語、訊息大小與群組為鍵，每次呼叫執行搜尋的一步。在這種線上調校中，每次試驗都在真實的 GEMM 旁邊執行。

Figure 4. Online tuning (after Figures 4–6 of the paper). Each call carries one search step (yellow); later calls use the best setting (green). The Optimizer line is the paper's example.圖 4。線上調校（依論文 Figures 4–6 繪製）。每次呼叫承載一步搜尋（黃色）；之後的呼叫使用最佳設定（綠色）。Optimizer 那一行是論文的例子。Drawn for this primer from Xu et al., NSDI 2025, §5.1–§5.3 and Figures 4–6.

AutoCCL adds one Leader per group (Figure 4, and Figure 4(b) of the paper below). Its Optimizer times the calls and searches alone; its Coordinator broadcasts each new setting with synchronous semantics, so no rank runs a mixed version. In the paper's example, NT 16 → 32 cut a 40 MB AllGather from 16 to 14 ms, so the Optimizer tried NT = 64. After convergence the search does not restart.

AutoCCL 為每個群組設一個 Leader（圖 4，以及下方論文的 Figure 4(b)）。它的 Optimizer 為呼叫計時並獨自搜尋；Coordinator 以同步語意廣播每個新設定，所以不會有 rank 跑混合版本。在論文的例子中，NT 16 → 32 讓 40 MB 的 AllGather 從 16 ms 降到 14 ms，於是 Optimizer 試 NT = 64。收斂後搜尋不會重新開始。

Figure 4(b) of the AutoCCL paper: a communication group with a Leader holding Optimizer, Coordinator, Executor and config table, and a Worker with Executor and config table — Figure 4(b) of the paper: AutoCCL's architecture (a design diagram). Each rank's Executor looks up a task's default or tuned configuration; only the Leader runs the Optimizer and the Coordinator.論文的 Figure 4(b)：AutoCCL 的架構（設計示意圖）。每個 rank 的 Executor 查詢任務的預設或調校後設定；只有 Leader 執行 Optimizer 與 Coordinator。Screenshot of Xu et al., NSDI 2025, Figure 4(b) (PDF p. 9).

4. Implementation4. 實作

AutoCCL adds 9,176 lines of C++ to NCCL 2.18.3, in its configuration-generation and transport-initialization modules (§5.4). Extra threads on one node per group tune and time the kernels; the Coordinator uses Linux sockets. Users preload the library with no code changes (PyTorch 2.1, Megatron-LM). It supports AllGather, ReduceScatter and AllReduce and is open source.

AutoCCL 在 NCCL 2.18.3 的組態產生與傳輸初始化模組中加入 9,176 行 C++（§5.4）。每個群組一個節點上的額外執行緒負責調校並為 kernel 計時；Coordinator 使用 Linux socket。使用者以 preload 載入函式庫，不必改程式（PyTorch 2.1、Megatron-LM）。它支援 AllGather、ReduceScatter 與 AllReduce，並已開源。

5. What was measured5. 量到了什麼

All results come from two A40 testbeds (§6.1). Cluster A: 2 nodes of 8 PCIe GPUs, joined in pairs by 400 Gb/s NVLink bridges ("NVLink" below, not NVSwitch), 2 × 400 Gb/s InfiniBand per node. Cluster B: 4 machines of 8 PCIe GPUs, 100 Gb/s InfiniBand. Baselines: NCCL 2.18.3 with its built-in tuning, and AFNFA as reproduced by the authors.

所有結果都來自兩個 A40 測試平台（§6.1）。Cluster A：2 個節點，各 8 顆 PCIe GPU，兩兩以 400 Gb/s NVLink 橋接相連（下文的「NVLink」，不是 NVSwitch），每節點 2 × 400 Gb/s InfiniBand。Cluster B：4 台機器，各 8 顆 PCIe GPU，100 Gb/s InfiniBand。基準：帶內建調校的 NCCL 2.18.3，以及作者重現的 AFNFA。

Test測試	Setup設定	Measured實測結果
AllGather, ReduceScatterAllGather、ReduceScatter	8 GPUs, 1 MB–1 GB8 顆 GPU，1 MB–1 GB	PCIe: +22.66%, +27.52% average over NCCL (AFNFA ≈ NCCL). NVLink: 1.38×, 1.39×PCIe：平均比 NCCL 高 22.66%、27.52%（AFNFA ≈ NCCL）。NVLink：1.38、1.39 倍
AllReduceAllReduce	PCIe 8–32, NVLink 8–16 GPUsPCIe 8–32、NVLink 8–16 顆 GPU	1.28× NCCL, 1.15× AFNFA on average平均為 NCCL 的 1.28 倍、AFNFA 的 1.15 倍
With a GEMM有 GEMM 時	128 MB; GEMM 3,456 × 128 × 3,456128 MB；GEMM 3,456 × 128 × 3,456	AllGather 1.29×, ReduceScatter 1.50×, AllReduce 1.38× NCCL; AFNFA 1.00×, 1.00×, 1.02×AllGather 1.29 倍、ReduceScatter 1.50 倍、AllReduce 1.38 倍（相對 NCCL）；AFNFA 1.00、1.00、1.02 倍
GEMM, size sweep有 GEMM，掃描大小	8 GPUs, NVLink, above 32 MB8 顆 GPU、NVLink、32 MB 以上	AllGather 1.11–1.76×, AllReduce 1.16–1.39× NCCLAllGather 1.11–1.76 倍、AllReduce 1.16–1.39 倍（相對 NCCL）
Training訓練	4 models, up to 32 GPUs, tuning included4 個模型、最多 32 顆 GPU、含調校	1.02–1.32× per iteration; average +16.7% vs NCCL, +14.5% vs AFNFA每次迭代 1.02–1.32 倍；平均比 NCCL 快 16.7%、比 AFNFA 快 14.5%
Convergence收斂	LLMs; VGG-19LLM；VGG-19	A few iterations; at most 10 minutes幾次迭代；最多 10 分鐘

The paper's Figure 11 shows the training runs. The number over each bar is the speedup over NCCL. AutoCCL reached 1.10–1.32× on Phi-2-2B, 1.07–1.18× on Llama-3.1-8B, 1.25× on Yi-1.5-34B and 1.02–1.20× on VGG-19, ahead of AFNFA (0.95–1.18×) in every setup. The abstract's 1.07–1.32× covers the three LLMs; VGG-19 on 8 PCIe GPUs gained only 1.02×.

論文的 Figure 11 呈現訓練結果，每根長條上的數字是相對 NCCL 的加速倍數。AutoCCL 在 Phi-2-2B 上達 1.10–1.32 倍，Llama-3.1-8B 1.07–1.18 倍，Yi-1.5-34B 1.25 倍，VGG-19 1.02–1.20 倍，在每種設定下都領先 AFNFA（0.95–1.18 倍）。摘要的 1.07–1.32 倍涵蓋三個 LLM；8 顆 PCIe GPU 上的 VGG-19 只快 1.02 倍。

Figure 11 of the AutoCCL paper: iteration time of Phi-2-2B, Llama-3.1-8B, Yi-1.5-34B and VGG-19 with NCCL, AFNFA and AutoCCL on PCIe and NVLink clusters, with speedup labels — Figure 11 of the paper: iteration time (lower is better) with NCCL, AFNFA and AutoCCL ("Ours"), measured; labels are speedups over NCCL. Panels re-arranged: Phi-2-2B and Llama-3.1-8B on top, Yi-1.5-34B and VGG-19 (milliseconds) below.論文的 Figure 11：使用 NCCL、AFNFA 與 AutoCCL（圖中「Ours」）時的迭代時間（越低越好），實測；標籤是相對 NCCL 的加速倍數。子圖重新排列：上列 Phi-2-2B 與 Llama-3.1-8B，下列 Yi-1.5-34B 與 VGG-19（毫秒）。Screenshot of Xu et al., NSDI 2025, Figure 11 (PDF p. 13), panels re-arranged.

On PCIe, training gains exceeded the microbenchmark gains, which the authors attribute to stronger resilience to interference. On NVLink they were modest: when computation dominates, over-optimizing communication can slow the computation. The arithmetic behind the convergence times:

在 PCIe 上，訓練增益超過微基準測試的增益，作者歸因於對干擾更強的韌性。在 NVLink 上增益不大：計算占主導時，過度最佳化通訊可能拖慢計算。收斂時間背後的算術如下：

Question問題	Calculation算式	Result結果
VGG-19 iterations in 10 minutes, at 150–200 ms each (§6.4)VGG-19 在 10 分鐘內的迭代數，每次 150–200 ms（§6.4）	$600 \div (0.15 \text{ to } 0.2)$	$3{,}000$ – $4{,}000$ iterations次迭代
Calls of Phi-2-2B's 80 MB AllGather in the 10 iterations of Figure 12 (3,120 per iteration, Table 1)Figure 12 的 10 次迭代中，Phi-2-2B 80 MB AllGather 的呼叫數（每次迭代 3,120 次，Table 1）	$3{,}120 \times 10$	$31{,}200$ possible trials of one task同一任務可做的試驗數

An LLM iteration offers thousands of trials of one task; VGG-19 offers each task about one. Hence a few iterations against thousands.

LLM 的一次迭代提供同一任務數千次試驗；VGG-19 每個任務大約只提供一次。因此一個只需幾次迭代，另一個需要數千次。

6. Limits and open questions6. 限制與未解的問題

The paper's limits (§8): no transport-specific parameters; some settings deadlock or crash training; failures and new algorithms are future work. A careful reader adds:

論文的限制（§8）：不含與特定傳輸相關的參數；某些設定會讓訓練死結或崩潰；故障處理與新演算法留待未來。細心的讀者還應加上：

Tuning stops. Nothing restarts the search when congestion, tenants or failures change.調校會停止。壅塞、租戶或故障改變時，沒有東西重新啟動搜尋。
Scale. At most 32 A40 GPUs in one rack: no NVSwitch, Hopper, RoCE or multi-tier fabric, and most gains are inside a node.規模。最多 32 顆 A40 GPU、同一機櫃：沒有 NVSwitch、Hopper、RoCE 或多層網路，大部分增益在節點內。
Baselines. AFNFA was reproduced, not run from its code, and NCCL's tuner plugins are not compared.基準。AFNFA 是重現的，不是用原始程式碼；也沒有和 NCCL 的 tuner plugin 比較。
Many sizes. The key includes the exact message size, so dynamic shapes give many tasks with few calls each, like VGG-19. All-to-all is not supported.大量大小。任務鍵包含精確的訊息大小，所以動態形狀會產生許多各只呼叫幾次的任務，就像 VGG-19。all-to-all 不支援。
Headline. The abstract's "up to 1.80×" with interference is not in the evaluation text, whose largest stated value is 1.76×.標題數字。摘要中有干擾時「最多 1.80 倍」不在評估正文中，正文寫出的最大值是 1.76 倍。

7. How it connects7. 與其他內容的關聯

Deep dive, sections 2, 5, 7: the phases are the $\alpha$ – $\beta$ model applied twice; Table 5 measures section 5's warning; Table 6 is the price of overlap.第 4 章第 2、5、7 節：兩個階段就是套用兩次的 $\alpha$ – $\beta$ 模型；Table 5 實測了第 5 節的警告；Table 6 是重疊的代價。
Theseus swaps whole schedules while the job runs, beyond tuning knobs. ResCCL saves the same resource, thread blocks, by changing the execution model offline.Theseus 在任務執行中抽換整個排程，而不只是調參數。ResCCL 節省同一種資源——thread block——做法是離線改變執行模型。
Guided readings: Meta RoCE and Llama 3 tuned NCCL by hand for one fabric; MegaScale overlaps tensor-parallel collectives with GEMMs, the case of Table 6.論文導讀：Meta RoCE 與 Llama 3 針對單一網路手動調校 NCCL；MegaScale 把 tensor parallel 集體操作與 GEMM 重疊，正是 Table 6 的情況。

Key numbers關鍵數字

Number	數字	Meaning (A40 testbeds)
158 / 93 / 28	NCCL parameters: studied / undocumented / performance-sensitive	NCCL 參數：研究 / 無文件 / 影響效能
4.0 → 8.9 GB/s	64 MB AllReduce, 16 GPUs: NCCL's Ring vs tuned Tree	16 顆 GPU 的 64 MB AllReduce：NCCL 的 Ring 對調校後的 Tree
30.08 → 18.26 GB/s	80 MB AllGather default: no GEMM vs heavy GEMM (tuned 38.62 → 32.44)	80 MB AllGather 預設：無 GEMM 對重量 GEMM（調校後 38.62 → 32.44）
1.29× · 1.50× · 1.38×	AllGather · ReduceScatter · AllReduce vs NCCL, 128 MB with a GEMM	有 GEMM 時 128 MB 的 AllGather · ReduceScatter · AllReduce（相對 NCCL）
1.07–1.32× · 1.02–1.20×	Training iteration speedup, tuning included: three LLMs · VGG-19	訓練迭代加速，含調校時間：三個 LLM · VGG-19

Terms used on this page本頁用到的術語

Implementation parameters實作參數: Algorithm, protocol, transport: few values; they decide what runs.; 演算法、協定、傳輸方式：選項少，決定執行什麼。
Resource parameters資源參數: NC, NT, C: many values; they set how much hardware runs.; NC、NT、C：選項多，決定用多少硬體。
Unimodal, sweet point單峰、最佳點: Bandwidth rises, peaks, then levels off or falls as one knob grows.; 某個參數變大時，頻寬先升、達到峰值，再持平或下降。
Online tuning線上調校: Profiling candidate settings on the job's own repeated collectives.; 用任務自己重複的集體操作量測候選設定。
Leader: The GPU per group that searches and broadcasts each new setting.; 每個群組中負責搜尋並廣播新設定的那顆 GPU。