Collectives research: solvers, new algorithms, the library, MoE and compression集體通訊研究：求解器、新演算法、函式庫、MoE 與壓縮

Collective communication is now a networking research topic of its own. SIGCOMM 2026 gave it a research session, "Collective Communication Algorithms", with OptCCL, Theseus, Trivance, ZipCCL and DynamiQ. Its session on in-network aggregation had EPIC, HyNA, PReCCL and UBEP, and an experience session had Meta's paper on its NCCLX library. SIGCOMM 2025 had SyCCL and ResCCL, SIGCOMM 2024 had TE-CCL and MCCS, NSDI 2026 had FAST, NSDI 2025 had AutoCCL, and NSDI 2024 had Swing. Each paper attacks one mechanism from Section 4.3–8, so we read them as answers to five questions. Each of the 16 papers of this chapter also has its own digest of about ten minutes, with figures, a panel and a quiz.

集體通訊現在本身就是一個網路研究主題。SIGCOMM 2026 給了它一個研究場次「Collective Communication Algorithms」，其中有 OptCCL、Theseus、Trivance、ZipCCL 與 DynamiQ。網內聚合的場次有 EPIC、HyNA、PReCCL 與 UBEP，經驗論文場次則有 Meta 介紹其 NCCLX 函式庫的論文。SIGCOMM 2025 有 SyCCL 與 ResCCL，SIGCOMM 2024 有 TE-CCL 與 MCCS，NSDI 2026 有 FAST，NSDI 2025 有 AutoCCL，NSDI 2024 有 Swing。每篇論文各自針對第 2–8 節的某一個機制，所以我們把它們當成五個問題的答案來讀。本章的 16 篇論文也各有一篇約十分鐘的導讀，附圖、面板與測驗。

Two rules hold in this section. Every paper is peer-reviewed, and what we say about it comes from the paper itself, not from its title or from a summary by others. Every result we quote was measured, on a testbed, a hardware prototype or in production, and we give its setup. Where a paper argues with a cost model or a proof, we say so. Where its main evidence is simulation, we say that in one sentence and give no simulated speedups.

本節遵守兩條規則。每篇論文都經過同儕審查，我們對它的描述都來自論文本身，而不是標題或他人的摘要。我們引用的每個結果都是量測得來的，來自實驗平台、硬體原型或生產環境，並附上它的設定。如果論文改用成本模型或證明來論證，我們會說明；如果論文的主要證據是模擬，我們只用一句話交代，不列出模擬得到的加速。

Reading map. The digests are numbered in the order below. Read them in this order if you can: each one uses words that the earlier ones define. Each line gives the question that the digest answers and the subsection of this page that introduces it.

閱讀地圖。導讀依下列順序編號。可以的話請依序閱讀：每一篇都會用到前面幾篇定義過的詞。每一行列出該篇導讀回答的問題，以及本頁介紹它的小節。

1. TE-CCL: how can a solver describe a collective at all, and why does copying a chunk force integer variables? (9.1)TE-CCL：求解器要怎麼描述一個集體操作？為什麼「複製 chunk」會逼出整數變數？（9.1）
2. SyCCL: how does symmetry cut synthesis from hours to seconds, and why do the gains shrink at the training step? (9.1)SyCCL：對稱性怎樣把合成時間從數小時縮到數秒？為什麼增益到了訓練步就縮小？（9.1）
3. OptCCL: why do step-based solvers trade optimality for scale, and how does "route first, schedule second" avoid it? (9.1)OptCCL：為什麼以步為單位的求解器要拿最佳性換規模？「先路由、後排程」怎樣避開這個取捨？（9.1）
4. MCCS: in a shared cloud, who may see the topology, and who may change the ring while jobs run? (9.1)MCCS：在共用的雲端裡，誰可以看到拓樸？誰可以在任務執行時改變環？（9.1）
5. AutoCCL: which NCCL knobs set a collective's speed, and why does the best setting move when a GEMM runs beside it? (9.3)AutoCCL：哪些 NCCL 旋鈕決定集體操作的速度？為什麼旁邊跑著 GEMM 時最佳設定會改變？（9.3）
6. ResCCL: what does one thread block per connection cost, and how can a compiler share thread blocks? (9.3)ResCCL：每條連線一個 thread block 的代價是什麼？編譯器怎樣讓 thread block 共用？（9.3）
7. NCCLX: what breaks in NCCL at 100,000 GPUs, and how does Meta's library fix it? (9.3, 9.4)NCCLX：NCCL 在 100,000 顆 GPU 時哪裡會壞？Meta 的函式庫怎樣修好它？（9.3、9.4）
8. PReCCL: how can the library move bytes away from a slow or failed NIC while the job runs? (9.3)PReCCL：函式庫怎樣在任務執行時，把位元組從變慢或故障的 NIC 移開？（9.3）
9. Theseus: how can all GPUs change to a new schedule at the same request, and what does a change cost? (9.3, 9.4)Theseus：所有 GPU 怎樣在同一個請求換上新排程？換一次的代價是多少？（9.3、9.4）
10. Swing: why do two all-reduce schedules with equal steps and bytes run at different speeds on a torus? (9.2)Swing：為什麼步數與位元組都相同的兩個 all-reduce 排程，在環面上跑出不同的速度？（9.2）
11. Trivance: how do two ports per node cut an all-reduce to $\lceil \log_3 n \rceil$ steps, and what does that cost in link load? (9.2)Trivance：每個節點兩個埠，怎樣把 all-reduce 縮到 $\lceil \log_3 n \rceil$ 步？鏈路負載要付出什麼？（9.2）
12. ZipCCL: when does lossless compression make a collective faster, and when slower? (9.5)ZipCCL：無損壓縮什麼時候讓集體操作變快，什麼時候變慢？（9.5）
13. DynamiQ: why does compression error grow hop by hop in a ring all-reduce, and how can it stay small? (9.5)DynamiQ：為什麼壓縮誤差在環形 all-reduce 中一跳一跳地增長？怎樣讓它保持很小？（9.5）
14. EPIC: how can switches from different vendors add gradients for standard RoCE NICs? (9.5)EPIC：不同廠商的交換器，怎樣替標準 RoCE NIC 把梯度相加？（9.5）
15. HyNA: what should a switch do with hash collisions and overflows, so that rare exceptions do not stall every worker? (9.5)HyNA：交換器該怎樣處理雜湊碰撞與溢位，才不會讓少數例外拖住每個 worker？（9.5）
16. FAST: how can a schedule for a skewed all-to-all be computed at every call, and why do one-to-one stages avoid incast? (9.4)FAST：怎樣在每次呼叫時為偏斜的 all-to-all 算出排程？為什麼一對一的階段能避開 incast？（9.4）

The figure shows where each paper intervenes.

下圖標出每篇論文介入的位置。

9.1 Can a solver write a better schedule than ring and tree, and finish in time?9.1 求解器能寫出比環與樹更好的排程嗎？來得及算完嗎？

The problem. NCCL chooses from a short menu, ring or tree times three protocols, with a cost model close to the one in Section 4.3 (Section 4.5). One shape does not fit a fabric with two tiers (Section 4.4). SyCCL works out the cost analytically with the parameters of Alibaba's H800 cluster. Each GPU has 180 GB/s of NVLink and one 50 GB/s NIC, a ratio of 3.6 to 1, but NCCL's ring all-gather crosses 7 NVLink hops for each network hop. So the NIC waits about half of the time, and about 10.6% of all the bandwidth is wasted. At small sizes the ring's many hops cost latency instead. The research answer is schedule synthesis: a solver gets the topology and the demand, and decides which chunk crosses which link at which time. A runtime such as MSCCL then runs the result.

問題。NCCL 從一份很短的菜單中選擇——環或樹，乘上三種協定——用的是接近第 4.3 節的成本模型（第 4.5 節）。單一形狀不適合兩層的網路（第 4.4 節）。SyCCL 用阿里 H800 叢集的參數以解析方式算出代價。每顆 GPU 有 180 GB/s 的 NVLink 與一張 50 GB/s 的 NIC，比例是 3.6 比 1，但 NCCL 的環形 all-gather 每經過一跳網路就要經過 7 跳 NVLink。所以 NIC 約有一半的時間在等，全部頻寬中約有 10.6% 被浪費。小尺寸時，環的跳數多則改成付出延遲。研究界的回答是排程合成：把拓樸與需求交給求解器，由它決定哪個 chunk 在什麼時間走哪條鏈路，再由 MSCCL 之類的執行期執行結果。

Exact but slow. SCCL (PPoPP 2021) uses an SMT solver and finds near-optimal algorithms, but only for about one chassis. TACCL (NSDI 2023) scales further with a heuristic that human-written "communication sketches" guide, and it decides routing and scheduling separately. We describe these two only through what TE-CCL and SyCCL report about them. In TE-CCL's runs, TACCL ran out of memory or timed out on large topologies.

精確但慢。SCCL（PPoPP 2021）使用 SMT 求解器找出接近最佳的演算法，但只適用於大約一台機箱。TACCL（NSDI 2023）用一個由人寫的「通訊草圖」（communication sketch）引導的啟發式方法擴展得更遠，並把路由與排程分開決定。我們只透過 TE-CCL 與 SyCCL 對這兩者的描述來介紹它們。在 TE-CCL 的實驗中，TACCL 在大拓樸上會耗盡記憶體或逾時。

TE-CCL. TE-CCL (SIGCOMM 2024; University of Pennsylvania, Microsoft Research and others) treats the collective as traffic engineering: a multi-commodity flow problem with three additions. Transfers are finite (a fixed number of bytes, not a rate), GPUs can store and forward, and a GPU can copy, which means it keeps a chunk and also forwards it. Time is cut into epochs; in most of the evaluation, one epoch is the time that the fastest link needs for one chunk. A chunk sent on link $(i,j)$ at the start of epoch $k$ arrives by the end of epoch $k + \lceil \alpha_{ij}/\tau \rceil$, where $\tau$ is the epoch length, so $\alpha$ is part of the model. The result is a time-expanded graph. In our sketch the latency is shorter than one epoch:

TE-CCL。TE-CCL（SIGCOMM 2024；賓州大學、微軟研究院等）把集體操作當成流量工程：一個多商品流（multi-commodity flow）問題，加上三項擴充。傳輸量是有限的（固定的位元組數，而不是速率），GPU 可以儲存轉送，GPU 也可以複製——留下一個 chunk 的同時把它轉送出去。時間被切成時段（epoch）；在大部分評估中，一個時段是最快的鏈路送一個 chunk 所需的時間。在第 $k$ 個時段開始時送上鏈路 $(i,j)$ 的 chunk，會在第 $k + \lceil \alpha_{ij}/\tau \rceil$ 個時段結束前到達（$\tau$ 是時段長度），所以 $\alpha$ 是模型的一部分。結果是一張時間展開圖（time-expanded graph）。我們的草圖中，延遲短於一個時段：

The variables $F_{s,i,j,k,c}$ (chunks of source $s$ on link $(i,j)$ in epoch $k$) and $B_{s,i,k,c}$ (chunks in GPU $i$'s buffer at the start of epoch $k$) multiply sources, links, epochs and chunks, so the problem grows fast (TE-CCL §3.1). Copy is the reason for integer variables: with fractional chunks, a destination could be credited twice for the same half chunk. TE-CCL therefore has three solvers: an exact MILP for collectives that need copy, such as all-gather; an LP that stays exact when no GPU copies, such as all-to-all, and scales further; and an A*-like search in rounds, which gives up optimality to reach larger topologies. On small cases the solvers are fast. On an all-to-all with 8 chunks, SCCL timed out after 10,032.7 s, while TE-CCL finished in 1.88 s (TE-CCL §6.1), at least $10{,}032.7 / 1.88 \approx 5{,}300$ times faster. At scale they need hours, as the table below shows. On a testbed of 32 AMD GPUs in two chassis, TE-CCL's all-gather schedules reached 2.14× the algorithm bandwidth of TACCL and 3.18× that of RCCL. Its other bandwidth results are the solver's own $\alpha$–$\beta$ estimates, not hardware runs.

變數 $F_{s,i,j,k,c}$（第 $k$ 個時段在鏈路 $(i,j)$ 上屬於來源 $s$ 的 chunk）與 $B_{s,i,k,c}$（第 $k$ 個時段開始時在 GPU $i$ 緩衝區中的 chunk）是來源數、鏈路數、時段數與 chunk 數的乘積，所以問題長得很快（TE-CCL §3.1）。整數變數的來源是複製：如果 chunk 可以是分數，同一個半 chunk 可能被目的地算成兩次。因此 TE-CCL 有三種求解器：給需要複製的集體操作（例如 all-gather）用的精確 MILP；在沒有 GPU 複製時（例如 all-to-all）仍然精確、而且擴展得更遠的 LP；以及以回合進行、類似 A* 的搜尋，它放棄最佳性以換取更大的拓樸。小案例時這些求解器很快。在一個 8 個 chunk 的 all-to-all 上，SCCL 在 10,032.7 s 後逾時，而 TE-CCL 在 1.88 s 內完成（TE-CCL §6.1），至少快了 $10{,}032.7 / 1.88 \approx 5{,}300$ 倍。到了大規模，它們要花數小時，如下表所示。在兩台機箱共 32 顆 AMD GPU 的實驗平台上，TE-CCL 的 all-gather 排程達到 TACCL 的 2.14 倍、RCCL 的 3.18 倍演算法頻寬。它其他的頻寬結果是求解器自己的 $\alpha$–$\beta$ 估計，不是硬體上的執行。

SyCCL. SyCCL (SIGCOMM 2025; Alibaba Cloud and Tsinghua University) shrinks the problem with symmetry. In a one-to-all collective every non-source GPU has the same demand, an all-gather is $N$ relabeled broadcasts, and an all-to-all is $N$ relabeled scatters. Clos and multi-rail fabrics organize into dimensions and groups, and the groups in one dimension are isomorphic. A sketch splits the collective into stages of sub-demands, each on one dimension and one group. SyCCL solves each distinct sub-demand once, with TE-CCL's MILP, and reuses the result for the isomorphic groups. A small linear system mixes several sketches, so that each dimension's share of the data equals its share of the bandwidth. SyCCL prunes inconsistent combinations, estimates the completion time of each merged schedule with an $\alpha$–$\beta$ event simulator, and keeps the fastest. The paper's Figure 5 shows the idea:

SyCCL。SyCCL（SIGCOMM 2025；阿里雲與清華大學）用對稱性把問題變小。在一對多的集體操作中，每顆非來源 GPU 的需求都相同；all-gather 是 $N$ 個重新編號的 broadcast，all-to-all 是 $N$ 個重新編號的 scatter。Clos 與多 rail 網路可以組織成維度與群組，同一維度中的群組彼此同構。一份草圖（sketch）把集體操作切成由子需求組成的階段，每個子需求落在一個維度、一個群組上。SyCCL 對每個不同的子需求只用 TE-CCL 的 MILP 解一次，再把結果重用到同構群組。一個小的線性方程組把幾份草圖混合起來，讓每個維度分到的資料比例等於它的頻寬比例。SyCCL 剪掉不一致的組合，用 $\alpha$–$\beta$ 事件模擬器估算每份合併排程的完成時間，留下最快的。論文的 Figure 5 呈現了這個想法：

Every number in the table is a measured run time of a solver. The 64-, 256- and 512-GPU cases are topologies given to the solver; their schedules were not run.

表中每個數字都是求解器實際執行的時間。64、256 與 512 顆 GPU 的案例是交給求解器的拓樸，它們的排程並沒有實際執行。

SyCCL synthesized schedules 915× to 17,286× faster than TE-CCL on average (SyCCL Table 5). Its testbed has 4 servers of 8 A800 GPUs, with four 200 Gb/s NICs per server. There, all-gather bus bandwidth on 16 GPUs was 17–108% above NCCL's for messages above 1 MB, and on 32 GPUs it was up to 104% above NCCL's and 20–88% above TE-CCL's. But training iterations were only 0.4–6.3% faster than with NCCL, and 1.6–4.0% faster than with TE-CCL (Table 6). Section 4.7 explains the gap: only exposed communication adds to the step.

SyCCL 合成排程的速度平均比 TE-CCL 快 915 倍到 17,286 倍（SyCCL Table 5）。它的實驗平台有 4 台伺服器，每台 8 顆 A800 GPU 與四張 200 Gb/s NIC。在這個平台上，16 顆 GPU 的 all-gather 在 1 MB 以上的訊息，匯流排頻寬比 NCCL 高 17–108%；32 顆 GPU 時比 NCCL 高最多 104%、比 TE-CCL 高 20–88%。但訓練迭代只比用 NCCL 快 0.4–6.3%，比用 TE-CCL 快 1.6–4.0%（Table 6）。第 4.7 節解釋了這個差距：只有暴露的通訊會加到步時間上。

OptCCL. OptCCL (SIGCOMM 2026; Cornell University) asks why these solvers must trade optimality for scale. Any formulation with $T$ time steps of size $s$ has the completion time $s \cdot T$. A small $T$ keeps the problem small, a small $s$ gives good pipelining, and the two are tied. OptCCL decouples space from time. A linear program first chooses the routes and the bytes on each, under bandwidth limits that hold only on average over the whole collective; the paper proves that this loses nothing over all step sizes (its Theorem 2). Weighted fair queueing then turns the routes into fixed-size chunks, and a symmetry-aware Dantzig–Wolfe decomposition solves one subproblem per class of identical GPUs. The quality of OptCCL's schedules was only simulated, and OptCCL is not yet part of a library. What was measured is synthesis time on a 32-core server: within 3 hours, TE-CCL found no schedule beyond 16–32 GPUs and SyCCL none beyond 64–128, while OptCCL needed about 30 minutes at 256 GPUs and was 3–32× faster than SyCCL at 64 GPUs or more.

OptCCL。OptCCL（SIGCOMM 2026；康乃爾大學）問的是：為什麼這些求解器必須拿最佳性換規模？任何以 $T$ 個大小為 $s$ 的時間步來建模的方法，完成時間都是 $s \cdot T$。$T$ 小則問題小，$s$ 小則管線化好，而兩者互相綁住。OptCCL 把空間與時間拆開。先用一個線性規劃選出路徑與每條路徑上的位元組數，頻寬限制只要在整個集體操作的期間「平均」成立；論文證明這樣做對所有步長都不損失最佳性（它的 Theorem 2）。接著用加權公平佇列（weighted fair queueing）把路徑切成固定大小的 chunk，再用感知對稱性的 Dantzig–Wolfe 分解，對每一類相同的 GPU 只解一個子問題。OptCCL 排程的品質只做了模擬，它也還沒有放進任何函式庫。量測的是在一台 32 核心伺服器上的合成時間：3 小時內，TE-CCL 超過 16–32 顆 GPU、SyCCL 超過 64–128 顆就找不到排程，而 OptCCL 在 256 顆 GPU 時約需 30 分鐘；在 64 顆 GPU 以上，它比 SyCCL 快 3–32 倍。

Who may see the topology? Every solver above assumes that its user knows the topology. In a public cloud the tenant does not. NCCL runs inside the tenant's process, sees one big virtual switch, orders its ring by rank, and never changes it; in Meta's production trace this put up to 2× the necessary flows across racks. MCCS (SIGCOMM 2024; Duke University and Meta) moves the collective into a service that the cloud provider runs on every host. Tenants keep the NCCL API through a thin shim. The provider chooses the ring order, the NIC and the path of every flow, and changes them between collectives: all ranks exchange the number of their last launched collective, and switch after the largest one. The testbed is small, and its fabric is emulated: 8 RTX 3090 GPUs, with one switch that emulates a 2:1 spine-leaf. Averaged over 8–512 MB, MCCS reached 2.4× NCCL's algorithm bandwidth on 8 GPUs. Its service path adds 50–80 µs to each collective, which hurts below 8 MB.

誰可以看到拓樸？上面每個求解器都假設使用者知道拓樸。在公有雲裡，租戶並不知道。NCCL 在租戶的行程內執行，看到的是一台巨大的虛擬交換器，依 rank 排列它的環，而且從不改變；在 Meta 的生產紀錄中，這讓跨機櫃的流最多達到必要數量的 2 倍。MCCS（SIGCOMM 2024；杜克大學與 Meta）把集體操作移到雲端供應商在每台主機上執行的服務裡。租戶透過一層很薄的 shim 繼續使用 NCCL 的 API。供應商決定環的順序、每條流用哪張 NIC 與哪條路徑，並在集體操作之間改變它們：所有 rank 交換各自最後啟動的集體操作編號，在最大的那一個之後一起切換。實驗平台很小，網路是模擬出來的：8 顆 RTX 3090，用一台交換器模擬 2:1 的 spine-leaf。在 8–512 MB 的平均下，MCCS 在 8 顆 GPU 上達到 NCCL 的 2.4 倍演算法頻寬。它的服務路徑讓每次集體操作多花 50–80 µs，8 MB 以下就吃虧。

Is it fast enough? For a dense model with fixed traffic, a synthesis of 30–40 minutes once per job is acceptable. Theseus measured what a slow solver costs in practice: TE-CCL's intermediate schedules for a 1 GB all-gather on 32 GPUs needed 13 hours to beat NCCL and 49 hours to reach the best one (last row of the table). And every solver here needs the demand before it starts. In an MoE all-to-all the split sizes are known only when the collective starts (9.4), so a schedule that was synthesized before the job cannot follow them.

夠快嗎？對流量固定的稠密模型，每個任務合成一次、花 30–40 分鐘可以接受。Theseus 量到慢速求解器在實務上的代價：TE-CCL 為 32 顆 GPU、1 GB 的 all-gather 產生的中間排程，要 13 小時才贏過 NCCL，49 小時才到最佳的一份（表中最後一列）。而這裡的每個求解器都必須在開始前知道需求。在 MoE 的 all-to-all 中，切分大小要到集體操作開始時才知道（9.4），所以在任務開始前合成的排程無法跟上它們。

Caveat. The bandwidth and training gains come from small testbeds: 32 AMD GPUs for TE-CCL, 16–32 A800 GPUs for SyCCL, and 8 GPUs on an emulated fabric for MCCS. OptCCL's schedule quality is simulation only. Every schedule also assumes a known, stable topology; a failed NIC (9.3) or a skewed all-to-all (9.4) breaks that assumption.

保留。頻寬與訓練的增益都來自小型實驗平台：TE-CCL 用 32 顆 AMD GPU，SyCCL 用 16–32 顆 A800，MCCS 用模擬網路上的 8 顆 GPU。OptCCL 的排程品質只有模擬。每份排程也都假設拓樸已知且穩定；一張故障的 NIC（9.3）或一個偏斜的 all-to-all（9.4）就會打破這個假設。

9.2 New algorithms for fabrics with low bisection9.2 為低二分頻寬網路設計的新演算法

The problem. Section 4.3 gave Rabenseifner's algorithm a condition: every pair of ranks can talk at full rate at the same time. A non-blocking Clos is close to that. A torus is not: its bisection is low, and distance is a number of hops. Recursive doubling's peer in step $s$ is $2^s$ ranks away, so the late steps cross many hops and many flows share each link. The ring talks only to neighbors but needs $O(N)$ steps (Swing, NSDI 2024).

問題。第 4.3 節給 Rabenseifner 演算法加了一個條件：每一對 rank 都能同時以全速通訊。非阻塞的 Clos 接近這個條件。環面（torus）則不是：它的二分頻寬很低，距離是跳數。遞迴倍增在第 $s$ 步的夥伴相距 $2^s$ 個 rank，所以後面幾步要跨很多跳，很多條流共用每條鏈路。環只與鄰居通訊，卻需要 $O(N)$ 步（Swing，NSDI 2024）。

Swing (Sapienza University of Rome, ETH Zurich, RWTH Aachen) keeps $\log_2 N$ steps and shortens the distance. In step $s$, a node exchanges with the peer at the signed offset

Swing（羅馬大學、蘇黎世聯邦理工學院、亞琛工業大學）保留 $\log_2 N$ 步，但縮短距離。在第 $s$ 步，節點與位於下列帶號偏移量的夥伴交換：

Even ranks add $\rho(s)$ and odd ranks subtract it (Swing §3.1.1), so the peer swings left and right; in the late steps it is about two-thirds as far as recursive doubling's $2^s$. The reduce-scatter plus all-gather form sends the minimum bytes, and $D$ plain plus $D$ mirrored copies keep all $2D$ ports of a $D$-dimensional torus busy. Swing also adds a third factor to the $\alpha$–$\beta$ model, the congestion deficiency $\Xi$: how much longer the byte term takes because messages of the same collective share the busiest link. By the paper's model (its Table 2), bandwidth-optimal Swing has $\Xi \approx 1.19$, 1.03 and 1.008 on 2D, 3D and 4D tori, against 1.5, 1.17 and 1.07 for Rabenseifner's algorithm tuned for tori.

偶數 rank 加上 $\rho(s)$、奇數 rank 減去它（Swing §3.1.1），所以夥伴在左右之間擺盪；在後面幾步，距離約為遞迴倍增 $2^s$ 的三分之二。reduce-scatter 加 all-gather 的形式送出最少的位元組，而 $D$ 份一般版加 $D$ 份鏡像版讓 $D$ 維環面的 $2D$ 個埠全都忙碌。Swing 也在 $\alpha$–$\beta$ 模型上加了第三個因子，壅塞不足度（congestion deficiency）$\Xi$：同一個集體操作的訊息共用最忙的鏈路，使位元組項變長的倍數。依論文的模型（它的 Table 2），頻寬最佳的 Swing 在二維、三維、四維環面上的 $\Xi$ 約為 1.19、1.03 與 1.008，針對環面調校的 Rabenseifner 演算法則是 1.5、1.17 與 1.07。

Trivance (SIGCOMM 2026; TU Berlin and Weizenbaum Institute, Purdue University) starts from two older algorithms. Bruck needs only $\log_3 n$ steps on a bidirectional ring but sends all traffic in one direction, so congestion is high. Recursive doubling and Swing take $\log_2 n$ steps and use the second port only for a mirrored copy on half the data, which does not cut the steps. In Trivance, each node exchanges with the peers at distance $3^k$ in both directions at once and reduces both incoming messages together, so its reach triples in each step. The latency-optimal variant needs $\lceil \log_3 n \rceil$ steps, the bandwidth-optimal one $2 \lceil \log_3 n \rceil$. In step $k$ every link carries $3^k$ chunks, so the load is uniform, and the congestion is a third of Bruck's (Trivance §4.1 and Appendix B).

Trivance（SIGCOMM 2026；柏林工業大學與 Weizenbaum 研究所、普渡大學）從兩個較舊的演算法出發。Bruck 在雙向環上只需 $\log_3 n$ 步，但所有流量都往同一個方向送，所以壅塞很高。遞迴倍增與 Swing 需要 $\log_2 n$ 步，第二個埠只拿來跑處理一半資料的鏡像版，步數並沒有減少。在 Trivance 中，每個節點同時與兩個方向上距離 $3^k$ 的夥伴交換，並把兩則收到的訊息一起歸約，所以涵蓋範圍每一步變成三倍。延遲最佳的版本需要 $\lceil \log_3 n \rceil$ 步，頻寬最佳的版本需要 $2 \lceil \log_3 n \rceil$ 步。在第 $k$ 步每條鏈路承載 $3^k$ 個 chunk，所以負載是均勻的，而壅塞是 Bruck 的三分之一（Trivance §4.1 與附錄 B）。

Which design wins, by the papers' cost model. Trivance's Table 1 compares Trivance, Bruck, recursive doubling and Swing on a bidirectional ring with one model. Here $K$ is the number of steps, $m_k$ the bytes that each message carries in step $k$, and $c_k$ the congestion, the number of messages that share the busiest link in that step:

依論文的成本模型，哪個設計勝出。Trivance 的 Table 1 用同一個模型比較雙向環上的 Trivance、Bruck、遞迴倍增與 Swing。其中 $K$ 是步數，$m_k$ 是第 $k$ 步每則訊息攜帶的位元組，$c_k$ 是壅塞，也就是該步共用最忙鏈路的訊息數：

In the latency-optimal forms every message carries the whole buffer, $m_k = S$, so the byte term is $\beta S$ times the congestion summed over the steps. That sum is about $n$ for recursive doubling, $n/3$ for Swing and $n/2$ for Trivance. So Trivance has the fewest steps, but it puts more bytes on the busiest links than Swing. The model predicts that Trivance wins when $\alpha$ dominates, with small messages, and that Swing wins when the byte term dominates, with large ones; the paper gives no closed-form crossover. Each step costs one $\alpha$, so the step count is the latency term. For $n = 4{,}096$:

在延遲最佳形式中，每則訊息都攜帶整個緩衝區，$m_k = S$，所以位元組項是 $\beta S$ 乘上各步壅塞的總和。這個總和在遞迴倍增約為 $n$、在 Swing 約為 $n/3$、在 Trivance 約為 $n/2$。所以 Trivance 的步數最少，但它在最忙鏈路上放的位元組比 Swing 多。模型預測：$\alpha$ 主導時（小訊息）Trivance 勝出，位元組項主導時（大訊息）Swing 勝出；論文沒有給出交叉點的封閉式。每一步付一次 $\alpha$，所以步數就是延遲項。以 $n = 4{,}096$ 為例：

Caveat. Both papers target tori and multiport networks. Both evaluate only in packet-level simulation, with the SST simulator, and report no hardware run, so this page and their digests give the cost models, not the simulated speedups. On a non-blocking Clos, where any two ranks are a few switch hops apart, distance is a much smaller lever; Swing's own discussion expects no gain there.

保留。兩篇論文都針對環面與多埠網路。兩者都只在封包層級的模擬器 SST 中評估，沒有任何硬體上的執行，所以本頁與它們的導讀只介紹成本模型，不列出模擬得到的加速。在非阻塞的 Clos 上，任兩個 rank 只隔幾跳交換器，距離這個槓桿小得多；Swing 自己的討論也預期那裡沒有增益。

9.3 The library itself: tuning, thread blocks, 100K GPUs and change at run time9.3 函式庫本身：調校、thread block、10 萬顆 GPU 與執行期的變更

The problem. Section 4.5 and Section 4.6 described NCCL's machinery: channels with staging buffers, a GPU copy at each end, a CPU proxy that waits for the clear-to-send, and buffers allocated in advance. NCCL makes its choices once, when a communicator starts, and the same way for every job: knob settings from a cost model, one thread block per channel, buffers and connections for every peer, the same share of bytes for every channel, and a fixed set of schedules. Five digests each attack one of these static choices.

問題。第 4.5 節與第 4.6 節描述了 NCCL 的機制：帶分段緩衝區的 channel、兩端各一次 GPU 複製、等待 clear-to-send 的 CPU proxy，以及事先配置的緩衝區。NCCL 在 communicator 建立時就做好所有選擇，而且對每個任務都一樣：由成本模型決定的旋鈕設定、每個 channel 一個 thread block、為每個對端準備的緩衝區與連線、每個 channel 分到相同的位元組，以及一組固定的排程。五篇導讀各自針對其中一個靜態的選擇。

Tune the knobs: AutoCCL. AutoCCL (NSDI 2025; USTC and Microsoft Research) tunes six NCCL knobs per collective: algorithm, protocol, transport, number of channels, threads per block and chunk size. On A40 servers the defaults were far from the best. A 64 MB all-reduce on 16 GPUs ran at 4.0 GB/s with NCCL's ring and at 8.9 GB/s with a tuned tree. A GEMM beside an 80 MB all-gather cut NCCL's default from 30.08 to 18.26 GB/s, while a setting tuned under that GEMM kept 32.44 GB/s. AutoCCL tries the 12 combinations of the three few-valued knobs and searches the three resource knobs of each by coordinate descent. Every trial runs on the job's own repeated collectives. On up to 32 A40 GPUs, training iterations were 1.07–1.32× faster for three LLMs, tuning included, and 1.02–1.20× faster for VGG-19.

調校旋鈕：AutoCCL。AutoCCL（NSDI 2025；中國科學技術大學與微軟研究院）為每個集體操作調校六個 NCCL 旋鈕：演算法、協定、傳輸方式、channel 數、每個 block 的執行緒數與 chunk 大小。在 A40 伺服器上，預設值離最佳值很遠。16 顆 GPU 上 64 MB 的 all-reduce，用 NCCL 的環跑出 4.0 GB/s，用調校過的樹跑出 8.9 GB/s。旁邊跑著一個 GEMM 時，80 MB all-gather 的 NCCL 預設值從 30.08 掉到 18.26 GB/s，而在這個 GEMM 下調校出的設定還保有 32.44 GB/s。AutoCCL 嘗試三個取值很少的旋鈕的 12 種組合，在每一種裡用座標下降法搜尋三個資源旋鈕。每次試驗都跑在任務自己重複出現的集體操作上。在最多 32 顆 A40 上，三個 LLM 的訓練迭代快了 1.07–1.32 倍（含調校時間），VGG-19 快了 1.02–1.20 倍。

Compile the schedule: ResCCL. NCCL runs a ring all-reduce as $2k-1$ primitive steps for $k$ ranks (send, recvReduceSend, …, recv), on one thread block per channel (Demystifying NCCL, HOTI 2025). ResCCL (SIGCOMM 2025; Northeastern University, SIAT and Alibaba Cloud) measured what one thread block per connection costs on A100 servers. The thread blocks of extra channels sat idle 98.2% of the time, thread blocks waited on synchronization up to 67.1% of the time, and interpreting a custom schedule at run time cost 17.1% on average. ResCCL compiles an algorithm offline. It schedules at the primitive level, so that one thread block runs the same task for every micro-batch; it lets connections that are never active at the same time share one thread block; and it generates plain kernels. The testbed has 4 servers of 8 A100 GPUs with four 200 Gb/s RoCE NICs per server. Against MSCCL, which runs the same algorithm, ResCCL reached up to 2.5× the bandwidth, and for a synthesized all-gather on 2 servers of 8 GPUs it needed 4 thread blocks per GPU instead of 18 (77.8% fewer). Against NCCL the algorithm changes too, because NCCL runs only its built-in algorithms. With ResCCL, Megatron-LM throughput was 18–39% higher than with NCCL for T5 and 11–20% higher for GPT-3. Compiling for 1,024 GPUs takes about 11 minutes, offline.

編譯排程：ResCCL。NCCL 把 $k$ 個 rank 的環形 all-reduce 執行成 $2k-1$ 個基本步驟（send、recvReduceSend、…、recv），每個 channel 一個 thread block（Demystifying NCCL，HOTI 2025）。ResCCL（SIGCOMM 2025；東北大學、中國科學院深圳先進技術研究院與阿里雲）在 A100 伺服器上量到每條連線一個 thread block 的代價：額外 channel 的 thread block 有 98.2% 的時間閒置，thread block 等待同步的時間最多達 67.1%，而執行時直譯自訂排程平均損失 17.1%。ResCCL 離線編譯演算法。它在基本步驟層級排程，讓一個 thread block 對每個 micro-batch 執行同一個任務；讓從不同時活動的連線共用一個 thread block；並產生一般的 kernel。實驗平台有 4 台伺服器，每台 8 顆 A100 與四張 200 Gb/s 的 RoCE NIC。相對於執行相同演算法的 MSCCL，ResCCL 的頻寬最多達 2.5 倍；在 2 台各 8 顆 GPU 上的合成 all-gather，每顆 GPU 只需 4 個 thread block，而不是 18 個（少 77.8%）。相對於 NCCL，演算法也跟著改變，因為 NCCL 只執行它內建的演算法。用 ResCCL 時，Megatron-LM 的吞吐在 T5 上比 NCCL 高 18–39%，在 GPT-3 上高 11–20%。為 1,024 顆 GPU 編譯約需 11 分鐘，離線進行。

Scale to 100K GPUs: NCCLX. Meta's experience paper at SIGCOMM 2026, "Connecting 100K+ GPUs: Building the Communication Stack for Large-Scale LLM Training" (Hongyi Zeng, Min Si, Pavan Balaji et al., Meta and Harvard University), explains why each part of NCCL strains beyond 100,000 GPUs, and describes Meta's fork, NCCLX. Our digest reads the arXiv preprint, which has a different title ("Collective Communication for 100k+ GPUs") and a different author list. FIFOs and staging copies make memory grow with thread blocks (the channel figure in Section 4.5 shows one block, one SM and one staging buffer per channel). Copy kernels take SMs from concurrent GEMMs. Buffers and QPs are allocated eagerly, host-side arguments do not fit CUDA graphs, the clear-to-send sits on the critical path, and initialization is serialized. The fabric also spans several buildings: relative to one rack, latency is 7× across racks, 15× across AI zones and 30× across datacenters.

擴展到 10 萬顆 GPU：NCCLX。Meta 在 SIGCOMM 2026 的經驗論文〈Connecting 100K+ GPUs: Building the Communication Stack for Large-Scale LLM Training〉（Hongyi Zeng、Min Si、Pavan Balaji 等，Meta 與哈佛大學）說明了 NCCL 的每個部分在超過 100,000 顆 GPU 時為何吃緊，並介紹 Meta 的分支 NCCLX。我們的導讀讀的是 arXiv 預印本，它的標題（〈Collective Communication for 100k+ GPUs〉）與作者名單都不同。FIFO 與分段複製讓記憶體隨 thread block 增長（第 4.5 節的 channel 圖：每個 channel 一個 block、一個 SM、一塊分段緩衝區）。複製 kernel 從同時執行的 GEMM 手中搶走 SM。緩衝區與 QP 被提前配置，主機端的參數與 CUDA graph 不相容，clear-to-send 位於關鍵路徑上，初始化又是序列化的。網路也跨越數棟建築：相對於機櫃內，跨機櫃的延遲是 7 倍、跨 AI zone 是 15 倍、跨資料中心是 30 倍。

CTran: host-driven and zero-copy. NCCLX's transport is host-driven: one CPU thread per communicator issues the RDMA operations, and the kernel and the CPU synchronize through host-pinned flags in less than 1 µs. It is zero-copy: data moves from the user's source buffer straight into the user's destination buffer, and buffers are registered the first time a collective uses them. The copy-based path adds a constant tax to every transfer, which can double the transfer time across hosts; zero-copy point-to-point was 1.09–2.7× faster for messages of 1–128 MB. Transfers can also be SM-free, through the copy engine or RDMA, so no copy kernel takes SMs from the GEMMs that hide the collectives in Section 4.7's overlap timeline.

CTran：主機驅動、零複製。NCCLX 的傳輸層是主機驅動（host-driven）的：每個 communicator 一條 CPU 執行緒發出 RDMA 操作，kernel 與 CPU 透過 host-pinned 旗標同步，耗時不到 1 µs。它是零複製（zero-copy）的：資料從使用者的來源緩衝區直接搬進使用者的目的緩衝區，緩衝區在第一次被集體操作使用時註冊。以複製為基礎的路徑對每次傳輸都加上一筆固定的稅，跨主機時可讓傳輸時間加倍；零複製的點對點傳輸對 1–128 MB 的訊息快 1.09–2.7 倍。傳輸也可以不佔 SM（SM-free），由 copy engine 或 RDMA 搬資料，所以沒有複製 kernel 會從第 4.7 節重疊時間軸裡那些藏住集體操作的 GEMM 手中搶走 SM。

Flow control, startup, memory. DQPLB limits the outstanding data per connection by topology class: it is Section 4.6's bytes-in-flight bound (see the slot diagram there), set per distance. Together with load-balancing changes and tuned virtual output queues (VOQs) in the spine switches, it cut switch buffer build-up by an order of magnitude against Llama 3. Stock NCCL needed more than 4 minutes to initialize at 96K GPUs: its topology computation is $O(N^2)$, about 10 s at 48K ranks, and TCP queues overflow beyond 64K ranks. One global communicator split with ncclCommSplit, a TCPStore bootstrap and $O(N)$ CPU algorithms made process-group creation up to 11× faster. In Llama 4 pre-training, NCCL used about 10 GB of each H100's HBM over 10+ communicators, with 2 MiB of metadata per channel, because it allocates buffers and QPs before they are used; lazy connections, lazy channels and a slab allocator roughly halved it.

流量控制、啟動、記憶體。DQPLB 依拓樸類別限制每條連線未完成的資料量：它就是第 4.6 節的在途位元組上限（見該節的槽位時空圖），按距離設定。配合負載平衡的改進與 spine 交換器上調校過的虛擬輸出佇列（VOQ），它讓交換器緩衝區的堆積比 Llama 3 降低了一個數量級。原版 NCCL 在 96K 顆 GPU 時初始化要超過 4 分鐘：它的拓樸計算是 $O(N^2)$，在 48K 個 rank 時約 10 s，而 TCP 佇列在超過 64K 個 rank 時會溢位。用一個全域 communicator 搭配 ncclCommSplit 切分、以 TCPStore 啟動，再加上 $O(N)$ 的 CPU 演算法，讓 process group 的建立快了最多 11 倍。在 Llama 4 預訓練中，NCCL 在 10 個以上的 communicator 中用掉每顆 H100 約 10 GB 的 HBM，每個 channel 有 2 MiB 的中繼資料，因為它在使用之前就配置好緩衝區與 QP；延遲建立連線、延遲建立 channel 與 slab 配置器讓用量大約減半。

Faults. A job of this size must survive failed links and NICs, so NCCLX also has FTAR, a fault-tolerant ring all-reduce between the replica groups of hybrid sharded data parallelism. FTAR moves 8 MB chunks, which saturate Meta's network, with only 2 thread blocks of 512 threads; NCCL's default is 4 blocks of 544. When NCCL is limited to the same 2 blocks, FTAR's latency is 9–18% lower; against NCCL's default it is about equal. So FTAR's gain is fewer thread blocks at the same speed.

故障。這種規模的任務必須撐過鏈路與 NIC 的故障，所以 NCCLX 還有 FTAR：在混合分片資料平行（HSDP）的各個副本群組之間執行的容錯環形 all-reduce。FTAR 以 8 MB 的 chunk 搬資料，這個大小足以讓 Meta 的網路飽和，而且只用 2 個各 512 執行緒的 thread block；NCCL 的預設是 4 個各 544 執行緒。當 NCCL 被限制成同樣的 2 個 block 時，FTAR 的延遲低 9–18%；與 NCCL 的預設相比則大約相同。所以 FTAR 的收穫是用更少的 thread block 跑出相同的速度。

Results. In production, the steady-state step latency of Llama 4 training fell by up to 12%. With AllToAllvDynamic (9.4), Llama 4 Maverick decode latency fell by 15–80%. The paper's Table 3 measured 4, 8 and 16 inference hosts, and the gain grows with the hosts and with the number of experts per token, $k$. The table below takes two rows at 16 hosts, plus three ratios from this subsection:

結果。在生產環境中，Llama 4 訓練的穩態步延遲降低最多 12%。有了 AllToAllvDynamic（9.4），Llama 4 Maverick 的解碼延遲降低 15–80%。論文的 Table 3 量了 4、8 與 16 台推論主機，增益隨主機數與每個 token 的專家數 $k$ 增加。下表取 16 台主機的兩列，再加上本小節的三個比值：

At 16 hosts Table 3 reaches about 83%, slightly above the headline range. The 3.3% row shows why a bound per distance makes sense: one credit size cannot fit paths whose latencies differ by 30×.

在 16 台主機時，Table 3 的範圍到約 83%，略高於標題式的範圍。3.3% 那一列說明了按距離設定上限的道理：同一個額度大小無法適合延遲相差 30 倍的路徑。

Change at run time: PReCCL and Theseus. AutoCCL tunes once and then stops, and ResCCL and NCCLX fix their choices before the job. PReCCL and Theseus (both SIGCOMM 2026, both with co-authors from Alibaba Cloud) keep deciding while the job runs. PReCCL (Tsinghua University, Zhongguancun Laboratory and Alibaba Cloud) attacks NCCL's equal split of bytes across channels, which the paper calls virtual topologies. When one channel's NIC or path is congested, that channel sets the completion time, and a dead NIC aborts the job. PReCCL counts how often each channel's sender waits for FIFO credit (the slots of Section 4.6), shares the counts on the collective's own traffic, and at every collective boundary lets all ranks compute the same new byte split. A failed NIC becomes a channel with zero bytes. On a testbed of 32 Hopper GPUs shared by three jobs, PReCCL raised bus bandwidth above 64 MB 1.8× over NCCL for ring all-reduce and 2.1× for all-to-all, and made training iterations 1.18–1.21× faster. After one of eight NICs failed, it kept 78% of the bus bandwidth, where NCCL aborted.

執行期的變更：PReCCL 與 Theseus。AutoCCL 調校一次就停下，ResCCL 與 NCCLX 則在任務開始前就固定它們的選擇。PReCCL 與 Theseus（都在 SIGCOMM 2026，作者都包括阿里雲的研究者）在任務執行時持續做決定。PReCCL（清華大學、中關村實驗室與阿里雲）針對 NCCL 把位元組平均分給各 channel 的做法，論文把 channel 稱為虛擬拓樸（virtual topology）。當某個 channel 的 NIC 或路徑壅塞時，那個 channel 決定完成時間；而一張壞掉的 NIC 會讓任務中止。PReCCL 計算每個 channel 的發送端等待 FIFO 額度（第 4.6 節的槽位）的次數，把這些計數搭在集體操作自己的流量上分享，並在每次集體操作的交界讓所有 rank 算出相同的新位元組分配。故障的 NIC 變成分到零位元組的 channel。在 32 顆 Hopper GPU、由三個任務共用的實驗平台上，PReCCL 讓 64 MB 以上的匯流排頻寬在環形 all-reduce 上達到 NCCL 的 1.8 倍、在 all-to-all 上達到 2.1 倍，並讓訓練迭代快了 1.18–1.21 倍。八張 NIC 壞掉一張後，它保有 78% 的匯流排頻寬，NCCL 則中止。

Theseus (Peking University, UCAS, NUS and Alibaba Cloud) changes whole schedules instead. Programs add schedules with policies and publish signals such as NIC health. Every $K = 100$ requests, one small all-reduce on the bootstrap network makes all GPUs see the same schedules and the same signal versions: it takes the intersection of the sets and the minimum of each version. Each GPU then selects locally, and a change builds only the resources that differ. The testbed has 4 servers of 8 A100 GPUs, each server with four dual-port 100 Gb/s NICs, and the slow NIC is emulated by packet pacing. With one of the 16 NICs capped at 10% of its rate, collectives slowed more than 3× without adaptation and less than 1.5× with it. Llama3-8B iterations took 1.75 s normally and 3.76 s with that cap and no adaptation; with adaptation they took 2.03 s even with the NIC at 0%, where the static schedule stops.

Theseus（北京大學、中國科學院大學、新加坡國立大學與阿里雲）改為抽換整份排程。程式把排程連同策略加進來，並發布 NIC 健康度之類的訊號。每 $K = 100$ 個請求，就在 bootstrap 網路上執行一次小的 all-reduce，讓所有 GPU 看到相同的排程與相同的訊號版本：它取各集合的交集與每個版本的最小值。之後每顆 GPU 在本地選擇，而一次變更只建立有差異的資源。實驗平台有 4 台伺服器，每台 8 顆 A100 與四張雙埠 100 Gb/s 的 NIC，變慢的 NIC 以封包節流（packet pacing）模擬。16 張 NIC 中有一張被限制在 10% 的速率時，沒有自適應的集體操作變慢 3 倍以上，有自適應時不到 1.5 倍。Llama3-8B 的迭代正常時為 1.75 s，在這個限制下沒有自適應時為 3.76 s；有自適應時，即使 NIC 降到 0%，也只要 2.03 s，而靜態排程在 0% 時會停住。

Caveat. NCCLX's numbers are production measurements on Meta's workloads and fabric: strong evidence, but hard to reproduce elsewhere, and some of its microbenchmarks do not name the hardware. The other four papers measure on 32 GPUs or fewer (AutoCCL on A40, ResCCL on A100 and V100, PReCCL on Hopper, Theseus on A100). PReCCL adds a two-week trial on 1,024 H200 GPUs, an observational comparison with the two weeks before it.

保留。NCCLX 的數字是在 Meta 的工作負載與網路上的生產量測：證據很強，但很難在別處重現，而且有些微基準測試沒有說明硬體。另外四篇論文都在 32 顆或更少的 GPU 上量測（AutoCCL 用 A40，ResCCL 用 A100 與 V100，PReCCL 用 Hopper，Theseus 用 A100）。PReCCL 另有一次在 1,024 顆 H200 上為期兩週的試驗，那是與前兩週相比的觀察性比較。

9.4 MoE all-to-all: skewed, and known only at run time9.4 MoE 的 all-to-all：偏斜，而且要到執行時才知道

The problem. Section 3.2's $\tfrac{N-1}{N}\,S$ per rank for all-to-all assumes that every rank sends the same amount to every other rank. Expert parallelism breaks this. An MoE layer runs two all-to-alls in each pass, four per training step with the backward pass: dispatch sends each token to the GPUs that hold its experts, and combine brings the expert outputs back. The router chooses the experts for every token in every step, so the byte matrix is skewed and changes from step to step. Section 4.4 added that a hierarchy cannot remove all-to-all bytes from the network, and that the PXN detour cost 8.2–11.2%. Skew also creates incast, the many-to-one case that Section 4.6's receiver-driven admission must absorb.

問題。第 3.2 節中 all-to-all 每 rank $\tfrac{N-1}{N}\,S$ 的下限，假設每個 rank 送給其他每個 rank 的量都一樣。專家平行打破了這一點。一個 MoE 層在前向與反向各有兩次 all-to-all（每個訓練步共四次）：dispatch 把每個 token 送到持有其專家的 GPU，combine 把專家的輸出送回來。路由器在每一步為每個 token 選擇專家，所以位元組矩陣是偏斜的，而且每一步都在變。第 4.4 節還指出，階層無法把 all-to-all 的位元組從網路上移除，而 PXN 繞路造成 8.2–11.2% 的減速。偏斜也會造成 incast，也就是第 4.6 節的接收端驅動許可必須吸收的多對一情況。

Sizes known only at run time. Section 9.3 listed one weakness of NCCL's interface: the host passes the arguments, and a CUDA graph fixes them when it is captured. For an all-to-all whose sizes change every step, the host must then reserve and send the worst case. NCCLX's AllToAllvDynamic takes the split sizes by reference, in GPU memory, and reads them when the collective runs, so it sends the actual counts, not the maximum. Inside a node it copies over NVLink with several thread blocks; between nodes a CPU thread posts RDMA writes. NCCLX uses it to move MoE tokens in inference; its decode results are in 9.3.

大小要到執行時才知道。第 9.3 節列出了 NCCL 介面的一個弱點：參數由主機傳入，而 CUDA graph 在擷取時就把它們固定下來。對於大小每一步都在變的 all-to-all，主機只好預留並傳送最壞情況。NCCLX 的 AllToAllvDynamic 以參照方式取得放在 GPU 記憶體中的切分大小，並在集體操作執行時才讀取，所以它送的是實際數量，而不是最大值。在節點內，它用多個 thread block 經 NVLink 複製；在節點之間，由 CPU 執行緒提交 RDMA write。NCCLX 用它在推論中搬移 MoE 的 token；解碼的結果列在 9.3。

Skew on the NICs. Skew also loads the NICs unevenly, and the busiest NIC sets the time. On Theseus's 32-GPU testbed, a static all-to-all took up to 2.4× as long at a skew of 2.2 (the busiest expert's load over the average) as with uniform load, although every GPU sent the same number of tokens. Theseus switched to a schedule that balances the NICs whenever the predicted skew passed 1.4. At skew 2.2 this made a 64 MB all-to-all 1.73× faster than NCCL with PXN, and Qwen3-30B-A3B training about 1.07× faster. PReCCL treats a slow pair in an all-to-all differently: it relays part of that pair's traffic through another GPU of the same node, over NVLink.

NIC 上的偏斜。偏斜也讓各張 NIC 的負載不均，而最忙的 NIC 決定耗時。在 Theseus 的 32 顆 GPU 實驗平台上，靜態的 all-to-all 在偏斜度 2.2（最忙專家的負載除以平均）時，耗時最多是負載均勻時的 2.4 倍，雖然每顆 GPU 送出的 token 數都相同。每當預測的偏斜度超過 1.4，Theseus 就換成一份平衡各 NIC 的排程。在偏斜度 2.2 時，這讓 64 MB 的 all-to-all 比使用 PXN 的 NCCL 快 1.73 倍，Qwen3-30B-A3B 的訓練快約 1.07 倍。PReCCL 處理 all-to-all 中變慢的一對 GPU 的方式不同：它把這一對的部分流量經 NVLink 轉給同一節點的另一顆 GPU 中繼。

A schedule for every call. FAST (NSDI 2026; CMU and others) computes a new schedule for each skewed all-to-all. First it moves bytes between the GPUs of one server over the scale-up links, so that every NIC of the server carries the same load. Then it splits the server-to-server byte matrix into one-to-one stages, a Birkhoff decomposition: in each stage every server sends to exactly one other server, so no receiver sees incast, and the busiest server never waits, so the total time reaches the lower bound set by the largest row or column sum. Every rank computes the same schedule on its host CPU from the per-expert token counts that Megatron-LM already exchanges; this took 25 µs for 32 GPUs and 221 µs for 64. On 32 H200 GPUs (4 servers × 8, 400 Gb/s InfiniBand) with Zipf-skewed loads of 100 MB to 1 GB per GPU, the all-to-all ran 1.2–1.3× faster than NCCL with PXN and 1.2–1.5× faster than DeepEP; with no skew it gained nothing. Megatron-LM MoE training on 32 MI300X GPUs over 100 Gb/s RoCE ran 1.18× (EP16) to 4.48× (EP32) faster than with RCCL.

每次呼叫一份排程。FAST（NSDI 2026；CMU 等）為每一次偏斜的 all-to-all 算一份新排程。它先經 scale-up 鏈路在同一台伺服器的 GPU 之間搬移位元組，讓這台伺服器的每張 NIC 負載相同。接著把伺服器對伺服器的位元組矩陣拆成一對一的階段（Birkhoff 分解）：每個階段裡每台伺服器只送給另一台伺服器，所以沒有接收端遇到 incast；最忙的伺服器也從不等待，所以總時間達到由最大列和或行和決定的下界。每個 rank 都在自己的主機 CPU 上，從 Megatron-LM 本來就會交換的各專家 token 數算出同一份排程；32 顆 GPU 花 25 µs，64 顆花 221 µs。在 32 顆 H200（4 台伺服器 × 8，400 Gb/s InfiniBand）上，每顆 GPU 100 MB 到 1 GB、呈 Zipf 偏斜的負載下，all-to-all 比使用 PXN 的 NCCL 快 1.2–1.3 倍，比 DeepEP 快 1.2–1.5 倍；沒有偏斜時則沒有增益。在 32 顆 MI300X、100 Gb/s RoCE 上，Megatron-LM 的 MoE 訓練比使用 RCCL 快 1.18 倍（EP16）到 4.48 倍（EP32）。

UBEP (SIGCOMM 2026; Nanjing University and Huawei; its digest is listed with the parallelism research) targets superpods with large scale-up domains, such as NVL72/576 and CloudMatrix384. Expert-parallel libraries there orchestrate the all-to-all in the bulk-synchronous parallel (BSP) way: each phase starts only after every rank finishes the previous one. UBEP names three bottlenecks of this approach; the table maps each one to a mechanism from earlier sections. On 64 NPUs of a production CloudMatrix384, UBEP cut dispatch latency by up to 52.4% against CANN EP, a DeepEP-like BSP library for Ascend, and it cut the P99 time per output token (TPOT) of MoE inference by up to 11.1%.

UBEP（SIGCOMM 2026；南京大學與華為；它的導讀列在平行化研究那一頁）針對具有大型 scale-up domain 的 superpod，例如 NVL72/576 與 CloudMatrix384。那裡的專家平行函式庫以大量同步平行（BSP）的方式編排 all-to-all：每個階段都要等所有 rank 完成前一個階段才開始。UBEP 指出這種做法的三個瓶頸；下表把每個瓶頸對應到前面各節的機制。在一套生產環境 CloudMatrix384 的 64 顆 NPU 上，UBEP 讓 dispatch 延遲比 CANN EP（類似 DeepEP、為 Ascend 設計的 BSP 函式庫）降低最多 52.4%，MoE 推論的 P99 每個輸出 token 時間（TPOT）降低最多 11.1%。

Caveat. UBEP and AllToAllvDynamic were measured in inference, and UBEP measured only dispatch. Theseus's training gain from balancing the NICs is about 7%. FAST's largest training gains are against RCCL on 100 Gb/s RoCE, where scale-up is 35 times faster than scale-out; against NCCL on H200 its all-to-all gain is 1.2–1.3×, and it needs scale-up links fast enough to hide the redistribution. SyCCL's symmetry does not hold for all-to-all with skewed sizes, PReCCL leaves it out, and EPIC leaves it to future work.

保留。UBEP 與 AllToAllvDynamic 都是在推論中量測，而 UBEP 只量了 dispatch。Theseus 靠平衡 NIC 得到的訓練增益約 7%。FAST 最大的訓練增益是相對 100 Gb/s RoCE 上的 RCCL，那裡 scale-up 比 scale-out 快 35 倍；在 H200 上相對 NCCL，它的 all-to-all 增益是 1.2–1.3 倍，而且它需要夠快的 scale-up 鏈路來藏住重新分配。SyCCL 的對稱性對大小偏斜的 all-to-all 不成立，PReCCL 不處理它，EPIC 則留待未來的工作。

9.5 Shrink the bytes, or reduce them in the network9.5 讓位元組變少，或在網路中歸約

The problem. Section 3.2 and Section 4.3 called $\tfrac{2(N-1)}{N}\,S$ per rank the lower bound for an all-reduce. It holds only for endpoints that do all the additions and send every bit. Two ways around it remain: the ranks can send fewer bits for the same values, or a switch can add, which cuts the traffic to about $S$ per direction (Section 4.4).

問題。第 3.2 節與第 4.3 節把每 rank $\tfrac{2(N-1)}{N}\,S$ 稱為 all-reduce 的下限。它只在「所有加法都由端點做、每個位元都要送」時成立。還剩兩條繞過它的路：讓各 rank 用更少的位元送出同樣的值；或者讓交換器做加法，把流量減到每個方向約 $S$（第 4.4 節）。

ZipCCL (SIGCOMM 2026; HIT Shenzhen and HKUST) compresses without loss. A BF16 value has 1 sign bit, 8 exponent bits and 7 mantissa bits. LLM tensors are close to Gaussian, so their exponents concentrate on a few values; for values sampled from a normal distribution, the paper's Table 1 computes that the 7 most frequent exponents hold 97.5% of them. ZipCCL maps these 7 values to the 3-bit codes 001–111; code 000 is an escape, and the full 8-bit exponent goes into a separate "zero-point" list. Sign and mantissa stay raw, which keeps a fixed stride in memory, and the top-7 set comes from the tensor's standard deviation, so no histogram is computed at run time. We call this exponent coding.

ZipCCL（SIGCOMM 2026；哈爾濱工業大學深圳與香港科技大學）做無損壓縮。一個 BF16 值有 1 個符號位元、8 個指數位元與 7 個尾數位元。LLM 的張量接近高斯分佈，所以指數集中在少數幾個值；對於從常態分佈取樣的值，論文的 Table 1 算出最常見的 7 個指數涵蓋其中 97.5%。ZipCCL 把這 7 個值對應到 3 位元代碼 001–111；代碼 000 是跳脫碼，完整的 8 位元指數另外放進一份「zero-point」清單。符號與尾數保持原樣，讓記憶體中的步距固定；前 7 名的集合由張量的標準差推得，所以執行時不必算直方圖。我們稱之為指數編碼。

The measured ratio on Qwen3-A22B all-to-all traffic was 1.33×, below the 1.45× ceiling, because escaped exponents, indices and padding add bytes. Every addition stays in high precision: ZipCCL's reduce-scatter is a compressed all-to-all followed by a local reduction, and an adaptive switcher picks it only when that path is faster than the native reduce-scatter. On 64 RTX A6000 GPUs, 8 per server on a 200 Gb/s network, communication was up to 1.35× faster (Llama3-8B all-gather) and about 1.25× faster for all-to-all (DeepSeek-V3, Qwen3-MoE), with end-to-end speedups of 1.16–1.18×. On 16 H800 GPUs with a 400 Gb/s network the end-to-end gain was 1.13×. Over NVLink the same codec made collectives slower: on 8 H800 GPUs a 1 GB all-to-all took 5.05 ms instead of 2.96 ms, while over RDMA on 16 H800 GPUs it took 76.71 ms instead of 101.76 ms. The paper's model explains both: compression pays only while the link is slower than a break-even bandwidth that grows with the codec's throughput, measured at 1.2–1.3 TB/s.

在 Qwen3-A22B 的 all-to-all 流量上量到的壓縮比是 1.33 倍，低於 1.45 倍的上限，因為跳脫的指數、索引與填補都會增加位元組。每一次加法都維持高精度：ZipCCL 的 reduce-scatter 是一個壓縮的 all-to-all 加上本地歸約，而自適應切換器只有在這條路徑比原生 reduce-scatter 快時才選它。在 64 顆 RTX A6000（每台伺服器 8 顆、200 Gb/s 網路）上，通訊快最多 1.35 倍（Llama3-8B 的 all-gather），all-to-all 快約 1.25 倍（DeepSeek-V3、Qwen3-MoE），端到端加速 1.16–1.18 倍。在 400 Gb/s 網路上的 16 顆 H800，端到端增益是 1.13 倍。經 NVLink 時，同一套編解碼器反而讓集體操作變慢：8 顆 H800 上 1 GB 的 all-to-all 從 2.96 ms 變成 5.05 ms；而在 16 顆 H800 上經 RDMA 時，則從 101.76 ms 降到 76.71 ms。論文的模型同時解釋了兩者：只有當鏈路比某個損益平衡頻寬慢時，壓縮才划算，而這個頻寬隨編解碼器的吞吐量增加；量到的吞吐量是 1.2–1.3 TB/s。

DynamiQ (SIGCOMM 2026; UCL, Broadcom and Harvard University) compresses with loss, and asks what happens in a multi-hop all-reduce. Compressors built for a parameter server compress once, and the server adds in high precision. In a ring or a butterfly, every intermediate GPU adds to a partial sum and must recompress it, so the error grows hop by hop. DynamiQ first runs a small all-reduce of per-block statistics, so that all GPUs agree on 2, 4 or 8 bits for each block of 256 values, within an average of 5 bits. Then one fused kernel per hop decompresses, adds and recompresses, with non-uniform levels, two-level scales and correlated rounding. On four servers with two RTX A6000 GPUs each and one 100 Gb/s NIC per server, fine-tuning LLaMA 1B reached 99% of BF16's accuracy 34.5% faster than MXFP8 and 40.8% faster than BF16, and the final accuracy stayed within 0.1% of BF16. The hardware path has 4 workers; the paper's results for more workers are simulation.

DynamiQ（SIGCOMM 2026；倫敦大學學院、Broadcom 與哈佛大學）做有損壓縮，並問多跳 all-reduce 裡會發生什麼事。為參數伺服器設計的壓縮器只壓縮一次，由伺服器以高精度相加。在環或 butterfly 中，每顆中間 GPU 都要把資料加進部分和，再把它重新壓縮，所以誤差一跳一跳地增長。DynamiQ 先對每個區塊的統計值做一次小的 all-reduce，讓所有 GPU 對每個 256 值的區塊取得一致的 2、4 或 8 位元寬度，平均不超過 5 位元。接著每一跳用一個融合的 kernel 解壓縮、相加、再壓縮，並採用非均勻的量化等級、兩層的縮放因子與相關的捨入。在四台伺服器（每台兩顆 RTX A6000、一張 100 Gb/s NIC）上微調 LLaMA 1B，達到 BF16 精度的 99% 比 MXFP8 快 34.5%、比 BF16 快 40.8%，最終精度與 BF16 的差距在 0.1% 以內。硬體上的路徑只有 4 個 worker；論文中更多 worker 的結果是模擬。

EPIC (SIGCOMM 2026; Peking University with Alibaba Cloud and vendors) asks why in-network collectives are rare on Ethernet. SHARP is proprietary and uses vendor headers, SwitchML and ATP rely on host stacks such as DPDK, and the feature spans the library, the NIC, the switch and a controller, so no single vendor can build it. EPIC defines the IncTree: ranks are the leaves, switches are the inner nodes, and endpoints are named by IP address and QP. On it EPIC offers AllReduce, Reduce and Broadcast directly, and derives Barrier, ReduceScatter and AllGather from them. Hosts keep standard RoCE NICs, and a switch runs in one of three modes:

EPIC（SIGCOMM 2026；北京大學與阿里雲及多家廠商）問的是：為什麼網內集體操作在乙太網路上很少見？SHARP 是專有的、使用廠商自訂標頭，SwitchML 與 ATP 依賴 DPDK 之類的主機協定堆疊，而這個功能橫跨函式庫、NIC、交換器與控制器，所以沒有單一廠商能獨力做出來。EPIC 定義了 IncTree：rank 是葉節點，交換器是內部節點，端點以 IP 位址與 QP 命名。在它上面，EPIC 直接提供 AllReduce、Reduce 與 Broadcast，再由它們衍生出 Barrier、ReduceScatter 與 AllGather。主機沿用標準 RoCE NIC，交換器以三種模式之一運作：

EPIC's hardware testbed is small: four servers, each with two RTX 3090 GPUs and two 100 GbE NICs, form 8 ranks behind one Tofino switch, which runs Mode II with integer arithmetic. On it a 1 GB AllReduce reached 65.3 Gb/s. SwitchML reached the same, 65.4 Gb/s, but needed 6 CPU cores against EPIC's 1. Against NCCL, and in training, the ratios are:

EPIC 的硬體實驗平台很小：四台伺服器，每台有兩顆 RTX 3090 與兩張 100 GbE NIC，組成一台 Tofino 交換器後面的 8 個 rank；這台交換器以整數運算執行 Mode II。在這個平台上，1 GB 的 AllReduce 達到 65.3 Gb/s。SwitchML 也達到相同的 65.4 Gb/s，但需要 6 個 CPU 核心，EPIC 只要 1 個。與 NCCL 相比以及在訓練中的比值如下：

The paper's "1.59×" is the first row; against NCCL's default it is 1.36×. Tensor parallelism gains most because communication is a large share of its step; the best case, OPT-350m with TP = 8, was 1.38×. Production systems usually keep tensor parallelism inside the NVLink domain, so the data-parallel row is closer to their case.

論文的「1.59 倍」是第一列；相對 NCCL 的預設是 1.36 倍。張量平行的增益最大，因為通訊佔它每一步的比例很大；最好的情況是 TP = 8 的 OPT-350m，達 1.38 倍。生產系統通常把張量平行留在 NVLink 網域內，所以資料平行那一列更接近它們的情況。

The switch state is small. By the paper's analysis, Mode II needs about 250 KB per job at 100 Gb/s and a 10 µs round trip, twice the path's bandwidth-delay product, and the prototype platforms had 1–16 MB of on-chip SRAM. This answers, for this design, Section 4.4's concern about switch state per collective. A Xilinx VU13P FPGA ran the engine at the 100 Gb/s line rate and measured 96–244 ns per 1 KB packet; the 53 ns that the paper's Table 1 lists for an FPGA comes from a cycle-accurate RTL simulation of an Intel Agilex design. The authors also model-checked Modes II and III with TLA+ for AllReduce, Reduce and Broadcast on a small tree (depth 3, two branches, three packets per rank, at most one loss at a time). On networks that lose and reorder packets, every explored run gave the correct sum and terminated, and the check found a buffer-recycling bug when Mode II was extended to Mode III.

交換器的狀態很小。依論文的分析，在 100 Gb/s、往返 10 µs 時，Mode II 每個任務約需 250 KB，也就是路徑頻寬延遲積的兩倍；而各原型平台的晶片內 SRAM 有 1–16 MB。這對它自己的設計回答了第 4.4 節關於交換器要為每個集體操作保存狀態的疑慮。一片 Xilinx VU13P FPGA 以 100 Gb/s 線速執行這個引擎，量到每個 1 KB 封包 96–244 ns；論文 Table 1 為 FPGA 列出的 53 ns，則來自 Intel Agilex 設計的週期精確 RTL 模擬。作者也以 TLA+ 在一棵小樹（深度 3、兩個分支、每個 rank 三個封包、任何時刻最多遺失一個）上，對 Mode II 與 III 的 AllReduce、Reduce 與 Broadcast 做了模型檢查。在會遺失與重排封包的網路上，每一條探索過的執行路徑都得到正確的總和並且終止；從 Mode II 擴充成 Mode III 時，這項檢查還找到一個緩衝區回收的錯誤。

HyNA (SIGCOMM 2026; Beijing University of Posts and Telecommunications) attacks the cost that Section 4.4 warned about: switch state. A switch pipeline adds only integers, at line rate, in fixed slots of SRAM. When two jobs' gradients hash to the same slot, or a sum overflows, ATP sends the packet to a host server; the paper puts that detour at 10–50 µs, against about 500 ns in the pipeline. In synchronous training the slowest packet holds every worker. HyNA keeps these exceptions on the switch die: RISC-V cores beside the pipeline take each one over an on-chip bus (under 200 ns by design), resolve collisions by cuckoo hashing into the switch's packet-buffer memory, and move an overflowed slot to FP32. On an Alveo U280 FPGA prototype at 100 GbE, where two servers replay traffic for up to 256 workers, HyNA reached 84.5 Gb/s of goodput, against 60 Gb/s for ATP re-implemented on the same FPGA, and kept at least 94.8% of it at a 5% collision rate. It speeds up only the gradient all-reduce of MoE training, not the all-to-all.

HyNA（SIGCOMM 2026；北京郵電大學）針對第 4.4 節警告過的代價：交換器的狀態。交換器管線只能以線速做整數加法，而且用的是 SRAM 中固定的槽位。當兩個任務的梯度雜湊到同一個槽位，或者總和溢位時，ATP 會把封包送到主機伺服器；論文估計這段繞路要 10–50 µs，而管線內只要約 500 ns。在同步訓練中，最慢的封包會拖住每個 worker。HyNA 把這些例外留在交換器晶片上：管線旁的 RISC-V 核心經晶片內匯流排（設計值不到 200 ns）接手每一個例外，以 cuckoo 雜湊把碰撞移到交換器的封包緩衝記憶體，並把溢位的槽位改成 FP32。在 100 GbE 的 Alveo U280 FPGA 原型上（兩台伺服器重播最多 256 個 worker 的流量），HyNA 達到 84.5 Gb/s 的有效吞吐，在同一片 FPGA 上重新實作的 ATP 則是 60 Gb/s；碰撞率 5% 時，HyNA 仍保有至少 94.8%。它只加速 MoE 訓練中梯度的 all-reduce，不加速 all-to-all。

Caveat. The hardware evidence here is small. EPIC's tree has one switch and 8 ranks; HyNA's is one FPGA switch fed by two traffic-replay servers; ZipCCL tops out at 64 GPUs; DynamiQ's multi-hop path has 4 workers. ZipCCL is exactly lossless, but its ratio depends on how close to Gaussian the tensors are. DynamiQ changes the numbers and must control the error, and switches add integers, so EPIC's Tofino path and HyNA quantize floats first. Compression pays most where the link is slow and communication is a large share of the step: ZipCCL's end-to-end gain was larger on A6000 than on H800 GPUs, and over NVLink it was a loss.

保留。這裡的硬體證據規模都很小。EPIC 的樹只有一台交換器與 8 個 rank；HyNA 是由兩台重播流量的伺服器餵資料的一台 FPGA 交換器；ZipCCL 最多到 64 顆 GPU；DynamiQ 的多跳路徑只有 4 個 worker。ZipCCL 是完全無損的，但壓縮比取決於張量有多接近高斯分佈。DynamiQ 會改變數值，必須控制誤差；交換器做的是整數加法，所以 EPIC 的 Tofino 路徑與 HyNA 都要先把浮點數量化。通訊佔一步的比例越大、鏈路越慢，壓縮的回報越高：ZipCCL 在 A6000 上的端到端增益比在 H800 上大，而經 NVLink 時則是虧損。

What the papers agree on, and what is still open論文的共識，以及仍未解決的問題

1. Leave the GPU to compute. NCCLX moves data with the copy engine and RDMA, without staging copies, and FTAR needs half of NCCL's default thread blocks. ResCCL shares thread blocks between connections that are never active together (18 → 4 in one case). AutoCCL measured how far a GEMM beside a collective moves its best settings. The reason is Section 4.7: overlap pays only if the collective does not take the resources that the compute needs.把 GPU 留給計算。NCCLX 用 copy engine 與 RDMA 搬資料、不做分段複製，FTAR 只需 NCCL 預設一半的 thread block。ResCCL 讓從不同時活動的連線共用 thread block（其中一例從 18 個降到 4 個）。AutoCCL 量到集體操作旁邊的 GEMM 會讓最佳設定移動多少。理由在第 4.7 節：只有當集體操作不搶走計算所需的資源時，重疊才划算。
2. Use the structure of the fabric. SyCCL solves one group of a Clos or multi-rail fabric and reuses the schedule for the isomorphic ones, and OptCCL solves one subproblem per class of identical GPUs. MCCS orders the ring by rack. Swing and Trivance choose peers by distance on a torus. NCCLX sets DQPLB's bound per distance class, and UBEP names distance-unaware token routing as a bottleneck. A schedule that ignores where the ranks sit leaves bandwidth unused, as SyCCL's analysis of NCCL's ring shows.利用網路的結構。SyCCL 只解 Clos 或多 rail 網路中的一個群組，再把排程重用到同構的群組；OptCCL 對每一類相同的 GPU 只解一個子問題。MCCS 依機櫃排列環。Swing 與 Trivance 在環面上依距離挑選夥伴。NCCLX 按距離類別設定 DQPLB 的上限，UBEP 則把不考慮距離的 token 路由列為瓶頸。忽略 rank 位置的排程會讓頻寬閒置，正如 SyCCL 對 NCCL 環的分析所示。
3. Decide at run time, and agree on when. NCCLX reads all-to-all sizes on the GPU when the collective runs, ZipCCL's switcher uses the compressed reduce-scatter only when it is faster, AutoCCL tunes on the job's own collectives, and PReCCL and Theseus keep adapting for the whole job. NCCL's auto-selection (Section 4.5) was the first such run-time choice. Every change while a job runs needs all ranks to switch at the same collective. MCCS switches after the largest collective number that any rank has launched, PReCCL tags each collective with an epoch and computes the new split the same way on every rank, and Theseus agrees on versions every 100 requests. The solvers, TE-CCL, SyCCL and OptCCL, still compute the schedule before the job, from a known demand.留到執行時決定，並約定何時切換。NCCLX 在集體操作執行時於 GPU 上讀取 all-to-all 的大小，ZipCCL 的切換器只在壓縮版 reduce-scatter 比較快時才選它，AutoCCL 在任務自己的集體操作上調校，PReCCL 與 Theseus 則在整個任務期間持續自適應。NCCL 的自動選擇（第 4.5 節）是第一個這類的執行期選擇。任務執行中的每一次變更，都需要所有 rank 在同一個集體操作切換。MCCS 在任一 rank 已啟動的最大集體操作編號之後切換，PReCCL 為每次集體操作標上 epoch，並在每個 rank 上以相同方式算出新的分配，Theseus 則每 100 個請求就對版本達成一致。求解器 TE-CCL、SyCCL 與 OptCCL 仍在任務開始前、依已知的需求算出排程。
4. Go below the endpoint bound. EPIC and HyNA add in the switch, ZipCCL sends 11 bits instead of 16 without loss, and DynamiQ sends about 5 bits per value with a controlled error. All four go below the byte counts of Section 3.2, which assume that the endpoints do every addition and send every bit.突破端點的下限。EPIC 與 HyNA 在交換器裡做加法，ZipCCL 無損地用 11 位元取代 16 位元，DynamiQ 則在誤差受控下每個值只送約 5 位元。四者都低於第 3.2 節的位元組數，因為那些數字假設所有加法都由端點做、每個位元都要送。
5. Collective gains shrink at the step. SyCCL's bus bandwidth rose by up to 108%, but its iterations got only 0.4–6.3% faster. Theseus made a skewed 64 MB all-to-all 1.73× faster, but MoE training only about 1.07× faster. EPIC's data-parallel training gained about 1.2%. Section 4.7 explains all three: only exposed communication adds to the step.集體操作的增益到了訓練步就縮小。SyCCL 的匯流排頻寬提高最多 108%，迭代卻只快了 0.4–6.3%。Theseus 讓偏斜的 64 MB all-to-all 快 1.73 倍，MoE 訓練卻只快約 1.07 倍。EPIC 的資料平行訓練只多了約 1.2%。第 4.7 節解釋了這三者：只有暴露的通訊會加到步時間上。

Three questions remain open. How fast must a schedule be? SyCCL needs about 37 minutes at 512 GPUs and OptCCL about 30 minutes at 256, while the sizes of an MoE all-to-all are known only when it starts. FAST answers this for all-to-all alone: its schedule takes microseconds to milliseconds on the host (221 µs at 64 GPUs, 77 ms at 320), but it assumes one job on a full-bisection scale-out tier. For the other collectives, Theseus can swap in better schedules while a job runs, but no paper here synthesizes one for each step. How strong is the evidence? The testbeds are small (8 to 64 GPUs, and one switch for EPIC and HyNA), the torus algorithms and OptCCL's schedule quality rest on cost models and simulation, and production results such as NCCLX's are hard to reproduce. Do the techniques combine? No paper here synthesizes a schedule for compressed data or adds compressed values inside a switch, and none tests many adaptive jobs on shared links; PReCCL tests two.

仍有三個問題懸而未決。排程必須多快？SyCCL 在 512 顆 GPU 上需要約 37 分鐘，OptCCL 在 256 顆上約 30 分鐘，而 MoE all-to-all 的大小要到它開始時才知道。FAST 只為 all-to-all 回答了這個問題：它在主機上算排程只要微秒到毫秒（64 顆 GPU 221 µs，320 顆 77 ms），但它假設單一任務跑在全對分頻寬的 scale-out 層上。至於其他集體操作，Theseus 能在任務執行時換上更好的排程，但這裡沒有一篇論文為每一步合成排程。證據有多強？實驗平台都很小（8 到 64 顆 GPU，EPIC 與 HyNA 只有一台交換器），環面演算法與 OptCCL 的排程品質靠的是成本模型與模擬，而 NCCLX 這類生產結果又很難重現。技術能結合嗎？這裡沒有一篇論文為壓縮過的資料合成排程，或在交換器內把壓縮過的值相加，也沒有一篇測試共用鏈路上的許多個自適應任務；PReCCL 只測了兩個。

The 16 digests side by side, in the order of the reading map. "Measured" means a testbed, a hardware prototype or production; model values are labelled.

16 篇導讀並列，依閱讀地圖的順序。「實測」指實驗平台、硬體原型或生產環境；模型值另有標示。

Panel R4.1 picks the measured results of this table that apply to one collective and one message size.

面板 R4.1 從這張表中挑出適用於某一種集體操作與某個訊息大小的實測結果。

Each card below opens one digest.

下面每張卡片打開一篇導讀。