Research digest 9 of 16 · Chapter 4: Collective communication

研究導讀 9 / 16 · 第 4 章：集體通訊

Theseus: Runtime-Adaptive GPU Collective Communication with Hot-Swappable SchedulesTheseus：以可熱抽換排程實現執行期自適應的 GPU 集體通訊

SIGCOMM 2026Rui Ding, Xiandong Lu, Jiajun Wang, …, Qun Huang, Jiamin Cao, Jiaqi Gao (Peking University, UCAS, NUS, Alibaba Cloud) · paper論文

After reading, you can explain how Theseus swaps a collective's schedule mid-job: who signals the change, how all GPUs agree, and what a switch costs.讀完後，你能說明 Theseus 如何在任務執行中更換集體操作的排程：由誰發出訊號、所有 GPU 如何取得一致，以及一次切換的代價。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 4 ↗第 4 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

NCCL fixes its schedules and its selection rule when a communicator starts, but long jobs change: MoE loads shift, NICs slow down, solvers find better schedules hours later. Theseus, a drop-in NCCL replacement, makes the schedule a run-time variable. Programs add schedules with policies and publish signals such as NIC health. Every $K$ requests, one small AllReduce aligns the inputs of all GPUs; each GPU then selects locally, and a switch builds only what changed. On 32 A100 GPUs with one NIC capped at 10%, collectives slowed more than 3× without adaptation and less than 1.5× with it.

NCCL 在 communicator 建立時就固定了排程與選擇規則，但長時間的任務會變：MoE 負載會移動、NIC 會變慢、求解器幾小時後才找到更好的排程。Theseus 是 NCCL 的直接替換品，把排程變成執行期的變數。程式可以加入帶有策略的排程，並發布 NIC 健康之類的訊號。每 $K$ 個請求，一次小型 AllReduce 讓所有 GPU 的輸入一致；接著每顆 GPU 在本地選擇，切換時只建立改變的部分。在 32 顆 A100 上把一張 NIC 限速到 10%，不調適時集體操作慢了 3 倍以上，調適後慢不到 1.5 倍。

Where this paper sits這篇論文的位置

Section 4.5 showed how NCCL chooses: a cost model ranks every (algorithm, protocol) pair for the message size. Panel 4 drew schedules as fixed step sequences; section 9 showed solvers that write better ones. But NCCL's menu is fixed at start (v2.25 has 13 predefined schedules, out of 21 combinations of 7 algorithms and 3 protocols), its inputs ignore NIC health and expert loads, and a new schedule needs a restart. Section 9 asked whether synthesis can follow run-time traffic or a runtime must pick from schedules computed in advance; Theseus builds the second option.

第 4.5 節說明了 NCCL 怎麼選：成本模型依訊息大小為每個（演算法、協定）組合排名。面板 4 把排程畫成固定的步驟序列；第 9 節則介紹能寫出更好排程的求解器。但 NCCL 的選單在啟動時就固定（v2.25 有 13 個預先定義的排程，取自 7 種演算法乘 3 種協定共 21 種組合），輸入不含 NIC 健康與專家負載，換新排程還必須重啟。第 9 節問過：排程合成跟得上執行時的流量嗎，還是必須由執行期從事先算好的排程中挑選？Theseus 實作的是第二條路。

The problem, measured問題，用量測說明

On their testbed (4 servers, each with 8 A100 GPUs and 4 dual-port 100 Gb/s NICs), the authors measured four cases. Shifting loads: a static MoE all-to-all took up to 2.4× longer at a skewness of 2.2 (maximum over average expert load) than at uniform load, because the busiest NIC sets the time. A fail-slow NIC: with one NIC capped at 10%, AllReduce throughput fell to 16%; near 0 Gb/s it stopped. Slow synthesis: on a 32-GPU, 1 GB AllGather, TE-CCL needed 13 hours to beat NCCL and 49 hours to reach its best schedule. A wrong cost model: NCCL's estimates for Ring AllReduce were off by up to $\pm 42\%$ , so at 1 MB it picked the wrong protocol:

作者在測試平台上（4 台伺服器，每台 8 顆 A100 GPU 與 4 張雙埠 100 Gb/s NIC）量測了四種情況。負載移動：偏斜度（專家負載的最大值除以平均值）為 2.2 時，靜態的 MoE all-to-all 最多比均勻負載時慢 2.4 倍，因為最忙的 NIC 決定了時間。fail-slow 的 NIC：一張 NIC 限速 10% 時，AllReduce 吞吐量掉到 16%；接近 0 Gb/s 時完全停住。合成太慢：在 32 顆 GPU、1 GB 的 AllGather 上，TE-CCL 花了 13 小時才勝過 NCCL，49 小時才找到最佳排程。成本模型錯誤：NCCL 對 Ring AllReduce 的估計誤差可達 $\pm 42\%$ ，所以在 1 MB 時選錯了協定：

Question (1 MB Ring AllReduce)問題（1 MB Ring AllReduce）	Calculation算式	Result結果
LL128, picked: estimated 204 µs, measured 261 µsLL128（被選）：估計 204 µs，實測 261 µs	$(204 - 261) \div 261$	$\approx -22\%$
LL, rejected: estimated 285 µs, measured 247 µsLL（落選）：估計 285 µs，實測 247 µs	$(285 - 247) \div 247$	$\approx +15\%$
Cost of the wrong pick選錯的代價	$261 \div 247$	$\approx 1.06\times$ slower than LL比 LL 慢

So a library needs selection driven by outside signals, and hot swapping: changing the schedule between calls without stopping the job.

所以函式庫需要由外部訊號驅動的選擇，以及熱抽換，也就是在呼叫之間更換排程而不停下任務。

Figures 2 to 5 of the Theseus paper: relative all-to-all CCT against skewness, AllReduce throughput against a NIC rate cap, TE-CCL schedule quality against solver hours, and NCCL's CCT estimation error per protocol — Figures 2–5 of the paper: all-to-all time against skew; AllReduce throughput against one NIC's cap; TE-CCL's schedules against solver hours (red: NCCL); NCCL's estimation error per protocol. All measured.論文的 Figure 2–5：all-to-all 時間隨偏斜度的變化；AllReduce 吞吐量隨單張 NIC 上限的變化；TE-CCL 的排程隨求解時數的變化（紅線：NCCL）；NCCL 依協定的估計誤差。全部是量測值。Screenshot of Ding et al., SIGCOMM 2026, Figures 2–5 (p. 224).

The design設計

A communicator is the library's endpoint on one GPU; a request is one collective call on it. Matching requests on all GPUs must run the same schedule.

communicator 是函式庫在一顆 GPU 上的端點；請求是在它上面的一次集體操作呼叫。所有 GPU 上互相對應的請求必須執行同一個排程。

1. Signals, schedules and policies1. 訊號、排程與策略

The inputs form the selection context. An endogenous attribute is a library metric, such as a request counter, that a hook updates; it is equal on all GPUs. An exogenous attribute is an outside signal, such as a NIC health map, with a new version at every update. Each schedule has a policy: a usability predicate, a score, and an optional replacement that names the schedule it adapts. Asynchronous calls (NewAttr, SetExo, NewSched) write the context into logs that a daemon replicates with Raft, off the request path. The selector computes

這些輸入構成選擇情境。內生屬性是由 hook 更新的函式庫指標，例如請求計數器，在所有 GPU 上都相同。外生屬性是外部訊號，例如 NIC 健康圖，每次更新都有新版本。每個排程帶一個策略：可用性判斷式、分數，以及選用的取代（replacement），指出它改編自哪個排程。非同步呼叫（NewAttr、SetExo、NewSched）把情境寫進日誌，由 daemon 以 Raft 複製，不在請求路徑上。選擇器計算

\begin{gathered} \mathcal{U}(r) = \{\, S : \mathrm{usable}_S(r, C_{\mathcal{A}}) \,\} \\ S^{*} = \arg\max_{S \in \mathcal{U}(r)} \mathrm{score}_S(C_{\mathcal{A}}) \end{gathered}

for request $r$ over the agreed context $C_{\mathcal{A}}$ . If no schedule is usable, a predefined one runs, and a tie goes to the schedule that replaces the others. The fail-slow policy relies on that tie-break (figure below): $S_\Delta$ has the same score as $S$ , replaces it, and is usable only while some health value is below $\theta$ . A monitor, not Theseus, detects slow NICs.

其中 $r$ 是請求， $C_{\mathcal{A}}$ 是取得一致後的情境。若沒有可用的排程就執行預先定義的排程；平手時，選取代了其他排程的那一個。fail-slow 策略就靠這個平手規則（見下圖）： $S_\Delta$ 的分數與 $S$ 相同、宣告取代 $S$ ，而且只在某個健康值低於 $\theta$ 時可用。偵測慢 NIC 的是監測器，不是 Theseus。

The paper's Policy 2 on one node. NIC 2's health is below

\theta

, so both schedules are usable with equal scores, and the replacement link picks

S_\Delta

.論文的 Policy 2（一個節點）。NIC 2 的健康值低於

\theta

，所以兩個排程都可用且分數相同，由取代關係選出

S_\Delta

。Drawn for this digest from §4.1–§4.2 of the paper; health values are illustrative.

2. Agreeing on inputs, not on decisions2. 對輸入取得一致，而不是對決定

Logs reach nodes at different times, so GPUs could select from different versions. Instead of a barrier or per-decision consensus, Theseus uses two facts: all GPUs issue requests in the same order, so the request ID is a shared clock, and the context changes far less often than requests arrive. When the request ID is a multiple of $K$ , all communicators run one AllReduce over the bootstrap network (the TCP side channel for setup) with this reducer:

日誌到達各節點的時間不同，所以兩顆 GPU 可能依不同版本選擇。Theseus 不用 barrier，也不為每個決定跑一輪共識，而是利用兩個事實：每顆 GPU 以相同順序發出請求，所以請求編號是共享的時鐘；情境變動的頻率也遠低於請求到達的頻率。當請求編號是 $K$ 的倍數時，所有 communicator 透過 bootstrap 網路（建立連線用的 TCP 旁路通道）跑一次 AllReduce，使用以下 reducer：

\begin{gathered} C[A] = C_1[A] \cap C_2[A] \\ C[S] = C_1[S] \cap C_2[S] \\ C[v_a] = \min\big(C_1[v_a],\, C_2[v_a]\big) \end{gathered}

Here $A$ is the attribute set, $S$ the schedule set and $v_a$ the version of attribute $a$ . The result is the newest context that every GPU already holds, so no GPU fetches anything and one step suffices. The AllReduce carries version numbers, not attribute values. Appendix E of the paper proves that all GPUs then select the same schedule.

其中 $A$ 是屬性集合， $S$ 是排程集合， $v_a$ 是屬性 $a$ 的版本。結果是每顆 GPU 都已持有的最新情境，所以沒有 GPU 需要抓資料，一步就夠。AllReduce 只帶版本號，不帶屬性值。論文附錄 E 證明所有 GPU 因此選出同一個排程。

The paper's Figure 8 example. M1 has read schedule S2 and A2's v1–v2; M0 has not. Both install only what both hold.論文 Figure 8 的例子。M1 已讀到排程 S2 與 A2 的 v1–v2，M0 還沒有。兩者只安裝雙方都有的內容。Drawn for this digest from §5 and Figure 8 of the paper.

The price is delay: a new value takes effect at the first agreement where every GPU has it.

代價是延遲：新數值要到每顆 GPU 都已持有它的第一次協議才生效。

One switch (illustrative request numbers). GPU 2 gets v8 after the agreement at request 200, so that round keeps

S

; at request 300 all GPUs switch together. The first request on each path pays a delta migration (mean 8 ms).一次切換（請求編號只是示意）。GPU 2 在第 200 個請求的協議之後才收到 v8，所以那一輪維持

S

；到第 300 個請求，所有 GPU 一起切換。每條路徑上的第一個請求要付一次 delta migration（平均 8 ms）。Drawn for this digest from Algorithm 1 and §6 of the paper.

3. Switching cheaply: the resource tree and delta migration3. 便宜地切換：資源樹與 delta migration

Building a schedule's resources (scratch buffer, synchronization flags, peers' buffer handles, channels, low-level operations or OPs) takes many control messages, and NCCL builds them all at start. Theseus caches them in a four-layer tree, each at the highest layer where it stays valid: L0 global (scratch buffer), L1 per schedule (synchronization), L2 per buffer (handles, channels) and L3 per buffer range (OPs, which hold address offsets). A request walks from root to leaf: a miss builds, a hit reuses, and an unselected schedule uses no GPU memory. When a schedule replaces another, delta migration builds only the difference, lazily, along the current request's path: a vertex whose schedule tag differs from the selected schedule is migrated.

建立排程的資源（scratch 緩衝區、同步旗標、對端的緩衝區 handle、channel、低階操作 OP）需要許多控制訊息，而 NCCL 在啟動時就全部建好。Theseus 把它們快取在一棵四層的樹中，每個資源放在仍然有效的最高一層：L0 全域（scratch 緩衝區）、L1 每個排程（同步）、L2 每個緩衝區（handle、channel）、L3 每個緩衝區範圍（OP，因為 OP 內含位址偏移）。請求從樹根走到葉子：未命中就建立，命中就重用，未被選中的排程不占 GPU 記憶體。當排程取代另一個排程時，delta migration 只建立差異，而且是惰性的，只沿目前請求的路徑進行：排程標籤與被選排程不符的節點才遷移。

The paper's Figures 9–10 example; each vertex shows its key and its tag. After S2 replaced S0, R5 migrated its whole path and R6 only L2–L3; the leaf of R3 keeps S0 until a request visits it.論文 Figure 9–10 的例子；每個節點標出它的鍵與標籤。S2 取代 S0 之後，R5 遷移了整條路徑，R6 只遷移 L2–L3；R3 的葉子保持 S0，直到有請求造訪。Drawn for this digest from §6.1–§6.2 and Figures 9–10 of the paper.

Implementation實作

Theseus is 20.9K lines of CUDA/C++ on MSCCL++ v0.7 and exports all NCCL v2.19 APIs, so applications switch by relinking. Send and receive go to the real NCCL, loaded as a fallback. The code is public.

Theseus 是建在 MSCCL++ v0.7 上的 20.9K 行 CUDA/C++，匯出全部 NCCL v2.19 API，應用程式只要重新連結即可。send 與 receive 交給真正的 NCCL，作為備援載入。程式碼已公開。

What was measured量了什麼

All runs use the same testbed, with one ToR per NIC in a two-tier Clos. The baselines are NCCL v2.19 with its tuner plugin and MSCCL++ v0.8; $K = 100$ .

所有實驗都在同一個測試平台，每張 NIC 接一台 ToR，組成兩層 Clos。比較基準是開啟 tuner plugin 的 NCCL v2.19 與 MSCCL++ v0.8； $K = 100$ 。

Stable conditions: better schedules, not adaptation穩定條件：靠的是更好的排程，不是調適

Nodes, GPUs each節點數、每節點 GPU	Start-up vs NCCL啟動對 NCCL	AllGather	Broadcast	ReduceScatter
(1, 2)	116× faster快 116 倍	1.51×	1.22×	1.61×
(2, 8)	400× faster快 400 倍	1.47×	1.13×	1.16×
(4, 8)	341× faster快 341 倍	1.23×	1.12×	1.33×

With a fixed context and 2 MB per GPU, Theseus starts in the order of 10 ms against NCCL's 3–5 s, because it builds resources on first use. The bandwidth gains come from the authors' custom schedules: flexible thread-block allocation inside a node, and rail-wise hierarchical algorithms across nodes. So the abstract's "up to 1.61× in stable environments" measures schedule quality, not adaptation.

在固定情境、每顆 GPU 2 MB 的條件下，Theseus 的啟動時間在 10 ms 的量級，NCCL 要 3–5 s，因為 Theseus 在第一次使用時才建立資源。頻寬增益來自作者的自訂排程：節點內彈性配置 thread block，跨節點用按 rail 分層的演算法。所以摘要中「穩定環境下最多 1.61 倍」量的是排程品質，不是調適。

Overhead: rare but heavy setups額外負擔：少見但沉重的建立

With 10 to 1,000 schedules, over 95% of times to execution (TTE, call to kernel launch) stayed below 10 µs; selection took at most 17 µs. Agreement lifts the 99th percentile to the order of 100 µs, and first-time setups exceed 1 ms. After 500 selected schedules, Theseus used under 500 MB of GPU memory; MSCCL++ needed 1.6–8 GB for tens to hundreds.

情境有 10 到 1,000 個排程時，超過 95% 的執行前時間（TTE，從呼叫到 kernel 啟動）低於 10 µs；選擇最多 17 µs。協議把第 99 百分位數拉到 100 µs 的量級，第一次建立則超過 1 ms。選過 500 個排程後，Theseus 用的 GPU 記憶體不到 500 MB；MSCCL++ 在數十到數百個排程時需要 1.6–8 GB。

Figure 12 and Table 3 of the Theseus paper: CDF of time to execution for 10, 100 and 1,000 schedules, GPU memory against selected schedules, and the TTE breakdown per executor event — Figure 12 and Table 3 of the paper: CDF of TTE; GPU memory against selected schedules (MS = MSCCL++); each event's latency in ms and its share of over 500K requests in a 32-GPU job. All measured.論文的 Figure 12 與 Table 3：TTE 的累積分布；GPU 記憶體隨已選排程數的變化（MS = MSCCL++）；32 GPU 任務超過 50 萬個請求中，每種事件的延遲（ms）與出現比例。全部是量測值。Screenshot of Ding et al., SIGCOMM 2026, Figure 12 and Table 3 (p. 231).

In Table 3, 98.4% of requests skip agreement and hit the cache. Weighting each latency by its share gives the 0.13 ms mean:

在 Table 3 中，98.4% 的請求不觸發協議且命中快取。把每種延遲乘上它的比例，就得到 0.13 ms 的平均：

Event (share × mean latency)事件（比例 × 平均延遲）	Calculation算式	Result結果
Normal setup一般建立	$0.03\% \times 206$	$\approx 0.062$ msabout half約一半
Delta migrationdelta migration	$0.43\% \times 8$	$\approx 0.034$ ms
Scratch reallocationscratch 重新配置	$0.14\% \times 15$	$\approx 0.021$ ms
Selection選擇	$100\% \times 0.01$	$0.010$ ms
Agreement協議	$1\% \times 0.19$	$\approx 0.002$ ms
Sum合計	rows above以上各列	$\approx 0.13$ ms

Rare setups dominate, and the agreement is almost free. Without delta migration, the paper reports more than 10× the GPU memory and 5.3× the mean TTE. NCCL's mean TTE is about 10 µs; the authors argue that 0.13 ms hides behind earlier GPU work.

主導平均的是少見的建立，協議幾乎不花成本。論文指出，若沒有 delta migration，GPU 記憶體會超過 10 倍，平均 TTE 是 5.3 倍。NCCL 的平均 TTE 約 10 µs；作者認為 0.13 ms 可以藏在 GPU 先前的工作之後。

A slow NIC一張變慢的 NIC

Llama3-8B trains with data parallelism, and one of the 16 NICs is capped by packet pacing. An agent publishes each NIC's health (peak throughput over capacity) every 20 ms, and the policy switches to one of 16 prepared rerouting schedules when health falls below 0.5. Rerouting moves a fraction $f$ of the slow NIC's bytes over NVLink to GPUs behind healthy NICs. With $M$ bytes per NIC, bandwidth $B$ and cap $p$ , equal finish times give the paper's rule:

Llama3-8B 以資料平行訓練，16 張 NIC 中的一張以 packet pacing 限速。代理程式每 20 ms 發布每張 NIC 的健康值（峰值吞吐量除以容量），健康值低於 0.5 時，策略切換到事先備好的 16 個繞道排程之一。繞道把慢 NIC 的一部分位元組 $f$ 經 NVLink 移到健康 NIC 後面的 GPU。設每張 NIC 送 $M$ 位元組、頻寬為 $B$ 、上限為 $p$ ，讓完成時間相等就得到論文的規則：

\begin{gathered} \frac{(1-f)\,M}{pB} = \frac{(1 + f/3)\,M}{B} \\ \Longrightarrow\quad f = \frac{3(1-p)}{3+p} \end{gathered}

The NIC phase then takes $4/(3+p)$ of the healthy time, never more than 4/3 (our derivation), against $1/p$ without rerouting.

於是 NIC 階段只需健康時的 $4/(3+p)$ 倍，最多 4/3（我們的推導），不繞道則是 $1/p$ 倍。

One test node: 8 GPUs, 4 NICs, one ToR per NIC. NIC 2 is capped at

p

. Its GPU keeps

1-f

of the bytes and relays

f

over NVLink,

f/3

to a GPU behind each healthy NIC.一個測試節點：8 顆 GPU、4 張 NIC，每張 NIC 一台 ToR。NIC 2 被限制在

p

。它的 GPU 留下

1-f

的位元組，把

f

經 NVLink 轉送，每張健康 NIC 後面的 GPU 各收

f/3

。Drawn for this digest from §8.3 and Appendix F.3 of the paper.

In the measurement (figure below), AllGather and AllReduce slowed more than 3× at a 10% cap without adaptation and stalled at 0%; with adaptation they slowed less than 1.5×, even at 0%. An iteration took 1.75 s normally and 3.76 s at 10% without adaptation; with it, 2.03 s at 0%, only 1.16×, because 1.2 s of it is computation. The paper does not say whether the unadapted run used NCCL or Theseus's normal schedule.

在量測中（見下圖），不調適時限速 10% 讓 AllGather 與 AllReduce 慢了 3 倍以上，0% 時停住；調適後即使在 0%，也慢不到 1.5 倍。一次迭代正常是 1.75 s，不調適、限速 10% 是 3.76 s；調適後 0% 時是 2.03 s，只是 1.16 倍，因為其中 1.2 s 是計算。論文沒有說明不調適的那組用的是 NCCL 還是 Theseus 的正常排程。

Figure 14 of the Theseus paper: CCT slowdown of AllGather and AllReduce with and without adaptation against the NIC rate limit, and Llama3-8B iteration time — Figure 14 of the paper: slowdown of AllGather (AG) and AllReduce (AR) without (w/o) and with (w/) adaptation as one NIC's cap falls (no bar: could not finish), and Llama3-8B iteration time (dashed: uncapped). Measured.論文的 Figure 14：單張 NIC 上限下降時，AllGather（AG）與 AllReduce（AR）不調適（w/o）與調適（w/）的變慢倍數（無長條：無法完成），以及 Llama3-8B 的迭代時間（虛線：未限速）。量測值。Screenshot of Ding et al., SIGCOMM 2026, Figure 14 (p. 232).

Panel 1 lets you try other caps and thresholds.

面板 1 讓你試試其他上限與門檻。

Panel 1面板 1

A NIC slows down: keep the schedule, or switch?一張 NIC 變慢：維持排程，還是切換？

NIC rate cap

p

NIC 速率上限

p

Health threshold

\theta

健康門檻

\theta

NICs per node

n

每節點 NIC 數

n

Agreement every

K

requests每

K

個請求協議一次

Policy runs

策略選出

Bytes moved off the slow NIC,

f

移出慢 NIC 的位元組

f

NIC phase, static

S

NIC 階段，靜態

S

NIC phase, with Theseus

NIC 階段，使用 Theseus

Agreement cost per request

每個請求的協議成本

Model: the paper's rule for 4 NICs, generalized by us to $n$ NICs: $f = (n-1)(1-p)/(n-1+p)$ . The node's NIC phase takes $1/p$ of the healthy time without rerouting and $n/(n-1+p)$ with it. Health $h \approx p$ assumes that the NIC is busy. Defaults are the paper's slow-NIC run: 4 NICs per node, $\theta = 0.5$ , a 10% cap, $K = 100$ , and 0.19 ms per agreement (Table 3). The orange marks are measured collective slowdowns from Figure 14. They differ from the model because a collective also has NVLink phases that the NIC does not touch, and the model ignores relay costs (our reading). Try $\theta$ below the cap, then $n = 8$ .

模型：論文針對 4 張 NIC 的規則，由我們推廣到 $n$ 張： $f = (n-1)(1-p)/(n-1+p)$ 。不繞道時，節點的 NIC 階段是健康時的 $1/p$ 倍，繞道時是 $n/(n-1+p)$ 倍。健康值 $h \approx p$ 假設該 NIC 正忙。預設值取自論文的慢 NIC 實驗：每節點 4 張 NIC、 $\theta = 0.5$ 、限速 10%、 $K = 100$ ，每次協議 0.19 ms（Table 3）。橘色標記是 Figure 14 量到的集體操作變慢倍數。它們與模型不同，因為集體操作還有 NIC 碰不到的 NVLink 階段，而模型忽略了轉送成本（這是我們的解讀）。試著把 $\theta$ 調到低於上限，再試 $n = 8$ 。

Other changing conditions其他會變的條件

Case and setup情況與設定	Measured result量測結果
MoE loads (Qwen3-30B-A3B): two all-to-all schedules, switch at predicted skew 1.4; baseline NCCL with PXNMoE 負載（Qwen3-30B-A3B）：兩個 all-to-all 排程，預測偏斜度 1.4 時切換；基準是開 PXN 的 NCCL	At skew 2.2: 1.05× at 2 MB per GPU, 1.73× at 64 MB; training 1.07× faster偏斜度 2.2：每顆 GPU 2 MB 快 1.05 倍、64 MB 快 1.73 倍；訓練快 1.07 倍
Anytime synthesis (heFFTe, 16 GPUs): two TE-CCL schedule sets, swapped in as they arrive隨時可用的合成（heFFTe、16 顆 GPU）：TE-CCL 的兩組排程一到就換上	Over 150 s saved against the default schedule and against waiting比預設排程、也比等待省下超過 150 s
Online learning: 9 thread-block settings explored for 18 iterations線上學習：探索 9 種 thread block 設定 18 次迭代	Then 1.11× faster than NCCL on a 224 MB-per-GPU AllGather之後在每顆 GPU 224 MB 的 AllGather 上比 NCCL 快 1.11 倍

The abstract's 2.46× (communication) and 1.84× (job time) are not tied to a run in section 8 (the introduction places 1.84× in fail-slow scenarios); we infer that both come from the 10% case of Figure 14.

摘要中的 2.46 倍（通訊）與 1.84 倍（任務時間）在第 8 節沒有對應到特定執行（引言把 1.84 倍歸在 fail-slow 的情境）；我們推斷兩者都來自 Figure 14 的 10% 情況。

Limits and open questions限制與未解的問題

The paper's own. Only collectives are optimized, faults need manual group operations, and the bootstrap network must never partition. Selection is consistent but not always stable: noisy signals need smoothing or hysteresis. A careful reader's. All runs use at most 32 A100 GPUs at 100 Gb/s each; the TCP AllReduce (0.19 ms here) is untested at 10,000 ranks. Apart from the synthesis run, no run compares adaptive Theseus with its best static schedule, so part of the other dynamic gains may be schedule quality. The slow-NIC baseline does nothing; evicting the node or rebalancing channels is not compared. Peak-throughput health may flag an idle NIC (our concern). A change can wait $K$ requests, so the MoE run predicts loads.

論文自己提到的。只最佳化集體操作，故障需要手動的群組操作，bootstrap 網路也不能分割。選擇保證一致，但不一定穩定：雜訊大的訊號需要平滑或遲滯。細心讀者會問的。所有實驗最多只用 32 顆 A100、每顆 100 Gb/s；走 TCP 的 AllReduce（這裡 0.19 ms）在 1 萬個 rank 時尚未測試。除了排程合成那組實驗，沒有任何執行把調適中的 Theseus 與它最佳的靜態排程相比，所以其他動態增益的一部分可能仍是排程品質。慢 NIC 實驗的基準什麼都不做，沒有與踢除節點或重新分配 channel 比較。以峰值吞吐量計算的健康值可能把閒置的 NIC 也標成不健康（這是我們的疑慮）。變更最多要等 $K$ 個請求，所以 MoE 實驗要預測負載。

How it connects與其他內容的關聯

Theseus replaces the cost-model choice of Section 4.5 with policies, and section 9's solvers become schedule producers. AutoCCL tunes NCCL's parameters inside the fixed menu, with a leader that broadcasts the choice; Theseus has no leader and changes the menu. PReCCL keeps one schedule and shifts bytes between its channels at collective boundaries, also with a deterministic rule on every rank. Like NCCLX, it connects lazily; its hierarchical AllGather comes from ResCCL. MegaScale evicts a failed node and restarts; Theseus routes around a slow NIC through NVLink and another rail, the PXN path of the Rail-only paper.

Theseus 用策略取代第 4.5 節的成本模型選擇，第 9 節的求解器則變成排程的生產者。AutoCCL 在固定選單內調整 NCCL 參數，由 leader 廣播選擇；Theseus 沒有 leader，改變的是選單本身。PReCCL 保留單一排程，在集體操作的邊界於 channel 之間移動位元組，同樣在每個 rank 上用確定性規則。它像 NCCLX 一樣惰性建立連線；它的階層式 AllGather 來自 ResCCL。MegaScale 踢除故障節點並重啟；Theseus 經 NVLink 與另一條 rail 繞過慢 NIC，也就是 Rail-only 論文中的 PXN 路徑。

Key numbers關鍵數字

Number數字	What was measured (32 A100 GPUs)量了什麼（32 顆 A100）
≤ 2.4×	Static all-to-all time at skew 2.2 vs uniform load偏斜度 2.2 時靜態 all-to-all 的時間，相對於均勻負載
13 h / 49 h	TE-CCL time to beat NCCL / to its best scheduleTE-CCL 勝過 NCCL／找到最佳排程的時間
$O(10)$ ms / 3–5 s	Communicator start-up, Theseus / NCCLcommunicator 啟動時間，Theseus／NCCL
0.13 ms / ≈ 10 µs	Mean time to execution, Theseus / NCCL平均執行前時間，Theseus／NCCL
> 3× / < 1.5×	Collective slowdown with one NIC capped, without / with adaptation一張 NIC 被限速時集體操作的變慢倍數，不調適／調適
1.75 / 3.76 / 2.03 s	Llama3-8B iteration: normal / 10% cap, no adaptation / 0% cap, adaptedLlama3-8B 迭代：正常／限速 10% 不調適／限速 0% 調適

Terms used on this page本頁用到的術語

Selection context: Theseus's inputs: attributes, schedules and policies, kept in replicated logs.; Theseus 的輸入：屬性、排程與策略，存在複製的日誌中。
Endogenous / exogenous attribute: A library metric updated by a hook / an outside signal with versions.; 由 hook 更新的函式庫指標／帶版本的外部訊號。
Agreement: An AllReduce every $K$ requests that intersects sets and keeps minimum versions.; 每 $K$ 個請求一次的 AllReduce，對集合取交集、保留最小版本。
Delta migration: Building only what differs from the replaced schedule, one path at a time.; 只建立與被取代排程不同的部分，一次一條路徑。
Fail-slow: A device that still works, but below its nominal speed.; 仍能運作、但低於標稱速度的裝置。