Topology research: wire smarter, or re-wire with optics拓樸研究：更聰明地接線，或用光學重新接線
ZCube and BST, which wire the network more cleverly, and three optical designs that re-wire it for the collective that is running.ZCube 與 BST 以更聰明的方式接線，另有三個依正在執行的集體操作重新接線的光學設計。
This page assumes Chapters 1–6. Every term it uses is defined there; the glossary on the contents page links to each definition. The papers here are recent and most of their results come from small testbeds or simulation, so read each number with its label.
本頁假設你已讀過第 1–6 章。這裡用到的每個名詞都在那些章節中定義過；目錄頁的詞彙表連到每個定義。這些論文都很新，多數結果來自小型測試平台或模擬，所以請連同標籤一起讀每個數字。
Terms from later chapters used here (2)本頁用到的後面章節名詞（2 個）
- Packet spraying, reordering封包噴灑、亂序 Ch. 8 Sending consecutive packets of one flow over different paths; they may arrive out of order, so the NIC must accept that.把同一條流的連續封包送到不同路徑；封包可能亂序抵達，NIC 必須能接受。
- Simulation, packet-level simulation模擬、封包層級模擬 Ch. 12 Running a model of the workload and network in software; a packet-level simulator follows every packet through every queue.以軟體執行工作負載與網路的模型；封包層級模擬器追蹤每個封包經過每個佇列。
This page sizes a Clos for the worst case: every tier adds switch ports, two transceivers per link, and hops. Training traffic is not the worst case. It follows rails, parallelism phases and collective steps, and MoE traffic is skewed and local. Five recent papers turn that structure into a cheaper or faster network. They form two families. Two keep the network electrical and static but wire it more cleverly. Three use optical circuit switches and change the wiring during training, each at a different moment.
本頁以最壞情況來規劃 Clos：每多一層就多出交換器埠、每條鏈路兩個光模組，以及更多跳數。訓練流量並不是最壞情況。它依循 rail、平行化階段與集體操作的步驟，而 MoE 流量則偏斜且具區域性。五篇近期論文把這種結構變成更便宜或更快的網路，分成兩個家族。兩篇讓網路保持電子式、靜態，但接得更聰明。三篇使用光電路交換器，在訓練進行中改變接線，各自選在不同的時刻。
Static and electrical: wire smarter. ZCube came out of an automated search over topology parameters. Its cost-effective design gives each GPU two half-speed NIC ports, one to a group switch and one to an index switch, and meshes the two switch layers fully, so any two GPUs are at most two switches apart. BST builds from a Steiner system: each spine joins only a few ToRs, yet every pair of ToRs shares exactly one spine. Both remove a tier of Section 6.1's arithmetic, and both need help from routing or placement: ZCube relies on PXN (its large-scale speedups also use packet spraying), BST on threshold-gated detours, two virtual lanes and a rank table. Their hardware tests are small, 16 and 24 accelerators, and their large-scale cost claims come from cost models and simulation.
靜態、電子：接得更聰明。ZCube 來自對拓樸參數的自動搜尋。它最具成本效益的設計讓每顆 GPU 有兩個半速 NIC 埠，一個接群組交換器、一個接索引交換器，並讓兩層交換器完全互連，所以任兩顆 GPU 最多相隔兩台交換器。BST 從 Steiner 系統出發：每台 spine 只接少數幾台 ToR，但每一對 ToR 恰好共用一台 spine。兩者都省掉第 6.1 節算術中的一層，也都需要路由或擺放的配合：ZCube 依賴 PXN（大規模的加速結果也用了封包噴灑），BST 依賴有門檻的繞路、兩條虛擬通道與 rank 表。它們的硬體測試規模很小，分別是 16 與 24 顆加速器，大規模的成本主張來自成本模型與模擬。
Optical: re-wire at the right moment. An optical circuit switch (OCS) has no packet logic and no per-port transceiver on the switch side, but it connects each port to only one other port at a time, and every change costs time. The TPU v4 torus of Section 6.9 changes once per job. MixNet keeps a small electrical fat tree and adds a regional OCS per expert group, re-wired inside every MoE layer while the experts compute. Opus keeps rail wiring but makes each rail one OCS, re-wired between parallelism phases in idle windows. Harvest goes inside one collective in a photonic scale-up domain and chooses before which steps to re-wire, trading the reconfiguration delay against congestion.
光學：在對的時刻重新接線。光電路交換器（OCS）沒有封包邏輯，交換器端也不需要每埠光模組，但它同一時間只能把每個埠接到另一個埠，而且每次改變都要花時間。第 6.9 節的 TPU v4 環面每個任務改變一次。MixNet 保留一個小型電子 fat tree，並為每個專家群組加一台區域 OCS，在每個 MoE 層內趁專家計算時重新接線。Opus 保留 rail 接線，但讓每條 rail 變成一台 OCS，在平行化階段之間的閒置空檔重新接線。Harvest 深入單一次集體操作、在光子 scale-up 域中，決定在哪幾步之前重新接線，以重新配置延遲換取壅塞。
| Paper · venue · problem論文 · 會議 · 問題 | Mechanism機制 | Measured result (setup)實測結果（設定） |
|---|---|---|
| ZCube SIGCOMM 2025 A Clos costs tiers of switches and opticsClos 要花好幾層交換器與光模組 | Searched design: two half-speed NIC ports per GPU, group and index switches, two switch hops搜尋出的設計：每顆 GPU 兩個半速 NIC 埠、群組與索引交換器、兩個交換器跳數 | 16 H800 GPUs: same NCCL all-reduce and all-to-all bus bandwidth as a rail-optimized fat tree, with 48 × 200G links instead of 32 × 400G. One month in production: 35% of rail-aligned traffic crossed spines.16 顆 H800：NCCL all-reduce 與 all-to-all 的 bus bandwidth 與 rail-optimized fat tree 相同，但用 48 條 200G 鏈路取代 32 條 400G。生產環境一個月：35% 的同 rail 流量經過 spine。 |
| BST SIGCOMM 2026 Reach large scale with two tiers只用兩層達到大規模 | Steiner design, one shared spine per ToR pair; gated detours, two virtual lanes, rank tableSteiner 設計，每對 ToR 共用一台 spine；有門檻的繞路、兩條虛擬通道、rank 表 | 24 Ascend 910B NPUs, 4 ToRs (6 spines of radix 2 against 3 of radix 4), 1 GB HCCL: ring all-reduce 69,648 µs against 76,914 µs on Clos; pairwise all-to-all 37,676 against 36,793 µs.24 顆 Ascend 910B NPU、4 台 ToR（6 台基數 2 的 spine 對 3 台基數 4），1 GB HCCL：環狀 all-reduce 69,648 µs，Clos 為 76,914 µs；成對 all-to-all 37,676 對 36,793 µs。 |
| MixNet SIGCOMM 2025 MoE all-to-all is large, sparse and localMoE 的 all-to-all 量大、稀疏且具區域性 | Regional OCS per expert group beside a small fat tree, re-wired twice per MoE layer每個專家群組一台區域 OCS，旁邊配小型 fat tree，每個 MoE 層重新接線兩次 | 128 H800 in production: all-to-all is 33–55% of Mixtral 8×7B iteration time. 32 A100 with a Polatis OCS: about 9.8 s against 11.5 s per iteration on 4 × 100G electrical, excluding about 5.7 s of NIC activation per switch.生產環境 128 顆 H800：all-to-all 占 Mixtral 8×7B 迭代時間的 33–55%。32 顆 A100 加 Polatis OCS：每次迭代約 9.8 s，4 × 100G 電子網路為 11.5 s，不含每次切換約 5.7 s 的 NIC 啟動。 |
| Opus SIGCOMM 2026 Rail switches and their optics cost powerrail 交換器與光模組耗電 | One OCS per rail; NIC ports time-shared between parallelism phases, re-wired in idle windows每條 rail 一台 OCS；NIC 埠在平行化階段之間分時共用，在閒置空檔重新接線 | A100 runs on Perlmutter: over 75% of windows between phases exceed 1 ms. 4 servers with 2 L40 each, Polatis OCS, 2 rails: 4 re-wirings per step at 200 Gb/s per GPU; NIC link-up about 6 s (about 3 s without auto-negotiation) against about 25 ms for the OCS.Perlmutter 上的 A100 執行：超過 75% 的階段間空檔大於 1 ms。4 台伺服器、每台 2 顆 L40、Polatis OCS、2 條 rail：每步重新接線 4 次，每顆 GPU 200 Gb/s；NIC 建立連線約 6 s（關閉自動協商約 3 s），OCS 約 25 ms。 |
| Harvest SIGCOMM 2026 When to re-wire during one collective在一次集體操作中何時重新接線 | α–β cost plus propagation and congestion; dynamic program over the steps; closed form for recursive doublingα–β 成本加上傳播與壅塞；對步驟做動態規劃；遞迴倍增有封閉形式 | No photonic switch. 8 GPUs on a BlueField-3 ring at 100G: per-hop α 30.32 µs and β 85.11 Gb/s; synthesis within 20 µs up to 64 GPUs; synchronization about 4 µs for 8 GPUs.沒有光子交換器。8 顆 GPU 接成 100G 的 BlueField-3 環：每跳 α 30.32 µs、β 85.11 Gb/s；64 顆 GPU 以內排程合成在 20 µs 內；8 顆 GPU 同步約 4 µs。 |
Where they agree. All five treat the traffic as known collectives, not arbitrary flows. All five count cost as ports plus transceivers: savings come from removing a tier, or from replacing packet ports with circuit ports. The optical papers agree that a change of wiring must be hidden or paid back: behind expert compute (MixNet), inside idle windows and started early (Opus), or only where the congestion saved exceeds the delay (Harvest). And all five rest their large-scale claims on simulation or cost models; the hardware tests use 8 to 32 accelerators.
它們的共識。五篇都把流量當成已知的集體操作，而不是任意的流。五篇都以埠數加光模組計算成本：節省來自去掉一層，或用電路埠取代封包埠。三篇光學論文都同意，改變接線必須藏起來或賺回來：藏在專家計算後面（MixNet）、放在閒置空檔裡並提早開始（Opus），或只在省下的壅塞超過延遲時才做（Harvest）。五篇的大規模主張都建立在模擬或成本模型上；硬體測試只有 8 到 32 顆加速器。
What real hardware showed. The two papers that switched real optics found the same thing. The optical switch was not the slow part: the Polatis OCS changed in about 25 ms in Opus and 41–47 ms in MixNet. The NICs and transceivers were: they needed about 5.7 s (MixNet) and about 6 s, or 3 s without auto-negotiation (Opus), to bring the link back up. Neither paper puts that time into its results. MixNet's prototype iteration times exclude it, and Opus's timing results come from emulation and simulation that inject only the switch delay. Harvest's regime, nanoseconds to microseconds, needs optics that re-lock in that time, which are not yet commercial.
真實硬體顯示了什麼。兩篇真正切換過光學元件的論文發現同一件事。慢的不是光交換器：Polatis OCS 在 Opus 中約 25 ms、在 MixNet 中 41–47 ms 完成切換。慢的是 NIC 與光模組：它們需要約 5.7 s（MixNet）與約 6 s、關閉自動協商時 3 s（Opus），才能把鏈路重新建立起來。兩篇論文都沒有把這段時間放進結果。MixNet 原型的迭代時間不含它，Opus 的時間結果則來自只注入交換器延遲的模擬硬體與模擬。Harvest 所需的奈秒到微秒區間，需要能在這麼短時間內重新鎖定的光學元件，而這種元件尚未商用。
Where they differ. Rings divide them. An OCS port connects to one peer, so Opus's per-phase circuits are rings; Harvest proves that rings never gain from re-wiring, while recursive doubling and Bruck do when switching is fast. MoE all-to-all divides them too: MixNet builds its design around it, Opus spreads it over different rings per rail, and ZCube assumes rail-aligned traffic, which PXN produces even for all-to-all. Read ZCube and BST first, as the next step from rail-optimized and rail-only wiring, then MixNet and Opus, which re-wire in milliseconds, and Harvest last, because it builds on the α–β model of Section 4.3.
它們的分歧。環把它們分開。一個 OCS 埠只能接一個夥伴，所以 Opus 每個階段的電路都是環；Harvest 證明環永遠不會因重新接線而受益，而遞迴倍增與 Bruck 在切換夠快時會受益。MoE 的 all-to-all 也讓它們分歧：MixNet 以它為設計核心，Opus 把它分散到每條 rail 不同的環上，ZCube 則假設流量同 rail，而 PXN 連 all-to-all 也能做到這點。先讀 ZCube 與 BST，作為 rail-optimized 與 rail-only 接線的下一步；再讀以毫秒級重新接線的 MixNet 與 Opus，最後讀 Harvest，因為它建立在第 4.3 節的 α–β 模型之上。