Research digest 1 of 3 · Chapter 8: Routing and load balancing

研究導讀 1 / 3 · 第 8 章：路由與負載平衡

SGLB: Scalable and Robust Global Load Balancing in Commodity AI ClustersSGLB：商用 AI 叢集中可擴展且穩健的全域負載平衡

SIGCOMM 2025Chenchen Qi, Wenfei Wu, Yongcan Wang et al. (ByteDance, Peking University, Shanghai Jiao Tong University, Broadcom) · paper論文

After reading, you can explain why a switch that sees only its own queues still picks congested paths, what remote state SGLB's switches exchange, and how it reroutes around a failed link in microseconds.讀完後，你能說明為何只看自己佇列的交換器仍會選到壅塞的路徑、SGLB 的交換器彼此交換哪些遠端狀態，以及它如何在微秒內繞過故障鏈路。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 8 ↗第 8 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

ByteDance trains on commodity Ethernet switches that run BGP and ECMP. ECMP ignores load, so a few large flows collide, and BGP needs more than a second to route around a failed link. New Broadcom switch chips contain a GLB engine (global load balancing) that trades queue and link state with neighbour switches and sends each packet on the best-scored path. SGLB makes it work in a three-tier AI fabric: it scores only the first three hops, adds a fast link-state protocol beside BGP, and switches off links to keep paths equal. On 14 switches and 32 H800 GPUs with background traffic, All-to-All ran 60% faster than with ECMP and 15% faster than with local adaptive routing (DLB). Rerouting took 1–45 µs instead of more than 1 s, and LLaMA-3 training took 12.3% less time.

字節跳動用執行 BGP 與 ECMP 的商用乙太網路交換器做訓練。ECMP 不看負載，所以少數幾條大流會碰撞；BGP 繞過故障鏈路需要超過一秒。新的 Broadcom 交換晶片內建 GLB 引擎（global load balancing，全域負載平衡），它與鄰近交換器交換佇列與鏈路狀態，並把每個封包送上評分最好的路徑。SGLB 讓它能在三層 AI 網路中運作：它只為前三跳評分，在 BGP 旁加上一個快速的鏈路狀態協定，並關掉部分鏈路讓路徑保持相等。在 14 台交換器、32 顆 H800 GPU、加上背景流量的環境下，All-to-All 比 ECMP 快 60%，比本地自適應路由（DLB）快 15%。重新選路只花 1–45 µs，而不是超過 1 秒；LLaMA-3 訓練時間少了 12.3%。

1. Where this paper sits1. 這篇論文的位置

Section 8.2 computes why hashing fails for training (Panel 2 there). A few large flows often share an uplink, and the busiest one sets the collective's speed. Its answers: more entropy (Meta, §3), no choice (dual-plane, §4), and per-packet spraying with reorder-tolerant NICs (§5, Panel 5).

第 8.2 節算出雜湊為何對訓練無效（那裡的面板 2）。少數幾條大流常常共用一條上行鏈路，而最忙的那條決定集體通訊的速度。它的答案是：增加熵（Meta，§3）、拿掉選擇（dual-plane，§4），以及搭配能容忍亂序的 NIC 做逐封包噴灑（§5、面板 5）。

Missing is a switch that looks at load: §5 mentions adaptive routing on InfiniBand without numbers. SGLB gives testbed numbers for Ethernet. A local decision uses only the sending switch's own egress queues, as Broadcom's DLB (dynamic load balancing) does. A global decision also uses the state of switches further along the path.

還缺的是一台會看負載的交換器：§5 提到 InfiniBand 上的自適應路由，但沒有數字。SGLB 為乙太網路提供了測試平台的數字。本地（local）決策只用發送端交換器自己的出口佇列，Broadcom 的 DLB（dynamic load balancing，動態負載平衡）就是如此。全域（global）決策還會用到路徑上更遠處交換器的狀態。

2. The problem, measured2. 問題，用量測說話

The paper has no measurement study of its own; it cites its motivation. One production AI cluster sees 5K–60K link flaps per day (HPN guided reading). BGP needs 5–10 s to converge in AI clusters (an IETF draft), and about 1 s in the best industry report (AWS).

論文沒有自己的量測研究；它的動機來自引用。一個生產 AI 叢集每天有 5K–60K 次鏈路抖動（HPN 導讀）。AI 叢集中 BGP 收斂要 5–10 秒（一份 IETF 草案），業界最好的報告（AWS）約 1 秒。

Figure A shows why a local view is not enough. Switch S00's own queues A, B and C look equal, so DLB may route through S11, where queue Y is full.

圖 A 說明為何只看本地不夠。交換器 S00 自己的佇列 A、B、C 看起來一樣，所以 DLB 可能經由 S11 送出，而那裡的佇列 Y 已滿。

Figure A. Figure 1 of the paper redrawn, with queue levels set to show the case. DLB compares only A, B and C, while the GLB engine adds the X, Y and Z that GLB packets report (blue arrows) and so avoids S11.圖 A。重畫論文 Figure 1，佇列高度設定成呈現這個情況。DLB 只比較 A、B、C，而 GLB 引擎加上 GLB 封包回報的 X、Y、Z（藍色箭頭），因此避開 S11。Drawn for this digest from Qi et al., SIGCOMM 2025, §2.2 and Figure 1.

Failures show the same blindness. If link S11–S02 fails, S00 still sees a healthy uplink to S11 and keeps sending until BGP withdraws the route.

故障也暴露同樣的盲點。若鏈路 S11–S02 故障，S00 看到的仍是一條通往 S11 的健康上行鏈路，會一直往那裡送，直到 BGP 撤回路由。

3. The design3. 設計

In one SGLB switch (Figure 3 of the paper), BGP finds which subnets are reachable and SyncMesh tracks nearby links. The orchestrator merges both into a path group per destination ToR. The GLB engine keeps a path quality profile for each group: one score per path.

在一台 SGLB 交換器中（論文 Figure 3），BGP 找出哪些子網路可達，SyncMesh 追蹤附近的鏈路。orchestrator（協調器）把兩者合成每台目的 ToR 的路徑群組（path group）。GLB 引擎為每個群組保存一份路徑品質 profile：每條路徑一個分數。

Figure 3 of the SGLB paper: BGP and SyncMesh above an orchestrator in the control plane; below a dashed line, a destination to path group table, a group path quality table and the GLB engine — Figure 3 of the paper, a design diagram of one switch: BGP and SyncMesh feed the orchestrator, which writes path groups; the GLB engine keeps the quality column current.論文 Figure 3，一台交換器的設計示意圖：BGP 與 SyncMesh 把資訊交給 orchestrator，由它寫入路徑群組；GLB 引擎持續更新品質欄。Qi et al., SIGCOMM 2025, Figure 3 (ACM PDF, p. 5).

3.1 The global state that the GLB engine carries3.1 GLB 引擎攜帶的全域狀態

The GLB engine reads the queue length and link utilization of each port. User programs on its embedded processor (EDK programs) send these states to chosen neighbours in small GLB packets. SGLB scores a path with five factors (Equation 1 of the paper):

GLB 引擎讀取每個埠的佇列長度與鏈路使用率。引擎內嵌處理器上的使用者程式（EDK 程式）把這些狀態裝進小的 GLB 封包，送給指定的鄰居。SGLB 用五個因子為一條路徑評分（論文 Equation 1）：

\begin{gathered} V = \alpha_1 Q^{(L)} + \alpha_2 L^{(L)} \\ {} + \alpha_3 Q^{(R)} + \alpha_4 L^{(R)} + \alpha_5 B^{(R)} \end{gathered}

$Q^{(L)}$ and $L^{(L)}$ are the queue and utilization of the local egress port. $Q^{(R)}$ and $L^{(R)}$ are the same at the next hop on that path, and $B^{(R)}$ is the next hop's mean queue over all its ports. The weights are not published. Local state refreshes every 1 µs and remote state every 15 µs. So "global" means one hop of remote state.

$Q^{(L)}$ 與 $L^{(L)}$ 是本地出口埠的佇列與使用率。 $Q^{(R)}$ 與 $L^{(R)}$ 是該路徑下一跳的同樣數值， $B^{(R)}$ 是下一跳所有埠的平均佇列。權重沒有公開。本地狀態每 1 µs 更新，遠端狀態每 15 µs 更新。所以「全域」指的是一跳的遠端狀態。

If all packets took the best path, it would soon be the worst. So SGLB rounds scores to a few levels, keeps the top-K paths, and picks at random among equal ones. The panel compares the three decisions on a small model.

如果所有封包都走最好的路徑，它很快就會變成最差的。所以 SGLB 把分數捨入成少數幾個等級，保留前 K 條路徑，並在同分路徑中隨機挑選。下面的面板在一個小模型上比較三種決策。

Panel 1面板 1

Hash, local or global: four paths, a few large flows雜湊、本地或全域：四條路徑、幾條大流

Path choice選路方式

Large flows (400 Gb/s each)大流數（每條 400 Gb/s）

Remote background S2→L1遠端背景 S2→L1

Local background L0→S3本地背景 L0→S3

Link S1→L1鏈路 S1→L1

ECMP drawECMP 抽樣

Most-loaded link

負載最重的鏈路

Delivered / offered

送達 / 送出

Collective finishes

集體通訊完成

Speed-up vs ECMP (300 hashes)

相對 ECMP 的加速（300 次雜湊）

Reroute after the failure

故障後重新選路

A teaching model, not the paper's testbed or simulator. Four 800 Gb/s paths lead from leaf L0 through spines S1–S4 to leaf L1; each flow sends 256 MB at up to 400 Gb/s. ECMP puts each whole flow on a hashed path. Local (DLB) sprays packets so that L0's own uplinks are equally loaded. Global (SGLB) sprays so that the busier hop of every path is equally loaded. When a link is overloaded, congestion control slows every flow that uses it, so a sprayed collective runs at its worst path's rate. The defaults copy the testbed's background flows: 400 Gb/s on one remote link and 500 Gb/s on one local uplink. Try: set the remote background to 0, and DLB matches SGLB; set the local one to 0 instead, and DLB falls behind. Then fail S1→L1. SGLB drops the path in about 45 µs; ECMP and DLB keep sending into the dead link until BGP converges, which took more than 1 s on the testbed.

這是教學模型，不是論文的測試平台或模擬器。四條 800 Gb/s 路徑從 leaf L0 經 spine S1–S4 到 leaf L1；每條流以最多 400 Gb/s 送 256 MB。ECMP 把整條流放在雜湊選出的路徑上。本地（DLB）噴灑封包，讓 L0 自己的上行鏈路負載相等。全域（SGLB）噴灑封包，讓每條路徑上較忙的那一跳負載相等。鏈路超載時，壅塞控制會讓所有用到它的流變慢，所以噴灑的集體通訊以最差路徑的速率前進。預設值照抄測試平台的背景流量：一條遠端鏈路 400 Gb/s、一條本地上行 500 Gb/s。試試：把遠端背景設為 0，DLB 就和 SGLB 一樣；改把本地背景設為 0，DLB 就落後。接著讓 S1→L1 故障。SGLB 約 45 µs 就拿掉這條路徑；ECMP 與 DLB 會一直往斷掉的鏈路送，直到 BGP 收斂，這在測試平台上花了超過 1 秒。

3.2 Fitting global state into small tables3.2 把全域狀態塞進小表格

Scores for every path to every destination do not fit. The paper's reference cluster has 64-port switches, 8 rails of 32 ToRs and 32 leaves, and 8 planes of 32 spines:

每個目的地的每條路徑都評分是放不下的。論文的參考叢集用 64 埠交換器，有 8 個各含 32 台 ToR 與 32 台 leaf 的 rail，以及 8 個各含 32 台 spine 的 plane：

Question問題	Calculation算式	Result結果
Paths between ToRs of two rails兩個 rail 的 ToR 之間的路徑數	$32 \times 32 \times 4$	$4{,}096$ paths條路徑
Profiles one ToR would need一台 ToR 需要的 profile 數	$8 \times 32 - 1$	$255$ a chip holds 64 profiles of 32 values晶片只放得下 64 個 profile，每個 32 個值
ToR: GLB inside the rail, plus one 1-hop profileToR：rail 內用 GLB，另加一個 1 跳 profile	$31 + 1$	$32$ profiles × 32 leaves個 profile × 32 台 leaf
Leaf: one per same-plane leaf of another railLeaf：其他 rail 的每台同 plane leaf 一個	$7 \times 4$	$28$ profiles × 32 spines個 profile × 32 台 spine
Spine: 8 same-index ToRs merged, 4 segments eachSpine：8 台同編號 ToR 合併，各 4 條路段	$8 \times 4$	$32$ values; 32 profiles個值；32 個 profile

Every switch scores only a path segment, the first three hops. The ToR applies GLB selectively, the leaf shares profiles, and the spine merges them (Figure B). Each GLB engine then holds at most 1,024 segments. The price: a ToR scores traffic to other rails like DLB ( $\alpha_3 = \alpha_4 = \alpha_5 = 0$ ), and leaves and spines do the global part. SyncMesh areas follow the segments, so link state reaches only the switches that use it.

每台交換器只為路段（path segment）評分，也就是前三跳。ToR 選擇性地使用 GLB，leaf 共用 profile，spine 合併 profile（圖 B）。於是每個 GLB 引擎最多存 1,024 條路段。代價是：ToR 對送往其他 rail 的流量像 DLB 一樣評分（ $\alpha_3 = \alpha_4 = \alpha_5 = 0$ ），全域的部分交給 leaf 與 spine。SyncMesh 的區域（area）沿著路段劃分，所以鏈路狀態只送到用得到它的交換器。

Figure B. One GLB table (64 profile rows, to scale, × 32 values) per tier. Unreduced, a ToR would need 255 profiles of up to 4,096 values; reduced, every tier fits (§4.3 of the paper).圖 B。每一層一張 GLB 表（64 列 profile，依比例，× 32 個值）。不縮減時，一台 ToR 需要 255 個 profile、每個最多 4,096 個值；縮減後每一層都放得下（論文 §4.3）。Drawn for this digest from Qi et al., SIGCOMM 2025, §4.3 and Figures 4–6.

3.3 The failure path3.3 故障處理的路徑

BGP is distance-vector: it advertises a path per destination subnet. One failed link therefore triggers new advertisements for every subnet routed over it. If the link between S-0-0 and L-7-0 fails, each rail-7 ToR waits for $32 \times 7 = 224$ updates. SyncMesh is link-state: one message per failed link reaches each switch in the area.

BGP 是距離向量（distance-vector）協定：它為每個目的子網路通告一條路徑。因此一條故障鏈路會讓經過它的每個子網路重新通告。若 S-0-0 與 L-7-0 之間的鏈路故障，rail 7 的每台 ToR 要等 $32 \times 7 = 224$ 個更新。SyncMesh 是鏈路狀態（link-state）協定：每條故障鏈路只有一則訊息送到區域內的每台交換器。

A shortcut is faster still. The EDK program that sees the link go down messages the neighbour's GLB engine directly, and that switch's orchestrator deletes the affected paths from a precomputed link-to-path map. This takes about 1 µs when only the link's ends react and about 45 µs when the event travels one hop (Figure C).

一條捷徑還要更快。看到鏈路斷線的 EDK 程式直接通知鄰居的 GLB 引擎，該交換器的 orchestrator 再依事先算好的鏈路到路徑對照表刪除受影響的路徑。只有鏈路兩端要反應時約需 1 µs，事件要往外傳一跳時約需 45 µs（圖 C）。

Figure C. Time from link shutdown until no more packet loss, log scale. The first three bars are measured (Table 1 of the paper); 5–10 s is cited. "One hop" is a leaf–ToR failure or the two-tier case.圖 C。從關閉鏈路到不再丟包的時間，對數刻度。前三條是實測（論文 Table 1）；5–10 秒是引用值。「傳一跳」是 leaf–ToR 故障或兩層網路的情況。Drawn for this digest from Qi et al., SIGCOMM 2025, Table 1 and §5.1.

BGP stays for its proven correctness. When the two protocols disagree, SGLB trusts "unreachable":

BGP 因為正確性久經驗證而保留。兩個協定不一致時，SGLB 相信「不可達」：

SyncMesh	SyncMesh	BGP	BGP	Path	路徑	Reason	理由
up	正常	reachable	可達	enabled	啟用	both agree	兩者一致
down	斷線	still reachable	仍可達	disabled at once	立即停用	trusting BGP would black-hole packets	相信 BGP 會讓封包進入黑洞
up	正常	not yet reachable	尚未可達	disabled	停用	costs only an idle healthy path	代價只是一條閒置的健康路徑

Flaps get the same bias. A link-down is reported at once; a link-up only when the link, its BGP session and the switch's boot are all complete.

抖動也受到同樣的偏向。鏈路斷線立即回報；鏈路恢復則要等鏈路、其上的 BGP 會話與交換器開機都完成才回報。

3.4 When a bundle loses one link3.4 當一束鏈路少了一條

A LAG (link aggregation group) bundles several links into one. When one LAG link fails, destinations stay reachable but some paths lose bandwidth, and unlike weighted ECMP, the GLB engine cannot split by weight. In Figure D, link Z fails, so paths C and D share 800 Gb/s. Above 2,400 Gb/s of load they drop packets, and congestion control slows the whole flows, also on A and B. About 1,600 Gb/s gets through instead of 2,400: throughput suppression.

LAG（link aggregation group，鏈路聚合群組）把數條鏈路綁成一條。LAG 中一條鏈路故障時，目的地仍可達，但有些路徑的頻寬變少了；而且與加權 ECMP 不同，GLB 引擎不能依權重分流。在圖 D 中，鏈路 Z 故障，所以路徑 C 與 D 共用 800 Gb/s。負載超過 2,400 Gb/s 時它們開始丟包，壅塞控制讓整條流變慢，連 A 與 B 上的部分也一樣。只有約 1,600 Gb/s 通過，而不是 2,400：這就是吞吐壓抑（throughput suppression）。

SGLB makes the paths equal again by switching links off, segment by segment from downstream. With $x$ downstream and $y$ upstream links, after one downstream failure it keeps the largest $y'$ with

SGLB 用關掉鏈路的方式讓路徑重新相等，從下游開始逐段處理。設下游有 $x$ 條、上游有 $y$ 條鏈路，下游一條故障後，它保留滿足下式的最大 $y'$ ：

\frac{y'}{x-1} \le \frac{y}{x}, \qquad 1 \le y' \le y

and disables the other $y - y'$ . In Figure D, $x = y = 2$ gives $y' = 1$ , so SGLB disables link C.

並停用其餘 $y - y'$ 條。在圖 D 中， $x = y = 2$ 得到 $y' = 1$ ，所以 SGLB 停用鏈路 C。

Figure D. Figure 8 of the paper redrawn with four 800 Gb/s paths and link Z failed. On the left, congestion control holds every flow to the rate of the C–D side, while on the right SGLB disables C. The loads come from the paper's analysis.圖 D。重畫論文 Figure 8，四條 800 Gb/s 路徑中的鏈路 Z 故障。左邊的壅塞控制把每條流壓到 C–D 那一側的速率，右邊的 SGLB 則停用 C。負載取自論文的分析。Drawn for this digest from Qi et al., SIGCOMM 2025, §4.4 and Figures 7–8.

4. Implementation4. 實作

SGLB runs on commodity switches with a GLB-capable Broadcom chip; the paper cites Tomahawk 5's "cognitive routing" but does not name the testbed chip, and the EDK programs and table sizes are vendor-specific. SyncMesh and the orchestrator are about 15K lines of Go and C++. Leaves of a rail share one BGP AS number, as do spines of a plane, so one AS path names a path group.

SGLB 執行在搭載支援 GLB 的 Broadcom 晶片的商用交換器上；論文引用 Tomahawk 5 的「cognitive routing」，但沒有說出測試平台的晶片，而 EDK 程式與表格大小都依廠商而定。SyncMesh 與 orchestrator 約 15K 行 Go 與 C++。同一 rail 的 leaf 共用一個 BGP AS 編號，同一 plane 的 spine 也是，所以一條 AS 路徑就代表一個路徑群組。

5. What was measured5. 量到了什麼

The testbed is a three-tier RoCEv2 cluster of 14 switches: 2 rails of 2 ToRs and 4 leaves, plus 2 spines, with 800 Gb/s switch links. Hosts have 8 H800 GPUs and 8 NICs. The baselines are ECMP and DLB on the same switches, and NCCL-tests reports bus bandwidth. The background load is 400 Gb/s on one leaf-to-ToR link and 500 Gb/s on one ToR-to-leaf link.

測試平台是 14 台交換器的三層 RoCEv2 叢集：2 個 rail，各有 2 台 ToR 與 4 台 leaf，另有 2 台 spine，交換器之間是 800 Gb/s 鏈路。主機有 8 顆 H800 GPU 與 8 張 NIC。對照組是同一批交換器上的 ECMP 與 DLB，匯流排頻寬由 NCCL-tests 報告。背景負載是一條 leaf 往 ToR 鏈路 400 Gb/s、一條 ToR 往 leaf 鏈路 500 Gb/s。

Without background, all three reach 56.7–58.8 Gb/s of All-to-All bus bandwidth, because empty links leave nothing to avoid. With it, SGLB is 60% faster than ECMP and 15% faster than DLB (Figure 9 of the paper). Figure 10 of the paper sweeps the load on one leaf-to-ToR link.

沒有背景時，三者的 All-to-All 匯流排頻寬都在 56.7–58.8 Gb/s，因為空的鏈路上沒有東西要避開。加上背景後，SGLB 比 ECMP 快 60%，比 DLB 快 15%（論文 Figure 9）。論文 Figure 10 掃描一條 leaf 往 ToR 鏈路上的負載。

Figure 10 of the SGLB paper: normalized All-to-All bus bandwidth against link load from 50 to 82 percent for SGLB, DLB and ECMP; SGLB stays highest — Figure 10 of the paper, measured on the testbed: All-to-All bus bandwidth against background load on one leaf-to-ToR link, normalized to SGLB without background.論文 Figure 10，測試平台實測：All-to-All 匯流排頻寬對一條 leaf 往 ToR 鏈路上的背景負載，以 SGLB 無背景時為基準。Qi et al., SIGCOMM 2025, Figure 10 (ACM PDF, p. 10).

At 82% load SGLB loses 48% of its unloaded bandwidth, DLB 55% and ECMP 65%. SGLB's lead over DLB and ECMP grows from 6.7% and 8.3% at 50% load to 14% and 46% at 82%. AllReduce shows the local–global gap more sharply.

在 82% 負載時，SGLB 損失無負載頻寬的 48%，DLB 55%，ECMP 65%。SGLB 對 DLB 與 ECMP 的領先，從 50% 負載時的 6.7% 與 8.3%，增加到 82% 時的 14% 與 46%。AllReduce 把本地與全域的差距顯示得更清楚。

Figure 11 of the SGLB paper: normalized AllReduce bus bandwidth against link load from 74 to 85 percent; SGLB flat near 100, DLB drops after 77, ECMP lowest — Figure 11 of the paper, measured on the testbed: AllReduce bus bandwidth as the load on one leaf-to-ToR link rises from 74% to 85%.論文 Figure 11，測試平台實測：一條 leaf 往 ToR 鏈路的負載從 74% 升到 85% 時的 AllReduce 匯流排頻寬。Qi et al., SIGCOMM 2025, Figure 11 (ACM PDF, p. 10).

Hierarchical AllReduce works mostly inside the rack, so little of its traffic needs the loaded link, and SGLB moves that part elsewhere. DLB cannot see the load one hop away. At 85% load, normalized bus bandwidth is 99.5% for SGLB, 67.6% for DLB and 57% for ECMP.

階層式 AllReduce 主要在機櫃內完成，所以只有少量流量需要經過有負載的鏈路，而 SGLB 把這部分移到別處。DLB 看不見一跳之外的負載。在 85% 負載時，正規化匯流排頻寬 SGLB 為 99.5%，DLB 為 67.6%，ECMP 為 57%。

In LLaMA-3 training on 32 GPUs, RDMA writes loaded one ToR–leaf link at 500 Gb/s each way. The measured times give the paper's 12.3%:

在 32 顆 GPU 的 LLaMA-3 訓練中，RDMA 寫入讓一條 ToR–leaf 鏈路兩個方向各有 500 Gb/s 負載。量到的時間得出論文的 12.3%：

Question問題	Calculation算式	Result結果
Iteration with ECMP: 2.66 s compute + 1.177 s communicationECMP 一次迭代：計算 2.66 秒 + 通訊 1.177 秒	$2.66 + 1.177$	$3.837$ s
Iteration with SGLB: 0.704 s communicationSGLB 一次迭代：通訊 0.704 秒	$2.66 + 0.704$	$3.364$ s
Training time saved節省的訓練時間	$1 - 3.364 \div 3.837$	$\approx 12.3\%$ assumes no overlap假設沒有重疊

Per flowlet instead of per packet, SGLB wins by less, because a flowlet keeps its first packet's path (Figure 12 of the paper). Failover times are in Figure C. Table 2 of the paper fails one link of a LAG under AllReduce:

改成逐 flowlet 而非逐封包時，SGLB 勝出的幅度較小，因為一個 flowlet 會沿用第一個封包的路徑（論文 Figure 12）。故障切換時間見圖 C。論文 Table 2 在 AllReduce 下讓 LAG 的一條鏈路故障：

Scheme, link Z	方案，鏈路 Z	Bus bw (Gb/s)	匯流排頻寬（Gb/s）	ToR uplinks (Gb/s)
DLB, up	DLB，正常	47.97	4 × 792.5	396.5
DLB, failed	DLB，故障	22.23	4 × 370	185
SGLB, up	SGLB，正常	47.92	4 × 791.5	396
SGLB, failed	SGLB，故障	32.38	3 × 722	270

DLB keeps feeding C and D and falls 54%. SGLB shuts C and keeps 46% more than DLB: removing a healthy link raised throughput. Not shown here: the SyncMesh start-up benchmark (emulated on two switches) and the comparisons with DRILL, ConWeave, CONGA and NDP (simulation only, against simplified re-implementations).

DLB 繼續把流量送進 C 與 D，下降 54%。SGLB 關掉 C，比 DLB 多保住 46%：拿掉一條健康的鏈路，反而提高了吞吐。本頁不列：SyncMesh 啟動成本的測試（在兩台交換器上模擬），以及與 DRILL、ConWeave、CONGA、NDP 的比較（只有模擬，而且對手是簡化的重新實作）。

6. Limits and open questions6. 限制與未解的問題

Small scale. 32 GPUs, 14 switches, background on one or two links. The O(10K)-GPU claim is table counting; there are no production numbers.規模小。32 顆 GPU、14 台交換器，背景流量只在一兩條鏈路上。O(10K) GPU 的說法靠的是表格計數；沒有生產環境的數字。
Reordering. Per-packet mode needs NICs that accept out-of-order packets; the paper does not say which it used (compare IRN).亂序。逐封包模式需要能接受亂序封包的 NIC；論文沒有說用了哪一種（對照 IRN）。
One hop of remote state. Congestion further away shows only through $B^{(R)}$ , and ToRs score cross-rail traffic locally.只有一跳的遠端狀態。更遠處的壅塞只能透過 $B^{(R)}$ 看到，而 ToR 對跨 rail 的流量只做本地評分。
Capacity thrown away. After the LAG failure only 3 of 4 uplinks carry traffic; splitting by weight would do better.丟掉的容量。LAG 故障後 4 條上行只有 3 條在用；依權重分流會做得更好。
Units. Table 2 says Gb/s, but NCCL-tests prints GB/s, and 47.97 is close to the NIC's 396.5 Gb/s ÷ 8. The ratios are unaffected.單位。Table 2 標的是 Gb/s，但 NCCL-tests 輸出 GB/s，而 47.97 接近 NIC 的 396.5 Gb/s ÷ 8。比例不受影響。

7. How it connects7. 與其他內容的關聯

Chapter 8: a fourth answer beside entropy (§3), no choice (§4) and spraying (§5). Its per-packet mode still needs the NIC of §5.第 8 章：在增加熵（§3）、拿掉選擇（§4）與噴灑（§5）之外的第四種答案。它的逐封包模式仍需要 §5 的那種 NIC。
Alibaba's Stellar sprays each connection over 128 paths from its own NIC; SGLB keeps the choice in the switch. HyNA also adds processor cores beside a switch pipeline, for in-network aggregation.阿里的 Stellar 從自家 NIC 把每條連線噴灑到 128 條路徑；SGLB 把選擇留在交換器。HyNA 同樣在交換器管線旁加上處理器核心，用於網內聚合。
Chapter 10: R-Pingmesh, also from ByteDance, finds the bad link. PReCCL moves bytes off a slow path from the host side.第 10 章：同樣來自字節跳動的 R-Pingmesh 找出壞掉的鏈路。PReCCL 從主機端把位元組移離慢路徑。
Chapter 6: the reference cluster is one more rail-based Clos beside HPN and Meta's.第 6 章：這個參考叢集是 HPN 與 Meta 之外又一個以 rail 為基礎的 Clos。

Key numbers關鍵數字

Number	數字	Meaning (testbed unless noted)
+60% / +15%	All-to-All vs ECMP / DLB, with background	有背景時 All-to-All 對 ECMP / DLB
48% / 55% / 65%	Loss of SGLB / DLB / ECMP at 82% load	82% 負載時 SGLB / DLB / ECMP 的損失
99.5% / 67.6% / 57%	AllReduce at 85% load: SGLB / DLB / ECMP	85% 負載時 AllReduce：SGLB / DLB / ECMP
−12.3%	LLaMA-3 training time, SGLB vs ECMP	LLaMA-3 訓練時間，SGLB 對 ECMP
~1 µs / ~45 µs / > 1 s	Failover: link ends / one hop / BGP	故障切換：鏈路兩端 / 一跳 / BGP
32.38 vs 22.23	Bus bandwidth after a LAG link fails, SGLB vs DLB	LAG 一條鏈路故障後的匯流排頻寬，SGLB 對 DLB

Terms used on this page本頁用到的術語

GLB engine: A switch-chip block that scores paths from local and neighbour state and picks a path per packet.; 交換晶片中的一個區塊，依本地與鄰居狀態為路徑評分，並為每個封包挑路徑。
SyncMesh: SGLB's link-state protocol: versioned link records shared within areas.; SGLB 的鏈路狀態協定：帶版本號的鏈路記錄，在區域內共享。
LAG: Several links between two switches used as one logical link.; 兩台交換器之間當成一條邏輯鏈路使用的數條鏈路。
Throughput suppression: Loss on a weakened path makes congestion control slow whole flows, so healthy paths go underused.; 變弱路徑上的丟包讓壅塞控制拖慢整條流，使健康路徑用不滿。