Load-balancing research: in the switch, in the NIC, and watching負載平衡研究：在交換器、在 NIC，以及監看

This page ends with per-packet spraying as the end state, and with two open points: spraying needs NICs that tolerate out-of-order arrival, and Section 8.5 had no production paper with numbers for adaptive routing on training traffic. Three recent papers on this page, and a fourth on the transport research page, take spraying as given and ask what it costs. They are easiest to read by where the path decision is made: in the switch, in the NIC, or nowhere, with the network only watched. Figure R1 places them, and the table compares them.

本頁以逐封包噴灑作為終點，並留下兩個未解的問題：噴灑需要能容忍亂序到達的 NIC，而第 8.5 節找不到一篇針對訓練流量、給出自適應路由數字的生產環境論文。本頁的三篇近期論文，以及傳輸研究頁的第四篇，都把噴灑當成前提，追問它的代價。最容易的讀法是看路徑決定在哪裡做：在交換器、在 NIC，或者不做決定、只負責觀察網路。圖 R1 標出它們的位置，表格則加以比較。

In the switch: SGLB. ByteDance's SGLB uses the global load-balancing engine of a commodity Broadcom switch chip. Each switch scores a path from its own queue and link load and from the state its next hop reports, and steers each packet, flowlet or flow to a good path. A link-state protocol beside BGP, with a data-plane shortcut and a rule that "unreachable" always wins, handles failures; after a partial link-group failure, SGLB disables just enough links to keep paths symmetric. On a testbed of 32 H800 GPUs and 14 switches, All-to-All bus bandwidth was 60% higher than with ECMP under injected background flows, failures converged in 1–45 µs against more than 1 s with BGP, and LLaMA-3 training time fell 12.3% against ECMP. Its comparisons with other research load balancers are simulation only.

在交換器：SGLB。字節跳動的 SGLB 使用商用 Broadcom 交換晶片內的全域負載平衡引擎。每台交換器依自己的佇列與鏈路負載，加上下一跳回報的狀態為路徑評分，把每個封包、flowlet 或流導向好的路徑。與 BGP 並行的鏈路狀態協定，加上資料平面捷徑與「不可達永遠優先」的規則，負責處理故障；鏈路群組部分故障後，SGLB 只停用剛好足夠的鏈路讓路徑保持對稱。在 32 顆 H800 GPU、14 台交換器的測試平台上，注入背景流量時 All-to-All 的匯流排頻寬比 ECMP 高 60%，故障在 1–45 µs 內收斂（BGP 超過 1 s），LLaMA-3 訓練時間比 ECMP 少 12.3%。它與其他研究型負載平衡器的比較只有模擬。

Animation R6.1 plays one link failure with SGLB's numbers, once with BGP and once with a local reroute.

動畫 R6.1 用 SGLB 的數字播放一次鏈路故障，一次用 BGP，一次用本地改路由。

In the NIC, many paths: Stellar. Alibaba's Stellar, whose digest is listed on the transport research page, is the production data point that Section 8.5 lacked. Its FPGA-based RNIC sprays each connection obliviously over 128 paths, more than the 60 aggregation switches of HPN, with one shared congestion window and a 250 µs retransmission timeout. On a 1,024-GPU testbed against ConnectX-7 it trained 6% faster on average, and up to 14%, with random rank placement, but only 0.72% faster with locality-aware placement. With 128 paths, a link dropping 1% or 3% of packets caused almost no slowdown of a 960-GPU AllReduce. The strongest lever of Section 8.6, placement, still shows through.

在 NIC、多路徑：Stellar。阿里的 Stellar 的導讀列在傳輸研究頁，正是第 8.5 節缺少的生產環境數據。它以 FPGA 為基礎的 RNIC 把每條連線不看狀態地噴灑到 128 條路徑上（多於 HPN 的 60 台聚合交換器），共用一個壅塞窗口，並設 250 µs 的重傳逾時。在 1,024 顆 GPU 的測試平台上對比 ConnectX-7，隨機配置 rank 時訓練平均快 6%、最多 14%；依局部性配置時只快 0.72%。有 128 條路徑時，一條丟掉 1% 或 3% 封包的鏈路，幾乎不拖慢 960 顆 GPU 的 AllReduce。第 8.6 節最強的槓桿——配置——在這裡依然看得出來。

In the NIC, few lanes: PSN-PATH. PSN-PATH looks inside the NIC at a cost Stellar does not discuss: a receiver that takes every sequence gap as a loss resends data that was only late. It stripes a queue pair into four lanes, each pinned to one UDP source port, so a gap inside a lane is a real loss and is NACKed at once. Nothing ran on real switches or NICs: FPGA synthesis estimates 55.3% fewer LUTs and 83.7% less block RAM than MP-RDMA, and its throughput results come from FPGA-emulated switches and ns-3. It needs switches that keep per-flow ECMP.

在 NIC、少數通道：PSN-PATH。PSN-PATH 看進 NIC 內部一個 Stellar 沒有討論的代價：把每個序號缺口都當遺失的接收端，會重送只是晚到的資料。它把一個 queue pair 切成四條通道，每條固定一個 UDP 來源埠，於是通道內的缺口就是真正的遺失，立刻送 NACK。整篇論文沒有在真實交換器或 NIC 上執行：FPGA 合成估計它比 MP-RDMA 少用 55.3% 的 LUT 與 83.7% 的 block RAM，吞吐量結果則來自 FPGA 模擬的交換器與 ns-3。它需要維持逐流 ECMP 的交換器。

Watching: OmniPath Ping. Once switches spray, a probe takes one random path and a TTL no longer names a switch, so the service tracing of R-Pingmesh loses its meaning. OmniPath Ping has switches copy one probe onto every equal-cost path, drop duplicates where paths merge, merge the ACKs on the way back and name the link that never answered. On a 4-switch Tofino testbed (2 leaves, 2 spines, 4 servers, 100 Gb/s), one probe per task covered 100% of paths against about 66.5% for Ping-based baselines, with 576 probe and ACK packets per switch against 2,688. Its scale results are NS-3. It answers Stellar's objection that switch spraying hides paths, but only for switch spraying.

觀察：OmniPath Ping。交換器一旦噴灑，探測只走一條隨機路徑，TTL 也不再對應一台交換器，R-Pingmesh 的服務追蹤便失去意義。OmniPath Ping 讓交換器把一個探測複製到每條等價路徑、在路徑匯合處丟棄重複、在回程合併 ACK，並指出始終沒有回應的鏈路。在 4 台交換器的 Tofino 測試平台上（2 台 leaf、2 台 spine、4 台伺服器、100 Gb/s），每個任務一個探測涵蓋 100% 的路徑，基於 Ping 的基準約 66.5%；每台交換器的探測與 ACK 封包為 576 個，基準 2,688 個。它的規模結果來自 NS-3。它回應了 Stellar「交換器噴灑會隱藏路徑」的疑慮，但只限於交換器噴灑。

Where they agree, and where they differ. All four agree that per-flow ECMP is not enough for training traffic, that something must tolerate reordering, and that path changes must react faster than BGP: SGLB in microseconds, Stellar by resending on another path, PSN-PATH by an immediate NACK and a lane move, OmniPath Ping by naming the failed link. They differ on who decides and on how many paths. SGLB trusts the switch because the switch sees the queues; Stellar trusts its own NIC and commodity switches. Stellar uses 128 paths, PSN-PATH argues for four, and SGLB scores all paths of a three-hop segment and sprays among the best few. The evidence differs as much as the answers: production and testbeds for SGLB and Stellar, emulation and simulation for PSN-PATH. The designs can also collide: PSN-PATH's lanes break under SGLB's packet mode, and OmniPath Ping covers only switch-side spraying.

相同之處與不同之處。四篇都同意：逐流 ECMP 對訓練流量不夠，總得有某個元件容忍亂序，而路徑變更必須比 BGP 快——SGLB 以微秒計，Stellar 在另一條路徑重送，PSN-PATH 立刻送 NACK 並搬移通道，OmniPath Ping 則指出故障的鏈路。它們的分歧在於誰來決定、以及用幾條路徑。SGLB 信任交換器，因為交換器看得到佇列；Stellar 信任自家 NIC 與商用交換器。Stellar 用 128 條路徑，PSN-PATH 主張四條，SGLB 則為三跳區段的所有路徑評分、在最好的少數幾條之間噴灑。證據的差異和答案一樣大：SGLB 與 Stellar 有生產環境與測試平台，PSN-PATH 只有模擬。這些設計也會互相衝突：PSN-PATH 的通道在 SGLB 的封包模式下會失效，而 OmniPath Ping 只涵蓋交換器端的噴灑。

Three questions stay open: how to combine switch-side awareness with host-side recovery without both reacting to the same loss; how many paths are enough, measured on one real fabric; and how to monitor a fabric where the NIC sprays. Read SGLB first, then Stellar for the production numbers, then PSN-PATH as mechanism, and OmniPath Ping last.

還有三個問題未解：如何結合交換器端的感知與主機端的恢復，而不讓兩者對同一次遺失重複反應；在同一個真實網路上量測，到底幾條路徑才夠；以及如何監控由 NIC 噴灑的網路。先讀 SGLB，再讀 Stellar 看生產環境數字，接著把 PSN-PATH 當成機制來讀，最後讀 OmniPath Ping。