Sharing and simulation research: schedule, be fair, place, simulate共用與模擬研究：排程、公平、擺放、模擬

Five recent papers take this chapter's questions further. They start from one fact: a training fabric is shared, and its behavior can neither be predicted from one job nor tested on the real cluster. Their common ground truth is Alibaba's 2023 trace, released with Crux: more than 2,000 GPUs, more than 5,000 jobs in two weeks, more than 30 jobs at once at peak, and 36.3% of jobs, holding 51% of the GPUs, exposed to contention with other jobs, mostly through ECMP hash collisions (Section 11.1). That figure is exposure inferred from placement, not measured slowdown. Three papers ask what to do about the contention. Two ask how to know what a design will do before anyone builds it.

五篇近期論文把本章的問題往前推。它們都從同一個事實出發：訓練網路是共享的，它的行為既無法從單一任務預測，也無法在真實叢集上測試。共同的基準是阿里與 Crux 一起公開的 2023 年紀錄：超過 2,000 顆 GPU、兩週內超過 5,000 個任務、尖峰時同時超過 30 個任務，而且 36.3% 的任務（占 51% 的 GPU）暴露在與其他任務的競爭中，主要來自 ECMP 雜湊碰撞（第 11.1 節）。這個數字是從配置推得的暴露程度，不是量到的減速。三篇論文問的是如何處理這種競爭；兩篇問的是在任何人動手建置之前，如何知道一個設計會怎樣。

Sharing: schedule, be fair, or place. Crux keeps the placement and decides who gets the link. Its rule is GPU intensity, $I_j = W_j / t_j$, the computation a job completes per second of communication on its bottleneck link. The most intensive job picks the least-congested ECMP path first, and priorities follow intensity, compressed into about eight hardware levels. LEVELLER uses the same knobs with the opposite objective. It gives strict priority to the job with the lowest normalized progress rate $R_j$, the iterations a job completed divided by those it would have completed alone, and proves this greedy rule within a factor of two of max-min fairness. MonkeyTree changes the placement instead. With ring collectives, a rack's uplink demand is the number of fragmented jobs in it; at most two per rack is always achievable for data parallelism, and a small integer program restores that bound with the fewest worker migrations.

共享：排程、公平，還是配置。Crux 不動配置，決定誰取得鏈路。它的規則是 GPU 密集度 $I_j = W_j / t_j$：任務在瓶頸鏈路上每一秒通訊所完成的計算量。最密集的任務先挑壅塞最少的 ECMP 路徑，優先權也依密集度排定，再壓縮進約八個硬體層級。LEVELLER 用同樣的旋鈕，目標卻相反。它把嚴格優先權給正規化進度率 $R_j$ 最低的任務（任務完成的迭代數除以它獨自執行時會完成的迭代數），並證明這個貪婪規則與 max-min 公平相差不到兩倍。MonkeyTree 改的是配置。使用環狀集體通訊時，機櫃的上行需求就是其中被碎片化的任務數；對資料平行，每個機櫃最多兩個永遠做得到，而一個小型整數規劃以最少的 worker 遷移恢復這個上限。

The three agree that a scheduler's placement leaves contention behind as jobs come and go, that routing alone cannot help once a rack's egress exceeds its uplinks, and that commodity knobs are enough to act: the UDP source port steers ECMP, the DSCP traffic class sets priority, and checkpoint and restore moves a worker. They differ in what they optimize: cluster GPU utilization (Crux), the slowest job's progress (LEVELLER), or bounded load per rack (MonkeyTree). The answers can stack: MonkeyTree bounds the load, and a scheduler then rations what is left. The paper tries this only with Crux's routing, in simulation; no one has paired it with LEVELLER. Also open is how to weigh job size, which Crux counts and LEVELLER ignores.

三者都同意：任務來來去去時，排程器的配置會留下競爭；一旦機櫃的出口流量超過上行，單靠路由幫不上忙；而現成的旋鈕足以採取行動：UDP 來源埠導引 ECMP，DSCP 流量類別設定優先權，檢查點與還原搬動 worker。它們的差異在於最佳化的對象：叢集 GPU 使用率（Crux）、最慢任務的進度（LEVELLER），或每個機櫃的有界負載（MonkeyTree）。這些答案可以疊加：MonkeyTree 限制負載，再由排程器分配剩下的競爭。論文只在模擬中與 Crux 的路由組合過；還沒有人把它與 LEVELLER 搭配。同樣未解的是如何衡量任務大小：Crux 把它算進去，LEVELLER 則忽略它。

Simulation: fidelity against speed. Section 12.2 describes the two simulator levels; these papers measure them. SimAI makes the simulation of one job trustworthy by running the real Megatron or DeepSpeed and the real NCCL logic on one host, then replaying the exact flows in a multi-threaded ns-3. Against two 128-host testbeds it matched iteration time within 3.9% up to 1,024 GPUs, where ASTRA-sim's components erred by tens to hundreds of percent. Its remaining cost is the packet-level event count. Wormhole, with two SimAI co-authors, removes most of it without leaving the packet level: it memoizes contention patterns that repeat and fast-forwards through the steady states of congestion control. On a 56-core server it processed 227 to 745 times fewer events than ns-3 for GPT workloads of 64 to 1,024 GPUs, and on a real 256-GPU trace it matched the measured training time within 3.02%.

模擬：精確度對上速度。第 12.2 節描述了兩種模擬器層級；這兩篇論文量測了它們。SimAI 讓單一任務的模擬值得信賴：在一台主機上執行真實的 Megatron 或 DeepSpeed 與真實的 NCCL 邏輯，再於多執行緒的 ns-3 中重播精確的流。對照兩座 128 台主機的測試平台，它在最多 1,024 顆 GPU 時，迭代時間的誤差在 3.9% 以內，而 ASTRA-sim 的各元件誤差達數十到數百個百分點。它剩下的成本是封包級的事件數。與 SimAI 有兩位共同作者的 Wormhole，在不離開封包層級的前提下去掉了大部分事件：記住重複出現的競爭模式，並快轉壅塞控制的穩態。在一台 56 核心伺服器上，它處理 64 到 1,024 顆 GPU 的 GPT 工作負載時，事件數比 ns-3 少 227 到 745 倍；在一份真實的 256 顆 GPU 紀錄上，它與實測訓練時間的誤差在 3.02% 以內。

The two threads meet uncomfortably. The large-scale evidence of the sharing papers comes from coarse models: an $\alpha$–$\beta$ model in Crux, a flow-level simulator fed by ASTRA-sim in LEVELLER, a max-min flow-level simulator in MonkeyTree. Wormhole compared a flow-level simulator with ns-3 on the same simulated LLM training traffic over rail-optimized fat-trees and found about 20% average error in flow completion time. That is one flow-level simulator against a packet-level reference, not a hardware measurement and not the three papers' own simulators. Still, it tells a reader how far to trust a simulated gain of a few percent. Neither simulator has been validated under multi-tenant contention either: SimAI's validation runs one job at a time, and Wormhole expects smaller gains when traffic from many tenants repeats less. So the table lists only measured results, and their scale is small: 12 to 96 GPUs on testbeds, one migrated worker on a prototype.

兩條主線的交會並不順利。共享類論文的大規模證據來自粗略的模型：Crux 用 $\alpha$–$\beta$ 模型，LEVELLER 用以 ASTRA-sim 為輸入的流級模擬器，MonkeyTree 用 max-min 流級模擬器。Wormhole 在 rail-optimized fat-tree 上、以同一份模擬的 LLM 訓練流量比較流級模擬器與 ns-3，發現流完成時間的平均誤差約 20%。這是一個流級模擬器對照封包級基準的結果，不是硬體量測，也不是那三篇論文自己的模擬器。但它仍告訴讀者，模擬出的幾個百分點增益能信到什麼程度。兩個模擬器也都沒有在多租戶競爭下驗證過：SimAI 的驗證一次只跑一個任務，Wormhole 則預期在多租戶流量重複較少時增益會變小。所以下表只列實測結果，而且規模很小：測試平台上 12 到 96 顆 GPU，原型上一個被遷移的 worker。