What clusters and jobs actually carry: Acme and Hermod叢集與任務實際承載了什麼：Acme 與 Hermod
Six months of traces from a 4,704-GPU LLM datacenter, and the order in which one job's collectives should go.一個 4,704 顆 GPU 的 LLM 資料中心六個月的追蹤紀錄，以及一個任務內的集體操作應以什麼順序進行。
This page assumes Chapters 1–5. Every term it uses is defined there; the glossary on the contents page links to each definition. The papers here are recent and most of their results come from small testbeds or simulation, so read each number with its label.
本頁假設你已讀過第 1–5 章。這裡用到的每個名詞都在那些章節中定義過；目錄頁的詞彙表連到每個定義。這些論文都很新，多數結果來自小型測試平台或模擬，所以請連同標籤一起讀每個數字。
Terms from later chapters used here (5)本頁用到的後面章節名詞（5 個）
- Lossless network無損網路 Ch. 7 A network whose switches pause senders instead of dropping packets when a buffer fills.緩衝區滿時，交換器會暫停傳送端而不丟封包的網路。
- Hash collision雜湊碰撞 Ch. 8 Two large flows hashed onto the same uplink, so each gets half its rate.兩條大流被雜湊到同一條上行，各只得到一半速率。
- Standby pool, spare備用池、備援 Ch. 10 Spare machines kept ready so a failed one can be replaced without waiting for a repair.預備的備用機器，故障時可立即替換，不必等待修理。
- Placement, scheduler放置、排程器 Ch. 11 Choosing which GPUs a job gets and which rank runs where; it decides which links the traffic crosses.決定工作拿到哪些 GPU、各 rank 放在哪裡；它決定流量經過哪些鏈路。
- Simulation, packet-level simulation模擬、封包層級模擬 Ch. 12 Running a model of the workload and network in software; a packet-level simulator follows every packet through every queue.以軟體執行工作負載與網路的模型；封包層級模擬器追蹤每個封包經過每個佇列。
This page argues from first principles. A step is arithmetic plus an exchange whose bytes do not shrink with scale, every collective waits for the slowest rank, and the traffic arrives in synchronized line-rate bursts. Two recent papers test that argument from opposite ends. Acme (NSDI 2024) measures a whole LLM datacenter for six months. Hermod (SIGCOMM 2025) looks inside one job, at the moments when two kinds of its traffic want the same link.
本頁從第一原理論證。一步是算術加上一次交換，交換的位元組不隨規模縮小；每次集體操作都等最慢的 rank；流量以同步的滿線速突發抵達。兩篇近期論文從兩端檢驗這個論證。Acme（NSDI 2024）量測一整座 LLM 資料中心六個月。Hermod（SIGCOMM 2025）則看進單一任務內部，看它的兩種流量同時要用同一條鏈路的時刻。
Acme: what the cluster carries. Shanghai AI Laboratory traced 4,704 A100 GPUs in two clusters, about 1.09 million jobs. The first lesson is that count is not cost. In the Kalos cluster, evaluation is 92.9% of jobs but 0.8% of GPU time, and pretraining is 3.2% of jobs but 94.0% of GPU time; jobs with 256 or more GPUs use over 96% of it. So the network is built for a few large synchronous jobs, the subject of this page. The second lesson is about averages. Seren's InfiniBand NICs are idle over 60% of the time and rarely exceed 25% of line rate in 15-second samples. That is what the bursts of Section 5.6 look like through a coarse average, not proof of spare bandwidth. The third lesson is the barrier of Section 1.4 at cluster scale: infrastructure failures are 11% of failed jobs but over 82% of the GPU time that failures waste, because they strike the largest jobs midway. Acme also separates storage: each Kalos node has four InfiniBand NICs for training and a fifth for storage, the separation that Meta and HPN make with whole networks.
Acme：叢集承載的是什麼。上海人工智慧實驗室記錄了兩個叢集共 4,704 顆 A100 GPU、約 109 萬個任務。第一課是數量不等於成本。在 Kalos 叢集，評估占任務數的 92.9%、GPU 時間只占 0.8%；預訓練占任務數的 3.2%、GPU 時間卻占 94.0%；256 顆 GPU 以上的任務用掉其中超過 96%。所以網路是為少數大型同步任務而建，也就是本頁的主題。第二課關於平均值。Seren 的 InfiniBand NIC 超過 60% 的時間閒置，以 15 秒取樣時很少超過線速的 25%。這正是第 5.6 節的突發透過粗糙平均值看起來的樣子，不能證明頻寬有餘。第三課是叢集規模下的第 1.4 節柵欄：基礎設施故障只占失敗任務的 11%，卻占故障浪費的 GPU 時間超過 82%，因為它們在最大的任務執行到一半時發生。Acme 也把儲存分開：每個 Kalos 節點有四張訓練用的 InfiniBand NIC，第五張給儲存，正是 Meta 與 HPN 用整個網路做的那種分離。
Hermod: who goes first inside a job. In an MoE job trained with ZeRO-2 and 1F1B, the expert-parallel all-to-all, the pipeline sends and the data-parallel reduce-scatter and all-gather are coflows: groups of flows that a computation waits for as a unit. The training graph fixes when they overlap: across micro-batches, in the backward pass, and at the iteration boundary. Hermod gives every coflow a static key (micro-batch ID, then type with EP and PP before DP, then layer ID) and asks the whole cluster to serve that order. Its claim is about order, not bandwidth: fair sharing gives a link to whoever has bytes, but training needs it to go to whatever the next computation waits for. The short paper has no implementation and evaluates only in simulation, so its digest gives the mechanism and no speedups.
Hermod：任務內誰先走。在以 ZeRO-2 與 1F1B 訓練的 MoE 任務中，專家平行的 all-to-all、管線傳送，以及資料平行的 reduce-scatter 與 all-gather 都是 coflow：一組流，計算把它當成一個整體來等待。訓練圖固定了它們何時重疊：跨 micro-batch、在反向期間、在迭代交界。Hermod 給每個 coflow 一個靜態鍵值（micro-batch 編號，再比類型，EP 與 PP 先於 DP，最後比層編號），並要求整個叢集依此順序服務。它的主張關於順序而非頻寬：公平分享把鏈路給有位元組要送的人，訓練卻需要把它給下一段計算正在等的東西。這篇短論文沒有實作，只在模擬中評估，所以它的導讀只講機制，不給加速數字。
| Acme | Hermod | ||||
|---|---|---|---|---|---|
| Venue | 會議 | NSDI 2024 | SIGCOMM 2025 (short paper) | SIGCOMM 2025（短論文） | |
| Problem | 問題 | What LLM development does to a GPU datacenter: job mix, utilization, failures | LLM 開發對 GPU 資料中心造成什麼：任務組成、使用率、故障 | EP, PP and DP coflows of one MoE job contend for links at predictable moments | 同一個 MoE 任務的 EP、PP 與 DP coflow 在可預測的時刻搶鏈路 |
| Mechanism | 機制 | Six-month trace of 4,704 A100s; fault-tolerant pretraining and an evaluation scheduler built on its findings | 4,704 顆 A100 的六個月紀錄；據此建立的容錯預訓練與評估排程器 | Static cluster-wide key: micro-batch ID, then type (EP and PP before DP), then layer ID | 全叢集的靜態鍵值：micro-batch 編號，再比類型（EP 與 PP 先於 DP），最後比層編號 |
| Measured result | 實測結果 | Kalos pretraining: 3.2% of jobs, 94.0% of GPU time. NICs idle over 60% of the time (15 s samples). Infrastructure failures: 11% of failed jobs, over 82% of their GPU time. Evaluation makespan up to 1.8× shorter. | Kalos 預訓練：任務數 3.2%、GPU 時間 94.0%。NIC 超過 60% 的時間閒置（15 秒取樣）。基礎設施故障：失敗任務的 11%、其 GPU 時間超過 82%。評估總時長最多縮短 1.8 倍。 | None: the evaluation is simulation only and is not reported here | 沒有：評估只有模擬，本站不列出 |
Where they agree, and where they differ. The two papers agree on three points. The unit of design is the large synchronous job, not the average job. MoE all-to-all is where the network shows: Acme measured much lower GPU utilization for an MoE model on servers with one InfiniBand NIC, and Hermod puts EP coflows ahead of DP coflows. And averages mislead: a 15-second NIC average hides bursts, and a fair share hides which flow is urgent. They differ in evidence and in lever. Acme is production measurement, and its answers are provisioning, separation and fast recovery. Hermod is a design argued from the training graph and checked only in simulation, and its answer is ordering on the bandwidth that exists.
相同之處與不同之處。兩篇論文在三點上一致。設計的單位是大型同步任務，而不是平均的任務。MoE 的 all-to-all 是網路最吃緊的地方：Acme 量到 MoE 模型在只有一張 InfiniBand NIC 的伺服器上 GPU 使用率低得多，Hermod 則把 EP coflow 排在 DP coflow 之前。還有，平均值會誤導：15 秒的 NIC 平均值掩蓋突發，公平分享掩蓋哪條流最急。兩者的差別在證據與手段。Acme 是生產環境量測，它的答案是充足的頻寬、分離與快速恢復。Hermod 是從訓練圖推論出的設計，只在模擬中檢驗，它的答案是在現有頻寬上排順序。
Neither paper closes the loop. Acme's clusters give 25 to 100 Gb/s per GPU, a quarter or less of the 400 Gb/s per GPU in the Meta and HPN designs, and its counters are too coarse to see a burst. Hermod has not met a real switch: Ethernet has eight priority classes, operators make only a few of them lossless (Microsoft made two), and the paper does not say how its keys fit into them. Read Acme first for what the cluster carries, then Hermod for how one job's traffic should be ordered.
兩篇論文都沒有把問題收尾。Acme 的叢集每顆 GPU 只有 25 到 100 Gb/s，是 Meta 與 HPN 設計中每顆 GPU 400 Gb/s 的四分之一或更少，而且它的計數器太粗，看不到突發。Hermod 還沒有碰過真正的交換器：Ethernet 有八個優先權類別，營運者只會把其中少數設為無損（Microsoft 設了兩個），而論文沒有說它的鍵值如何放進這些類別。先讀 Acme，了解叢集承載什麼；再讀 Hermod，了解單一任務的流量該如何排序。