Research digest 4 of 5 · Chapter 11: Sharing the cluster

研究導讀 4 / 5 · 第 11 章：共用叢集

SimAI: Unifying Architecture Design and Performance Tuning for Large-Scale Large Language Model Training with Scalability and PrecisionSimAI：兼顧可擴展性與精確度，統一大規模大型語言模型訓練的架構設計與效能調校

NSDI 2025Xizheng Wang, Qingxu Li, Yichi Xu et al. (Alibaba Cloud, Tsinghua University, Zhongguancun Laboratory, South China University of Technology) · paper論文

After reading, you can explain how SimAI predicts a training iteration within 4% of a real 1,024-GPU cluster: real framework and NCCL code, measured kernels, packet-level network, and what each part fixes.讀完後，你能說明 SimAI 如何把一次訓練迭代的預測做到與真實 1,024 顆 GPU 叢集相差 4% 以內：真實的框架與 NCCL 程式碼、實測的 kernel、封包級網路，以及每一部分修正了什麼。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 11 ↗第 11 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Alibaba wanted one simulator to choose hardware before buying it and to tune NCCL on running clusters. Earlier tools rebuild the training software as a model, and the model drifts: ASTRA-sim with its built-in workload missed Alibaba's testbeds by 92.1–143.9%. SimAI reuses the real software. It runs Megatron or DeepSpeed on one host that believes it is a whole cluster, takes kernel times from measurements, expands collectives with NCCL's own code, and sends the flows through packet-level ns-3. On 128-host A100 and H100 testbeds, simulated iteration time was within 3.9% of the real one (under 4% at 1,024 GPUs); collective bandwidth was off by 3.9% and 2.3%, against 74.8% and 51.7% for ASTRA-sim. The price is speed: an early version needed more than 24 hours for a GPT-3 iteration that 128 GPUs run in 2 seconds.

阿里希望用一個模擬器在採購前選定硬體，並在運作中的叢集上調校 NCCL。早期工具把訓練軟體重寫成模型，而模型會偏離實際：用內建工作負載的 ASTRA-sim 與阿里測試平台相差 92.1–143.9%。SimAI 改為重用真實軟體。它讓 Megatron 或 DeepSpeed 在一台「以為自己是整個叢集」的主機上執行，kernel 時間取自實測，集體操作用 NCCL 自己的程式碼展開，再把流送進封包級的 ns-3。在 128 台主機的 A100 與 H100 測試平台上，模擬的迭代時間與真實值相差 3.9% 以內，1,024 顆 GPU 時低於 4%；集體操作頻寬偏差為 3.9% 與 2.3%，ASTRA-sim 則是 74.8% 與 51.7%。代價是速度：早期版本模擬 GPT-3 的一次迭代要超過 24 小時，而 128 顆 GPU 只需 2 秒。

1. Where this paper sits1. 這篇論文的位置

Section 12.2 ("What a simulator adds") lists what a simulator needs: the kernels and communication calls, the collective algorithm that turns calls into flows, and a network that carries them. It warns that simulated MFU is a comparison, not a prediction, because simulators depend on their compute and collective models. SimAI tests that warning with measurements. Before it, Alibaba's flow-level planning and packet-level tuning disagreed with each other and with production (§1). Chakra traces replay only their recorded scale, ns-3 has no collective logic, GPGPU-Sim is too slow, and ASTRA-sim lacks precise GPU timing and NCCL's behavior (§2.3).

第 12.2 節（「模擬器多了什麼」）列出模擬器需要的東西：kernel 與通訊呼叫、把呼叫變成流的集體通訊演算法，以及承載這些流的網路。它警告：模擬出的 MFU 是比較，不是預測，因為模擬器依賴自己的計算與集體模型。SimAI 用量測來檢驗這個警告。在它之前，阿里流級的規劃與封包級的調校彼此不一致，也與生產環境不一致（§1）。Chakra 軌跡只能重播錄下它時的規模，ns-3 沒有集體通訊邏輯，GPGPU-Sim 太慢，ASTRA-sim 則缺少精確的 GPU 計時與 NCCL 的行為（§2.3）。

2. The problem, measured2. 問題，用量測說話

Alibaba's six-month task mix (Table 2) sets the scale. 35.29% of tasks used more than 1,024 GPUs, 41.10% trained 100B–500B models, and more than 94% were GPT-3 or LLaMA variants on Megatron or DeepSpeed (Figure 1). A useful simulator must handle more than 1,000 GPUs and follow new versions of two framework families.

阿里六個月的任務組成（Table 2）定下了規模。35.29% 的任務使用超過 1,024 顆 GPU，41.10% 訓練 100B–500B 的模型，超過 94% 是跑在 Megatron 或 DeepSpeed 上的 GPT-3 或 LLaMA 變體（圖 1）。有用的模擬器必須處理超過 1,000 顆 GPU，並跟上兩個框架家族的新版本。

Figure 1. The motivating measurements, redrawn: Alibaba's production task mix (① ②), ASTRA-sim's measured error with its own workload (③), and the measured run time of an early simulator against the real iteration (④).圖 1。重畫的動機量測：阿里生產環境的任務組成（① ②）、ASTRA-sim 使用自帶工作負載時的實測誤差（③），以及早期模擬器的實測執行時間對比真實的一次迭代（④）。Drawn for this primer from Wang et al., NSDI 2025, Table 2, §2.3, §3.5 and §6.1.

Existing tools failed on accuracy (ASTRA-sim 92.1–143.9% off, §6.1) and on speed (about a day per simulated GPT-3 iteration on 128 GPUs, §2.3). The paper's goals follow: an exact workload, NCCL-faithful collectives, precise compute across GPU generations, and more than 1,000 GPUs.

現有工具在精確度（ASTRA-sim 差 92.1–143.9%，§6.1）與速度（128 顆 GPU 上模擬 GPT-3 一次迭代約一天，§2.3）上都不行。論文的目標由此而來：精確的工作負載、忠於 NCCL 的集體操作、在各代 GPU 上都精確的計算，以及超過 1,000 顆 GPU。

3. The design3. 設計

SimAI splits an iteration into three fidelity problems: workload, compute and communication. It solves each with the cheapest method that is still exact or measured (the paper's Figure 1, below).

SimAI 把一次迭代拆成三個保真度問題：工作負載、計算與通訊。每個問題都用「仍然精確或來自實測」的方法中最便宜的那一種解決（論文的 Figure 1，見下圖）。

Block diagram: input description, workload generator, execution engine with computation simulator and communication simulator, simulation speedup — The paper's Figure 1, SimAI's architecture: the job and cluster description, the workload generator, and the execution engine with its computation and communication simulators.論文的 Figure 1，SimAI 的架構：任務與叢集的描述、工作負載產生器，以及含計算與通訊模擬器的執行引擎。Screenshot of Wang et al., NSDI 2025 (USENIX open-access PDF).

A discrete-event execution engine walks the dependency graph. It sends compute events to SimAI-CP and communication events through SimCCL to SimAI-CM. Figure 2 marks which parts run real code, look up measurements, or model.

離散事件的執行引擎（execution engine）走訪相依圖，把計算事件交給 SimAI-CP，把通訊事件經 SimCCL 交給 SimAI-CM。圖 2 標出哪些部分執行真實程式碼、查詢實測資料或屬於模型。

Figure 2. SimAI's pipeline, after the paper's Figure 1. Green parts run real software, blue looks up measurements, and only orange is a model. The packet-level network is the part that costs run time.圖 2。SimAI 的流程，依論文 Figure 1 重畫。綠色執行真實軟體，藍色查詢實測資料，只有橘色是模型。花掉執行時間的是封包級網路。Drawn for this primer from Wang et al., NSDI 2025, §3.1–§3.5 and Figure 1.

3.1 The workload comes from the real framework3.1 工作負載來自真實框架

SimAI-WG (the workload generator) "hijacks" Megatron or DeepSpeed. The framework runs on one host but is told that it runs on the target number of GPUs, and NCCL moves no data. It writes a workload file with every compute submodule (Table 1 marks memory-bound ones such as LayerNorm), every collective with its group (TP, DP, PP or EP) and size, and the dependencies, which the authors checked with Nsight Systems on 1,024 GPUs. Workload error disappears by construction: the list is what the framework does, not what a modeler believes it does, and a new framework version needs no new model.

SimAI-WG（工作負載產生器）「劫持」Megatron 或 DeepSpeed。框架在一台主機上執行，卻被告知自己跑在目標數量的 GPU 上，而 NCCL 不搬任何資料。它寫出一份工作負載檔，內含每個計算子模組（Table 1 標出受記憶體限制的，例如 LayerNorm）、每個集體操作及其群組（TP、DP、PP 或 EP）與大小，以及相依關係；作者在 1,024 顆 GPU 上用 Nsight Systems 檢查了這些相依關係。工作負載誤差依構造就消失了：清單就是框架真正做的事，而不是建模者以為它做的事，框架出新版也不必重建模型。

3.2 Kernel times are measured, and scaled only for GPUs that do not exist yet3.2 Kernel 時間來自實測，只有尚未上市的 GPU 才用縮放

SimAI-CP (the computation simulator) looks up each submodule in a database of measured runs per GPU type. On A100 with one parameter set, Embedding takes 3,425 µs and LayerNorm 715 µs (Table 4). A new GPU type needs one pass over all operations. For an unreleased GPU, SimAI-CP-Model scales from a similar known GPU. One factor for all kernels erred by up to 25.1%, so it scales per kernel class, following the roofline model: a compute-bound kernel (time $T_{\mathrm{c}}$ ) with peak compute $F$ , a memory-bound one ( $T_{\mathrm{m}}$ ) with memory bandwidth $B$ . A prime marks the new GPU, which must take less time if it is faster:

SimAI-CP（計算模擬器）從各型 GPU 實測結果組成的資料庫查出每個子模組的時間。在 A100 上用一組參數時，Embedding 花 3,425 µs，LayerNorm 花 715 µs（Table 4）。新的 GPU 型號只需把所有操作跑一遍。對尚未上市的 GPU，SimAI-CP-Model 從架構相近的已知 GPU 縮放。所有 kernel 用同一個因子，誤差最多 25.1%，所以它依循 roofline 模型按 kernel 類別縮放：受計算限制的 kernel（時間 $T_{\mathrm{c}}$ ）依峰值算力 $F$ ，受記憶體限制的（ $T_{\mathrm{m}}$ ）依記憶體頻寬 $B$ 。加撇號的是新 GPU；它若比較快，就必須花比較少的時間：

T'_{\mathrm{c}} = T_{\mathrm{c}} \times \frac{F}{F'}, \qquad T'_{\mathrm{m}} = T_{\mathrm{m}} \times \frac{B}{B'}

We corrected the direction of this formula. The paper prints both ratios as new over known, which would make a GPU with twice the bandwidth twice as slow. The method is otherwise the paper's. For a hypothetical GPU with twice the A100's memory bandwidth:

我們修正了這個公式的方向。論文把兩個比值都印成「新除以已知」，那會讓頻寬加倍的 GPU 慢一倍。除此之外，方法與論文相同。假設有一顆記憶體頻寬是 A100 兩倍的 GPU：

Question問題	Calculation算式	Result結果
LayerNorm, 715 µs on A100LayerNorm，A100 上 715 µs	$715 \times 1/2$	$\approx 358$ µs
Embedding, 3,425 µs on A100Embedding，A100 上 3,425 µs	$3{,}425 \times 1/2$	$\approx 1{,}713$ µs

On GPT-3 175B kernels the database missed by 0.5–3.1% (A100, H100, H20) and the roofline variant by 13–15%, most on the attention QKV and MLP linear kernels. So the roofline is only a fallback for GPUs that do not exist yet. Cycle-level GPU simulators were rejected as too slow.

在 GPT-3 175B 的 kernel 上，資料庫的誤差為 0.5–3.1%（A100、H100、H20），roofline 變體為 13–15%，以 attention QKV 與 MLP 線性層 kernel 最大。所以 roofline 只是尚未上市 GPU 的備案。週期級 GPU 模擬器因太慢而被排除。

3.3 Collectives are expanded by NCCL's own code3.3 集體操作由 NCCL 自己的程式碼展開

NCCL picks the algorithm, channels and relays of a collective from the node count, the message size and about 50 environment variables, and a hand-written model of those rules drifts with every release. SimCCL is NCCL itself, changed in 572 lines. A shim library intercepts communicator creation and builds virtual communicators that stand for every rank of a group. SimCCL reads the topology from a file, builds the channels, and turns each collective into transfers with size, sender, receiver and route, without moving data.

NCCL 依節點數、訊息大小與約 50 個環境變數，挑選集體操作的演算法、channel 與轉送，而手寫的規則模型會隨每一版偏離。SimCCL 就是 NCCL 本身，只改了 572 行。一個墊片函式庫攔截 communicator 的建立，建出代表群組中每個 rank 的虛擬 communicator。SimCCL 從檔案讀取拓樸、建立 channel，再把每個集體操作變成一串傳輸，記下大小、傳送端、接收端與路徑，而不搬資料。

Figure 3. The PXN example of the paper's §3.4 on two rail-optimized hosts. A model that emitted one flow 1 → 8 would load the wrong NIC and the wrong rail switch.圖 3。論文 §3.4 的 PXN 例子，畫在兩台 rail-optimized 主機上。若模型只送出一條 1 → 8 的流，就會讓錯的 NIC 與錯的 rail 交換器承受負載。Drawn for this primer from Wang et al., NSDI 2025, §3.4 and Figure 3.

PXN shows why this matters (Figure 3; collectives Section 12.2). With NCCL_P2P_PXN_LEVEL set, NCCL relays rank 1's data to rank 8 over NVLink through rank 0, whose NIC is on rank 8's rail. SimCCL therefore emits 1→0 on NVLink and 0→8 on rail 0. Adaptive routing and CollNet/SHARP are not supported (Table 6), and expert parallelism assumes balanced gates.

PXN 說明了這為何重要（圖 3；集體通訊第 12.2 節）。設定 NCCL_P2P_PXN_LEVEL 時，NCCL 把 rank 1 給 rank 8 的資料經 NVLink 轉給 rank 0，而 rank 0 的 NIC 在 rank 8 的 rail 上。因此 SimCCL 送出 NVLink 上的 1→0 與 rail 0 上的 0→8。adaptive routing 與 CollNet/SHARP 不支援（Table 6），專家平行則假設 gate 是均衡的。

3.4 Packets in ns-3, with threads that share no locks3.4 在 ns-3 中模擬封包，執行緒之間不共用鎖

SimAI-CM (the communication simulator) replays the flows in ns-3's packet-level RDMA model, so queues, PFC, ECN and DCQCN or HPCC act on every packet. It is the only level that reproduces hash polarization or a victim flow, and the expensive one. The authors parallelized ns-3 with UNISON, which partitions the network over threads automatically. Threads still serialized on locks around global state such as switch queue lengths. SimAI moves that state into a table indexed by node ID. Each node runs on one thread, so each row has one writer and needs no lock (Figure 4). That gave 23× over one thread and 15% over locked threads.

SimAI-CM（通訊模擬器）在 ns-3 的封包級 RDMA 模型中重播這些流，讓佇列、PFC、ECN 與 DCQCN 或 HPCC 作用在每個封包上。這是唯一能重現 hash polarization 或受害者流的層級，也是昂貴的那一個。作者用會自動把網路分給各執行緒的 UNISON 把 ns-3 平行化。但執行緒仍在包住全域狀態（例如交換器佇列長度）的鎖上排隊。SimAI 把這些狀態移到以節點 ID 為索引的表。每個節點只在一個執行緒上執行，所以每列只有一個寫入者，不需要鎖（圖 4）。這比單執行緒快 23 倍，比有鎖的多執行緒快 15%。

Figure 4. The lock-free state of the paper's §3.5. Left: threads queue at locks around shared simulator state. Right: each node's row has one owner thread, which the paper measured as 23× faster than one thread.圖 4。論文 §3.5 的無鎖狀態。左：各執行緒在包住共用模擬狀態的鎖前排隊。右：每個節點的那一列只屬於一個執行緒，論文實測比單執行緒快 23 倍。Drawn for this primer from Wang et al., NSDI 2025, §3.5.

The paper states neither the host nor the thread count of the 23× measurement. It reports no absolute run time for the final simulator at any scale, so a reader cannot tell how many design points fit in a day.

論文沒有說明 23 倍這項量測的主機與執行緒數，也沒有報告最終版模擬器在任何規模下的絕對執行時間，所以讀者無法得知一天能跑多少個設計點。

4. Implementation4. 實作

SimAI hooks CUDA functions, so simulations run on hosts without GPUs; only the kernel database needs a GPU, once per type. Alibaba runs it as a Kubernetes service, and the code is open source (aliyun/SimAI). The building blocks took more than 10,000 lines of porting code. Production runs fluctuated by at most 5%, so averaged parameters are used (§6). Re-implementing the frameworks by hand was dropped as months of hard-to-maintain work.

SimAI 攔截 CUDA 函式，所以模擬可在沒有 GPU 的主機上執行；只有 kernel 資料庫需要 GPU，每種一次。阿里把它做成 Kubernetes 服務，程式碼已開源（aliyun/SimAI）。各組件共寫了超過 10,000 行移植程式碼。生產環境的執行波動最多 5%，所以使用平均後的參數（§6）。手動重寫框架則因為要花好幾個月、又難以維護而放棄。

5. What was measured5. 量到了什麼

Ground truth is real training with matching settings on two clusters of 128 hosts × 8 GPUs: A100 hosts with four 2×100 Gb/s NICs and 600 GB/s NVLink, and H100 hosts with eight 2×200 Gb/s NICs and 900 GB/s NVLink, both on multi-rail fat-tree RoCEv2 like HPN. Jobs ran GPT-3 13B, LLaMA 65B and GPT-3 175B at 128–1,024 GPUs; H20 joined the kernel tests. The baseline, ASTRA-sim, reads SimAI-WG's workload files. Its compute comes from SCALE-sim matrix kernels scaled by an NVIDIA-to-TPU FLOPS ratio, and it models no runtime software or NIC pipeline. Metrics: bus bandwidth (collectives Chapter 11), kernel time and iteration time.

基準真值是在兩座 128 台主機 × 8 顆 GPU 的叢集上、以相同設定進行的真實訓練：A100 主機有四張 2×100 Gb/s NIC 與 600 GB/s NVLink，H100 主機有八張 2×200 Gb/s NIC 與 900 GB/s NVLink，兩者都是類似 HPN 的多 rail fat-tree RoCEv2。任務在 128–1,024 顆 GPU 上執行 GPT-3 13B、LLaMA 65B 與 GPT-3 175B；kernel 測試另加入 H20。比較基準 ASTRA-sim 讀取 SimAI-WG 的工作負載檔。它的計算來自以 NVIDIA 對 TPU 的 FLOPS 比例縮放的 SCALE-sim 矩陣 kernel，也不模擬執行期軟體與 NIC 管線。指標：bus bandwidth（集體通訊第 11 章）、kernel 時間與迭代時間。

Test (paper figure)測試（論文圖）	Setup設定	SimAISimAI	ASTRA-simASTRA-sim
Collectives inside a host (Fig. 6)主機內的集體操作（Fig. 6）	8 MB–8 GB, average deviation of bus bandwidth8 MB–8 GB，bus bandwidth 的平均偏差	3.9% A100 · 2.3% H100	74.8% A100 · 51.7% H100
AllGather across hosts (Fig. 7)跨主機的 AllGather（Fig. 7）	8 MB messages, A1008 MB 訊息，A100	close to the real run; values only in the figure貼近真實；數值只在圖中	45.9% at 128 GPUs · 530.2% at 512128 顆 GPU 45.9% · 512 顆 530.2%
Kernels (Fig. 8)Kernel（Fig. 8）	GPT-3 175B, A100 · H100 · H20GPT-3 175B，A100 · H100 · H20	0.5–3.1% (CP-Model: 13–15%)0.5–3.1%（CP-Model：13–15%）	117.9% · 49.8% · 224%
Iteration time (Fig. 9)迭代時間（Fig. 9）	GPT-3 13B, LLaMA 65B, GPT-3 175B; 128–1,024 GPUsGPT-3 13B、LLaMA 65B、GPT-3 175B；128–1,024 顆 GPU	< 3.9%; < 4% at 1,024< 3.9%；1,024 顆時 < 4%	36.1× SimAI's errorSimAI 誤差的 36.1 倍

Six panels of stacked bars of iteration time for three models, comparing AstraSim, SimAI and ground truth on H100 and A100 clusters — The paper's Figure 9: iteration time on the H100 (top) and A100 (bottom) clusters. Ground-truth bars are measured, the others predicted; ASTRA-sim's excess sits mostly in computation.論文的 Figure 9：H100（上）與 A100（下）叢集的迭代時間。Groundtruth 長條是實測值，其餘是預測；ASTRA-sim 多出的部分大多在計算。Screenshot of Wang et al., NSDI 2025 (USENIX open-access PDF).

Over all scenarios SimAI matched reality to 98.1% on average. Precision rose from GPT-3 13B to 175B because larger collective messages are simulated more precisely. The "36.1×" is a ratio of two errors; ASTRA-sim's own end-to-end error is not in the text. Section 5 uses SimAI to size NICs and tensor parallelism for future hosts; those are simulated what-ifs, so they are not reported here. The panel lays Figure 2's parts out as rungs: switch one to a coarser method and see which error source opens.

在所有情境平均下，SimAI 與真實結果吻合 98.1%。精確度從 GPT-3 13B 到 175B 逐步提升，因為較大的集體操作訊息模擬得更精確。「36.1 倍」是兩個誤差的比值；正文沒有給出 ASTRA-sim 本身的端到端誤差。第 5 節用 SimAI 為未來的主機規劃 NIC 與張量平行度；那些是模擬出的假設情境，所以這裡不報告。下面的面板把圖 2 的各部分排成階梯：把某一階換成較粗略的方法，看看會打開哪一種誤差來源。

Panel 1面板 1

Accuracy-vs-cost ladder: switch one rung to a coarser method精確度與成本的階梯：把某一階換成較粗略的方法

① Workload① 工作負載

② Kernel times② Kernel 時間

③ Collective expansion③ 集體操作展開

④ Network④ 網路

GPU of the measurement量測所用的 GPU

Error sources left open

仍未消除的誤差來源

Nearest measured whole run

最接近的整體實測結果

Run-time cost

執行時間成本

What you must obtain

需要先取得的東西

Bars: measured deviation from the real cluster, on a log scale. Outline: the full-fidelity rung (SimAI). Filled: the method you chose. Every number is the paper's measurement. The workload bar (92.1–143.9%) is a whole ASTRA-sim run with its own workload (§6.1). Kernel bars are GPT-3 175B kernels (Fig. 8); 25.1% is the paper's "up to" for one-factor scaling. Collective bars are the in-host average over 8 MB–8 GB (Fig. 6); they measure the whole communication stack, expansion and network together. The paper measured no flow-level back end and no H20 collectives, so those rungs have no bar. The run-time tile is qualitative: the paper gives only "more than 24 hours" for an early version and 23× from lock-free threads.

長條：相對真實叢集的實測偏差，對數尺度。空心框：完整保真度的那一階（SimAI）。實心：你選的方法。每個數字都是論文的量測。工作負載長條（92.1–143.9%）是 ASTRA-sim 用自帶工作負載的整體執行（§6.1）。kernel 長條是 GPT-3 175B 的 kernel（Fig. 8）；25.1% 是論文給單一因子縮放的「最多」值。集體操作長條是 8 MB–8 GB 的主機內平均（Fig. 6），量的是整個通訊堆疊，展開與網路一起。論文沒有量測流級後端，也沒有量測 H20 的集體操作，所以那些階沒有長條。執行時間這一格是定性的：論文只給早期版本「超過 24 小時」，以及無鎖多執行緒的 23 倍。

Stop and think停下來想一想

Set kernels to SCALE-sim and collectives to the generic model: this is the paper's ASTRA-sim baseline. Now set kernels back to "Measured". On A100, which open error source is largest, and why is there no measured whole run to compare with?

把 kernel 切到 SCALE-sim、集體操作切到通用模型：這就是論文的 ASTRA-sim 基準。現在把 kernel 切回「實測」。在 A100 上，哪一個仍未消除的誤差來源最大？為什麼沒有可以比較的整體實測結果？

6. Limits and open questions6. 限制與未解的問題

The paper's own gaps: no adaptive routing or SHARP, balanced expert gates only, re-profiling for new workloads, 13–15% kernel error for modeled GPUs, a SimCCL re-port per NCCL version, and no fault simulation. A careful reader adds:

論文自己的缺口：不支援 adaptive routing 與 SHARP、只支援均衡的專家 gate、新工作負載需重新剖析、以模型推估 GPU 時 kernel 誤差 13–15%、每版 NCCL 都要重新移植 SimCCL，以及沒有故障模擬。細心的讀者還會補上：

One tenant. Every validation ran one job on a dedicated cluster; cross-job collisions and PFC coupling were not validated.單一租戶。所有驗證都是一個任務獨占叢集；跨任務碰撞與 PFC 耦合沒有驗證。
Iteration time, not flows. Per-flow completion times and queues under congestion were not compared with hardware.迭代時間，不是流。壅塞下逐流的完成時間與佇列沒有和硬體比較。
Scale. Two 128-host rail-optimized fat-trees; oversubscribed or multi-pod fabrics are extrapolation.規模。兩座 128 台主機的 rail-optimized fat-tree；超額認購或多 pod 的網路屬於外推。
Run time. No absolute figure. The Wormhole paper later reports 9 hours of ns-3 time for a GPT-13B workload on 128 GPUs.執行時間。沒有絕對數字。後來的 Wormhole 論文報告：128 顆 GPU 上的 GPT-13B 工作負載要 9 小時的 ns-3 時間。
Baseline setup. ASTRA-sim's compute setup was the authors' choice; part of its error reflects it.基準設定。ASTRA-sim 的計算設定是作者選的；它的誤差有一部分反映這一點。

7. How it connects7. 與其他內容的關聯

Chapter 11, section 4: for one job up to 1,024 GPUs, "a comparison, not a prediction" can be sharpened to "under 4% error, measured". Tenants and failures remain unvalidated.第 11 章第 4 節：對最多 1,024 顆 GPU 的單一任務，「是比較，不是預測」可以說得更精確：「實測誤差低於 4%」。租戶與故障仍未驗證。
Chapter 4: SimCCL reproduces PXN (section 4), NCCL's choices (section 5) and bus bandwidth (section 8). Chapter 7: the packet network runs DCQCN and HPCC.第 4 章：SimCCL 重現 PXN（第 4 節）、NCCL 的選擇（第 5 節）與 bus bandwidth（第 8 節）。第 7 章：封包網路執行 DCQCN 與 HPCC。
Wormhole (NSDI 2026, two shared authors) removes the remaining cost, packet events. Crux, LEVELLER and MonkeyTree evaluate at scale with $\alpha$ – $\beta$ or flow-level models, the panel's coarser rungs.Wormhole（NSDI 2026，有兩位共同作者）去除剩下的成本：封包事件。Crux、LEVELLER 與 MonkeyTree 用 $\alpha$ – $\beta$ 或流級模型做大規模評估，也就是面板中較粗略的幾階。
Guided readings: HPN is the same team's fabric; Meta RoCE's 22 µs RTT against NCCL's assumed 10 µs is an effect only a simulator running NCCL's logic sees.論文導讀：HPN 是同一團隊的網路；Meta RoCE 的 22 µs RTT 對比 NCCL 假設的 10 µs，只有執行 NCCL 邏輯的模擬器看得到。

Key numbers關鍵數字

Number	數字	Meaning (setup)
35.29%	Alibaba tasks on more than 1,024 GPUs (Table 2)	阿里使用超過 1,024 顆 GPU 的任務（Table 2）
92.1–143.9%	ASTRA-sim error with its built-in workload (§6.1)	ASTRA-sim 用內建工作負載時的誤差（§6.1）
> 24 h vs 2 s	Early simulator vs real cluster, one GPT-3 iteration, 128 GPUs	早期模擬器對真實叢集，GPT-3 一次迭代，128 顆 GPU
23× · +15%	Lock-free ns-3 threads vs one thread · vs locked threads	無鎖 ns-3 多執行緒對單執行緒 · 對有鎖多執行緒
3.9% · 2.3%	In-host collective deviation, A100 · H100 (ASTRA-sim 74.8% · 51.7%)	主機內集體操作偏差，A100 · H100（ASTRA-sim 74.8% · 51.7%）
0.5–3.1% · 13–15%	Kernel error, measured database · roofline variant	kernel 誤差，實測資料庫 · roofline 變體
< 3.9% · < 4% · 98.1%	Iteration-time deviation · at 1,024 GPUs · average alignment	迭代時間偏差 · 1,024 顆 GPU 時 · 平均吻合度

Terms used on this page本頁用到的術語

SimAI-WG: The real training framework on one host with a faked world size; it records kernels, collectives and dependencies.; 在單一主機上以假造的總規模執行的真實訓練框架；記錄 kernel、集體操作與相依關係。
SimAI-CP / CP-Model: Kernel times from measurements, or roofline scaling for a GPU that cannot be measured.; kernel 時間取自實測，或對無法量測的 GPU 用 roofline 縮放。
SimCCL: NCCL changed in 572 lines to plan collectives with virtual communicators and emit flows instead of moving data.; 修改 572 行的 NCCL，用虛擬 communicator 規劃集體操作，送出流而不搬資料。
SimAI-CM: Packet-level RDMA network in ns-3, parallelized with UNISON and lock-free per-node state.; ns-3 中的封包級 RDMA 網路，以 UNISON 與無鎖的逐節點狀態平行化。