Parallelism research: when ranks are not equal平行化研究：當各 rank 並不相同
Recent work on uneven pipelines, expert placement and long sequences, and what each does to the traffic.關於不均勻管線、專家擺放與長序列的近期研究，以及它們各自對流量的影響。
This page assumes Chapters 1–6. Every term it uses is defined there; the glossary on the contents page links to each definition. The papers here are recent and most of their results come from small testbeds or simulation, so read each number with its label.
本頁假設你已讀過第 1–6 章。這裡用到的每個名詞都在那些章節中定義過；目錄頁的詞彙表連到每個定義。這些論文都很新，多數結果來自小型測試平台或模擬，所以請連同標籤一起讀每個數字。
Terms from later chapters used here (3)本頁用到的後面章節名詞（3 個）
- ECMPECMP Ch. 8 Equal-cost multi-path: a switch picks one of several equal paths by hashing each packet's header.等價多路徑：交換器以封包標頭的雜湊值在多條等價路徑中選一條。
- Packet spraying, reordering封包噴灑、亂序 Ch. 8 Sending consecutive packets of one flow over different paths; they may arrive out of order, so the NIC must accept that.把同一條流的連續封包送到不同路徑；封包可能亂序抵達，NIC 必須能接受。
- Placement, scheduler放置、排程器 Ch. 11 Choosing which GPUs a job gets and which rank runs where; it decides which links the traffic crosses.決定工作拿到哪些 GPU、各 rank 放在哪裡；它決定流量經過哪些鏈路。
Recent research at SIGCOMM, NSDI, ISCA and EuroSysSIGCOMM、NSDI、ISCA 與 EuroSys 的近期研究
Every formula on this page assumes that each rank does the same work at the same time, over a healthy link. Six recent papers measure what happens when that is false. In each one, a single piece becomes the slowest: a slow pipeline link, a long sequence, the NIC that EP must cross, a hot expert and the NIC behind it, or a barrier. A synchronous step then runs at the speed of that piece. The map below places the papers by parallelism and by workload. Four are training papers, and two of those describe how production MoE models are trained. Two study the expert-parallel all-to-all of MoE serving, which training runs too.
本頁每個公式都假設：每個 rank 在同一時間做同樣多的工作，而且鏈路健康。六篇近期論文量測了這個假設不成立時會發生什麼。在每一篇中，都有一個部件變成最慢的：一條慢的管線鏈路、一條長序列、EP 必須跨越的 NIC、一個熱門專家與它背後的 NIC，或是一道 barrier。同步的步驟於是以那個部件的速度前進。下圖依平行化維度與工作負載標出這些論文的位置。四篇是訓練論文，其中兩篇描述生產環境的 MoE 模型怎麼訓練；兩篇研究的是 MoE 服務中的專家平行 all-to-all，而訓練也跑同樣的 all-to-all。
Attack of the Bubbles (NSDI 2026) adds one variable to the bubble of Section 3.5: link delay. A delay on the link after a stage adds only about to the iteration while . Here the slackness is the difference between the warm-up forward counts of neighbouring stages, and is the time of one operation. Beyond that bound, bubbles cascade through the pipeline. The paper's system, PipeMorph, re-plans the warm-up counts so that the slow link gets more slack. It also moves that link's sends to CPU processes over host RDMA, so a stuck send can no longer block the launch of compute kernels.
Attack of the Bubbles（NSDI 2026）為第 3.5 節的氣泡加上一個變數：鏈路延遲。某個階段之後的鏈路若有延遲 ，只要 ，迭代時間就只增加約 。其中 slackness 是相鄰階段暖身正向次數之差， 是一個運算的時間。超過這個界限，氣泡就會在管線中連鎖擴散。論文的系統 PipeMorph 重新規劃暖身次數，讓慢的鏈路得到更多 slack。它也把那條鏈路的傳送移到經主機 RDMA 傳送的 CPU 行程，讓卡住的傳送不再擋住計算 kernel 的啟動。
ByteScale (SIGCOMM 2025) starts from the context parallelism of Section 3.6, which this page treats as one fixed degree per job. In long-context training, sequence lengths are highly skewed. A CP degree sized for the longest sequence makes short sequences pay ring traffic that they do not need. Micro-batches with equal token counts also carry unequal attention work, because attention grows with the square of each sequence's length, so DP ranks and pipelines wait for each other. ByteScale merges DP and CP into one pool of ranks that hold equal numbers of tokens. Each sequence gets only the ranks its length needs, so short sequences send no ring traffic, and all ranks reuse one NCCL communicator for their point-to-point sends. A balance scheduler then gives lighter ranks and pipelines more micro-batches, which a token-level loss keeps exact. On ByteDance's production cluster (hardware undisclosed), throughput rose up to 7.89× over a static DP × CP mesh, for LLaMA-7B at 2M tokens on 1,024 GPUs. On all 16,384 GPUs (Mistral-8×22B, 2M tokens) it rose 2.41× on production data.
ByteScale（SIGCOMM 2025）的起點是第 3.6 節的上下文平行，本頁把它當成每個任務一個固定的度數。在長上下文訓練中，序列長度高度偏斜。依最長序列決定的 CP 度數，讓短序列付出它們不需要的環狀流量。token 數相同的 micro-batch 也帶著不等量的 attention 工作，因為 attention 隨每條序列長度的平方成長，於是 DP rank 與各條管線互相等待。ByteScale 把 DP 與 CP 合併成一個 rank 池，每個 rank 持有相同數量的 token。每條序列只拿到它的長度需要的 rank，所以短序列不產生環狀流量，所有 rank 也重用同一個 NCCL communicator 做點對點傳送。平衡排程器再讓較輕的 rank 與管線多跑幾個 micro-batch，而 token 層級的損失讓結果保持精確。在字節跳動的生產叢集上（硬體未公開），LLaMA-7B 在 2M token、1,024 顆 GPU 時，吞吐比靜態 DP × CP 網格最高提升 7.89 倍。在全部 16,384 顆 GPU 上（Mistral-8×22B、2M token），生產資料的吞吐提升 2.41 倍。
Insights into DeepSeek-V3 (ISCA 2025) and MegaScale-MoE (EuroSys 2026, ByteDance) are the training side of EP, and Section 3.8 of this page is built on them. In both, the slow piece is the NIC: on DeepSeek-V3's H800 nodes, NVLink is only about 4 times faster than one 400 Gb/s NIC. DeepSeek-V3 keeps EP across 8 nodes but sends each token to at most 4 of them, once per node over InfiniBand, and then copies it over NVLink; dispatch goes in FP8. Its eight network planes share no switch, and the full 2,048-GPU job trained at 272.80 billion tokens per day, against 272.52 billion on a re-wired network whose rails meet in shared spines. MegaScale-MoE keeps EP, and Ulysses-style sequence parallelism for attention, inside one 8-GPU node. It uses only PP and DP across nodes and fuses each all-to-all with its GEMM. On a 352B model it reached 1.65–1.88× Megatron-LM's throughput on 240 to 1,440 H800 GPUs.
Insights into DeepSeek-V3（ISCA 2025）與 MegaScale-MoE（EuroSys 2026，字節跳動）是 EP 的訓練面，本頁第 3.8 節就建立在它們之上。兩篇中最慢的部件都是 NIC：在 DeepSeek-V3 的 H800 節點上，NVLink 只比一張 400 Gb/s NIC 快約 4 倍。DeepSeek-V3 讓 EP 跨 8 個節點，但每個 token 最多只送往其中 4 個，每個節點經 InfiniBand 一次，再由 NVLink 複製；分派以 FP8 傳送。它的八個網路平面不共用交換器，完整的 2,048 顆 GPU 任務每天訓練 2,728.0 億個 token，而重新接線成各 rail 在共用 spine 交會的網路時是 2,725.2 億個。MegaScale-MoE 把 EP 以及注意力用的 Ulysses 式序列平行留在一個 8-GPU 節點內。它跨節點只用 PP 與 DP，並把每個 all-to-all 與對應的 GEMM 融合。在 352B 模型上，它在 240 到 1,440 顆 H800 上的吞吐是 Megatron-LM 的 1.65–1.88 倍。
Balancing and Beyond (SIGCOMM 2026, Alibaba Cloud) measures EP on scale-out H20 and H800 clusters. In a sampled production layer the busiest expert received about 8× the median expert's tokens. The paper balances first, with NIC-aware expert placement and per-step expert swaps over NVLink. Then it sends one copy per remote host through the same-index GPU, which keeps the traffic on its rail. UBEP (SIGCOMM 2026, Nanjing University and Huawei) puts the whole EP group inside one CloudMatrix384 superpod, where a load/store fabric replaces the NIC. There, bytes are cheap, and barriers and hop distance set the time: a two-hop access costs 11.5× a local one. UBEP runs sending and counting on separate groups of cores at once, carries the arrival signal inside 512-byte atomic writes, and gives every core the same mix of near and far destinations.
Balancing and Beyond（SIGCOMM 2026，阿里雲）在 scale-out 的 H20 與 H800 叢集上量測 EP。在一個抽樣的生產層中，最忙的專家收到的 token 約是中位數專家的 8 倍。論文先做平衡：考慮 NIC 的專家配置，以及每一步經 NVLink 交換專家。接著經由同編號的 GPU，每台遠端主機只送一份複本，讓流量留在自己的 rail 上。UBEP（SIGCOMM 2026，南京大學與華為）把整個 EP 群組放進一個 CloudMatrix384 superpod，在那裡以 load/store 網路取代 NIC。在那裡位元組很便宜，決定耗時的是 barrier 與跳數：跳兩次的存取是本地存取的 11.5 倍。UBEP 讓傳送與計數由不同的核心群組同時執行，把到達訊號放進 512 位元組的原子寫入中，並讓每個核心分到相同比例的近端與遠端目的地。
| Paper and problem論文與問題 | Mechanism機制 | Measured result實測結果 |
|---|---|---|
| Attack of the Bubbles · NSDI 2026 A slow PP link breaks the schedule once its delay exceeds the slack; a stuck NCCL send blocks compute慢的 PP 鏈路一旦延遲超過 slack 就打亂排程；卡住的 NCCL 傳送擋住計算 | More slack on the slow link; its sends move to CPU processes over host RDMA給慢的鏈路更多 slack；它的傳送改由 CPU 行程經主機 RDMA 進行 | 1.36× ZeroBubble's throughput on 128 H800 GPUs, while a one-hour production delay trace was replayed as injected delays在 128 顆 H800 GPU 上，以注入延遲重播一小時的生產延遲紀錄時，吞吐為 ZeroBubble 的 1.36 倍 |
| ByteScale · SIGCOMM 2025 Skewed sequence lengths: a fixed CP degree wastes ring traffic on short sequences, and unequal attention work leaves ranks waiting序列長度偏斜：固定的 CP 度數在短序列上浪費環狀流量，不均的 attention 工作讓 rank 互相等待 | One pool of DP × CP ranks with equal tokens; each sequence gets only the ranks it needs over one NCCL communicator; unequal micro-batch counts per rank一個 token 數相等的 DP × CP rank 池；每條序列經同一個 NCCL communicator 只用它需要的 rank；各 rank 的 micro-batch 數不相等 | Up to 7.89× the static mesh's throughput (LLaMA-7B, 2M tokens, 1,024 GPUs, open data); 2.41× on production data at 16,384 GPUs (Mistral-8×22B); production cluster, hardware undisclosed吞吐最高為靜態網格的 7.89 倍（LLaMA-7B、2M token、1,024 顆 GPU、公開資料）；16,384 顆 GPU 的生產資料上為 2.41 倍（Mistral-8×22B）；生產叢集，硬體未公開 |
| Insights into DeepSeek-V3 · ISCA 2025 EP over 8 nodes on H800, whose NVLink is only about 4× one NIC在 H800 上跨 8 個節點做 EP，而它的 NVLink 只有一張 NIC 的約 4 倍 | At most 4 nodes per token, one IB copy per node, then NVLink forwarding; FP8 dispatch; eight planes that share no switch每個 token 最多 4 個節點，每節點一份 IB 複本，再經 NVLink 轉送；FP8 分派；八個不共用交換器的平面 | 272.80 vs 272.52 billion tokens/day on the multi-plane vs a shared-spine network (full 2,048-GPU training); DeepEP 42–58 GB/s per GPU完整 2,048 顆 GPU 的訓練：多平面網路每天 2,728.0 億個 token，共用 spine 的網路為 2,725.2 億；DeepEP 每顆 GPU 42–58 GB/s |
| MegaScale-MoE · EuroSys 2026 TP or EP across nodes puts traffic on the network in every layer跨節點的 TP 或 EP 讓每一層都有流量上網路 | EP and SP inside the node, only PP and DP across nodes; whole-layer scheduling and fused all-to-all + GEMM kernelsEP 與 SP 留在節點內，跨節點只用 PP 與 DP；整層排程，以及 all-to-all 與 GEMM 的融合 kernel | 1.65–1.88× Megatron-LM's throughput (352B model, 240–1,440 H800); 1.28× from SP + EP and both overlaps on 240 GPUs吞吐為 Megatron-LM 的 1.65–1.88 倍（352B 模型，240–1,440 顆 H800）；在 240 顆 GPU 上 SP + EP 加上兩種重疊為 1.28 倍 |
| Balancing and Beyond · SIGCOMM 2026 Hot experts and the NICs they share become stragglers; one copy per expert wastes NIC bytes (MoE serving)熱門專家與它們共用的 NIC 成為 straggler；每個專家一份複本浪費 NIC 位元組（MoE 服務） | NIC-aware placement, per-step swaps, a same-index relay with one copy per host, per-expert overlap考慮 NIC 的配置、逐步交換、經同編號 GPU 轉送且每台主機一份、逐專家重疊 | Communication 190.21 → 134.00 µs against DeepEP's default (EP32, H20)通訊時間 190.21 → 134.00 µs，相對 DeepEP 預設（EP32，H20） |
| UBEP · SIGCOMM 2026 Inside a superpod, barriers and far hops, not bytes, set the dispatch time (MoE serving)在 superpod 內，決定分派時間的是 barrier 與遠的跳數，而非位元組（MoE 服務） | Concurrent core groups, the arrival flag inside 512-byte atomic writes, an equal hop mix per core同時運作的核心群組、放在 512 位元組原子寫入中的到達旗標、每個核心相同的跳數組合 | Dispatch latency up to 52.4% below Huawei's CANN EP (64 NPUs); P99 time per output token up to 11.1% lower分派延遲比華為的 CANN EP 最多低 52.4%（64 顆 NPU）；P99 每個輸出 token 的時間最多低 11.1% |
What they agree on. First, a policy that is fixed before the data or the network is seen is wrong. Three of the papers end with a choice per workload: warm-up counts per link, a transport and overlap table per batch size, a core split that follows the expert count and the batch. Second, the slowest piece, not the mean, must be measured and attacked. This is the order-statistics view of Section 10.6, now caused by data and paths instead of sick machines. Third, network-layer fixes alone are not enough. PipeMorph argues that ECMP and packet spraying cannot see pipeline dependencies; the EP library finds that a better transport pays only after balancing; UBEP finds that more bandwidth does not cure a phase bound by synchronization. Two papers also keep communication off the GPU's compute units: CPU-side sends in PipeMorph, SM-free GPU-initiated RDMA in the EP library. DeepSeek-V3 spends up to 20 SMs per GPU on it and asks for hardware to take that work over.
它們的共識。第一，在看到資料或網路之前就固定的策略是錯的。其中三篇最後都按工作負載做選擇：每條鏈路的暖身次數、每個批次大小的傳輸與重疊表、隨專家數與批次調整的核心分配。第二，必須量測並對付的是最慢的部件，而不是平均。這就是第 10.6 節的順序統計觀點，只是原因從生病的機器換成了資料與路徑。第三，只在網路層修正並不夠。PipeMorph 主張 ECMP 與封包噴灑看不到管線的相依關係；EP 函式庫發現更好的傳輸要在平衡之後才有回報；UBEP 發現更多頻寬治不了受同步限制的階段。其中兩篇也讓通訊不佔用 GPU 的計算單元：PipeMorph 改由 CPU 傳送，EP 函式庫使用不佔 SM、由 GPU 發起的 RDMA。DeepSeek-V3 每顆 GPU 最多花 20 個 SM 在通訊上，並要求由硬體接手這份工作。
Where they differ. The two serving papers are about inference. Their small decode batches make expert GEMMs bound by memory bandwidth, so their overlap verdicts may not hold for training micro-batches; their lessons on skew, duplicate bytes and synchronization do. They also answer this page's "EP breaks the rail assumption" in opposite ways. One keeps the scale-out fabric and stays on a rail by relaying through the same-index GPU. The other puts the EP group inside one scale-up domain, where rails do not exist. The two training papers split the same way: DeepSeek-V3 keeps EP on the scale-out fabric and stays on its plane by forwarding over NVLink, and MegaScale-MoE keeps EP inside the node. The evidence differs as well: injected delays on 128 H800 GPUs, production claims plus H20 and H800 runs, one partition of a production superpod, a full production training job on 2,048 H800 GPUs, and runs on 240 to 1,440 H800 GPUs.
它們的分歧。兩篇服務論文談的是推論。它們的 decode 批次很小，使專家 GEMM 受記憶體頻寬限制，所以它們對重疊的結論不一定適用於訓練的 micro-batch；但關於偏斜、重複位元組與同步的教訓仍然適用。它們也以相反的方式回應本頁的「EP 打破 rail 假設」。一篇保留 scale-out 網路，經同編號 GPU 轉送而留在 rail 上。另一篇把 EP 群組放進一個 scale-up 域，那裡根本沒有 rail。兩篇訓練論文也以同樣的方式分歧：DeepSeek-V3 讓 EP 留在 scale-out 網路上，經 NVLink 轉送而留在自己的平面上；MegaScale-MoE 則把 EP 留在節點內。證據也不同：在 128 顆 H800 GPU 上注入延遲、生產環境的宣稱加上 H20 與 H800 的實驗、生產 superpod 中的一個分區、2,048 顆 H800 上一個完整的生產訓練任務，以及 240 到 1,440 顆 H800 上的實驗。