Research digest 4 of 6 · Chapter 3: Splitting the model

研究導讀 4 / 6 · 第 3 章：切分模型

UBEP: Re-architecting Expert Parallelism Communication Library for Production SuperpodsUBEP：為生產環境 superpod 重新架構專家平行通訊函式庫

SIGCOMM 2026Yipeng Liu, Chang Liu, Si Shen et al. (Nanjing University, Huawei) · paper論文

After reading, you can explain why MoE dispatch inside a superpod is limited by barriers and hop distance rather than bandwidth, and how UBEP splits cores, balances hops and fuses flags into data.讀完後，你能說明為何 superpod 內的 MoE dispatch 受限於 barrier 與跳數距離而非頻寬，以及 UBEP 如何拆分核心、平衡跳數，並把旗標融入資料。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 3 ↗第 3 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

UBEP is a communication library for expert parallelism (EP) in mixture-of-experts (MoE) inference on Huawei's CloudMatrix384 (CM384), a superpod of 384 NPUs on one load/store fabric, the Unified Bus (UB). At 392 GB/s per link, bytes are cheap, and the software around them sets the time. Libraries built for InfiniBand clusters run dispatch as phases split by global barriers. One barrier takes about 15% of the time, and half of the vector cores already fill the link while the rest wait. Equal token counts also do not give equal times, because a two-hop access costs 11.5× a local one. UBEP runs a sending group and a counting group of cores at once, gives each core the same mix of near and far tokens, and carries the arrival signal inside each 512-byte write. On a production CM384, dispatch latency fell by up to 52.4% (at 64 NPUs) against Huawei's CANN EP, and P99 time per output token by up to 11.1%.

UBEP 是一個通訊函式庫，用於華為 CloudMatrix384（CM384）上混合專家（MoE）推論的專家平行（EP）；CM384 是一個 superpod，384 顆 NPU 共用一個 load/store 網路，即 Unified Bus（UB）。每條鏈路有 392 GB/s，位元組很便宜，決定時間的是位元組周圍的軟體。為 InfiniBand 叢集打造的函式庫把 dispatch 分成以全域 barrier 隔開的階段。一次 barrier 約占 15% 的時間，而一半的向量核心就已填滿鏈路，其餘只能等待。相同的 token 數也不代表相同的時間，因為兩跳存取的代價是本地的 11.5 倍。UBEP 讓一組傳送核心與一組計數核心同時執行，給每個核心相同的遠近 token 組合，並把「資料已到」的訊號放進每一次 512 位元組的寫入。在生產環境的 CM384 上，dispatch 延遲比華為的 CANN EP 最多降低 52.4%（64 顆 NPU 時），P99 每個輸出 token 的時間最多降低 11.1%。

1. Where this paper sits1. 這篇論文的位置

Section 3.7 sizes the EP all-to-all at $\text{tokens} \times k \times h \times 2\,\text{B}$ per MoE layer, with destinations set token by token. Those tokens go to GPUs of any rank, so EP breaks the rail assumption, and a rail-only network pays 8.2–11.2% for the NVLink detour. A superpod answers the other way. It makes the scale-up domain large enough to hold the whole EP group: 72 GPUs in NVL72, 384 NPUs in CM384. No NIC or RDMA queue pair takes part; ranks write into each other's memory with load and store instructions.

第 3.7 節把 EP all-to-all 估為每個 MoE 層 $\text{token 數} \times k \times h \times 2\,\text{B}$ ，目的地逐 token 決定。這些 token 會送到任何 rank 的 GPU，所以 EP 打破了 rail 假設，rail-only 網路要為 NVLink 繞路付出 8.2–11.2%。superpod 從另一個方向回答。它把 scale-up domain 做得夠大，能容納整個 EP 群組：NVL72 有 72 顆 GPU，CM384 有 384 顆 NPU。這裡沒有 NIC 或 RDMA queue pair 參與；各 rank 用 load 與 store 指令直接寫進彼此的記憶體。

What remains is the fixed cost per phase, the $\alpha$ term of the $\alpha$ – $\beta$ model in the Chapter 4. UBEP's baseline is CANN EP, Huawei's Ascend adaptation of DeepEP for CM384. The scope is narrow: the paper measures decode-time inference, and only the dispatch half of the all-to-all. It calls combine analogous and does not evaluate it.

剩下的是每個階段的固定成本，也就是第 4 章 $\alpha$ – $\beta$ 模型中的 $\alpha$ 項。UBEP 的基準是 CANN EP，即華為為 CM384 改寫的 Ascend 版 DeepEP。範圍很窄：論文量測的是解碼階段的推論，而且只有 all-to-all 的 dispatch 這一半。它說 combine 可比照處理，但沒有評估。

2. The problem, measured2. 問題，用量測說話

CM384 has a two-tier fabric. NPUs on one board meet at first-tier switches, and boards meet at second-tier switches. Each NPU has two compute dies and several AI Cores, each with a matrix unit and several vector cores (AIVs), which move the data. Table 2 of the paper gives the access latency by distance (Diagram A); the paper does not say how it was measured.

CM384 是兩層的網路。同一塊板上的 NPU 經第一層交換器相連，板與板之間經第二層交換器相連。每顆 NPU 有兩個計算 die 與多個 AI Core，每個 AI Core 含一個矩陣單元與數個負責搬資料的向量核心（AIV）。論文 Table 2 給出依距離區分的存取延遲（示意圖 A）；論文沒有說明是怎麼量的。

Diagram A. The two-tier UB fabric of CM384 (simplified; each blue square is one die) and the access latency at each distance, from Table 2 of the paper (measured on CM384; method not stated). A token sent two hops away costs 11.5× one sent to the other die of the same NPU.示意圖 A。CM384 的兩層 UB 網路（簡化；每個藍色方塊是一個 die），以及各距離的存取延遲，取自論文 Table 2（在 CM384 上量測；方法未說明）。送到兩跳外的 token，代價是送到同一顆 NPU 另一個 die 的 11.5 倍。Drawn for this primer from Liu et al., SIGCOMM 2026, §2.3 and Table 2.

The baseline dispatch runs five phases. (1) Every rank pre-allocates a receive region, split by source and expert. (2) Each AIV writes its tokens straight into the destination experts' slots. (3) A global barrier, SyncAll; then AIVs write per-expert flags and token counts. (4) AIVs poll the flags, then SyncAll again. (5) A prefix sum over the counts, CalCumSum, gives each token's offset, and the tokens are reordered.

基準的 dispatch 有五個階段。(1) 每個 rank 預先配置接收區，依來源與專家切分。(2) 每個 AIV 把自己的 token 直接寫進目的專家的位置。(3) 全域 barrier SyncAll；之後 AIV 寫入每個專家的旗標與 token 數。(4) AIV 輪詢旗標，再做一次 SyncAll。(5) 對 token 數做前綴和（CalCumSum），得到每個 token 的位移，再重排 token。

The paper draws three insights from CM384. Barriers waste cores. UB gives 392 GB/s against 50 GB/s for IB/RoCE, so half the AIVs fill the link: going from 24 to 48 communication cores gave "minimal benefit", because the extra cores wait at barriers. Synchronization dominates. One SyncAll costs about 15% of the execution time. Balance must see distance. With two hops at 11.5× local, equal token counts leave stragglers. The paper does not state the batch or message size behind the first two measurements.

論文從 CM384 得出三個洞見。barrier 浪費核心。UB 提供 392 GB/s，IB/RoCE 只有 50 GB/s，所以一半的 AIV 就填滿鏈路：通訊核心從 24 個增加到 48 個只有「極小的好處」，因為多出的核心都在 barrier 等待。同步占主導。一次 SyncAll 約占執行時間的 15%。平衡必須看距離。兩跳是本地的 11.5 倍，相同的 token 數仍會留下 straggler。論文沒有說明前兩項量測所用的批次或訊息大小。

3. The design3. 設計

Three mechanisms work together. Decomposition creates concurrent tasks, scheduling balances them, and Data-as-Flag synchronizes them without barriers. The paper's Figure 4 (below) puts the three together.

三個機制一起運作。分解產生並行的工作，排程讓它們平衡，Data-as-Flag 則不用 barrier 就讓它們同步。論文的 Figure 4（下圖）把三者畫在一起。

Diagram of two NPUs with AIV cores and token paths, a 'No SyncAll' band between them, and boxes for kernel decomposition, hierarchical token-level scheduling and Data-as-Flag synchronization. — Figure 4 of the paper: an overview of the UBEP architecture (a design diagram). The three parts work together to remove the 'SyncAll' barrier: tokens flow from Init to Output across NPUs without waiting for every core to finish each phase.論文圖 4：UBEP 架構總覽（設計示意圖）。三個部分合力拿掉「SyncAll」屏障：token 從 Init 流到 Output、跨越 NPU，不必等每個核心都完成每個階段。Liu et al., SIGCOMM 2026, Figure 4 (arXiv 2607.06202 HTML, vector figure).

3.1 Kernel decomposition: split the cores by job3.1 核心分解：依工作拆分核心

UBEP splits the AIVs into two groups (Diagram B). The token group sends tokens. The count group writes the per-expert token counts and then computes CalCumSum. The groups run at the same time, so the counting hides behind the transmission. A point-to-point signal replaces SyncAll: the count group writes the offsets into shared global memory, and the other AIVs start reordering as soon as they see the update.

UBEP 把 AIV 拆成兩組（示意圖 B）。token 組負責傳送 token。計數組寫入每個專家的 token 數，再計算 CalCumSum。兩組同時執行，所以計數藏在傳輸後面。點對點訊號取代了 SyncAll：計數組把位移寫進共享的全域記憶體，其他 AIV 一看到更新就開始重排。

Diagram B. Dispatch on one die, after Figures 2 and 5 of the paper. (a) In BSP, AIVs that finish sending early wait at the barrier, and counting starts only after it. (b) UBEP counts on a few AIVs while the rest send, and one signal in shared memory starts the reordering.示意圖 B。一個 die 上的 dispatch，依論文 Figures 2 與 5 重畫。(a) 在 BSP 中，先傳完的 AIV 在 barrier 等待，計數要等 barrier 之後才開始。(b) UBEP 讓少數 AIV 計數、其餘同時傳送，共享記憶體中的一個訊號就啟動重排。Drawn for this primer from Liu et al., SIGCOMM 2026, §2.3, §3.2 and Figures 2, 5.

Appendix A sizes the groups. With $n$ AIVs, $N_{\text{exp}}$ experts, top- $K$ routing and batch $B$ , the count group's time grows with $N_{\text{exp}}/n_{\text{count}}$ and the token group's with $KB/(n - n_{\text{count}})$ . With linear costs and $n - n_{\text{count}} \approx n$ , equal times give

附錄 A 決定兩組的大小。設 $n$ 個 AIV、 $N_{\text{exp}}$ 個專家、top- $K$ 路由、批次 $B$ ，計數組的時間隨 $N_{\text{exp}}/n_{\text{count}}$ 成長，token 組的時間隨 $KB/(n - n_{\text{count}})$ 成長。在線性成本且 $n - n_{\text{count}} \approx n$ 的近似下，令兩者相等得到

n_{\text{count}} \approx \frac{\beta}{\alpha}\, n\, \frac{N_{\text{exp}}}{K\,B}

Here $\alpha$ and $\beta$ are the costs per token sent and per expert counted; the paper never measures their ratio. More experts need more counting cores, and larger batches move cores to sending. Appendix C.3 confirms this trend.

其中 $\alpha$ 與 $\beta$ 是每傳一個 token 與每計數一個專家的成本；論文從未量測兩者的比值。專家越多，需要越多計數核心；批次越大，核心就移往傳送。附錄 C.3 證實了這個趨勢。

3.2 Hierarchical token scheduling: equal hop mix3.2 階層式 token 排程：相同的跳數組合

One die sends $m$ tokens with $n$ AIVs. AIV $j$ gets the set $\mathcal{M}_j$ ; token $i$ costs $\ell_i$ to send and $\mathrm{RTT}_i$ to confirm. The paper's Program 1 minimizes the slowest AIV, with at most $\lceil m/n \rceil$ tokens per AIV:

一個 die 用 $n$ 個 AIV 送出 $m$ 個 token。AIV $j$ 分到集合 $\mathcal{M}_j$ ；token $i$ 的傳送成本是 $\ell_i$ ，確認成本是 $\mathrm{RTT}_i$ 。論文的 Program 1 在每個 AIV 最多 $\lceil m/n \rceil$ 個 token 的條件下，讓最慢的 AIV 最快：

\min\ \max_j \Big( \sum_{i \in \mathcal{M}_j} \ell_i + \max_{i \in \mathcal{M}_j} \mathrm{RTT}_i \Big)

The problem is NP-hard (a reduction from Subset Sum), and a solver is too slow for a microsecond budget. UBEP uses a rule instead: give every AIV the same hop mix. If $V_h$ tokens go $h$ hops, each AIV gets at most $\lceil V_h/n \rceil$ of them. A mapper reads a virtual AIV-by-expert matrix column by column and cuts the token sequence into equal slices. A four-way search, which tests three pivots in one SIMD instruction, finds each slice's first expert; the paper states 4–5 steps and under 1 µs.

這個問題是 NP-hard（由 Subset Sum 歸約），求解器對微秒級的預算來說太慢。UBEP 改用一條規則：給每個 AIV 相同的跳數組合。若有 $V_h$ 個 token 要走 $h$ 跳，每個 AIV 最多分到 $\lceil V_h/n \rceil$ 個。對應器逐欄讀取一個虛擬的「AIV × 專家」矩陣，並把 token 序列切成等長的片段。一個四分搜尋在一條 SIMD 指令中測試三個樞紐點，找出每個片段的第一個專家；論文說只需 4–5 步、不到 1 µs。

Panel 1面板 1

One die's dispatch: barriers or two groups一個 die 的 dispatch：barrier 還是兩組並行

Library函式庫

EP ranks (dies)EP rank 數（die）

Batch per rank

B

每個 rank 的批次

B

Count-group AIVs

n_{\text{count}}

計數組 AIV 數

n_{\text{count}}

SyncAll cost (assumed)SyncAll 成本（假設）

Dispatch time (model)

dispatch 時間（模型）

Send phase

傳送階段

Time outside sending

傳送以外的時間

Best count group (model)

最佳計數組（模型）

Destinations near / far

目的地近 / 遠

Send, one hop or less傳送，一跳以內Send, 2 hops傳送，2 跳SyncAllSyncAllFlags / counts旗標 / 計數Prefix sum前綴和Idle閒置

Our model of one die's dispatch, not the paper's code. Each AIV writes its tokens one after another, and each write costs the access latency of its destination (Table 2: 218 ns on the same NPU, 929 ns one hop, 2,500 ns two hops), the argument of the paper's Insight 3. Tokens spread evenly over all ranks; 16 ranks per server, one expert per rank, top-8, and 48 AIVs, the core count of the paper's Figures 3b and 9. CANN EP (BSP): all AIVs send contiguous slices, wait for the slowest, SyncAll, write flags and counts, wait one far round trip, SyncAll, then compute the prefix sum. UBEP: the count group writes counts, waits for every rank's counts and computes the prefix sum while the token group sends; one shared-memory signal replaces both barriers. "w/o mapping" keeps the hop-blind slices. The SyncAll cost is our assumption, set so that the two barriers take about 15% of the BSP time at the defaults (Insight 2). Reordering, the same in both designs, and link bandwidth are left out; in Table 4 UBEP used only 17–26% of the link. Compare shapes, not microseconds. Defaults: 64 ranks and batch 64, as in the paper's Figure 8 and sensitivity tests. Try 256 ranks: counting 256 experts needs many AIVs, and the gain shrinks, as the paper measured. Then set the count group far from the "best" tile.

這是我們對一個 die 的 dispatch 所建的模型，不是論文的程式碼。每個 AIV 依序寫出自己的 token，每次寫入的代價是目的地的存取延遲（Table 2：同一顆 NPU 218 ns、一跳 929 ns、兩跳 2,500 ns），也就是論文 Insight 3 的論點。token 平均分散到所有 rank；每台伺服器 16 個 rank、每個 rank 一個專家、top-8，以及 48 個 AIV，即論文 Figures 3b 與 9 的核心數。CANN EP（BSP）：所有 AIV 傳送連續的片段、等最慢的那個、SyncAll、寫旗標與計數、等一次遠端往返、再 SyncAll，最後計算前綴和。UBEP：token 組傳送的同時，計數組寫出計數、等所有 rank 的計數到齊並計算前綴和；共享記憶體中的一個訊號取代兩次 barrier。「無對應」保留不看跳數的片段。SyncAll 成本是我們的假設，設成在預設值下兩次 barrier 約占 BSP 時間的 15%（Insight 2）。兩種設計相同的重排，以及鏈路頻寬，都不列入；在 Table 4 中 UBEP 只用了鏈路的 17–26%。請比較形狀，而非微秒數。預設值：64 個 rank、批次 64，與論文 Figure 8 及敏感度測試相同。試試 256 個 rank：計數 256 個專家需要很多 AIV，增益隨之縮小，與論文量到的一致。再把計數組設成遠離「最佳」那一格的值。

3.3 Data-as-Flag: the write is the signal3.3 Data-as-Flag：寫入本身就是訊號

A receiver must know when data has arrived. A separate flag needs a fence between payload and flag. UB guarantees that a 512-byte write is atomic, so no receiver sees half a block, and the data can carry its own signal (Diagram C). Token-Flag Fusion puts a 32 B flag and 480 B of payload in each block, at 93.75% efficiency. Data-for-Checksum sends a checksum after the batch, which blocks token-level pipelining. Sentinel Polling pre-fills the buffer with a reserved value and watches a 32 B field. It reaches 100% efficiency, but buffers must be reset, and data equal to the sentinel would deadlock; the paper bounds that chance by $2^{-256}$ under a uniform-value assumption. Without 512 B atomics, the fallback is the checksum variant or explicit fences.

接收端必須知道資料何時到達。獨立的旗標需要在資料與旗標之間加一道 fence。UB 保證 512 位元組的寫入是原子的，所以接收端永遠看不到寫了一半的區塊，資料就能攜帶自己的訊號（示意圖 C）。Token-Flag Fusion 在每個區塊放 32 B 旗標與 480 B 資料，效率 93.75%。Data-for-Checksum 在整批之後送一個 checksum，這會擋住 token 層級的管線化。Sentinel Polling 預先在緩衝區填入保留值，並監看一個 32 B 欄位。它的效率是 100%，但緩衝區必須重置，而資料若恰好等於 sentinel 就會死結；論文在均勻分布的假設下把這個機率界定為 $2^{-256}$ 。沒有 512 B 原子寫入時，退路是 checksum 變體或明確的 fence。

Diagram C. The three Data-as-Flag variants, after Figure 6 of the paper. Each wide box is one 512 B block. The flag (accent) travels in the same atomic write as the data, the checksum variant adds one packet per batch, and Sentinel Polling checks a field of the data itself against a pre-filled value, so the empty outlined block on the right is one that has not arrived yet.示意圖 C。三種 Data-as-Flag 變體，依論文 Figure 6 重畫。每個寬方塊是一個 512 B 區塊。旗標（強調色）與資料在同一次原子寫入中傳送；checksum 變體每批多送一個封包；Sentinel Polling 則拿資料本身的一個欄位與預先填入的值比對，因此右側只有外框、沒有填色的區塊就是尚未到達的資料。Drawn for this primer from Liu et al., SIGCOMM 2026, §3.4 and Figure 6.

4. Implementation4. 實作

UBEP is about 10K lines of Ascend C on Huawei's CANN stack. It uses UB load/store primitives and the SIMD operations VectorCount, CompareScalar and FindFirstValue. The receive space is partitioned at initialization, so senders write tokens directly into the destination's memory. End-to-end tests use vLLM-Ascend v0.14.0.

UBEP 約有 10K 行 Ascend C，建在華為的 CANN 堆疊上。它使用 UB 的 load/store 原語，以及 VectorCount、CompareScalar 與 FindFirstValue 等 SIMD 操作。接收空間在初始化時就切分好，所以傳送端直接把 token 寫進目的地的記憶體。端到端測試使用 vLLM-Ascend v0.14.0。

5. What was measured5. 量到了什麼

All tests ran on 16 Ascend servers allocated from a production CM384, 256 NPU dies in all. The models are Qwen3-30B, GLM-4.7, DeepSeek-R1 and DeepSeek-V3.2, all with top-8 routing. The baselines are CANN EP and "UBEP w/o mapping", which keeps everything except the distance-aware mapping. Table 4 compares dispatch on identical hardware (Diagram D).

所有測試都在從生產環境 CM384 撥出的 16 台 Ascend 伺服器上執行，共 256 個 NPU die。模型是 Qwen3-30B、GLM-4.7、DeepSeek-R1 與 DeepSeek-V3.2，都用 top-8 路由。基準是 CANN EP 與「UBEP 無對應」；後者保留除了距離感知對應以外的一切。Table 4 在相同硬體上比較 dispatch（示意圖 D）。

Diagram D. Dispatch latency and effective bandwidth (CANN EP → UBEP) from 16 to 256 EP ranks, redrawn from Table 4 of the paper (measured on CM384; the paper does not state the configuration). UBEP is faster at every size, but both run far below the link rate.示意圖 D。16 到 256 個 EP rank 的 dispatch 延遲與有效頻寬（CANN EP → UBEP），依論文 Table 4 重畫（在 CM384 上量測；論文未說明設定）。UBEP 在每種規模都比較快，但兩者都遠低於鏈路速率。Drawn for this primer from Liu et al., SIGCOMM 2026, §5.2 and Table 4.

Derived from Table 4:

由 Table 4 推導：

Question問題	Calculation算式	Result結果
Latency reduction at EP16EP16 的延遲降幅	$1 - 73 \div 103$	$\approx 29.1\%$
Latency reduction at EP256EP256 的延遲降幅	$1 - 112 \div 151$	$\approx 25.8\%$
UBEP's bandwidth as a share of the 392 GB/s linkUBEP 頻寬占 392 GB/s 鏈路的比例	$66 \div 392$ … $100 \div 392$	$\approx 17\text{–}26\%$
Data per dispatch, latency × bandwidth (same in every row)每次 dispatch 的資料量，延遲 × 頻寬（每列都相同）	$73\,\mu\text{s} \times 100\,\text{GB/s}$	$\approx 7.3$ MB

So dispatch runs far below the link rate: latency and synchronization, not bytes, set the time. The paper reports a bandwidth gain of 35.3–40.8% over CANN EP. Table 4 also has a DeepEP row, labeled "a protocol-capability reference on H800". Its values equal the dispatch table in DeepEP's own README (H800 with 400 Gb/s InfiniBand, 128 tokens, hidden 7,168, top-8, FP8), and the paper does not say that it measured them. We therefore do not compare UBEP with DeepEP.

所以 dispatch 遠低於鏈路速率：決定時間的是延遲與同步，而不是位元組。論文報告頻寬比 CANN EP 提高 35.3–40.8%。Table 4 還有一列 DeepEP，標示為「H800 上的協定能力參考」。這些數值與 DeepEP 自己 README 中的 dispatch 表相同（H800 搭配 400 Gb/s InfiniBand、128 個 token、hidden 7,168、top-8、FP8），論文也沒有說是自己量的。因此我們不拿 UBEP 與 DeepEP 比較。

Test測試	Setup設定	Measured (dispatch)實測結果（dispatch）
Phase breakdown (Fig. 8)階段分解（Fig. 8）	64 NPUs, 1 expert each, three batch sizes64 顆 NPU、各 1 個專家、三種批次	34.7–52.4% lower latency than CANN EP延遲比 CANN EP 低 34.7–52.4%
Batch sweep (Fig. 7a)批次掃描（Fig. 7a）	128 ranks128 個 rank	46.4% on average for batch 8–64; 35.9% at batch 128批次 8–64 平均 46.4%；批次 128 時 35.9%
Rank sweep (Fig. 7b)rank 掃描（Fig. 7b）	batch 64批次 64	43.9% on average; 35.1% at 256 ranks平均 43.9%；256 個 rank 時 35.1%
Expert sweep (Fig. 7c)專家掃描（Fig. 7c）	expert counts not stated專家數未說明	Synchronization about 70% of runtime in CANN EP, about 30% in UBEP; 38.6–41.9% faster同步約占 CANN EP 執行時間的 70%、UBEP 的 30%；快 38.6–41.9%
Scheduling (Fig. 9)排程（Fig. 9）	64 NPUs, 256 experts, top-8, 8 hot experts64 顆 NPU、256 個專家、top-8、8 個熱門專家	Slowest AIV: CANN EP 62.2 µs, w/o mapping 48.1 µs, UBEP 43.5 µs最慢的 AIV：CANN EP 62.2 µs、無對應 48.1 µs、UBEP 43.5 µs
Data-as-Flag (Fig. 10)Data-as-Flag（Fig. 10）	vs stop-and-wait flags對照 stop-and-wait 旗標	31.0–57.1% lower latency; about 1 µs saved at about 10 KB granularity延遲低 31.0–57.1%；在約 10 KB 粒度下約省 1 µs

Figure 8 of the paper (below) breaks the dispatch into its phases, the measured counterpart of the panel above.

論文 Figure 8（下圖）把 dispatch 拆成各個階段，是上面面板的實測對照。

Three panels of horizontal stacked bars of dispatch phases for UBEP and CANN EP at batch sizes 32, 64 and 128. — Figure 8 of the paper: where the time goes in one MoE dispatch, UBEP (top half of each panel) vs CANN EP (bottom half), at batch sizes 32, 64 and 128 on 64 NPUs of the CloudMatrix384 (measured). Each row is a group of AIV cores (0–41 averaged, then 42–45, 46 and 47). UBEP has no separate flag-and-verify phase, and its phases overlap instead of running one after another; its bars end at roughly 20–30, 35–50 and 60–90 µs against about 60, 82 and 130 µs for CANN EP.論文圖 8：在 CloudMatrix384 的 64 顆 NPU 上，一次 MoE dispatch 的時間花在哪裡，UBEP（每個面板上半）對 CANN EP（下半），batch 大小 32、64、128（實測）。每一列是一組 AIV 核心（0–41 取平均，接著是 42–45、46、47）。UBEP 沒有獨立的旗標與驗證階段，各階段重疊執行而不是一個接一個；它的長條約在 20–30、35–50、60–90 µs 結束，CANN EP 約為 60、82、130 µs。Liu et al., SIGCOMM 2026, Figure 8 (arXiv 2607.06202 HTML, vector figure).

End to end, with vLLM-Ascend, P99 time per output token (TPOT) fell by up to 11.1%. Profiling explains the gap: MoE communication is about 50% of per-token latency but only about 20% of the hardware execution time. Dependency stalls and runtime overhead fill the rest. The paper gives no absolute TPOT, input lengths or concurrency.

端到端使用 vLLM-Ascend 時，P99 每個輸出 token 的時間（TPOT）最多降低 11.1%。剖析解釋了這個落差：MoE 通訊約占每個 token 延遲的 50%，卻只占硬體實際執行時間的約 20%。其餘是依賴停頓與執行期開銷。論文沒有給出 TPOT 的絕對值、輸入長度或並行度。

6. Limits and open questions6. 限制與未解的問題

The paper's own: gains shrink at batch 128 and at 256 ranks, where 12 cores are reserved for routing and checksums; raw transmission is slightly slower, because cores move to counting; Sentinel Polling's deadlock risk is never zero. Its portability checklist (Table 5) needs per-core concurrency, hop-dependent latency and 512 B atomic writes. A careful reader adds:

論文自己的：在批次 128 與 256 個 rank 時增益縮小，此時有 12 個核心保留給路由與 checksum；原始傳輸稍微變慢，因為有核心移去計數；Sentinel Polling 的死結風險永遠不是零。它的可移植性清單（Table 5）需要核心層級的並行、隨跳數變化的延遲，以及 512 B 原子寫入。細心的讀者還應該補上：

Inference, dispatch only. No training, no backward pass, no combine. Training's larger batches move toward the regime where the gains already shrink.只有推論、只有 dispatch。沒有訓練、沒有反向、沒有 combine。訓練的批次較大，會走向增益已經縮小的區間。
One platform, one baseline. There is no same-hardware comparison with Hybrid-EP, the closest design.一個平台、一個基準。沒有與最接近的設計 Hybrid-EP 在相同硬體上比較。
Unstated setups. The 15% SyncAll share, the 24-to-48-core test and Table 4 have no stated batch or message size.未說明的設定。15% 的 SyncAll 占比、24 到 48 個核心的測試與 Table 4 都沒有說明批次或訊息大小。
Mixed headline. The 52.4% comes from Figure 8 (64 NPUs), while Table 4 shows 25.8–29.1%.混合的標題數字。52.4% 來自 Figure 8（64 顆 NPU），而 Table 4 顯示的是 25.8–29.1%。

7. How it connects7. 與其他內容的關聯

Deep dive, section 5: the EP byte formula still holds, but inside a superpod the bytes are cheap, and barriers and hop distance set the time.第 3.7 節：EP 的位元組公式依然成立，但在 superpod 內位元組很便宜，決定時間的是 barrier 與跳數距離。
Chapter 4, section 2: going from 50 to 392 GB/s shrinks only the $\beta$ term; a phase made of barriers keeps its $\alpha$ .第 4.3 節：從 50 到 392 GB/s 只縮小 $\beta$ 項；由 barrier 組成的階段仍保有它的 $\alpha$ 。
The EPIC digest attacks the same dispatch on scale-out H20/H800 clusters, by removing duplicate inter-host bytes and expert imbalance.EPIC 導讀在 scale-out 的 H20/H800 叢集上處理同樣的 dispatch，做法是去除重複的跨主機位元組與專家不平衡。
The PipeMorph digest shows the same rule for pipelines: the slowest element sets the pace.PipeMorph 導讀在管線上呈現同一條規則：最慢的元素決定步調。
Rail-only measured the cross-rail detour of EP; inside a superpod, EP traffic never reaches the rails. TPU v4 is another very large scale-up domain.Rail-only 量過 EP 跨 rail 的繞路成本；在 superpod 內，EP 流量根本不會碰到 rail。TPU v4 是另一個非常大的 scale-up domain。

Key numbers關鍵數字

Number	數字	Meaning (measured on CM384 unless noted)
218 / 929 / 2,500 ns	Access latency: same NPU / one hop / two hops	存取延遲：同一 NPU / 一跳 / 兩跳
≈ 15%	Share of execution time in one SyncAll (setup not stated)	一次 SyncAll 占執行時間的比例（設定未說明）
52.4%	Largest dispatch-latency cut against CANN EP; 64 NPUs	相對 CANN EP 的最大 dispatch 延遲降幅；64 顆 NPU
151 → 112 µs	CANN EP → UBEP dispatch at 256 ranks (Table 4)	256 個 rank 時 CANN EP → UBEP 的 dispatch（Table 4）
62.2 / 48.1 / 43.5 µs	Slowest AIV: CANN EP / w/o mapping / UBEP	最慢的 AIV：CANN EP / 無對應 / UBEP
11.1%	Largest P99 TPOT cut, end to end in vLLM-Ascend	vLLM-Ascend 端到端的最大 P99 TPOT 降幅

Terms used on this page本頁用到的術語

Superpod: Tens to hundreds of accelerators on one scale-up fabric with a shared load/store address space.; 數十到數百顆加速器共用一個 scale-up 網路與 load/store 位址空間。
AIV: A vector core of an Ascend AI Core; UBEP schedules work per AIV.; Ascend AI Core 中的向量核心；UBEP 以 AIV 為單位排程。
Dispatch: Sending each token to the ranks of its top- $k$ experts.; 把每個 token 送到其 top- $k$ 專家所在的 rank。
SyncAll: A global barrier across all cores in the Ascend programming model.; Ascend 程式模型中跨所有核心的全域 barrier。
CalCumSum: The prefix sum over per-expert token counts that gives each token's offset.; 對每個專家的 token 數做前綴和，得到每個 token 的位移。
Data-as-Flag: Synchronization carried by the data write itself, relying on 512 B atomicity.; 依賴 512 B 原子性、由資料寫入本身攜帶的同步。