Research digest 1 of 6 · Chapter 3: Splitting the model

研究導讀 1 / 6 · 第 3 章：切分模型

Attack of the Bubbles: Straggler-Resilient Pipeline Parallelism for Large Model Training氣泡來襲：能抵抗 straggler 的大型模型訓練管線平行

NSDI 2026Tianyuan Wu, Lunxi Cao, Hanfeng Lu et al. (HKUST, Alibaba Group) · paper論文

After reading, you can explain why one slow pipeline link costs more than its delay, compute how much delay a schedule absorbs, and follow how PipeMorph re-plans warm-up and moves sends to the CPU.讀完後，你能說明為何一條變慢的管線鏈路造成的損失超過延遲本身，算出排程能吸收多少延遲，並理解 PipeMorph 如何重新規劃暖機、把傳送交給 CPU。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 3 ↗第 3 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

Pipeline parallelism (PP) sends small activations between stages, so designers put it on long, oversubscribed paths. When one inter-stage link slows, the delay costs about itself up to a threshold; above it, idle slots (bubbles) cascade from stage to stage. NCCL adds head-of-line (HOL) blocking: a stuck send stops ready computation. The threshold is $c \le (\Delta-1)t$ , where $t$ is one operator's time and the slackness $\Delta$ counts the extra warm-up forwards of the upstream stage. PipeMorph gives the slow link more slackness and sends its activations through host memory with CPU-side RDMA. With injected delays on up to 128 H800 GPUs, iteration time improved 1.2–3.5×; with a replayed production delay trace, throughput was 1.36× that of ZeroBubble.

管線平行（PP）在階段之間傳送小小的激活值，所以設計者把它放在又長又超額認購的路徑上。某一條階段間鏈路變慢時，在門檻以內延遲的代價約等於它本身；超過門檻，閒置時段（氣泡）會從一個階段連鎖傳到下一個階段。NCCL 還會造成 head-of-line（HOL）阻塞：一個卡住的 send 擋住已經可以執行的計算。門檻是 $c \le (\Delta-1)t$ ，其中 $t$ 是一個運算子的時間，slackness $\Delta$ 是上游階段多跑的暖機正向數。PipeMorph 給慢鏈路更多 slackness，並把它的激活值經主機記憶體以 CPU 端 RDMA 傳送。在最多 128 顆 H800 GPU 上注入延遲時，迭代時間改善 1.2–3.5 倍；重播生產環境的延遲紀錄時，吞吐是 ZeroBubble 的 1.36 倍。

1. Where this paper sits1. 這篇論文的位置

Section 3.5 gives the bubble, $(p-1)/(m+p-1)$ of the step, and says that small PP messages tolerate long paths. That is why Alibaba HPN lets only PP traffic cross its 15:1 core links. The formulas assume that a message arrives at once. This paper adds a delay $c_i$ on the link between stages $S_i$ and $S_{i+1}$ .

第 3.5 節給出氣泡，占一步的 $(p-1)/(m+p-1)$ ，並說小小的 PP 訊息能忍受長路徑。這就是 Alibaba HPN 只讓 PP 流量跨越 15:1 核心鏈路的原因。那些公式假設訊息一送出就到達。這篇論文加上階段 $S_i$ 與 $S_{i+1}$ 之間鏈路上的延遲 $c_i$ 。

The paper builds on ZeroBubble (ZB), which splits each backward pass into two operators. B computes the input gradient, which the previous stage waits for. W computes the weight gradient, which no stage waits for, so W fills idle slots. 1F1B runs B and W together. Stage $S_i$ runs $x_i$ forwards before its first backward, its warm-up count; each one holds a micro-batch of activations. The arXiv preprint called the system Adaptra. Its baseline "Falcon" is a straggler-mitigation system by the same first author, not Google's Falcon transport.

論文建立在 ZeroBubble（ZB）之上，它把每次反向拆成兩個運算子。B 計算輸入梯度，前一個階段要等它。W 計算權重梯度，沒有階段要等它，所以 W 用來填補閒置時段。1F1B 把 B 與 W 一起執行。階段 $S_i$ 在第一次反向之前執行 $x_i$ 個正向，這是它的暖機數；每一個都保留一個 micro-batch 的激活值。arXiv 預印本把系統叫作 Adaptra。它的基準「Falcon」是同一位第一作者的 straggler 緩解系統，不是 Google 的 Falcon 傳輸協定。

2. The problem, measured2. 問題，用量測說話

Communication stragglers, links that stay slower than their peers, are frequent and can last hours. In Alibaba's clusters, more than 36% of the jobs that used over 50% of the GPUs slowed down because of communication (cited from Crux). The authors injected delays with an NCCL plugin. Figure 1 redraws their Figure 3: the same 30 ms delay slowed GPT-2 7B 8.04× on a DP link but 2.24× on a PP link.

通訊 straggler 是持續比同儕慢的鏈路，它們很常見，而且可能持續數小時。在阿里巴巴的叢集中，使用超過 50% GPU 的任務有超過 36% 因通訊而變慢（引自 Crux）。作者用一個 NCCL plugin 注入延遲。圖 1 重畫他們的 Figure 3：同樣 30 ms 的延遲在 DP 鏈路上讓 GPT-2 7B 慢 8.04 倍，在 PP 鏈路上則是 2.24 倍。

Figure 1. GPT-2 7B (ZB, DP 2 × PP 4, 8 H800 GPUs) with a delay injected on one link, redrawn from Figure 3 of the paper (measured). A DP delay stalls every replica's sync.圖 1。某一條鏈路注入延遲時的 GPT-2 7B（ZB、DP 2 × PP 4、8 顆 H800 GPU），依論文 Figure 3 重畫（實測）。DP 延遲會卡住每個副本的同步。Drawn for this primer from Wu et al., NSDI 2026, §2.2 and Figure 3.

Even on PP, 2.24× for 30 ms is large. Figure 1 of the paper splits such a slowdown into its two causes.

即使在 PP 上，30 ms 就慢 2.24 倍也很大。論文 Figure 1 把這種變慢拆成兩個原因。

Stacked bar chart of time per iteration for 1F1B and ZB at communication delays of 0, 0.02, 0.04 and 0.06 s. — Figure 1 of the paper, measured: GPT-2 14B on 8 H800 nodes, with a delay injected between stages 6 and 7. Bars split each iteration into base time, dependency bubbles and blocking. At 60 ms, ZB grows from 0.77 to 3.26 s, mostly from blocking.論文 Figure 1，實測：在 8 台 H800 節點上訓練 GPT-2 14B，於第 6、7 階段之間注入延遲。長條把每次迭代拆成基本時間、依賴氣泡與阻塞。延遲 60 ms 時，ZB 從 0.77 s 增加到 3.26 s，增加的部分大多是阻塞。Wu et al., NSDI 2026, Figure 1 (USENIX PDF, p. 2).

Dependency bubbles. A stage cannot start an operator before its input arrives. A short delay fits into existing slack. A longer one makes a stage wait, then its neighbor, and the bubble spreads. In the paper's worked example (4 stages, 12 micro-batches, 10 ms operators, 390 ms), a 10 ms delay on the first link adds 10 ms, but 20 ms adds 50 ms. The panel in section 3.2 reproduces this.

依賴氣泡。一個階段在輸入到達之前無法開始運算子。短延遲裝得進原有的餘裕。長一點的延遲讓某個階段等待，接著是它的鄰居，氣泡就擴散開來。在論文的範例中（4 個階段、12 個 micro-batch、運算子 10 ms、390 ms），第一條鏈路上 10 ms 的延遲多出 10 ms，但 20 ms 卻多出 50 ms。第 3.2 節的面板重現了這個結果。

HOL blocking. NCCL sends are GPU kernels, issued in one fixed order with the computation. On a slow link the NCCL queue fills, the next send cannot launch, and the ready compute kernel behind it waits (Figure 2, top). Extra CUDA streams share the same queue, so they do not help. With 30 ms on one PP link, a 7B iteration went from 710 ms to 1,350 ms.

HOL 阻塞。NCCL 的 send 是 GPU kernel，與計算以單一固定順序發出。鏈路慢時 NCCL 佇列會滿，下一個 send 無法啟動，排在它後面已備妥的計算 kernel 只能等待（圖 2 上半）。多開 CUDA stream 會共用同一個佇列，所以沒有幫助。一條 PP 鏈路加 30 ms 時，7B 的一次迭代從 710 ms 增加到 1,350 ms。

Figure 2. Top: under NCCL, the stuck send SF2 fills the queue, so SF3 cannot launch and the ready F4 waits (after Figures 2 and 6 of the paper). Bottom: delegation; the GPU only copies to host memory (grey), and delegates D1–D3 send.圖 2。上半：在 NCCL 下，卡住的 send SF2 塞滿佇列，所以 SF3 無法啟動，已備妥的 F4 只能等待（依論文 Figures 2 與 6）。下半：委派；GPU 只把資料複製到主機記憶體（灰色），由委派行程 D1–D3 送出。Drawn for this primer from Wu et al., NSDI 2026, §3.2, §5.2 and Figures 2, 6, 10.

ECMP and packet spraying cannot fix either effect: they do not know the pipeline's critical path and cannot reach inside the GPU. The fix belongs in the framework, which sees the schedule.

ECMP 與 packet spraying 修不好這兩個效應：它們不知道管線的關鍵路徑，也碰不到 GPU 內部。解法屬於看得到排程的框架。

3. The design3. 設計

PipeMorph analyzes how much delay a schedule absorbs, moves slack to the slow link, and takes sends out of the GPU's issue order. A slow DP link is first re-mapped to carry PP traffic, as in Falcon.

PipeMorph 分析排程能吸收多少延遲、把餘裕移到慢鏈路，並把 send 移出 GPU 的發出順序。慢的 DP 鏈路會先像 Falcon 那樣重新對應，改送 PP 流量。

3.1 Slackness is a delay budget3.1 Slackness 就是延遲預算

The slackness of link $i$ is $\Delta_i = x_i - x_{i+1}$ , the extra warm-up forwards of the upstream stage; Lemma 1 proves $\Delta_i \ge 0$ . Follow one steady-state cycle (Figure 3). After $S_{i+1}$ finishes the backward $B_a$ , four steps must happen before it can run a later forward $F_b$ . The gradient travels up ( $c_i$ ), $S_i$ runs $B_a$ and $F_b$ ( $2t$ ), and the activation travels down ( $c_i$ ). The schedule leaves this chain a window of $2\Delta_i t$ , so it fits if $2c_i + 2t \le 2\Delta_i t$ .

鏈路 $i$ 的 slackness 是 $\Delta_i = x_i - x_{i+1}$ ，即上游階段多跑的暖機正向數；Lemma 1 證明 $\Delta_i \ge 0$ 。跟著穩定期的一個週期走（圖 3）。 $S_{i+1}$ 完成反向 $B_a$ 之後，必須先發生四個步驟，它才能執行較晚的正向 $F_b$ 。梯度往上傳（ $c_i$ ）， $S_i$ 執行 $B_a$ 與 $F_b$ （ $2t$ ），激活值再往下傳（ $c_i$ ）。排程留給這條依賴鏈 $2\Delta_i t$ 的窗口，所以只要 $2c_i + 2t \le 2\Delta_i t$ 就裝得下。

Figure 3. One steady-state cycle with slackness 2, after Figure 7 of the paper (analysis). The chain from

B_a

F_b

needs

2c + 2t

; the schedule leaves

2\Delta t

. Here

c = t

, the largest delay that fits.圖 3。slackness 為 2 時穩定期的一個週期，依論文 Figure 7 重畫（分析）。從

B_a

到

F_b

的依賴鏈需要

2c + 2t

；排程留給它

2\Delta t

。此處

c = t

，是裝得下的最大延遲。Drawn for this primer from Wu et al., NSDI 2026, §4.1, Theorem 1 and Figure 7.

Theorem 1 gives the added iteration time with $N$ micro-batches:

Theorem 1 給出 $N$ 個 micro-batch 時多出的迭代時間：

D(c_i) = \begin{cases} \Theta(c_i), & c_i \le (\Delta_i - 1)\,t \\[6pt] \Theta\!\left(\dfrac{N\,c_i}{\Delta_i + 1}\right), & c_i \gt (\Delta_i - 1)\,t \end{cases}

Below the threshold, the delay costs itself once. Above it, every window stretches, and the cost grows with $N$ . Equation 1 states the condition for unequal operator times:

門檻以下，延遲只花它本身一次。門檻以上，每個窗口都被拉長，代價隨 $N$ 成長。Equation 1 寫出運算子時間不相等時的條件：

t^F_i + t^B_i + 2c_i \le \Delta_i \big(t^F_{i+1} + t^B_{i+1}\big)

So a link with $\Delta_i = 1$ , as in 1F1B, has no budget at all. Interleaving divides $t$ by $v$ , so it also divides the budget by $v$ .

所以像 1F1B 那樣 $\Delta_i = 1$ 的鏈路完全沒有預算。交錯把 $t$ 除以 $v$ ，預算也就跟著除以 $v$ 。

3.2 Re-planning the warm-up when a link slows3.2 鏈路變慢時重新規劃暖機

Slackness costs memory: each extra warm-up forward holds another micro-batch of activations. Algorithm 1 plans for no straggler. The first stage gets $x_0 = \lfloor M/M_F \rfloor$ , the activations that fit in GPU memory, and the slack is spread evenly down to $x_{S-1} = 1$ . Algorithm 2 runs after detection. From the last stage up, it gives each link the smallest slackness that satisfies Equation 1, within two clips:

slackness 要用記憶體換：每多一個暖機正向就多保留一個 micro-batch 的激活值。Algorithm 1 在沒有 straggler 時規劃。第一個階段拿到 $x_0 = \lfloor M/M_F \rfloor$ ，也就是 GPU 記憶體裝得下的激活值數，餘裕平均分配，一直到 $x_{S-1} = 1$ 。Algorithm 2 在偵測到 straggler 後執行。它從最後一個階段往上，給每條鏈路滿足 Equation 1 的最小 slackness，並限制在兩個界限之內：

\begin{gathered} r_i = \left\lceil \frac{t^F_i + t^B_i + 2c_i}{t^F_{i+1} + t^B_{i+1}} \right\rceil \\[6pt] \Delta_i = \min\big( N - 2S,\ \max(r_i,\ 2) \big) \\[6pt] x_i = x_{i+1} + \Delta_i \end{gathered}

The lower clip keeps a budget on every link; the upper clip leaves micro-batches for the other stages. The GPU memory limit is dropped, because extra activations go to host memory (section 3.3). For the worked example with 20 ms on the first link:

下限讓每條鏈路都保有預算；上限讓其他階段保有 micro-batch。這裡不再受 GPU 記憶體限制，因為多出的激活值會放到主機記憶體（第 3.3 節）。以第一條鏈路有 20 ms 的範例來算：

Question問題	Calculation算式	Result結果
Budget of the no-delay plan $x = (7, 5, 3, 1)$ , $\Delta_0 = 2$ 無延遲規劃 $x = (7, 5, 3, 1)$ 、 $\Delta_0 = 2$ 的預算	$(2 - 1) \times 10$	10 msexceeded by 20 ms被 20 ms 超出
Slackness from Algorithm 2 for link $S_0$ – $S_1$ Algorithm 2 給鏈路 $S_0$ – $S_1$ 的 slackness	$\lceil (10 + 10 + 2 \times 20) \div (10 + 10) \rceil$	3cap $N - 2S = 4$ 上限 $N - 2S = 4$
New budget of that link該鏈路的新預算	$(3 - 1) \times 10$	20 msthe delay fits延遲裝得下
New warm-up counts, last stage first新暖機數，從最後一個階段算起	$1,\ 1 + 2,\ 3 + 2,\ 5 + 3$	$x = (8, 5, 3, 1)$

Algorithms 3–4 turn the $x_i$ into a full schedule with a greedy simulator, because an exact MILP takes hours. After its warm-up, each idle stage runs the available operator with the highest priority: B, which frees memory and unblocks the upstream stage, then F, then W, which fills gaps. Schedules change only at iteration boundaries.

Algorithms 3–4 用貪婪模擬器把 $x_i$ 變成完整排程，因為精確的 MILP 要花數小時。暖機之後，每個閒置的階段從可執行的運算子中挑優先權最高的：先是 B（釋放記憶體並解除上游的等待），再來是 F，最後是填補空隙的 W。排程只在迭代邊界更換。

Panel 1面板 1

One slow link, three schedules一條慢鏈路、三種排程

Schedule排程

Delay

c

on the slow link慢鏈路上的延遲

c

Slow link慢鏈路

Stages

S

階段數

S

Micro-batches

N

Micro-batch 數

N

Operator time

t

運算子時間

t

GPU activation budget

x_0

GPU 激活值預算

x_0

Iteration time

迭代時間

Bubble rate

氣泡率

Slow link slackness

慢鏈路 slackness

Warm-up counts

x

暖機數

x

Activations on

S_0

S_0

的激活值

F (forward)F（正向）B (1F1B: B + W, 2t)B（1F1B：B + W，2t）W (weight gradient)W（權重梯度）Bubble氣泡

Our model of the paper's schedules. F, B and W each take $t$ ; 1F1B runs B and W together in $2t$ . The delay $c$ applies to both directions of one link. 1F1B and ZeroBubble are planned once for no delay and then run in that fixed order. PipeMorph re-plans with Algorithm 2 and the greedy scheduler, using the delay that the profiler would measure (after the warm-up, an F runs only while the stage holds fewer than $x_i$ activations). ZeroBubble and the no-delay PipeMorph plan come from Algorithm 1 with budget $x_0$ . The panel models dependency bubbles only, with no HOL blocking, so its ZeroBubble behaves like the paper's PipeMorph-CPU. Defaults are the paper's worked example (Figures 4–5): 390 ms without delay, 400 ms at 10 ms, 440 ms at 20 ms. Try $x_0 = 4$ (slackness 1): every millisecond of delay cascades. Then halve $t$ : the budget $(\Delta-1)t$ halves too. Measured on H800s with injected delays: 30 ms on the last link of a 4-stage 7B pipeline gave ZB 703 ms, PipeMorph-CPU 579 ms and PipeMorph 398 ms (Figure 14). 60 ms on an 8-stage 14B pipeline slowed 1F1B 2.70×, ZB 4.24×, PipeMorph-CPU 2.15× and PipeMorph 1.13× (Figure 15).

這是我們對論文排程所建的模型。F、B、W 各花 $t$ ；1F1B 把 B 與 W 一起做，花 $2t$ 。延遲 $c$ 作用在同一條鏈路的兩個方向。1F1B 與 ZeroBubble 只在無延遲時規劃一次，之後依那個固定順序執行。PipeMorph 用 profiler 會量到的延遲，以 Algorithm 2 與貪婪排程器重新規劃（暖機之後，只有在階段保留的激活值少於 $x_i$ 時才執行 F）。ZeroBubble 與無延遲時的 PipeMorph 規劃來自預算為 $x_0$ 的 Algorithm 1。面板只模擬依賴氣泡，沒有 HOL 阻塞，所以面板中的 ZeroBubble 表現像論文的 PipeMorph-CPU。預設值就是論文的範例（Figures 4–5）：無延遲 390 ms、延遲 10 ms 時 400 ms、20 ms 時 440 ms。試試 $x_0 = 4$ （slackness 1）：每一毫秒的延遲都會連鎖擴散。再把 $t$ 減半：預算 $(\Delta-1)t$ 也跟著減半。在 H800 上注入延遲的實測：4 個階段的 7B 管線最後一條鏈路加 30 ms，ZB 703 ms、PipeMorph-CPU 579 ms、PipeMorph 398 ms（Figure 14）。8 個階段的 14B 管線加 60 ms，1F1B 慢 2.70 倍、ZB 4.24 倍、PipeMorph-CPU 2.15 倍、PipeMorph 1.13 倍（Figure 15）。

3.3 Taking sends out of the GPU's issue order3.3 把 send 移出 GPU 的發出順序

Re-planning does not remove HOL blocking. A straw man ran the NCCL sends in separate GPU processes. It removed the launch gaps but slowed the computation: one backward pass took 61.9 ms instead of 31.1 ms, and the end-to-end slowdown stayed about 1.9×.

重新規劃消除不了 HOL 阻塞。一個稻草人方案把 NCCL 的 send 放到獨立的 GPU 行程中執行。它消除了啟動空檔，卻拖慢了計算：一次反向要 61.9 ms 而不是 31.1 ms，端到端仍慢約 1.9 倍。

PipeMorph instead delegates the affected stages' transfers to the CPU (Figure 2, bottom). The trainer copies a tensor to pinned host memory and sets a flag. A delegate process polls the flag and sends the tensor with CPU-side RDMA (Gloo); the receiving side does the reverse. Several delegates with their own queues keep up with the GPU. Delegation removes HOL blocking and keeps the extra activations of Algorithm 2 in host memory. It could also bypass a failed RNIC, but the paper does not test that. Figure 10 of the paper shows the path in seven steps.

PipeMorph 改為把受影響階段的傳輸委派給 CPU（圖 2 下半）。訓練行程把張量複製到 pinned 主機記憶體並設定旗標。委派行程（delegate）輪詢這個旗標，再用 CPU 端 RDMA（Gloo）把張量送出；接收端則反向進行。多個各帶佇列的委派行程讓傳送跟得上 GPU。委派消除了 HOL 阻塞，並把 Algorithm 2 多出的激活值放在主機記憶體。它也能繞過故障的 RNIC，但論文沒有測試這一點。論文 Figure 10 以七個步驟畫出這條路徑。

Diagram of receiver and sender delegates on the CPU between RNICs and the GPU, with seven numbered steps. — Figure 10 of the paper (design): receiver delegates fetch data through the RNICs (1) and hand it to the GPU (2–3); after F, B or W (4), results return to host memory (5–6) and sender delegates transmit them (7).論文 Figure 10（設計）：接收端委派行程經 RNIC 取得資料（1），交給 GPU（2–3）；執行 F、B 或 W（4）之後，結果回到主機記憶體（5–6），由傳送端委派行程送出（7）。Wu et al., NSDI 2026, Figure 10 (USENIX PDF, p. 9).

4. Implementation4. 實作

PipeMorph is 5.3K lines of code, mostly Python with CUDA transfer kernels, on Megatron-LM and ZeroBubble. A profiler times operators with CUDA Events and reports through Redis; straggler detection is reused from Falcon and XPUTimer. When a PP link slows, the scheduler deploys a new plan at the next iteration boundary and turns on delegation for the affected nodes.

PipeMorph 有 5.3K 行程式碼，主要是 Python，傳輸 kernel 用 CUDA，建在 Megatron-LM 與 ZeroBubble 之上。profiler 用 CUDA Events 為運算子計時並透過 Redis 回報；straggler 偵測沿用 Falcon 與 XPUTimer。某條 PP 鏈路變慢時，排程器在下一個迭代邊界部署新的規劃，並為受影響的節點開啟委派。

5. What was measured5. 量到了什麼

The testbed has 128 H800 GPUs, 8 per node, with 400 Gb/s InfiniBand. GPT-2-style models range from 7B (PP 4, 4 GPUs) to 140B (TP 8, DP 2, PP 8, 128 GPUs). Delays are injected into one link; one test replays a recorded production trace. PipeMorph-CPU, which uses delegation with the original schedule, isolates the effect of adaptation.

測試平台有 128 顆 H800 GPU，每節點 8 顆，400 Gb/s InfiniBand。GPT-2 風格的模型從 7B（PP 4、4 顆 GPU）到 140B（TP 8、DP 2、PP 8、128 顆 GPU）。延遲注入某一條鏈路；其中一個測試重播生產環境錄下的紀錄。PipeMorph-CPU 用委派但保留原本的排程，用來單獨看出調適的效果。

Test測試	Setup設定	Measured實測結果
One slow link (Fig. 14)一條慢鏈路（Fig. 14）	7B, 30 ms on the last link7B，最後一條鏈路 30 ms	ZB 703 ms, 57.4% idle; PipeMorph-CPU 579 ms, 48.8%; PipeMorph 398 ms, 25.3%ZB 703 ms、閒置 57.4%；PipeMorph-CPU 579 ms、48.8%；PipeMorph 398 ms、25.3%
Delay sweep (Fig. 15)延遲掃描（Fig. 15）	14B, 60 ms on link 6–714B，鏈路 6–7 上 60 ms	Slowdown: 1F1B 2.70×, ZB 4.24×, PipeMorph-CPU 2.15×, PipeMorph 1.13×變慢：1F1B 2.70 倍、ZB 4.24 倍、PipeMorph-CPU 2.15 倍、PipeMorph 1.13 倍
Model sizes (Fig. 16)模型大小（Fig. 16）	7B–60B, 30 or 60 ms7B–60B，30 或 60 ms	Up to 3.71× faster than 1F1B and 3.49× than ZB最多比 1F1B 快 3.71 倍、比 ZB 快 3.49 倍
Slow DP link (Fig. 19)慢 DP 鏈路（Fig. 19）	7B, DP 2 × PP 4, 10–30 ms7B、DP 2 × PP 4、10–30 ms	Falcon's re-mapping 3.6× faster than ZB; PipeMorph 1.96× faster than FalconFalcon 的重新對應比 ZB 快 3.6 倍；PipeMorph 再比 Falcon 快 1.96 倍
Production trace (Fig. 22)生產環境紀錄（Fig. 22）	140B, 128 GPUs, one hour140B、128 顆 GPU、一小時	45.9 iterations/min, against 41.4 (1F1B) and 33.6 (ZB)每分鐘 45.9 次迭代，1F1B 41.4、ZB 33.6
No straggler (Fig. 21)無 straggler（Fig. 21）	Every PP transfer delegated所有 PP 傳輸都委派	+1.0% to +16.4% iteration time迭代時間 +1.0% 到 +16.4%

The schedule generator stayed within 1% of the MILP optimum on simulated profiles and ran in under 100 ms. Derived ratios:

排程產生器在模擬的設定下與 MILP 最佳解相差不到 1%，執行不到 100 ms。推導出的比值：

Question問題	Calculation算式	Result結果
ZB against PipeMorph, one link at 30 ms一條鏈路 30 ms 時 ZB 相對 PipeMorph	$703 \div 398$	$\approx 1.77\times$
PipeMorph against ZB, slow DP link慢 DP 鏈路下 PipeMorph 相對 ZB	$3.6 \times 1.96$	$\approx 7.1\times$

Figure 16 of the paper holds the single-link tests behind the 3.71× and 3.49× above.

論文 Figure 16 是上面 3.71 倍與 3.49 倍背後的單一鏈路測試。

Four grouped bar charts of time per iteration for 7B to 60B models under four delay settings and four systems. — Figure 16 of the paper, measured on 128 H800 GPUs: iteration time with 30 or 60 ms injected between the first or the last two stages, for four model sizes. PipeMorph (red, right bar) is fastest in every group.論文 Figure 16，在 128 顆 H800 GPU 上實測：在最前或最後兩個階段之間注入 30 或 60 ms，四種模型大小的迭代時間。PipeMorph（紅色、最右的長條）在每一組都最快。Wu et al., NSDI 2026, Figure 16 (USENIX PDF, p. 13).

The production replay is the most realistic test (Figure 4b): the trace is bursty (median 13.0 ms, P99 79.3 ms), and in severe congestion both baselines fell below 30 iterations per minute while PipeMorph stayed above 40.

重播生產環境紀錄是最貼近真實的測試（圖 4b）：這段紀錄是突發的（中位數 13.0 ms、P99 79.3 ms），嚴重壅塞時兩個基準都掉到每分鐘 30 次迭代以下，PipeMorph 則維持在 40 次以上。

Figure 4. Measured results, redrawn from Figures 14 and 22 of the paper: (a) one link delayed 30 ms; (b) a one-hour production delay trace, replayed on 128 H800 GPUs for each method.圖 4。實測結果，依論文 Figures 14 與 22 重畫：(a) 一條鏈路延遲 30 ms；(b) 一小時的生產環境延遲紀錄，對每個方法在 128 顆 H800 GPU 上重播。Drawn for this primer from Wu et al., NSDI 2026, §7.2, §7.5 and Figures 14, 22.

We omit the paper's "up to 2.8×" for fast-changing delays (Figure 18): we could not reproduce it from the figure's values.

我們略去論文對快速變化延遲報告的「最多 2.8 倍」（Figure 18）：無法從該圖的數值重現它。

6. Limits and open questions6. 限制與未解的問題

The paper's own limits: delegation costs up to 16.4% (PCIe copies, slower CPU-side RDMA); rescheduling once per iteration leaves about a 5% gap when latency changes very often; TP stragglers are excluded. A careful reader adds:

論文自己的限制：委派最多帶來 16.4% 的開銷（PCIe 複製、較慢的 CPU 端 RDMA）；每次迭代只重新排程一次，在延遲極頻繁變化時約有 5% 的差距；TP 的 straggler 被排除在外。細心的讀者還應該補上：

Injected delays. The plugin adds fixed latency. Real congestion also cuts bandwidth, so $c$ grows with message size.人為注入的延遲。plugin 加上的是固定延遲。真實的壅塞還會降低頻寬，使 $c$ 隨訊息大小增加。
Dense models only. No interleaving, which would shrink the budget, and no MoE or context parallelism.只有稠密模型。沒有測試會縮小預算的交錯，也沒有 MoE 或上下文平行。
Host side. Host-memory capacity and PCIe contention with the NIC are not measured.主機端。沒有量測主機記憶體容量，也沒有量測與 NIC 之間的 PCIe 爭用。
RNIC fallback. The conclusion claims zero restart overhead on RNIC failures, but no experiment tests it.RNIC 備援。結論宣稱 RNIC 故障時重啟開銷為零，但沒有任何實驗測試它。

7. How it connects7. 與其他內容的關聯

Deep dive, section 4: "PP tolerates long paths" holds for bandwidth. For latency it holds only up to the budget $(\Delta-1)t$ .第 3.5 節：「PP 能忍受長路徑」對頻寬成立。對延遲則只在預算 $(\Delta-1)t$ 以內成立。
Llama 3 traded memory against bubble with the number of in-flight micro-batches. PipeMorph trades memory against delay tolerance with the same knob.Llama 3 用在途 micro-batch 數拿記憶體換氣泡。PipeMorph 用同一個旋鈕拿記憶體換延遲容忍度。
Alibaba HPN lets only PP cross its 15:1 links. PipeMorph measures the limit of that rule, and its DP-to-PP re-mapping applies the rule at run time.Alibaba HPN 只讓 PP 跨越 15:1 鏈路。PipeMorph 量出這條規則的極限，而它把 DP 改對應成 PP，就是在執行期套用這條規則。
The UBEP digest shows the same lesson for expert parallelism: the slowest element sets the pace, and synchronization must leave the critical path.UBEP 導讀在專家平行上看到同樣的教訓：最慢的元素決定步調，同步必須離開關鍵路徑。

Key numbers關鍵數字

Number	數字	Meaning (H800 testbed, injected delay)
8.04× / 2.24×	30 ms on one DP link / one PP link; GPT-2 7B	一條 DP / PP 鏈路加 30 ms；GPT-2 7B
710 → 1,350 ms	HOL blocking, 30 ms on one PP link; 7B	HOL 阻塞，一條 PP 鏈路加 30 ms；7B
$c \le (\Delta-1)t$	Delay absorbed at linear cost (Theorem 1, analysis)	以線性代價吸收的延遲（Theorem 1，分析）
703 → 398 ms	ZB → PipeMorph, 7B, 30 ms on one link	ZB → PipeMorph，7B，一條鏈路 30 ms
4.24× vs 1.13×	Slowdown of ZB vs PipeMorph at 60 ms; 14B	60 ms 時 ZB 與 PipeMorph 的變慢；14B
1.36×	Throughput over ZB, replayed production trace; 140B, 128 GPUs	重播生產環境紀錄時相對 ZB 的吞吐；140B、128 顆 GPU

Terms used on this page本頁用到的術語

ZeroBubble (ZB): A schedule that splits the backward into B (input gradient) and W (weight gradient).; 把反向拆成 B（輸入梯度）與 W（權重梯度）的排程。
Warm-up count $x_i$: Forwards that stage $S_i$ runs before its first backward.; 階段 $S_i$ 在第一次反向之前執行的正向數。
Slackness $\Delta_i$: $x_i - x_{i+1}$ ; the link absorbs up to $(\Delta_i - 1)t$ of delay.; $x_i - x_{i+1}$ ；鏈路最多能吸收 $(\Delta_i - 1)t$ 的延遲。
HOL blocking (kernel level): A blocked NCCL send launch that keeps ready kernels from being issued.; 被擋住的 NCCL send 啟動，使已備妥的 kernel 無法發出。
CPU delegation: Sending PP tensors from pinned host memory by delegate processes with CPU-side RDMA.; 由委派行程以 CPU 端 RDMA 從 pinned 主機記憶體送出 PP 張量。