Research digest 1 of 3 · Chapter 9: Front end and checkpoints

研究導讀 1 / 3 · 第 9 章：前端網路與檢查點

ByteCheckpoint: A Unified Checkpointing System for Large Foundation Model DevelopmentByteCheckpoint：為大型基礎模型開發打造的統一檢查點系統

NSDI 2025Borui Wan, Mingji Han, Yiyao Sheng et al. (The University of Hong Kong, ByteDance) · paper論文

After reading, you can explain how ByteCheckpoint saves training state once and loads it into any parallel layout, and why its stall is under a second while a save still takes 51 s.讀完後，你能說明 ByteCheckpoint 如何把訓練狀態存一次就能載入任何平行配置，以及為何它的停頓不到一秒，一次儲存卻仍要 51 秒。

≈ 11 min · 1 interactive panel · 5-question quiz≈ 11 分鐘 · 1 個互動面板 · 5 題測驗Chapter 9 ↗第 9 章 ↗Jump to the quiz ↓跳到測驗 ↓

If you read nothing else

只讀這一段也可以

ByteDance's checkpoints rarely load back into the layout that wrote them: in six months its platform needed 34,794 such conversions, called resharding, each an offline job of 593 to 1,870 s. ByteCheckpoint stores every tensor shard with its position in the global tensor, so any new layout reads only the bytes it needs, at load time. It spreads writes over the data-parallel replicas, reads each byte once per data-parallel group, and shares it with an all-to-all on the GPU network. Against PyTorch DCP and Megatron MCP on 32 to 4,800 GPUs, the stall per checkpoint fell 12.13× to 161.50×. In production, a 405B model on 8,960 GPUs stalls 0.59 s per checkpoint, but the checkpoint is safe in storage only after 51.06 s. At that scale, metadata, planning and barriers set the limits, not link bandwidth.

字節跳動的檢查點很少載回寫出它的那個配置：六個月內，平台需要 34,794 次這類轉換，稱為重新分片（resharding），每次都是 593 到 1,870 秒的離線任務。ByteCheckpoint 把每個張量分片連同它在全域張量中的位置一起存下，所以任何新配置都能在載入時只讀它需要的位元組。它把寫入分散到各資料平行副本，每個資料平行群組只讀每個位元組一次，再經 GPU 網路以 all-to-all 分享。在 32 到 4,800 顆 GPU 上與 PyTorch DCP 和 Megatron MCP 相比，每個檢查點的停頓降低 12.13 倍到 161.50 倍。在正式環境中，一個 405B 模型在 8,960 顆 GPU 上每次檢查點停頓 0.59 秒，但檢查點要到 51.06 秒後才安全存進儲存。在這個規模下，決定極限的是中繼資料、規劃與 barrier，而不是鏈路頻寬。

1. Where this paper sits1. 這篇論文的位置

Section 9.3 ("Checkpoint traffic") treats a checkpoint as a burst of bytes. MegaScale's two-stage checkpoint copies GPU state to pinned host memory, so the stall is the per-host bytes divided by the copy rate. After a failure, one worker per data-parallel group reads from HDFS and broadcasts. Three things were missing. Other jobs read a checkpoint in other parallel layouts. The choice of writer decides the slowest writer. And at 10,000 GPUs, the control messages around the bytes (plans, barriers, file metadata) cost as much as the bytes. This paper, from MegaScale's company, measures all three.

第 9.3 節（「檢查點流量」）把檢查點當成一波位元組突發。MegaScale 的兩階段檢查點把 GPU 狀態複製到釘住的主機記憶體，所以停頓是每台主機位元組數除以複製速率。故障後，每個資料平行群組由一個 worker 從 HDFS 讀取再廣播。缺了三件事。其他任務會以別的平行配置讀檢查點。由誰寫入，決定了最慢的寫入者。而在 10,000 顆 GPU 時，位元組周圍的控制訊息（計畫、barrier、檔案中繼資料）花的時間和位元組本身一樣多。這篇來自 MegaScale 同一家公司的論文，量測了這三點。

2. The problem, measured2. 問題，用量測說話

A parallel layout splits a model over GPUs by tensor (TP), pipeline (PP) and data parallelism (DP). Resharding converts saved state from one layout to another. Figure 1 shows why ByteDance needs it all the time.

平行配置（parallel layout）以張量（TP）、管線（PP）與資料平行（DP）把模型切到 GPU 上。重新分片（resharding）把已存的狀態從一種配置轉成另一種。圖 1 說明為什麼字節跳動時時都需要它。

Figure 1. One checkpoint, many readers, after the paper's Figure 2. The counts are six months of resharding events, and the times are the average offline job before ByteCheckpoint (§2.2; Table 1: 1,870.38, 650.34 and 593.21 s). Both are production measurements.圖 1。一份檢查點、許多讀者，依論文 Figure 2 重畫。次數是六個月的重新分片事件，時間是導入 ByteCheckpoint 之前離線任務的平均耗時（§2.2；Table 1：1,870.38、650.34 與 593.21 秒）。兩者都是正式環境的量測。Drawn for this primer from Wan et al., NSDI 2025, Figure 2, §2.2 and Table 1.

Before ByteCheckpoint, each conversion was an offline job (download, convert, upload) while training or evaluation waited, and the largest script had 3,193 lines of Python. Frameworks multiply the work: Megatron-LM jobs averaged 301 GPUs, FSDP jobs 25 and DDP jobs 6 (Table 2), each with its own format. Saving a GPT 175B model on 4,096 GPUs to HDFS took about 200 s, longer than one iteration. An asynchronous save hides the stall, but a checkpoint is a recovery point only once it is in storage. Bursts of file requests also overload the HDFS metadata server, the NameNode.

在 ByteCheckpoint 之前，每次轉換都是一個離線任務（下載、轉換、上傳），訓練或評估就在一旁等，最大的腳本有 3,193 行 Python。框架讓工作量倍增：Megatron-LM 任務平均 301 顆 GPU、FSDP 任務 25 顆、DDP 任務 6 顆（Table 2），各有自己的格式。在 4,096 顆 GPU 上的 GPT 175B 模型存到 HDFS 約要 200 秒，比一次迭代還長。非同步儲存能遮住停頓，但檢查點要進了儲存才是恢復點。大量檔案請求還會壓垮 HDFS 的中繼資料伺服器，也就是 NameNode。

3. The design3. 設計

ByteCheckpoint has two calls, save and load. A planner per framework (Megatron-LM, FSDP, DDP, veScale) turns the framework's sharding into a plan. One execution engine per worker runs any plan, and the storage backends (memory, local disk, HDFS, NAS) share one interface.

ByteCheckpoint 只有兩個呼叫：save 與 load。每個框架（Megatron-LM、FSDP、DDP、veScale）各有一個規劃器，把框架的分片方式轉成計畫。每個 worker 上一個執行引擎，執行任何計畫；各儲存後端（記憶體、本地磁碟、HDFS、NAS）共用一個介面。

Layer diagram: user code of Megatron-LM, DDP, FSDP and veScale; the ByteCheckpoint API; four planners; one execution engine; memory, disk, HDFS and NAS I/O back ends. — Figure 4 of the paper: ByteCheckpoint's layers (a design diagram). One API serves four frameworks; only the planner layer knows each framework's sharding, and the execution engine and the storage I/O back ends are shared.論文 Figure 4：ByteCheckpoint 的分層（設計示意圖）。同一套 API 服務四種框架；只有規劃器這一層知道各框架的分片方式，執行引擎與儲存 I/O 後端則共用。Screenshot of Wan et al., NSDI 2025, Figure 4 (USENIX PDF).

3.1 Shards in global coordinates3.1 以全域座標記錄分片

Each saved shard gets a ShardMeta record (tensor name, n-D offsets, n-D lengths), which is its place in the global tensor, and a ByteMeta record (file, byte offset, size). All records go into one global metadata file. At load time, a new rank intersects its own box with the saved boxes. In one dimension, a saved shard $[a_s, a_s+\ell_s)$ and a needed range $[a_t, a_t+\ell_t)$ share

每個已存分片有一筆 ShardMeta 紀錄（張量名稱、n 維偏移、n 維長度），也就是它在全域張量中的位置，以及一筆 ByteMeta 紀錄（檔案、位元組偏移、大小）。所有紀錄放進一個全域中繼資料檔。載入時，新 rank 拿自己的方塊與已存的方塊求交集。在一維中，已存分片 $[a_s, a_s+\ell_s)$ 與所需範圍 $[a_t, a_t+\ell_t)$ 的共同部分是

\big[\max(a_s,\, a_t),\ \min(a_s+\ell_s,\, a_t+\ell_t)\big)

and an n-D overlap applies this in each dimension. ByteMeta turns the overlap into a byte range of one file, and the rank reads only that range (Figure 2, left). The same files serve every later layout.

n 維的重疊就是每一維各取這個區間。ByteMeta 把重疊轉成某個檔案的一段位元組範圍，rank 只讀那一段（圖 2 左）。同一批檔案服務之後的每一種配置。

ZeRO-style sharding breaks the boxes: Megatron's distributed optimizer and FSDP flatten a group's tensors, concatenate them and cut equal slices, so a slice can end mid-row. FSDP with DCP all-gathers such tensors before saving. ByteCheckpoint instead stores each irregular tensor slice as several regular boxes (Figure 2, right): 0.21 s and 0.19 s on 32 and 64 GPUs, against 4.12 s and 5.84 s with the all-gather (Table 7). The text calls these times "microsecond-level"; the table shows about 0.2 s.

ZeRO 式的分片會打破方塊：Megatron 的分散式優化器與 FSDP 把一個群組的張量攤平、串接，再切成等長的切片，所以切片可能在一列中間結束。FSDP 搭配 DCP 會在儲存前對這類張量做 all-gather。ByteCheckpoint 則把每個不規則張量切片存成幾個規則方塊（圖 2 右）：在 32 與 64 顆 GPU 上花 0.21 秒與 0.19 秒，用 all-gather 則是 4.12 秒與 5.84 秒（Table 7）。正文說這些時間是「微秒等級」；表上是約 0.2 秒。

Figure 2. Left: new rank 1 at TP = 4 needs rows 2–3 of tensor W, which lie inside file f0, so it reads one byte range of f0. Right: the paper's Figure 7 example; flatten-and-slice cuts tensor B in the middle of row 1, and each slice is stored as two regular boxes.圖 2。左：TP = 4 的新 rank 1 需要張量 W 的第 2–3 列，它們都在檔案 f0 裡，所以只讀 f0 的一段位元組範圍。右：論文 Figure 7 的例子；攤平後切片在第 1 列中間切斷張量 B，每個切片存成兩個規則方塊。Drawn for this primer from Wan et al., NSDI 2025, §3.2–3.3 and Figures 6–8.

3.2 Load-time resharding, step by step3.2 載入時重新分片，一步一步

Every rank calls load and reads the global metadata file.每個 rank 呼叫 load，讀取全域中繼資料檔。
Each planner finds the saved shards that overlap its new shards.每個規劃器找出與自己新分片重疊的已存分片。
A coordinator (rank 0) gathers the local plans and removes redundant reads.協調者（rank 0）收集各本地計畫，去掉重複的讀取。
The coordinator scatters the final plans.協調者把最終計畫分送回去。
Each engine reads, deserializes, copies to the GPU and runs the all-to-all.每個引擎讀檔、反序列化、複製到 GPU，再執行 all-to-all。
An asynchronous, tree-shaped barrier confirms that the load is complete.一個非同步、樹狀的 barrier 確認載入已完成。

Dataloader state is split the same way. If DP changes, each worker's token buffers are split or merged, so that no sample is dropped or read twice. Panel 1 runs the overlap rule and the read rule of the next section.

資料載入器的狀態也同樣拆開。若 DP 改變，每個 worker 的 token 緩衝區被切開或合併，不丟掉任何樣本，也不重複讀取。面板 1 讓你操作重疊規則與下一節的讀取規則。

Panel 1面板 1

Who reads which bytes at load time載入時誰讀哪些位元組

Preset預設情境

Saved TP degree存檔時的 TP

New TP degree新的 TP

New DP degree新的 DP

Read rule讀取規則

New ranks per stage

每個階段的新 rank 數

Saved files per new shard

每個新分片涉及的檔案

Byte-range reads

位元組範圍讀取次數

Read from storage

從儲存讀出

Largest read by one rank

單一 rank 的最大讀取量

Received by all-to-all, per rank

每個 rank 經 all-to-all 收到

One tensor of one pipeline stage, drawn in global coordinates. The saved files split it by the saved TP degree. Each new rank needs $1/\mathrm{TP}$ of it, and every DP replica needs the same part. Filled: bytes that the rank reads from storage, coloured by the file they come from. Outline only: bytes that arrive by the all-to-all. Sizes are fractions of one copy of the tensor, because the paper gives no checkpoint sizes. Defaults: the paper's Figure 8 (saved at TP 2, DP 2; loaded at TP 4, DP 2). The second preset is the measured 70B change (TP 4, PP 8, DP 75 → 150). Try a new TP above the saved TP, then raise DP and watch the storage reads stay at one copy.

一個管線階段中的一個張量，以全域座標畫出。已存的檔案依存檔時的 TP 切開它。每個新 rank 需要其中的 $1/\mathrm{TP}$ ，而每個 DP 副本需要同一部分。實心：該 rank 從儲存讀出的位元組，顏色代表來源檔案。只有外框：經 all-to-all 收到的位元組。大小以「一份張量」的比例表示，因為論文沒有給檢查點大小。預設值：論文的 Figure 8（以 TP 2、DP 2 儲存；以 TP 4、DP 2 載入）。第二個預設情境是實測的 70B 配置變更（TP 4、PP 8、DP 75 → 150）。試著把新的 TP 設得比存檔時大，再提高 DP，看儲存讀取量維持在一份。

3.3 Spread the writes and the reads over the data-parallel group3.3 把寫入與讀取分散到資料平行群組

Under DP, every replica holds the same model state, and MCP and DCP let the first replica write all of it. ByteCheckpoint gives each replicated shard to the replica $r$ that has the fewest bytes so far, $S_r$ :

在 DP 下，每個副本持有相同的模型狀態，而 MCP 與 DCP 讓第一個副本全部寫入。ByteCheckpoint 把每個複製的分片交給目前累積位元組 $S_r$ 最少的副本 $r$ ：

r^{\star} = \arg\min_{r \,\in\, \text{DP group}} S_r, \qquad S_{r^{\star}} \leftarrow S_{r^{\star}} + \lvert \text{shard} \rvert

The paper calls this worst-fit. At rank 0 of the 70B model, the model-state upload took 6.67 s at DP = 75 and 2.20 s at DP = 150 (Table 9). The save speedup over MCP grew from 2.21× at 2,400 GPUs to 8.87× at 4,800 GPUs.

論文稱之為 worst-fit。在 70B 模型的 rank 0 上，模型狀態的上傳在 DP = 75 時花 6.67 秒，DP = 150 時花 2.20 秒（Table 9）。相對 MCP 的儲存加速，從 2,400 顆 GPU 時的 2.21 倍增加到 4,800 顆時的 8.87 倍。

The read side mirrors this (Figure 3). One rank of each DP group reads a given byte from storage, the group splits the work evenly, and an all-to-all inside the group gives every replica its full shard. In the paper's words, idle inter-GPU bandwidth replaces redundant storage reads. MegaScale used one reader and a broadcast; ByteCheckpoint balances the reads.

讀取端是鏡像（圖 3）。同一個位元組在每個 DP 群組只由一個 rank 從儲存讀出，群組平均分攤工作，再由群組內的 all-to-all 讓每個副本拿到完整分片。用論文的話說，閒置的 GPU 間頻寬取代了重複的儲存讀取。MegaScale 用一個讀者加廣播；ByteCheckpoint 把讀取平均分攤。

Figure 3. One DP group of 4 replicas; numbers are units of replicated state. Worst-fit cuts the slowest writer from 4 units to 1. On load, storage serves each byte once, and the all-to-all on the GPU network fills in the rest.圖 3。一個有 4 個副本的 DP 群組；數字是複製狀態的單位。worst-fit 讓最慢的寫入者從 4 單位降到 1 單位。載入時，儲存對每個位元組只服務一次，其餘由 GPU 網路上的 all-to-all 補齊。Drawn for this primer from Wan et al., NSDI 2025, §4.1 and Table 9.

The save pipeline copies GPU state into pinned host memory with two alternating ("ping-pong") buffers; parallel processes then serialize, dump and upload it. Only the copy blocks training. Planning for the 405B model on 8,960 GPUs took 62 s, so plans are cached after the first save.

儲存管線把 GPU 狀態複製到釘住的主機記憶體，使用兩個輪流的（「乒乓」）緩衝區；接著由平行的行程序列化、寫入共享記憶體、上傳。只有複製會阻塞訓練。405B 模型在 8,960 顆 GPU 上的規劃要 62 秒，所以計畫在第一次儲存後就被快取。

Gantt chart of save phases on rank 0 for model, optimizer, loader and extra state; bytecheckpoint.save() returns at a red dashed line after planning and device-to-host copies. — Figure 12 of the paper: one save on rank 0 (901 MB in all), drawn by ByteCheckpoint's monitoring from measured phase times. Training waits only until save() returns at the red dashed line, after planning and the device-to-host copies; serialization, dumps and uploads continue in the background. Read the bars: the tooltip's numbers do not agree with each other (672.08 MB in 223 ms is not 332.22 MB/s).論文 Figure 12：rank 0 的一次儲存（共 901 MB），由 ByteCheckpoint 的監控依實測的各階段時間畫出。訓練只等到 save() 在紅色虛線處返回，也就是規劃與從裝置到主機的複製完成時；序列化、寫入共享記憶體與上傳都在背景繼續。請看長條：提示框裡的數字彼此不一致（223 毫秒傳 672.08 MB 並不是 332.22 MB/s）。Screenshot of Wan et al., NSDI 2025, Figure 12 (USENIX PDF).

3.4 HDFS and the control path at 10,000 GPUs3.4 HDFS 與 10,000 顆 GPU 時的控制路徑

HDFS reads at any offset but writes only by appending. Multi-threaded range reads raised one file from 400 MB/s to 2–3 GB/s (H800 server, 200 Gb/s NIC). Parallel sub-files, joined by a metadata-only concatenation, write one file at 3 GB/s. ByteDance also rewrote HDFS in C++ and added a NameNode proxy; the deployed system serves about 10 TB/s and 100,000 metadata operations per second.

HDFS 可以從任意偏移讀取，寫入卻只能附加。多執行緒的範圍讀取讓單一檔案從 400 MB/s 提高到 2–3 GB/s（H800 伺服器、200 Gb/s NIC）。平行寫入子檔案、再以只動中繼資料的串接合併，單一檔案可寫到 3 GB/s。字節跳動還用 C++ 重寫了 HDFS，並加了一層 NameNode 代理；部署中的系統提供約 10 TB/s，每秒約 100,000 次中繼資料操作。

The control path failed first. At 8,960 GPUs, NCCL-based gather and scatter of plans set up channels slowly, needed a connection to every GPU and ran out of GPU memory. gRPC fixed the memory use, and a tree (GPUs → local rank 0 → group roots → coordinator) removed the central hotspot. A similar tree replaced Megatron's integrity barrier, which stalled about 20 s at about 10,000 GPUs.

最先出問題的是控制路徑。在 8,960 顆 GPU 時，用 NCCL 收集與分送計畫，建立 channel 很慢、需要連到每一顆 GPU，還用光了 GPU 記憶體。改用 gRPC 解決了記憶體問題，再用一棵樹（GPU → 本地 rank 0 → 群組的根 → 協調者）去掉中央熱點。類似的樹也取代了 Megatron 用 torch.distributed 做的完整性 barrier，它在約 10,000 顆 GPU 時停頓約 20 秒。

4. Implementation4. 實作

ByteCheckpoint is about 20,000 lines of Python on PyTorch DCP, with an export to Safetensors. Monitoring records duration, I/O size, rank and step for every operation; it drew the timeline above, and per-rank heat maps such as the paper's Figure 11, where the ranks that hold dataloader state save longest. The paper does not say which network carries HDFS, gRPC and barrier traffic; the all-to-all uses the inter-GPU links.

ByteCheckpoint 約 20,000 行 Python，建在 PyTorch DCP 之上，可匯出 Safetensors。監控為每個操作記錄耗時、I/O 大小、rank 與步數；上面的時間軸就是它畫的，還有像論文 Figure 11 那樣的各 rank 熱圖，圖中持有資料載入器狀態的 rank 存得最久。論文沒有說 HDFS、gRPC 與 barrier 流量走哪個網路；all-to-all 使用 GPU 間的鏈路。

5. What was measured5. 量到了什麼

Setup: vDiT, a 4B video model, uses FSDP (ZeRO-2) on A100s and reshards from 32 to 64 and from 128 to 64 GPUs. tGPT, a 70B text model, uses Megatron-LM on H800s at TP = 4, PP = 8 and DP = 75 or 150 (2,400 or 4,800 GPUs). Runs take 500 steps and save to HDFS every 100. The baselines, DCP and MCP, save only GPU state. $T_{\text{Block}}$ is the stall, $T_{\text{Save}}$ ends at the integrity check, $T_{\text{Load}}$ loads the same layout and $T_{\text{Reshard}}$ a new one (Table 4, seconds):

設定：vDiT 是 4B 的影片模型，在 A100 上用 FSDP（ZeRO-2），從 32 顆重新分片到 64 顆、從 128 顆到 64 顆。tGPT 是 70B 的文字模型，在 H800 上用 Megatron-LM，TP = 4、PP = 8，DP = 75 或 150（2,400 或 4,800 顆 GPU）。每次執行 500 步，每 100 步存到 HDFS。基準 DCP 與 MCP 只存 GPU 狀態。 $T_{\text{Block}}$ 是停頓， $T_{\text{Save}}$ 到完整性檢查結束為止， $T_{\text{Load}}$ 載入同一配置， $T_{\text{Reshard}}$ 載入新配置（Table 4，單位秒）：

vDiT 4B: DCP → ByteCheckpointvDiT 4B：DCP → ByteCheckpoint	32 GPUs32 顆 GPU	128 GPUs128 顆 GPU
$T_{\text{Block}}$	16.25 → 0.54 30.09×	61.37 → 0.38 161.50×
$T_{\text{Save}}$	86.82 → 27.47 3.16×	236.34 → 23.74 9.96×
$T_{\text{Load}}$	50.12 → 11.69 4.29×	105.74 → 12.01 8.80×
$T_{\text{Reshard}}$	74.89 → 16.01 4.68×	91.01 → 13.64 6.67×

tGPT 70B: MCP → ByteCheckpointtGPT 70B：MCP → ByteCheckpoint	2,400 GPUs2,400 顆 GPU	4,800 GPUs4,800 顆 GPU
$T_{\text{Block}}$	4.73 → 0.39 12.13×	4.70 → 0.36 13.06×
$T_{\text{Save}}$	28.97 → 13.11 2.21×	76.21 → 8.59 8.87×
$T_{\text{Load}}$	69.87 → 49.48 1.41×	123.80 → 64.39 2.00×
$T_{\text{Reshard}}$	126.30 → 62.10 2.03×	64.62 → 54.31 1.19×

The stall range is 12.13× to 161.50×; the evaluation section's "13.06× to 161.50×" leaves out the 2,400-GPU row. The abstract's 54.20× is the mean of the four stall ratios, and its 6.05× (save) and 3.88× (load) are means of ratios too.

停頓降幅的範圍是 12.13 倍到 161.50 倍；評估一節寫的「13.06 倍到 161.50 倍」漏掉了 2,400 顆 GPU 那一列。摘要的 54.20 倍是四個停頓比值的平均，6.05 倍（儲存）與 3.88 倍（載入）也是比值的平均。

With dataloader state, which the baselines cannot save, the 70B reshard took 401.21 s at 2,400 GPUs: token buffers reach 20 GB, and only some workers hold them. In ablations on 13B and 30B models (Tables 5–6), the asynchronous pipeline alone made saves 1.45× and 1.81× faster, and balancing raised this to 2.48× and 2.46×. A 175B production job on 2,080 H800 GPUs resumed with bit-wise identical loss.

若包含基準無法儲存的資料載入器狀態，70B 模型在 2,400 顆 GPU 上重新分片要 401.21 秒：token 緩衝區可達 20 GB，而且只有部分 worker 持有它們。在 13B 與 30B 模型的消融實驗中（Tables 5–6），單靠非同步管線，儲存快 1.45 倍與 1.81 倍，加上平衡提高到 2.48 倍與 2.46 倍。一個在 2,080 顆 H800 GPU 上的 175B 正式任務恢復後，損失逐位元相同。

Production (Table 8, Figure 4): a 7B vision model with FSDP on 1,488 GPUs, and a 405B text model (TP = 8, DP = 70, PP = 16) on 8,960 GPUs. Only the stall is under a second; the text's "typically within a few seconds" does not match the 20.13 s and 51.06 s saves. Parallel uploads of small dataloader files (once 73.16% of the 7B save time) and NameNode fixes (worst metadata cost per file write 3 s → 150 ms) mattered most.

正式環境（Table 8、圖 4）：一個以 FSDP 在 1,488 顆 GPU 上訓練的 7B 視覺模型，以及一個在 8,960 顆 GPU 上的 405B 文字模型（TP = 8、DP = 70、PP = 16）。只有停頓不到一秒；正文的「通常在幾秒內」與 20.13 秒、51.06 秒的儲存時間不符。最重要的修正是平行上傳小的資料載入器檔案（曾佔 7B 模型儲存時間的 73.16%），以及修改 NameNode（每次檔案寫入最壞的中繼資料成本 3 秒 → 150 毫秒）。

Figure 4. Table 8's production measurements on one time axis. Red (drawn 3 px wide): the only time training waits. Amber: the time until the checkpoint can restore the job. Blue: the read cost of a restart.圖 4。Table 8 的正式環境量測畫在同一條時間軸上。紅色（畫成 3 px 寬）：訓練唯一等待的時間。琥珀色：檢查點能用來恢復任務之前的時間。藍色：重啟時讀取的成本。Drawn for this primer from Wan et al., NSDI 2025, Table 8.

6. Limits and open questions6. 限制與未解的問題

The paper's only stated gap is the dataloader, whose large token buffers still slow full-state resharding. A careful reader should add:

論文明說的缺口只有資料載入器：它的大型 token 緩衝區仍讓完整狀態的重新分片很慢。細心的讀者還應補上：

Stall is not durability. At 8,960 GPUs, training resumes after 0.59 s, but the recovery point exists 51 s later, and a restart reads for 129 s. $T_{\text{Block}}$ is Chapter 9's Young–Daly $C$ ; $T_{\text{Save}}$ puts a floor under the interval.停頓不等於持久。在 8,960 顆 GPU 時，訓練 0.59 秒後就恢復，但恢復點要 51 秒後才存在，而重啟要讀 129 秒。 $T_{\text{Block}}$ 是第 9 章 Young–Daly 的 $C$ ； $T_{\text{Save}}$ 則是間隔的下限。
ETTR is pessimistic by construction. Table 4 also lists an effective training time ratio from Appendix C. It assumes one failure in every interval and counts the save as lost, so it stays below 50% (28.80–48.92%). Read it as a relative score.ETTR 在構造上就悲觀。Table 4 還列出依附錄 C 算出的有效訓練時間比。它假設每個間隔都有一次故障，並把儲存時間算成損失，所以一定低於 50%（28.80–48.92%）。請把它當成相對分數。
Unstated network, costs and baselines. Whether HDFS and gRPC traffic share links with training, the pinned memory, and the metadata size are not given. The authors added the baselines' HDFS access themselves, and no baseline ran beyond 4,800 GPUs.沒說的網路、成本與基準。HDFS 與 gRPC 流量是否與訓練共用鏈路、釘住的記憶體用量、中繼資料大小，論文都沒有給。基準的 HDFS 存取是作者自己加的，而且沒有任何基準跑超過 4,800 顆 GPU。

Bandwidth is not the limit. With Chapter 9's 14 bytes per parameter (the paper gives no size) and 8 GPUs per server:

頻寬不是極限。用第 9 章的每參數 14 位元組（論文沒有給大小），並假設每台伺服器 8 顆 GPU：

Question問題	Calculation算式	Result結果
Checkpoint of the 405B model (our assumption: 14 bytes per parameter)405B 模型的檢查點（我們的假設：每參數 14 位元組）	$405 \times 10^{9} \times 14$	$\approx 5.67$ TB
Average rate during the 51.06 s save51.06 秒儲存期間的平均速率	$5.67\,\text{TB} \div 51.06\,\text{s}$	$\approx 111$ GB/sabout 1.1% of the 10 TB/s HDFS figure約為 HDFS 10 TB/s 的 1.1%
Per server, 8,960 GPUs ÷ 8 = 1,120 servers每台伺服器，8,960 顆 GPU ÷ 8 = 1,120 台	$111\,\text{GB/s} \div 1{,}120$	$\approx 99$ MB/sunder 1 Gb/s on a 200 Gb/s NIC在 200 Gb/s NIC 上不到 1 Gb/s

Per-file metadata, planning and stragglers set the 51 s, which is the paper's own production lesson.

決定那 51 秒的是每個檔案的中繼資料、規劃與落後者，這正是論文自己在正式環境學到的教訓。

7. How it connects7. 與其他內容的關聯

Chapter 9: section 3's two-stage checkpoints and recovery reads in production form; section 4's HDFS limits, engineered around. Against section 1's rule, the recovery all-to-all runs on the back end, but at load time, when no collective waits.第 9 章：第 3 節的兩階段檢查點與恢復時的讀取，在這裡成為正式環境的形式；第 4 節的 HDFS 限制，在這裡被繞過。相對於第 1 節的規則，恢復時的 all-to-all 走後端，但發生在載入時，沒有集體通訊在等。
Chapter 10, section 3: $T_{\text{Block}}$ sets the stall term of the lost time; detection and restart set the rest.第 10.3 節： $T_{\text{Block}}$ 決定損失時間中的停頓項；偵測與重啟決定其餘部分。
Guided readings: MegaScale (same company, the design this extends), Llama 3 (Tectonic sized for bursty checkpoint writes) and Alibaba HPN (checkpoints every 2 to 4 hours).論文導讀：MegaScale（同一家公司，這篇延伸的設計）、Llama 3（為突發的檢查點寫入而設計的 Tectonic）與 Alibaba HPN（每 2 到 4 小時存一次檢查點）。
Next digest: Checkmate removes the copy out of GPU memory altogether.下一篇導讀：Checkmate 完全拿掉從 GPU 記憶體往外的複製。

Key numbers關鍵數字

Number	數字	Meaning and setup
34,794 · 593–1,870 s	Resharding events in six months · average offline job before ByteCheckpoint; production	六個月的重新分片事件 · 導入前離線任務的平均耗時；正式環境
12.13–161.50×	Stall reduction vs DCP and MCP, 32 to 4,800 GPUs	相對 DCP 與 MCP 的停頓降幅，32 到 4,800 顆 GPU
0.59 / 51.06 / 129.49 s	Stall / save / load, 405B model on 8,960 GPUs; production	停頓 / 儲存 / 載入，8,960 顆 GPU 上的 405B 模型；正式環境
6.67 → 2.20 s	Rank-0 model-state upload, DP 75 → 150 (70B model)	rank 0 的模型狀態上傳，DP 75 → 150（70B 模型）
62 s	Uncached save planning, 405B on 8,960 GPUs; measured run time	未快取的儲存規劃，8,960 顆 GPU 上的 405B；實測執行時間
2–3 GB/s · 3 GB/s	One-file HDFS read (from 400 MB/s) · write; 200 Gb/s NIC	單檔 HDFS 讀（原為 400 MB/s）· 寫；200 Gb/s NIC
≈20 s · 3 s → 150 ms	Barrier stall at ≈10,000 GPUs · worst metadata cost per file write, before → after	約 10,000 顆 GPU 時的 barrier 停頓 · 每次檔案寫入最壞的中繼資料成本，修正前 → 後

Terms used on this page本頁用到的術語

Resharding: Converting saved state from one parallel layout to another.; 把已存的狀態從一種平行配置轉成另一種。
ShardMeta / ByteMeta: A shard's place in the global tensor / its place in a file.; 分片在全域張量中的位置／它在檔案中的位置。
Irregular tensor: A ZeRO slice that is not one box of the original tensor.; 不是原張量中單一方塊的 ZeRO 切片。
Worst-fit balancing: Giving each replicated shard to the replica with the fewest bytes so far.; 把每個複製的分片交給目前位元組最少的副本。
$T_{\text{Block}}$ / $T_{\text{Save}}$: The training stall of one save / the time until the checkpoint passes its integrity check.; 一次儲存的訓練停頓／檢查點通過完整性檢查所需的時間。
NameNode: The HDFS metadata server.; HDFS 的中繼資料伺服器。