What training computes, and why it needs thousands of GPUs訓練在算什麼，又為何需要上千顆 GPU

1.1What one training step does一個訓練步驟做了什麼

A language model is a large set of numbers, called parameters. Llama 3 405B, the example this book uses most, has 405 billion of them. The model reads text in pieces called tokens; a token is about three quarters of an English word. Training adjusts the parameters so that the model predicts the next token better. It does so in small steps. Each training step works on one batch: a fixed set of training text, millions of tokens for a large model. The arithmetic runs on GPUs (graphics processing units), chips built to do many multiplications at once. The NVIDIA H100 is the GPU in most of the systems in this book.

語言模型是一大組數字，稱為參數。本書最常用的例子 Llama 3 405B 有 4,050 億個參數。模型以稱為 token（詞元）的小段落讀入文字；一個 token 約為英文單字的四分之三。訓練就是調整參數，讓模型更準確地預測下一個 token。訓練一小步一小步地進行，每一個訓練步驟處理一個批次（batch）：一組固定的訓練文字，大型模型一次有數百萬個 token。這些算術在 GPU（圖形處理器）上執行；GPU 是專為同時做大量乘法而設計的晶片。本書大多數系統用的是 NVIDIA H100。

One training step turns one batch of tokens into one update of every parameter. It has four phases, and they always run in the same order. The forward pass pushes the batch through every layer and produces a loss: one number that says how wrong the model was. The backward pass walks the layers in reverse and produces a gradient for every parameter: how sensitive the loss is to that parameter, which tells which way the parameter should be adjusted. In other words, it is the rate of change of the loss with respect to the parameter, and its opposite direction is the one in which the loss falls fastest locally. The optimizer step reads each gradient and moves the parameter a small distance. The synchronization makes every copy of the model agree on the result. The first three phases need only computation on a single GPU; the fourth involves communication between GPUs. Chapter 5 shows that the first three phases shrink when you add GPUs and the fourth does not; that is why this book is about the network.

一個訓練步把一批 token 變成對每一個參數的一次更新。它有四個階段，順序永遠一樣。前向傳播把這批資料推過每一層，產生損失值：一個表示模型錯多少的數字。反向傳播反向走過各層，為每個參數算出梯度：描述參數對損失有多敏感，並指出參數應往哪個方向調整。換言之，即損失函數對參數的變化率，其反方向是局部降低損失最快的方向。優化器步驟讀取每個梯度，把參數移動一小段距離。同步讓模型的每一份副本對結果取得一致。前三個階段只需要在單顆 GPU 上運算，第四項則牽扯到 GPU 之間的通訊。第 5 章會說明：加 GPU 時前三個階段會縮小，第四個不會；這就是本書談網路的原因。

Section 1.2 shows that one GPU would need thousands of years for a model of this size, so thousands of GPUs share each step. The simplest way to share it is to give every GPU its own copy of the model and a different part of the batch. Each GPU computes gradients for its own part. Before the optimizer step, the gradients from all GPUs must be added together, so that every copy makes the same update and the copies stay identical. An operation called an all-reduce computes this sum: every GPU sends its gradients in, and every GPU receives the total. That exchange is the synchronization phase, and it is the reason the GPUs need a network. This way of splitting the work is called data parallelism. Chapter 3 introduces it properly, together with four other ways to split a job, and Chapter 4 shows how an all-reduce is done. In a program that runs on many GPUs, the process that drives one GPU is called a rank; ranks are numbered 0, 1, 2 and so on.

第 1.2 節會算出：這種規模的模型若只用一顆 GPU，要花上數千年，所以每一步都由數千顆 GPU 分擔。最簡單的分擔方式，是讓每顆 GPU 各持有一份完整的模型，並處理批次中不同的一部分。每顆 GPU 為自己那一部分算出梯度。在優化器步驟之前，必須把所有 GPU 的梯度加總，讓每份副本做出相同的更新、彼此保持一致。這個加總由一種稱為 all-reduce 的操作完成：每顆 GPU 送出自己的梯度，每顆 GPU 都收到總和。這次交換就是同步階段，也是 GPU 需要網路的原因。這種分工方式稱為資料平行。第 3 章會正式介紹它以及另外四種切分方式，第 4 章說明 all-reduce 實際上怎麼做。在跨多顆 GPU 執行的程式中，驅動一顆 GPU 的行程稱為 rank，編號為 0、1、2 等等。

1.2How much arithmetic: the 6ND rule算術有多少：6ND 法則

Training is arithmetic, so the first question is how much of it there is. Count it in FLOPs, floating-point operations: one multiplication or one addition on numbers with a fractional part. A GPU's speed is counted in FLOP per second, FLOP/s; one TFLOP/s is $10^{12}$ of them per second.

訓練就是算術，所以第一個問題是：算術有多少？以 FLOP（浮點運算）計數：對帶小數的數字做一次乘法或一次加法。GPU 的速度以每秒 FLOP 數，即 FLOP/s 計；1 TFLOP/s 是每秒 $10^{12}$ 次。

Start with one weight matrix with $d_{\text{in}}$ inputs and $d_{\text{out}}$ outputs, applied to one token. The forward pass computes $d_{\text{out}}$ sums, each of $d_{\text{in}}$ products: $d_{\text{in}} \times d_{\text{out}}$ multiply-adds, which is $2\,d_{\text{in}}\,d_{\text{out}}$ floating-point operations, or exactly 2 FLOPs per weight. Sum over all the weight matrices in the model and the forward pass costs about $2N$ FLOPs per token, where $N$ is the number of parameters the token passes through. In a dense model that is every parameter. In a mixture-of-experts (MoE) model, each layer holds many small sub-networks and each token passes through only a few of them, so $N$ is the active parameter count, much smaller than the total (Chapter 3). The backward pass does two matrix multiplies of the same size for each one in the forward pass: one to get the gradient with respect to the layer's input, which the layer below needs, and one to get the gradient with respect to the weights. That is $4N$. Together, $6N$ FLOPs per token, and for a run over $D$ tokens

從一個有 $d_{\text{in}}$ 個輸入、$d_{\text{out}}$ 個輸出的權重矩陣開始，作用在一個 token 上。前向算 $d_{\text{out}}$ 個和，每個和有 $d_{\text{in}}$ 個乘積：$d_{\text{in}} \times d_{\text{out}}$ 次乘加，即 $2\,d_{\text{in}}\,d_{\text{out}}$ 次浮點運算，恰好每個權重 2 FLOPs。把模型裡所有權重矩陣加起來，前向每個 token 約 $2N$ FLOPs，$N$ 是這個 token 經過的參數量。dense 模型就是全部參數；混合專家（MoE）模型的每一層含有許多小型子網路，每個 token 只經過其中少數幾個，所以 $N$ 是啟用參數量，遠小於總參數量（第 3 章）。反向對前向的每一次矩陣乘法都做兩次同樣大小的乘法：一次算對該層輸入的梯度（往回的下一層需要它），一次算對權重的梯度。這是 $4N$。合計每個 token $6N$ FLOPs，跑過 $D$ 個 token 就是

Figure 1.1 shows the three products for one weight matrix.

圖 1.1 畫出一個權重矩陣的三次乘法。

where $N$ is the number of parameters and $D$ is the number of training tokens. For Llama 3 405B, with about 15 trillion tokens, this is about $3.8 \times 10^{25}$ floating-point operations. One GPU cannot do this in a human lifetime. The worked example below shows by how much.

其中 $N$ 是參數量，$D$ 是訓練 token 數。以 Llama 3 405B、約 15 兆 token 計算，大約是 $3.8 \times 10^{25}$ 次浮點運算。一顆 GPU 在人的一生之內做不完。下面的算例會說明差距有多大。

The rule ignores the attention scores, which add about $12 \times \text{layers} \times \text{sequence length} \times \text{hidden size}$ per token and matter when sequences are very long. It ignores recomputation, extra work that some systems do to save memory (Chapter 3). It also ignores the optimizer, which touches each parameter a handful of times per step, keeps about 16 bytes of numbers per parameter (Chapter 3 counts them), and is limited by memory speed, not by arithmetic.

這條法則忽略了注意力分數——每個 token 另外約需 $12 \times \text{層數} \times \text{序列長度} \times \text{隱藏維度}$，序列很長時才重要。它忽略重算，也就是某些系統為了省記憶體而多做的工作（第 3 章）。它也忽略優化器：優化器每步只碰每個參數少數幾次、每個參數約保留 16 位元組的數字（第 3 章會逐項計算），受限於記憶體速度而非算術。

To turn FLOPs into time, divide by the rate at which a GPU computes. Its peak rate is the most it can do in theory. The table gives the peaks of three GPU generations for BF16, the 16-bit number format of training (Chapter 3). "Dense" means without the 2:1 structured-sparsity figure that product sheets often quote, which training does not use. Chapter 2 explains the memory columns and the NVLink column, the connections between the GPUs inside one machine.

要把 FLOP 換成時間，就除以 GPU 的運算速率。它的峰值速率是理論上的上限。下表列出三個 GPU 世代在 BF16（訓練用的 16 位元數字格式，第 3 章）下的峰值。「稠密」是指不含產品規格表常引用的 2:1 結構化稀疏數字，訓練用不到那個。第 2 章會解釋記憶體兩欄，以及 NVLink（一台機器內 GPU 之間的連接）一欄。

Now the worked example: Llama 3 405B, pre-trained on about 15.6 trillion tokens. The arithmetic, one question per row; a GPU-hour is one GPU busy for one hour, the unit in which training time is bought:

現在看實例：Llama 3 405B，以約 15.6 兆 token 預訓練。逐列計算如下；GPU 小時是一顆 GPU 忙碌一小時，也是購買訓練時間的單位：

So pure stepping on 16,384 GPUs would take about 67 days. The real run took longer: averaged over everything, each GPU delivered about 340 TFLOP/s, about 34% of peak. The difference is a long-context phase that ran less efficiently (Section 1.3), the restarts after 466 interruptions in 54 days, and the work lost between each failure and the last saved copy of the training state, called a checkpoint (Chapter 9). Keep two numbers from this section: a GPU-year of H100 time at 40% of peak is about $1.2 \times 10^{22}$ FLOP, and a frontier model needs thousands of them.

所以 16,384 顆 GPU 的純步進時間約 67 天。實際的運行更久：全程平均下來，每顆 GPU 只發揮約 340 TFLOP/s，約為峰值的 34%。差距來自效率較低的長上下文階段（第 1.3 節）、54 天內 466 次中斷之後的重啟，以及每次故障與上一份訓練狀態存檔之間損失的工作；這種存檔稱為檢查點（第 9 章）。從這一節記住兩個數字：一個 H100 GPU 年在 40% 峰值下約為 $1.2 \times 10^{22}$ FLOP，而一個前沿模型需要數千個。

Figure 1.2 puts the three per-GPU rates side by side, and Panel 1.1 repeats the calculation for other models and clusters.

圖 1.2 把每顆 GPU 的三種速率並排，面板 1.1 則對其他模型與叢集重做這個計算。

1.3Utilization: how much of the peak a run uses使用率：一次訓練用到多少峰值

The number that tells you how well a run uses its GPUs is MFU, model FLOPs utilization: the tokens per second the job actually trains, multiplied by $6N$, divided by the number of GPUs times their peak FLOP/s. It counts only the arithmetic the model needs. HFU, hardware FLOPs utilization, counts every FLOP the hardware executed, including recomputation. When a system saves memory by keeping fewer intermediate results and recomputing them in the backward pass (activation checkpointing, Chapter 3), it performs an extra forward pass, about $8N$ per token instead of $6N$, and HFU credits it. HFU is therefore always at least MFU, and the gap is extra computation (recomputing the forward pass during the backward pass) traded for less memory use. The PaLM authors, who proposed MFU as the metric to report, gave 46.2% MFU and 57.8% HFU for the same run. The TPU v4 paper quotes the 57.8% figure and reports that TPU v4 sustains about 60% of peak on large language models (TPU v4 guided reading). When you compare systems, check which of the two is being quoted, and against which peak: dense or sparse, BF16 or FP8.

衡量一次訓練把 GPU 用得多好的數字是 MFU（模型 FLOPs 使用率）：任務實際訓練的每秒 token 數乘以 $6N$，除以 GPU 數乘上它們的峰值 FLOP/s。它只計入模型需要的算術。HFU（硬體 FLOPs 使用率）計入硬體執行的每一次 FLOP，包括重算。當系統為了省記憶體而少存中間結果、在反向傳播時重算它們（激活值檢查點，第 3 章），它多做一次前向，每個 token 約 $8N$ 而非 $6N$，HFU 會把它算進去。所以 HFU 永遠不低於 MFU，兩者的差距，就是用額外的運算（在反向傳播時重算前向）換取更少的記憶體使用。提議以 MFU 作為回報指標的 PaLM 作者，對同一次運行給出 46.2% MFU 與 57.8% HFU。TPU v4 論文引用的是 57.8% 這個數字，並回報 TPU v4 在大型語言模型上持續達到約 60% 的峰值（TPU v4 導讀）。比較系統時，要確認引用的是兩者中的哪一個，以及相對於哪個峰值：稠密或稀疏、BF16 或 FP8。

Figure 1.3 shows what each metric counts.

圖 1.3 畫出兩種指標各計入什麼。

Why does MFU remain so far below peak GPU performance? Three factors account for most of the gap. First, GPU execution is imperfect. Small matrix operations fail to fully utilize the hardware, while layer normalization, softmax and optimizer updates are often memory-bound, limited by how fast data moves in and out of GPU memory rather than by arithmetic. Second, communication introduces idle time, because GPUs must wait for data transfers across devices (Chapter 4 and Chapter 5). Third, synchronization amplifies stragglers, GPUs that run persistently slower than the rest. Faster GPUs remain idle until the slowest worker finishes (Section 1.4). These overheads reduce MFU within each training step. Beyond that, checkpointing, data loading and failure recovery introduce additional downtime, including the 466 interruptions recorded over 54 days. Because reported MFU typically excludes this downtime, actual end-to-end training efficiency can be even lower. Effective training time, the share of wall-clock time in which the job makes progress it keeps, measures it separately; it was above 90% for both MegaScale and Llama 3 (Chapter 10). Multiply the two to get the fraction of purchased GPU-time that became model: about $0.41 \times 0.9 \approx 37\%$ for the Llama 3 run, which is consistent with the 34% that the model card's GPU-hours imply.

為什麼 MFU 會遠低於 GPU 的峰值效能？差距大部分來自三個因素。第一，GPU 的執行並不完美。小型矩陣運算無法充分利用硬體，而 layer normalization、softmax 與優化器更新往往受記憶體限制，速度取決於資料進出 GPU 記憶體有多快，而不是算術有多快。第二，通訊帶來閒置時間，因為 GPU 必須等待跨裝置的資料傳輸（第 4 章與第 5 章）。第三，同步會放大落後者（持續比其他 GPU 慢的 GPU）的影響。較快的 GPU 必須閒置，直到最慢的那一個完成（第 1.4 節）。這些開銷都在每個訓練步之內降低 MFU。除此之外，檢查點、資料載入與故障恢復還帶來額外的停機時間，包括 54 天內記錄到的 466 次中斷。由於回報的 MFU 通常不含這些停機時間，實際端到端的訓練效率可能更低。有效訓練時間——工作在牆鐘時間中產生可保留進度的比例——把它分開量；MegaScale 與 Llama 3 都高於 90%（第 10 章）。把兩者相乘就得到買來的 GPU 時間中真正變成模型的比例：Llama 3 這次運行約 $0.41 \times 0.9 \approx 37\%$，與模型卡的 GPU 小時所隱含的 34% 一致。

Figure 1.4 follows the purchased GPU time down to that 37%.

圖 1.4 把買來的 GPU 時間一路追到這 37%。

Here is what one MFU point buys on the $3.79 \times 10^{25}$ FLOP of Llama 3 405B. The price is about $2 per H100-hour, a round number in the range of public cloud rates in 2024–2025, and the power is 700 W per GPU:

一個 MFU 百分點值多少，以 Llama 3 405B 的 $3.79 \times 10^{25}$ FLOP 算給你看。價格以每 H100 小時約 2 美元計——這是 2024–2025 年公有雲價格範圍內的一個整數——功率以每顆 GPU 700 W 計：

One point is therefore about 620,000 GPU-hours, about 38 hours of the whole cluster, about $1.2 million, and about 430 MWh of GPU power alone, before cooling and the rest of the machine. These are the stakes behind the gains in later chapters: 14.9% from a better way of wiring the network at Alibaba (Chapter 6), more than 2× for the exchanges between GPUs from tuning the network and the communication library together at Meta (Chapter 4), and 0.7 points from removing 0.5% of slow machines at ByteDance (Section 1.4). The network is a few percent of a cluster's cost. The utilization it decides is worth far more than the network itself.

所以一個百分點約為 62 萬 GPU 小時，相當於整個叢集約 38 小時，約值 120 萬美元，光是 GPU 電力就約 430 MWh，還不含冷卻與機器的其他部分。這就是後面各章那些增益背後的賭注：Alibaba 靠更好的網路接法得到 14.9%（第 6 章）、Meta 靠同時調校網路與通訊函式庫讓 GPU 之間的交換快了超過 2 倍（第 4 章）、ByteDance 靠移除 0.5% 的慢機器拿回 0.7 個百分點（第 1.4 節）。網路只占叢集成本的幾個百分點。它所決定的使用率，價值遠超過網路本身。

Panel 1.2 recomputes the value of one point for other MFU levels, prices and power draws.

面板 1.2 對其他 MFU、價格與功率重算一個百分點的價值。

1.4Many GPUs, one step: the barrier多顆 GPU、一個步驟：柵欄

Every exchange of this kind is a barrier: a point that no rank can pass until every rank has reached it. The all-reduce cannot produce its result until the last GPU's gradients have arrived, and the optimizer cannot update until the sum is complete. So the time of a step is not the average time of its participants but the maximum. If one GPU in 16,384 is 10% slower, the other 16,383 wait 10% of a step, every step, for as long as that GPU stays in the job. MegaScale found that about 0.5% of its machines were substantially slower than the rest, in one case about 10% longer on identical forward computations, and that removing them regained about 0.7 MFU points and made peak MFU consistent from run to run (MegaScale §6.3). Llama 3 states the rule in one sentence: a single straggler, a rank that is persistently slower than the others, can slow thousands of GPUs. That is why Meta built tools to find the slowest ranks (Llama 3 §3.3.4). The network is one source of such delay. A part of the network that is overloaded makes one rank's data arrive late in exactly the way a slow GPU does, and the rest of the job cannot tell the difference.

每一次這類交換都是一道柵欄（barrier）：所有 rank 都抵達之前，誰也不能通過的點。all-reduce 要等最後一顆 GPU 的梯度到達才能產生結果，優化器要等總和完成才能更新。所以一步的時間不是參與者的平均時間，而是最大值。如果 16,384 顆 GPU 中有一顆慢 10%，其他 16,383 顆每一步都等 10% 的步時間，只要那顆 GPU 還在任務裡就一直如此。MegaScale 發現約 0.5% 的機器明顯比其他慢，其中一例在相同的前向計算上多花約 10%，而移除它們拿回約 0.7 個 MFU 百分點，並讓各次運行的峰值 MFU 變得一致（MegaScale §6.3）。Llama 3 用一句話說出這條規則：單一個 straggler（持續比其他 rank 慢的 rank）就能拖慢數千顆 GPU。所以 Meta 建了工具來找出最慢的 rank（Llama 3 §3.3.4）。網路是這類拖慢的來源之一。網路中某個超載的部分讓某個 rank 的資料晚到，效果和一顆慢 GPU 完全相同，任務的其餘部分分不出差別。

Figure 1.5 draws one step of eight ranks with one straggler.

圖 1.5 畫出八個 rank、其中一個是 straggler 的一步。

Animation 1.1 plays the step of Figure 1.5 in time order: each rank reaches the barrier and waits there.

動畫 1.1 依時間順序播放圖 1.5 的那一步：每個 rank 抵達柵欄，然後在那裡等待。

Random variation adds to this. Every rank's step time varies a little from step to step, and the step waits for the largest of thousands of such values, which grows with the number of ranks. Chapter 10 works out how large that effect is, and how operators find and remove stragglers. The network is part of the same story: a part of the network that is slow for one step delays every rank, exactly as a slow GPU does.

隨機變動會讓情況更糟。每個 rank 的步驟時間每一步都略有不同，而一步要等數千個這種數值中的最大者，這個最大值會隨 rank 數增加而變大。第 10 章會算出這個效應有多大，以及營運者如何找出並移除 straggler。網路也在同一個故事裡：只要網路的某一部分在某一步變慢，所有 rank 都會被拖慢，和一顆慢 GPU 的效果完全一樣。

Guided readings for this chapter本章相關的論文導讀