MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUsMegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

Reading plan (45 minutes)閱讀計畫（45 分鐘）

The Meta and Alibaba papers are network papers. MegaScale is a systems paper that treats the model, the training framework, the collective library, the network and the operations tooling as one object to optimize. For a network reader its value is seeing how the network is used: what the collectives are, when they run, and what the system does when a link flaps. §3 is the performance half, §4–§5 the reliability half, §6 the measurements.

Meta 與阿里的論文是網路論文。MegaScale 是系統論文，把模型、訓練框架、集體通訊函式庫、網路與營運工具當成一個整體來最佳化。對網路讀者而言，它的價值在於看到網路如何被使用：集體操作是什麼、何時執行、鏈路抖動時系統做什麼。§3 是效能的一半，§4–§5 是可靠性的一半，§6 是量測。

1. The problem: scale makes everything worse at once1. 問題：規模讓一切同時變糟

The paper opens with two challenges. The first is efficiency. MFU is the ratio of the throughput you observe to the throughput you would get if every GPU ran at 100% of its peak FLOPs for the whole time. Every microsecond a GPU waits for data from another GPU lowers MFU, and the fraction of time spent waiting grows with the number of GPUs, because the collectives involve more participants and the pipeline has more stages. The second is stability. A run on more than 10,000 GPUs lasts weeks, and in that time hardware fails constantly. Any single failure stops the whole synchronous job. Worse, a machine that is merely slow (a straggler) also stops the job, in a softer way: every other GPU waits for it at every synchronization point. The paper's position is that these two challenges must be solved together and across the entire stack, which is why a single paper covers optimizer choice and NIC retransmit timers.

論文一開始提出兩個挑戰。第一是效率。MFU 是你觀察到的吞吐，與每顆 GPU 全程以 100% 峰值 FLOPs 運行時能得到的吞吐之比。GPU 每等待另一顆 GPU 的資料一微秒，MFU 就下降；而等待時間的比例隨 GPU 數量增加而增長，因為集體操作涉及更多參與者、管線也有更多階段。第二是穩定性。超過 10,000 顆 GPU 的運行持續數週，期間硬體不斷故障。任何單一故障都會讓整個同步任務停下。更糟的是，一台只是比較慢的機器（straggler）也會用較溫和的方式讓任務停擺：其他所有 GPU 在每個同步點都等它。論文的立場是這兩個挑戰必須一起、跨整個堆疊解決，這就是為什麼一篇論文同時涵蓋最佳化器的選擇與 NIC 的重傳計時器。

2. Changing the model to help the system2. 修改模型來幫助系統

Three algorithmic changes reduce either the amount of work or the number of times GPUs must wait for each other. Parallel transformer block: in a standard transformer layer, the attention sub-layer runs first and the feed-forward sub-layer consumes its output. The parallel form feeds both from the same input and sums their outputs, which removes a serial dependency inside every layer and lets the two be computed and communicated together. Sliding-window attention: each token attends only to a fixed window of nearby tokens instead of the whole sequence; stacking layers still lets information travel far, and the attention cost falls. LAMB optimizer: normally, raising the batch size beyond a point hurts final accuracy. LAMB adapts the learning rate per layer and, in their experiments, allowed a 4× larger batch without loss of accuracy. A larger batch means more micro-batches per step, and more micro-batches means the pipeline's warm-up and cool-down bubbles are a smaller fraction of the step; the paper computes an 87.5% reduction in pipeline bubbles. In the ablation these three contribute 4.6, 1.0 and 3.0 MFU points respectively (the ablation used a 3× batch).

三個演算法層面的改動減少了工作量，或減少了 GPU 必須互相等待的次數。平行 transformer 區塊：標準 transformer 層中，注意力子層先跑，前饋子層接著消耗它的輸出。平行形式讓兩者接收同一個輸入並把輸出相加，消除了每一層內的序列依賴，讓兩者可以一起計算、一起通訊。滑動視窗注意力：每個 token 只關注附近固定視窗內的 token 而非整個序列；堆疊多層後資訊仍能傳得很遠，注意力的成本則下降。LAMB 最佳化器：通常批次大小超過某個點就會傷害最終精度。LAMB 逐層調整學習率，在他們的實驗中允許 4 倍的批次而不損失精度。批次更大表示每步有更多 micro-batch，更多 micro-batch 表示管線的暖機與收尾氣泡占一步的比例更小；論文算出管線氣泡減少 87.5%。消融實驗中這三項分別貢獻 4.6、1.0 與 3.0 個 MFU 百分點（消融用的是 3 倍批次）。

3. Overlap in all three dimensions3. 三個維度都做重疊

The paper's central performance idea is that communication should never be the only thing happening. Each parallelism dimension gets its own overlap scheme.

論文效能方面的核心想法是：通訊永遠不該是唯一在發生的事。每個平行化維度都有自己的重疊方案。

Data parallelism (ZeRO-2 style). Optimizer state and gradients are sharded across the data-parallel group, so the usual all-reduce of gradients becomes a reduce-scatter after the backward pass and an all-gather of parameters before the forward pass. MegaScale divides the model into chunks (the same chunks used for interleaved pipelining). The all-gather for one chunk runs while another chunk computes forward; the reduce-scatter for one chunk runs while another computes backward. The all-gather for the first chunk of an iteration has nothing to hide behind, so it is prefetched during data loading at the start of the iteration. Communication is launched in priority order, where priority is set by which computation needs the result soonest.

資料平行（ZeRO-2 風格）。最佳化器狀態與梯度分片到資料平行群組，所以常見的梯度 all-reduce 變成反向之後的 reduce-scatter 與正向之前的參數 all-gather。MegaScale 把模型切成多個 chunk（與交錯式管線用的 chunk 相同）。某個 chunk 的 all-gather 在另一個 chunk 做正向時執行；某個 chunk 的 reduce-scatter 在另一個 chunk 做反向時執行。一次迭代第一個 chunk 的 all-gather 沒有東西可以藏，所以在迭代開始載入資料時預先抓取。通訊依優先權順序發出，優先權由哪個計算最快需要結果決定。

Pipeline parallelism (interleaved 1F1B). In the steady state of a 1F1B schedule each stage alternates one forward and one backward micro-batch, and the point-to-point sends and receives are issued asynchronously so they overlap with the next micro-batch. The warm-up phase is harder: a stage must receive activations before it can compute, but it does not need to wait for its own send to complete. MegaScale decouples send from receive so the send runs in the background while the stage computes on what it already received; the cool-down phase applies the mirror image.

管線平行（交錯式 1F1B）。在 1F1B 排程的穩態中，每個階段交替執行一個正向與一個反向 micro-batch，點對點的 send 與 receive 以非同步方式發出，與下一個 micro-batch 重疊。暖機階段比較難：一個階段必須先收到激活值才能計算，但不需要等自己的 send 完成。MegaScale 把 send 與 receive 解耦，讓 send 在背景執行、階段同時對已收到的資料計算；收尾階段則是鏡像的做法。

Tensor and sequence parallelism. Within a transformer layer, tensor parallelism splits the big matrix multiplications across the 8 GPUs of a server, and sequence parallelism splits the layer-norm and dropout along the sequence; the two require an all-gather before the parallel linear layer and a reduce-scatter after it. MegaScale fuses these collectives with the linear layers on the feed-forward path, where the matrices are largest and give the most computation to hide behind, and cuts each matrix multiply into chunks so that chunk $k$ computes while chunk $k+1$'s data is still in flight. The same is done in the backward pass. In Table 3, the three overlaps together add 6.2 MFU points (TP 2.2, PP 2.5, DP 1.5).

張量與序列平行。在一個 transformer 層內，張量平行把大型矩陣乘法切到伺服器的 8 顆 GPU，序列平行把 layer-norm 與 dropout 沿序列切分；兩者需要在平行線性層之前做 all-gather、之後做 reduce-scatter。MegaScale 把這些集體操作與前饋路徑上的線性層融合——那裡的矩陣最大、有最多計算可供隱藏——並把每個矩陣乘法切成多段，讓第 $k$ 段計算時第 $k+1$ 段的資料仍在傳輸中。反向也一樣處理。在 Table 3 中三種重疊合計增加 6.2 個 MFU 百分點（TP 2.2、PP 2.5、DP 1.5）。

4. Operators, data pipeline, and starting up4. 運算子、資料管線與啟動

Operators. Attention uses FlashAttention-2, which partitions work across GPU thread blocks more evenly; LayerNorm and GeLU are fused into single kernels to cut launch overhead and memory traffic. Data pipeline. Preprocessing of the next step's data runs during the gradient synchronization at the end of the current step. On each machine, the eight GPUs in a tensor-parallel group need identical inputs, so one loader per machine reads data into shared memory and each GPU copies from there, instead of eight loaders reading the same bytes. Initialization. Before the first step, every GPU must join its collective groups. Megatron-LM's default took 1,047 seconds on 2,048 GPUs, because workers registered through a single-threaded blocking TCP key-value store and then ran a global barrier whose cost grew as the square of the worker count. Replacing the store with non-blocking Redis cut the time to 361 s; reordering the group setup to make the barriers linear brought it under 5 s on 2,048 GPUs and under 30 s on more than 10,000. This matters beyond the first step: every automatic restart in §5 pays the initialization cost again.

運算子。注意力使用 FlashAttention-2，它把工作更均勻地分配到 GPU 的執行緒區塊；LayerNorm 與 GeLU 融合成單一 kernel 以減少啟動開銷與記憶體流量。資料管線。下一步資料的前處理在這一步結尾的梯度同步期間執行。每台機器上張量平行群組的八顆 GPU 需要相同的輸入，所以每台機器一個載入器把資料讀進共享記憶體、每顆 GPU 從那裡複製，而不是八個載入器讀同樣的位元組。初始化。第一步之前，每顆 GPU 必須加入它的集體通訊群組。Megatron-LM 預設在 2,048 顆 GPU 上花 1,047 秒，因為 worker 透過單執行緒、阻塞式的 TCP 鍵值儲存註冊，然後執行一個成本隨 worker 數平方增長的全域 barrier。把儲存換成非阻塞的 Redis 後縮到 361 秒；重新排列群組設定讓 barrier 成為線性後，2,048 顆 GPU 低於 5 秒、超過 10,000 顆低於 30 秒。這不只關乎第一步：§5 的每一次自動重啟都要再付一次初始化成本。

5. The network5. 網路

§3.6 is short but every sentence maps onto a problem from the Meta and Alibaba papers. Topology. Switches are built on the Broadcom Tomahawk 4 chip: 25.6 Tb/s, 64 ports of 400 Gb/s, with 32 ports down and 32 up at every layer (1:1). Three layers form a Clos that spans more than 10,000 GPUs. Fewer ECMP collisions. At the ToR, each 400G downlink is split with a cable into two 200G links to servers, so the uplink bandwidth per ToR is twice the downlink bandwidth; collisions among the few large flows then rarely saturate an uplink. This is the same 1:2 under-subscription Meta used as a mitigation. Each server's eight 200G NICs connect to eight different ToR switches (multi-rail), and up to 64 servers share one set of ToRs; the scheduler places communication-heavy ranks under the same ToR set so most traffic crosses one switch. Congestion control. With default DCQCN, all-to-all communication at scale produced excessive PFC pauses and head-of-line blocking. ByteDance built a controller that combines Swift's precise RTT measurement with DCQCN's fast ECN-based response. Link flaps. NICs occasionally go down for a few seconds and recover. NCCL's retransmit timer and retry count were tuned so a flap does not kill the job, and the NIC's adaptive retransmission feature was enabled so retries happen at shorter intervals.

§3.6 很短，但每句話都對應到 Meta 與阿里論文中的某個問題。拓樸。交換器基於 Broadcom Tomahawk 4 晶片：25.6 Tb/s、64 個 400 Gb/s 埠，每層 32 埠向下、32 埠向上（1:1）。三層構成一個橫跨超過 10,000 顆 GPU 的 Clos。更少的 ECMP 碰撞。在 ToR，每個 400G 下行用線纜拆成兩條到伺服器的 200G 鏈路，所以每台 ToR 的上行頻寬是下行的兩倍；少數大流之間的碰撞因此很少塞滿一條上行。這與 Meta 用作緩解的 1:2 欠額認購相同。每台伺服器的八張 200G NIC 接到八台不同的 ToR 交換器（多 rail），最多 64 台伺服器共用一組 ToR；排程器把通訊密集的 rank 放在同一組 ToR 之下，讓多數流量只跨一台交換器。壅塞控制。使用預設 DCQCN 時，大規模的 all-to-all 通訊產生過多的 PFC 暫停與隊頭阻塞。字節跳動建了一個結合 Swift 的精確 RTT 量測與 DCQCN 快速 ECN 反應的控制器。鏈路抖動。NIC 偶爾會斷線幾秒再恢復。NCCL 的重傳計時器與重試次數經過調校，讓一次抖動不會殺死任務，並啟用 NIC 的自適應重傳功能，讓重試以更短的間隔進行。

6. Fault tolerance as a workflow6. 把容錯做成工作流程

A driver process requests resources from Kubernetes and starts one executor per node, which launches the training processes. Each executor runs a daemon that sends a heartbeat to the driver carrying the node's IP and pod name, hardware information, the training process status, its stdout/stderr logs, and RDMA traffic counters. The RDMA counters are the key signal: training traffic is regular, so a sudden drop raises an alert and traffic that stops entirely triggers recovery automatically. When the driver sees an anomaly, or no heartbeat at all, it suspends the job and runs a battery of diagnostics on the suspect nodes: an intra-host loopback test from every NIC to host memory and to the GPUs, a NIC-to-NIC connectivity and bandwidth test, an NCCL all-to-all within the node, and an all-reduce with the neighboring nodes under the same ToR. Nodes that fail are evicted and replaced from a standby pool; then the job restarts from the last checkpoint.

一個driver行程向 Kubernetes 申請資源，並在每個節點啟動一個executor，由它啟動訓練行程。每個 executor 跑一個 daemon，向 driver 送出心跳，內含節點的 IP 與 pod 名稱、硬體資訊、訓練行程狀態、stdout/stderr 日誌，以及 RDMA 流量計數器。RDMA 計數器是關鍵訊號：訓練流量有規律，突然下降就發出警報，完全停止就自動觸發恢復。當 driver 看到異常或完全收不到心跳，它暫停任務並對可疑節點執行一組診斷：每張 NIC 到主機記憶體與 GPU 的主機內迴路測試、NIC 對 NIC 的連通性與頻寬測試、節點內的 NCCL all-to-all，以及與同一 ToR 下鄰近節點的 all-reduce。失敗的節點被逐出並從備用池換新；然後任務從最後的檢查點重啟。

Checkpoints in two stages. Writing a full checkpoint of a 175B model directly to a distributed file system takes long enough to lose noticeable training time, and the file system would be hammered by thousands of GPUs at once. MegaScale first copies each GPU's state to pinned host memory, which takes seconds and pauses training only briefly; a background process then writes from host memory to HDFS without blocking the GPUs. Because the pause is short, checkpoints can be taken more often, which shortens the work lost per failure. On recovery, reading is also made cheap: one worker in each data-parallel group reads its shard from HDFS and broadcasts it to the others over the fast network, so HDFS serves a few readers rather than all of them.

兩階段檢查點。把 175B 模型的完整檢查點直接寫進分散式檔案系統耗時夠久、足以損失可觀的訓練時間，而且數千顆 GPU 同時寫會把檔案系統打垮。MegaScale 先把每顆 GPU 的狀態複製到釘住的主機記憶體，只需數秒、訓練僅短暫暫停；再由背景行程把主機記憶體寫到 HDFS，不阻塞 GPU。因為暫停很短，檢查點可以更頻繁，每次故障損失的工作就更少。恢復時讀取也很省：每個資料平行群組中只有一個 worker 從 HDFS 讀取它的分片，再透過高速網路廣播給其他成員，所以 HDFS 只服務少數讀者而非全部。

What it achieved. In a production run of several weeks on more than 10,000 GPUs, training restarted more than 100 times. Over 90% of the software and hardware faults were handled with no human involvement. Detection plus diagnosis took under 10 minutes on average, and the job caught up to its pre-failure progress within 15 minutes of the last checkpoint, keeping effective training time above 90%.

達到的成果。在超過 10,000 顆 GPU 上數週的生產運行中，訓練重啟超過 100 次。超過 90% 的軟硬體故障在沒有人介入的情況下處理完畢。偵測加診斷平均低於 10 分鐘，任務在最後檢查點之後 15 分鐘內追回故障前的進度，有效訓練時間維持在 90% 以上。

7. Finding the slow machine7. 找出那台慢機器

Even with failures handled, MFU varied between runs of the identical job. Two tools found out why. The CUDA event monitor times critical code segments on every rank using CUDA events rather than host-side timers; since it does not force the GPU to synchronize, it costs almost nothing and can stay on in production. Its heat map of per-rank step times showed that about 0.5% of machines were substantially slower than the rest. Because every collective waits for its slowest member, that 0.5% set the pace for everyone. Excluding those machines made peak MFU consistent run to run. The 3-D parallel visualizer draws the logical topology of tensor, pipeline and data parallel groups and the data flow between ranks. When a collective times out, each healthy rank logs the operation it was waiting on, while the faulty rank has hung and logs nothing; comparing the logs against the topology picture points at the silent rank.

即使故障被處理了，同一個任務不同次運行的 MFU 仍有差異。兩個工具找出了原因。CUDA 事件監控器用 CUDA 事件而非主機端計時器為每個 rank 的關鍵程式段計時；因為它不強迫 GPU 同步，幾乎沒有成本，可以在生產環境常開。它的逐 rank 步時間熱圖顯示約 0.5% 的機器明顯比其他慢。因為每個集體操作都等最慢的成員，那 0.5% 決定了所有人的節奏。排除那些機器後峰值 MFU 在各次運行間變得一致。3-D 平行可視化工具畫出張量、管線與資料平行群組的邏輯拓樸以及 rank 之間的資料流。當集體操作逾時，每個健康的 rank 記錄它正在等待的操作，而故障的 rank 已經卡住、什麼都沒記；把日誌對照拓樸圖就能指出那個沉默的 rank。

Three cases from §6.3. First, some hosts took about 10% longer on identical forward computations; isolating and removing them gave about 0.7 MFU points. Second, MFU declined slowly over a long run because some ranks launched their gradient reduce-scatter later than others, making the fast ranks wait; the cause was irregular Python garbage collection and a few PyTorch operations on the critical path, and removing them stopped the decline. Third, NIC flapping: interfaces went down for several seconds and came back, dropping the packets in flight. The default NCCL timeout expired before the NIC recovered, so the job died; setting a larger timeout explicitly let the job ride through. The root cause was poor signal quality across the NIC, the active optical cable and the switch port, fixed with tighter quality control on cabling.

§6.3 的三個案例。第一，某些主機在相同的正向計算上多花約 10%；隔離並移除它們帶來約 0.7 個 MFU 百分點。第二，長時間運行中 MFU 緩慢下滑，因為某些 rank 比其他 rank 更晚發出梯度 reduce-scatter，讓快的 rank 等待；原因是不規律的 Python 垃圾回收與關鍵路徑上的幾個 PyTorch 操作，移除後下滑就停止。第三，NIC 抖動：介面斷線數秒後恢復，在途封包全部丟失。預設的 NCCL 逾時在 NIC 恢復前就到期，任務因此死亡；明確設定較大的逾時讓任務得以撐過。根本原因是 NIC、主動光纜與交換器埠之間的訊號品質不佳，以更嚴格的線纜品質管控修正。

8. Results8. 結果

Experiments use a 175B model (96 layers, hidden size 12,288, 128 heads, TP=8, PP=8, sequence length 2,048) on NVIDIA Ampere GPUs. Table 2 scales from 256 to 12,288 GPUs: MegaScale reaches 65.3% MFU at 256 GPUs and 55.2% at 12,288, against Megatron-LM's 53.0% and 41.2%; the speedup grows from 1.23× to 1.34× with scale, because Megatron-LM's exposed communication grows faster than MegaScale's. A 530B model (105 layers, PP=35) is also tested. Table 3 is the ablation on 256 GPUs. It starts from 47.7% MFU; each row adds one optimization and its gain in MFU points:

實驗使用 175B 模型（96 層、隱藏維度 12,288、128 個 head、TP=8、PP=8、序列長度 2,048）在 NVIDIA Ampere GPU 上進行。Table 2 從 256 擴展到 12,288 顆 GPU：MegaScale 在 256 顆達到 65.3% MFU、在 12,288 顆達到 55.2%，Megatron-LM 則是 53.0% 與 41.2%；加速比隨規模從 1.23 倍增到 1.34 倍，因為 Megatron-LM 暴露的通訊增長得比 MegaScale 快。另外也測試了 530B 模型（105 層、PP=35）。Table 3 是 256 顆 GPU 的消融：從 47.7% MFU 開始，每一列加上一項最佳化及其 MFU 增益（百分點）：

The total, 65.3%, is the 256-GPU figure of Table 2.

合計 65.3%，正是 Table 2 在 256 顆 GPU 的數字。

9. Critique and context9. 評論與脈絡

MegaScale's contribution is breadth, and breadth is also its limit: no single mechanism is analyzed deeply, and the network section gives no measurements of the Swift+DCQCN controller or the collision rate. Some of the algorithmic changes (sliding-window attention, parallel blocks) alter the model, so the MFU comparison with Megatron-LM is partly a comparison of different models. The results are for a dense model on Ampere GPUs; ByteDance's later MegaScale-MoE paper shows that mixture-of-experts traffic changes the overlap story. What has lasted is the operational recipe: heartbeat with traffic counters, automatic diagnosis, two-stage checkpoints and always-on per-rank timing. The Llama 3 paper reports the same shape of operation at 16K GPUs with a published failure table; read it next.

MegaScale 的貢獻是廣度，廣度也是它的限制：沒有任何單一機制被深入分析，網路章節也沒有給出 Swift+DCQCN 控制器或碰撞率的量測。部分演算法改動（滑動視窗注意力、平行區塊）改變了模型，所以與 Megatron-LM 的 MFU 比較部分是不同模型之間的比較。結果針對 Ampere GPU 上的稠密模型；字節跳動後來的 MegaScale-MoE 論文顯示混合專家流量改變了重疊的故事。留下來的是營運配方：帶流量計數器的心跳、自動診斷、兩階段檢查點與常開的逐 rank 計時。Llama 3 論文報告了 16K 顆 GPU 上同樣形態的營運，並附有公開的故障表；接著讀它。

Key numbers關鍵數字

Terms used in this paper本篇用到的術語

MFU

Model FLOPs utilization: observed throughput divided by the throughput at 100% of peak FLOPs on every GPU.

模型 FLOPs 使用率：觀察到的吞吐除以每顆 GPU 都在 100% 峰值 FLOPs 時的吞吐。

1F1B (interleaved)

A pipeline schedule that alternates one forward and one backward micro-batch per stage; interleaving gives each stage several non-adjacent model chunks.

每個階段交替執行一個正向與一個反向 micro-batch 的管線排程；交錯式讓每個階段擁有數個不相鄰的模型 chunk。

ZeRO-2

Sharding optimizer state and gradients across data-parallel ranks, turning all-reduce into reduce-scatter plus all-gather.

把最佳化器狀態與梯度分片到資料平行的 rank，使 all-reduce 變成 reduce-scatter 加 all-gather。

Straggler

A rank or machine that is slower than its peers; in synchronous training it sets the pace for all of them.

比同儕慢的 rank 或機器；在同步訓練中它決定所有人的節奏。

Two-stage checkpoint

Copy GPU state to pinned host memory first (seconds), then write to HDFS in the background.

先把 GPU 狀態複製到釘住的主機記憶體（數秒），再在背景寫到 HDFS。

Multi-rail

Each of a server's eight NICs connects to a different ToR switch.

伺服器的八張 NIC 各接到不同的 ToR 交換器。

Swift

Google's delay-based congestion control; MegaScale borrows its precise RTT measurement and combines it with DCQCN's ECN response.

Google 基於延遲的壅塞控制；MegaScale 借用它的精確 RTT 量測，並與 DCQCN 的 ECN 反應結合。