Glossary名詞表

Grouped by the chapter that defines each term. Every name links to the sentence where the term is introduced.依定義它的章分組。每個名稱都連到介紹該名詞的那一句。

No term matches.沒有符合的名詞。

01What training computes訓練在算什麼

Parameter參數參數ParameterOne number in the model that training adjusts. A large language model has billions of them.模型中由訓練調整的一個數字。大型語言模型有數十億個參數。
Token詞元（token）詞元（token）TokenA piece of text, about three quarters of a word, that the model reads as one unit.文字的一小段，約四分之三個英文單字，模型把它當成一個單位讀入。
Batch批次（batch）批次（batch）BatchThe set of training examples (tokens) that one training step processes.一個訓練步驟處理的一組訓練資料（詞元）。
Training step訓練步驟訓練步驟Training stepOne forward pass, one backward pass and one optimizer step over one batch. A run repeats it millions of times.對一個批次做一次前向傳播、一次反向傳播與一次最佳化器更新。整個訓練重複它數百萬次。
Forward pass前向傳播前向傳播Forward passThe model reads the batch layer by layer and computes its prediction and its error (the loss).模型逐層讀入批次，算出預測與誤差（損失）。
Loss損失（loss）損失（loss）LossOne number that says how wrong the model's prediction was.表示模型預測錯得多嚴重的一個數字。
Backward pass反向傳播反向傳播Backward passThe model walks the layers in reverse and computes a gradient for every parameter.模型反向走過各層，為每個參數算出梯度。
Gradient梯度梯度GradientHow sensitive the loss is to each parameter: the rate of change of the loss with respect to it. Its opposite direction lowers the loss fastest locally.描述損失對每個參數有多敏感，也就是損失函數對參數的變化率；其反方向是局部降低損失最快的方向。
Optimizer step優化器步驟優化器步驟Optimizer stepThe part of a step that applies the gradients to the parameters. Adam is the common optimizer.把梯度套用到參數上的那一段。常用的優化器是 Adam。
FLOP, FLOP/sFLOP、FLOP/sFLOP、FLOP/sFLOP, FLOP/sA floating-point operation (one multiply or one add). FLOP/s is the rate; a TFLOP/s is 10¹² per second.一次浮點運算（一次乘法或加法）。FLOP/s 是速率；1 TFLOP/s 是每秒 10¹² 次。
GPUGPUGPUGPUGraphics processing unit: the chip that does the arithmetic of training. An NVIDIA H100 is one.圖形處理器：負責訓練中算術運算的晶片，例如 NVIDIA H100。
Peak rate峰值速率峰值速率Peak rateThe most FLOP/s a GPU can do in theory. Real training reaches 30–60% of it.GPU 理論上最多能做的 FLOP/s。實際訓練約達到其 30–60%。
Synchronization同步同步SynchronizationThe part of a step in which GPUs exchange results so that every copy of the model stays the same.一個步驟中 GPU 交換結果、讓每份模型副本保持一致的那一段。
RankrankrankRankOne process of a distributed job, numbered 0, 1, 2 …; in training, one rank drives one GPU.分散式工作中的一個行程，編號 0、1、2……；訓練中一個 rank 驅動一顆 GPU。
Barrier柵欄（barrier）柵欄（barrier）BarrierA point that no rank may pass until every rank has reached it. The step lasts as long as the slowest rank.所有 rank 都抵達前，誰都不能通過的點。步驟時間由最慢的 rank 決定。
Straggler落後者（straggler）落後者（straggler）StragglerA rank or machine that is persistently slower than the others, and so sets the pace of the whole job.持續比其他 rank 慢的 rank 或機器，因而決定整個工作的速度。
MFUMFUMFUMFUModel FLOPs utilization: the arithmetic the model needs, per second, divided by the GPUs' peak rate.模型 FLOPs 使用率：模型所需的運算量每秒除以 GPU 峰值速率。
HFUHFUHFUHFUHardware FLOPs utilization: like MFU, but it also counts recomputed work, so HFU ≥ MFU.硬體 FLOPs 使用率：與 MFU 類似，但也計入重算的運算，所以 HFU ≥ MFU。
GPU-hourGPU 小時GPU 小時GPU-hourOne GPU busy for one hour; the unit in which training runs are bought and priced.一顆 GPU 忙碌一小時；訓練用量以此單位購買與計價。

02The hardware硬體

Server (host, node)伺服器（主機、節點）伺服器（主機、節點）Server (host, node)One computer in the cluster. A training server holds 8 GPUs, 2 CPUs and 8–9 network cards.叢集中的一台電腦。訓練伺服器有 8 顆 GPU、2 顆 CPU 與 8–9 張網路卡。
HBMHBMHBMHBMHigh-bandwidth memory: the GPU's own memory, stacked beside the chip; about 80 GB at 3.35 TB/s on an H100.高頻寬記憶體：疊在 GPU 晶片旁的專屬記憶體；H100 約 80 GB、3.35 TB/s。
NVLink, NVSwitchNVLink、NVSwitchNVLink、NVSwitchNVLink, NVSwitchNVIDIA's GPU-to-GPU links inside a server, and the switch chips that join them; about 450 GB/s per GPU each way on H100.NVIDIA 在伺服器內連接 GPU 的鏈路，以及把它們接在一起的交換晶片；H100 每顆 GPU 每個方向約 450 GB/s。
High-bandwidth domain (HBD)高頻寬域（HBD）高頻寬域（HBD）High-bandwidth domain (HBD)The set of GPUs joined by NVLink: one 8-GPU server, or a whole rack in systems such as NVL72.以 NVLink 相連的一組 GPU：一台 8-GPU 伺服器，或 NVL72 這類系統中的整個機櫃。
PCIePCIePCIePCIeThe bus that connects GPUs and network cards to the CPUs inside a server; about 64 GB/s each way for Gen5 x16.伺服器內連接 GPU、網路卡與 CPU 的匯流排；Gen5 x16 每個方向約 64 GB/s。
NICNIC（網路卡）NIC（網路卡）NICNetwork interface card: the device that sends and receives a server's network traffic; 400 Gb/s per GPU in current clusters.網路介面卡：負責伺服器網路收發的裝置；現行叢集每顆 GPU 配 400 Gb/s。
RDMARDMARDMARDMARemote direct memory access: the NIC reads and writes memory on another server without copies through the CPU.遠端直接記憶體存取：NIC 直接讀寫另一台伺服器的記憶體，不經 CPU 複製。
GPUDirect RDMAGPUDirect RDMAGPUDirect RDMAGPUDirect RDMARDMA straight into GPU memory: the NIC moves data to and from HBM over PCIe with no copy in host memory.直接進出 GPU 記憶體的 RDMA：NIC 經 PCIe 讀寫 HBM，不經主機記憶體複製。
Queue pair (QP)佇列對（QP）佇列對（QP）Queue pair (QP)One RDMA connection: a send queue and a receive queue whose state the NIC keeps.一條 RDMA 連線：一個送出佇列加一個接收佇列，狀態存在 NIC 內。
Packet, MTU封包、MTU封包、MTUPacket, MTUA packet is one unit of data on the wire. The MTU is the largest payload one packet may carry (4 KB for RDMA).封包是線路上的一個資料單位。MTU 是一個封包能攜帶的最大資料量（RDMA 為 4 KB）。
Link鏈路（link）鏈路（link）LinkOne cable between two ports, with its rate in each direction, such as 400 Gb/s.兩個埠之間的一條線纜，具有各方向的速率，例如 400 Gb/s。
Switch, port交換器、埠交換器、埠Switch, portA switch forwards packets between its ports; a port is one socket for one link.交換器在各埠之間轉送封包；埠是接一條鏈路的插口。
Radix基數（radix）基數（radix）RadixThe number of ports on one switch chip at a given speed: 128 × 400G for a 51.2 Tb/s chip.一顆交換晶片在某速率下的埠數：51.2 Tb/s 晶片為 128 × 400G。
SerDes laneSerDes 通道SerDes 通道SerDes laneOne high-speed serial signal; a port is 4 or 8 lanes, and a chip's capacity is lanes × lane rate.一路高速串列訊號；一個埠由 4 或 8 個通道組成，晶片容量是通道數 × 通道速率。
Breakout分接（breakout）分接（breakout）BreakoutSplitting one fast port into several slower links, such as one 400G port into two 200G links.把一個高速埠拆成多條較慢的鏈路，例如一個 400G 埠拆成兩條 200G。
Switch buffer (shallow, deep)交換器緩衝（淺、深）交換器緩衝（淺、深）Switch buffer (shallow, deep)Memory in a switch that holds packets waiting for a busy port: tens of MB on one chip (shallow), gigabytes in a chassis (deep).交換器內暫存等待忙碌埠之封包的記憶體：單晶片為數十 MB（淺），機框式為數 GB（深）。
Transceiver, optics光模組光模組Transceiver, opticsThe plug-in module that turns a port's electrical signal into light and back; about 9 W and $199 in one paper's model.把埠的電訊號轉成光、再轉回來的可插拔模組；某論文模型中約 9 W、199 美元。
DAC, AOC, multi-mode, single-modeDAC、AOC、多模、單模DAC、AOC、多模、單模DAC, AOC, multi-mode, single-modeCable types by reach: copper (about 3 m), active optical cable and multi-mode fiber (about 100 m), single-mode fiber (500 m and more).依距離分的線纜：銅線（約 3 m）、主動光纜與多模光纖（約 100 m）、單模光纖（500 m 以上）。
Link flap鏈路抖動（link flap）鏈路抖動（link flap）Link flapA link that goes down and comes back within seconds, usually from marginal optics.在數秒內斷線又恢復的鏈路，通常因光學元件品質不佳。
Latency延遲延遲LatencyThe time before the first byte arrives, as opposed to how fast bytes flow once they start.第一個位元組抵達前的時間，相對於開始後位元組流動的速度。
Round-trip time (RTT)往返時間（RTT）往返時間（RTT）Round-trip time (RTT)The time for a packet to reach the far end and for the answer to come back; about 22 µs in Meta's fabric.封包抵達對端、回覆再傳回所需的時間；Meta 的網路約 22 µs。
Bandwidth-delay product (BDP)頻寬延遲乘積（BDP）頻寬延遲乘積（BDP）Bandwidth-delay product (BDP)Link rate × round-trip time: the bytes a sender must have in flight to keep a link busy; 1.1 MB at 400 Gb/s and 22 µs.鏈路速率 × 往返時間：要讓鏈路保持忙碌，傳送端必須在途的位元組數；400 Gb/s、22 µs 時為 1.1 MB。
Roofline, arithmetic intensityRoofline、算術強度Roofline、算術強度Roofline, arithmetic intensityTime ≥ max(operations ÷ peak rate, bytes ÷ bandwidth). Intensity is operations per byte; the balance point is where the two terms tie.時間 ≥ max(運算量 ÷ 峰值速率, 位元組 ÷ 頻寬)。強度是每位元組的運算量；兩項相等處是平衡點。
Scale-up, scale-outScale-up、scale-outScale-up、scale-outScale-up, scale-outScale-up is the fast fabric inside a high-bandwidth domain (NVLink); scale-out is the network between servers.Scale-up 指高頻寬域內的快速互連（NVLink）；scale-out 指伺服器之間的網路。
Flow流（flow）流（flow）FlowA stream of packets between two endpoints over one connection; in training, often one queue pair.兩個端點之間經一條連線傳送的一串封包；訓練中常對應一個佇列對。
DPUDPUDPUDPUData processing unit: a NIC with its own processor cores for storage, networking and virtualization work.資料處理單元：帶有自身處理器核心的 NIC，用於儲存、網路與虛擬化工作。
Ethernet, InfiniBandEthernet、InfiniBandEthernet、InfiniBandEthernet, InfiniBandThe two network families used for training: Ethernet, the data-center standard, and InfiniBand, built for RDMA.訓練所用的兩大網路家族：資料中心標準 Ethernet，與為 RDMA 而設計的 InfiniBand。
Bytes in flight在途位元組在途位元組Bytes in flightData sent but not yet acknowledged. A link stays busy only if bytes in flight ≥ the BDP.已送出但尚未確認的資料。只有在途位元組 ≥ BDP，鏈路才會保持忙碌。
Hop跳（hop）跳（hop）HopOne switch that a packet crosses on its way.封包途中經過的一台交換器。
Uplink, downlink上行、下行上行、下行Uplink, downlinkA switch's ports toward the tier above (uplinks) and toward hosts or the tier below (downlinks).交換器朝上一層的埠（上行）與朝主機或下一層的埠（下行）。
Oversubscription超額認購超額認購OversubscriptionBandwidth entering a tier from below divided by bandwidth leaving upward; 1:1 is non-blocking.從下方進入一層的頻寬除以向上離開的頻寬；1:1 為非阻塞。
Go-back-N, selective retransmissionGo-back-N、選擇性重傳Go-back-N、選擇性重傳Go-back-N, selective retransmissionTwo ways to recover a lost packet: resend it and everything after it (go-back-N), or resend only the missing one (selective).兩種遺失恢復方式：重送遺失封包及其後所有封包（go-back-N），或只重送缺的那一個（選擇性重傳）。
Congestion, congestion control壅塞、壅塞控制壅塞、壅塞控制Congestion, congestion controlA link or port asked to carry more than its rate, so a queue grows; congestion control slows the senders.鏈路或埠被要求承載超過其速率，佇列因而增長；壅塞控制會讓傳送端降速。

03Splitting the model切分模型

Mixed precision (BF16, FP32, FP8)混合精度（BF16、FP32、FP8）混合精度（BF16、FP32、FP8）Mixed precision (BF16, FP32, FP8)Doing the arithmetic in 16-bit numbers (BF16, 2 bytes) while keeping 32-bit copies (FP32, 4 bytes) for the optimizer.以 16 位元數字（BF16，2 位元組）做運算，同時為優化器保留 32 位元副本（FP32，4 位元組）。
Optimizer state最佳化器狀態最佳化器狀態Optimizer stateThe extra numbers the optimizer keeps per parameter: for Adam, an FP32 master weight and two moments, 12 bytes.優化器為每個參數保留的額外數字：Adam 為一個 FP32 主權重與兩個動量，共 12 位元組。
Activations激活值（activation）激活值（activation）ActivationsThe intermediate results of each layer, kept from the forward pass because the backward pass needs them.每一層的中間結果；反向傳播需要它們，所以前向傳播時必須保存。
Activation checkpointing激活值檢查點（activation checkpointing）激活值檢查點（activation checkpointing）Activation checkpointingKeeping only some activations and recomputing the rest in the backward pass; saves memory for about one extra forward pass.只保存部分激活值，其餘在反向傳播時重算；以約一次額外的前向傳播換取記憶體。
Micro-batch微批次（micro-batch）微批次（micro-batch）Micro-batchA slice of the batch that one GPU group processes at a time; a step runs many of them.一個 GPU 群組一次處理的批次切片；一個步驟會跑很多個。
Parallelism dimension, degree平行化維度與程度平行化維度與程度Parallelism dimension, degreeA way to split a training job over GPUs (DP, TP, PP, CP, EP). Its degree is how many GPUs share one split.把訓練工作分散到 GPU 的方式（DP、TP、PP、CP、EP）。程度是共同分擔一份切分的 GPU 數。
Data parallelism (DP)資料平行（DP）資料平行（DP）Data parallelism (DP)Every GPU group holds the same model and reads a different part of the batch; their gradients are summed every step.每個 GPU 群組持有相同模型、讀取批次的不同部分；每一步都要把梯度加總。
Collective集體操作（collective）集體操作（collective）CollectiveA communication operation in which every rank of a group takes part, such as an all-reduce.群組中每個 rank 都參與的通訊操作，例如 all-reduce。
All-reduceAll-reduceAll-reduceAll-reduceA collective in which every rank ends with the sum of all ranks' buffers. It averages DP gradients.每個 rank 最後都得到所有 rank 緩衝區總和的集體操作；用於平均 DP 梯度。
Reduce-scatterReduce-scatterReduce-scatterReduce-scatterA collective in which every rank ends with the sum of one slice (1/N) of all buffers.每個 rank 最後得到所有緩衝區某一切片（1/N）總和的集體操作。
All-gatherAll-gatherAll-gatherAll-gatherA collective in which each rank starts with one slice and every rank ends with all slices.每個 rank 起初各有一個切片、最後都擁有全部切片的集體操作。
All-to-allAll-to-allAll-to-allAll-to-allA collective in which every rank sends a different block to every other rank.每個 rank 都送不同資料區塊給其他每個 rank 的集體操作。
Send/receive (point-to-point)Send/receive（點對點）Send/receive（點對點）Send/receive (point-to-point)A message from one rank to one other rank, as between pipeline stages.從一個 rank 送到另一個 rank 的訊息，例如管線階段之間。
ZeRO, FSDPZeRO、FSDPZeRO、FSDPZeRO, FSDPData parallelism that divides (shards) optimizer state, gradients and weights among the DP group; FSDP is PyTorch's ZeRO-3.把最佳化器狀態、梯度與權重分散（分片）到 DP 群組的資料平行；FSDP 是 PyTorch 的 ZeRO-3。
Shard分片（shard）分片（shard）ShardOne GPU's slice of a tensor or of the model state when it is divided among GPUs.張量或模型狀態被分散到多顆 GPU 時，一顆 GPU 持有的那一片。
Tensor parallelism (TP)張量平行（TP）張量平行（TP）Tensor parallelism (TP)Each weight matrix of a layer is split across GPUs; the partial results are combined by collectives inside every layer.把每層的權重矩陣切到多顆 GPU；每一層內都要用集體操作合併部分結果。
Sequence parallelism (SP)序列平行（SP）序列平行（SP）Sequence parallelism (SP)Splitting some per-token work of a layer along the sequence across the TP group; turns each all-reduce into an all-gather plus a reduce-scatter.把層內部分逐詞元的工作沿序列分給 TP 群組；把 all-reduce 變成 all-gather 加 reduce-scatter。
Pipeline parallelism (PP), stage管線平行（PP）、階段管線平行（PP）、階段Pipeline parallelism (PP), stageThe layers are divided into consecutive stages on different GPUs, and activations are passed from stage to stage.把各層分成連續的階段放在不同 GPU 上，活化值在階段之間傳遞。
Pipeline bubble管線氣泡管線氣泡Pipeline bubbleIdle time at the start and end of a pipelined step, while stages wait for the first or last micro-batch.管線化步驟開頭與結尾的閒置時間，各階段等待第一個或最後一個微批次。
1F1B, interleaved schedule1F1B、交錯排程1F1B、交錯排程1F1B, interleaved schedulePipeline schedules: 1F1B alternates one forward and one backward micro-batch; interleaving gives each GPU several chunks of layers.管線排程：1F1B 交替一個前向與一個反向微批次；交錯排程讓每顆 GPU 持有多段層。
Context parallelism (CP)上下文平行（CP）上下文平行（CP）Context parallelism (CP)One long sequence is split across GPUs; attention gathers the keys and values of the whole sequence.把一條長序列切到多顆 GPU；注意力運算要收集整條序列的 key 與 value。
Mixture of experts (MoE), expert混合專家（MoE）、專家混合專家（MoE）、專家Mixture of experts (MoE), expertA model whose feed-forward layers are many small networks (experts); each token uses only a few of them.前饋層由許多小網路（專家）組成的模型；每個詞元只使用其中少數幾個。
Router, top-k路由器（router）、top-k路由器（router）、top-kRouter, top-kThe small layer that scores the experts for each token and keeps the best k of them.為每個詞元替專家打分、保留最好的 k 個的小型層。
Active parameters啟用參數啟用參數Active parametersThe parameters one token actually uses; compute follows them, memory follows the total.一個詞元實際用到的參數；運算量看它，記憶體看總數。
Expert parallelism (EP)專家平行（EP）專家平行（EP）Expert parallelism (EP)Different experts sit on different GPUs, so tokens are sent to their experts and back with all-to-alls.不同專家放在不同 GPU 上，詞元以 all-to-all 送到專家再送回。
Dispatch, combine分派（dispatch）、合併（combine）分派（dispatch）、合併（combine）Dispatch, combineThe two all-to-alls of an MoE layer: tokens to their experts, then expert outputs back.MoE 層的兩次 all-to-all：把詞元送到專家，再把專家輸出送回。
Local rank本地 rank本地 rankLocal rankA GPU's position inside its server, 0 to 7. GPUs with the same local rank in different servers often talk to each other.GPU 在其伺服器內的位置，0 到 7。不同伺服器中本地 rank 相同的 GPU 常互相通訊。
Model replica模型副本模型副本Model replicaOne complete copy of the model, spread over a TP × PP group of GPUs; DP joins the replicas.一份完整的模型，分布在一個 TP × PP 的 GPU 群組上；DP 連接各個副本。
Traffic matrix流量矩陣流量矩陣Traffic matrixA table of how many bytes each GPU sends to each other GPU in one step.記錄一個步驟中每顆 GPU 送給其他每顆 GPU 多少位元組的表格。
Critical path關鍵路徑關鍵路徑Critical pathThe chain of steps that must finish one after another; anything on it adds directly to the step time.必須依序完成的一串工作；位於其上的任何東西都直接加到步驟時間。

04Collective communication集體通訊

Ring all-reduce環狀 all-reduce環狀 all-reduceRing all-reduceRanks pass chunks to their neighbor around a ring: a reduce-scatter then an all-gather. Each sends about 2× its buffer.各 rank 沿環把資料塊傳給鄰居：先 reduce-scatter 再 all-gather。每個 rank 約送出緩衝區的 2 倍。
Bandwidth-optimal頻寬最優頻寬最優Bandwidth-optimalAn algorithm whose busiest rank sends the fewest bytes possible; for all-reduce, 2(N−1)/N times the buffer.最忙的 rank 送出最少位元組的演算法；all-reduce 為緩衝區的 2(N−1)/N 倍。
Tree, recursive doubling and halving樹、遞迴加倍與減半樹、遞迴加倍與減半Tree, recursive doubling and halvingAlgorithms with about log₂N steps instead of 2(N−1): fewer steps, more bytes per step.約 log₂N 步、而非 2(N−1) 步的演算法：步數少，每步位元組多。
Rabenseifner's algorithmRabenseifner 演算法Rabenseifner 演算法Rabenseifner's algorithmRecursive halving then doubling: the ring's byte count in only 2 log₂N steps, on a network where all pairs can talk at full rate.先遞迴減半、再遞迴加倍：只用 2 log₂N 步達到環狀的位元組數；需要所有配對都能全速通訊的網路。
α–β cost modelα–β 成本模型α–β 成本模型α–β cost modelTime of a collective ≈ α × (dependent steps) + bytes sent ÷ bandwidth; α is the latency of one step.集體操作時間 ≈ α × （相依步數）+ 送出位元組 ÷ 頻寬；α 是每一步的延遲。
Hierarchical all-reduce階層式 all-reduce階層式 all-reduceHierarchical all-reduceReduce-scatter inside each server over NVLink, all-reduce one slice across servers among same-local-rank GPUs, then all-gather inside.先在伺服器內以 NVLink 做 reduce-scatter，再在各伺服器本地 rank 相同的 GPU 間 all-reduce 一個切片，最後在伺服器內 all-gather。
NCCLNCCLNCCLNCCLNVIDIA's collective communication library; PyTorch calls it for every collective.NVIDIA 的集體通訊函式庫；PyTorch 的每個集體操作都呼叫它。
Communicator, channel通訊器（communicator）、通道（channel）通訊器（communicator）、通道（channel）Communicator, channelA communicator is one group of ranks; a channel is one independent ring or tree inside it, with its own buffers and connection.通訊器是一組 rank；通道是其中一個獨立的環或樹，有自己的緩衝區與連線。
Clear-to-send (CTS), receiver-driven admission允許傳送（CTS）、接收端准入允許傳送（CTS）、接收端准入Clear-to-send (CTS), receiver-driven admissionA small message in which the receiver names a free buffer slot; the sender may write only after it, which caps bytes in flight.接收端指出一個空閒緩衝槽的小訊息；傳送端收到後才能寫入，因此限制了在途位元組。
Overlap, exposed communication重疊、暴露的通訊重疊、暴露的通訊Overlap, exposed communicationRunning communication while the GPU computes; the part that outlasts the computation is exposed and adds to the step time.在 GPU 運算時同時進行通訊；比運算更久的部分即「暴露」，會加到步驟時間。
algbw, busbwalgbw、busbwalgbw、busbwalgbw, busbwThe two numbers nccl-tests reports: buffer size ÷ time (algbw), and that rescaled by the algorithm's bytes so it compares with the link rate (busbw).nccl-tests 報告的兩個數字：緩衝區大小 ÷ 時間（algbw），以及依演算法位元組換算後可與鏈路速率比較的值（busbw）。
SM (streaming multiprocessor)SM（串流多處理器）SM（串流多處理器）SM (streaming multiprocessor)One of the ~130 compute units inside a GPU. NCCL runs each channel on one SM, taking it from computation.GPU 內約 130 個運算單元之一。NCCL 每個通道占用一個 SM，從運算中拿走。
In-network reduction網內歸約網內歸約In-network reductionSwitches that add the data themselves, so each rank sends its buffer up once and receives the sum once.由交換器自己做加總，每個 rank 只需送出一次緩衝區、接收一次總和。

05The step-time budget步驟時間帳

Strong scaling強擴展強擴展Strong scalingRunning the same job (same global batch) on more GPUs: compute per GPU falls, DP bytes per GPU do not.在更多 GPU 上跑同一個工作（全域批次不變）：每顆 GPU 的運算下降，DP 位元組不變。
Elephant flow, low entropy大象流、低熵大象流、低熵Elephant flow, low entropyA flow large enough to fill a link by itself; low entropy means few distinct flows for switches to spread.大到能獨自塞滿一條鏈路的流；低熵指交換器可分散的不同流很少。
Burstiness, duty cycle突發性、工作週期突發性、工作週期Burstiness, duty cycleTraining traffic is on at full rate or off; the duty cycle is the share of a step a NIC spends sending.訓練流量不是全速就是靜止；工作週期是 NIC 在一個步驟中傳送的時間比例。

06Topology拓樸

PXNPXNPXNPXNNCCL's PCIe × NVLink path: data first hops over NVLink to the GPU on the destination's rail, whose NIC sends it.NCCL 的 PCIe × NVLink 路徑：資料先經 NVLink 移到目的地那條 rail 上的 GPU，再由它的 NIC 送出。
Topology拓樸拓樸TopologyHow the switches and links are wired together.交換器與鏈路如何接在一起。
Clos network, fat treeClos 網路、胖樹（fat tree）Clos 網路、胖樹（fat tree）Clos network, fat treeTiers of switches in which every switch of one tier connects to every switch of the next, giving many equal paths.多層交換器，每層的每台都連到下一層的每台，提供許多等價路徑。
Leaf, spine, core, tierLeaf、spine、core、層（tier）Leaf、spine、core、層（tier）Leaf, spine, core, tierA leaf (top-of-rack) switch connects hosts; spines connect leaves; a core tier connects groups of spines.Leaf（機櫃頂）交換器接主機；spine 連接 leaf；core 層再連接多組 spine。
ToRToRToRToRTop-of-rack switch: the leaf switch that the servers of a rack (or a rail group) plug into.機櫃頂交換器：一個機櫃（或一組 rail）的伺服器所接的 leaf 交換器。
Non-blocking非阻塞非阻塞Non-blockingEvery host can send at full rate at once, because each tier has as much bandwidth up as down.每台主機能同時全速傳送，因為每層向上的頻寬等於向下的頻寬。
Bisection bandwidth對分頻寬對分頻寬Bisection bandwidthThe smallest total bandwidth across any cut of the GPUs into two equal halves.把 GPU 切成相等兩半的任何切法中，跨越切口的最小總頻寬。
Pod, segmentPod、segmentPod、segmentPod, segmentA pod is one two-tier Clos joined to others by a thinner tier; a segment is HPN's tier-1 group of 1,024 GPUs.Pod 是一個兩層 Clos，以較細的一層與其他 pod 相連；segment 是 HPN 第一層的 1,024 顆 GPU 群組。
RailRailRailRailAll GPUs with the same local rank across servers, and the network that joins them.跨伺服器中本地 rank 相同的所有 GPU，以及連接它們的網路。
Rail-optimized wiringRail 最佳化接線Rail 最佳化接線Rail-optimized wiringNIC k of every server goes to the same leaf (rail k), so same-rank traffic crosses one switch; spines join the rails.每台伺服器的第 k 張 NIC 接到同一台 leaf（rail k），同 rank 流量只過一台交換器；spine 把各 rail 連起來。
Rail-onlyRail-onlyRail-onlyRail-onlyA proposal that keeps one full network per rail and removes the spine; cross-rail traffic first crosses NVLink.每條 rail 保留一個完整網路、移除 spine 的提案；跨 rail 流量先經 NVLink。
Dual-ToR雙 ToR（dual-ToR）雙 ToR（dual-ToR）Dual-ToREach NIC has two ports wired to two different ToR switches, so one switch or link failure does not cut the host off.每張 NIC 的兩個埠接到兩台不同的 ToR，單一交換器或鏈路故障不會讓主機斷線。
Dual-plane雙平面（dual-plane）雙平面（dual-plane）Dual-planeThe tier above the ToRs is split into two separate halves, and each ToR of a pair connects to only one of them.ToR 之上的那一層分成兩個獨立的半邊，一對 ToR 各只連到其中一半。
Aggregation layer聚合層聚合層Aggregation layerHPN's name for the second tier of switches, above the ToRs.HPN 對 ToR 之上第二層交換器的稱呼。
AI Zone (RTSW, CTSW, ATSW)AI Zone（RTSW、CTSW、ATSW）AI Zone（RTSW、CTSW、ATSW）AI Zone (RTSW, CTSW, ATSW)Meta's non-blocking two-tier block of rack switches (RTSW) and cluster switches (CTSW); aggregator switches (ATSW) join the zones.Meta 的非阻塞兩層區塊，由機櫃交換器（RTSW）與叢集交換器（CTSW）組成；聚合交換器（ATSW）連接各 zone。
Torus環面（torus）環面（torus）TorusChips wired to their neighbors in a grid whose rows wrap around; TPU v4 uses three dimensions, six links per chip.晶片在網格中與鄰居相連、每列首尾相接；TPU v4 用三維，每顆晶片六條鏈路。
Optical circuit switch (OCS)光路交換器（OCS）光路交換器（OCS）Optical circuit switch (OCS)A switch of tiny mirrors that connects one fiber to another; it never reads packets and changes connections in milliseconds.以微鏡連接兩條光纖的交換器；不讀封包，可在毫秒級改變連線。
Slice切片（slice）切片（slice）SliceThe set of TPU cubes given to one job and wired into one torus by the OCSes.分配給一個工作、由 OCS 接成一個環面的一組 TPU cube。
ICIICIICIICIInter-chip interconnect: the direct links between TPU chips.晶片間互連：TPU 晶片之間的直接鏈路。
DragonflyDragonflyDragonflyDragonflyA topology family from HPC: groups of routers, each group joined to every other group by a long link.源自 HPC 的拓樸家族：路由器分成群組，每個群組以長鏈路連到其他每個群組。
Multi-plane fat-tree (MPFT)多平面 fat tree（MPFT）多平面 fat tree（MPFT）Multi-plane fat-tree (MPFT)Separate fabrics per NIC index: GPU i and NIC i of every node attach to plane i, and no switch joins two planes.依 NIC 編號分開的網路：每個節點的 GPU i 與 NIC i 接到平面 i，沒有交換器連接兩個平面。
Node-limited routing節點受限路由節點受限路由Node-limited routingAn MoE rule that sends each token's experts to at most M nodes, so a token crosses the network at most M times.MoE 規則：每個詞元的專家最多分布在 M 個節點，詞元最多跨網路 M 次。

07Transport傳輸層

RoCEv2RoCEv2RoCEv2RoCEv2RDMA over Converged Ethernet: RDMA packets carried inside UDP/IP so that ordinary Ethernet switches can forward them.融合乙太網路上的 RDMA：把 RDMA 封包包在 UDP/IP 內，讓一般 Ethernet 交換器可以轉送。
Packet sequence number (PSN)封包序號（PSN）封包序號（PSN）Packet sequence number (PSN)The number in each RDMA packet that loss recovery uses to find gaps.每個 RDMA 封包中的編號，遺失恢復用它找出缺口。
Memory registration, MTT記憶體註冊、MTT記憶體註冊、MTTMemory registration, MTTBefore RDMA, a buffer is registered: pinned, and its address map loaded into the NIC's memory translation table (MTT).RDMA 前必須註冊緩衝區：鎖定記憶體，並把位址對應載入 NIC 的記憶體轉換表（MTT）。
Goodput有效吞吐（goodput）有效吞吐（goodput）GoodputUseful data delivered per second, after resent and wasted bytes are removed.扣掉重送與浪費的位元組後，每秒交付的有用資料量。
Lossless network無損網路無損網路Lossless networkA network whose switches pause senders instead of dropping packets when a buffer fills.緩衝區滿時，交換器會暫停傳送端而不丟封包的網路。
PFCPFCPFCPFCPriority flow control: a switch port sends PAUSE upstream before its buffer overflows, per traffic class.優先權流量控制（PFC）：交換器埠在緩衝區溢位前，依流量類別向上游送出 PAUSE。
Headroom預留空間（headroom）預留空間（headroom）HeadroomBuffer reserved for the bytes that still arrive after a PAUSE has been sent.為送出 PAUSE 後仍會抵達的位元組所保留的緩衝空間。
Congestion spreading, victim flow壅塞擴散、受害者流壅塞擴散、受害者流Congestion spreading, victim flowPFC side effects: pauses spread upstream hop by hop and slow flows that cross no congested link.PFC 的副作用：暫停逐跳往上游擴散，拖慢沒經過壅塞鏈路的流。
IncastIncast（多對一匯聚）Incast（多對一匯聚）IncastMany senders converge on one receiver at once, so one port receives more than it can drain.許多傳送端同時匯聚到一個接收端，一個埠收到的超過它能送出的。
ECN, CNPECN、CNPECN、CNPECN, CNPExplicit congestion notification: a switch marks packets when its queue grows; the receiver returns a congestion notification packet (CNP).明確壅塞通知：佇列增長時交換器標記封包；接收端回傳壅塞通知封包（CNP）。
DCQCNDCQCNDCQCNDCQCNThe standard RoCE congestion control: ECN marks at the switch, CNPs from the receiver, rate cuts at the sender NIC.RoCE 的標準壅塞控制：交換器做 ECN 標記、接收端回 CNP、傳送端 NIC 降速。
BDP-FCBDP-FCBDP-FCBDP-FCIRN's cap of one BDP of unacknowledged bytes per queue pair, enforced in the NIC.IRN 在 NIC 內施加的上限：每個佇列對最多一個 BDP 的未確認位元組。
Ultra EthernetUltra EthernetUltra EthernetUltra EthernetAn industry transport specification (1.0 in 2025) that sprays packets and retransmits selectively.業界傳輸規格（2025 年 1.0 版），會噴灑封包並做選擇性重傳。

08Routing and load balancing路由與負載平衡

ECMPECMPECMPECMPEqual-cost multi-path: a switch picks one of several equal paths by hashing each packet's header.等價多路徑：交換器以封包標頭的雜湊值在多條等價路徑中選一條。
Hash, five-tuple雜湊、五元組雜湊、五元組Hash, five-tupleA function that turns header fields (source and destination address and port, protocol: the five-tuple) into a path choice.把標頭欄位（來源與目的位址和埠、協定：五元組）轉成路徑選擇的函式。
Hash collision雜湊碰撞雜湊碰撞Hash collisionTwo large flows hashed onto the same uplink, so each gets half its rate.兩條大流被雜湊到同一條上行，各只得到一半速率。
Hash polarization雜湊極化雜湊極化Hash polarizationThe same flows colliding again at the next tier because each tier hashes the same fields the same way.因每層以相同方式雜湊相同欄位，同一批流在下一層再次碰撞。
QP scaling, E-ECMPQP 擴增、E-ECMPQP 擴增、E-ECMPQP scaling, E-ECMPOpening several queue pairs per connection and hashing on the destination QP number, so switches see more distinct flows.每條連線開多個佇列對，並以目的 QP 編號做雜湊，讓交換器看到更多不同的流。
Path pinning, traffic engineering路徑固定、流量工程路徑固定、流量工程Path pinning, traffic engineeringChoosing paths by rule or by a central planner instead of by hash.以規則或中央規劃器、而非雜湊，來選擇路徑。
Packet spraying, reordering封包噴灑、亂序封包噴灑、亂序Packet spraying, reorderingSending consecutive packets of one flow over different paths; they may arrive out of order, so the NIC must accept that.把同一條流的連續封包送到不同路徑；封包可能亂序抵達，NIC 必須能接受。
Flowlet, adaptive routingFlowlet、自適應路由Flowlet、自適應路由Flowlet, adaptive routingMoving a flow to a less loaded path at a gap between its bursts, or letting the switch choose by load.在流的突發之間空檔把它移到較空的路徑，或讓交換器依負載選擇。

09Front end and checkpoints前端網路與檢查點

Front-end and back-end networks前端網路與後端網路前端網路與後端網路Front-end and back-end networksThe back end carries only GPU-to-GPU training traffic; the front end carries data, checkpoints, logs and management.後端只承載 GPU 之間的訓練流量；前端承載資料、檢查點、日誌與管理流量。
Checkpoint檢查點（checkpoint）檢查點（checkpoint）CheckpointA saved copy of the training state (weights and optimizer state) from which a job can resume after a failure.訓練狀態（權重與最佳化器狀態）的存檔，工作故障後可從此處恢復。
Two-stage checkpoint兩階段檢查點兩階段檢查點Two-stage checkpointCopy the state to host memory first (the GPUs pause for seconds), then write it to storage in the background.先把狀態複製到主機記憶體（GPU 暫停數秒），再在背景寫入儲存。
Distributed storage分散式儲存分散式儲存Distributed storageMany storage servers that act as one file system or object store; its throughput is set by the number of servers.許多儲存伺服器合為一個檔案系統或物件儲存；吞吐量由伺服器數決定。
Erasure coding糾刪碼糾刪碼Erasure codingProtecting data with k data fragments and m parity fragments instead of three full copies.以 k 個資料片段加 m 個同位片段保護資料，而非存三份完整副本。
Out-of-band management頻外管理頻外管理Out-of-band managementA separate slow network that reaches each server's management controller even when the host is down.一條獨立的低速網路，即使主機當機也能連到每台伺服器的管理控制器。
Telemetry遙測（telemetry）遙測（telemetry）TelemetryCounters and logs collected from hosts and switches to watch the network.從主機與交換器收集的計數器與日誌，用來監看網路。

10Failures and stragglers故障與落後者

MTBFMTBF（平均故障間隔）MTBF（平均故障間隔）MTBFMean time between failures; for a job that stops when any part fails, the failure rates of all parts add.平均故障間隔；若任一零件故障就停機，所有零件的故障率相加。
Effective training time有效訓練時間有效訓練時間Effective training timeThe share of wall-clock time in which the job makes progress that it keeps.工作在牆鐘時間中產生可保留之進度的比例。
Young–Daly intervalYoung–Daly 間隔Young–Daly 間隔Young–Daly intervalThe checkpoint interval that minimizes lost time: about √(2 × checkpoint pause × MTBF).使損失時間最小的檢查點間隔：約 √(2 × 檢查點暫停 × MTBF)。
Silent data corruption (SDC)靜默資料損毀（SDC）靜默資料損毀（SDC）Silent data corruption (SDC)A GPU that produces wrong results without raising an error.GPU 產生錯誤結果卻不報錯。
Watchdog, heartbeat看門狗（watchdog）、心跳看門狗（watchdog）、心跳Watchdog, heartbeatTimers and periodic status messages that detect a hung or dead rank.偵測卡住或失效 rank 的計時器與定期狀態訊息。
Jitter, order statistics抖動、順序統計抖動、順序統計Jitter, order statisticsRandom step-to-step variation of each rank; the step waits for the maximum, which grows with the number of ranks.每個 rank 每步的隨機變動；步驟等待最大值，而最大值隨 rank 數成長。
Standby pool, spare備用池、備援備用池、備援Standby pool, spareSpare machines kept ready so a failed one can be replaced without waiting for a repair.預備的備用機器，故障時可立即替換，不必等待修理。

11Sharing the cluster共用叢集

Multi-tenancy, tenant多租戶、租戶多租戶、租戶Multi-tenancy, tenantSeveral jobs or customers sharing one cluster and its network.多個工作或客戶共用一個叢集及其網路。
Placement, scheduler放置、排程器放置、排程器Placement, schedulerChoosing which GPUs a job gets and which rank runs where; it decides which links the traffic crosses.決定工作拿到哪些 GPU、各 rank 放在哪裡；它決定流量經過哪些鏈路。
Fragmentation碎片化碎片化FragmentationFree GPUs that exist but not in a shape a job can use without crossing racks or pods.空閒 GPU 存在，但其形狀無法讓工作不跨機櫃或 pod 就使用。
Isolation, contention隔離、爭用隔離、爭用Isolation, contentionKeeping one job's traffic from slowing another's; contention is two jobs wanting the same link.避免一個工作的流量拖慢另一個；爭用是兩個工作想用同一條鏈路。

12Models and simulation模型與模擬

Simulation, packet-level simulation模擬、封包層級模擬模擬、封包層級模擬Simulation, packet-level simulationRunning a model of the workload and network in software; a packet-level simulator follows every packet through every queue.以軟體執行工作負載與網路的模型；封包層級模擬器追蹤每個封包經過每個佇列。
Analytical model解析模型解析模型Analytical modelA formula that predicts a quantity, such as step time, from a few parameters.以少數參數預測某個量（例如步驟時間）的公式。
Validation驗證驗證ValidationChecking a model or simulator against measurements of the real system.以真實系統的量測檢驗模型或模擬器。