How to use this primer如何使用這份入門
The reading order, entry points by question, how the chapters depend on each other, further reading, and what changed in this edition.閱讀順序、依問題找入口、各章的相依關係、延伸閱讀,以及這一版改了什麼。
How to read it怎麼讀
Read the thirteen chapters in order. Each one opens with a "Before you start" box listing the earlier terms it uses most, and a list of the new terms it defines; any term links back to its definition. Chapters 1–5 need no background. The research digests and guided readings are optional: read a digest after the chapter its topic names.
請依序讀這十三章。每章開頭有「開始之前」,列出它最常用到的前面名詞,以及它新定義的名詞;任何名詞都可以點回它的定義。第 1–5 章不需要任何背景。研究導讀與論文導讀是選讀:讀完主題所標明的那一章後再讀。
Start with a question從一個問題開始
If you came with a specific question, these sections answer it.
如果你帶著一個具體問題來,下列各節會回答它。
| Why does training need a special network at all?訓練為什麼需要專用的網路? | §1.4 · §5.4第 1.4 節 · 第 5.4 節 |
| How many bytes do the GPUs exchange in one step?一步之中 GPU 之間交換多少位元組? | §3.7 · §5.3第 3.7 節 · 第 5.3 節 |
| What is an all-reduce, and how long does it take?all-reduce 是什麼,要花多久? | §4.2 · §4.3第 4.2 節 · 第 4.3 節 |
| Why does tensor parallelism stay inside one server?張量平行為什麼留在一台伺服器內? | §2.2 · §3.4第 2.2 節 · 第 3.4 節 |
| What do mixture-of-experts models change for the network?混合專家模型對網路改變了什麼? | §3.8 · §5.5 · §6.8第 3.8 節 · 第 5.5 節 · 第 6.8 節 |
| What is rail-optimized wiring, and why would anyone delete the spine?什麼是 rail-optimized 接線?為什麼有人要拿掉 spine? | §6.3 · §6.6第 6.3 節 · 第 6.6 節 |
| Why do RDMA networks try never to drop a packet?RDMA 網路為什麼盡量一個封包都不丟? | §7.2 · §7.3第 7.2 節 · 第 7.3 節 |
| Why does ECMP hashing fail for training traffic?ECMP 雜湊為什麼對訓練流量失效? | §8.1 · §8.2第 8.1 節 · 第 8.2 節 |
| How big is a checkpoint, and which network carries it?檢查點有多大,由哪個網路承載? | §9.3第 9.3 節 |
| How often does a large job fail, and how often should it checkpoint?大型任務多久故障一次?該多久存一次檢查點? | §10.1 · §10.3第 10.1 節 · 第 10.3 節 |
| What goes wrong when many jobs share one network?許多任務共用一個網路時會出什麼問題? | §11.1 · §11.2第 11.1 節 · 第 11.2 節 |
| Can I trust a simulator's numbers?模擬器的數字可以相信嗎? | §12.1 · §12.4第 12.1 節 · 第 12.4 節 |
| Which design should I build?我該建哪一種設計? | §13.1 · §13.7第 13.1 節 · 第 13.7 節 |
How the chapters connect各章的關係
Each arc joins a chapter to an earlier chapter it relies on. Every arc points backwards: no chapter needs a later one.
每一條弧線把一章連到它依賴的較早章節。所有弧線都指向前面:沒有任何一章需要後面的章節。
Reading list and courses閱讀清單與課程
Read in this order. Each entry names the chapters of this primer it extends and what to look for.
請依此順序閱讀。每一條都標明它延伸了本入門的哪幾章,以及要注意什麼。
Start here: production experience從這裡開始:量產經驗
Foundations you will need需要的基礎
Going deeper更深入
Courses and venues課程與會議
What changed from the first edition與第一版相比改了什麼
The first edition taught every topic twice, once quickly on the overview and again in eleven deep dives, and the two orders disagreed. Read in order, 129 terms appeared before they were explained, in 2,219 paragraphs, and 42 had no marked definition at all.第一版每個主題教兩次:總覽快速講一次,十一個深入章節再講一次,而兩者的順序互相衝突。依序閱讀時,有 129 個名詞在解釋之前就出現,共 2,219 個段落,另有 42 個從未被明確定義。
A register of 162 terms gives each one a home chapter. A scan of every English paragraph finds no term used before its definition; the Chinese copy was checked the same way.一份 162 個名詞的登錄表為每個名詞指定一個所屬章。逐段掃描英文,沒有任何名詞在定義之前被使用;中文版也以同樣方式檢查過。
Recent papers now live on eleven frontier pages, each placed after the chapter it needs, so the main path stays readable from start to finish.近期論文改放在十一個研究前沿頁,各自放在它需要的那一章之後,讓主線從頭到尾都讀得下去。
Where a source was checked, the text now says what the paper reports: Rail-only's figures are computed, HPN's hosts send ARP and the ToRs speak BGP, Llama 3's failure table does not add up as printed, Crux's theorem has limit 1, DCP's control queue is not guaranteed lossless. 14 research digests were later checked line by line against the full papers and corrected.凡是核對過原文的地方,內文都改為論文實際寫的內容:Rail-only 的圖是計算值、HPN 由主機送 ARP 而由 ToR 跑 BGP、Llama 3 故障表的印刷數字對不起來、Crux 定理的極限是 1、DCP 的控制佇列並非保證無損。14 篇研究導讀後來逐條對照論文全文核對並修正。
Where each old section went →舊版各節搬到哪裡了 → · Open the first edition ↗開啟第一版 ↗