←Contents目錄
How to use使用說明
About this primer
關於這份入門

How to use this primer如何使用這份入門

The reading order, entry points by question, how the chapters depend on each other, further reading, and what changed in this edition.閱讀順序、依問題找入口、各章的相依關係、延伸閱讀,以及這一版改了什麼。

How to read it怎麼讀

Read the thirteen chapters in order. Each one opens with a "Before you start" box listing the earlier terms it uses most, and a list of the new terms it defines; any term links back to its definition. Chapters 1–5 need no background. The research digests and guided readings are optional: read a digest after the chapter its topic names.

請依序讀這十三章。每章開頭有「開始之前」,列出它最常用到的前面名詞,以及它新定義的名詞;任何名詞都可以點回它的定義。第 1–5 章不需要任何背景。研究導讀與論文導讀是選讀:讀完主題所標明的那一章後再讀。

Start with a question從一個問題開始

If you came with a specific question, these sections answer it.

如果你帶著一個具體問題來,下列各節會回答它。

Why does training need a special network at all?訓練為什麼需要專用的網路?§1.4 · §5.4第 1.4 節 · 第 5.4 節
How many bytes do the GPUs exchange in one step?一步之中 GPU 之間交換多少位元組?§3.7 · §5.3第 3.7 節 · 第 5.3 節
What is an all-reduce, and how long does it take?all-reduce 是什麼,要花多久?§4.2 · §4.3第 4.2 節 · 第 4.3 節
Why does tensor parallelism stay inside one server?張量平行為什麼留在一台伺服器內?§2.2 · §3.4第 2.2 節 · 第 3.4 節
What do mixture-of-experts models change for the network?混合專家模型對網路改變了什麼?§3.8 · §5.5 · §6.8第 3.8 節 · 第 5.5 節 · 第 6.8 節
What is rail-optimized wiring, and why would anyone delete the spine?什麼是 rail-optimized 接線?為什麼有人要拿掉 spine?§6.3 · §6.6第 6.3 節 · 第 6.6 節
Why do RDMA networks try never to drop a packet?RDMA 網路為什麼盡量一個封包都不丟?§7.2 · §7.3第 7.2 節 · 第 7.3 節
Why does ECMP hashing fail for training traffic?ECMP 雜湊為什麼對訓練流量失效?§8.1 · §8.2第 8.1 節 · 第 8.2 節
How big is a checkpoint, and which network carries it?檢查點有多大,由哪個網路承載?§9.3第 9.3 節
How often does a large job fail, and how often should it checkpoint?大型任務多久故障一次?該多久存一次檢查點?§10.1 · §10.3第 10.1 節 · 第 10.3 節
What goes wrong when many jobs share one network?許多任務共用一個網路時會出什麼問題?§11.1 · §11.2第 11.1 節 · 第 11.2 節
Can I trust a simulator's numbers?模擬器的數字可以相信嗎?§12.1 · §12.4第 12.1 節 · 第 12.4 節
Which design should I build?我該建哪一種設計?§13.1 · §13.7第 13.1 節 · 第 13.7 節

How the chapters connect各章的關係

Each arc joins a chapter to an earlier chapter it relies on. Every arc points backwards: no chapter needs a later one.

每一條弧線把一章連到它依賴的較早章節。所有弧線都指向前面:沒有任何一章需要後面的章節。

Which earlier chapters each chapter builds onPart I第一部Part II第二部Part III第三部01020304050607080910111213

builds on建立在其上is needed by被其使用

Reading list and courses閱讀清單與課程

Read in this order. Each entry names the chapters of this primer it extends and what to look for.

請依此順序閱讀。每一條都標明它延伸了本入門的哪幾章,以及要注意什麼。

Start here: production experience從這裡開始:量產經驗

Ch. 6–9第 6–9 章
The routing evolution (ECMP → path pinning → E-ECMP) and the decision to run 400G without DCQCN. Read sections 4 and 6 twice.
路由的演進(ECMP → 路徑固定 → E-ECMP)以及在 400G 不用 DCQCN 的決定。第 4 與第 6 節值得讀兩遍。
Ch. 6, 8第 6, 8 章
The most complete topology paper: dual-ToR, dual-plane, single-chip switches, cooling. Section 2 is the best short description of why AI traffic differs.
最完整的拓樸論文:dual-ToR、dual-plane、單晶片交換器、散熱。第 2 節是對「AI 流量為何不同」最好的簡短描述。
Ch. 3, 10第 3, 10 章
The systems view: parallelism, overlap, network tuning and diagnosis in one paper. Section 3.4 for network, section 5 for fault tolerance.
系統觀點:平行化、重疊、網路調校與診斷集於一篇。第 3.4 節看網路,第 5 節看容錯。
Ch. 3, 10第 3, 10 章
The Llama 3 Herd of Models — Meta, 2024 — section 3.3 only · guided reading →導讀 →
The 24K-GPU cluster description, 4-D parallelism, NCCLX, and the 54-day failure table. Ten pages that every infrastructure engineer should know.
24K GPU 叢集描述、4-D 平行、NCCLX,以及 54 天的故障表。每位基礎設施工程師都該知道的十頁。
Ch. 6第 6 章
The non-Clos answer: torus plus optical circuit switches. Compare its assumptions with HPN's.
非 Clos 的答案:環面加光路交換。把它的假設和 HPN 的做比較。
Ch. 3, 6第 3, 6 章
Short and provocative. The traffic heatmaps alone are worth the read.
短而有挑釁性。光是那幾張流量熱圖就值得一讀。

Foundations you will need需要的基礎

Ch. 7第 7 章
RDMA over Commodity Ethernet at Scale — Guo et al., Microsoft, SIGCOMM 2016 · guided reading →導讀 →
PFC deadlocks, pause storms, and the operational reality of lossless Ethernet. The warning that every later paper answers.
PFC 死鎖、暫停風暴,以及無損 Ethernet 的營運現實。後來每篇論文都在回應的那個警告。
Ch. 7第 7 章
The default RoCE congestion control. Know the ECN threshold, CNP, and rate-decrease rules.
RoCE 預設的壅塞控制。要知道 ECN 門檻、CNP 與降速規則。
Ch. 7第 7 章
Selective retransmission and a bytes-in-flight cap make PFC unnecessary. The intellectual root of Ultra Ethernet.
選擇性重傳加在途位元組上限,讓 PFC 不再必要。Ultra Ethernet 的思想源頭。
Ch. 8第 8 章
How to spray packets without breaking go-back-N NICs: reorder inside the switch.
如何在不破壞 go-back-N NIC 的前提下噴灑封包:在交換器內重排。
Ch. 4第 4 章
Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations — Patarasuk & Yuan, JPDC 2009
The proof that the ring sends the minimum bytes. Read it after Chapter 4.
證明環形演算法送出的位元組最少。讀完第 4 章再讀。
Ch. 4第 4 章
Collectives built for a specific topology instead of the default ring.
為特定拓樸量身打造的集體通訊,而不是預設的環。
Ch. 6第 6 章
A Scalable, Commodity Data Center Network Architecture — Al-Fares et al., SIGCOMM 2008; Technology-Driven, Highly-Scalable Dragonfly Topology — Kim et al., ISCA 2008
Fat-tree and Dragonfly: the two topology families every later design cites.
Fat-tree 與 Dragonfly:後來每個設計都會引用的兩大拓樸家族。

Going deeper更深入

Ch. 11第 11 章
Multi-tenant scheduling, and the simulator used to design HPN.
多租戶排程,以及用來設計 HPN 的模擬器。
Ch. 10第 10 章
11 months, 4 million jobs: how failures scale with job size.
11 個月、四百萬個任務:故障如何隨任務規模增長。
Ch. 6–8第 6–8 章
Where the field is going: new transports that tolerate loss and reordering.
領域的走向:能容忍丟包與亂序的新傳輸層。
Ch. 3第 3 章
How to Scale Your Model — Google DeepMind (online book); The Ultra-Scale Playbook — Hugging Face
The parallelism side in depth, with roofline arithmetic for every collective.
深入平行化這一側,為每種集體操作做 roofline 計算。

Courses and venues課程與會議

course
A seminar built on exactly these papers: Meta RoCE and HPN in week 2, then collectives, photonics, congestion control and load balancing. The public reading list is a syllabus for this primer.
完全以這些論文為核心的研討課:第二週就讀 Meta RoCE 與 HPN,接著是集體通訊、光子學、壅塞控制與負載平衡。公開的閱讀清單就是這份入門的課綱。
workshop
Keynotes from Meta and HKUST plus short papers on simulation, circuit switching and multi-NIC communication. The fastest way to see what practitioners worry about this year.
Meta 與 HKUST 的主題演講,加上關於模擬、電路交換與多 NIC 通訊的短論文。了解業界今年在煩惱什麼的最快方式。
course
Stanford CS 244, Berkeley CS 268, CMU 15-744
General graduate networking: data center fabrics, congestion control, and RDMA in one or two weeks each. Good for the foundations above.
一般的研究所網路課:資料中心網路、壅塞控制與 RDMA 各佔一兩週。適合補上面的基礎部分。
industry
NVIDIA GTC sessions on NCCL and Spectrum-X; OCP Summit networking track; Ultra Ethernet Consortium webinars
Where the hardware details live: NIC features, switch telemetry, and the transport specifications.
硬體細節所在:NIC 功能、交換器遙測與傳輸層規範。

What changed from the first edition與第一版相比改了什麼

One path instead of overview plus deep dives一條路徑,取代總覽加深入章節

The first edition taught every topic twice, once quickly on the overview and again in eleven deep dives, and the two orders disagreed. Read in order, 129 terms appeared before they were explained, in 2,219 paragraphs, and 42 had no marked definition at all.第一版每個主題教兩次:總覽快速講一次,十一個深入章節再講一次,而兩者的順序互相衝突。依序閱讀時,有 129 個名詞在解釋之前就出現,共 2,219 個段落,另有 42 個從未被明確定義。

Each term has one home每個名詞只有一個家

A register of 162 terms gives each one a home chapter. A scan of every English paragraph finds no term used before its definition; the Chinese copy was checked the same way.一份 162 個名詞的登錄表為每個名詞指定一個所屬章。逐段掃描英文,沒有任何名詞在定義之前被使用;中文版也以同樣方式檢查過。

Research moved off the main path研究移出主線

Recent papers now live on eleven frontier pages, each placed after the chapter it needs, so the main path stays readable from start to finish.近期論文改放在十一個研究前沿頁,各自放在它需要的那一章之後,讓主線從頭到尾都讀得下去。

Checked against the sources對照原文修正

Where a source was checked, the text now says what the paper reports: Rail-only's figures are computed, HPN's hosts send ARP and the ToRs speak BGP, Llama 3's failure table does not add up as printed, Crux's theorem has limit 1, DCP's control queue is not guaranteed lossless. 14 research digests were later checked line by line against the full papers and corrected.凡是核對過原文的地方,內文都改為論文實際寫的內容:Rail-only 的圖是計算值、HPN 由主機送 ARP 而由 ToR 跑 BGP、Llama 3 故障表的印刷數字對不起來、Crux 定理的極限是 1、DCP 的控制佇列並非保證無損。14 篇研究導讀後來逐條對照論文全文核對並修正。

Where each old section went →舊版各節搬到哪裡了 → · Open the first edition ↗開啟第一版 ↗