Chapter 10 of 13 · Part III · AI in orbit
第 10 章（共 13 章）· 第三部 · 在軌 AI
What AI training asks of a network, and what a constellation offersAI 訓練對網路的要求，以及星座能給什麼
This chapter relates synchronous training’s bandwidth, latency and reliability requirements to bisection bandwidth, injection rate and exposed communication. It compares satellite-link assumptions with terrestrial systems, then identifies which companion-primer designs transfer to orbit and which require adaptation or replacement.本章以對切頻寬、注入速率與暴露通訊說明同步訓練對網路的要求，並將各論文的衛星鏈路假設與地面系統比較。接著逐項檢視姊妹教材的設計，說明哪些能沿用到軌道，哪些需要調整或替換。
Before you start
開始之前
This chapter builds on Chapters 2, 5, 6 and 8, plus the companion AI Training Network Primer. It recalls the orbital-relevant parts of companion Chapters 3–11: parallelism, collectives, step budgets, topology, transport, routing, front ends, failures and sharing. The links below open definitions of recurring terms.
本章建立在第 2、5、6、8 章之上，也建立在姊妹教材 AI Training Network Primer 之上。該教材第 3 到 11 章（平行化、集體通訊、步時間預算、拓樸、傳輸層、路由、前端、故障、共用）只回顧到軌道論證所需的程度。以下名詞在本章經常出現，點選即可回到定義。
New terms in this chapter (8)本章新名詞（8）
Each link opens the definition at the term’s first appearance.
各名詞在首次出現處都有定義，點選連結即可前往。
- all-reduce全歸約（all-reduce） The collective in which every worker ends with the element-wise sum of all workers' vectors. 讓每個工作者最後都拿到所有工作者向量逐元素總和的集體通訊。
- step time步時間 The wall-clock time of one training step, compute plus exposed communication. 一個訓練步驟的實際耗時，等於運算時間加上暴露的通訊時間。
- exposed communication暴露的通訊 The communication that extends beyond compute and therefore lengthens the step. 一步之中超出運算重疊範圍、因此直接拉長步時間的通訊。
- oversubscription超額訂閱 The ratio of capacity entering a network tier to capacity leaving it upward. 進入某一層網路的容量，與該層向上送出容量之間的比值。
- ICI晶片間互連（ICI） The inter-chip interconnect that links accelerators inside one host or pod directly over ICI. 直接連接同一主機或同一 pod 內加速器的晶片間互連。
- torus環面（torus） A grid in which each node links to its neighbors in every dimension and the edges wrap around. 每個節點都與各維度的鄰居相連、且邊緣首尾相接的網格。
- Clos networkClos 網路 A layered switch fabric in which compute nodes attach to leaf switches and every leaf reaches every other through spine switches. 一種分層交換架構：運算節點接到葉交換器，每個葉交換器都透過脊交換器到達其他所有葉交換器。
- bisection intensity對切強度 Floating-point operations per step divided by bytes that must cross the bisection per step. 每一步的浮點運算量，除以每一步必須穿越對切面的位元組數。
10.1Recap from the companion primer: parallelisms, collectives, the step budget回顧姊妹教材：平行化、集體通訊、步時間預算
Recall
回顧
Section 5.2 defines bisection as the capacity cut when a graph is halved. Section 2.1 gives vacuum propagation time of 3.336 µs/km. Both help assess training clusters.
The companion primer explains three ways to split a large model across workers. Their traffic patterns determine how well they fit an orbital network. Tensor parallelism splits each layer’s weight matrices across a small accelerator group, which exchanges partial activations several times per layer. It requires the highest bandwidth and lowest latency, so Meta’s Llama 3 report keeps it within a host at TP = 8 [77, §3.3.2] preprint. Pipeline parallelism assigns consecutive layers to stages.
Adjacent stages exchange forward activations and backward gradients once per micro-batch, moving few bytes on each step’s critical path. Data parallelism replicates the model and assigns each replica a separate batch slice. It combines gradients through an all-reduce, which gives every worker the element-wise sum of all workers’ vectors so every replica applies the same update. The volume of that collective is proportional to model size with batch size independent of that volume. This explains why it is the term that a slow inter-satellite fabric will feel first.
姊妹教材花了三章說明大型模型如何切分到多個工作者上，而軌道問題只需要知道每一種切法會產生什麼流量。張量平行把每一層的權重矩陣切到一小群加速器上，每層要交換部分激活值好幾次，因此它對頻寬與延遲的要求是三者之中最高的，通常被限制在一台主機之內，Meta 的 Llama 3 報告把它設為 TP = 8[77, §3.3.2]preprint。流水線平行把連續的層分配到各個階段，相鄰階段之間每個微批次傳一次前向激活值與反向梯度，位元組數較少，卻落在每一步的關鍵路徑上。資料平行則把整個模型複製多份，各副本處理各自的批次切片，然後必須合併梯度讓每個副本套用相同的更新，這件事靠的就是全歸約（all-reduce），也就是讓每個工作者最後都拿到所有工作者向量逐元素總和的集體通訊。這個集體通訊的資料量與模型大小成正比，批次大小則維持獨立，所以它會是慢速星間網路最先感受到的那一項。
The all-reduce cost follows directly from the bytes that a ring must move. A ring all-reduce has N workers, each holding M bytes, and starts with reduce-scatter. Each worker sends one N-th of its vector right and adds the vector arriving from the left. After N − 1 repetitions, each worker holds the complete sum of one shard. All-gather then circulates the finished shards for another N − 1 steps.
Each step sends M/N bytes from every worker. Both phases therefore move 2(N − 1)M/N bytes per worker, approaching 2M from below as N grows. A cut halving the group carries M/2 bytes per direction during reduce-scatter. All-gather returns another M/2 bytes per direction as finished shards. The total is M bytes per direction.
Divide per-worker bytes by injection rate to obtain the all-reduce injection bound. Divide cut bytes by bisection bandwidth to obtain the cut bound. Injection governs when bisection exceeds half of one worker’s injection rate, as the table shows. Ring all-reduce therefore stresses worker links, while all-to-all expert dispatch stresses the cut. Section 10.4 uses this distinction.
全歸約的成本可以從「環必須搬多少資料」直接算出來，以下直接列出推導。設 N 個工作者各持有 M 位元組的向量，環狀全歸約先做歸約散佈（reduce-scatter）：每個工作者把自己向量的 N 分之一送給右鄰，並把左鄰送來的加進來，重複 N − 1 次之後每個工作者手上都有某一個分片的完整總和。接著做全收集（all-gather），讓完成的分片再繞行 N − 1 步。每一步從每個工作者送出 M/N 位元組，所以兩階段合計每個工作者送出 2(N − 1)M/N 位元組，隨 N 增大趨近於 2M ，上限為 2M。同一套計數也決定了任何把群組對半切開的切面上會流過多少資料：歸約散佈必須把另一半的部分和送過來（每個方向 M/2 位元組），全收集又必須把完成的分片送回去（每個方向再 M/2），因此切面每個方向總共承載 M 位元組。
把每工作者位元組除以工作者的注入速率就得到全歸約時間的注入下界，把切面位元組除以對切頻寬就得到切面下界，而下方的算式表顯示：只要拓樸的對切頻寬超過單一工作者注入速率的一半，起作用的就是注入下界。第 10.4 節會倚賴的實際結論是：環狀全歸約吃緊的是工作者自己擁有的鏈路，切面上的流量則較少。反之，像混合專家模型裡的專家分派這類全對全（all-to-all）模式吃緊的才是切面。
| Question問題 | Calculation算式 | Result結果 |
|---|---|---|
| Bytes a worker sends in the reduce-scatter phase歸約散佈階段每個工作者送出的位元組 | \((N-1)\times \dfrac{M}{N}\) | \(\dfrac{N-1}{N}M\)computed. N − 1 steps of one shard each計算值。N − 1 步、每步一個分片 |
| Bytes a worker sends in the all-gather phase全收集階段每個工作者送出的位元組 | \((N-1)\times \dfrac{M}{N}\) | \(\dfrac{N-1}{N}M\)computed. Same count, finished shards計算值。同樣的計數，換成完成的分片 |
| Total per worker, and its limit每個工作者合計，及其極限 | \(\dfrac{2(N-1)}{N}M \to 2M\) | 1.975 M at N = 81（N = 81）computed. 1.9998 M at N = 8,192計算值。N = 8,192 時為 1.9998 M |
| Bytes crossing a halving cut, per direction穿越對半切面的位元組，每個方向 | \(\tfrac{M}{2}+\tfrac{M}{2}\) | \(M\)computed. Partial sums over, finished shards back計算值。部分和送過去、完成分片送回來 |
| Injection bound vs cut bound注入下界與切面下界 | \(\dfrac{2M}{B_{\text{inj}}}\ \text{vs}\ \dfrac{M}{B_{\text{bis}}}\) | injection binds when當 \(B_{\text{bis}} > B_{\text{inj}}/2\)時注入下界起作用computed. True for a 2-D torus whenever 2√N·B > 2B, i.e. N > 1計算值。對 2-D 環面只要 2√N·B > 2B，即 N > 1，就成立 |
Step time combines compute and communication that extends beyond it. Measured terrestrial utilization provides the comparison point for an orbital design.
步時間由運算與超出運算的暴露通訊組成，地面系統的實測利用率可作為軌道設計的比較基準。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | The companion's step budget reduces the whole network question to two lines. |
| Result and scope | Qualitative scope | The step time is one training step’s wall-clock duration, expressed as \(T_{\text{step}} = T_{\text{compute}} + T_{\text{exposed}}\). Its second term, exposed communication, is communication extending beyond the arithmetic that overlaps it. |
| Result and scope | Qualitative scope | Communication fully overlapped with compute adds zero step time. |
| Cost assumptions | Qualitative scope | The relevant network cost is the communication that extends beyond compute. |
| Model and compute | 55.2%, 12,288, 41.2%, 59.1%, 3,072, 3, 38–43%, 16 | Model-FLOPs utilization measures the share of peak arithmetic used. MegaScale reaches 55.2% at 12,288 GPUs, versus Megatron-LM’s 41.2% on the same hardware. MegaScale’s value falls from 59.1% at 3,072 GPUs [98, Tabs. 2–3]. Llama 3 reports 38–43% BF16 MFU on up to 16K H100s [77, §3.3.3] preprint. |
| Result and scope | Qualitative scope | Even purpose-built clusters leave roughly half their arithmetic idle. |
| Result and scope | Qualitative scope | An orbital design must keep exposed communication near tens of percent rather than several hundred percent. |
| 結果與適用範圍 | 定性範圍 | 姊妹教材的步時間預算把整個網路問題縮成兩行。 |
| 結果與適用範圍 | 定性範圍 | 步時間是一個訓練步驟的實際耗時，可分解為 \(T_{\text{step}} = T_{\text{compute}} + T_{\text{exposed}}\)，其中第二項是暴露的通訊，也就是超出算術運算重疊範圍的那部分通訊。 |
| 結果與適用範圍 | 定性範圍 | 藏在運算底下進行的通訊的額外步時間成本為零，因此評估網路時，應看有多少通訊時間超出運算。 |
| 模型與運算 | 12,288, 55.2%, 41.2%, 3,072, 59.1% | 模型 FLOPs 利用率（MFU），即峰值算力實際被用到的比例，是最能揭露這個餘數的指標，而地面數字給了我們期待值：ByteDance 的 MegaScale 在 12,288 顆 GPU 上達到 55.2%，同一硬體上的 Megatron-LM 只有 41.2%，而它自己的數字也從 3,072 顆 GPU 時的 59.1% 往下掉[98, Tabs. 2–3]。 |
| 結果與適用範圍 | 3, 16, 38–43% | Llama 3 則在最多 16K 顆 H100 上回報 38–43% 的 BF16 MFU[77, §3.3.3]preprint。 |
| 結果與適用範圍 | 定性範圍 | 連專門打造的叢集都讓大約一半的算力閒置，軌道設計應以這些實測值為基準，將暴露通訊維持在幾十個百分點的量級。 |
Oversubscription measures the capacity reduction between network tiers. Training systems keep their heaviest parallel traffic within the tier that supplies enough bandwidth.
超額訂閱描述網路層級之間的容量縮減，訓練系統會將最重的平行流量保留在頻寬足夠的層級內。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | The last piece of vocabulary is how a fabric is sized against injection. |
| Capacity and rate | 1:1, 1:7 | In a tiered network, oversubscription divides incoming capacity from below by available upward capacity. A 1:1 fabric carries every worker’s full injection across the bisection simultaneously. A 1:7 fabric carries one seventh. |
| Capacity and rate | 3, 3,072, 1:1, 1:7 | Llama 3 uses 3,072-GPU pods with full, 1:1 bisection bandwidth. The aggregation tier joins eight pods at 1:7 oversubscription. Scheduling keeps tensor-parallel and most pipeline traffic within pods, leaving lighter data-parallel traffic to cross the restricted tier [77, §3.3.1] preprint. |
| Capacity and rate | Qualitative scope | The model partition follows the bandwidth hierarchy. Each tier supplies the bandwidth needed by the parallel traffic placed there. |
| Result and scope | Qualitative scope | Satellite clusters follow the same rule and add another tier: the connection from an accelerator board to the optical terminal leaving its spacecraft. |
| 結果與適用範圍 | 定性範圍 | 最後一個詞彙是網路架構如何相對於注入速率來定尺寸。 |
| 結果與適用範圍 | 1:1, 1:7 | 在分層網路裡，超額訂閱是從下方進入某一層的容量與該層能向上送出容量的比值，所以 1:1 的架構能同時把每個工作者的全部注入速率送過對切面，而 1:7 的架構只能送七分之一。 |
| 模型與運算 | 3, 3,072, 1:1, 1:7 | Llama 3 的叢集展示了標準的折衷：3,072 顆 GPU 的 pod 以全對切頻寬、1:1 容量的方式接線，而把八個這種 pod 接在一起的匯聚層則超額訂閱 1:7，排程器被要求把張量平行與大部分流水線流量留在 pod 內，只讓較輕的資料平行流量穿越那層薄的網路[77, §3.3.1]preprint。 |
| 結果與適用範圍 | 定性範圍 | 本章會一再回到的模式是：頻寬花在平行化需要的地方，而模型切分配合網路層級，以可用頻寬為依據。 |
| 結果與適用範圍 | 定性範圍 | 衛星叢集完整繼承這套設計紀律，只是在最底層多加了一層：加速器板與離開太空船的雷射終端之間的那一層。 |
10.2The terrestrial reference numbers地面參考數字
Terrestrial training clusters provide high bandwidth per accelerator and place switches in a hierarchy. Their operational failure measurements also define a reliability reference.
地面訓練叢集為每顆加速器提供高頻寬，並以交換器構成階層，其營運故障量測也提供了可靠度基準。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | Three documents and one product page define the terrestrial reference. Each orbital proposal needs comparison with their per-accelerator training-network capabilities. |
| Result and scope | 405 B, 3, 16, 700 W, 80 GB, 24, 400, 3,072 | Meta trained the 405 B Llama 3 on up to 16K H100s within a 24K-GPU RoCE cluster. Each H100 has 700 W power and 80 GB HBM3. A three-layer Clos joins 3,072-GPU pods with “400 Gbps interconnects between GPUs”. Pipeline- and data-parallel traffic “may communicate through a multi-hop network, with latency up to tens of microseconds” [77, §3.3.1–3.3.3] preprint. |
| Capacity and rate | 400, 2, 200 | Alibaba's HPN gives each of the eight GPUs in a host its own 400 Gbps RDMA NIC built from 2 × 200 Gbps ports |
| Capacity and rate | 3.2, 15,360, 51.2 | so a host injects 3.2 Tbps into the back-end fabric, keeps 15,360 GPUs in one pod on 51.2 Tbps single-chip switches |
| Result and scope | Qualitative scope | and observes that training traffic consists of "very few |
| Result and scope | Qualitative scope | but periodically bursty flows" that saturate the NIC for seconds to tens of seconds at a time |
| Capacity and rate | 400–900 GB/s, 4–9 | while intra-host NVLink at 400–900 GB/s bidirectional runs 4–9× faster than the NIC [65, §2.2, §3, §5.2]. |
| Reliability and accuracy | 10.5, 0.057%, 0.051%, 5, 60, 1–2 | The paper’s operating statistics provide Section 10.5’s reliability reference. Monthly rates are 0.057% failures for NIC-to-ToR links and 0.051% critical errors for ToR switches. Daily link flaps total 5K–60K, producing 1–2 training-job crashes monthly [65, §2.3, Fig. 5]. |
| 結果與適用範圍 | 定性範圍 | 每一個軌道提案都必須拿來與前沿訓練叢集實際提供給每顆加速器的資源相比，而三份文件加一個產品頁面提供了這些數字。 |
| 結果與適用範圍 | 24, 16, 700 W, 80 GB, 405 B, 3, 400, 3,072 | Meta 在一個 24K GPU 的 RoCE 叢集內用最多 16K 顆 H100（700 W、80 GB HBM3）訓練 405 B 的 Llama 3，GPU 之間是「400 Gbps 互連」，由 3,072 顆 GPU 的 pod 組成三層 Clos，並說明流水線與資料平行流量「可能經過多跳網路通訊，延遲最多達數十微秒」[77, §3.3.1–3.3.3]preprint。 |
| 容量與速率 | 2, 200, 400, 3.2, 51.2, 15,360, 400–900 GB/s, 4–9 | 阿里巴巴的 HPN 給主機內八顆 GPU 每顆一張由 2 × 200 Gbps 埠組成的 400 Gbps RDMA 網路卡，因此一台主機往後端網路注入 3.2 Tbps，一個 pod 用 51.2 Tbps 單晶片交換器容納 15,360 顆 GPU，並觀察到訓練流量是「極少數但週期性爆發的流」，一次就能讓網路卡飽和數秒到數十秒，而主機內的 NVLink 以 400–900 GB/s 雙向速率跑得比網路卡快 4–9 倍[65, §2.2, §3, §5.2]。 |
| 結果與適用範圍 | 10.5, 0.057%, 0.051%, 5, 60, 1–2 | 同一篇論文的營運統計是第 10.5 節的可靠度量尺：每月有 0.057% 的網路卡到 ToR 鏈路失效、0.051% 的 ToR 交換器發生嚴重錯誤、每天有 5K–60K 次鏈路抖動，結果是每月 1–2 次訓練工作崩潰[65, §2.3, Fig. 5]。 |
TPU v6e provides the on-board interconnect reference for Suncatcher. Its torus differs from the Clos fabrics through which GPU clusters scale bisection bandwidth.
TPU v6e 提供 Suncatcher 機上互連的參考，其環面架構與 GPU 叢集用來擴展對切頻寬的 Clos 各有特性。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Model and compute | Qualitative scope | The accelerator-side numbers come from Google's TPU v6e, because the Suncatcher proposal is built around that chip. |
| Compute and memory | 918 TFLOPS, 32 GB, 1,638 GB/s | Each Trillium chip delivers 918 TFLOPS in bf16, carries 32 GB of HBM at 1,638 GB/s |
| Capacity and rate | 800 GB/s | ICI directly connects accelerator chips. Each chip provides 800 GB/s bidirectional bandwidth over four ports. |
| Capacity and rate | 6.4, 256, 2 | This converts to 6.4 Tbps. A pod has 256 chips wired as a 2-D torus. Each chip connects to neighbors in each dimension, with wraparound edges. |
| Capacity and rate | 25.6 | and the pod has 25.6 Tbps of data-center network capacity [99] company. |
| Result and scope | Qualitative scope | The torus provides a terrestrial precedent for Suncatcher’s lattice. |
| Capacity and rate | 3 | Llama 3 and HPN use Clos networks. Compute nodes attach to leaf switches, and spine switches connect every leaf to every other. Bisection then grows with node count, compared with the torus’s square-root scaling. |
| Result and scope | 10.4 | Section 10.4 puts numbers on that difference. |
| Latency comparison | Qualitative scope | The three papers leave per-cluster round-trip time unspecified, so "tens of microseconds" is the available latency reference. |
| Result and scope | 10.1 | Section 10.1 directly derives the ring all-reduce rule because its primary-source statement remains to be located. |
| 模型與運算 | 定性範圍 | 加速器端的數字來自 Google 的 TPU v6e，因為 Suncatcher 提案就是圍繞這顆晶片建立的。 |
| 運算與記憶體 | 918 TFLOPS, 32 GB, 1,638 GB/s, 800 GB/s, 6.4 | 每顆 Trillium 晶片提供 918 TFLOPS 的 bf16 算力、32 GB 的 HBM 以 1,638 GB/s 存取，並透過四個埠提供每顆晶片 800 GB/s 雙向的晶片間互連（ICI），也就是加速器之間的直接互連，換算為 6.4 Tbps。 |
| 容量與速率 | 256, 2, 25.6 | 256 顆晶片組成一個 pod，接成 2-D 環面（torus），即每顆晶片都與各維度鄰居相連、邊緣首尾相接的網格，而整個 pod 有 25.6 Tbps 的資料中心網路容量[99]company。 |
| 結果與適用範圍 | 定性範圍 | 環面值得特定點名，因為它是 Suncatcher 所畫晶格的地面前身。 |
| 結果與適用範圍 | 3 | 而 Llama 3 與 HPN 的網路則是 Clos 網路，一種分層交換架構，運算節點接到葉交換器，每個葉交換器都透過脊交換器到達其他所有葉交換器，因此對切頻寬隨節點數線性成長。 |
| 結果與適用範圍 | 10.4 | 第 10.4 節會為這個差異填上數字。 |
| 結果與適用範圍 | 定性範圍 | 三篇論文的叢集內往返時間仍待補充，目前可用的延遲參考是「數十微秒」。 |
| 結果與適用範圍 | 10.1 | 環狀全歸約規則的一手來源亦待補充，因此第 10.1 節直接列出推導。 |
| System系統 | Per-accelerator network每顆加速器的網路 | Fabric網路架構 | Latency / reliability延遲／可靠度 | Source and tag來源與標籤 |
|---|---|---|---|---|
| Llama 3 (Meta) | 400 Gbps per GPU. TP = 8, PP = 16, DP ≤ 128每顆 GPU 400 Gbps。TP = 8、PP = 16、DP ≤ 128 | 3-layer Clos, 3,072-GPU pods at full bisection, 1:7 at aggregation三層 Clos，3,072 GPU 的 pod 全對切，匯聚層 1:7 | "up to tens of microseconds" multi-hop. 38–43% MFU多跳「最多數十微秒」。MFU 38–43% | arXiv 2407.21783 §3.3 preprint |
| HPN (Alibaba) | 400 Gbps NIC per GPU (2 × 200). 3.2 Tbps per host. NVLink 4–9× NIC每顆 GPU 一張 400 Gbps 網路卡（2 × 200）。每主機 3.2 Tbps。NVLink 為網路卡 4–9 倍 | 15,360 GPUs per pod. 51.2 Tbps switches每 pod 15,360 顆 GPU。51.2 Tbps 交換器 | 0.057% links fail/month. 1–2 job crashes/month. 5K–60K flaps/day每月 0.057% 鏈路失效。每月 1–2 次工作崩潰。每天 5K–60K 次抖動 | SIGCOMM 2024 §2–3, §5.2 peer-reviewed |
| MegaScale (ByteDance) | 8 × 200G NICs per server每台伺服器 8 × 200G 網路卡 | 3-layer Clos on 25.6 Tbps Tomahawk 4, 1:125.6 Tbps Tomahawk 4 三層 Clos，1:1 | 55.2% MFU at 12,288 GPUs (59.1% at 3,072)12,288 顆 GPU 時 MFU 55.2%（3,072 顆時 59.1%） | NSDI 2024 §3.6, Tables 2–3 peer-reviewed |
| TPU v6e (Google) | ICI 800 GB/s bidirectional per chip, 4 ports (6.4 Tbps). 918 TFLOPS bf16每顆晶片 ICI 800 GB/s 雙向、4 埠（6.4 Tbps）。918 TFLOPS bf16 | 256-chip 2-D torus pod. 25.6 Tbps DCN per pod256 晶片 2-D 環面 pod。每 pod 25.6 Tbps DCN | publication pending待公布 | Google Cloud docsGoogle Cloud 文件 company |
10.3Mapping the companion's chapters onto a constellation把姊妹教材的章節對應到星座上
Recall
回顧
Section 2.4 gives 2–30 s acquisition and three terminals per Starlink satellite. Section 8.4 supplies Trillium radiation rates, and Section 8.3 gives thermal duty cycles. These facts change specific companion chapters.
The parallelism hierarchy must follow the available bandwidth hierarchy in orbit. Predictable geometry changes collective schedules, while the step-time identity remains valid.
在軌平行化層級應配合可用頻寬層級。可預測的幾何會改變集體通訊排程，步時間恆等式則保持成立。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Terrestrial design assumptions | Qualitative scope | The companion’s design chapters assume a room of racks. Review each chapter against the satellite facts to identify which assumptions require changes. |
| Capacity and rate | Qualitative scope | The parallelism chapter survives intact, because its lesson is that the parallelism hierarchy must match the bandwidth hierarchy |
| Capacity and rate | 6.4 | Satellites add a bandwidth tier. Tensor parallelism stays on one spacecraft’s ICI, which provides 6.4 Tbps per Trillium chip. |
| Capacity and rate | 10, 10–30 | Pipeline and data parallelism cross optical terminals. Suncatcher targets “on the order of 10 Tbps” per link. With Google’s stated “dozens of TPU chips” per satellite, this tier is roughly 10–30× oversubscribed against on-board ICI [100] company. |
| Capacity and rate | 100, 200 m, 3 | Collective algorithms need adaptation because their usual graph is static with uniform rates. Formation geometry cycles twice per orbit, changing neighbor distances between 100 and 200 m [36, p. 4]. Spatial multiplexing then makes rate depend on distance (p. 3). |
| Orbital configuration | Qualitative scope | The step-budget chapter is kept verbatim, since the identity remains valid in orbit \(T_{\text{step}} = T_{\text{compute}} + T_{\text{exposed}}\). |
| Result and scope | Qualitative scope | Only the inputs change. |
| 結果與適用範圍 | 定性範圍 | 姊妹教材的設計章節是為一間裝滿機架的房間寫的，要看清軌道改變了什麼，最乾淨的方法就是逐章走過去，問哪一個衛星事實（如果有的話）打破了該章的假設。 |
| 容量與速率 | 6.4 | 平行化那一章完整存活，因為它的教訓是平行化層級必須配合頻寬層級，而衛星只是多加一層：張量平行留在同一艘太空船內的 ICI 上，一顆 Trillium 晶片有 6.4 Tbps 的 ICI，而流水線與資料平行則穿越雷射終端。 |
| 容量與速率 | 10, 10–30 | 若每顆衛星真的載有 Google 現在所說的「數十顆 TPU 晶片」[100]company，即使是 Suncatcher 每條鏈路「10 Tbps 量級」的目標相對於機上 ICI 也會超額訂閱約 10–30 倍。 |
| 結果與適用範圍 | 100, 200 m, 3 | 集體通訊那一章需要調整，因為它的演算法假設靜態圖與一致的鏈路速率，而編隊的幾何每圈週期變化兩次、鄰居距離在 100 到 200 m 之間振盪[36, p. 4]，一旦使用空間多工，鏈路速率還會隨距離改變（p. 3）。 |
| 結果與適用範圍 | 定性範圍 | 步時間預算那一章可以逐字保留，因為軌道仍遵循 \(T_{\text{step}} = T_{\text{compute}} + T_{\text{exposed}}\) 這個恆等式，改變的只有輸入值。 |
A satellite’s terminal count limits its network degree. Optical fading and acquisition also require transport assumptions suited to free-space links.
衛星的終端數限制了網路度數，光學衰落與鏈路擷取則要求傳輸層採用適合自由空間鏈路的假設。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | The topology chapter is the first that must be replaced. |
| Capacity and rate | 51.2, 64 | The companion’s Clos uses 51.2 Tbps switch ASICs with at least 64 ports. A satellite’s degree instead follows its independently pointed terminal count. Current Starlink satellites have three [39, §2.1] company, while van Berkel’s tori use four to six [101, §IV-A] preprint. Suncatcher depicts eight neighbors [36, p. 4, Fig. 2], and Pénot and Balakrishnan sweep up to twelve ports [102] preprint. |
| Fabric roles | 10.4 | With degree that low, the choice is between a torus or lattice in which every satellite is both compute and router. A Clos in which some satellites give up their payload to become switches, which is the subject of Section 10.4. |
| Reliability and accuracy | 2–30 s, 6 | The transport chapter must also be replaced, because lossless RDMA with priority flow control assumes zero packet loss and microsecond recovery. In comparison, a free-space optical link fades, re-acquires over 2–30 s [33, Tab. 1], and still awaits congestion-control measurements in the papers covered by Chapter 6. |
| 結果與適用範圍 | 定性範圍 | 拓樸那一章是第一個必須整個換掉的。 |
| 容量與速率 | 64, 51.2 | 它的 Clos 網路建立在有 64 個以上埠的 51.2 Tbps 交換晶片上，而衛星的節點度數就是它能獨立指向的雷射終端數：目前的 Starlink 衛星是三個[39, §2.1]company，van Berkel 的環面分析用四到六個[101, §IV-A]preprint，Suncatcher 的晶格鄰域是八個[36, p. 4, Fig. 2]，而 Pénot 與 Balakrishnan 的 Clos 掃描最多到十二個[102]preprint。 |
| 結果與適用範圍 | 定性範圍 | 度數這麼低，選擇就落在兩者之間：一種是環面或晶格，每顆衛星既是運算節點也是路由器。 |
| 結果與適用範圍 | 10.4 | 另一種是 Clos，部分衛星放棄運算酬載變成交換器，這是第 10.4 節的主題。 |
| 結果與適用範圍 | 2–30 s, 6 | 傳輸層那一章也必須換掉，因為帶優先流量控制的無損 RDMA 預設鏈路封包遺失率為零、且能在微秒內恢復，而自由空間光鏈路會衰落、重新擷取要 2–30 s[33, Tab. 1]，而且在第 6 章能找到的任何論文裡，ISL 壅塞控制演算法的實測仍待開展。 |
Routing can use predicted geometry, while the front end depends on ground visibility and downlink capacity. Failure handling must include radiation and thermal pauses.
路由可利用預測幾何，前端則取決於地面可見性與下行容量。故障處理還需納入輻射與散熱暫停。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | The remaining chapters split between adaptation and replacement. |
| Predictable routing | Qualitative scope | Routing is adapted and in one respect becomes easier. This occurs because the geometry of a formation is known from ephemerides seconds to hours ahead, so paths can be computed from ephemerides. The equal-cost multipath hashing that terrestrial fabrics use to spread bursty flows gives way to schedules computed on a predictable time-varying graph. |
| Capacity and rate | 200, 4.8 TB | The front-end chapter covers dataset arrival and checkpoint departure, so orbital ground transfer requires a replacement design. TBIRD demonstrates 200 Gbps and 4.8 TB in roughly five minutes [37, §I–IV]. Suncatcher’s pilot plans radio ground links and calls optical links “eventually” necessary [36, pp. 2, 7]. |
| Radiation and thermal inputs | 17 rad, 5 krad, 150 rad, 15–20 | Failure analysis keeps checkpoint-and-restart arithmetic but changes its inputs. Tests report about one silent corruption per 17 rad and one functional interrupt per 5 krad per chip [36, pp. 5, 9]. The environment is roughly 150 rad yearly. Press reports add prototype chip shutdowns every 15–20 minutes for cooling (Payload. |
| Result and scope | Qualitative scope | IEEE ComSoc blog citing the NYT) press. |
| Result and scope | Qualitative scope | The sharing chapter, finally, is the one that mostly disappears. This occurs because an orbital cluster is a single-tenant machine whose owner controls the fabric. Thus, the fairness and isolation mechanisms it describes address a single owner’s needs. |
| 結果與適用範圍 | 定性範圍 | 剩下的章節各自落在調整與替換兩類。 |
| 結果與適用範圍 | 定性範圍 | 路由屬於調整，而且在某個面向上反而變簡單，因為編隊的幾何可以從星曆提前數秒到數小時得知，路徑可以由星曆算出來，地面網路用來分散爆發流量的等價多路徑雜湊也讓位給在可預測的時變圖上計算出的排程。 |
| 容量與速率 | 200, 4.8 TB | 前端那一章談的是資料集如何進入叢集、檢查點如何離開，它必須被換掉，因為通往地面的唯一路徑是下行鏈路，而其光學下行已展示的最高水準是 TBIRD 的 200 Gbps，在大約五分鐘的一次過頂內傳送 4.8 TB[37, §I–IV]，而 Suncatcher 的先導任務計畫用射頻做地面鏈路，並稱光學地面鏈路「終將」必要[36, pp. 2, 7]。 |
| 可靠度與準確率 | 17 rad, 5 krad, 150 rad, 15–20 | 故障那一章屬於調整，保留它的檢查點與重啟算術，但換上新的速率：約每 17 rad 一次無聲資料損毀、每顆晶片約每 5 krad 一次功能中斷，環境劑量約每年 150 rad[36, pp. 5, 9]，再加上據報已升空的 MVP 原型機每 15–20 分鐘就必須把晶片關機降溫的熱工作週期（Payload。 |
| 結果與適用範圍 | 定性範圍 | IEEE ComSoc 部落格引述 NYT）press。 |
| 結果與適用範圍 | 定性範圍 | 最後，共用那一章幾乎整個消失，因為軌道叢集是單一租戶的機器，擁有者同時擁有網路，它所描述的公平與隔離機制僅需處理單一擁有者的需求。 |
| Companion chapter姊妹教材章節 | Verdict結論 | Satellite fact that forces the change迫使改變的衛星事實 | Where it is treated在哪裡處理 |
|---|---|---|---|
| 3 Parallelism3 平行化 | keep保留 | One more tier: ICI 6.4 Tbps per chip inside, laser terminals outside (TPU v6e docs company. [36, p. 3])多一層：機內每晶片 ICI 6.4 Tbps，機外雷射終端（TPU v6e 文件 company。[36, p. 3]） | 10.4 |
| 4 Collectives4 集體通訊 | adapt調整 | Geometry cycles twice per orbit, 100–200 m spacing. Rate depends on range with spatial multiplexing [36, pp. 3–4]幾何每圈變化兩次，間距 100–200 m。空間多工下速率隨距離改變[36, pp. 3–4] | 13.2 |
| 5 Step budget5 步時間預算 | keep保留 | Identity unchanged. Inputs change恆等式保持成立，只有輸入改變 | 10.4 |
| 6 Topology6 拓樸 | replace替換 | Degree = terminals: 3 (Starlink company), 4–6 (van Berkel preprint), 8 (Suncatcher), ≤12 (Pénot preprint). Switch ASIC absent度數＝終端數：3（Starlink company）、4–6（van Berkel preprint）、8（Suncatcher）、≤12（Pénot preprint）。交換晶片待納入設計 | 10.4, 11.5 |
| 7 Transport7 傳輸層 | replace替換 | FSO fades. Acquisition 2–30 s ([33, Tab. 1]). ISL congestion-control measurements pending光鏈路會衰落。擷取 2–30 s（[33, Tab. 1]）。ISL 壅塞控制待實測 | 6.6 |
| 8 Routing8 路由 | adapt調整 | Geometry predictable from ephemerides. Schedules replace ECMP hashing幾何可由星曆預測。排程取代 ECMP 雜湊 | 5.4 |
| 9 Front end9 前端 | replace替換 | Ground path is a downlink: TBIRD 200 Gbps, 4.8 TB per pass [37]. Suncatcher pilot uses radio [36, p. 2]地面路徑是下行鏈路：TBIRD 200 Gbps、每次過頂 4.8 TB[37]。Suncatcher 先導任務用射頻[36, p. 2] | 13.5 |
| 10 Failures10 故障 | adapt調整 | SDC 1/17 rad, SEFI 1/5 krad per chip, 150 rad/yr [36, pp. 5, 9]. 15–20 min duty cycle pressSDC 每 17 rad 一次、SEFI 每晶片每 5 krad 一次、每年 150 rad[36, pp. 5, 9]。15–20 分鐘工作週期 press | 10.5 |
| 11 Sharing11 共用 | drop省略 | Single tenant owns compute and fabric單一租戶同時擁有運算與網路 | — |
Original figure note原始圖說出處
Drawn for this primer from Google Cloud TPU v6e docs, Agüera y Arcas et al. Joule 2026, Chaudhry & Yanikomeroglu OJCOMS 2022, Schieler et al. ICSOS 2023, van Berkel arXiv 2607.14172, Pénot & Balakrishnan arXiv 2605.15335.10.4Bisection: torus vs Clos in orbit對切頻寬：軌道上的環面與 Clos
Recall
回顧
Section 5.2 treated bisection as a cut in a constellation graph with a few hundred Gbps per edge. The same quantity now carries the whole gradient traffic of a training job. The edge rate is whatever a laser terminal can close at 100 m.
第 5.2 節把對切當成星座圖上每條邊幾百 Gbps 的一個切面。現在同一個量要承載整個訓練工作的梯度流量，而邊的速率是雷射終端在 100 m 距離能閉合的任何速率。
Van Berkel’s comparison combines topology, port count and link rate. Separating these factors identifies the torus scaling penalty and the engineering choices behind the headline gap.
van Berkel 的比較同時包含拓樸、埠數與鏈路速率。分解這些因素，才能辨識環面的縮放代價及主要差距背後的工程選擇。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | The one quantitative comparison of orbital fabrics in the literature is van Berkel's. It is worth reading closely because the headline ratio hides three separate factors. |
| Capacity and rate | 8,000, 72, 2:1, 100 GB/s, 2, 89, 90, 3, 20, 100 Gb/s | At N = 8,000, the terrestrial Clos has 8,000 racks, W = 72 ToR ports, 2:1 oversubscription and 100 GB/s links. The compared tori have 89 × 90 satellites in 2-D and 20 × 20 × 20 in 3-D. Both use 100 Gb/s laser ISLs. |
| Latency comparison | 28,800 TB/s, 2.25 TB/s, 10 TB/s, 1, 12,880, 2,880, 4 µs, 1,800 µs, 600 µs, 1 µs, 20 µs | Bisection bandwidths are 28,800 TB/s for Clos, 2.25 TB/s for the 2-D torus and 10 TB/s for the 3-D torus. The stated ratio is 1 : 12,880 : 2,880. Corresponding diameter latencies are 4 µs, 1,800 µs and 600 µs. Assumed forwarding times are 1 µs per Clos hop and 20 µs per torus hop [101, §IV-A, Tab. 3] preprint. |
| Result and scope | 2, 1/2, 2/3, 3 | The torus formulas are \(2N^{1/2}B\) in two dimensions and \(2N^{2/3}B\) in three. Their exponents match Pénot and Balakrishnan’s independent mesh results [102, Tab. 2] preprint. Van Berkel’s 2-D text omits the factor of two that Table 3 includes. |
| Result and scope | Qualitative scope | The calculations here follow the table. |
| Capacity and rate | 12,880, 8, 100 GB/s, 100 Gb/s, 18, 72, 4, 89 | The 12,880 ratio separates into three factors in the table. Link rate contributes 8, comparing 100 GB/s with 100 Gb/s. Link count contributes 18, comparing 72 ports with 4 terminals. Topology contributes the remaining roughly 89, or \(\sqrt{N}\). |
| Torus topology cost | 100 | The first two factors are engineering choices that Suncatcher-class links partly reverse. The intrinsic torus factor is \(\sqrt{N}\). It underlies the paper’s “100+” cost increase. The paper therefore places torus-wired frontier training within two to three years outside its credible scenarios (§IV-E, §V). |
| 結果與適用範圍 | 定性範圍 | 文獻中對軌道網路架構唯一的定量比較來自 van Berkel，值得細讀，因為那個醒目的比值裡藏著三個獨立的因子。 |
| 結果與適用範圍 | 8,000, 72, 2:1, 100 GB/s, 89, 90, 2, 20, 3, 100 Gb/s, 28,800 TB/s, 2.25 TB/s, 10 TB/s, 1, 12,880, 2,880, 1 µs, 20 µs, 4 µs, 1,800 µs, 600 µs | 對 N = 8,000 個節點，該文把一個地面 Clos（8,000 個機架、每個機櫃頂交換器 W = 72 埠、2:1 超額訂閱、100 GB/s 鏈路）與一個 89 × 90 衛星的 2-D 環面及一個 20 × 20 × 20 的 3-D 環面相比，後兩者都用 100 Gb/s 雷射 ISL 接線，得到的對切頻寬各自是 28,800 TB/s、2.25 TB/s 與 10 TB/s，比值 1 : 12,880 : 2,880，網路直徑則在 Clos 每跳 1 µs、環面每跳 20 µs 的假設下各自為 4 µs、1,800 µs 與 600 µs[101, §IV-A, Tab. 3]preprint。 |
| 結果與適用範圍 | 2, 1/2, 2/3, 3 | 環面的公式是二維 \(2N^{1/2}B\)、三維 \(2N^{2/3}B\)，與 Pénot 和 Balakrishnan 獨立推得的網格縮放指數一致[102, Tab. 2]preprint，van Berkel 的正文省略了 2-D 的 2 倍因子，Table 3 則包含它，本章計算採用表中的數值。 |
| 容量與速率 | 12,880, 8, 100 GB/s, 100 Gb/s, 18, 72, 4, 89 | 下方算式表把 12,880 分解之後可以看到：8 倍來自鏈路速率（100 GB/s 對 100 Gb/s），18 倍來自每節點鏈路數（72 埠對 4 個終端），只有剩下的約 89 倍，也就是 \(\sqrt{N}\)，才是拓樸本身。 |
| 結果與適用範圍 | 定性範圍 | 這個分解對軌道很重要，因為前兩個因子是工程選擇，Suncatcher 等級的鏈路能部分扭轉。 |
| 結果與適用範圍 | 100 | 而 \(\sqrt{N}\) 是任何環面的本質，也是該文斷定環面接線的前沿訓練會帶來「100 倍以上」成本增加、兩三年內超出其可信情境的原因（§IV-E、§V）。 |
Bisection intensity relates a workload’s arithmetic to the traffic crossing its cut. A Clos increases cut capacity by assigning part of the satellite fleet to switching.
對切強度將工作負載的運算量與穿越切面的流量連結起來。Clos 透過讓部分衛星專責交換來增加切面容量。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | van Berkel frames the requirement through a single ratio. |
| Capacity and rate | 6 FLOP, 12, 1, 500 | bisection intensity divides floating-point work per step by bytes crossing the bisection per step. Data-parallel inputs are k = 6 FLOP per parameter per token and c ≈ 12 cut-crossing bytes per parameter. With about 1 M batch tokens, the result is \(I = kB_{\text{tokens}}/c \approx 500\) kFLOP per byte [101, §IV-C, Eqs. 2–3] preprint. |
| Capacity and rate | Qualitative scope | A cluster whose total compute rate divided by its bisection bandwidth exceeds that figure must expose some gradient exchange even with complete overlap |
| Capacity and rate | 8,000, 918 TFLOPS, 14.7 TB/s | and the calc table shows that 8,000 Trillium chips at 918 TFLOPS each would need about 14.7 TB/s of bisection |
| Result and scope | 2, 3 | which the 2-D torus misses by a factor of six, the 3-D torus by a factor of one and a half |
| Result and scope | Qualitative scope | and the Clos exceeds by three orders of magnitude. |
| Result and scope | 12, 10.1 | The accounting is his, including the choice of 12 bytes per parameter compared with the bf16 gradient volume Section 10.1 used. The panel below lets you vary each input. |
| Result and scope | Qualitative scope | Pénot and Balakrishnan add the compute cost of a Clos. A switch satellite dedicates its payload to switching. |
| Orbital configuration | 4, 6, 2, 3, 1, 9 | Their VL2-like fabric has k ports per switch satellite and L layers. Compute satellites comprise the fraction \(r(k,L) = k/(k+4L-6)\) [102, Eq. 8]. Line-of-sight mapping is usually infeasible at L ≤ 2 and feasible across their sweep at L ≥ 3. The smallest usable L satisfies \((k/2)^{L-1} + (2L-3)(k/2)^{L-2} \ge N\) (Eq. 9) preprint. |
| Capacity and rate | 8, 28, 3, 144, 4, 81, 36 | At k = 8, capacity is 28 satellites for L = 3 and 144 for L = 4. An 81-satellite cluster therefore needs four layers. Only 36 satellites remain available for compute, which qualifies the table’s bisection advantage. |
| 結果與適用範圍 | 定性範圍 | van Berkel 用一個比值來表述需求。 |
| 結果與適用範圍 | 定性範圍 | 一個工作負載的對切強度是每一步的浮點運算量除以每一步必須穿越對切面的位元組數。 |
| 模型與運算 | 6 FLOP, 12, 1, 500 | 對資料平行訓練，他取每參數每 token k = 6 FLOP、每參數穿越切面 c ≈ 12 位元組、批次約 1 M tokens，得到 \(I = kB_{\text{tokens}}/c \approx 500\) kFLOP 每位元組[101, §IV-C, Eqs. 2–3]preprint。 |
| 結果與適用範圍 | 定性範圍 | 一個叢集若總算力除以對切頻寬超過這個數字，即使充分重疊，仍會暴露部分梯度交換。 |
| 容量與速率 | 8,000, 918 TFLOPS, 14.7 TB/s, 2, 3 | 算式表顯示 8,000 顆各 918 TFLOPS 的 Trillium 晶片需要約 14.7 TB/s 的對切頻寬，2-D 環面差了六倍、3-D 環面差了一倍半，而 Clos 多出三個數量級。 |
| 結果與適用範圍 | 12, 10.1 | 這套會計是他的，包括選用每參數 12 位元組，第 10.1 節則採 bf16 梯度量，下方面板讓你逐一改動每個輸入。 |
| 結果與適用範圍 | 定性範圍 | Pénot 與 Balakrishnan 補上了 Clos 故事缺少的另一半：擔任交換器的衛星僅承擔交換功能。 |
| 結果與適用範圍 | 4, 6, 2, 3, 1, 9 | 在他們類 VL2 的架構中，交換衛星有 k 個埠、共 L 層，仍是運算節點的衛星比例為 \(r(k,L) = k/(k+4L-6)\)[102, Eq. 8]，對應到視線圖上在 L ≤ 2 時通常超出可行範圍、在 L ≥ 3 時整個掃描範圍都可行，而最小可用的 L 是第一個滿足 \((k/2)^{L-1} + (2L-3)(k/2)^{L-2} \ge N\) 的值（Eq. 9）preprint。 |
| 結果與適用範圍 | 8, 3, 28, 4, 144, 81, 36 | k = 8 時這個容量在 L = 3 是 28 顆、L = 4 是 144 顆，所以 81 顆衛星的叢集需要四層，而 81 顆中只剩 36 顆是運算節點，讀算式表裡的對切數字時必須對照這個代價。 |
| Question問題 | Calculation算式 | Result結果 |
|---|---|---|
| van Berkel's 2-D torus bisection, reproduced重算 van Berkel 的 2-D 環面對切頻寬 | \(2\sqrt{8000}\times 12.5\ \text{GB/s}\) | 2.24 TB/scomputed. Table 3 gives 2.25. 100 Gb/s = 12.5 GB/s計算值。Table 3 為 2.25。100 Gb/s = 12.5 GB/s |
| 3-D torus and Clos, reproduced重算 3-D 環面與 Clos | \(2\cdot 8000^{2/3}\cdot 12.5;\ \tfrac12\cdot 8000\cdot 72\cdot 100\) | 10.0 TB/s. 28,800 TB/scomputed from van Berkel §IV-A inputs由 van Berkel §IV-A 的輸入計算 |
| Decomposing the 12,880 ratio分解 12,880 這個比值 | \(\sqrt{N}\times\dfrac{W}{4}\times\dfrac{B_{\text{Clos}}}{B_{\text{torus}}}\) | 89.4 × 18 × 8 = 12,880computed. Topology × link count × link rate計算值。拓樸 × 鏈路數 × 鏈路速率 |
| Bisection needed at van Berkel's 500 kFLOP/B for 8,000 Trillium chips以 van Berkel 的 500 kFLOP/B 計，8,000 顆 Trillium 所需對切頻寬 | \(\dfrac{8000\times 918\ \text{TFLOPS}}{5\times 10^{5}\ \text{FLOP/B}}\) | 14.7 TB/scomputed. 918 TFLOPS from TPU v6e docs [company]. 2-D torus short 6.5×, 3-D short 1.5×計算值。918 TFLOPS 取自 TPU v6e 文件 [company]。2-D 環面差 6.5 倍，3-D 差 1.5 倍 |
| Pénot's compute fraction at L = 3Pénot 在 L = 3 時的運算比例 | \(r = \dfrac{k}{k+6}\) | 0.40 · 0.50 · 0.57 · 0.63 · 0.67 for k = 4, 6, 8, 10, 12（k = 4、6、8、10、12）computed from Eq. 8由 Eq. 8 計算 |
| Layers and compute satellites for N = 81, k = 8N = 81、k = 8 時的層數與運算衛星數 | \((k/2)^{L-1}+(2L-3)(k/2)^{L-2}\ge 81\) | L = 4 (28 at L = 3, 144 at L = 4). R = 0.44. 36 computecomputed from Pénot Eq. 9 and Eq. 8由 Pénot Eq. 9 與 Eq. 8 計算 |
| 81 satellites, 8 apertures of 9.6 Tb/s each: 2-D torus vs Clos bisection81 顆衛星、各 8 個 9.6 Tb/s 孔徑：2-D 環面與 Clos 的對切 | \(2\sqrt{81}\cdot 9.6\cdot\tfrac{8}{4};\ \ 36\cdot 8\cdot 9.6/2\) | 346 Tb/s vs 1,382 Tb/scomputed with the Panel 10.1 model. Per compute satellite 4.3 vs 38 Tb/s以面板 10.1 的模型計算。每顆運算衛星 4.3 對 38 Tb/s |
Original figure note原始圖說出處
Drawn for this primer from van Berkel arXiv 2607.14172 Table 3 and Pénot & Balakrishnan arXiv 2605.15335 Fig. 12, Table 3, Eq. 8.Panel 10.1 applies the two-line model. Select node count, terminals per satellite, aperture rate, topology, model size, compute rate and batch. The three published aperture rates are presets. The panel computes bisection from the stated formulas and all-reduce bytes per worker from Section 10.1. It divides those bytes by worker injection rate to obtain all-reduce time.
Compute time uses six FLOP per parameter per token divided by cluster compute rate. Selected overlap then determines exposed communication and the MFU-like ratio. The chart plots bisection versus node count for four topologies at the selected terminal count and rate. It marks the selected cluster and van Berkel’s three Table 3 references at N = 8,000. Their positions differ because his Clos uses 72 ports at 100 GB/s, while his tori use 100 Gb/s links.
面板 10.1 把這些零件放進兩行模型。你選擇節點數、每顆衛星的終端數、每孔徑速率（三個已發表數值做為預設）、拓樸、模型大小、每節點算力與批次大小，面板就用上面的公式算出對切頻寬、用第 10.1 節算出每工作者的全歸約位元組、以工作者的注入速率算出全歸約時間、以每參數每 token 六次浮點運算除以叢集算力算出運算時間，最後在你給排程器的任何重疊比例下算出暴露比例與類 MFU 的比值。圖表在你選的終端數與速率下畫出四種拓樸的對切頻寬對節點數曲線，標出你的叢集，並在 N = 8,000 處放上 van Berkel Table 3 的三個參考點。它們與曲線分開標示，因為他的 Clos 用的是 72 個 100 GB/s 的埠、環面用的是 100 Gb/s 鏈路，這正是上面那個分解想說明的事。
Stop and think停下來想一想
Select a 2-D torus with 4 terminals and the 100 Gb/s Starlink preset. Use a 405 B model and 16 M tokens per step, matching Llama 3’s batch. Raise N from 16 toward 8,192 and find where exposed communication exceeds one half with 50% overlap. Determine whether Clos reduces exposure or exchanges it for lost compute satellites. Repeat with the Suncatcher preset and 8 terminals, then explain the gain beyond the link-rate ratio.
把 2-D 環面設為 4 個終端、100 Gb/s 的 Starlink 預設、405 B 模型、每步 16 M tokens（Llama 3 的批次），然後把 N 從 16 往 8,192 調。在 50% 重疊下，暴露比例在哪個 N 超過一半？換成 Clos 能解決它，還是只是把暴露時間換成損失的運算衛星？接著改用 Suncatcher 預設、8 個終端重做一次，想想為什麼答案的變化遠大於鏈路速率的比值。
10.5Orbital propagation advantage and reliability cost軌道傳播優勢與可靠度成本
Recall
回顧
Section 2.1 gives vacuum light-time of 3.336 µs/km, 47% faster than fiber. Chapter 8 defines silent data corruption and SEFI. These radiation effects corrupt or halt computation while leaving the chip physically intact.
A compact formation has short propagation paths. Per-hop forwarding inside the satellites can dominate its latency budget, while constellation-scale paths add much longer propagation delays.
緊密編隊的傳播路徑短，衛星內部的逐跳轉送可能主導延遲預算。全星座路徑則有大得多的傳播延遲。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Latency comparison | Qualitative scope | Latency is the one dimension on which an orbital cluster has favorable propagation conditions. This occurs because vacuum is faster than glass and a formation is physically small. |
| Orbital configuration | 2 km, 81, 1 km, 6.7 µs | Light crosses the 2 km diameter of a Suncatcher-sized formation, whose 81 satellites sit within a 1 km radius [36], in about 6.7 µs |
| Result and scope | 100–200 m | and it crosses a single 100–200 m neighbor link in a third to two-thirds of a microsecond, as the calc table below computes |
| Latency comparison | Qualitative scope | This cluster propagation time fits within Meta’s reported “tens of microseconds” for multi-hop pipeline- and data-parallel paths [77, §3.3.2] preprint. |
| Latency comparison | 20 µs, 1 µs | The comparison excludes forwarding time inside each satellite, which still awaits measurement. Van Berkel assumes 20 µs per torus hop and 1 µs per Clos hop [101, §IV-B] preprint. |
| Result and scope | 81, 160 µs, 6.7 µs | An 81-satellite eight-neighbor lattice has an eight-hop diameter. Under these assumptions, crossing the cluster takes about 160 µs of forwarding and 6.7 µs of propagation. |
| Latency comparison | Qualitative scope | so the latency budget of a formation is set by the router inside the spacecraft |
| Latency comparison | Qualitative scope | with vacuum propagation contributing a smaller share |
| Onboard switching delay | Qualitative scope | and the design question becomes how fast an on-board switch can be made rather than how far apart the satellites are. |
| Result and scope | 360, 1,970 km, 2.1, 1.2, 6.6 ms, 10.6 | Constellation neighbor links span roughly 360–1,970 km (Section 2.1), giving 1.2–6.6 ms of light-time per hop. That exceeds an entire terrestrial training fabric’s diameter latency by two orders of magnitude. Section 10.6 therefore separates workloads by regime alongside rate. |
| 結果與適用範圍 | 定性範圍 | 延遲是軌道叢集具備有利傳播條件的面向，因為真空比玻璃快，而編隊在物理上又很小。 |
| 軌道配置 | 81, 1 km, 2 km, 6.7 µs, 100–200 m | Suncatcher 規模的編隊把 81 顆衛星放在 1 km 半徑之內[36]，光穿越它 2 km 的直徑約需 6.7 µs，穿越單一條 100–200 m 的鄰居鏈路只需三分之一到三分之二微秒，下方計算表會算出這些值。 |
| 結果與適用範圍 | 定性範圍 | 這讓整個叢集輕鬆落在 Meta 為其流水線與資料平行流量所經多跳路徑報告的「數十微秒」之內[77, §3.3.2]preprint。 |
| 結果與適用範圍 | 20 µs, 1 µs | 這個比較遺漏的是封包待在每顆衛星「裡面」，相對於衛星「之間」的時間，這項時間仍待論文提供量測，van Berkel 以假設的方式把環面每跳設為 20 µs、Clos 每跳設為 1 µs[101, §IV-B]preprint。 |
| 結果與適用範圍 | 81, 160 µs, 6.7 µs | 在這個假設下，一個直徑為八跳的 81 顆衛星八鄰居晶格，把封包轉送過整個叢集要花約 160 µs，而封包真正在飛行的時間只有 6.7 µs，所以編隊的延遲預算是由太空船內部的路由器、所主導，外部真空傳播的佔比較小，設計問題也就從衛星相距多遠，變成機上交換器能做得多快。 |
| 軌道配置 | 360, 1,970 km, 2.1, 1.2, 6.6 ms, 10.6 | 全星座體制則是另一回事，因為約 360 到 1,970 km 的鄰居跳距（第 2.1 節）光是光行時間每跳就要 1.2 到 6.6 ms，比地面訓練網路的整個直徑還多兩個數量級，這也是第 10.6 節按各種體制的條件來分類工作負載的原因之一。 |
Synchronous collectives require every contribution and stall after a link failure. Independent link-loss assumptions therefore affect training differently from reroutable inference requests.
同步集體通訊需要全部參與者的貢獻，鏈路失效會使其停擺。因此獨立鏈路失效假設對訓練與可改道的推論請求各有影響。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Result and scope | Qualitative scope | Reliability reverses the picture, and the reason lies in what a synchronous step needs. |
| Reliability comparison | Qualitative scope | An all-reduce completes only when every participant's contribution has arrived. Thus, a single failed link stalls every worker until the collective is re-routed around it or the job falls back to its last checkpoint. This means that the rate at which links fail sets how often a training job pays that cost. |
| Result and scope | 0.057%, 5, 60, 1–2, 0.95 | Alibaba’s production fabric loses 0.057% of NIC-to-ToR links monthly. Together with ToR critical errors and 5K–60K daily flaps, this causes 1–2 monthly training-job crashes [65, §2.3, Fig. 5]. SpaceMoE is the satellite paper here with a link-failure model. It assumes independent laser-ISL survival of 0.95 per topology snapshot [78, §II] preprint. |
| Result and scope | 88 | The per-event probability is already 88 times higher |
| Result and scope | 272, 9, 13.6 | The time units widen the gap: HPN measures monthly failure, while SpaceMoE specifies snapshot failure. A 9 × 9 eight-neighbor lattice has 272 links. SpaceMoE’s assumption leaves about 13.6 down per snapshot. The chance of all links operating together falls below one in a million. |
| Reliability comparison | 0.16 | while the HPN rate predicts about 0.16 failures across the same links in a whole month. |
| Model and compute | Qualitative scope | SpaceMoE’s loss assumption suits reroutable inference requests, which can reach another copy of an expert after a failure. Synchronous training needs each worker’s original gradient shard. This difference makes the same loss model tolerable for inference and fatal for training. |
| 結果與適用範圍 | 定性範圍 | 可靠度則把局面翻轉過來，原因在於同步訓練步需要什麼。 |
| 結果與適用範圍 | 定性範圍 | 全歸約要等每個參與者的貢獻都到齊才算完成，所以只要一條鏈路失效，所有工作者都得停下來，直到集體通訊繞過它，或整個工作退回上一個檢查點。 |
| 結果與適用範圍 | 定性範圍 | 也就是說，鏈路失效的頻率決定了訓練工作多常要付出這個代價。 |
| 結果與適用範圍 | 0.057%, 5, 60, 1–2 | 阿里巴巴的生產網路每月有 0.057% 的網路卡到 ToR 鏈路失效，再加上 ToR 嚴重錯誤與每天 5K–60K 次鏈路抖動，每月造成 1–2 次訓練工作當機[65, §2.3, Fig. 5]。 |
| 結果與適用範圍 | 0.95 | 而唯一真的為鏈路失效建模的衛星論文 SpaceMoE，讓每條雷射 ISL 在每個拓樸快照中各自獨立地以 0.95 的機率存活[78, §II]preprint。 |
| 結果與適用範圍 | 88, 9, 272, 13.6 | 單一事件的機率已經高了 88 倍，而單位又把差距拉得更大，因為一個是每月的比例、另一個是每個快照的比例：套用在 9 × 9 八鄰居晶格的 272 條鏈路上，SpaceMoE 的假設讓每個快照平均有約 13.6 條鏈路斷線，所有鏈路同時正常的機率低於百萬分之一。 |
| 結果與適用範圍 | 0.16 | 而 HPN 的失效率預測同樣這些鏈路整整一個月只會壞約 0.16 條。 |
| 結果與適用範圍 | 定性範圍 | SpaceMoE 的假設就它自己的用途而言是合理的，因為推論請求遇到斷鏈時可以改送到該專家的另一個副本，但同步訓練步需要各工作者原本的梯度分片，正是這種差異，讓同一個失效模型對推論可以容忍、對訓練卻是致命的。 |
Formation-link failure rates await measurement. Existing evidence instead supplies acquisition delays, chip radiation rates and reported thermal pauses, which define separate recovery requirements.
編隊鏈路的失效率待量測。現有證據提供了擷取延遲、晶片輻射事件率與媒體報導的散熱暫停，各自對應各種恢復需求。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Model and compute | Qualitative scope | These figures describe production ground links and a constellation model. |
| Reliability comparison | 100 m | The literature leaves the failure rate of a 100 m orbital optical link unspecified. |
| Result and scope | 2–30 s | Chaudhry and Yanikomeroglu report 2–30 s reacquisition for next-generation terminals [33, Tab. 1]. A lost lock therefore interrupts many training steps. Terrestrial transport absorbs link flaps in milliseconds. |
| Result and scope | Qualitative scope | Radiation adds orbital failure modes, and Google's proton tests on Trillium supply the only accelerator rates. |
| Result and scope | 650 km, 150 rad, 17 rad, 5 krad, 0.03 | A shielded chip at 650 km absorbs about 150 rad yearly. Tests found about one silent corruption per 17 rad and one functional interrupt per 5 krad per chip [36, pp. 5, 9]. The table converts these rates to roughly nine corruptions and 0.03 interrupts per chip-year. |
| Reliability comparison | 8,000, 20, 1–2 | Across van Berkel’s 8,000 chips, interrupts reach about 20 resets monthly, an order above HPN’s 1–2 crashes. SDC is more frequent and leaves the error flag clear. At that scale, one occurs every seven to eight minutes. |
| Model and compute | 5 | A data-parallel job would incorporate wrong gradients several times hourly unless it verifies results. Detection requires redundant computation or checksummed reductions. The Suncatcher authors state that the effect on training “requires further study” (p. 5). |
| Result and scope | 8, 15–20 | Chapter 8’s thermal duty cycle adds periodic interruptions. The flown MVP reportedly powers down its chips every 15–20 minutes (Payload. |
| Result and scope | Qualitative scope | IEEE ComSoc blog citing the NYT) press. Such interruptions are predictable enough for checkpoint scheduling, while limiting each continuous run’s length. |
| 結果與適用範圍 | 100 m | 這兩個數字各自描述地面生產鏈路與星座模型，軌道上 100 m 光學鏈路的失效率仍待量測。 |
| 結果與適用範圍 | 2–30 s | 確實存在的是新一代終端的重新擷取時間，Chaudhry 與 Yanikomeroglu 的綜述列為 2–30 s[33, Tab. 1]，這讓每一次失鎖都變成持續好幾個訓練步的中斷，而地面上的鏈路抖動在幾毫秒內就被傳輸層吸收了。 |
| 結果與適用範圍 | 定性範圍 | 輻射還會帶來額外的軌道失效模式，Google 對 Trillium 所做的質子測試提供了唯一的加速器數據。 |
| 軌道配置 | 650 km, 150 rad, 17 rad, 5 krad, 0.03 | 在 650 km、有屏蔽的軌道上，一顆晶片每年吸收約 150 rad，而測試發現每約 17 rad 發生一次無聲資料損毀、每顆晶片每約 5 krad 發生一次功能中斷[36, pp. 5, 9]，計算表據此換算出每顆晶片每年約九次無聲損毀與 0.03 次中斷。 |
| 結果與適用範圍 | 8,000, 20, 1–2 | 中斷率單看每顆晶片的值較小，直到把它乘上整個叢集：在 van Berkel 的 8,000 顆晶片規模下，它相當於每月約 20 次重置，比 HPN 的 1–2 次當機高一個數量級。 |
| 結果與適用範圍 | 定性範圍 | 無聲損毀則更糟，因為 SDC 發生時錯誤旗標仍維持空白，而在同樣規模下每七到八分鐘就發生一次。 |
| 結果與適用範圍 | 5 | 因此資料平行工作每小時會把好幾個錯誤的梯度併進共享模型，因此需要投入冗餘計算或帶校驗和的歸約來偵測，而 Suncatcher 的作者自己也說單事件效應對訓練的影響「需要進一步研究」（p. 5）。 |
| 結果與適用範圍 | 8, 15–20 | 第 8 章的熱工作週期再加上第三種週期性中斷，因為據報導已發射的 MVP 每 15–20 分鐘就讓晶片斷電一次（Payload。 |
| 結果與適用範圍 | 定性範圍 | 引用 NYT 的 IEEE ComSoc 部落格）press。 |
| 結果與適用範圍 | 定性範圍 | 這種中斷雖然可預測到足以安排檢查點，卻限制了任何一次連續執行的長度。 |
| Question問題 | Calculation算式 | Result結果 |
|---|---|---|
| Light-time across a 2 km formation光穿越 2 km 編隊的時間 | \(2\ \text{km}\times 3.336\ \mu\text{s/km}\) | 6.7 µscomputed. R = 1 km from [36]. Vacuum delay from Section 2.1計算值。R = 1 km 取自 [36]。真空延遲取自第 2.1 節 |
| Light-time across one 100–200 m neighbor link光穿越一條 100–200 m 鄰居鏈路的時間 | \((0.1\text{–}0.2)\ \text{km}\times 3.336\ \mu\text{s/km}\) | 0.33–0.67 µscomputed. Spacing from [36, p. 4]計算值。間距取自 [36, p. 4] |
| Forwarding time across an 81-satellite lattice at van Berkel's per-hop assumption以 van Berkel 每跳假設，轉送穿越 81 顆衛星晶格的時間 | \(8\ \text{hops}\times 20\ \mu\text{s}\) | 160 µscomputed. 9 × 9 eight-neighbor lattice has diameter 8. 20 µs per hop is an assumption [101, §IV-B]計算值。9 × 9 八鄰居晶格直徑為 8。每跳 20 µs 是假設[101, §IV-B] |
| Light-time of one constellation hop全星座一跳的光行時間 | \((363\text{–}1969)\ \text{km}\times 3.336\ \mu\text{s/km}\) | 1.2–6.6 mscomputed. Hop lengths from Section 2.1計算值。跳距取自第 2.1 節 |
| Ratio of per-event link-failure probabilities, SpaceMoE vs HPN單一事件鏈路失效機率之比：SpaceMoE 對 HPN | \(0.05 / 0.00057\) | 88×computed. Per snapshot vs per month, so the true gap is wider計算值。一個是每快照、一個是每月，真正的差距更大 |
| Links down per snapshot, and chance all are up, in a 9 × 9 eight-neighbor lattice (272 links) under SpaceMoE's model在 SpaceMoE 模型下，9 × 9 八鄰居晶格（272 條鏈路）每快照斷線數與全部正常的機率 | \(272\times 0.05;\ \ 0.95^{272}\) | 13.6. 8.7 × 10⁻⁷computed. 272 = 72 + 72 + 128 straight and diagonal links, open boundaries計算值。272 = 72 + 72 + 128 條直線與對角鏈路，邊界開放 |
| Expected failures in the same 272 links over a month at HPN's rate以 HPN 的失效率計，同樣 272 條鏈路一個月的預期失效數 | \(272\times 0.00057\) | 0.16computed. Rate from [65, §2.3]計算值。失效率取自 [65, §2.3] |
| Silent data corruptions per chip-year每顆晶片每年的無聲資料損毀次數 | \(150\ \text{rad/yr}\ \div\ 17\ \text{rad}\) | ≈ 9 (8.8)computed. Both inputs from [36, pp. 5, 9]計算值。兩個輸入都取自 [36, p. 5] |
| SDC interval across 8,000 chips8,000 顆晶片的 SDC 間隔 | \(\dfrac{365\times 1440\ \text{min}}{8000\times 8.8}\) | one per 7.4 min (≈ 193 per day)computed. N = 8,000 is van Berkel's cluster size計算值。N = 8,000 為 van Berkel 的叢集規模 |
| Functional interrupts per chip-year, and per month across 8,000 chips每顆晶片每年的功能中斷次數，以及 8,000 顆晶片每月的次數 | \(150/5000;\ \ 8000\times 0.03/12\) | 0.03. ≈ 20 per monthcomputed. Compare HPN's 1–2 job crashes per month計算值。對照 HPN 每月 1–2 次工作當機 |
The table compares the two failure regimes by failure mode. Terrestrial entries measure production fleets. Orbital entries are assumptions or ground chip tests. The comparison therefore provides lower bounds on the problem, with actual orbital behavior still requiring measurement.
下表把兩種失效體制並排比較，每一列是一種失效模式。它呈現的模式是，每個地面數字都是生產機群的量測，而軌道數字則來自模型假設或晶片地面測試，因此這項比較界定了問題的下界，軌道系統的實際表現仍待量測。
| Failure mode失效模式 | Terrestrial reference (measured)地面參考（實測） | Orbital figure軌道數字 | Effect on a synchronous step對同步訓練步的影響 |
|---|---|---|---|
| Link failure鏈路失效 | 0.057% of NIC–ToR links per month [65]每月 0.057% 的網路卡–ToR 鏈路[65] | 5% per snapshot, i.i.d., assumed (SpaceMoE) preprint. Formation-link rates await measurement每快照 5%、獨立同分佈，為假設（SpaceMoE）preprint。編隊鏈路的失效率待實測 | stalls the all-reduce until re-routed or restarted全歸約停擺，直到繞路或重新啟動 |
| Link flap and recovery鏈路抖動與恢復 | 5K–60K flaps per day across the fleet, absorbed by transport整個機群每天 5K–60K 次抖動，由傳輸層吸收 | re-acquisition 2–30 s per terminal [33]每個終端重新擷取 2–30 s[33] | an outage of many steps rather than a hiccup一次持續好幾步的中斷，持續時間高於短暫停頓 |
| Switch failure交換器失效 | 0.051% of ToR switches hit critical errors per month每月 0.051% 的 ToR 交換器發生嚴重錯誤 | data pending. A Clos switch satellite is itself a spacecraft exposed to SEFI資料待補充。Clos 的交換衛星本身就是會遭遇 SEFI 的太空船 | removes every path through that switch移除經過該交換器的所有路徑 |
| Job crash工作當機 | 1–2 training-job crashes per month每月 1–2 次訓練工作當機 | orbital training jobs await demonstration軌道訓練工作待示範 | rollback to the last checkpoint退回上一個檢查點 |
| Silent data corruption無聲資料損毀 | quantification pending in the three reference papers三篇參考論文的量化結果待補充 | ≈ 1 per 17 rad, ≈ 9 per chip-year at 150 rad/yr ([36]. Computed)每約 17 rad 一次，在每年 150 rad 下約每顆晶片每年 9 次（[36]。計算值） | a wrong gradient enters the model with the error flag clear錯誤梯度在錯誤旗標保持空白的情況下進入模型 |
| Functional interrupt功能中斷 | included in other categories in the reference papers參考論文將其併入其他類型 | ≈ 1 per 5 krad per chip, ≈ 20 per month at 8,000 chips ([36]. Computed)每顆晶片每約 5 krad 一次，8,000 顆晶片每月約 20 次（[36]。計算值） | a node reset, then rejoin and resynchronise節點重置，之後重新加入並重新同步 |
| Thermal duty cycle熱工作週期 | continuous cooling持續冷卻 | MVP reportedly powers down every 15–20 min press據報導 MVP 每 15–20 分鐘斷電一次 press | a predictable pause that bounds run length可預測的暫停，限制連續執行長度 |
10.6Three regimes: constellation-wide, formation cluster, single satellite三種體制：全星座、編隊叢集、單顆衛星
Recall
回顧
Chapter 9 found that satellite federated learning is limited by contact gaps measured in hours rather than by link rate. That every satellite-FL paper confines its ISLs to intra-plane rings.
第 9 章發現衛星聯邦學習受限於以小時計的接觸空檔，其鏈路容量則有餘裕，而且每篇衛星 FL 論文都把 ISL 限制在同軌道面的環上。
Constellation-wide systems combine long propagation paths with lower link rates. They fit contact-driven learning, content delivery and compute-bound inference under the stated assumptions.
全星座系統的傳播路徑長、鏈路速率較低。在所列假設下，適用工作包括依接觸安排的學習、內容遞送與運算受限推論。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Latency comparison | 10.3 | The three regimes differ by orders of magnitude in both rate and latency. Figure 10.3 therefore uses logarithmic axes, and workload assessment needs both measures. |
| Capacity and rate | 2, 5, 100, 200, 0.5 Gbit/s, 16 Mb/s | The constellation-wide regime uses Chapters 2 and 5’s laser links over hundreds to thousands of kilometers. Current Starlink satellites have three terminals. SpaceX claims 100 Gbps sustained and 200 Gbps peak company. Assumed rates are 200 Gbps in TinyLEO [39, §6.1] and “≥100 Gbps” in SpaceMoE [78] preprint. CollabLLM-LEO uses 0.5 Gbit/s [96, Tab. II] preprint, OrbitalBrain about 100 Mbps [94] and AsyncFLEO 16 Mb/s [92, Tab. II]. |
| Latency comparison | Qualitative scope | with every hop costing milliseconds and the graph changing as planes cross and terminals re-acquire. |
| Capacity and rate | 400 | All of those rates sit below the 400 Gbps a single terrestrial GPU owns |
| Latency comparison | Qualitative scope | so the regime suits work that exchanges one model update or one request at a time and tolerates seconds to hours of delay |
| Result and scope | 9 | which describes federated learning, whose binding constraint Chapter 9 showed to be contact gaps, content delivery |
| Result and scope | Qualitative scope | and the relaying of results from inference that one satellite has already completed. |
| Capacity and rate | 1.02–1.07 s, 7.28 GFLOPS, 100 | SpaceMoE’s expert-parallel inference belongs here. Its 7.28 GFLOPS flight board largely determines the reported 1.02–1.07 s per token. The ≥100 Gbps links therefore carry little traffic relative to capacity (Section 12.2). |
| 結果與適用範圍 | 10.3 | 已發表的研究可以分成三種體制，它們在兩個軸上同時相差好幾個數量級，這就是圖 10.3 用對數座標畫鏈路速率對每跳延遲的原因，也是為什麼判斷工作負載的適用性，需要同時考慮鏈路速率與每跳延遲。 |
| 容量與速率 | 2, 5, 100, 200 | 在全星座體制中，鏈路就是第 2 章與第 5 章研究的那些：相距數百到數千公里的衛星之間的雷射終端，目前的 Starlink 衛星每顆三個，SpaceX 聲稱持續 100 Gbps、峰值 200 Gbps company。 |
| 容量與速率 | 200, 100, 0.5 Gbit/s, 16 Mb/s | TinyLEO 以 200 Gbps 建模[39, §6.1]，SpaceMoE 以「≥100 Gbps」建模[78]preprint，CollabLLM-LEO 以 0.5 Gbit/s 建模[96, Tab. II]preprint，OrbitalBrain 約 100 Mbps[94]，AsyncFLEO 則是 16 Mb/s[92, Tab. II]。 |
| 結果與適用範圍 | 定性範圍 | 每一跳都要花上毫秒，圖也隨軌道面交會與終端重新擷取而改變。 |
| 容量與速率 | 400, 9 | 這些速率全都低於單顆地面 GPU 獨享的 400 Gbps，因此這個體制適合一次交換一個模型更新或一個請求、能容忍數秒到數小時延遲的工作，也就是聯邦學習（第 9 章已說明它的限制是接觸空檔）、內容傳遞，以及轉送單顆衛星已經完成的推論結果。 |
| 容量與速率 | 1.02–1.07 s, 7.28 GFLOPS, 100 | SpaceMoE 的專家平行推論也屬於這裡，而它之所以可行，是因為每 token 1.02–1.07 s 幾乎完全由 7.28 GFLOPS 的飛行級板卡決定，所以它 ≥100 Gbps 的鏈路使用量極低（第 12.2 節）。 |
Formation links approach on-board interconnect rates, making distributed training plausible under additional bisection and reliability conditions. Single-satellite inference keeps collective traffic local and has flight demonstrations.
編隊鏈路接近機上互連速率，在對切頻寬與可靠度條件成立時，具備分散式訓練的可能。單衛星推論則把集體流量留在本地，已有飛行示範。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Formation network | 100–200 m, 10, 24, 400, 9.6, 100, 800 | Suncatcher represents the formation cluster regime. Bounded relative motion keeps 100–200 m neighbor links continuously visible, with propagation below one microsecond per hop. The target is roughly 10 Tbps per link. Its 24 × 400G DWDM channels supply 9.6 Tbps on a 100 GHz grid. A bench demonstrator using commercial components achieved 800 Gbps in one direction [36, pp. 3, 8]. |
| Capacity and rate | 6.4, 10.4, 10.1 | Formation links alone reach the same rate decade as TPU v6e’s 6.4 Tbps ICI. Data- and pipeline-parallel training therefore become arithmetically plausible here. Van Berkel, Pénot and Balakrishnan, Section 10.4 and Panel 10.1 all examine this regime. |
| Result and scope | 10–30, 3 | The tier boundary persists within formations. Dozens of chips behind a few apertures create roughly 10–30× oversubscription against on-board ICI. As in Llama 3, tensor parallelism should stay within the spacecraft. Lighter pipeline and data-parallel collectives cross optical links. |
| Capacity and rate | 6.4 | The single-satellite regime keeps every byte of collective traffic on ICI, where a Trillium chip has 6.4 Tbps and light-time is negligible |
| Model and compute | 1, 2, 2026 | The single-satellite regime alone has flight heritage: Φ-Sat-1’s Myriad 2, Starcloud-1’s H100 and the four-TPU Suncatcher MVP run inference on board. The MVP launched on 1 October 2026 (Chapter 8, Section 11.1) press. Van Berkel’s roofline analysis gives satellite-local inference the same limits as the same terrestrial chip [101, §IV] preprint. |
| Result and scope | Qualitative scope | so this regime depends on the on-board interconnect. |
| 結果與適用範圍 | 100–200 m | 編隊叢集體制就是 Suncatcher 的體制：衛星相距 100–200 m，處於有界的相對幾何中，讓每個鄰居都保持可見。 |
| 結果與適用範圍 | 定性範圍 | 每跳光行時間低於一微秒。 |
| 容量與速率 | 10, 24, 400, 100, 9.6, 800 | 鏈路目標是每條鏈路 10 Tbps 量級，由 24 × 400G 的 DWDM 通道組成（在 100 GHz 柵格上為 9.6 Tbps），並已用現成元件搭建的實驗台規模展示器展示單向 800 Gbps[36, pp. 3, 8]。 |
| 容量與速率 | 6.4, 10.4, 10.1 | 它是唯一一個鏈路速率與 TPU v6e 的 6.4 Tbps ICI 落在同一個數量級的體制，因此也是唯一一個資料平行或流水線平行訓練在算術上說得通的體制，這就是為什麼 van Berkel 與 Pénot、Balakrishnan 的網路架構分析、第 10.4 節以及面板 10.1 全都落在這裡。 |
| 結果與適用範圍 | 10–30, 3 | 即使在這個體制中，層級邊界仍然存在，因為一顆在少數幾個孔徑後面裝了數十顆晶片的衛星，相對於它自己的 ICI 大約超額訂閱 10–30 倍，所以平行化必須像 Llama 3 在地面上的安排一樣：張量平行留在太空船內部，較輕的流水線與資料平行集體通訊才跨出去。 |
| 容量與速率 | 6.4, 1, 2, 2026, 10 | 單顆衛星體制把每一位元組的集體通訊流量都留在 ICI 上，Trillium 晶片在此有 6.4 Tbps，光行時間可以忽略，而且它是唯一有飛行實績的體制：Φ-Sat-1 的 Myriad 2、Starcloud-1 的 H100，以及 2026 年 10 月 1 日發射、搭載四顆 TPU 的 Suncatcher MVP，都在機上執行推論（第 8 章、第 11.1 節）press。 |
| 結果與適用範圍 | 定性範圍 | van Berkel 的 roofline 分析證實，侷限在單顆衛星內的推論與同一顆晶片在地面上的上限相同[101, §IV]preprint，因此這個體制依賴的是機上互連。 |
Original figure note原始圖說出處
Drawn for this primer. Rates: AsyncFLEO (IEEE BigData 2022), OrbitalBrain (NINeS 2026), CollabLLM-LEO and SpaceMoE (preprints), TinyLEO (SIGCOMM 2025), Starlink (company claim), Suncatcher (Joule 2026, p. 8), TPU v6e (Google docs), Llama 3 (arXiv 2407.21783). Hop latencies computed at 3.336 µs/km in the calc table of this section. Llama 3 as reported.Where geometry is stated, Figure 10.3 derives hop latency from that geometry, as the calculation table shows. OrbitalBrain, CollabLLM-LEO and TinyLEO specify rates while reproducible hop geometries remain open. Their hollow markers therefore occupy the constellation band at an unspecified distance. The best constellation rate, 200 Gbps, is about fifty times below the formation target. Its hops are about ten thousand times longer. Both differences follow from roughly 1,000 km separation through Section 2.2’s link budget, leaving terminal-only improvements outside the formation regime.
圖 10.3 中的每跳延遲，凡是論文有給出幾何的，都依幾何計算（見下方計算表）。空心標記則誠實記錄 OrbitalBrain、CollabLLM-LEO 與 TinyLEO 只固定了速率、可重現的跳距幾何待補充，所以把它們放在全星座區帶內，呈現其所屬體制的範圍。速率與延遲兩個軸說明，衛星間距同時決定了全星座與編隊之間的兩種差距。最佳的全星座數字 200 Gbps 比編隊目標低約五十倍，而它的每跳距離長約一萬倍，兩個因素都透過第 2.2 節的鏈路預算，源自同樣那上千公里的間距，因此僅改良終端時，全星座網路仍處於編隊體制之外。
| Question問題 | Calculation算式 | Result結果 |
|---|---|---|
| AsyncFLEO intra-plane hop (8 satellites per plane at 2,000 km)AsyncFLEO 同軌道面一跳（2,000 km 高度每面 8 顆） | \(2(6371+2000)\sin(\pi/8)\ \text{km}\times 3.336\ \mu\text{s/km}\) | 6,407 km · 21.4 mscomputed. 40 satellites in 5 planes from arXiv 2212.11522計算值。5 個軌道面共 40 顆衛星取自 arXiv 2212.11522 |
| SpaceMoE intra-plane hop (32 per plane at 550 km)SpaceMoE 同軌道面一跳（550 km 高度每面 32 顆） | \(2(6921)\sin(\pi/32)\ \text{km}\times 3.336\ \mu\text{s/km}\) | 1,357 km · 4.5 mscomputed. 33 × 32 at 550 km from arXiv 2605.00515計算值。550 km 的 33 × 32 取自 arXiv 2605.00515 |
| Rate gap, best constellation link to formation target速率差距：最佳全星座鏈路對編隊目標 | \(9.6\ \text{Tb/s}\ /\ 200\ \text{Gb/s}\) | 48×computed. TinyLEO §6.1 and Suncatcher (Joule p. 8)計算值。TinyLEO §6.1 與 Suncatcher（Joule p. 8） |
| Oversubscription of one 9.6 Tb/s aperture against on-board ICI for 16–48 chips一個 9.6 Tb/s 孔徑相對於 16–48 顆晶片機上 ICI 的超額訂閱 | \((16\text{–}48)\times 6.4\ \text{Tb/s}\ /\ 9.6\ \text{Tb/s}\) | ≈ 11–32×computed. "dozens of chips" read as 16–48, an assumption of this table. ICI from TPU v6e docs計算值。「數十顆晶片」解讀為 16–48，為本表的假設。ICI 取自 TPU v6e 文件 |
The three regimes define the scope of van Berkel’s verdict. He places two-to-three-year orbital frontier training outside credible scenarios and judges a ten-year horizon optimistic [101, §V] preprint. A torus sacrifices two to three bisection orders against terrestrial Clos (Section 10.4). Orbital Clos instead sacrifices compute satellites, while reliability assumptions exceed terrestrial references (Section 10.5). Flown hardware uses local interconnects only.
The question for Chapter 11 is a formation fabric meeting Figure 10.3’s rate requirements. It must recover torus capacity, place tensor-parallel boundaries across apertures and preserve its compute advantage after corruption recovery and redundancy costs.
對照這三種體制來讀，van Berkel 的結論比它的標題所暗示的範圍更窄，也因此更站得住腳。他的結論是，兩到三年內的太空前沿訓練超出其可信情境，十年的時程則仍偏樂觀[101, §V]preprint。而理由與本章各節一一對應：唯一有足夠鏈路速率的體制，若接成環面，對切頻寬會比地面 Clos 少兩到三個數量級（第 10.4 節），若接成 Clos，則要用運算衛星來付代價，而且還繼承了高於地面參考的失效假設（第 10.5 節）。至於真的飛上去過的體制，僅使用本地互連。因此本章交給第 11 章的開放問題，聚焦於編隊網路如何補回環面的容量損失、正確安排跨孔徑的張量平行邊界，以及在計入冗餘成本後保有運算優勢，並撐過無聲損毀。
| Regime體制 | Link rate and hop latency鏈路速率與每跳延遲 | Graph behavior圖的行為 | Workloads that fit合適的工作負載 | Status現況 |
|---|---|---|---|---|
| Constellation-wide全星座 | 16 Mb/s–200 Gb/s. 1.2–21 ms per hop16 Mb/s–200 Gb/s。每跳 1.2–21 ms | 3–4 terminals, crossings, 2–30 s re-acquisition3–4 個終端、軌道面交會、2–30 s 重新擷取 | federated learning, CDN, relaying single-satellite inference, compute-bound MoE inference聯邦學習、CDN、轉送單顆衛星推論結果、受運算限制的 MoE 推論 | links flown. AI workloads simulated鏈路已飛行。AI 工作負載為模擬 |
| Formation cluster編隊叢集 | 0.8 Tb/s/dir bench, 9.6–12.8 Tb/s target. 0.33–0.67 µs per hop實驗台每方向 0.8 Tb/s、目標 9.6–12.8 Tb/s。每跳 0.33–0.67 µs | bounded 100–200 m geometry, 8 neighbors, cycles twice per orbit有界的 100–200 m 幾何、8 個鄰居、每軌道循環兩次 | pipeline and data-parallel training, if the bisection and SDC problems close流水線與資料平行訓練，前提是對切與 SDC 問題能解決 | proposal. ISL flight demonstration pending提案，ISL 在軌示範待完成 |
| Single satellite單顆衛星 | ICI 6.4 Tb/s per TPU v6e chip. Light-time negligible每顆 TPU v6e 晶片 ICI 6.4 Tb/s。光行時間可忽略 | static, on board靜態、機上 | inference. Tensor parallelism within a spacecraft推論。太空船內的張量平行 | flown (Φ-Sat-1, Starcloud-1, Suncatcher MVP)已飛行（Φ-Sat-1、Starcloud-1、Suncatcher MVP） |
Chapter summary
本章摘要
Ring all-reduce stresses each worker’s injection rate, while all-to-all traffic stresses bisection. Orbital designs must account for terminal degree and the compute capacity assigned to switching.
環狀全歸約主要受工作者注入速率限制，全對全流量則受對切頻寬限制。軌道設計需計入終端度數，以及專責交換所佔用的運算容量。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Capacity and rate | 2, 1 | Synchronous step time is \(T_{\text{compute}} + T_{\text{exposed}}\). Data-parallel all-reduce dominates its network term, moving \(2(N-1)/N\) times model bytes per worker in a ring. Injection rate governs ring patterns, while bisection governs all-to-all traffic. |
| Latency comparison | 400, 3, 6.4, 0.057%, 1–2, 38–55% | Terrestrial references provide 400 Gbps per GPU in Llama 3 and HPN, plus 6.4 Tbps ICI per TPU v6e. Multi-hop latency is “tens of microseconds”. Link failures are 0.057% monthly, with 1–2 monthly job crashes and 38–55% MFU. |
| Orbital configuration | Qualitative scope | Mapped onto orbit, the companion primer's parallelism and step-budget chapters survive, collectives, routing and failures must be adapted |
| Capacity and rate | 64, 1, 12,880, 2,880, 8, 18, 89 | and topology, transport and front end must be replaced. This occurs because satellite degree is a handful of laser terminals rather than a 64-port switch. Van Berkel's 1 : 12,880 : 2,880 bisection ratio decomposes into link rate (8×), link count (18×) and the \(\sqrt{N}\) intrinsic to a torus (89×) |
| Compute and switching trade-off | 4, 6 | and a Clos avoids the last factor only by turning satellites into switches at a compute fraction \(r = k/(k+4L-6)\). |
| 結果與適用範圍 | 2, 1 | 一個同步訓練步的成本是 \(T_{\text{compute}} + T_{\text{exposed}}\)，其中網路項由資料平行的全歸約主導，環狀演算法以讓每個工作者送出模型位元組 \(2(N-1)/N\) 倍的方式完成它，所以對環狀模式而言限制資源是每個工作者的注入速率，對全對全模式則是對切頻寬。 |
| 可靠度比較 | 400, 3, 6.4, 0.057%, 1–2, 38–55% | 地面參考給每顆 GPU 400 Gbps（Llama 3、HPN）、每顆 TPU v6e 晶片 6.4 Tbps 的 ICI、「數十微秒」的多跳延遲，以及每月 0.057% 鏈路失效、每月 1–2 次工作當機的失效率，同時達到 38–55% 的 MFU。 |
| 結果與適用範圍 | 64 | 對應到軌道上，姊妹教材的平行化與步時間預算章節可以保留，集體通訊、路由與故障章節必須調整，拓樸、傳輸層與前端則必須替換，因為衛星的度數只有少數幾個雷射終端，地面交換器則有 64 埠。 |
| 結果與適用範圍 | 1, 12,880, 2,880, 8, 18, 89, 4, 6 | van Berkel 的 1 : 12,880 : 2,880 對切比值可以分解成鏈路速率（8 倍）、鏈路數（18 倍）以及環面本質上的 \(\sqrt{N}\)（89 倍），而 Clos 只能靠把衛星變成交換器來避開最後一個因素，代價是運算比例 \(r = k/(k+4L-6)\)。 |
Formation propagation is favorable, but forwarding time and failure recovery require measurements. The published regimes separate constellation services, formation training proposals and flown single-satellite inference.
編隊具備有利的傳播條件，轉送時間與故障恢復則待量測。現有研究可分為星座服務、編隊訓練提案與已飛行的單衛星推論。
| Item項目 | Values數值 | Conditions and source條件與來源 |
|---|---|---|
| Latency comparison | 2 km, 6.7 µs | Light crosses a 2 km formation in 6.7 µs. The favorable propagation budget leaves satellite forwarding time as the main unmeasured latency component. |
| Reliability comparison | 5%, 88, 2–30 s, 8,000 | SpaceMoE’s 5% snapshot loss is 88 times HPN’s monthly rate before reconciling units. Reacquisition takes 2–30 s. Trillium radiation data imply about nine silent corruptions per chip-year, or one every seven to eight minutes across 8,000 chips. |
| Capacity and rate | Qualitative scope | Published work separates into constellation-wide Gbps and millisecond links, formation Tbps and sub-microsecond hops, and single-satellite ICI. Only single-satellite inference has flown, while formations alone could support training. Van Berkel therefore places two-to-three-year orbital frontier training outside credible scenarios. |
| 軌道配置 | 2 km, 6.7 µs | 編隊具備有利的傳播延遲，因為光穿越 2 km 的編隊只要 6.7 µs，真正的延遲預算是每顆衛星內部仍待量測的每跳轉送時間。 |
| 結果與適用範圍 | 5%, 88, 2–30 s, 8,000 | 可靠度則是它輸最多之處，因為 SpaceMoE 每快照 5% 的鏈路損失，在按原始單位比較時就已是 HPN 每月失效率的 88 倍，重新擷取要 2–30 s，而 Trillium 的輻射數據意味著每顆晶片每年約九次無聲損毀，在 8,000 顆晶片上每七到八分鐘就有一次。 |
| 容量與速率 | 定性範圍 | 已發表的研究落在三種體制：Gbps 與毫秒級的全星座、Tbps 與次微秒每跳的編隊，以及靠 ICI 的單顆衛星。 |
| 結果與適用範圍 | 定性範圍 | 只有最後一種飛上去過，而只有第二種可能承載訓練，這就是 van Berkel 將兩到三年內的在軌前沿訓練列於可信情境之外的原因。 |
Where these ideas go next
這些概念接下來用在哪裡
| Parameter參數 | Value數值 | Conditions and interpretation條件與解讀 |
|---|---|---|
| Formation count | 81 | Suncatcher’s illustrative bounded formation. |
| 編隊衛星數 | 81 | Suncatcher 的有界編隊示意設計。 |
| Neighbor spacing | 100–200 m | Bounded geometry under HCW dynamics and J2. |
| 鄰居間距 | 100–200 m | HCW 動力學與 J2 下的有界幾何。 |
| Aperture diameter | 10 cm | Near-field optical-link example. |
| 孔徑直徑 | 10 cm | 近場光學鏈路範例。 |
| Link rate | ≈10 Tbps | Per-link near-field design target. |
| 鏈路速率 | ≈10 Tbps | 每條近場鏈路的設計目標。 |
Chapter 11 examines bounded formation geometry under Hill–Clohessy–Wiltshire dynamics and J2, plus near-field optics using the parameters above. It separates published specifications from open choices and revisits Section 10.4’s torus-versus-Clos comparison under range-dependent rates and seconds-long repointing. Chapter 12 returns to inference, while Chapter 13 develops the Section 10.5 reliability gap into research questions.
第 11 章以表列參數拆解編隊體制，說明 HCW 動力學與 J2 下的有界幾何、近場光學，以及 Suncatcher 已指定與待定的設計。當鏈路速率隨距離變化、重新指向需數秒時，第 10.4 節的環面與 Clos 之爭也需重新評估。推論部分接續於第 12 章，第 10.5 節的可靠度差距則在第 13 章發展為研究問題。
Papers behind this chapter本章所依據的論文
arXiv 2607.14172The Cost and Network Limits of Space-Based AI Compute
arXiv 2605.15335Designing Dense Satellite Clusters for Distributed Space-based Datacenters
Joule 2026 / arXiv 2511.19468Toward a Future Space-Based, Highly Scalable AI Infrastructure System Design (Project Suncatcher)
The companion AI Training Network Primer covers Llama 3, Alibaba HPN and ByteDance MegaScale in its topology, step-budget and failures chapters.
地面參考（Llama 3、阿里巴巴 HPN、字節跳動 MegaScale）在姊妹教材 AI Training Network Primer 的拓樸、步時間預算與故障章節中有詳細介紹。