A frontier model is far too big to train on one chip. The work is split across thousands of GPUs, and after each step they must swap results before anyone can continue. If the links between them are slow, expensive chips sit idle waiting for data. So the network is not an accessory to the computers; it decides how many chips can usefully work together.
Engineers talk about two layers. The “scale-up” network joins a small group of accelerators, a server or a whole rack, so tightly that they behave almost like one giant chip; NVIDIA’s NVLink is the best-known example. The “scale-out” network joins those racks into a cluster across a building, traditionally with InfiniBand and increasingly with Ethernet tuned for AI, which the Ultra Ethernet Consortium is standardizing. Optical links are pushing some clusters across several buildings.
For buyers, the interconnect shows up as hidden cost and lock-in. Two clusters with the same chips can train at very different speeds depending on networking, and proprietary links tie a customer to one vendor’s systems. Inference benefits too: serving a large mixture-of-experts model across many chips depends on fast links between them.
In episode 44 at 25:38, Cisco’s Jeetu Patel describes networking that lets two data centers “hundreds of kilometers apart operate as one giant cluster,” which he calls “scale across.” Cisco sells that networking, so read it as his company’s strategy. During the episode, Conor notes that the show’s sponsor at the time, Galileo, is an NVIDIA AI factory partner (31:57). In episode 28 at 3:07, AMD’s Anush Elangovan lists the interconnects among the hardware layers where chips have improved. Episode 28 is a Season 2 episode sponsored by Galileo (intro at 0:05).
See also AI factory and high-bandwidth memory, the other main limit on moving data to AI chips.
Hear it from the guest
“Operate as one giant cluster, one coherent cluster. And when you do that, that's what's called scale across.”
“Even in the hardware, at the microarchitecture level, at how they're brought together in the interconnects”
Quotes lightly edited to remove filler words.
Sources
- NVIDIA: NVLink and NVLink Switch — Vendor page describing NVLink as the GPU-to-GPU interconnect for rack-scale systems; the sixth generation is listed at 3 TB/s of bandwidth per GPU on the Vera Rubin platform; checked October 24, 2026.
- InfiniBand Trade Association: About InfiniBand — Describes InfiniBand as an industry-standard, channel-based, switched-fabric interconnect architecture for server and storage connectivity; checked October 24, 2026.
- Ultra Ethernet Consortium — Industry consortium developing an Ethernet-based networking specification for AI and high-performance computing workloads; checked October 24, 2026.
