CUDA

Most people who train or run models never write CUDA directly. They write Python in PyTorch or another framework, and the framework calls CUDA libraries underneath to split the math across thousands of GPU cores. A small group of specialists writes custom CUDA “kernels,” the low-level functions that do one operation as fast as the chip allows; FlashAttention began as one of those.

Years of libraries, tutorials and tuned code have built up around CUDA. That ecosystem is a large part of NVIDIA’s lead: a competing chip can be fast on paper and still lose if popular models and tools do not run well on it on day one. AMD’s alternative is ROCm, an open software stack; Google trains on TPUs through the XLA compiler instead. The switching cost in AI hardware is software covers that competition in depth.

In episode 72 at 15:13, AMD’s Anush Elangovan says a typical newcomer’s entry point is knowing “there has been something called CUDA and there’s something called Rokum” (the transcript’s spelling of ROCm); his goal is for users to deploy a model in natural language without knowing either. That is AMD’s aim, stated by an AMD executive. Episode 72 names Svix and Walrus Memory as presenting sponsors in its opening read. In episode 47 at 10:43, Google DeepMind’s Paige Bailey explains that early frameworks were built for CPUs and later “retrofitted” with separate code paths for CUDA and for XLA. Episode 47 was sponsored by Galileo (sponsor thanks at 49:12).

See also graphics processing unit.