Data Wall

Language models have improved partly by training on more text: books, articles, forums, code. The public web is large but finite, and the amount of text used to train frontier models has grown fast. Epoch AI researchers estimate the usable stock of public human-written text at around 300 trillion tokens and project that, if trends continue, training runs will use roughly all of it sometime between 2026 and 2032. That limit is what people mean by the data wall.

It is a forecast, not a fixed date, and it depends on assumptions about data quality, repetition and how much text sits behind logins or in private archives. It also does not mean progress stops. Labs are responding by generating synthetic data, licensing private data, training on other kinds of data such as video, reusing data for more passes, and moving effort into post-training, especially reinforcement learning with verifiable rewards, where the model learns from checked attempts instead of new human text. Each route has trade-offs; leaning too hard on model-generated text risks model collapse. The debate is closely tied to scaling laws, which describe how much data and compute bigger models need.

In episode 44 at 13:54, Cisco’s Jeetu Patel calls this a “data gap”: people expected models to run out of data, “which was actually true,” but he says evidence is building that synthetic data works well. Conor mentions Galileo as the show’s sponsor later in the episode.

In episode 54 at 45:48, Dan Klein says that once a model has consumed the declarative knowledge on the web, “the golden stuff is kind of used up.” His view is that synthetic data helps where outputs can be checked, as in Go, and is too noisy where a model just produces text and hopes it is right. Galileo sponsored the episode.