Existential Risk

Most AI risks are bad but recoverable: a biased loan model, a data leak, a costly outage. Existential risk is the narrower category where there is no recovery. Philosopher Nick Bostrom’s 2002 paper defines it as an outcome that would wipe out intelligent life from Earth or permanently and drastically curtail its potential. Worries about AI in this category usually involve a future system far more capable than people, such as superintelligence, pursuing goals we failed to specify well. That failure is the problem AI alignment research tries to solve.

p(doom) turns the worry into a number. Someone who says “my p(doom) is 10%” means they personally put a one-in-ten chance on a catastrophic outcome. Numbers vary widely and rest on judgment, not data. A 2023 survey of thousands of published AI researchers found that between 38% and 51% of respondents gave at least a 10% chance to advanced AI leading to outcomes as bad as human extinction. The same year, researchers and public figures signed the Center for AI Safety’s one-line statement that reducing extinction risk from AI should be a global priority alongside pandemics and nuclear war. Those are sourced facts about what experts say, not evidence of what will happen.

On the show, the term comes up as a guest’s personal estimate. In episode 45, at 25:46, writer Andy Masley describes the idea of p(doom) around AI and says that, “gun to my head,” his own is probably pretty low, while noting that he is agnostic about many specific effective-altruism arguments on the topic. Earlier, at 19:20, he separates doom scenarios from other concerns, such as AI and the surveillance state. That is his view. The episode was sponsored by Galileo.

Editorially, existential risk is a long-range policy and research question; it shouldn’t crowd out the near-term risks teams manage every day. Related entries: AI safety, technological singularity and effective accelerationism, the movement that most loudly rejects slowing AI down over these fears.