Why is P(Doom) such a bad argument?

Imagine being told there is a precise 12.7% chance you will get into your dream college, or a 38.2% chance of rain on a day with no clouds. In the world of Artificial Intelligence, scientists and philosophers frequently debate **P(Doom)**—the probability that advanced AI will eventually wipe out humanity. But here is the catch: this seemingly scientific number is actually a mathematical illusion. ## The Illusion of Precision In statistics, a true probability requires a track record. We know the chance of a coin flip landing on heads is 50% because we have flipped coins millions of times. This is called *frequentist probability*. With AI doom, we are dealing with a one-time, future event that has never happened before. Therefore, P(Doom) is not a calculation; it is a subjective guess disguised as math. As prominent AI researcher [Yann LeCun](https://en.wikipedia.org/wiki/Yann_LeCun), Chief AI Scientist at Meta, has pointed out, calculating P(Doom) before we even have Human-Level AI is like calculating the probability of a boiler explosion before the steam engine was even invented. ## The Psychological Trap of "Mathiness" By assigning a specific percentage to our fears, we fall into a trap that economists call "mathiness"—using the language of mathematics to make unproven ideas look like hard science. When people state a P(Doom) of 10% or 50%, they are not using a formula. They are expressing a gut feeling of anxiety. Renowned AI critic and researcher [Margaret Mitchell](https://en.wikipedia.org/wiki/Margaret_Mitchell_(scientist)) has argued that focusing on sci-fi scenarios of human extinction distracts us from the very real, measurable harms AI is causing right now, such as algorithmic bias and copyright theft. In his book [*The Precipice: Existential Risk and the Future of Humanity*](https://en.wikipedia.org/wiki/The_Precipice:_Existential_Risk_and_the_Future_of_Humanity), philosopher Toby Ord discusses how we estimate these risks: > "These subjective probabilities are not assertions of dogmatic certainty... They represent our best estimate of the strength of the evidence, given everything we know." However, when "everything we know" is based on speculation rather than data, the resulting percentage is more like creative writing than science. P(Doom) treats a complex, human-controlled future as if it were a roll of the dice, ignoring our own ability to regulate, program, and shape how AI develops. ## Keep Exploring To dive deeper into how we predict the future of technology, consider these questions: 1. If P(Doom) is a bad metric, how *should* we measure and prepare for the long-term risks of technology? 2. How does the psychological phenomenon of [automation bias](https://en.wikipedia.org/wiki/Automation_bias)—our tendency to trust automated systems—make us more likely to believe flawed mathematical predictions? 3. If we shift our focus away from extinction-level events, what are the most urgent AI ethical challenges we need to solve today?

Have you ever noticed how a single, precise number can instantly silence a room? If a doctor tells you there is a 73% chance a treatment will work, you probably won't ask to see their math. You trust the number. But this trust becomes dangerous when we export "mathiness" to other massive, unpredictable global challenges. The illusion of precision we see in AI's P(Doom) isn't unique to computer science. We make the same mistake whenever we try to calculate the "uncalculable" in climate change and global economics. ## The Climate "Tipping Point" Trap In climate science, researchers often try to calculate the exact probability of reaching specific "tipping points"—like the collapse of the West Antarctic Ice Sheet. While the greenhouse effect is proven science, assigning a precise percentage to a chaotic, planetary feedback loop is incredibly difficult. Climate scientist [Elisabeth Lloyd](https://history.indiana.edu/faculty_staff/faculty/lloyd-elisabeth.html) has written extensively about the limits of climate models. When we treat highly uncertain, subjective model runs as precise probabilities, we risk policy paralysis. If we tell the public there is a "reassurance" of a 65% chance we will avoid a climate catastrophe, we mistake a educated guess for a hard boundary. Just like with AI, this number-crunching can distract us from taking obvious, immediate actions—like reducing emissions today—because we are too busy arguing over whether the "probability" is 60% or 80%. ## The Financial "Black Swan" and the 2008 Crash Perhaps the most devastating real-world consequence of this mathematical illusion occurred on Wall Street. Before the 2008 global financial crisis, risk managers used a mathematical formula called the **Copula function** to predict the probability of mortgage defaults. The models calculated the chance of a total housing market collapse as practically zero. They treated the economy like a predictable casino game. In his groundbreaking book [*The Black Swan: The Impact of the Highly Improbable*](https://en.wikipedia.org/wiki/The_Black_Swan:_The_Impact_of_the_Highly_Improbable), scholar and statistician Nassim Nicholas Taleb explained why this approach is fundamentally flawed: > "We forecast risk paths using past data, assuming the future will look like the past... We build toy models that ignore wild uncertainty, mistaking the map for the territory." Taleb argues that in complex systems—like global finance or human history—the most impactful events are "Black Swans": outliers that are impossible to predict using past data. Applying a neat percentage to the likelihood of a market crash gives investors a false sense of security, making the eventual crash much worse. ## Why We Crave the Number Why do we keep doing this? Psychologists call it **uncertainty reduction**. Human beings hate ambiguity. A subjective guess of "50% chance of collapse" feels safer and more actionable than admitting, "We simply do not know, and we cannot calculate it." When we look at complex, open-ended systems—whether it is the future of advanced technology, the Earth's biosphere, or the global economy—we must learn to resist the comfort of a false percentage. True preparation means building resilience to face the unknown, rather than pretending we have a map of a territory we have never visited.

Imagine standing on the deck of the *Titanic*. Would you prefer a meteorologist who gives you a precise "37.2% probability" of striking an iceberg, or a look-out who simply shouts that you are sailing at full speed into a dark, unmapped field of ice? The true conflict between our two positions lies in a dangerous paradox of human psychology. Position A shows that we use "P(Doom)" as an **ego-driven narrative engine**—a way to turn abstract technological anxieties into a grand, apocalyptic story starring ourselves as either gods or martyrs. Position B reveals that we use precise percentages as a **cognitive sedative**—a mathematical security blanket designed to tame chaotic, non-linear systems like climate change and high-finance. The friction between them is profound: Are these catastrophic percentages an arrogant, hype-generating attempt to puff ourselves up, or are they a desperate, head-in-the-sand attempt to quiet our deepest fears? ``` [ Complex, Unmapped Systems ] / \ / \ Position A: Ego & Narrative Position B: Fear & Control (AI "P(Doom)" as story) (Finance/Climate as math) \ / \ / [ False Certainty Mirage ] ``` ## The Unspoken Overlap: The Tyranny of "Mathiness" While these perspectives seem to diagnose different social ills, they secretly share a single, corrosive diagnosis: the trap of **mathiness**. Coined by Nobel laureate economist [Paul Romer](https://en.wikipedia.org/wiki/Paul_Romer), "mathiness" refers to using the appearance of mathematical rigor to disguise ideological or subjective claims. Both AI doomers and Wall Street analysts suffer from the exact same statistical illusion. They mistake a subjective guess for an objective measurement. Whether calculated in a Silicon Valley forum or a Manhattan board room, an imaginary probability creates a toxic distraction. It pulls our focus away from present, observable harms—such as carbon emissions today or immediate algorithmic bias—and traps us in abstract debates over hypothetical futures. Statistician [Nassim Nicholas Taleb](https://en.wikipedia.org/wiki/Nassim_Nicholas_Taleb) diagnosed this exact intellectual failure in his book [*Fooled by Randomness*](https://en.wikipedia.org/wiki/Fooled_by_Randomness): > "Probability is not a mere computation of odds on a dice table; it is the acceptance of the lack of certainty in our knowledge and the development of methods for dealing with our ignorance." When we treat probability as a hard prediction rather than a confession of ignorance, we fail Taleb's test completely. ## A Unified Framework: Pragmatic Resilience To move beyond both false hype and false security, we must fuse these insights into a new paradigm: **Pragmatic Resilience**. Instead of trying to calculate the *probability* of an unprecedented disaster—which is mathematically impossible in complex systems—we must measure our **vulnerability** and build systems that can withstand shock. ``` +-----------------------------------------------------------------+ | PRAGMATIC RESILIENCE | +-----------------------------------------------------------------+ | 1. Abandon Map-Making: Stop calculating odds for black swans. | | 2. Audit Vulnerabilities: Identify fragile single points. | | 3. Build Deep Redundancy: Target immediate, observable harms. | +-----------------------------------------------------------------+ ``` This framework replaces the illusion of precise foresight with three actionable principles: 1. **Acknowledge the Blindspot:** Accept that for unique, hyper-complex events—whether an artificial general intelligence, a novel climate tipping point, or a global financial instrument—the true probability is fundamentally uncalculable. 2. **Shift from Risk to Fragility:** Stop asking *"What are the odds of doom?"* Start asking *"How fragile is our system if things go wrong?"* A bank that requires an accurate probability model to survive is inherently fragile. An AI system whose safety relies on predicting its future intelligence is inherently dangerous. 3. **Address Present Vulnerabilities:** Pivot our resources toward tangible, immediate safeguards. In climate policy, this means building resilient infrastructure today. In AI safety, as computer scientist [Timnit Gebru](https://en.wikipedia.org/wiki/Timnit_Gebru) argues, it means addressing the immediate, measurable impacts of data exploitation and algorithmic bias right now, rather than obsessing over sci-fi apocalypses. By stripping away the false comfort of "mathiness" and the ego of apocalyptic storytelling, Pragmatic Resilience forces us to stop predicting the unpredictable, and start preparing for it.