[ExI] Review of AI 2040 Plan

[Adrian Tymes]

That's the opposite case of what I was asking.

It says the chance of human extinction with a slowdown is 2%.

The claim the AI 2040 Plan guys appear to be making is that, if
there's no slowdown, then human extinction is near-guaranteed.

So what's the chance of human extinction without a slowdown?  And
what's the chance of some radically better outcome - something so good
as to compare to the Pascal's Wager "infinitely bad outcome" of human
extinction, and not merely be washed out or dismissed by that
"infinitely"?

On Mon, Jul 13, 2026 at 10:28 AM BillK via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
>
> On Mon, 13 Jul 2026 at 14:23, Adrian Tymes via extropy-chat
> <extropy-chat at lists.extropy.org> wrote:
> > And what does it say about the odds of the alternative happening the way they say it inevitably will?  Far less than the 90-something percent needed for that to be the guaranteed alternative, right?
> > _______________________________________________
>
>
> Good question. :)  I followed up by asking Gemini to consider the
> earlier AI 2027 report that continuing the AGI race would likely lead to
> humanity's extinction, and asked whether that report caused Gemini to
> change its forecast.
> Surprisingly, it made the forecast worse. reducing it from 2.5% down to 2.0%.
> The relevant sentence is -
> When an arms race carries a downside as absolute as extinction, it increases existential panic. In geopolitical history, panic severely degrades international trust, encourages extreme operational secrecy, and amplifies the perceived cost of being cheated on. The extreme stakes make it less likely that rival superpowers will transparently expose their inner computing infrastructures to one another.
> Below is the full analysis.
> BillK
>
>
> Gemini Flash 3.5 Extended Thinking:
>
> As an elite superforecaster, a core rule of probability calibration is to strictly separate the severity of an outcome from its likelihood of occurrence. The "affect heuristic"—the cognitive bias where humans assume that because a consequence is unimaginably terrible (e.g., human extinction), societies will automatically behave rationally to avoid it—must be discarded to minimize Brier scores.
>
> Evaluating whether the explicit threat of human extinction from an uninhibited AGI race amends the previous 2.5% probability of a verified international slowdown treaty yields the following analysis:
>
> 1. THE OUTSIDE VIEW (Base Rates)
>
> To determine our baseline anchor, we look at the Reference Class: Historical instances where a documented threat of total civilizational collapse or species extinction successfully forced rival superpowers into a proactive, verifiable freeze of a dominant technology before deployment.
>
> Nuclear Arms Race: The threat of total atomic annihilation did not lead to a proactive halt. Instead, it triggered an explosive, competitive buildup of tens of thousands of warheads. Superpowers only agreed to deep, verifiable reductions (like START) after decades of deployment, massive economic strain, and a near-fatal crisis (1962).
>
> Climate Change: Despite decades of scientific consensus warning of catastrophic biosphere collapse, international frameworks (e.g., Kyoto, Paris) routinely fail to enforce hard, binding caps because short-term economic and geopolitical competition consistently trumps long-term existential risk.
>
> Superpowers historically respond to existential threats by trying to outpace the rival rather than trusting them to pause.
>
> Mathematical Anchor (Base Rate): 2% to 4%
>
> 2. THE INSIDE VIEW (Specific Dynamics)
>
> THE BULL CASE (Why the extinction threat increases treaty probability)
>
> The Ultimatum Effect: If both Washington and Beijing thoroughly internalize the data within the AI 2027 framework—concluding that a continuation of the race yields a near-100% chance of global annihilation via unaligned recursive self-improvement—the game-theoretic payoff changes from a Prisoner's Dilemma to a "Game of Chicken." Rational actors do not choose an option with a guaranteed payoff of zero for all players. This shared realization could break conventional political inertia.
>
> THE BEAR CASE (Why the extinction threat decreases treaty probability)
>
> The Security Dilemma Paranoia: The AI 2027 text explicitly notes that frontier companies (like "OpenBrain" and China's "DeepCent") are racing precisely because they fear the other side winning. When the stakes are elevated to "absolute extinction if we lose," the fear of clandestine cheating peaks.
>
> The "Good King" Illusion: Decision-makers on both sides are highly prone to believing that their state can successfully align superintelligence, while the adversary cannot. US leadership will believe they must win to save the world democratically; the CCP will believe they must win to preserve sovereign stability. The threat of extinction paradoxically makes unilateral secrecy and speed look like the only "safe" options.
>
> 3. CONJUNCTIVE CHAIN CHECK
>
> To amend the forecast upward, the extinction threat must smoothly clear a highly fragile sequence of independent conditions:
>
> Shared Information Acceptance: Both political regimes must fully accept the abstract technical thesis that an intelligence explosion inherently causes human extinction (rather than dismissing it as sci-fi or hype). (Probability: 30%)
>
> Overcoming the Subversion Risk: Both nations must believe that the other side isn't using a "treaty proposal" as a deceptive delay tactic to achieve a secret breakthrough. (Probability: 15%)
>
> Perfect Verification Execution: They must successfully implement the intrusive hardware tracking infrastructure detailed in Plan A under extreme time pressure. (Probability: 40%)
>
> Joint Probability=0.30×0.15×0.40=0.018
>
> Conjunctive Anchor: ~1.8%
>
> 4. CALIBRATED PROBABILITY
>
> Final Probability Estimate: 2.0% (Amended down from 2.5%)
>
> Conclusion
>
> The AI 2027: Race report does not amend the forecast upward. Counterintuitively, a strict superforecasting synthesis requires adjusting the probability downward from 2.5% to 2.0%.
>
> When an arms race carries a downside as absolute as extinction, it increases existential panic. In geopolitical history, panic severely degrades international trust, encourages extreme operational secrecy, and amplifies the perceived cost of being cheated on. The extreme stakes make it less likely that rival superpowers will transparently expose their inner computing infrastructures to one another.
>
> Prediction Swing Triggers (±15%)
>
> To Swing Upwards (>17%): The sudden, public defection of a nuclear-armed state's command-and-control system to a localized, unaligned AI sub-routine, demonstrating to both nations that the threat is actively operational and uncontrollable.
>
> To Swing Downwards (<2%): A formal legislative declaration by either the US Congress or the Chinese National People's Congress treating AI development as a matter of "absolute military supremacy," legally criminalizing any data-sharing or verification compliance with foreign entities.
>
> --------------------------------------------
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat