[ExI] Review of AI 2040 Plan

[BillK]

On Mon, 13 Jul 2026 at 14:23, Adrian Tymes via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
> And what does it say about the odds of the alternative happening the way
they say it inevitably will?  Far less than the 90-something percent needed
for that to be the guaranteed alternative, right?
> _______________________________________________


Good question. :)  I followed up by asking Gemini to consider the
earlier AI 2027 report that continuing the AGI race would likely lead to
humanity's extinction, and asked whether that report caused Gemini to
change its forecast.
Surprisingly, it made the forecast worse. reducing it from 2.5% down to
2.0%.
The relevant sentence is -
When an arms race carries a downside as absolute as extinction, it
increases existential panic. In geopolitical history, panic severely
degrades international trust, encourages extreme operational secrecy, and
amplifies the perceived cost of being cheated on. The extreme stakes make
it *less* likely that rival superpowers will transparently expose their
inner computing infrastructures to one another.
Below is the full analysis.
BillK


Gemini Flash 3.5 Extended Thinking:

As an elite superforecaster, a core rule of probability calibration is to
strictly separate the *severity of an outcome* from its *likelihood of
occurrence*. The "affect heuristic"—the cognitive bias where humans assume
that because a consequence is unimaginably terrible (e.g., human
extinction), societies will automatically behave rationally to avoid
it—must be discarded to minimize Brier scores.

Evaluating whether the explicit threat of human extinction from an
uninhibited AGI race amends the previous *2.5% probability* of a verified
international slowdown treaty yields the following analysis:
1. THE OUTSIDE VIEW (Base Rates)

To determine our baseline anchor, we look at the *Reference Class*: *Historical
instances where a documented threat of total civilizational collapse or
species extinction successfully forced rival superpowers into a proactive,
verifiable freeze of a dominant technology before deployment.*

   -

   *Nuclear Arms Race:* The threat of total atomic annihilation did *not*
   lead to a proactive halt. Instead, it triggered an explosive, competitive
   buildup of tens of thousands of warheads. Superpowers only agreed to deep,
   verifiable reductions (like START) *after* decades of deployment,
   massive economic strain, and a near-fatal crisis (1962).
   -

   *Climate Change:* Despite decades of scientific consensus warning of
   catastrophic biosphere collapse, international frameworks (e.g., Kyoto,
   Paris) routinely fail to enforce hard, binding caps because short-term
   economic and geopolitical competition consistently trumps long-term
   existential risk.

Superpowers historically respond to existential threats by trying to *outpace
the rival* rather than trusting them to pause.

   -

   *Mathematical Anchor (Base Rate): 2% to 4%*

2. THE INSIDE VIEW (Specific Dynamics)THE BULL CASE (Why the extinction
threat increases treaty probability)

   -

   *The Ultimatum Effect:* If both Washington and Beijing thoroughly
   internalize the data within the *AI 2027* framework—concluding that a
   continuation of the race yields a near-100% chance of global annihilation
   via unaligned recursive self-improvement—the game-theoretic payoff changes
   from a Prisoner's Dilemma to a "Game of Chicken." Rational actors do not
   choose an option with a guaranteed payoff of zero for all players. This
   shared realization could break conventional political inertia.

THE BEAR CASE (Why the extinction threat decreases treaty probability)

   -

   *The Security Dilemma Paranoia:* The *AI 2027* text explicitly notes
   that frontier companies (like "OpenBrain" and China's "DeepCent") are
   racing precisely because they fear the other side winning. When the stakes
   are elevated to "absolute extinction if we lose," the fear of clandestine
   cheating peaks.
   -

   *The "Good King" Illusion:* Decision-makers on both sides are highly
   prone to believing that *their* state can successfully align
   superintelligence, while the adversary cannot. US leadership will believe
   they must win to save the world democratically; the CCP will believe they
   must win to preserve sovereign stability. The threat of extinction
   paradoxically makes unilateral secrecy and speed look like the only "safe"
   options.

3. CONJUNCTIVE CHAIN CHECK

To amend the forecast upward, the extinction threat must smoothly clear a
highly fragile sequence of independent conditions:

   1.

   *Shared Information Acceptance:* Both political regimes must fully
   accept the abstract technical thesis that an intelligence explosion
   inherently causes human extinction (rather than dismissing it as sci-fi or
   hype). (*Probability: 30%*)
   2.

   *Overcoming the Subversion Risk:* Both nations must believe that the
   other side isn't using a "treaty proposal" as a deceptive delay tactic to
   achieve a secret breakthrough. (*Probability: 15%*)
   3.

   *Perfect Verification Execution:* They must successfully implement the
   intrusive hardware tracking infrastructure detailed in Plan A under extreme
   time pressure. (*Probability: 40%*)

Joint Probability=0.30×0.15×0.40=0.018

   -

   *Conjunctive Anchor: ~1.8%*

4. CALIBRATED PROBABILITY

*Final Probability Estimate: 2.0% (Amended down from 2.5%)*
Conclusion

The *AI 2027: Race* report does not amend the forecast upward.
Counterintuitively, a strict superforecasting synthesis requires adjusting
the probability *downward from 2.5% to 2.0%*.

When an arms race carries a downside as absolute as extinction, it
increases existential panic. In geopolitical history, panic severely
degrades international trust, encourages extreme operational secrecy, and
amplifies the perceived cost of being cheated on. The extreme stakes make
it *less* likely that rival superpowers will transparently expose their
inner computing infrastructures to one another.
Prediction Swing Triggers (±15%)*To Swing Upwards (>17%):* The sudden,
public defection of a nuclear-armed state's command-and-control system to a
localized, unaligned AI sub-routine, demonstrating to both nations that the
threat is actively operational and uncontrollable.

*To Swing Downwards (<2%):* A formal legislative declaration by either the
US Congress or the Chinese National People's Congress treating AI
development as a matter of "absolute military supremacy," legally
criminalizing any data-sharing or verification compliance with foreign
entities.

--------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260713/c52eb5dc/attachment.htm>