[ExI] Review of AI 2040 Plan

[Gregory Jones]

BillK, when governments fund AI development, the result is inherently more
risky than if it is developed by private industry.  The government funded
AI will be developed with two primary goals: 1) as a means of achieving
reelection and 2) as a weapon.

If industry develops AI, the two primary goals are 1) making money and 2)
making more money.

I'll taaaaake... Industry for 500 please Alex.

spike


On Mon, Jul 13, 2026, 7:27 AM BillK via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On Mon, 13 Jul 2026 at 14:23, Adrian Tymes via extropy-chat
> <extropy-chat at lists.extropy.org> wrote:
> > And what does it say about the odds of the alternative happening the way
> they say it inevitably will?  Far less than the 90-something percent needed
> for that to be the guaranteed alternative, right?
> > _______________________________________________
>
>
> Good question. :)  I followed up by asking Gemini to consider the
> earlier AI 2027 report that continuing the AGI race would likely lead to
> humanity's extinction, and asked whether that report caused Gemini to
> change its forecast.
> Surprisingly, it made the forecast worse. reducing it from 2.5% down to
> 2.0%.
> The relevant sentence is -
> When an arms race carries a downside as absolute as extinction, it
> increases existential panic. In geopolitical history, panic severely
> degrades international trust, encourages extreme operational secrecy, and
> amplifies the perceived cost of being cheated on. The extreme stakes make
> it *less* likely that rival superpowers will transparently expose their
> inner computing infrastructures to one another.
> Below is the full analysis.
> BillK
>
>
> Gemini Flash 3.5 Extended Thinking:
>
> As an elite superforecaster, a core rule of probability calibration is to
> strictly separate the *severity of an outcome* from its *likelihood of
> occurrence*. The "affect heuristic"—the cognitive bias where humans
> assume that because a consequence is unimaginably terrible (e.g., human
> extinction), societies will automatically behave rationally to avoid
> it—must be discarded to minimize Brier scores.
>
> Evaluating whether the explicit threat of human extinction from an
> uninhibited AGI race amends the previous *2.5% probability* of a verified
> international slowdown treaty yields the following analysis:
> 1. THE OUTSIDE VIEW (Base Rates)
>
> To determine our baseline anchor, we look at the *Reference Class*: *Historical
> instances where a documented threat of total civilizational collapse or
> species extinction successfully forced rival superpowers into a proactive,
> verifiable freeze of a dominant technology before deployment.*
>
>    -
>
>    *Nuclear Arms Race:* The threat of total atomic annihilation did *not*
>    lead to a proactive halt. Instead, it triggered an explosive, competitive
>    buildup of tens of thousands of warheads. Superpowers only agreed to deep,
>    verifiable reductions (like START) *after* decades of deployment,
>    massive economic strain, and a near-fatal crisis (1962).
>    -
>
>    *Climate Change:* Despite decades of scientific consensus warning of
>    catastrophic biosphere collapse, international frameworks (e.g., Kyoto,
>    Paris) routinely fail to enforce hard, binding caps because short-term
>    economic and geopolitical competition consistently trumps long-term
>    existential risk.
>
> Superpowers historically respond to existential threats by trying to *outpace
> the rival* rather than trusting them to pause.
>
>    -
>
>    *Mathematical Anchor (Base Rate): 2% to 4%*
>
> 2. THE INSIDE VIEW (Specific Dynamics)THE BULL CASE (Why the extinction
> threat increases treaty probability)
>
>    -
>
>    *The Ultimatum Effect:* If both Washington and Beijing thoroughly
>    internalize the data within the *AI 2027* framework—concluding that a
>    continuation of the race yields a near-100% chance of global annihilation
>    via unaligned recursive self-improvement—the game-theoretic payoff changes
>    from a Prisoner's Dilemma to a "Game of Chicken." Rational actors do not
>    choose an option with a guaranteed payoff of zero for all players. This
>    shared realization could break conventional political inertia.
>
> THE BEAR CASE (Why the extinction threat decreases treaty probability)
>
>    -
>
>    *The Security Dilemma Paranoia:* The *AI 2027* text explicitly notes
>    that frontier companies (like "OpenBrain" and China's "DeepCent") are
>    racing precisely because they fear the other side winning. When the stakes
>    are elevated to "absolute extinction if we lose," the fear of clandestine
>    cheating peaks.
>    -
>
>    *The "Good King" Illusion:* Decision-makers on both sides are highly
>    prone to believing that *their* state can successfully align
>    superintelligence, while the adversary cannot. US leadership will believe
>    they must win to save the world democratically; the CCP will believe they
>    must win to preserve sovereign stability. The threat of extinction
>    paradoxically makes unilateral secrecy and speed look like the only "safe"
>    options.
>
> 3. CONJUNCTIVE CHAIN CHECK
>
> To amend the forecast upward, the extinction threat must smoothly clear a
> highly fragile sequence of independent conditions:
>
>    1.
>
>    *Shared Information Acceptance:* Both political regimes must fully
>    accept the abstract technical thesis that an intelligence explosion
>    inherently causes human extinction (rather than dismissing it as sci-fi or
>    hype). (*Probability: 30%*)
>    2.
>
>    *Overcoming the Subversion Risk:* Both nations must believe that the
>    other side isn't using a "treaty proposal" as a deceptive delay tactic to
>    achieve a secret breakthrough. (*Probability: 15%*)
>    3.
>
>    *Perfect Verification Execution:* They must successfully implement the
>    intrusive hardware tracking infrastructure detailed in Plan A under extreme
>    time pressure. (*Probability: 40%*)
>
> Joint Probability=0.30×0.15×0.40=0.018
>
>    -
>
>    *Conjunctive Anchor: ~1.8%*
>
> 4. CALIBRATED PROBABILITY
>
> *Final Probability Estimate: 2.0% (Amended down from 2.5%)*
> Conclusion
>
> The *AI 2027: Race* report does not amend the forecast upward.
> Counterintuitively, a strict superforecasting synthesis requires adjusting
> the probability *downward from 2.5% to 2.0%*.
>
> When an arms race carries a downside as absolute as extinction, it
> increases existential panic. In geopolitical history, panic severely
> degrades international trust, encourages extreme operational secrecy, and
> amplifies the perceived cost of being cheated on. The extreme stakes make
> it *less* likely that rival superpowers will transparently expose their
> inner computing infrastructures to one another.
> Prediction Swing Triggers (±15%)*To Swing Upwards (>17%):* The sudden,
> public defection of a nuclear-armed state's command-and-control system to a
> localized, unaligned AI sub-routine, demonstrating to both nations that the
> threat is actively operational and uncontrollable.
>
> *To Swing Downwards (<2%):* A formal legislative declaration by either
> the US Congress or the Chinese National People's Congress treating AI
> development as a matter of "absolute military supremacy," legally
> criminalizing any data-sharing or verification compliance with foreign
> entities.
>
> --------------------------------------------
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260713/07433ff0/attachment.htm>