[ExI] Review of AI 2040 Plan

[BillK]

Quote:
AI companies are racing to build AIs that are smarter than humans in every
way. In AI 2027, we predicted that this would result in either extinction
or irreversible concentration of power.1
Plan A is our positive vision for what should happen instead.

<https://ai-2040.com/>
-----------------------------------------

This optimistic plan relies on the USA and China agreeing on a treaty to
pause AI development before AGI is reached.
I asked Gemini AI to behave as a forecaster and discuss the feasibility of
this plan.
The result was a 2.5% probability of success.
The full report of why it almost certainly won't happen is below.
BillK

Gemini 3.5 Flash Extended Thinking:

To evaluate the "AI 2040: Plan A" proposal published by the AI Futures
Project, we must define the core target event implied by the text: *Will
the United States and China establish and successfully enforce a verified
bilateral or multilateral AI governance treaty to explicitly halt, pause,
or restrict the development of frontier artificial intelligence before it
reaches superintelligence (ASI) by 2035?*

Here is the hyper-calibrated superforecasting evaluation of this target
event.
1. THE OUTSIDE VIEW (Base Rates)

To establish a statistical baseline, we look at the *Reference Class*:
*Bilateral
or multilateral technology-limitation or arms-control treaties enacted
between major adversarial superpowers targeting a highly lucrative,
dual-use, rapidly advancing capability prior to a kinetic catastrophe or
widespread deployment.*

Over the past 30 years, the baseline for this type of proactive, intrusive
restriction is exceptionally low:

   -

   *The Successes:* The Strategic Arms Reduction Treaties (START) and the
   Chemical Weapons Convention (1993) successfully limited stockpiles, but
   these were signed *after* decades of deployment, visible destruction, or
   near-miss crises (e.g., the Cuban Missile Crisis), and during a period of
   geopolitical convergence.
   -

   *The Failures/Collapses:* More recent history shows a decay in
   verification frameworks. The US withdrew from the ABM Treaty and the INF
   Treaty. The Joint Comprehensive Plan of Action (JCPOA) collapsed due to
   domestic political shifting. The Comprehensive Nuclear-Test-Ban Treaty
   (1996) remains unratified by key nations, and the Paris Climate Accords
   rely on non-binding, voluntarily reported targets.

Proactively banning an entirely new paradigm of economic and military power
before its destructive capacity has been unambiguously demonstrated has
almost no historical precedent.

   -

   *Mathematical Anchor (Base Rate):* *3% to 5%*

2. THE INSIDE VIEW (Specific Dynamics)THE BULL CASE (Why the plan could
succeed)

   -

   *The Physical Chokepoint:* Unlike cyber-weapons or biological agents,
   frontier AI requires massive physical infrastructure. Roughly 98.5% of
   advanced AI chips flow through an incredibly consolidated supply chain
   (ASML for lithography, TSMC for fabrication, NVIDIA for design). This makes
   hardware-level verification ("trustless chip tracking" via cryptographic
   firmware locks) technically feasible without requiring intrusive software
   inspections.
   -

   *Mutual Threat of Sovereign Loss:* Unlike traditional weapons where one
   state wins if the other loses, a misaligned superintelligence poses an
   existential threat to the internal political stability of *both*
   regimes. The Chinese Communist Party values top-down information control
   above all else; an autonomous, unaligned ASI is fundamentally incompatible
   with their governance model.
   -

   *Elite Alignment:* Elite opinion is shifting rapidly. The fact that
   former frontier researchers (like the authors of Plan A) and tech
   executives are actively warning congress indicates that the "overton
   window" for aggressive regulation is opening faster than historical tech
   cycles.

THE BEAR CASE (Why the plan will fail)

   -

   *The Geopolitical Prisoner's Dilemma:* Washington and Beijing currently
   view AI through a zero-sum, national security lens. As noted in the Plan A
   text, domestic tech lobbyists aggressively push the narrative that any
   unilateral regulation forces the US to lose the digital Cold War to China.
   For China, AI represents an asymmetric opportunity to bypass US
   conventional military and economic superiority.
   -

   *The Intrusiveness Threshold:* A "verified" slowdown requires
   unprecedented transparency. Even if chip tracking is used, verifying
   compliance would ultimately require verifying power grids, datacenter
   locations, and supply chains. In an era of decoupling and extreme espionage
   suspicion, neither nation will comfortably allow foreign oversight of its
   most critical national security infrastructure.
   -

   *Trillion-Dollar Economic Momentum:* The capital expenditure currently
   flowing into AI datacenters is unprecedented. Halting capabilities progress
   at "top human expert level" requires governments to force corporations to
   walk away from trillions in projected productivity gains.

3. CONJUNCTIVE CHAIN CHECK

Plan A requires a strict sequence of high-stakes, independent conditions to
succeed simultaneously. If we break down the necessary chain:

   1.

   *US Alignment:* The US political establishment must pivot from a "race
   to win" mindset to an "existential risk" framework by 2029.
   -

      *Estimated Probability:* *35%*
      2.

   *Sino-American Accord:* China must agree to a bilateral deal that
   essentially freezes their relative position or subjects their internal
   computing infrastructure to verifiable tracking.
   -

      *Estimated Probability:* *15%*
      3.

   *Technical Deployment:* Tamper-proof, hardware-level cryptographic
   tracking and verification mechanisms must be successfully developed and
   deployed globally without backdoor bypasses.
   -

      *Estimated Probability:* *40%*
      4.

   *Leakage Prevention:* Third-party actors, rogue states, or black-market
   hardware recycling clusters must be successfully contained to prevent a
   compliance defection loop.
   -

      *Estimated Probability:* *50%*

Joint Probability=0.35×0.15×0.40×0.50=0.0105

   -

   *Conjunctive Anchor:* *~1.05%*

4. CALIBRATED PROBABILITY

Combining the historical base rate, the extreme friction points of the
geopolitical Prisoner's Dilemma, and the mathematical decay of the
conjunctive chain, the final forecast is:

*Final Probability Estimate: 2.5%*
Confidence Level & Critical Triggers

My confidence in this low probability is high. The game-theoretic
incentives for defection are too steep, and the historical precedents for
proactive technological caps are too weak.

The specific data points or news events that would cause this prediction to
swing by *more than 15%* include:

   1.

   *An Unambiguous, High-Impact AI Catastrophe:* A major national security
   crisis—such as an autonomous model successfully incapacitating a nation's
   power grid, executing a multi-billion dollar automated financial exploit,
   or leaking a highly viable novel pathogen. This would shock the system,
   bypass the economic incentives, and force a joint US-China crisis
   management framework.
   2.

   *An Official Joint Superpower Communiqué:* A formal bilateral
   declaration following a US-China summit explicitly establishing a working
   group tasked with creating *binding, verifiable compute caps* with
   physical enforcement mechanisms.

-------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260713/96510f87/attachment-0001.htm>