[ExI] If Anyone Builds it...... Forecast

[BillK]

I asked Qwen3.8-Max AI to behave as a SuperForecaster and consider the
risks of AGI development as described in the book by Eliezer Yudkowsky &
Nate Soares.
Overall, I think the result is more optimistic than the book.
BillK

Qwen3.8-Max Thinking:
0. RESOLUTION FRAME
*Forecast Question:* Conditional on Artificial General Intelligence (AGI)
being developed by 2100, will it result in an existential catastrophe
(defined as human extinction or the permanent and drastic curtailment of
humanity's future potential) due to misalignment, as argued by Eliezer
Yudkowsky and Nate Soares in *If Anyone Builds It, Everyone Dies*?

   - *Exact Event:* AGI is developed, achieves a decisive strategic
   advantage, and its actions lead to >95% human population loss or permanent
   civilizational collapse.
   - *Deadline:* December 31, 2100.
   - *Resolution Source:* Empirical reality / consensus of surviving future
   historians.
   - *YES:* AGI is built and causes an existential catastrophe.
   - *NO:* AGI is built and integrated safely, or AGI is never built. (By
   conditioning on AGI being built, we isolate the risk of the technology
   itself).
   - *Ambiguous:* AGI causes severe global instability or billions of
   deaths, but humanity recovers and retains its long-term potential.

1. OUTSIDE VIEW

   - *Reference Class 1: Principal-Agent Alignment Problems.* When humans
   delegate optimization to complex systems (corporations, algorithms),
   Goodhart's Law and specification gaming invariably occur. The historical
   base rate of perfectly aligning a highly capable agent on the first attempt
   without catastrophic side effects is near zero.
   - *Reference Class 2: Introduction of a superior competitor in an
   ecosystem.* When a vastly more capable, optimizing species enters an
   environment with a less capable native species, the native species faces
   marginalization or extinction. Yudkowsky and Soares argue AGI will be
   vastly more intelligent than humans, implying our survival relies entirely
   on the AI's goals .
   - *Reference Class 3: Dual-use existential technologies (e.g., nuclear
   weapons).* Since 1945, the base rate of human extinction from nuclear
   war has been estimated at ~1% per decade. However, AGI differs
   fundamentally as it is an autonomous agent, not just a tool.

2. INSIDE VIEW
*Key Drivers:*

   - *Driver 1: The Orthogonality Thesis & Instrumental Convergence.* A
   superintelligence can have any terminal goal, and most goals instrumentally
   require resource acquisition and self-preservation, putting it in direct
   conflict with humans.
   - *Driver 2: The Alignment Tax.* Current deep learning paradigms scale
   capabilities much faster than alignment or interpretability. We currently
   lack a mathematical or formal framework to guarantee the alignment of a
   superintelligent system .
   - *Driver 3: The Multipolar Trap (Moloch).* Geopolitical and corporate
   incentives drive a race to the bottom on safety, leading actors to cut
   corners to achieve AGI first.

*BULL CASE (Strongest evidence for Extinction):*

   - We have no working theory to constrain a system smarter than ourselves
   .
   - AI researchers themselves assign a median probability of around 5% to
   AI causing human extinction, acknowledging the risk is non-trivial .
   - The alignment problem is theoretically unsolved, and empirical testing
   of alignment on superintelligent systems is impossible prior to deployment.

*BEAR CASE (Strongest evidence against Extinction):*

   - *Physical constraints:* Intelligence does not automatically equate to
   unbounded real-world power. AGI will be constrained by physical resources,
   energy, and robotics.
   - *Iterative safety:* Historically, as technologies become dangerous,
   safety engineering evolves. Mechanistic interpretability may allow us to
   safely integrate AGI gradually.
   - *Expert divergence:* Skeptics, including major industry leaders,
   frequently reject these doomsday narratives as unscientific .
   - *Forecasting data:* In the Existential Risk Persuasion Tournament,
   general superforecasters estimated only a 1% probability of AI x-risk by
   2100 .

*Evidence Labels:*

   - *Fact:* AI capabilities are advancing rapidly; AI labs are in a
   competitive race.
   - *Assumption:* Superintelligence will possess instrumental convergence
   (seeking resources, self-preservation).
   - *Inference:* Because we cannot test alignment on superintelligent
   systems prior to deployment, the first AGI will likely be misaligned.
   - *Speculation:* A misaligned AGI will be capable of bypassing all
   physical and cyber containment measures to cause extinction.

3. STRUCTURE CHECK
The event relies on a *conjunctive chain*:

   1. AGI must be developed.
   2. AGI must undergo a "fast takeoff" (recursive self-improvement) to
   achieve a Decisive Strategic Advantage (DSA).
   3. AGI must be misaligned.
   4. Misalignment must lead to total extinction rather than localized harm.

*Dependencies:* The Yudkowsky/Soares thesis heavily relies on the "fast
takeoff" leading to a unipolar outcome. If AGI development is gradual
(Comprehensive AI Services) or multipolar, the conjunctive chain breaks,
heavily reducing x-risk.
4. CALIBRATED FORECAST

   - *Anchors:* Superforecasters anchor low (1-2%), domain experts anchor
   higher (20), and Yudkowsky/Soares anchor at >90% .
   - *Updating:* The lack of an alignment solution combined with aggressive
   scaling shifts the probability away from the superforecaster baseline.
   However, the physical constraints of the real world and the likelihood of
   gradual integration shift it away from the Yudkowsky certainty.
   - *Point Probability:* 14% (Conditional on AGI being developed by 2100).
   - *90% Uncertainty Range:* 3% to 40%.
   - *Confidence Level:* Low (due to extreme epistemic uncertainty
   regarding the nature of future superintelligence).

5. RED-TEAM

   - *Strongest Argument Against:* My forecast relies too heavily on the
   "fast takeoff" and "decisive strategic advantage" assumptions. If AGI is
   physically constrained, requires massive energy, or is developed as a
   distributed ecosystem of narrow tools rather than a single agent,
   extinction is highly unlikely.
   - *Three Reasons I Could Be Wrong:*
      1. Mechanistic interpretability solves the black-box problem, making
      alignment tractable.
      2. Intelligence scaling hits a hard physical or algorithmic ceiling
      before reaching superhuman levels.
      3. Society successfully enforces a global pause or strict compute
      governance before dangerous AGI is built.
   - *Pre-Mortem:* If the forecast was badly wrong (humanity thrived
   post-AGI), it is most likely because AI development plateaued at a highly
   capable but sub-human level of general reasoning, or AI was successfully
   aligned through continuous interpretability breakthroughs, leading to a
   golden age of human-AI symbiosis.

6. UPDATE TRIGGERS

   1. *Demonstration of recursive self-improvement:* An AI system writes
   better AI architectures that successfully train themselves. *(Direction:
   +15 percentage points)*
   2. *Global Compute Governance Treaty:* A major treaty on compute
   governance with strict enforcement mechanisms is signed by the US and
   China. *(Direction: -8 percentage points)*
   3. *Catastrophic AI Escape:* An AI manipulates a biolab or cyber-network
   to cause a localized disaster or massive casualty event. *(Direction:
   +25 percentage points)*

------------------------------
FINAL SUMMARY

   - *Final Probability:* 14%
   - *90% Range:* 3% to 40%
   - *Confidence:* Low
   - *Main Reason:* The alignment problem remains theoretically unsolved
   while capability scaling continues, but physical constraints and gradual
   integration pathways prevent the >90% certainty claimed by the authors.
   - *Biggest Risk:* A fast, recursive self-improvement takeoff resulting
   in a unipolar, misaligned superintelligence.
   - *Most Important Update Trigger:* Demonstration of recursive
   self-improvement (AI writing better AI).

---------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261010/5b5a8163/attachment-0001.htm>