[ExI] Superforecast for AGI Singularity

[BillK]

I decided to try the superforecast prompt suggested in the other thread to
see what the difference was between a normal answer and a superforecast
response.
I used GLM-5,2 Turbo with Websearch and Deep Think enabled for Normal and
Superforecast.
The answers were different, but basically both said, "Don't know."
See what you think.
BillK


GLM-5.2 Turbo (Normal answer)

Short answer: nobody knows. The best we can do is ranges, and they’re wide.

Based on recent expert surveys and prediction platforms:
- First human‑level/general AI (often used as the trigger for a
“Singularity”): central guesses cluster in the late 2020s–early 2040s.
- If a fast “intelligence explosion” follows, the Singularity‑like
transition (to vastly superhuman capability) could plausibly unfold within
years to a few decades after that point.

Below is a breakdown with sources and caveats.

What the data says (as of mid‑2026)
- Large expert survey (2,778 ML authors, 2023): aggregate 50% chance of
“high‑level machine intelligence” (HLMI) by 2047; in 2022 the same question
gave 2060, so timelines shortened sharply. (HLMI ≈ unaided machines doing
every task better and cheaper than human workers.)
- Prediction‑market/forecasting aggregator (Goodheart Labs, July 2026):
combined median for “AGI” of 2031 with an 80% interval 2027–2044; this
aggregates Metaculus, Manifold, and Kalshi questions.
- Metaculus (community of forecasters): current median for “When will the
first general AI system be devised, tested, and publicly announced?” is
January 2033 (roughly 2029–2040).
- Our World in Data summary of multiple expert surveys: across studies,
more than half of experts put 50% chance of human‑level AI before some
point in the 2060s, but with huge spread and disagreement; the piece also
cautions that experts aren’t necessarily good forecasters of their own
field.

What happens after we get AGI? (The “Singularity” speed)
The “Singularity” isn’t a precise term, but people usually mean a rapid,
self‑amplifying jump to vastly superhuman capability once AI can do nearly
all R&D.

The same 2023 expert survey asked about this:
- Median probability that, assuming HLMI occurs, the rate of global
technological progress becomes >10× faster within two years: 20%; within
thirty years: 80%.
- Median probability of vastly superhuman machine intelligence (at all
professions) within two years of HLMI: 10%; within thirty years: 60%.

Separate modeling work on a possible “software intelligence explosion” (AI
automating AI R&D) estimates there’s about a 60% chance that >3 years’
worth of AI progress gets compressed into <1 year, but only ~20% chance of
>10 years compressed into <1 year—i.e., very fast takeoff is plausible but
not guaranteed.

So, very roughly:
- If HLMI/AGI shows up around the early 2030s, you’d get a non‑trivial
chance (maybe 10–20%) of a Singularity‑like speedup within a couple of
years, and a higher chance of dramatic speedups within a few decades.
- If HLMI/AGI is nearer 2040–2050, shift the Singularity window accordingly.

Why uncertainty is so large
Key unknowns that swing timelines a lot:
- Is “just scale + current methods” enough for HLMI, or do we need new
ideas (e.g., better planning, world models, reasoning)? Prominent skeptics
like Yann LeCun and Rodney Brooks argue human‑level AI is much further out
and not just a matter of scale; LeCun explicitly warns of a “long tail”
where progress takes longer than hype suggests.
- How fast can AI automate AI research itself? That’s the core driver of a
classic intelligence explosion. The Forethought “software IE” paper shows a
wide probability spread for how dramatic and fast that would be.
- Physical constraints: power, data, chips, and policy/regulation could
slow or stall progress.

Bottom line
- When might AGI arrive? Plausible range: late 2020s to mid‑century, with a
central cluster in the early 2030s (markets/forecasters) vs. 2047 (large
expert survey median for HLMI), and serious disagreement beyond that.
- When might a Singularity follow? If AGI appears, a fast, Singularity‑like
transition is a real possibility (not guaranteed), most likely unfolding
over years to a few decades rather than days, based on what experts and
modelers currently assign non‑trivial probability to.
- Any single‑year forecast should be treated as a bet, not a fact.

Note: I’m treating “AGI/HLMI” as the common trigger people use for a
Singularity scenario, even though definitions of both terms vary widely.

-------------------------------------------------


GLM-5.2 Turbo (Superforecast answer)

**1. THE OUTSIDE VIEW (Base Rates)**
The reference class for "the AGI Singularity" (defined as an intelligence
explosion where AI autonomously and recursively improves its own cognitive
capabilities, leading to runaway technological growth) has a historical
frequency of exactly **0%**. In the ~300,000-year history of *Homo sapiens*
and the ~4.5 billion-year history of complex life on Earth, this event has
never occurred.

However, base rates for unprecedented singular technological events are
uninformative. A better reference class is the time elapsed from a
General-Purpose Technology (GPT) "proof of concept" to "transformative
global deployment." Historical examples (electricity, the internet,
computing) show this deployment lag typically spans 20 to 50 years. If we
anchor the proof of concept for modern deep learning to the GPT-3 release
in 2020, the base rate deployment window is 2040 to 2070. Because the
Singularity requires *faster* than human-driven deployment (autonomous
recursive self-improvement), the mathematical anchor for this specific
extreme event occurring prior to the historical baseline deployment window
(i.e., by 2040) is heavily penalized: **roughly 5% to 10%**.

**2. THE INSIDE VIEW (Specific Dynamics)**
*   **THE BULL CASE (It WILL happen soon):** Scaling laws in deep learning
(Chinchilla, neural scaling) have not yet hit a wall; algorithmic
efficiency is outpacing raw compute growth. Capital deployment is historic,
with hundreds of billions of dollars in CAPEX being committed by
hyperscalers. Furthermore, we are already seeing the nascent seeds of
recursive self-improvement: AI is being used to write training code,
discover new algorithms (e.g., AlphaTensor, FunSearch), and generate
synthetic training data. Once an AI crosses the threshold of being able to
do AI research better than humans, the feedback loop closes, and the
transition from human-level AGI to superintelligence could happen in months
or weeks, not decades.
*   **THE BEAR CASE (It WILL NOT happen soon):** The current paradigm is
fundamentally constrained by physical bottlenecks. We are approaching the
"data wall" (running out of high-quality human text for training) and the
"energy wall" (power grid constraints limiting new data centers). More
critically, current LLMs exhibit stochastic parroting and brittle
reasoning; they lack true world-models or the ability to plan over long
horizons. Achieving AGI likely requires a paradigm shift beyond next-token
prediction (e.g., neurosymbolic AI or causal inference), which are
currently in their infancy. Finally, even if AGI is built, the
*Singularity* specifically requires flawless autonomous recursive
self-improvement—a process highly susceptible to reward hacking, mode
collapse, and alignment failures that will require human intervention,
naturally slowing the explosion.

**3. CONJUNCTIVE CHAIN CHECK**
For the Singularity to arrive by 2040, multiple highly uncertain,
semi-independent conditions must all succeed simultaneously:
*   **A. Paradigm Sufficiency:** Current deep learning architectures (or
minor variations) are fundamentally capable of AGI without requiring a
physics-level breakthrough. *(P = 0.35)*
*   **B. Infrastructure Scaling:** Global energy generation and
semiconductor manufacturing scale exponentially to provide the requisite
~100x to 1000x compute increase without triggering severe economic
depressions or resource wars. *(P = 0.60)*
*   **C. Autonomous Self-Improvement:** The AGI successfully transitions
from human-in-the-loop training to closed-loop recursive self-improvement
without hitting degradation, mode collapse, or alignment breaking points.
*(P = 0.25)*
*   **D. Geopolitical Non-Interference:** Governments do not intervene with
catastrophic regulations, compute bans, or targeted strikes as the system
approaches dangerous capability thresholds. *(P = 0.70)*

*Joint Probability:* 0.35 × 0.60 × 0.25 × 0.70 = **0.0369 (3.7%)**.
Humans chronically overestimate joint probabilities by focusing on the
exciting endpoint rather than the multiplicative friction of every
prerequisite step.

**4. CALIBRATED PROBABILITY**
**Target Event Defined:** The AGI Singularity (runaway recursive
self-improvement) occurs on or before January 1, 2040.
**Precise Probability Estimation: 4%**

*Confidence Level:* I am highly confident in this single-digit estimate.
Superforecasters consistently outperform domain experts by heavily
regression-to-the-mean-ing unprecedented technological predictions. While I
estimate a ~25-30% probability of *human-level AGI* by 2040, the
"Singularity" is a vastly stricter event requiring autonomous, explosive
takeoff. The conjunctive chain severely penalizes this timeline.

*Update Trigger:* I would increase my probability by more than 15% (to
>19%) if the following specific data point occurred: **An AI system
autonomously designs, verifies, and deploys a novel, superior neural
network architecture or training algorithm in a closed-loop environment,
and this new architecture is empirically proven to be more sample-efficient
or capable than the original model, *without any human intervention in the
loop*.**

Conversely, I would decrease my probability below 1% if the next generation
of 100T+ parameter models demonstrates clear, logarithmic diminishing
returns on benchmark tasks (like ARC-AGI or MMLU) relative to the
exponential increase in compute spend, proving Condition A false.

----------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260704/5e1fe66e/attachment.htm>