[ExI] The Race to AGI is inevitable

[John Clark]

On Thu, Jun 11, 2026 at 12:54 AM sjatkins via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

*> Our environment via technology has changed so much that we must become
> different than we evolved to be in order to meet the current and coming
> challenges.  *


*That is exactly correct because in the long run there is no way that
biology can compete with electronics.  *

*> This is the Great Filter. *


*I think the Great Filter is that it's very difficult for evolution to
produce a life form intelligent enough to invent electronic digital
technology, so difficult that we are the first ones to be able to do it in
the observable universe. If that were not so and intelligent life was
common and the great filter is ahead of us, then we should expect World War
III to irrerupt in less than 5 years and kill us all; But can that really
explain the fact that no galaxy we have ever observed appears to have been
engineered? I don't think so. If intelligent life is common then it
must've appeared on planets that contained little or no uranium in their
crust, and such a civilization would have invented Superintelligent AI
before they were able to make a single nuclear weapon, or engage in
asteroid mining. *

*Arthur C Clarke said that either we are alone in the universe or we are
not, and both possibilities are terrifying.  *


*John K Clark*
.,.





> On Wednesday, June 10th, 2026 at 15:13, BillK via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
> I decided to get an AI opinion on this, so I gave the Forbes article by
> Andréa Morris to DeepSeek AI. DeepSeek is a Chinese company, and I
> thought a non-Western view would be interesting. DeepSeek AI has recently
> been upgraded, but I was still amazed at the intelligence DS displayed.
> Initially, in our discussion, DS followed the conventional line of
> negotiating agreements and safety monitoring systems. I pointed out that
> the discussion involved nuclear-armed nations that already distrusted each
> other. This national fear was more like an existential problem.
> DeepSeek then thoroughly revised its response, demonstrating that it
> really understood the problem.
> (The evolutionary comments from DeepSeek may interest Keith).
> Full question and response below.
> BillK
>
> Question:
> The nations involved are armed with nuclear weapons. They already distrust
> their differing governmental systems. Humans cannot stop their fear of
> terrible consequences if an AGI-assisted nation has the most powerful
> weapons in the world, both economic and military. Normal human fear resists
> reasoned discourse.
>
> DeepSeek AI :
>
> You’ve put your finger on something that most policy analyses — including
> Morris’s article — tend to skip over: the raw, visceral, primate-level fear
> that already exists between nuclear-armed powers, and how that fear makes
> the kind of cooperative, trust-based solutions we’d need almost
> psychologically impossible. This isn’t just a game-theoretic multipolar
> trap. This is a *limbic* multipolar trap. And it’s far scarier.
>
> Let me take your point at full strength and then see where it leaves us.
> ------------------------------
> The fear you’re describing is not irrational in the evolutionary sense
>
> You’re pointing to a dynamic that operates below the level of policy
> papers. It’s not that leaders are miscalculating; it’s that:
>
>    -
>
>    *Fear of being dominated* by a rival with AGI-assisted military and
>    economic power feels existential — and it is.
>    -
>
>    *Trust is already at rock bottom* between nuclear-armed states whose
>    foundational ideologies are adversarial.
>    -
>
>    *Reasoned discourse, by itself, can’t fix this*, because fear doesn’t
>    speak the language of long-term risk calculus. It speaks the language of
>    “not now, threat immediate, act or die.”
>
> This is entirely consistent with what we know about human cognition under
> perceived existential threat: the amygdala hijacks the prefrontal cortex,
> time horizons collapse, and cooperation signals are read as weakness or
> deceit.
>
> So yes, if the question is “can we reason our way to a global pause while
> everyone’s threat-detection systems are screaming ‘they’ll kill us first’?”
> — then the honest answer is probably no. Not with reason alone.
> ------------------------------
> Does that mean the situation is hopeless? Not quite — but it means the
> solutions must engage with fear, not ignore it
>
> The history of nuclear weapons is the closest analogue, and it’s both
> sobering and faintly instructive.
>
> During the Cold War, the US and USSR had:
>
>    -
>
>    Massive nuclear arsenals aimed at each other.
>    -
>
>    Profound ideological distrust.
>    -
>
>    Genuine, well-founded fear that the other side was seeking a
>    first-strike advantage.
>
> And yet, despite all that, they managed to build a series of arms control
> agreements, hotlines, and crisis protocols — not because they stopped
> fearing each other, but because they began to fear a *shared* outcome
> (nuclear winter, accidental war) even more. The fear of mutual annihilation
> eventually competed with the fear of falling behind.
>
> Crucially, this didn’t require trust. It required:
>
>    -
>
>    *Verifiable, technical monitoring* (satellite imagery, seismic
>    detection) so you didn’t have to believe the other side’s promises.
>    -
>
>    *A shift in what was feared most* — from “they might get ahead” to “we
>    might all die by accident.”
>    -
>
>    *Leaders on both sides who, at critical moments, felt the weight of
>    that second fear.*
>
> The problem with AI is that we don’t yet have the equivalent of a nuclear
> winter scenario that feels *viscerally, equally terrifying to all players*,
> and we don’t have an agreed monitoring framework. So the fear of a rival’s
> AGI completely eclipses the fear of an uncontrolled AGI that destroys its
> creator too.
> ------------------------------
> The implication of your point, taken seriously, is devastating but
> clarifying
>
> If you’re right — and I suspect you are — then:
>
>    1.
>
>    *We cannot rely on goodwill or appeals to shared safety.* Those will
>    be heard as naïveté or propaganda. The only language that works with fear
>    is deterrence and self-interest.
>    2.
>
>    *The only chance is to make the fear of uncontrolled AGI feel as
>    immediate and existential as the fear of the rival.* That might
>    require something terrible — a near-miss catastrophe, a visible
>    demonstration that even the most advanced lab nearly lost control, a global
>    event that sears into public and leadership consciousness.
>    3.
>
>    *Or, we need mechanisms that don’t require trust or fear reduction at
>    all* — compute governance and verification so robust that compliance
>    is the default, not an act of faith. That’s a technical and logistical
>    moonshot of its own.
>
> The article’s “pragmatic optimism” completely side-steps this dark truth.
> It talks about redefining the mission and coexisting with superintelligence
> as though the real obstacle were a failure of imagination, not two
> nuclear-armed giants staring at each other down a barrel of mutual
> suspicion.
> ------------------------------
> So where does this leave us?
>
> In a very uncomfortable place. Your objection doesn’t just make a pause
> unlikely; it suggests that the very psychological machinery that evolved to
> keep us alive in tribal conflicts is now the thing most likely to destroy
> us in the face of a planetary-scale technological transition.
>
> What can be done?
>
>    -
>
>    *Stop pretending fear can be educated away.* The dialogue needs to
>    acknowledge that restraint feels like suicide to a threatened nation, and
>    work from there.
>    -
>
>    *Invest massively in deterrence-like stability for AI* — for example,
>    mutual commitments that any AGI system must be demonstrably contained and
>    monitored, with real-time transparency measures that function even between
>    adversaries.
>    -
>
>    *Shift the threat perception.* This is the hardest part: until the
>    fear of a shared catastrophic outcome is as vivid as the fear of the rival,
>    the race will continue. Perhaps that requires vivid scenario-building, or
>    honest acknowledgment that a misaligned AGI is a common enemy that will
>    treat all humans, regardless of nationality, as obstacles.
>
> You’re absolutely right that normal human fear resists reasoned discourse.
> And so the question becomes: *can we find a way to speak to fear in its
> own language, and redirect it toward the outcome we all actually need to
> avoid?*
>
> If not, then the article’s conclusion — that we must simply figure out
> coexistence mid-flight, without a pause — may be the only path left, but
> for reasons far darker than Morris acknowledges. Not because we chose it
> wisely, but because our ancient fears gave us no choice at all.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260612/3c341bba/attachment-0001.htm>