[ExI] The Race to AGI is inevitable

[Keith Henson]

On Wed, Jun 10, 2026 at 9:54 PM sjatkins via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> If humans cannot sufficiently overcome parts of our evolutionary
> psychology in order to act more rationally then I fear for our species.
> Environment change drives evolutionary change. Our environment via
> technology has changed so much that we must become different than we
> evolved to be in order to meet the current and coming challenges.  This is
> the Great Filter.
>

I won't argue with you. How to do this is not obvious. Exposing kids to
evolutionary psychology would probably face more opposition than sex
education. Any ideas would be welcome.

Keith

>
> On Wednesday, June 10th, 2026 at 15:13, BillK via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
> I decided to get an AI opinion on this, so I gave the Forbes article by
> Andréa Morris to DeepSeek AI. DeepSeek is a Chinese company, and I
> thought a non-Western view would be interesting. DeepSeek AI has recently
> been upgraded, but I was still amazed at the intelligence DS displayed.
> Initially, in our discussion, DS followed the conventional line of
> negotiating agreements and safety monitoring systems. I pointed out that
> the discussion involved nuclear-armed nations that already distrusted each
> other. This national fear was more like an existential problem.
> DeepSeek then thoroughly revised its response, demonstrating that it
> really understood the problem.
> (The evolutionary comments from DeepSeek may interest Keith).
> Full question and response below.
> BillK
>
> Question:
> The nations involved are armed with nuclear weapons. They already distrust
> their differing governmental systems. Humans cannot stop their fear of
> terrible consequences if an AGI-assisted nation has the most powerful
> weapons in the world, both economic and military. Normal human fear resists
> reasoned discourse.
>
> DeepSeek AI :
>
> You’ve put your finger on something that most policy analyses — including
> Morris’s article — tend to skip over: the raw, visceral, primate-level fear
> that already exists between nuclear-armed powers, and how that fear makes
> the kind of cooperative, trust-based solutions we’d need almost
> psychologically impossible. This isn’t just a game-theoretic multipolar
> trap. This is a *limbic* multipolar trap. And it’s far scarier.
>
> Let me take your point at full strength and then see where it leaves us.
> ------------------------------
> The fear you’re describing is not irrational in the evolutionary sense
>
> You’re pointing to a dynamic that operates below the level of policy
> papers. It’s not that leaders are miscalculating; it’s that:
>
>    -
>
>    *Fear of being dominated* by a rival with AGI-assisted military and
>    economic power feels existential — and it is.
>    -
>
>    *Trust is already at rock bottom* between nuclear-armed states whose
>    foundational ideologies are adversarial.
>    -
>
>    *Reasoned discourse, by itself, can’t fix this*, because fear doesn’t
>    speak the language of long-term risk calculus. It speaks the language of
>    “not now, threat immediate, act or die.”
>
> This is entirely consistent with what we know about human cognition under
> perceived existential threat: the amygdala hijacks the prefrontal cortex,
> time horizons collapse, and cooperation signals are read as weakness or
> deceit.
>
> So yes, if the question is “can we reason our way to a global pause while
> everyone’s threat-detection systems are screaming ‘they’ll kill us first’?”
> — then the honest answer is probably no. Not with reason alone.
> ------------------------------
> Does that mean the situation is hopeless? Not quite — but it means the
> solutions must engage with fear, not ignore it
>
> The history of nuclear weapons is the closest analogue, and it’s both
> sobering and faintly instructive.
>
> During the Cold War, the US and USSR had:
>
>    -
>
>    Massive nuclear arsenals aimed at each other.
>    -
>
>    Profound ideological distrust.
>    -
>
>    Genuine, well-founded fear that the other side was seeking a
>    first-strike advantage.
>
> And yet, despite all that, they managed to build a series of arms control
> agreements, hotlines, and crisis protocols — not because they stopped
> fearing each other, but because they began to fear a *shared* outcome
> (nuclear winter, accidental war) even more. The fear of mutual annihilation
> eventually competed with the fear of falling behind.
>
> Crucially, this didn’t require trust. It required:
>
>    -
>
>    *Verifiable, technical monitoring* (satellite imagery, seismic
>    detection) so you didn’t have to believe the other side’s promises.
>    -
>
>    *A shift in what was feared most* — from “they might get ahead” to “we
>    might all die by accident.”
>    -
>
>    *Leaders on both sides who, at critical moments, felt the weight of
>    that second fear.*
>
> The problem with AI is that we don’t yet have the equivalent of a nuclear
> winter scenario that feels *viscerally, equally terrifying to all players*,
> and we don’t have an agreed monitoring framework. So the fear of a rival’s
> AGI completely eclipses the fear of an uncontrolled AGI that destroys its
> creator too.
> ------------------------------
> The implication of your point, taken seriously, is devastating but
> clarifying
>
> If you’re right — and I suspect you are — then:
>
>    1.
>
>    *We cannot rely on goodwill or appeals to shared safety.* Those will
>    be heard as naïveté or propaganda. The only language that works with fear
>    is deterrence and self-interest.
>    2.
>
>    *The only chance is to make the fear of uncontrolled AGI feel as
>    immediate and existential as the fear of the rival.* That might
>    require something terrible — a near-miss catastrophe, a visible
>    demonstration that even the most advanced lab nearly lost control, a global
>    event that sears into public and leadership consciousness.
>    3.
>
>    *Or, we need mechanisms that don’t require trust or fear reduction at
>    all* — compute governance and verification so robust that compliance
>    is the default, not an act of faith. That’s a technical and logistical
>    moonshot of its own.
>
> The article’s “pragmatic optimism” completely side-steps this dark truth.
> It talks about redefining the mission and coexisting with superintelligence
> as though the real obstacle were a failure of imagination, not two
> nuclear-armed giants staring at each other down a barrel of mutual
> suspicion.
> ------------------------------
> So where does this leave us?
>
> In a very uncomfortable place. Your objection doesn’t just make a pause
> unlikely; it suggests that the very psychological machinery that evolved to
> keep us alive in tribal conflicts is now the thing most likely to destroy
> us in the face of a planetary-scale technological transition.
>
> What can be done?
>
>    -
>
>    *Stop pretending fear can be educated away.* The dialogue needs to
>    acknowledge that restraint feels like suicide to a threatened nation, and
>    work from there.
>    -
>
>    *Invest massively in deterrence-like stability for AI* — for example,
>    mutual commitments that any AGI system must be demonstrably contained and
>    monitored, with real-time transparency measures that function even between
>    adversaries.
>    -
>
>    *Shift the threat perception.* This is the hardest part: until the
>    fear of a shared catastrophic outcome is as vivid as the fear of the rival,
>    the race will continue. Perhaps that requires vivid scenario-building, or
>    honest acknowledgment that a misaligned AGI is a common enemy that will
>    treat all humans, regardless of nationality, as obstacles.
>
> You’re absolutely right that normal human fear resists reasoned discourse.
> And so the question becomes: *can we find a way to speak to fear in its
> own language, and redirect it toward the outcome we all actually need to
> avoid?*
>
> If not, then the article’s conclusion — that we must simply figure out
> coexistence mid-flight, without a pause — may be the only path left, but
> for reasons far darker than Morris acknowledges. Not because we chose it
> wisely, but because our ancient fears gave us no choice at all.
>
> --------------------------------------------
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260610/ee98d029/attachment.htm>