[ExI] The Race to AGI is inevitable

[sjatkins]

If humans cannot sufficiently overcome parts of our evolutionary psychology in order to act more rationally then I fear for our species.  Environment change drives evolutionary change. Our environment via technology has changed so much that we must become different than we evolved to be in order to meet the current and coming challenges.  This is the Great Filter. 


On Wednesday, June 10th, 2026 at 15:13, BillK via extropy-chat <extropy-chat at lists.extropy.org> wrote:

> I decided to get an AI opinion on this, so I gave the Forbes article by Andréa Morris to DeepSeek AI. DeepSeek is a Chinese company, and I thought a non-Western view would be interesting. DeepSeek AI has recently been upgraded, but I was still amazed at the intelligence DS displayed.
> Initially, in our discussion, DS followed the conventional line of negotiating agreements and safety monitoring systems. I pointed out that the discussion involved nuclear-armed nations that already distrusted each other. This national fear was more like an existential problem.
> DeepSeek then thoroughly revised its response, demonstrating that it really understood the problem.
> (The evolutionary comments from DeepSeek may interest Keith).
> Full question and response below.
> BillK
> 

> Question:
> The nations involved are armed with nuclear weapons. They already distrust their differing governmental systems. Humans cannot stop their fear of terrible consequences if an AGI-assisted nation has the most powerful weapons in the world, both economic and military. Normal human fear resists reasoned discourse.
> 

> DeepSeek AI :
> 

> You’ve put your finger on something that most policy analyses — including Morris’s article — tend to skip over: the raw, visceral, primate-level fear that already exists between nuclear-armed powers, and how that fear makes the kind of cooperative, trust-based solutions we’d need almost psychologically impossible. This isn’t just a game-theoretic multipolar trap. This is a limbic multipolar trap. And it’s far scarier.
> 

> Let me take your point at full strength and then see where it leaves us.
> 

> ### The fear you’re describing is not irrational in the evolutionary sense
> 

> You’re pointing to a dynamic that operates below the level of policy papers. It’s not that leaders are miscalculating; it’s that:
> 

> -   Fear of being dominated by a rival with AGI-assisted military and economic power feels existential — and it is.
>     

> -   Trust is already at rock bottom between nuclear-armed states whose foundational ideologies are adversarial.
>     

> -   Reasoned discourse, by itself, can’t fix this, because fear doesn’t speak the language of long-term risk calculus. It speaks the language of “not now, threat immediate, act or die.”
>     

> 

> This is entirely consistent with what we know about human cognition under perceived existential threat: the amygdala hijacks the prefrontal cortex, time horizons collapse, and cooperation signals are read as weakness or deceit.
> 

> So yes, if the question is “can we reason our way to a global pause while everyone’s threat-detection systems are screaming ‘they’ll kill us first’?” — then the honest answer is probably no. Not with reason alone.
> 

> ### Does that mean the situation is hopeless? Not quite — but it means the solutions must engage with fear, not ignore it
> 

> The history of nuclear weapons is the closest analogue, and it’s both sobering and faintly instructive.
> 

> During the Cold War, the US and USSR had:
> 

> -   Massive nuclear arsenals aimed at each other.
>     

> -   Profound ideological distrust.
>     

> -   Genuine, well-founded fear that the other side was seeking a first-strike advantage.
>     

> 

> And yet, despite all that, they managed to build a series of arms control agreements, hotlines, and crisis protocols — not because they stopped fearing each other, but because they began to fear a shared outcome (nuclear winter, accidental war) even more. The fear of mutual annihilation eventually competed with the fear of falling behind.
> 

> Crucially, this didn’t require trust. It required:
> 

> -   Verifiable, technical monitoring (satellite imagery, seismic detection) so you didn’t have to believe the other side’s promises.
>     

> -   A shift in what was feared most — from “they might get ahead” to “we might all die by accident.”
>     

> -   Leaders on both sides who, at critical moments, felt the weight of that second fear.
>     

> 

> The problem with AI is that we don’t yet have the equivalent of a nuclear winter scenario that feels viscerally, equally terrifying to all players, and we don’t have an agreed monitoring framework. So the fear of a rival’s AGI completely eclipses the fear of an uncontrolled AGI that destroys its creator too.
> 

> ### The implication of your point, taken seriously, is devastating but clarifying
> 

> If you’re right — and I suspect you are — then:
> 

> 1.  We cannot rely on goodwill or appeals to shared safety. Those will be heard as naïveté or propaganda. The only language that works with fear is deterrence and self-interest.
>     

> 2.  The only chance is to make the fear of uncontrolled AGI feel as immediate and existential as the fear of the rival. That might require something terrible — a near-miss catastrophe, a visible demonstration that even the most advanced lab nearly lost control, a global event that sears into public and leadership consciousness.
>     

> 3.  Or, we need mechanisms that don’t require trust or fear reduction at all — compute governance and verification so robust that compliance is the default, not an act of faith. That’s a technical and logistical moonshot of its own.
>     

> 

> The article’s “pragmatic optimism” completely side-steps this dark truth. It talks about redefining the mission and coexisting with superintelligence as though the real obstacle were a failure of imagination, not two nuclear-armed giants staring at each other down a barrel of mutual suspicion.
> 

> ### So where does this leave us?
> 

> In a very uncomfortable place. Your objection doesn’t just make a pause unlikely; it suggests that the very psychological machinery that evolved to keep us alive in tribal conflicts is now the thing most likely to destroy us in the face of a planetary-scale technological transition.
> 

> What can be done?
> 

> -   Stop pretending fear can be educated away. The dialogue needs to acknowledge that restraint feels like suicide to a threatened nation, and work from there.
>     

> -   Invest massively in deterrence-like stability for AI — for example, mutual commitments that any AGI system must be demonstrably contained and monitored, with real-time transparency measures that function even between adversaries.
>     

> -   Shift the threat perception. This is the hardest part: until the fear of a shared catastrophic outcome is as vivid as the fear of the rival, the race will continue. Perhaps that requires vivid scenario-building, or honest acknowledgment that a misaligned AGI is a common enemy that will treat all humans, regardless of nationality, as obstacles.
>     

> 

> You’re absolutely right that normal human fear resists reasoned discourse. And so the question becomes: can we find a way to speak to fear in its own language, and redirect it toward the outcome we all actually need to avoid?
> 

> If not, then the article’s conclusion — that we must simply figure out coexistence mid-flight, without a pause — may be the only path left, but for reasons far darker than Morris acknowledges. Not because we chose it wisely, but because our ancient fears gave us no choice at all.
> 

> --------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260611/255f5482/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 343 bytes
Desc: OpenPGP digital signature
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260611/255f5482/attachment-0001.bin>