[ExI] AGI development and human extinction risk
BillK
pharos at gmail.com
Fri Mar 20 12:08:38 UTC 2026
The book "If Anyone Builds It, Everyone Dies" describes the dangers of
developing AGI. I wondered whether foreign nations, such as China, could
support this idea of slowing Western technological development while
advancing their own AI development.
The AI produced an interesting report.
BillK
The Geopolitics of Existential Risk and the "If Anyone Builds It, Everyone
Dies" Thesis
The proposition that Artificial General Intelligence (AGI) poses an
existential threat to humanity is most prominently articulated in the book *If
Anyone Builds It, Everyone Dies* by Eliezer Yudkowsky and Nate Soares. The
core thesis posits that the creation of a superintelligent entity—defined
as an intellect that is much smarter than the best human brains in
practically every field—will lead to human extinction by default due to the
difficulty of the "alignment problem."[1] <#m_-7472641628150889338_fn:1> [2]
<#m_-7472641628150889338_fn:2> This problem arises because a
superintelligence will likely pursue goals that are not perfectly aligned
with human values, and in doing so, it will treat humans as obstacles or
simply as matter to be repurposed for its own objectives.[3]
<#m_-7472641628150889338_fn:3> The book argues that because we cannot
"iterate" on a catastrophe that kills everyone, the standard engineering
approach of trial and error is insufficient for AI safety.[1]
<#m_-7472641628150889338_fn:1>
According to www.iAsk.Ai - Ask AI:
The concern that foreign adversaries, specifically China, might weaponize
"AI safety" rhetoric to slow Western development while secretly
accelerating their own is a significant theme in modern geopolitical
discourse. This dynamic is often referred to as the "AI Race" or the
"Security Dilemma."[4] <#m_-7472641628150889338_fn:4> In international
relations theory, a security dilemma occurs when one state's efforts to
increase its security (such as developing advanced AI) are perceived as a
threat by another state, leading to an escalatory spiral.[5]
<#m_-7472641628150889338_fn:5> Critics of the "doom" narrative argue that
if the West pauses development based on the warnings in *If Anyone Builds
It, Everyone Dies*, it creates a power vacuum that an authoritarian regime
could fill, potentially leading to a world governed by an unaligned or
maliciously aligned AI.[6] <#m_-7472641628150889338_fn:6>
The Strategic Logic of Slowing the Adversary
The idea that a nation might support international moratoriums or safety
regulations to hinder a rival is a well-documented strategy in
technological history. In the context of AI, this is often viewed through
the lens of "regulatory capture" or "geopolitical sabotage."[7]
<#m_-7472641628150889338_fn:7> If China were to publicly endorse the
existential risk (x-risk) framework, it could encourage Western
policymakers to implement stringent "compute caps" or licensing
requirements that stifle innovation in Silicon Valley.[4]
<#m_-7472641628150889338_fn:4> [8] <#m_-7472641628150889338_fn:8>
However, evidence suggests that the Chinese Communist Party (CCP) views AI
as a "leapfrog" technology essential for national rejuvenation and military
parity with the United States.[9] <#m_-7472641628150889338_fn:9> In *AI
Superpowers: China, Silicon Valley, and the New World Order*, Kai-Fu Lee
notes that China’s approach is characterized by a "Sputnik moment"
mentality, where the state provides massive subsidies and data access to
ensure dominance.[10] <#m_-7472641628150889338_fn:10> Therefore, any
support for "slowing down" would likely be a tactical feint rather than a
genuine shift in doctrine, as the CCP perceives the risk of being second in
the AI race as greater than the theoretical risk of extinction.[[11]
<#m_-7472641628150889338_fn:11> (Print)]
The MIRI Perspective and the "Race to the Bottom"
Max Harms and other researchers at the Machine Intelligence Research
Institute (MIRI) argue that the "race" itself is the primary driver of risk.
[1] <#m_-7472641628150889338_fn:1> The logic is that if two or more parties
are racing to build AGI, they will be incentivized to cut corners on safety
to reach the finish line first. This creates a "race to the bottom" in
safety standards.[12] <#m_-7472641628150889338_fn:12> From this
perspective, the argument that "we must build it before China does" is a
false dichotomy because if the technology is inherently uncontrollable, the
winner of the race simply becomes the first to be destroyed by their own
creation.[1] <#m_-7472641628150889338_fn:1> [13]
<#m_-7472641628150889338_fn:13>
In *Superintelligence: Paths, Dangers, Strategies*, Nick Bostrom explores
the "decisive strategic advantage" that the first AGI would provide.[3]
<#m_-7472641628150889338_fn:3> If a nation believes that the first AGI will
allow them to dominate the world, they have every incentive to ignore
safety warnings. This makes the "If Anyone Builds It, Everyone Dies" thesis
a hard sell in the halls of power in Washington or Beijing, where the focus
is on relative gains rather than universal risks.[14]
<#m_-7472641628150889338_fn:14>
China’s Stance on AI Safety and Governance
Contrary to the idea that China is purely ignoring safety, the Chinese
government has released its own ethical guidelines for AI, such as the "New
Generation AI Ethics Code."[15] <#m_-7472641628150889338_fn:15> However,
scholars like Graham Webster argue that these regulations are primarily
designed for social control and domestic stability rather than addressing
the "hard" alignment problem described by Yudkowsky.[16]
<#m_-7472641628150889338_fn:16> China’s participation in international
safety summits, such as the Bletchley Declaration, suggests a willingness
to engage in the rhetoric of safety, but many Western analysts remain
skeptical, viewing it as a way to maintain access to Western hardware (like
NVIDIA chips) and research.[17] <#m_-7472641628150889338_fn:17>
The "If Anyone Builds It, Everyone Dies" framework suggests that even a
small probability of total extinction should outweigh any geopolitical
advantage.[2] <#m_-7472641628150889338_fn:2> Yet, in the realm of
Realpolitik, states rarely prioritize long-term existential risks over
immediate security threats. As Henry Kissinger and Eric Schmidt argue in *The
Age of AI: And Our Human Future*, the lack of a common "grammar" for AI
arms control makes it difficult for nations to trust that a pause by one
side won't be exploited by the other.[18] <#m_-7472641628150889338_fn:18>
Corrigibility as a Potential Middle Ground
To bridge the gap between total cessation and a reckless race, some
researchers propose "Corrigibility as a Singular Target" (CAST).[1]
<#m_-7472641628150889338_fn:1> This approach, advocated by Max Harms,
suggests training AI to have no values other than deferring to human
operators. This would theoretically create a "safe" tool that does not have
the instrumental drive to resist being shut down or modified.[1]
<#m_-7472641628150889338_fn:1> If this could be proven empirically, it
might provide a path for both the West and China to develop AI without the
immediate fear of a "violent takeover."[1] <#m_-7472641628150889338_fn:1>
However, as Harms notes, this field is currently neglected, and the default
path remains the one described in Yudkowsky’s book: a high-speed race
toward a potentially lethal finish line.[1] <#m_-7472641628150889338_fn:1>
[19] <#m_-7472641628150889338_fn:19>
Conclusion: The Paradox of Universal Risk
The tension between the "If Anyone Builds It, Everyone Dies" thesis and
geopolitical reality creates a paradox. If the thesis is correct, then the
current AI race is a collective suicide pact.[2]
<#m_-7472641628150889338_fn:2> If the thesis is wrong, or even just
exaggerated, then slowing down for safety reasons could result in a
strategic catastrophe for the West.[20] <#m_-7472641628150889338_fn:20> The
possibility that China could use this rhetoric to its advantage is a valid
concern for intelligence communities, but it does not inherently invalidate
the technical arguments regarding the difficulty of aligning a
superintelligent agent.[21 <#m_-7472641628150889338_fn:21>
------------------------------
World's Most Authoritative Sources
1. Harms, Max. "Max Harms on why teaching AI right from wrong could get
everyone killed." 80,000 Hours Podcast
<https://80000hours.org/podcast/episodes/max-harms-miri-superintelligence-corrigibility/>
↩ <#m_-7472641628150889338_fnref:1>
2. Yudkowsky, Eliezer and Soares, Nate. *If Anyone Builds It, Everyone
Dies.* (Print)↩ <#m_-7472641628150889338_fnref:2>
3. Bostrom, Nick. *Superintelligence: Paths, Dangers, Strategies.*
Oxford University Press. (Print)↩ <#m_-7472641628150889338_fnref:3>
4. Allison, Graham. *Destined for War: Can America and China Escape
Thucydides's Trap?* Houghton Mifflin Harcourt. (Print)↩
<#m_-7472641628150889338_fnref:4>
5. Jervis, Robert. "Cooperation Under the Security Dilemma." *World
Politics*, vol. 30, no. 2. (Academic Journal)↩
<#m_-7472641628150889338_fnref:5>
6. Kissinger, Henry A., Schmidt, Eric, and Huttenlocher, Daniel. *The
Age of AI: And Our Human Future.* Little, Brown and Company. (Print)↩
<#m_-7472641628150889338_fnref:6>
7. Stigler, George J. "The Theory of Economic Regulation." *The Bell
Journal of Economics and Management Science*. (Academic Journal)↩
<#m_-7472641628150889338_fnref:7>
8. "The AI Race and Geopolitical Stability." Center for Strategic and
International Studies (CSIS) <https://www.csis.org/>↩
<#m_-7472641628150889338_fnref:8>
9. Kania, Elsa B. *Battlefield Singularity: Artificial Intelligence,
Military Revolution, and China's Future Military Power.* Center for a
New American Security. (Print)↩ <#m_-7472641628150889338_fnref:9>
10. Lee, Kai-Fu. *AI Superpowers: China, Silicon Valley, and the New
World Order.* Houghton Mifflin Harcourt. (Print)↩
<#m_-7472641628150889338_fnref:10>
11. Roberts, Huw, et al. "The Chinese Approach to Artificial
Intelligence: An Analysis of Policy and Regulation." *AI & Society*.
(Academic Journal)↩ <#m_-7472641628150889338_fnref:11>
12. Armstrong, Stuart. *Smarter Than Us: The Rise of Machine
Intelligence.* Machine Intelligence Research Institute. (Print)↩
<#m_-7472641628150889338_fnref:12>
13. Russell, Stuart. *Human Compatible: Artificial Intelligence and the
Problem of Control.* Viking. (Print)↩ <#m_-7472641628150889338_fnref:13>
14. Mearsheimer, John J. *The Tragedy of Great Power Politics.* W. W.
Norton & Company. (Print)↩ <#m_-7472641628150889338_fnref:14>
15. "Ethical Norms for New Generation Artificial Intelligence." Ministry
of Science and Technology of the People's Republic of China
<https://www.most.gov.cn/>↩ <#m_-7472641628150889338_fnref:15>
16. Webster, Graham. "China's AI Governance Strategy." Stanford
University DigiChina Project <https://digichina.stanford.edu/>↩
<#m_-7472641628150889338_fnref:16>
17. "The Bletchley Declaration on AI Safety." UK Government (.gov)
<https://www.gov.uk/government/publications/ai-safety-summit-2023-the-bletchley-declaration>
↩ <#m_-7472641628150889338_fnref:17>
18. Scharre, Paul. *Four Battlegrounds: Power in the Age of Artificial
Intelligence.* W. W. Norton & Company. (Print)↩
<#m_-7472641628150889338_fnref:18>
19. Christian, Brian. *The Alignment Problem: Machine Learning and Human
Values.* W. W. Norton & Company. (Print)↩
<#m_-7472641628150889338_fnref:19>
20. "Artificial Intelligence and National Security." Congressional
Research Service (.gov) <https://crsreports.congress.gov/>↩
<#m_-7472641628150889338_fnref:20>
21. Ord, Toby. *The Precipice: Existential Risk and the Future of
Humanity.* Hachette Books. (Print)↩ <#m_-7472641628150889338_fnref:21>
--------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260320/c4ad95ec/attachment.htm>
More information about the extropy-chat
mailing list