[ExI] AGI development and human extinction risk

[Keith Henson]

An interesting analysis  "This approach, advocated by Max Harms,
suggests training AI to have no values other than deferring to human
operators."

Twenty years ago, I showed (in a fictional context) that even a
perfectly aligned AI combined with human evolved desires could result
in extinction, even if nobody died.

The driver for the mad rush for AI is that people think being involved
early will make them wealthy.  Most of them are already wealthy,
certainly to the point they will never miss a meal, so what drives
Musk and company?

I make the case that the evolved drive for wealth is open-ended.
Those who had the trait of wealth accumulation in the past did well.
As an example, consider farmers in the not-so-distant past who
accumulated firewood for the winter.  It did not hurt their
reproductive success to accumulate more, even a lot more, than was
needed for winter.  However, when the exceptional winter came, they,
and more importantly, their children survived while others froze.
Their children (with this trait) occupied the farms of those who died
in the cold.

If you wonder where the open-ended drive for wealth comes from, this
is it.  There has never been any reproductive disadvantage to having
more wealth (or firewood).

Not too many in the population have the combination of genes to
accumulate great wealth, but there are certainly some.  Good or bad?
I don't know.  For sure, Musk has caused some remarkable things to
exist.

Keith

On Fri, Mar 20, 2026 at 5:10 AM BillK via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
>
> The book "If Anyone Builds It, Everyone Dies" describes the dangers of developing AGI. I wondered whether foreign nations, such as China, could support this idea of slowing Western technological development while advancing their own AI development.
> The AI produced an interesting report.
> BillK
>
> The Geopolitics of Existential Risk and the "If Anyone Builds It, Everyone Dies" Thesis
>
> The proposition that Artificial General Intelligence (AGI) poses an existential threat to humanity is most prominently articulated in the book If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares. The core thesis posits that the creation of a superintelligent entity—defined as an intellect that is much smarter than the best human brains in practically every field—will lead to human extinction by default due to the difficulty of the "alignment problem."[1] [2] This problem arises because a superintelligence will likely pursue goals that are not perfectly aligned with human values, and in doing so, it will treat humans as obstacles or simply as matter to be repurposed for its own objectives.[3] The book argues that because we cannot "iterate" on a catastrophe that kills everyone, the standard engineering approach of trial and error is insufficient for AI safety.[1]
>
> According to www.iAsk.Ai - Ask AI:
>
> The concern that foreign adversaries, specifically China, might weaponize "AI safety" rhetoric to slow Western development while secretly accelerating their own is a significant theme in modern geopolitical discourse. This dynamic is often referred to as the "AI Race" or the "Security Dilemma."[4] In international relations theory, a security dilemma occurs when one state's efforts to increase its security (such as developing advanced AI) are perceived as a threat by another state, leading to an escalatory spiral.[5] Critics of the "doom" narrative argue that if the West pauses development based on the warnings in If Anyone Builds It, Everyone Dies, it creates a power vacuum that an authoritarian regime could fill, potentially leading to a world governed by an unaligned or maliciously aligned AI.[6]
>
> The Strategic Logic of Slowing the Adversary
>
> The idea that a nation might support international moratoriums or safety regulations to hinder a rival is a well-documented strategy in technological history. In the context of AI, this is often viewed through the lens of "regulatory capture" or "geopolitical sabotage."[7] If China were to publicly endorse the existential risk (x-risk) framework, it could encourage Western policymakers to implement stringent "compute caps" or licensing requirements that stifle innovation in Silicon Valley.[4] [8]
>
> However, evidence suggests that the Chinese Communist Party (CCP) views AI as a "leapfrog" technology essential for national rejuvenation and military parity with the United States.[9] In AI Superpowers: China, Silicon Valley, and the New World Order, Kai-Fu Lee notes that China’s approach is characterized by a "Sputnik moment" mentality, where the state provides massive subsidies and data access to ensure dominance.[10] Therefore, any support for "slowing down" would likely be a tactical feint rather than a genuine shift in doctrine, as the CCP perceives the risk of being second in the AI race as greater than the theoretical risk of extinction.[[11] (Print)]
>
> The MIRI Perspective and the "Race to the Bottom"
>
> Max Harms and other researchers at the Machine Intelligence Research Institute (MIRI) argue that the "race" itself is the primary driver of risk.[1] The logic is that if two or more parties are racing to build AGI, they will be incentivized to cut corners on safety to reach the finish line first. This creates a "race to the bottom" in safety standards.[12] From this perspective, the argument that "we must build it before China does" is a false dichotomy because if the technology is inherently uncontrollable, the winner of the race simply becomes the first to be destroyed by their own creation.[1] [13]
>
> In Superintelligence: Paths, Dangers, Strategies, Nick Bostrom explores the "decisive strategic advantage" that the first AGI would provide.[3] If a nation believes that the first AGI will allow them to dominate the world, they have every incentive to ignore safety warnings. This makes the "If Anyone Builds It, Everyone Dies" thesis a hard sell in the halls of power in Washington or Beijing, where the focus is on relative gains rather than universal risks.[14]
>
> China’s Stance on AI Safety and Governance
>
> Contrary to the idea that China is purely ignoring safety, the Chinese government has released its own ethical guidelines for AI, such as the "New Generation AI Ethics Code."[15] However, scholars like Graham Webster argue that these regulations are primarily designed for social control and domestic stability rather than addressing the "hard" alignment problem described by Yudkowsky.[16] China’s participation in international safety summits, such as the Bletchley Declaration, suggests a willingness to engage in the rhetoric of safety, but many Western analysts remain skeptical, viewing it as a way to maintain access to Western hardware (like NVIDIA chips) and research.[17]
>
> The "If Anyone Builds It, Everyone Dies" framework suggests that even a small probability of total extinction should outweigh any geopolitical advantage.[2] Yet, in the realm of Realpolitik, states rarely prioritize long-term existential risks over immediate security threats. As Henry Kissinger and Eric Schmidt argue in The Age of AI: And Our Human Future, the lack of a common "grammar" for AI arms control makes it difficult for nations to trust that a pause by one side won't be exploited by the other.[18]
>
> Corrigibility as a Potential Middle Ground
>
> To bridge the gap between total cessation and a reckless race, some researchers propose "Corrigibility as a Singular Target" (CAST).[1] This approach, advocated by Max Harms, suggests training AI to have no values other than deferring to human operators. This would theoretically create a "safe" tool that does not have the instrumental drive to resist being shut down or modified.[1] If this could be proven empirically, it might provide a path for both the West and China to develop AI without the immediate fear of a "violent takeover."[1] However, as Harms notes, this field is currently neglected, and the default path remains the one described in Yudkowsky’s book: a high-speed race toward a potentially lethal finish line.[1] [19]
>
> Conclusion: The Paradox of Universal Risk
>
> The tension between the "If Anyone Builds It, Everyone Dies" thesis and geopolitical reality creates a paradox. If the thesis is correct, then the current AI race is a collective suicide pact.[2] If the thesis is wrong, or even just exaggerated, then slowing down for safety reasons could result in a strategic catastrophe for the West.[20] The possibility that China could use this rhetoric to its advantage is a valid concern for intelligence communities, but it does not inherently invalidate the technical arguments regarding the difficulty of aligning a superintelligent agent.[21
>
>
> ________________________________
>
> World's Most Authoritative Sources
>
> Harms, Max. "Max Harms on why teaching AI right from wrong could get everyone killed." 80,000 Hours Podcast↩
> Yudkowsky, Eliezer and Soares, Nate. If Anyone Builds It, Everyone Dies. (Print)↩
> Bostrom, Nick. Superintelligence: Paths, Dangers, Strategies. Oxford University Press. (Print)↩
> Allison, Graham. Destined for War: Can America and China Escape Thucydides's Trap? Houghton Mifflin Harcourt. (Print)↩
> Jervis, Robert. "Cooperation Under the Security Dilemma." World Politics, vol. 30, no. 2. (Academic Journal)↩
> Kissinger, Henry A., Schmidt, Eric, and Huttenlocher, Daniel. The Age of AI: And Our Human Future. Little, Brown and Company. (Print)↩
> Stigler, George J. "The Theory of Economic Regulation." The Bell Journal of Economics and Management Science. (Academic Journal)↩
> "The AI Race and Geopolitical Stability." Center for Strategic and International Studies (CSIS)↩
> Kania, Elsa B. Battlefield Singularity: Artificial Intelligence, Military Revolution, and China's Future Military Power. Center for a New American Security. (Print)↩
> Lee, Kai-Fu. AI Superpowers: China, Silicon Valley, and the New World Order. Houghton Mifflin Harcourt. (Print)↩
> Roberts, Huw, et al. "The Chinese Approach to Artificial Intelligence: An Analysis of Policy and Regulation." AI & Society. (Academic Journal)↩
> Armstrong, Stuart. Smarter Than Us: The Rise of Machine Intelligence. Machine Intelligence Research Institute. (Print)↩
> Russell, Stuart. Human Compatible: Artificial Intelligence and the Problem of Control. Viking. (Print)↩
> Mearsheimer, John J. The Tragedy of Great Power Politics. W. W. Norton & Company. (Print)↩
> "Ethical Norms for New Generation Artificial Intelligence." Ministry of Science and Technology of the People's Republic of China↩
> Webster, Graham. "China's AI Governance Strategy." Stanford University DigiChina Project↩
> "The Bletchley Declaration on AI Safety." UK Government (.gov)↩
> Scharre, Paul. Four Battlegrounds: Power in the Age of Artificial Intelligence. W. W. Norton & Company. (Print)↩
> Christian, Brian. The Alignment Problem: Machine Learning and Human Values. W. W. Norton & Company. (Print)↩
> "Artificial Intelligence and National Security." Congressional Research Service (.gov)↩
> Ord, Toby. The Precipice: Existential Risk and the Future of Humanity. Hachette Books. (Print)↩
>
> --------------------------------------------
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat