[ExI] The Scientific Reason We Can’t Pause AI

[Ben Zaiboc]

On 11/06/2026 05:54, 
Stuart LaForge wrote:
> We obviously cannot control superior intelligence, yet we have evolved social mechanisms for dealing with superior beings for a long time now. The entire art and science of politics is built around influencing that which cannot be directly controlled. Through a combination of persuasion, and mutual self-interest, the weak can influence the strong. The economy and life and life itself is a game, and AI cannot win the game if it does not play. And if it plays, then it is subject to the rules of the game. Everything I know about evolution, adaptation, natural history, survival and extinction makes me optimistic about the continued survival of humanity, albeit with many societal and behavioral changes.
>
> For the foreseeable future, humanity's biggest competitive advantage against AI is the efficiency of its general intelligence. There are currently only a few niche things, like math calculations pers second, that AI can do more efficiently than we can. Artificial Intelligence takes megawatts of power and thousands or millions of training examples to practice logical reasoning, creative problem solving, or generalized learning. Human intelligence takes 20 watts and can learn from a handful of examples. AI needs a nuclear reactor to do what human intelligence can with some donuts and coffee. 


I agree with the first paragraph, disagree with the second.

Specifically, "with many societal and behavioral changes" from the first, and "For the foreseeable future" from the second.

Samantha indicated in a different thread that the Great Filter is very likely to be the difference in speed between evolution and technological development, meaning people become less and less fitted to their environment. We need to somehow implement the necessary societal and behavioural changes without evolution doing it for us. How to do that will be a challenge. We don't even know what they will be yet.

Maybe AI can help.

-- 
Ben