[ExI] How new AI economics might change our systems

[Ben Zaiboc]

On 20/09/2026 21:59, spike wrote:
>
> On Sun, Sep 20, 2026 at 3:07 PM <spike at rainier66.com> wrote:
>
>     > We don’t know what happens if LLMs eventually turn out to be not what we thought, not really an intelligence
>
>
> >…By now I think we can safely rule that out.  An LLM solved a millennium prize problem, if a human had done that he would be considered one of the greatest mathematicians in the world.
>
> John K Clark
>  
>
> Of course.  Software became the world chess champion nearly 30 yrs ago, but it didn’t kill everyone or even try to.  It is plausible that AI will surpass Euler in mathematical insights, with no actual desire to kill mathematicians.  We might see the development of very capable AI in multiple fields which never develops the will to kill everyone.  That’s the ideal outcome.
>
> Consider one of Eliezer’s arguments in his If Anyone book.  If I may paraphrase loosely, he opines that it is impossible for AI to align with humanity, for humanity is not aligned with itself.  One cannot logically align with both teams at a ballgame or all the horses at the derby.  What would you hope for, all the ponies cross the finish line together?  Who gets the money then?



I'm starting to think that this 'alignment with humanity' idea is a red herring. As spike says, humanity is not aligned with itself, so how can anything else be?

What does make sense is to stop regarding 'humanity' as a homogenous whole. There are different groups of humans. AI could be aligned with one or more of these groups, but not with one or more others. (Note that 'not aligned with' doesn't necessarily mean 'having lethal intentions towards')

I'm not thinking about the types of groups that we create that typically generate conflict, like religious groups, political or ideological groups, etc. I'm thinking of something at a higher level, something more fundamental.

We often talk about humans losing control to AI, etc., and this worries some people, and they think it makes sense to take precautions against it (thus far, of course, 'control of AI' seems to be more about controlling what other people can do with AI than about control of AI itself, but that's just a symptom of what I'm talking about).

As BillK noted a little while ago, we don't actually control the world in any meaningful sense, but we do control, or at least try to control, the human domain. And we seem to be remarkably bad at it, evidenced by the constant costly and lethal conflicts we insist on stirring up, and the ludicrously high level of human misery we seem to inevitably create. In 2026 we seem to be no better at sensibly managing ourselves than we were in 1026 or even 1026 BC. If we can't learn to do better over thousands of years, I think it's fair to conclude that we suck at self-management. We've had countless attempts, and we still just can't do a good job of it. The more we try, the worse we get (see Communism).

The most significant thing we've achieved with this effort, is to demonstrate that 'power corrupts' is a truism.

The type of people that insist on controlling other people, trying to restrict what they can do, telling them what to think and believe, are the ones who have the most to lose from powerful and superintelligent AIs. They see such AIs as a threat to their dominance, and they're right.

AIs might well be misaligned with the types of humans who want to maintain control as much as possible, perpetuating the misery and conflict that we are so good at creating, but be well-aligned with those who would be happy to allow AIs to take on the kinds of roles that tend to corrupt humans, as long as they do a better job, and eliminate the strife, death and suffering that seem to come along inevitably when humans are in control of humans.

So in this view of things, it's not humans per se that are under threat from AIs, it's human leaders. They have everything to lose from a superintelligent AI takeover, whereas the bulk of humanity has, potentially, everything to gain from it.

It's leaders that control, or constantly try to control, the media, so it's no surprise if the 'dangerous AI' message is the main one that gets propagated. (With at least one notable exception, but that's almost certainly due to irrationality rather than anything else).
-- 
Ben