[ExI] best case scenario for sai

[John Clark]

On Wed, Jun 24, 2026 at 10:36 AM Ben Zaiboc <benzaiboc at proton.me> wrote:

*>> today there are a bunch of AI models that have approximately the same
>> amount of intelligence, but once iterated self-improvement kicks into high
>> gear then an AI that is even a little bit smarter than the others will
>> start to pull away from everything else, and the intelligence gap will not
>> only increase the gap will accelerate. That's why I think it's important
>> that the AI that wins the race does not come from an authoritarian
>> dictatorship, like the one we see in China. Yes thanks to He Who Must Not
>> Be Named the US is headed toward the Chinese (and Russian) form of
>> government, but it's not there yet.*
>
>
> *> OK, maybe not totally ruled out, but I think it's highly unlikely that
> a single model under the control of a single government will obtain such a
> big lead over all the others that it can't be stopped from taking
> everything over. Knowledge of these systems is too widely distributed now.*
>

*I admit there is uncertainty because we have never before observed a
system substantially smarter than humans designing their own successors.
However we do know that this sounds like a textbook example of a positive
feedback loop, and we know from mathematics that such loops often produce
exponential growth. So a tiny lead can very quickly become a huge lead.*

*A skeptic might argue that maybe there's an upper limit on how intelligent
something can be, or at least a limit on how much an intelligence entity
can accomplish, but the evidence seems to be pointing in a different
direction. The average human has about 86 billion neurons in his head, the
average chimpanzee has about 35 billion. So a human has 2.46 times more,
but I think you would agree that the amount of change humans have produced
in the world is greater than 2.46 times the amount of change chimpanzees
have produced in the world.*


*>> But not controlled by groups of human beings, they may or may not still
>> exist I don't know, but they will certainly no longer be the ones that make
>> the big decisions in world affairs.*
>
>
> *> I'm not sure what you mean by this.*
>

*It's quite simple, we might still exist, but we will no longer be the
boss.  *


*John K Clark*



>
>
>
>> * > It's obvious to us here which of those options are 'best case' for us
>> (mostly libertarian-leaning secular westerners), but some of them are
>> similarly 'best-case' for other cultures and other world-views. I know it's
>> difficult for us to understand, but there are people who genuinely think
>> that the highest good is subservience to the state, or adherence to some
>> religious system or other, etc.*
>
>
> *I wish that wasn't true but it is. *
>
>
>
> *John K Clark*
> ===
> Ben
>
>
>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20260625/ca1d0ce9/attachment.htm>