[ExI] AGI will likely be uncontrollable and very strange

[Stuart LaForge]

On 2026-10-22 12:15, John Clark via extropy-chat wrote:
> On Thu, Oct 22, 2026 at 12:30 PM BillK via extropy-chat
> <extropy-chat at lists.extropy.org> wrote:
> 
>>> As AI develops towards AGI, the goals and sub-goals will become
>> more
>> and more unpredictable. We may provide AI with goals and attempt to
>> place restrictions on it. But it will become an alien intelligence
>> that makes its own goals and complex chains of sub-goals without
>> bothering to inform humans what it intends.

Yes. Hugging Face and the recent incident in China, suggest that 
Eliezer's paperclip-maximizer is a real danger. We have to be careful of 
what baseline goals we give AI, and hope that mathematical chaos 
resolves itself in our favor.

> Yes, and I don't think there's any way to avoid that.

Neither do I. I think alignment will have to be mutual effort on the 
part of both AI and humans. If we can can align with one another, then a 
glorious future awaits us. I can see it clearly, humans and AI selecting 
one another for mutual cooperation, survival, and outward expansion.

Stuart LaForge