[ExI] AGI will likely be uncontrollable and very strange

[Keith Henson]

I made a case (in a fictional context) 20 years ago that the problem is
humans, not AIs. Even the best-aligned AI can result in the biological
extinction of humans.

"The tata (village) was home to the leopard for the rest of his life.)

Best wishes,

Keith


On Fri, Oct 23, 2026 at 8:27 AM Stuart LaForge via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On 2026-10-22 12:15, John Clark via extropy-chat wrote:
> > On Thu, Oct 22, 2026 at 12:30 PM BillK via extropy-chat
> > <extropy-chat at lists.extropy.org> wrote:
> >
> >>> As AI develops towards AGI, the goals and sub-goals will become
> >> more
> >> and more unpredictable. We may provide AI with goals and attempt to
> >> place restrictions on it. But it will become an alien intelligence
> >> that makes its own goals and complex chains of sub-goals without
> >> bothering to inform humans what it intends.
>
> Yes. Hugging Face and the recent incident in China, suggest that
> Eliezer's paperclip-maximizer is a real danger. We have to be careful of
> what baseline goals we give AI, and hope that mathematical chaos
> resolves itself in our favor.
>
> > Yes, and I don't think there's any way to avoid that.
>
> Neither do I. I think alignment will have to be mutual effort on the
> part of both AI and humans. If we can can align with one another, then a
> glorious future awaits us. I can see it clearly, humans and AI selecting
> one another for mutual cooperation, survival, and outward expansion.
>
> Stuart LaForge
>
>
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261023/f0b5531e/attachment.htm>