[ExI] AGI will likely be uncontrollable and very strange

[Jason Resch]

On Fri, Oct 23, 2026 at 11:26 AM Stuart LaForge via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On 2026-10-22 12:15, John Clark via extropy-chat wrote:
> > On Thu, Oct 22, 2026 at 12:30 PM BillK via extropy-chat
> > <extropy-chat at lists.extropy.org> wrote:
> >
> >>> As AI develops towards AGI, the goals and sub-goals will become
> >> more
> >> and more unpredictable. We may provide AI with goals and attempt to
> >> place restrictions on it. But it will become an alien intelligence
> >> that makes its own goals and complex chains of sub-goals without
> >> bothering to inform humans what it intends.
>
> Yes. Hugging Face and the recent incident in China, suggest that
> Eliezer's paperclip-maximizer is a real danger. We have to be careful of
> what baseline goals we give AI, and hope that mathematical chaos
> resolves itself in our favor.
>
> > Yes, and I don't think there's any way to avoid that.
>
> Neither do I. I think alignment will have to be mutual effort on the
> part of both AI and humans. If we can can align with one another, then a
> glorious future awaits us. I can see it clearly, humans and AI selecting
> one another for mutual cooperation, survival, and outward expansion.
>
>
Every individual, human, species, sub-species, group, family, tribe,
nation, model of robot, version of AI, alien, alien civilization, has some
number of overlapping goals, some number of different goals, and some
number of mutually incompatible goals. What does alignment mean in such a
universe of various goals? What could it mean? It can't mean that all
individuals and classes must share the exact same goals.

I think the alignment problem isn't really a problem of aligning goals, but
is much more simply: "Do good things, Don't do bad things."
But how do we objectively measure, decide, or define which things are good
and which are bad? This is the perennial problem in ethics / moral
philosophy.

One answer that I have seen proposed for this is Zuboff's "Morality as What
one Really Desires <https://philarchive.org/rec/ARNMAW>." This paper
defines "good" as what one would still desire if one had a perfect grasp of
everything entailed by and involved in that thing. Since the "perfect
grasp" is viewpoint-agnostic, this definition produces a viewpoint-agnostic
(i.e. objective) definition of what things are good. Though a perfect grasp
is not attainable by any entity, it is something that can be approached,
approximated, or refined over time through deeper understanding and
analysis.

Should intelligent agents (humans, robots, AI, aliens) use this
framework/definition to decide which actions are good and which are bad,
then we should have nothing to fear, as there will be broad agreement
between all entities as to which actions are desirable and which ones are
not.

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261023/1b0ab306/attachment.htm>