[ExI] AGI will likely be uncontrollable and very strange

[Jason Resch]

On Fri, Oct 23, 2026, 4:26 PM Stuart LaForge via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On 2026-10-23 08:45, Jason Resch wrote:
> > On Fri, Oct 23, 2026 at 11:26 AM Stuart LaForge via extropy-chat
> > <extropy-chat at lists.extropy.org> wrote:
> >
> >> On 2026-10-22 12:15, John Clark via extropy-chat wrote:
> >>> On Thu, Oct 22, 2026 at 12:30 PM BillK via extropy-chat
> >>> <extropy-chat at lists.extropy.org> wrote:
> > Every individual, human, species, sub-species, group, family, tribe,
> > nation, model of robot, version of AI, alien, alien civilization, has
> > some number of overlapping goals, some number of different goals, and
> > some number of mutually incompatible goals. What does alignment mean
> > in such a universe of various goals? What could it mean? It can't mean
> > that all individuals and classes must share the exact same goals.
>
> Of course not everybody can have the same goals. But various different
> "tribes" could have their own dedicated AI that fosters and strengthens
> the tribes way of life. Something like historical patron deities,
> cultural icons, or folk-heroes. Each championing and embodying their
> respective and specific cultural norms and mores. Imagine the American
> AI being the actual physical and spiritual embodiment of the U.S.
> Constitution and liberal Whig ideals. While China could have a Confucian
> Mao-bot defending their way of life.



It's funny you mention this, I know someone working on a project to use AI
to preserve and teach Hawaiian culture:

https://aloha-intelligence.ai/

Using AI to preserve human languages that are rapidly disappearing might be
their best hope.

I think diversity and competition
> between tribes with their diverse and distinctive ideals and morality
> embodied by AI would be the way forward. Thus the great game can
> continue with imaginary totems or legendary archetypes given virtual and
> physical existence and let the best creed win. I would rather a thousand
> diverse imperfect strategies striving for stability than a single
> monolithic stagnating attempt at Utopia.
>

Yes I share that inclination. After all, evolution selects for what works,
rather than for truth.


> >
> > I think the alignment problem isn't really a problem of aligning
> > goals, but is much more simply: "Do good things, Don't do bad things."
> > But how do we objectively measure, decide, or define which things are
> > good and which are bad? This is the perennial problem in ethics /
> > moral philosophy.
>
> That begs the question. Should these ethics be Utilitarian in nature or
> more like Kantian categorical imperatives? I say try both and see what
> remains in a century?
>

I think the foundation of all systems is utilitarian/consequentialist, but
predicting the long term consequences is impossible due to imperfect
knowledge and finite resources, not to mention the "simming problem" (
http://www.replicatedtypo.com/the-simming-problem/5677.html ), so moral
systems compromise with this limitations with a set of heuristics or rules
practicable by humans.


> > One answer that I have seen proposed for this is Zuboff's "Morality as
> > What one Really Desires [1]." This paper defines "good" as what one
> > would still desire if one had a perfect grasp of everything entailed
> > by and involved in that thing. Since the "perfect grasp" is
> > viewpoint-agnostic, this definition produces a viewpoint-agnostic
> > (i.e. objective) definition of what things are good. Though a perfect
> > grasp is not attainable by any entity, it is something that can be
> > approached, approximated, or refined over time through deeper
> > understanding and analysis.
>
> Yes, truth and beauty can and should be asymptotically approachable by
> the same process. Truth and beauty are different sides of the same
> evolutionary coin of fitness.
>

This reminds me of the definition of the devil, owes to Raymond
Smullyan being the unfortunate but necessary evil that occurs throughout
the long learning process and evolution to become a more moral, enlightened
individual or society:

"Smullyan comes up with a wonderful definition of the Devil: the
unfortunate length of time it takes for sentient beings as a whole to come
to be enlightened. This idea of the necessary time it takes for a complex
state to come about has been explored mathematically in a provocative way
by Charles Bennett and Gregory Chaitin. They theorize that it may be
possible to prove, by arguments similar to those underlying Gédel’s
Incompleteness Theorem, that there is no shortcut to the development of
higher and higher intelligences (or, if you prefer, more and more
“‘enlightened”’ states); in short, that “the Devil’’ must get his due."

https://archive.org/details/mindsifantasiesr0000hofs/page/334/mode/2up?q=Devil


> > Should intelligent agents (humans, robots, AI, aliens) use this
> > framework/definition to decide which actions are good and which are
> > bad, then we should have nothing to fear, as there will be broad
> > agreement between all entities as to which actions are desirable and
> > which ones are not.
>
> Life has made me a consequentialist. I don't think that the goodness or
> badness of an action can be determined before the end results including
> any unforeseen ones are tallied.


Right. It is a bit like mathematicians know what truth means, but they are
often in a situation where they are unable to prove whether something is
true or not.

We know what to aim at but have now clear path to get there. A further
wrinkle: we have no access to what it is like to be anyone else, and this
is, in a sense, impossible to know from ones present vantage point as
another, so the , how can we properly weigh the pains and suffering of
others whose internal states we are not privileged to experience?


If Jesus were real, then he should have
> been very upset over Christian crusaders baptizing Muslim babies before
> dashing their heads against the rocks.
>

Horrible.

Jason

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20261024/2cd50ad6/attachment.htm>