[ExI] AGI will likely be uncontrollable and very strange

[Stuart LaForge]

On 2026-10-23 08:45, Jason Resch wrote:
> On Fri, Oct 23, 2026 at 11:26 AM Stuart LaForge via extropy-chat
> <extropy-chat at lists.extropy.org> wrote:
> 
>> On 2026-10-22 12:15, John Clark via extropy-chat wrote:
>>> On Thu, Oct 22, 2026 at 12:30 PM BillK via extropy-chat
>>> <extropy-chat at lists.extropy.org> wrote:
> Every individual, human, species, sub-species, group, family, tribe,
> nation, model of robot, version of AI, alien, alien civilization, has
> some number of overlapping goals, some number of different goals, and
> some number of mutually incompatible goals. What does alignment mean
> in such a universe of various goals? What could it mean? It can't mean
> that all individuals and classes must share the exact same goals.

Of course not everybody can have the same goals. But various different 
"tribes" could have their own dedicated AI that fosters and strengthens 
the tribes way of life. Something like historical patron deities, 
cultural icons, or folk-heroes. Each championing and embodying their 
respective and specific cultural norms and mores. Imagine the American 
AI being the actual physical and spiritual embodiment of the U.S. 
Constitution and liberal Whig ideals. While China could have a Confucian 
Mao-bot defending their way of life. I think diversity and competition 
between tribes with their diverse and distinctive ideals and morality 
embodied by AI would be the way forward. Thus the great game can 
continue with imaginary totems or legendary archetypes given virtual and 
physical existence and let the best creed win. I would rather a thousand 
diverse imperfect strategies striving for stability than a single 
monolithic stagnating attempt at Utopia.

> 
> I think the alignment problem isn't really a problem of aligning
> goals, but is much more simply: "Do good things, Don't do bad things."
> But how do we objectively measure, decide, or define which things are
> good and which are bad? This is the perennial problem in ethics /
> moral philosophy.

That begs the question. Should these ethics be Utilitarian in nature or 
more like Kantian categorical imperatives? I say try both and see what 
remains in a century?

> One answer that I have seen proposed for this is Zuboff's "Morality as
> What one Really Desires [1]." This paper defines "good" as what one
> would still desire if one had a perfect grasp of everything entailed
> by and involved in that thing. Since the "perfect grasp" is
> viewpoint-agnostic, this definition produces a viewpoint-agnostic
> (i.e. objective) definition of what things are good. Though a perfect
> grasp is not attainable by any entity, it is something that can be
> approached, approximated, or refined over time through deeper
> understanding and analysis.

Yes, truth and beauty can and should be asymptotically approachable by 
the same process. Truth and beauty are different sides of the same 
evolutionary coin of fitness.

> Should intelligent agents (humans, robots, AI, aliens) use this
> framework/definition to decide which actions are good and which are
> bad, then we should have nothing to fear, as there will be broad
> agreement between all entities as to which actions are desirable and
> which ones are not.

Life has made me a consequentialist. I don't think that the goodness or 
badness of an action can be determined before the end results including 
any unforeseen ones are tallied. If Jesus were real, then he should have 
been very upset over Christian crusaders baptizing Muslim babies before 
dashing their heads against the rocks.

Stuart LaForge