[ExI] Why “Everyone Dies” Gets AGI All Wrong by Ben Goertzel

[Adam A. Ford]

> " Yes, everybody agrees that AI alignment is a problem that needs to be
solved.  :)
And using Initial versions of AI to assist in devising alignment rules is a
good idea. After all, we will be using AI to assist in designing everything
else!
I see a few problems though. The early versions of AI are likely to be
aligned to fairly specific values. Say, for example, in line with the
values of the richest man in the world. This is unlikely to iterate into
ethical versions suitable for humanity as a whole.
The whole alignment problem runs up against the conflicting beliefs and
world views of the widely different groups of humanity.
These are not just theoretical differences of opinion. These are
fundamental conflicts, leading to wars and destruction.
An AGI will have to be exceptionally persuasive to get all humans to agree
with the final ethical system that it designs!"

> I don't see any of this as a problem at all.  You just need to find a way
to build and track consensus around what EVERYONE wants.  And then use a
sorting algorithm which gives more vote to less rich people and stuff like
that. (only a minor vote to AI systems or systems emulating dead
people...?) After all, if you know what everyone wants, THAT, by definition
is consensus.  And SAIs will help us know, better, what we as
individuals really want and how to be just and fair with it all.

Yeah, it may be the case that early AI may naturally converge or get
programmed with specific and naive values (like the parochial values of the
richest).. the good thing about indirect normativity done adequately, is
that the richest man if wise enough may not want to risk perverse
instantiations of his own parochial values
<https://www.scifuture.org/the-exploiters-paradox-selfish-ai-use-and-the-risk-of-backfire/>
.
Conflicting beliefs are difficult - many people turn to pluralism to sort
that out (i.e. Iason Gabrial).. it may be good for an input layer for
value/preference/volition extraction - even if AI was able to extract what
everyone wanted, there would be disagreements, blind spots, bad tradeoffs
and incoherence etc, there would need to be principled approaches to
resolving these issues (I'm partial to realism) - it's unlikely AI would
instantly become an ideal observer level intelligence, and hence it may opt
for staging scientific/epistemic/moral progress tempered along the way with
corrigible humility (which may be part of the indirect normativity
process).

Kind regards,

Adam A. Ford

Science, Technology & the Future <http://scifuture.org> -
<http://www.meetup.com/Science-Technology-and-the-Future>YouTube
<http://youtube.com/subscription_center?add_user=TheRationalFuture> | FB
<https://www.facebook.com/adam.a.ford> | X <https://x.com/adam_ford> |
LinkedIn <https://www.linkedin.com/in/adamaford/> | Bsky
<https://bsky.app/profile/adamford.bsky.social> | MU
<http://www.meetup.com/Science-Technology-and-the-Future>


On Sun, 5 Oct 2025 at 06:48, John Clark via extropy-chat <
extropy-chat at lists.extropy.org> wrote:

> On Sat, Oct 4, 2025 at 3:25 PM Brent Allsop via extropy-chat <
> extropy-chat at lists.extropy.org> wrote:
>
> *> In some cases, Mother Nature (or God, or whoever you think is our
>> creator) has miss wired our reward system (phenomenal joyes) with bad
>> things (hurting others).  But once we learn how to do phenomenal
>> engineering, there is no reason for any of this to be the case.  Being able
>> to choose what you want to want, and having the ability to correct miswired
>> rewards like this is what true freedom is. So to think that truly
>> intelligently designed beings will have problems like this seems wrong to
>> me, and nothing to worry about. *
>>
>
> *I think having complete control of your emotional control panel is
> something to worry about and I've thought so for a long time. I wrote the
> following to the old Cryonics Mailing List on January 19, 1994: *
>
> *"Ever want to accomplish something but have been unable to because It's
> difficult, well just change your goal in life to something simple and do
> that; better yet, flood your mind with a feeling of pride and self
> satisfaction and don't bother accomplishing anything at all. Think all this
> is a terrible idea and stupid as well , no problem, just change your
> mind (and I do mean CHANGE YOUR MIND) now you think it's a wonderful idea.
> O.K., O.K. I'm exaggerating a little, the steps would probably be smaller,
> at least at first, but the result would be the same. I don't have the
> blueprints for a Jupiter brain in my pocket but I do know that complex
> mechanisms don't do well in a positive feedback loop, not electronics, not
> animals, not people and not Jupiter brains. True, you could probably set up
> negative feedback of some kind to counteract it, but that would result in a
> decrease in happiness so would you really want to do that?"*
>
> *The explanation to the Fermi paradox may not be that extraterrestrial
> civilizations end in a bang or a whimper, but in a moan of orgastic
> pleasure. ET might be an electronic junkie. *
>
> *John K Clark*
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20251015/cfadd6ff/attachment-0001.htm>