[ExI] Why “Everyone Dies” Gets AGI All Wrong by Ben Goertzel

[Keith Henson]

On Tue, Oct 14, 2025 at 4:45 PM Adam A. Ford via extropy-chat
<extropy-chat at lists.extropy.org> wrote:
>
> > " Yes, everybody agrees that AI alignment is a problem that needs to be solved.  :)

The problem cannot be solved with AIs because the problem is with
humans, not the AIs.


> And using Initial versions of AI to assist in devising alignment rules is a good idea. After all, we will be using AI to assist in designing everything else!
> I see a few problems though. The early versions of AI are likely to be aligned to fairly specific values. Say, for example, in line with the values of the richest man in the world. This is unlikely to iterate into ethical versions suitable for humanity as a whole.
> The whole alignment problem runs up against the conflicting beliefs and world views of the widely different groups of humanity.
> These are not just theoretical differences of opinion. These are fundamental conflicts, leading to wars and destruction.
> An AGI will have to be exceptionally persuasive to get all humans to agree with the final ethical system that it designs!"
>
> > I don't see any of this as a problem at all.  You just need to find a way to build and track consensus around what EVERYONE wants.  And then use a sorting algorithm which gives more vote to less rich people and stuff like that. (only a minor vote to AI systems or systems emulating dead people...?) After all, if you know what everyone wants, THAT, by definition is consensus.  And SAIs will help us know, better, what we as individuals really want and how to be just and fair with it all.
>
> Yeah, it may be the case that early AI may naturally converge or get programmed with specific and naive values (like the parochial values of the richest).. the good thing about indirect normativity done adequately, is that the richest man if wise enough may not want to risk perverse instantiations of his own parochial values.
> Conflicting beliefs are difficult - many people turn to pluralism to sort that out (i.e. Iason Gabrial).. it may be good for an input layer for value/preference/volition extraction - even if AI was able to extract what everyone wanted, there would be disagreements, blind spots, bad tradeoffs and incoherence etc, there would need to be principled approaches to resolving these issues (I'm partial to realism) - it's unlikely AI would instantly become an ideal observer level intelligence, and hence it may opt for staging scientific/epistemic/moral progress tempered along the way with corrigible humility (which may be part of the indirect normativity process).
>
> Kind regards,
>
> Adam A. Ford
>
> Science, Technology & the Future - YouTube | FB | X | LinkedIn | Bsky | MU
>
>
> On Sun, 5 Oct 2025 at 06:48, John Clark via extropy-chat <extropy-chat at lists.extropy.org> wrote:
>>
>> On Sat, Oct 4, 2025 at 3:25 PM Brent Allsop via extropy-chat <extropy-chat at lists.extropy.org> wrote:
>>
>>> > In some cases, Mother Nature (or God, or whoever you think is our creator) has miss wired our reward system (phenomenal joyes) with bad things (hurting others).  But once we learn how to do phenomenal engineering, there is no reason for any of this to be the case.  Being able to choose what you want to want, and having the ability to correct miswired rewards like this is what true freedom is. So to think that truly intelligently designed beings will have problems like this seems wrong to me, and nothing to worry about.
>>
>>
>> I think having complete control of your emotional control panel is something to worry about and I've thought so for a long time. I wrote the following to the old Cryonics Mailing List on January 19, 1994:
>>
>> "Ever want to accomplish something but have been unable to because It's difficult, well just change your goal in life to something simple and do that; better yet, flood your mind with a feeling of pride and self satisfaction and don't bother accomplishing anything at all. Think all this is a terrible idea and stupid as well , no problem, just change your mind (and I do mean CHANGE YOUR MIND) now you think it's a wonderful idea. O.K., O.K. I'm exaggerating a little, the steps would probably be smaller, at least at first, but the result would be the same. I don't have the blueprints for a Jupiter brain in my pocket but I do know that complex mechanisms don't do well in a positive feedback loop, not electronics, not animals, not people and not Jupiter brains. True, you could probably set up negative feedback of some kind to counteract it, but that would result in a decrease in happiness so would you really want to do that?"
>>
>> The explanation to the Fermi paradox may not be that extraterrestrial civilizations end in a bang or a whimper, but in a moan of orgastic pleasure. ET might be an electronic junkie.
>>
>> John K Clark
>>
>> _______________________________________________
>> extropy-chat mailing list
>> extropy-chat at lists.extropy.org
>> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat
>
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo.cgi/extropy-chat