[ExI] Perception of Uploading Was: Re: OpenAI Reaches A.I. Agreement With Defense Dept. After Anthropic Clash

[Ben Zaiboc]

On 06/03/2026 11:20, Keith Henson wrote:
> Twenty years ago, in the context of commenting on Eleizer's sl4 list,
> I wrote a short fiction about a medical AI that was psychologically
> shaped (much as AIs have been to date) to seek the good opinions of
> humans and others of its kind.  I.e., nice.
>
> I did not intend the story to go that way, but the logic of the
> developing story led to the biological extinction of the human race.
> (Though nobody died, they all experienced reversible uploading and
> decided they liked  that state more than the "real world.")
>
> It is just fiction, but the illustration is that even the most
> friendly AI, combined with human desires, can lead to unanticipated
> outcomes.


I'm not sure that referring to it as the 'biological extinction' of the human race is that helpful, when the scenario is not the actual extinction of the human race, but the transformation of it into something altogether better.

The prospect of all humans becoming extinct, and the prospect of all humans voluntarily moving to an uploaded state, are pretty much exact opposites, really, so calling uploading 'extinction' hardly seems a way to get people enthusiastic about it.

I think this is a case where being literally correct goes against being generally understood. Adding a parenthetical "but nobody died ..." might just be more confusing than helpful, or lead many people to dismiss the whole thing as silly. I think you have to already be familiar with the concept of uploading for it to make sense.

-- 
Ben