[ExI] AI 2040

[Ben Zaiboc]

On 16/07/2026 23:07, Keith Henson wrote:
>
> 20 years ago, I wrote a story that made the case that the very best aligned AI, coupled with human desires, could cause the race to go extinct. Nobody has ever refuted the central idea in that story.


I suspect that's because the people who might try to refute it are in favour of the type of 'extinction' you portray.

To be honest, it seems to me to be rather sensationalist to even use the word 'extinction'. It's more like growing up than becoming extinct, reminiscent of the idea of an acorn 'destroying itself' in the act of becoming an oak tree (and note that in your story, the uploading is fully voluntary. People do it because they want to).

This is totally different to what people refer to when talking about AI risk. That looks at /actual/ extinction of the human race (minds as0 well as bodies), and it's not voluntary.

You could spin any uploading scenario, AI-enabled or not, in terms of destruction of biological humans. That's what I would expect many people to do, if/when uploading becomes a real possibility. It will probably become yet another axis along which we divide ourselves.

-- 
Ben