[ExI] Why “Everyone Dies” Gets AGI All Wrong by Ben Goertzel

[Ben Zaiboc]

On 04/10/2025 04:00, Keith Henson wrote:
> Uploaded humans living in private spaces don't have to agree on
> anything.  Their simulated world can be anything they like, including
> simulated slaves to beat.  Not my ideal world, but I am sure there
> will be some who want it.

Uploading only solves the problem for people who want to and can upload 
(and have control of their simulations).

I expect it would be very likely that there will be plenty of people who 
want to subjugate 'real' people, in the 'real' world, and would reject 
uploading, as well as plenty of people who are prevented from uploading 
(like everyone in all the autocratic and theocratic regimes for a start, 
so several billion).

Some kind of consensus on the 'correct' (or even acceptable) values is 
extremely unlikely.

The 'Universal declaration of human rights' is a lot less universal than 
you might think, for example. Article 2* in particular is simply false, 
if you take it as stating a fact rather than expressing an aspiration. 
Certainly many, if not most, nations disagree with it in practice. And 
apart from anything else, 'Freedom to upload' isn't in it, of course. 
I'm now wondering if even some western nations might not enact 
legislation to outlaw uploading, once it becomes possible. It wouldn't 
surprise me, there are plenty of people who regard it as tantamount to 
suicide, even among people who regard themselves as transhumanists or 
materialists. It wouldn't be hard for a government to conclude that it 
should be illegal (or that uploads don't count as human, and don't have 
any rights** at all). Look at how most governments responded to 
cryptocurrencies. If anything threatens their control, they will oppose 
it. This is just as true of liberal democracies as it is of all the 
other forms of government, the main advantage of democracies is that 
they change quicker and with less disruption than more traditional 
dictatorships.

Voluntary agreement on a single set of values to cover all humans is not 
just unlikely, it's probably impossible (just another reason why 
Yudkowsky's book is wrong).

-- 
Ben

* "Everyone is entitled to all the rights and freedoms set forth in this
  Declaration, without distinction of any kind, such as race, colour,
sex, language, religion, political or other opinion, national or social
origin, property, birth or other status. Furthermore, no distinction
shall be made on the basis of the political, jurisdictional or
international status of the country or territory to which a person
belongs, whether it be independent, trust, non-self-governing or under
any other limitation of sovereignty"


** I don't remember who it was that pointed out that your rights only extend as far as your ability to enforce them.

Ben