[ExI] the ethics of the Vile Offspring

[Anders Sandberg]

Stefano Vaj wrote:
> > I suspect Stross got the idea partially from Nick Bostrom's paper 
> "The future of human evolution" where he discusses scenarios where 
> posthumanity evolves into something that completely lacks whatever it 
> is that actually gives existence value, for example a very capable and 
> expansive civilization where there is no consciousness.
>
> Yes, so do I. In fact, I was simply flabbergasted when I was first 
> exposed to Bostrom's inclination to justify what is exactly the 
> argument against trans-simianism by resorting to qualia.

He is not basing it on qualia, he is using qualia as an example. Maybe 
the real value resides in something else, but his point still stands: 
the kind of evolution we might engage in in the future might push us 
away from whatever the real value is.

If we think evolution has so far pushed us in the direction of value it 
does not follow that future new kinds of evolution will continue to push 
in the right way.

It might of course be that we now, being better aware of value, can push 
our evolution in an even more desireable direction. But given past 
experience with human planning and coordination ability for complex 
social systems as well as our wide spread of opinion on what is 
valuable, this does not look guaranteed in any way. In fact, such 
projects might develop their own accidental dynamics pushing in the 
wrong way (like many of our social institutions do).

The problem is not the overman, the problem is that it might be rational 
for everybody to become something that in the aggregate or individually 
has lower value.

Monkeys might disagree with us about what is really valuable in life and 
might think that we have evolved in a direction that produces less 
value. They could just be wrong about that, since we have sources of 
value they lack access to, like science, philosophy and culture. But 
that doesn't mean some forms of future evolution can't be value-eroding. 
Robin's paper on the cosmic commons points out one nasty possibility 
(turning into an interstellar locust swarm) that might be hard to avoid 
without rather serious coordination (perhaps necessitating other, 
dangerous social institutions).

-- 
Anders Sandberg,
Future of Humanity Institute 
James Martin 21st Century School 
Philosophy Faculty 
Oxford University