[extropy-chat] Prime Directive

[Anders Sandberg]

A B wrote:
>   I agree completely with what George Dvorsky wrote. I would also add that
> it is only through consciousness that pleasure is possible (which most
> beings would choose to maximize), and alternatively, it is only through
> consciousness that suffering is possible (which most beings would choose
> to minimize). That alone makes consciousness pretty darned important, in
> my book.

Well, what is so important about pleasure and suffering then? If we say
they are important because we have evolved that way, and that pleasure and
suffering are signals about imminent potential or risk for our fitness,
then it seems that we should try to maximize fitness instead of
consciousness or pleasure, or minimizing pain. If they are important
because they are subjective states of attraction/repulsion, then again it
begs the question why we would aim for a world where we follow our
subjective attractions and lack any repulsions.

I'm not denying that pleasure is a good thing, it is just that I have so
far never seen a really convincing justification of it.

> Another issue that's going to throw a wrench into
> the engine, is that before too long we will be able to technologically
> uplift the original non-rational beings to super-rational levels  -  the
> undeveloped *potential* of a conscious being will also become a major
> ethical issue.

Again, why is the potential important? There is undeveloped potential for
great art in blocks of marble, but I don't think it is a crime that
artists don't free it as much as they can.

>   "I wonder if we really would end up with a mostly unpleasant (it can't
> really become unjust in this case) universe if we acknowledged physics as
> the underlying basis for morality."...
>
>   I think you can be assured of that outcome. Just take a quick survey of
> what physics/Darwinian selection has given us so far - an extremely
> top-heavy ratio of total suffering : to total pleasure.

Acknowledging hydrodynamics as the basis for your engineering project
doesn't mean that you have to make it behave like water does in nature. A
computer built by fluidistors is utterly un-lakelike but yet based on the
same principles and material.

My point was that starting from physics and empirical observations we can
rationally construct ethical systems that work practically. Basing
morality on non-physics means basing it on non-reality. (It should be
noted that I regard logical reasoning and concepts as being physical
processes, rather than some kind of platonic abstractions. They are as
real as language and culture, patterns embedded in our physical neural
substrate).

>   "All the entities who would be badly off in the might makes right world
> would hence band together to ensure that it would not occur (e.g. by
> resisting the strong
> or by making it irrational to pursue that world because they would
> irrevocably blow up a doomsday device if it did occur, etc)."
>
>   That strategy might have worked in 16th century Earth, for example. But
> in the age of exponential recursive self-improvement (and other rapid
> bootstrapping techniques), methinks it ain't gonna happen.

Because the natural power distribution would be highly uneven? Assuming
this is true, it does not follow that it would be rational for the most
powerful entities to use their powers in a might makes right fashion. It
might be possible, it might be tempting, but it might not be a logical
necessity. I think a lot of game theory and economics actually suggests
the reverse. But this is always going to be scenario-dependent.

-- 
Anders Sandberg,
Oxford Uehiro Centre for Practical Ethics
Philosophy Faculty of Oxford University