[ExI] ethics vs intelligence

[Anders Sandberg]

On 12/09/2012 14:40, Ben Zaiboc wrote:
> Woh. Hang on, why do you conclude that 'ethical' equates to 'friendly 
> to humans'?

Exactly. Suppose pleasure hedonism is actually the one true moral system 
- any sufficiently smart entity will figure it out. In that case the 
best possible state of the universe would be with as much mass as 
possible converted to computronium running the maximally pleasant 
mindstate over and over again. It is very unlikely this is compatible 
with any human existence, since our atoms could be used for 
computronium. It would (given the assumption) actually be what we humans 
*should* aim for if we were smart enough to see it, but it is unlikely 
that most of us would like to do the moral thing.


> Whatever you happen to be, your ethics must be grounded in what you 
> are. There can't be any such thing as an objective moral code, and you 
> can't derive a morality based on something that you are not

Some ethicists would beg to differ. To take Kant as an example, he would 
argue that all rational moral agents would converge on the same ethics 
(his) by virtue of being rational moral agents. He would predict that 
aliens, uplifted animals and AIs would all come to the same conclusion 
if they thought about it long enough. They are all the same, from this 
perspective, and the moral code is not subjective.

Yes, it is possible to construct AIs that realize Kant is right and 
still do not want to follow his system:
http://www.aleph.se/andart/archives/2011/02/why_we_should_fear_the_paperclipper.html
But Kant would argue that they fail at being rational moral agents - 
they cannot change their motivations.

It is worth noting that professional philosophers are 56.3% for moral 
realism (there are true moral facts, independent of subjective opinion). 
This remains true when sampling meta-ethicists or normative ethicists:
http://philpapers.org/surveys/results.pl
Whether ethicists actually *know* anything about the subject is of 
course the basis of the cognitivism-noncognitivism battle. But at least 
they have read and thought a lot about it.


-- 
Anders Sandberg,
Future of Humanity Institute
Oxford Martin School
Faculty of Philosophy
Oxford University