[ExI] Should we teach Strong AI Ethics?

[Anders Sandberg]

On 2016-05-26 17:18, BillK wrote:
> <http://www.smbc-comics.com/index.php?id=4122>
>
> Serious point though.
> If we teach AI about ethical behaviour (for our own safety) what do we
> expect the AI to do when it sees humans behaving unethically (to a
> greater or lesser extent)?
>
> Can a totally ethical AI even operate successfully among humans?

What is "totally ethical"?

[Philosopher hat on!]

Normally when we say something like that, we mean somebody who follows 
the One True moral system perfectly. Or at least one moral system 
perfectly. There are no humans that do it, so we do not have reliable 
intuitions about what it would mean. Now, a caricature view  of moral 
perfection is somebody being a saintly wuss: super kind, but exploitable 
by imperfect and nasty actors.

But there is no reason to think this is the only choice. You could 
imagine a morally perfect Objectivist, following rules of enlightened 
selfishness. Or a perfect average utilitarian maximizing the average 
happiness of all entities in our future lightcone. Neither would be a 
pushover ("If I give you my wallet there will be less resources for my 
von Neumann probe program. So, no, I will not give it to you. In fact, I 
will now force you to give me your money - I see that this will enable a 
further quintillion minds. Thank you.") Convergent instrumental goal 
behavior likely tends to turn wussy nice agents non-wussy.

There is an interesting issue about what to do with imperfect moral 
agents if you are a perfect one. A Kantian agent would presumably 
respect their autonomy and try to guide them to see how to obey the 
categorical imperative. A consequentialist agent would try to manipulate 
them to behave better, but the means might be anything from incentives 
to persuation to brainwashing. A virtue agent might not care at all, 
just demonstrating its own excellence. A paperclip maximizing agent 
would find non-paperclip maximizers a waste of resources and work to 
remove them.

In fact, most pure moral systems are very bad at "live and let live". We 
humans tend to de facto behave like that because our power is about 
equal; entities that are orders of magnitude more powerful may not 
behave like that unless we get the value code just right.

-- 
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20160526/3d37f4bc/attachment.html>