[ExI] 'Friendly' AI won't make any difference

[Anders Sandberg]

On 2016-02-26 00:21, BillK wrote:
> As I understand Khannea's article, he isn't saying that Friendly AI is 
> not possible. He is saying that corporations are more interested in 
> their profits. i.e they want AI to implement their frauds and scams 
> better, so that they won't get caught out. The objectives of 
> corporation / governments are not altruistic. They want advanced AI to 
> follow orders. They don't want a 'Friendly' AI that refuses to develop 
> weapons or plan attacks on enemies.

One way of explaining the problem to people is to point out it is a 
principal-agent problem and a translation problem. The humans want to 
tell the AI what to do, but the goal they have is not the same goal the 
AI actually has - even if doing what it is told is part of this goal, 
there is a profound translation going on from human concepts to AI 
concepts. So when I tell my AI to make me paperclips or a doomsday 
weapon, it may be motivated to obey the order but not to give me what I 
truly want or need (the utility is maximized by number of paperclips or 
the doominess of the weapon, not safety, not that the weapon actually 
fits my plans, not that the resulting events will be to my advantage). 
This is in many ways exactly like criticisms of corporations pointing 
out that they maximize shareholder value at the expense of other 
important things.

Note that this is a huge problem for the AI-using corporations too. They 
don't want their tools to produce disaster for their plans! Khannea 
makes the assumptions that corporations are *stupid*, which is not 
always reliable.


The problem of AI-empowered humans is a serious one too, which I agree 
is understudied in the AI safety community. Mostly because the tools for 
thinking about it are not in the domain of AI.

-- 
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University