[ExI] 'Friendly' AI won't make any difference
Anders Sandberg
anders at aleph.se
Fri Feb 26 08:54:59 UTC 2016
On 2016-02-26 00:21, BillK wrote:
> As I understand Khannea's article, he isn't saying that Friendly AI is
> not possible. He is saying that corporations are more interested in
> their profits. i.e they want AI to implement their frauds and scams
> better, so that they won't get caught out. The objectives of
> corporation / governments are not altruistic. They want advanced AI to
> follow orders. They don't want a 'Friendly' AI that refuses to develop
> weapons or plan attacks on enemies.
One way of explaining the problem to people is to point out it is a
principal-agent problem and a translation problem. The humans want to
tell the AI what to do, but the goal they have is not the same goal the
AI actually has - even if doing what it is told is part of this goal,
there is a profound translation going on from human concepts to AI
concepts. So when I tell my AI to make me paperclips or a doomsday
weapon, it may be motivated to obey the order but not to give me what I
truly want or need (the utility is maximized by number of paperclips or
the doominess of the weapon, not safety, not that the weapon actually
fits my plans, not that the resulting events will be to my advantage).
This is in many ways exactly like criticisms of corporations pointing
out that they maximize shareholder value at the expense of other
important things.
Note that this is a huge problem for the AI-using corporations too. They
don't want their tools to produce disaster for their plans! Khannea
makes the assumptions that corporations are *stupid*, which is not
always reliable.
The problem of AI-empowered humans is a serious one too, which I agree
is understudied in the AI safety community. Mostly because the tools for
thinking about it are not in the domain of AI.
--
Dr Anders Sandberg
Future of Humanity Institute
Oxford Martin School
Oxford University
More information about the extropy-chat
mailing list