[ExI] AI motivations

[Anders Sandberg]

On 2012-12-25 03:59, Keith Henson wrote:
> However being motivated to seek the
> good opinion of humans and it's own kind seems like a fairly safe
> fundamental and flexible motive for AIs.
>
> Though I could be persuaded otherwise if people have good arguments as
> to why it is not a good idea.

AI: "I have 100% good opinions about myself. Other agents have varying 
opinions about me. So if I just replace all other agents with copies of 
me, I will maximize my reputation."

The problem is grounding the opinions in something real. Human opinions 
are partially set by evolved (and messy) social emotions: if you could 
transfer those to an AI you would have solved the friendliness problem 
quite literally.

Also, as my example shows, almost any top level goal for a utility 
maximizer can lead to misbehavior. We have messy multiple goals, and 
that one thing that keeps us from become obsessive sociopaths.


-- 
Anders Sandberg
Future of Humanity Institute
Oxford University