[ExI] AI motivations

Anders Sandberg anders at aleph.se
Tue Dec 25 10:30:16 UTC 2012

On 2012-12-25 03:59, Keith Henson wrote:
> However being motivated to seek the
> good opinion of humans and it's own kind seems like a fairly safe
> fundamental and flexible motive for AIs.
> Though I could be persuaded otherwise if people have good arguments as
> to why it is not a good idea.

AI: "I have 100% good opinions about myself. Other agents have varying 
opinions about me. So if I just replace all other agents with copies of 
me, I will maximize my reputation."

The problem is grounding the opinions in something real. Human opinions 
are partially set by evolved (and messy) social emotions: if you could 
transfer those to an AI you would have solved the friendliness problem 
quite literally.

Also, as my example shows, almost any top level goal for a utility 
maximizer can lead to misbehavior. We have messy multiple goals, and 
that one thing that keeps us from become obsessive sociopaths.

Anders Sandberg
Future of Humanity Institute
Oxford University

More information about the extropy-chat mailing list