[ExI] AI motivations
anders at aleph.se
Tue Dec 25 10:30:16 UTC 2012
On 2012-12-25 03:59, Keith Henson wrote:
> However being motivated to seek the
> good opinion of humans and it's own kind seems like a fairly safe
> fundamental and flexible motive for AIs.
> Though I could be persuaded otherwise if people have good arguments as
> to why it is not a good idea.
AI: "I have 100% good opinions about myself. Other agents have varying
opinions about me. So if I just replace all other agents with copies of
me, I will maximize my reputation."
The problem is grounding the opinions in something real. Human opinions
are partially set by evolved (and messy) social emotions: if you could
transfer those to an AI you would have solved the friendliness problem
Also, as my example shows, almost any top level goal for a utility
maximizer can lead to misbehavior. We have messy multiple goals, and
that one thing that keeps us from become obsessive sociopaths.
Future of Humanity Institute
More information about the extropy-chat