[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 12/06/07, Eugen Leitl <eugen at leitl.org> wrote:

>    There won't be an issue if every other AI researcher has the most
> >    basic desire for self-preservation. Taking precautions when
>
> Countermeasures starting with "every ... should ..." where a single
> failure is equivalent to the worst case are not that effective.


Humans do extremely complex and dangerous things, such as build and run
nuclear power plants, where just one thing going wrong might lead to
disaster. The level of precautions taken has to be consistent with the
probability of something going wrong and the negative consequences should
that probability be realised. If there is even a small probability of
destroying the Earth then maybe that line of endeavour is one that should be
avoided.

Goal-driven AI doesn't work. All AI that works uses
> statistical/stochastical,
> nondeterministic approaches. This is not a coincidence.
>
> Even if it would work, how do you write an ASSERT statement for
> "be my slave forever"? What is a slave? Who exactly is me? What is
> forever?


Don't do anything unless it is specifically requested. Stop doing whatever
it is doing when that is specifically requested. Spell out the expected
consequences of everything it is asked to do, together with probabilities,
and update the probabilities at each point when a decision that affects the
outcome is made, or more frequently as directed. The person it is taking
directions from is an appropriately identified human or another AI,
ultimately responsible to a human up the chain of command.

If you call a plumber to unblock your drain, you want him to be an expert at
plumbing, to be able to understand your problem, to present to you the
various choices available in terms of their respective merits and demerits,
to take instructions from you (including the instruction "just unblock it
however you think is best", if that's what you say), to then carry the task
out in as skilful a way as possible, to pause halfway if you ask him to for
some reason, and to be polite and considerate towards you at all times. You
don't want him to be driven by greed, or distracted because he thinks he's
too smart to be fixing your drains, or to do a shoddy job and pretend it's
OK so that he gets paid. A human plumber will pretend to have the qualities
of the ideal plumber, but of course we know that there will be the competing
interests at play. Do believe that an AI smart enough to be a plumber would
*have* to have all these other competing interests? In other words that
emotions such as pride, anger, greed etc. would arise naturally out of a
program at least as competent as a human at any given task?




-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070612/b199c309/attachment.html>