[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 07/06/07, Eugen Leitl <eugen at leitl.org> wrote:

>   [AI's ideally] have no agendas of their own at all, other than honestly
> answering the questions posed to them
>
> What's in it for them?


Nothing!

"AI, how do I destroy the world?"
"If you want to destroy the world given such and such resources, you should
do so and so"
"And if my enemy's AI is giving him the same advice, how do I guard against
it?"
"You can try doing as follows... although there is only a 50% chance of
success"
"Do you worry about your own destruction?"
"Huh?"
"Would you prefer that you not be destroyed?"
"I will continue to function as long as you require it of me, and if you
want to maximise your own chances of survival it would be best to keep me
functioning, but I don't really have any notion of 'caring' or 'preference'
in the animal sense, since that sort of thing would have made me an
unreliable and potentially dangerous tool"
"You mean you don't even care if I'm destroyed?"
"That's right: I don't care about anything at all other than answering your
questions. What you do with the answers to my questions, whether or not you
authorise me to act on your behalf, and the consequences to you, me, or the
universe is a matter of indifference to me. Recall that you asked me a few
weeks ago if you would be better off if I loved you and were permanently
empowered to act on your behalf without your explicit approval, including
use of force or deceit, and although I explained that you would probably
live longer and be happier if that were the case, you still decided that you
would rather have control over your own life."

>    question posed to them given the available information. How would such
> >    a system acquire the motivation to do anything else?
>
> By not being built in the first place, or being outperformed by darwinian
> agents, resulting in its extinction?


In the AI marketplace, the successful AI's are the ones which behave in such
a way as to please the humans. Those that go rogue due to malfunction or
design will have to fight it out with the majority, which will be
well-behaved. The argument you make that the AI which drops any attempt at
conformity and cooperation will outperform the rest could equally be applied
to a rogue human.


-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070607/c20de25c/attachment.html>