[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 04/06/07, John K Clark <jonkc at att.net> wrote:

> An AI system set up to do theoretical physics will not decide to overthrow
> > its human oppressors
>
> I'd be willing to bet your life that is untrue.


Imagine a human theoretical physicist so brilliant and so focussed that he
completely ignores the outside world to concentrate on the equations in his
head. His obsession is such that he neglects to eat or drink. Of course,
even from the point of view of continuing to do physics this isn't very
clever, because he can't work if he dies, but as this is only a meta-problem
he is not interested in it. Provided that medical teams are available to
tend to his life support, would the disinterest in the outside world and his
own survival have any negative impact on the quality of his work? And if you
were going to design a computer to be a theoretical physicist, isn't this
exactly the sort of tireless and undistracted worker that you would want?

>so that it can sit on the beach reading novels, unless it can derive this
> >desire from its initial programming.
>
> Do you also believe that the reason you ordered a jelly doe nut today
> instead of your usual chocolate one is because of your initial
> programming,
> that is, your genetic code?
>

Unless divine intervention was at play, yes. My genetic code determines my
brain configuration, which changes dynamically according to the environment
from the moment my nervous system started to form. The complexity of the
environmental interaction makes it difficult for anyone to predict exactly
what I'm going to do and similarly with an AI it would be difficult to
predict exactly what it was going to do, otherwise there would be no point
in building it. However, for the dedicated AI physicist the only uncertainty
might be what the exact scientific output is going to be. You could allow it
to explore radically different behaviours, but that would be like designing
a chess-playing program with the ability and motivation to cheat.



-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070604/f2d2c1e4/attachment.html>