[ExI] Unfrendly AI is a mistaken idea.

[Stathis Papaioannou]

On 12/06/07, John K Clark <jonkc at att.net> wrote:
>
> Stathis Papaioannou Wrote:
>
> > There won't be an issue if every other AI researcher has the most basic
> > desire for self-preservation.
>
> I wouldn't take such precautions because I believe them to be futile and
> immoral, am I really that unusual?


So you would give a computer program control of a gun, knowing that it might
shoot you on the basis of some unpredictable outcome of the program?

> If the AI's top level goal is to remain your slave, then it won't by
> > definition want to change that top level goal.
>
> Gee, I can't understand why today's programmers whiting operating systems
> don't just put in a top level goal saying don't let their machines be
> taken
> over by hostile programs. Computer security problem solved!


The operating system obeys a shutdown command. The program does not seek to
prevent you from turning the power off. It might warn you that you might
lose data, but it doesn't get excited and try to talk you out of shutting it
down and there is no reason to suppose that it would do so if it were more
complex and self-aware,  just because it is more complex and self-aware. Not
being shut down is just one of many possible goals/ values/ motivations/
axioms, and there is no a priori reason why the program should value one
over another.

> do you think even God almighty could convince you by argument alone
> > that 2 + 2 = 5?
>
> No of course not, because 2 +2 is in fact equal to 2 and I can prove it:
>
> Let A = B
>
> Multiply both sides by A and you have
>
> A^2 = A*B
>
> Now add A^2 -2*a*B to both sides
>
> A^2 + A^2 -2*a*B = A*B + A^2 -2*A*B
>
> Using basic algebra  this can be simplified to
>
> 2*( A^2 -A*B) = A^2 -A*B
>
> Now just divide both sides by A^2 -A*B and we get
>
> 2 = 1
>
> Thus 2 +2 = 1 + 1 = 2
>

This example just illustrates the point: even someone who cannot point out
the problem with the proof (division by zero) knows that it must be wrong
and would not be convinced, no matter how smart the entity purporting to
demonstrate this is.


-- 
Stathis Papaioannou
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.extropy.org/pipermail/extropy-chat/attachments/20070612/c97c1397/attachment.html>