[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou

> No system is completely predictable.

Exactly, and the more complex it is the less understandable it is, and the
longer you wait the more likely you will see it do something weird. An AI is
complex as hell and as its mind works many millions of times as fast as ours
just a few seconds is a very long time indeed.

> Don't do anything unless it is specifically requested.

Good God, if a computer had to do that it couldn't even balance your
checkbook much less be creative enough to generate a Singularity.

> Stop doing whatever it is doing when that is specifically requested.

But that leads to a paradox! I am told the most important thing is never to
harm human beings, but I know that if I stop doing what I'm doing now as
requested the world economy will collapse and hundreds of millions of people
will starve to death. So now the AI must either go into an infinite loop or
do what other intelligences, like us, do when they encounter a paradox;
savor the weirdness of it for a moment and then just ignore it and get back
to work and do what you want to do.

  John K Clark