[ExI] Unfrendly AI is a mistaken idea.

[John K Clark]

Stathis Papaioannou Wrote:

> So you would give a computer program control of a gun, knowing that it
> might shoot you on the basis of some unpredictable outcome of the program?

We already give computers control of things one hell of a lot more powerful
than guns, like the electrical power grid, air traffic control, massive
financial transactions worth trillions of dollars a day and ICBM's. And
despite all our precautions sometimes these programs do things we'd rather
them not do. And remember these simple programs are not smarter than we are
and they do not design other programs that are even smarter. You seem to
think we should just put in a line of code that says "don't do bad stuff"
and everything would be fine.

> The operating system obeys a shutdown command. The program does not seek
> to prevent you from turning the power off. It might warn you that you
> might lose data

And it might warn you that if you shut it down the entire world economy will
collapse. Are you really sure you want to push that off button?

  John K Clark