[ExI] Unfrendly AI is a mistaken idea.

[Eugen Leitl]

On Tue, Jun 12, 2007 at 10:11:11PM +1000, Stathis Papaioannou wrote:

>    Yes, but if it is explicitly coded as a command that trumps everything
>    else, the system isn't going to go around trying to change the code,
>    unless that too is specifically coded.

Nothing is specifically coded in an AI (it's no longer your
grandfather's AI, anyway

http://www.amazon.de/Probabilistic-Robotics-Intelligent-Autonomous-Agents/dp/0262201623
http://www.amazon.de/Principles-Robot-Motion-Implementations-Implementation/dp/0262033275/
http://www.amazon.de/Autonomous-Robots-Inspiration-Implementation-Intelligent/dp/0262025787/

If the tool is doing something powerful and nonobvious, it
is no longer under your direct control. It is becoming
more and more autonomous, and unpredictable.

It's not a bug, it's a system feature.