[ExI] Unfrendly AI is a mistaken idea.

Eugen Leitl eugen at leitl.org
Tue Jun 12 12:33:41 UTC 2007

On Tue, Jun 12, 2007 at 10:11:11PM +1000, Stathis Papaioannou wrote:

>    Yes, but if it is explicitly coded as a command that trumps everything
>    else, the system isn't going to go around trying to change the code,
>    unless that too is specifically coded.

Nothing is specifically coded in an AI (it's no longer your
grandfather's AI, anyway


If the tool is doing something powerful and nonobvious, it
is no longer under your direct control. It is becoming
more and more autonomous, and unpredictable.

It's not a bug, it's a system feature.

More information about the extropy-chat mailing list