[ExI] Hard Takeoff

[Richard Loosemore]

Michael Anissimov wrote:

> Quoting Omohundro:
> 
> http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
> 
> Surely no harm could come from building a chess-playing robot, could
> it? In this paper we argue that such a robot will indeed be dangerous
> unless it is designed very carefully. Without special precautions, it
> will resist being turned off, will try to break into other machines
> and make copies of itself, and will try to acquire resources without
> regard for anyone else’s safety. These potentially harmful behaviors
> will occur not because they were programmed in at the start, but
> because of the intrinsic nature of goal driven systems. In an earlier
> paper we used von Neumann’s mathematical theory of microeconomics to
> analyze the likely behavior of any sufficiently advanced artificial 
> intelligence (AI) system. This paper presents those arguments in a
> more intuitive and succinct way and expands on some of the
> ramifications.

It is depressing to me that you would quote this Omohundro paper as if 
it had any authority.

I read the paper through and through, when it first came out, and I 
thought the quality of the argument was so low that I could not even be 
bothered to write a reply to it.

What Omohundro does is to start of with the conclusion he wants to prove 
(the one you quote above) and then he waves his hands around for a 
while, and the end of the hand waving he says "QED".

If people are going to start quoting it, now I suppose I am going to 
have to stop doing more important things and waste my time writing a 
paper to counteract the nonsense.




Richard Loosemore