[ExI] Unfrendly AI is a mistaken idea.

[Eugen Leitl]

On Fri, Jun 01, 2007 at 10:06:15PM +1000, Stathis Papaioannou wrote:

>    We don't have human level AI, but we have lots of dumb AI. In nature,

There is a qualitative difference between human-designed AI, and
naturally evolved AI. Former will never go anywhere. Because of this
extrapolations from pocket calculators and chess computers to 
robustly intelligent (even insects can be that) systems are invalid.

>    dumb organisms are no less inclined to try to take over than smarter
>    organisms (and no less capable of succeeding, as a general rule, but
>    leave that point for the sake of argument). Given that dumb AI doesn't

Yes, pocket calculators are not known for trying to take over the world.

>    try to take over, why should smart AI be more inclined to do so? And

It doesn't have to  be smart, it does have to be able to survive in
its native habitat, be it the global network, or the ecosystem. We don't
have such systems yet.

>    why should that segment of smart AI which might try to do so, whether
>    spontaneously or by malicious design, be more successful than all the

There is no other AI. There is no AI at all.

>    other AI, which maintains its ancestral motivation to work and improve

I don't see how there could be a domain-specific AI which specializes
in self-improvement. 

>    itself for humans just as humans maintain their ancestral motivation

How do you know you're working for humans? What is a human, precisely?
If I'm no longer fitting the description, how do I upgrade that description,
and what is preventing anyone else from that?

>    to survive and multiply?

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE