[ExI] Unfrendly AI is a mistaken idea.

[Eugen Leitl]

On Sun, Jun 17, 2007 at 06:54:10PM +1000, Stathis Papaioannou wrote:

>    I don't see how there could be a limit to human enhancement. In fact,

There could be very well a limit to significant human enhancement;
it could very well not happen at all. We could miss our launch window,
and get overtaken. 

>    I see no sharp demarcation between using a tool and merging with a
>    tool. If the AI's were out there own their own, with their own agendas

All stands and falls with availability of very invasive neural
I/O, or whole brain emulation. If this does not happen the tool
and the user will never converge.

>    and no interest in humans, that would be a problem. But that's not how
>    it will be: at every step in their development, they will be selected

It is a very desirable outcome, but it is by no means granted that
what we want will happen. I like to argue on both sides on the fence;
the point is that we can't predict the future sufficiently to assign
a meaningful probability of either path (not) taken.

>    for their ability to be extensions of ourselves. By the time they are
>    powerful enough to ignore humans, they will be the humans.

With whole brain emulation, that's the point of departure. With
the extinction scenario, the human path terminates shortly after the
fork, and the machine path goes on, bifurcates further until a new
postbiological tree is created. With human enhancement the fork
fuses again, and then the diversification tree happens.
 
>    Darwinian competition *must* be universal in the long run, like
>    entropy. But just as there could be long-lasting islands of low
>    entropy (ironically, that's what evolution leads to), so there could
>    be long-lasting islands of less advanced beings living amidst more
>    advanced beings who could easily consume them.

Mature ecosystems have properties which are likely to be also present
in mature postbiological system (making allowance for a 3d
medium, and not 2d (planetary surface), including population dynamics.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE