[ExI] Unfrendly AI is a mistaken idea.

[Eugen Leitl]

On Fri, Jun 01, 2007 at 11:09:30AM +1000, Stathis Papaioannou wrote:
> 
>    I don't see how that's possible. How is the AI going to comandeer the
>    R&D facilities, organise manufacture of new hardware, make sure that

A few years ago a few people made the experiment of obtaining their
livelihood without leaving their room. They ordered stuff on the Internet,
and had it delivered right into their home. It worked. It would have
worked just as well if the credit card numbers were stolen.

How much hardware is there on the global network right now? You might
be surprised. How much networked hardware will be there 50, 80, 100
years from now? Most to all of it. Desktop fabs will be widespread.
Also, people would do about anything for money. Very few would resist
the temptation of a few quick megabucks on the side. 

I really see no issues breaking out of containment by remote hardware
takeover, using which to build more hardware. The old adage of
"we'll pull their plugs" has always sounded ill-informed to me. 

>    the the factories are kept supplied with components, make sure the

Of course most of the supply-chain management today is information-driven,
and many fabs are off-limit to people, because they're a major source of
contaminants.

>    component factories are supplied with raw materials, make sure the

How are component factories supplied with raw materials today?

>    mines produce the raw materials, make sure the dockworkers load the

A plant nees sunlight, water, air and trace amounts of minerals as
raw materials. A lot of what bottlenecks computational material
science is chemistry is intellectual difficulty, number of experts, 
availability of codes with adequate scaling, and computer power.
Given that it takes a 64 kNode Blue Gene/L to run a realtime
cartoon mouse, you can imagine how much hardware you need for a
human equivalent, and what else you could do with that hardware,
which will be all-purpose initially.

Use your imagination. The problem is not nearly as hard as you
think it is.

>    raw materials onto ships etc. etc. etc. etc. Perhaps I am sinning
>    against the singularity idea in saying this, but do you really think
>    it's just a matter of writing some code on a PC somewhere, which then
>    goes on to take over the world?

It's not a PC. We don't have the hardware yet, especially in small
facilities. It's not a program, not in what people write today.
 
>    It's possible that an individual human somewhere will develop a
>    superweapon, or mind-control abilities, or a viral vector that inserts

You can xerox superweapons. Pimply teenagers can run 100 kNode
botnets from their basements -- some 25% of all online systems
are compromised. I wouldn't understimate the aggregate power of
a billion petaflop game consoles on residential GBit a couple
decades from now.

>    his DNA into every living cell on the planet; it's just not very
>    likely. And why do you suppose that rapid self-improvement of the
>    world-dominating kind is more likely in an AI than in the
>    nanotechnology that has evolved naturally over billions of years? For

Because it can't do generation times in seconds. Linear biopolymers
are slow as far as information processing is concerned.

Also, AIs are just proxies for aggregated GYears of biological
evolution.

>    that matter, why do you suppose that human level intelligence has not
>    evolved before, to our knowledge, if it's so adaptive? I don't know

We're starting with human level, because we already have human level.
We don't start with cyanobacteria.

>    thwe answer to these questions, but when you look at the universe,
>    there isn't really any evidence that intelligence is as "adaptive" as
>    we might assume it to be.

We certainly managed some advances in a 50 kYrs time frame,
and without major changes to hardware.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE