[extropy-chat] Two draft papers: AI and existential risk; heuristics and biases

[Robin Hanson]

At 12:33 PM 6/4/2006, Eliezer S. Yudkowsky wrote:
>These are drafts of my chapters for Nick Bostrom's forthcoming edited
>volume _Global Catastrophic Risks_.
>_Cognitive biases potentially affecting judgment of global risks_
>    http://singinst.org/Biases.pdf
>An introduction to the field of heuristics and biases ...
>_Artificial Intelligence and Global Risk_
>    http://singinst.org/AIRisk.pdf
>The new standard introductory material on Friendly AI.

The chapter on cognitive biases was excellent.   Regarding the other 
chapter, while you seem to have thought lots about many related 
issues over the years, you don't seem to have worked much on the 
issue I get stuck on: the idea that a single relatively isolated AI 
system could suddenly change from negligible to overwhelmingly powerful.

You warn repeatedly about how easy is is to fool oneself into 
thinking one understands AI, and you want readers to apply this to 
their intuitions about the goals an AI may have.   But you seem to be 
relying almost entirely on unarticulated intuitions when you conclude 
that very large and rapid improvement of isolated AIs is likely.

You say that humans today and natural selection do not self-improve 
in the "strong sense" because humans "haven't rewritten the human 
brain," "its limbic core, its cerebral cortex, its prefrontal 
self-models" and natural selection has not "rearchitected" "the 
process of mutation and recombination and selection," with "its focus 
on allele frequencies" while an AI "could rewrite its code from 
scratch."  And that is pretty much the full extent of your relevant argument.

This argument seems to me to need a whole lot of elaboration and 
clarification to be persuasive, if it is to go beyond the mere 
logical possibility of rapid self-improvement.   The code of an AI is 
just one part of a larger system that would allow an AI to 
self-improve, just as the genetic code is a self-modifiable part of 
the larger system of natural selection, and human culture and beliefs 
are a self-modifiable part of human improvement today.

In principle every part of each system could be self-modified, while 
in practice some parts are harder to modify than others.  Perhaps 
there are concepts and principles which could help us to understand 
why the relative ease of self-modification of the parts of the AI 
improvement process are importantly different that in these other 
cases.   But you do not seem to have yet articulated any such 
concepts or principles.

A standard abstraction seems useful to me:  when knowledge 
accumulates in many small compatible representations, growth is in 
the largest system that can share such representations.   Since DNA 
is sharable mainly within a species, the improvements that any one 
small family of members can produce are usually small compared to the 
improvements transferred by sex within the species.  Since humans 
share their knowledge via language and copying practices, the 
improvements that a small group of people can make are small compared 
to the improvements transferred from others, and made available by 
trading with those others.

The obvious question about a single AI is why its improvements could 
not with the usual ease be transferred to other AIs or humans, or 
made available via trades with those others.   If so, this single AI 
would just be part of our larger system of self-improvement.   The 
scenario of rapid isolated self-improvement would seem to be where 
the AI found a new system of self-improvement, where knowledge 
production was far more effective, *and* where internal sharing of 
knowledge was vastly easier than external sharing.

While this is logically possible, I do not yet see a reason to think 
it likely.   Today a single human can share the ideas within his own 
head far easier than he can share those ideas with others - 
communication with other people is far more expensive and 
error-prone.   Yet the rate at which a single human can innovate is 
so small relative to the larger economy that most innovation comes 
from ideas shared across people.  So a modest advantage for the AI's 
internal sharing would not be enough - the advantage would have to be 
enormous.




Robin Hanson  rhanson at gmu.edu  http://hanson.gmu.edu
Associate Professor of Economics, George Mason University
MSN 1D3, Carow Hall, Fairfax VA 22030-4444
703-993-2326  FAX: 703-993-2323