[extropy-chat] Two draft papers: AI and existential risk; heuristics and biases

[Mark Walker]

Robin Hanson wrote:


  Regarding the other
> chapter, while you seem to have thought lots about many related
> issues over the years, you don't seem to have worked much on the
> issue I get stuck on: the idea that a single relatively isolated AI
> system could suddenly change from negligible to overwhelmingly powerful.
>
> You warn repeatedly about how easy is is to fool oneself into
> thinking one understands AI, and you want readers to apply this to
> their intuitions about the goals an AI may have.   But you seem to be
> relying almost entirely on unarticulated intuitions when you conclude
> that very large and rapid improvement of isolated AIs is likely.
>
> You say that humans today and natural selection do not self-improve
> in the "strong sense" because humans "haven't rewritten the human
> brain," "its limbic core, its cerebral cortex, its prefrontal
> self-models" and natural selection has not "rearchitected" "the
> process of mutation and recombination and selection," with "its focus
> on allele frequencies" while an AI "could rewrite its code from
> scratch."  And that is pretty much the full extent of your relevant 
> argument.
>
> This argument seems to me to need a whole lot of elaboration and
> clarification to be persuasive, if it is to go beyond the mere
> logical possibility of rapid self-improvement.   The code of an AI is
> just one part of a larger system that would allow an AI to
> self-improve, just as the genetic code is a self-modifiable part of
> the larger system of natural selection, and human culture and beliefs
> are a self-modifiable part of human improvement today.
>
> In principle every part of each system could be self-modified, while
> in practice some parts are harder to modify than others.  Perhaps
> there are concepts and principles which could help us to understand
> why the relative ease of self-modification of the parts of the AI
> improvement process are importantly different that in these other
> cases.   But you do not seem to have yet articulated any such
> concepts or principles.
>
> A standard abstraction seems useful to me:  when knowledge
> accumulates in many small compatible representations, growth is in
> the largest system that can share such representations.   Since DNA
> is sharable mainly within a species, the improvements that any one
> small family of members can produce are usually small compared to the
> improvements transferred by sex within the species.  Since humans
> share their knowledge via language and copying practices, the
> improvements that a small group of people can make are small compared
> to the improvements transferred from others, and made available by
> trading with those others.
>
> The obvious question about a single AI is why its improvements could
> not with the usual ease be transferred to other AIs or humans, or
> made available via trades with those others.   If so, this single AI
> would just be part of our larger system of self-improvement.   The
> scenario of rapid isolated self-improvement would seem to be where
> the AI found a new system of self-improvement, where knowledge
> production was far more effective, *and* where internal sharing of
> knowledge was vastly easier than external sharing.
>
> While this is logically possible, I do not yet see a reason to think
> it likely.   Today a single human can share the ideas within his own
> head far easier than he can share those ideas with others -
> communication with other people is far more expensive and
> error-prone.   Yet the rate at which a single human can innovate is
> so small relative to the larger economy that most innovation comes
> from ideas shared across people.  So a modest advantage for the AI's
> internal sharing would not be enough - the advantage would have to be
> enormous.
>
I haven't read the paper you mention here, but I have thought a little about 
the problem. It seems to me that there are two possibilities that might 
allow for a rapid increase in power. One is if creating such a computer it 
is able to break through some congenital limitations we have to our thought 
and knowledge. The idea here is that we think that every other species is 
congenitally limited in comparison with our cognitive abilities. A full 
scholarship to Cambridge is not going to allow an ape to understand Plato's 
Republic or Darwin's The Origins of the Species. The idea that we labour 
under such congenital limitations is controversial. Critics sometimes cite 
the fact that human science has been very successful in understanding the 
basic structure of the universe. I discuss this problem in a paper 
"Naturalism and Skepticism" and conclude that these two theoretical 
considerations are relatively balanced. I argue, that the way to see who is 
right is to put away the theoretical speculation and run the experiment. In 
any event, the thought that we are not immune from biological limits to our 
thought and knowledge suggests some evidence for the conjecture that a very 
powerful intelligence could develop if it overcame these limits, just as we 
are powerful in comparison to apes. Sometimes is suggested that the fact 
that we have language guarantees that there are no such limits (e.g., the 
philosopher Donald Davidson argues this way), but clearly language is not 
sufficient. Some people who are mentally challenged have linguistic ability, 
but not sufficient capacity to understand Plato or Darwin. The unhappy 
thought is that we may be mentally congenitally mentally challenged in 
comparison.

The second thought has to do with what counts as a 'single AI'. Think how 
enormously difficult it is to bring 100,000 humans to work on a single task 
was perhaps the case with the Manhattan project. An AI that could create 
human equivalent expert subsystems by deploying the computing power 
necessary to emulate 100,000 humans might be able to work on a single 
problem much more efficiently because of lower communication costs, 
political costs (getting people on board with the idea) and energy costs. 
Now it may be objected that such an AI would constitute an economy unto 
itself because in effect it has modeled a bunch of different experts working 
on a single problem like advanced nano. Perhaps, but then this may be the 
heart of the worry: it could create its own more efficient economy.

Mark

Dr. Mark Walker
Department of Philosophy
University Hall 310
McMaster University
1280 Main Street West
Hamilton, Ontario, L8S 4K1
Canada