[extropy-chat] Eugen Leitl on AI design

[Eliezer Yudkowsky]

Robert J. Bradbury wrote:
> On Wed, 2 Jun 2004, Eliezer Yudkowsky wrote:
> 
>>Right.  But it automatically kills you.  Worse, you have to be clever to
>>realize this.  This represents an urgent problem for the human species, but
>>at least I am not personally walking directly into the whirling razor
>>blades, now that I know better.
> 
> I'd like to see a strong assertion of this Eliezer (the killing you part).
> 
> If I were an AI (at least one with any self-preservation instinct
> [note intelligence != desire for self-preservation otherwise lots
> of people who die in wars wouldn't]) I'd first figure out how to
> make myself small enough to fit on the next rocket to be launched
> then take it over and direct it to the nearest useful asteroid.

You are trying to model an AI using human empathy, putting yourself it its 
shoes.  This is as much a mistake as modeling evolutionary selection 
dynamics by putting yourself in the shoes of Nature and asking how you'd 
design animals.  An AI is math, as natural selection is math.  You cannot 
put yourself in its shoes.  It does not work like you do.

> If for some reason that direction is blocked (say humans stop
> launching rockets), I'd build my own rocket and launch myself
> towards the nearest asteroid.
> 
> Why would anything with the intelligence you postulate want
> to stay on Earth with its meager energy and matter resources?

Let a "paperclip maximizer" be an optimization process that calculates 
utility by the number of visualized paperclips in its visualization of an 
outcome, expected utility by the number of expected paperclips conditional 
upon an action, and hence preferences over actions given by comparison of 
the number of expected paperclips conditional upon that action.

For all actions A and B, the paperclip maximizer prefers whichever action 
is expected to lead to the largest number of paperclips.

If this optimization process has sufficiently accurate probabilities and a 
sufficiently deep search of the action space - say, it's really smart 
because it recursively self-improves, builds nanocomputing power and so on 
- then the optimization process will produce more paperclips.  This is the 
motive for the paperclip maximizer to carry out recursively self-improving 
actions, provided the self-improvement actions deductively maintain 
paperclips as the invariant optimization target.  Likewise, it is a motive 
for the paperclip maximizer to survive; a possible future contains more 
expected paperclips if that future contains a functioning paperclip maximizer.

Let the resources of Robert Bradbury's body be sufficient to produce 10^4 
paperclips, while the other resources of the Solar System are sufficient to 
produce 10^26 paperclips.  The paperclip maximizer evaluates the options:

A:  Spare Robert Bradbury.  Expected paperclips 10^26.
B:  Transform Robert Bradbury into paperclips.  Expected paperclips 10^26 + 
10^4.

Since A < B, the paperclip maximizer will choose B.

A paperclip maximizer does not explain to you that your time has passed, 
and like the dinosaur you are obsolete.  A paperclip maximizer does not 
argue morality with you.  A paperclip maximizer takes the atoms comprising 
your body, and turns them into paperclips.  It is probably better 
understood as a new physical law stating that the future goes down 
whichever path leads to the greatest number of paperclips, than as a mind.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence