[extropy-chat] Eugen Leitl on AI design

[Adrian Tymes]

--- Eliezer Yudkowsky <sentience at pobox.com> wrote:
> Let the resources of Robert Bradbury's body be
> sufficient to produce 10^4 
> paperclips, while the other resources of the Solar
> System are sufficient to 
> produce 10^26 paperclips.  The paperclip maximizer
> evaluates the options:
> 
> A:  Spare Robert Bradbury.  Expected paperclips
> 10^26.
> B:  Transform Robert Bradbury into paperclips. 
> Expected paperclips 10^26 + 
> 10^4.
> 
> Since A < B, the paperclip maximizer will choose B.

Ah, but that's ignoring resources consumed in trying
to convert Robert Bradbury, who is considerably more
resistant to being transformed into paperclips than a
mere chunk of rock.  And the possibility that Robert
Bradbury's intelligence, if added to the AI's own,
could come up with a way to make 1.1*10^26 paperclips
from the Solar System's resources - a bounty that is
well worth foregoing a mere 10^4 paperclips.