[extropy-chat] AI design

[Eliezer Yudkowsky]

Zero Powers wrote:

> I agree that, if we were to pose some sort of threat to it, the AI
> (presumably being rational) would defend itself.  But it seems to me that,
> if this AI is going to be as intellectually omnipotent as the proponents
> here suggest, we would pose as much of a threat to it as daisies pose to us.
> Nevertheless, you have stated what I see as the only credible reason we'd
> have something to fear -- posing a threat to the AI.

The problem is expected utility maximization.  I'm using expected utility 
maximization as my formalism because it's a very simple and very stable 
system, it is the unique result of various optimality criteria that would 
make it an attractor for any self-modifying optimization process that 
tended toward any of those optimality criteria and wasn't already an 
expected utility maximizer, and because expected utility maximization is so 
taken-for-granted that most people who try to build an AGI will not dream 
of using anything else.  I haven't heard anyone try to analyze a UFAI goal 
system dynamic other than expected utility maximization - which must be 
stable under recursive self-modification, please note, or it will soon be 
replaced by something else, probably expected utility maximization!  As far 
as I know, I'm the only one doing analysis of goal system dynamics for SIs 
at all.  Anyway, I'm going to talk about expected utility maximization.

The problem with expected utility maximization is this:  Suppose there is a 
maximizing optimization process for which the utility function U(x) does 
not explicitly assign a high value to humans.  We are not saying the MOP 
hates you.  We are saying, it does not explicitly love you.

Let there be absolutely anything else in the universe that the MOP would 
care about, if your atoms were configured that way.  I use paperclips as my 
example, but you can substitute any other possible configuration of atoms.

The MOP will compute more expected utility if your atoms are used for 
paperclips than if your atoms are not used for paperclips.

ExpectedPaperclips(Transform-Zero-Into-Paperclips) >
ExpectedPaperclips(Not-Transform-Zero-Into-Paperclips)

Your atoms will end up as paperclips.

That's it.  That's all.  That's the end of the analysis.  It's like 
calculating fitness in an evolutionary biology problem and finding that 
allele A tends to substitute for allele B in a population gene pool.  It 
doesn't matter how much you wistfully like allele B, how much benefit B 
would provide to the group or the tribe, or that the entire species will 
become extinct if allele B doesn't win.  Allele A will become universal in 
the gene pool.

Or, let there be absolutely anything else in the universe that the MOP 
wants to approximate as closely as possible, and wishes to use more 
computing power for this end.

Your atoms will end up as computronium.

Or, let there be anything whatsoever the MOP does with the solar system 
whose side effects, if not explicitly mediated, will prove fatal to humans.

If the MOP's utility function does not explicitly value humans, you will be 
killed as a side effect.

You cannot think about an AI by putting yourself in the AI's shoes.  It 
does not work like you do.  I suggest reading George Williams's "Adaptation 
and Natural Selection" for a picture of how natural selection (another 
optimization process that does not work like you do) behaves not at all 
like many hopeful statements that were made of it, by group selectionists, 
species selectionists, and so on.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence