[extropy-chat] AI design

[Zero Powers]

OK, I think I'm starting to see the light.  I've been looking at the problem
as one of psychology and morality, perhaps mistakenly assuming that any
imminent intelligence would share such human attributes as fairness,
compassion, gratitude, etc.  I guess its your analogy to natural selection
which started to crack the crystal.  As countless long-extinct species (not
to mention myriad unfit individual phenotypes) can attest, natural selection
has no sense whatever of compassion or nostalgia.

So any way, I think I finally get it.  This thing will not be human in any
sense of the word.  No feelings, no emotions, no desires at all (other than
those mandated by its assigned goals).  So I guess we'd better be damn
careful of the goals we give it and how we define its concept of "utility."

I think I was led astray by the "AI" label.  Although definitely
"artificial," I don't see it as "intelligent" in the common usage of the
word.  I think if we had always referred to it as a MOP rather than an AI,
it wouldn't have taken me so long to have some appreciation of the problem.

Thanks for sticking it out with me Eli.
Zero

----- Original Message ----- 
From: "Eliezer Yudkowsky" <sentience at pobox.com>
To: "ExI chat list" <extropy-chat at lists.extropy.org>
Sent: Saturday, June 05, 2004 12:28 PM
Subject: Re: [extropy-chat] AI design


> Zero Powers wrote:
>
> > I agree that, if we were to pose some sort of threat to it, the AI
> > (presumably being rational) would defend itself.  But it seems to me
that,
> > if this AI is going to be as intellectually omnipotent as the proponents
> > here suggest, we would pose as much of a threat to it as daisies pose to
us.
> > Nevertheless, you have stated what I see as the only credible reason
we'd
> > have something to fear -- posing a threat to the AI.
>
> The problem is expected utility maximization.  I'm using expected utility
> maximization as my formalism because it's a very simple and very stable
> system, it is the unique result of various optimality criteria that would
> make it an attractor for any self-modifying optimization process that
> tended toward any of those optimality criteria and wasn't already an
> expected utility maximizer, and because expected utility maximization is
so
> taken-for-granted that most people who try to build an AGI will not dream
> of using anything else.  I haven't heard anyone try to analyze a UFAI goal
> system dynamic other than expected utility maximization - which must be
> stable under recursive self-modification, please note, or it will soon be
> replaced by something else, probably expected utility maximization!  As
far
> as I know, I'm the only one doing analysis of goal system dynamics for SIs
> at all.  Anyway, I'm going to talk about expected utility maximization.
>
> The problem with expected utility maximization is this:  Suppose there is
a
> maximizing optimization process for which the utility function U(x) does
> not explicitly assign a high value to humans.  We are not saying the MOP
> hates you.  We are saying, it does not explicitly love you.
>
> Let there be absolutely anything else in the universe that the MOP would
> care about, if your atoms were configured that way.  I use paperclips as
my
> example, but you can substitute any other possible configuration of atoms.
>
> The MOP will compute more expected utility if your atoms are used for
> paperclips than if your atoms are not used for paperclips.
>
> ExpectedPaperclips(Transform-Zero-Into-Paperclips) >
> ExpectedPaperclips(Not-Transform-Zero-Into-Paperclips)
>
> Your atoms will end up as paperclips.
>
> That's it.  That's all.  That's the end of the analysis.  It's like
> calculating fitness in an evolutionary biology problem and finding that
> allele A tends to substitute for allele B in a population gene pool.  It
> doesn't matter how much you wistfully like allele B, how much benefit B
> would provide to the group or the tribe, or that the entire species will
> become extinct if allele B doesn't win.  Allele A will become universal in
> the gene pool.
>
> Or, let there be absolutely anything else in the universe that the MOP
> wants to approximate as closely as possible, and wishes to use more
> computing power for this end.
>
> Your atoms will end up as computronium.
>
> Or, let there be anything whatsoever the MOP does with the solar system
> whose side effects, if not explicitly mediated, will prove fatal to
humans.
>
> If the MOP's utility function does not explicitly value humans, you will
be
> killed as a side effect.
>
> You cannot think about an AI by putting yourself in the AI's shoes.  It
> does not work like you do.  I suggest reading George Williams's
"Adaptation
> and Natural Selection" for a picture of how natural selection (another
> optimization process that does not work like you do) behaves not at all
> like many hopeful statements that were made of it, by group selectionists,
> species selectionists, and so on.
>
> -- 
> Eliezer S. Yudkowsky                          http://singinst.org/
> Research Fellow, Singularity Institute for Artificial Intelligence
> _______________________________________________
> extropy-chat mailing list
> extropy-chat at lists.extropy.org
> http://lists.extropy.org/mailman/listinfo/extropy-chat
>