[extropy-chat] Fools building AIs

Eliezer S. Yudkowsky sentience at pobox.com
Fri Oct 6 07:46:25 UTC 2006


Ben Goertzel wrote:
> Eli wrote:
> 
>>>I do not see why this attitude is inconsistent with a deep
>>>understanding of the nature of intelligence, and a profound
>>>rationality.
>>
>>Hell, Ben, you aren't willing to say publicly that Arthur T. Murray is a
>>fool.  In general you seem very reluctant to admit that certain things
>>are inconsistent with rationality - a charity level that *I* think is
>>inconsistent with rationality.
> 
> Heh....  Eli, this is a very humorous statement about me, and one that
> definitely would not be written by anyone who knew me well in person
> !!!!   I have to say that making this kind of generalization about me,
> based on your very limited knowledge of me as a human being, is rather
> irrational on your part ;-) ... My ex-wife would **really** get a kick
> out of the idea that I am "reluctant to admit that certain things are
> inconsistent with rationality" ;-)

Oh, I'm sure there are plenty of things which you admit are inconsistent 
with rationality.  Arthur T. Murray is apparently not one of them, which 
suggests your standards are a bit lax.  You are too charitable, and this 
is also a flaw.

> I think there are a LOT of things that are inconsistent with
> rationality ... though it's also true that some humans can be highly
> rational in some domains and highly irrational in others, and
> effectively maintain a strict separation between the domains.  (For
> example, I know some excellent scientists who are also deeply
> religious, but separate the two domains verrry strictly so their
> rationality in science is not pragmatically affected by their
> irrationality in personal and spiritual life.)
> 
> However, I don't think that advocating the creation of superhuman AI
> even in the face of considerable risk that it will annihilate
> humanity, is **irrational**.  It is simply a choice of goals that is
> different from your, Eliezer's, currently preferred choice.

Heh.  Considerable risk?  Your friend spoke of the *desirability* of 
such an outcome, not the risk of it.  Both of these are foolishness but 
they are foolish in different ways.  In the former case it is a moral 
error, pure and simple.  In the latter case, it is a result of vague 
ideas leading you to not know the rigorous reasons which show why rigor, 
i.e., theories strong enough to formally calculate extremely low failure 
probabilities, is required in order to succeed.  If you don't know the 
rules, you don't know the rule whereby you could see that not knowing 
the rules definitely kills you.  But that's a separate issue.

>>  -- "Cognitive biases potentially affecting judgment of global risks"
>>
>>I seriously doubt that your friend is processing that question with the
>>same part of his brain that he uses to decide e.g. whether to
>>deliberately drive into oncoming traffic or throw his three-year-old
>>daughter off a hotel balcony.
> 
> No, but so what?
> 
> The part of his mind that decides whether to throw someone off a
> balcony or to drive into traffic is his EMOTIONS ... the part of his
> mind that decides whether a potentially dangerous superhuman AI should
> be allowed to be created is his REASON which is more dispassionately
> making judgments based on less personal and emotional aspects of his
> value system...

The comfort of cynicism is an emotion, though it is mistaken by many for 
rationality.

"The second virtue is relinquishment.  P. C. Hodgell said:  "That which 
can be destroyed by the truth should be."  Do not flinch from 
experiences that might destroy your beliefs.  The thought you cannot 
think controls you more than thoughts you speak aloud.  Submit yourself 
to ordeals and test yourself in fire.  Relinquish the emotion which 
rests upon a mistaken belief, and seek to feel fully that emotion which 
fits the facts.  If the iron approaches your face, and you believe it is 
hot, and it is cool, the Way opposes your fear.  If the iron approaches 
your face, and you believe it is cool, and it is hot, the Way opposes 
your calm.  Evaluate your beliefs first and then arrive at your 
emotions.  Let yourself say:  "If the iron is hot, I desire to believe 
it is hot, and if it is cool, I desire to believe it is cool."  Beware 
lest you become attached to beliefs you may not want."

> It is not logically inconsistent to
> 
> a) value being alive more than being dead
> b) value a superhuman AI's life more than the human race's life

What kind of AI?  How hard is it to build this kind of AI?  Is that kind 
of AI likely to wipe out humans in the first place?  But this of course 
is a dispute of fact.

>>Your friend, I suspect, is carrying out a form of non-extensional
>>reasoning which consists of reacting to verbal descriptions of events
>>quite differently than how he would react to witnessing even a small
>>sample of the human deaths involved.
> 
> Well, but why do you consider it irrational for someone to make a
> considered judgment that contradicts their primal emotional reactions?

That depends on the *why* of the considered judgment.  We form 
deliberative moral principles through an essentially emotional process, 
even though those principles may later override other emotions.  If the 
moral principle itself is formed through flawed reasoning - reasoning 
which could be destroyed by some particular truth - then the process as 
a whole is irrational.  "A chain of a thousand links will arrive at a 
correct conclusion if every step is correct, but if one step is wrong it 
may carry you anywhere.  In mathematics a mountain of good deeds cannot 
atone for a single sin.  Therefore, be careful on every step."

> In this case, the person may just be making a decision to adopt a
> supergoal that contradicts their emotional reactions, even though they
> are not able to actually extinguish their emotional reactions...

And Arthur T. Murray *may* be secretly a genius, but it is not likely. 
I once constructed an incorrect moral ontology (Eliezer_1996-1999) that 
forced me to consider a similar tradeoff, but I never thought it was 
*likely* that a nice superintelligence would kill people because it was 
the right thing to do.  In those days I thought it was a serious 
possibility, because in those days I didn't understand how I was 
evaluating "right thing to do", so for all I knew it would end up 
anywhere.  But I never thought murder was a *likely* good deed.  That, 
to me, suggests a different category of mistake.

> No, he is not making this mistake.  He thinks they'll be better
> because he thinks our evolutionary "design" sucks

So it does.  For example, it leads people to try to solve problems by 
murder, without even looking for a third alternative.

> and appropriately
> engineered AI systems can be ethically as well as intellectually
> superior by design...

I believe the children are our future, but only if they're very 
carefully designed;
I believe I'm more ethical than my own parents, and somehow, I don't 
want to kill them because of that;
And if I did want to kill my parents for the crime of still having 
things yet to learn, then I would have gone wrong, and certainly my life 
would not be worth more than theirs, or so my present self judges.

Anyway.  I'm not interested in arguing with your friend secondhand.  Say 
what you yourself think is *optimal* reasoning, and I may be interested 
in disputing it.  As for what you think *may* be rational, that's a 
class large enough to include Mentifex, so I may have to say that we 
have different standards for what "may" be "rational" and close it at that.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence



More information about the extropy-chat mailing list