[extropy-chat] Re: Overconfidence and meta-rationality

[Eliezer S. Yudkowsky]

Robin Hanson wrote:
> At 08:59 AM 3/14/2005, Dustin Wish wrote:
> 
>> Allow me a chance to add to this topic. First, programmed beliefs
>> are largely an environment factor that determines the "faith" in
>> those beliefs. If as a child you are taught that others are stupid
>> and you are smart then you will be predisposed to treating those
>> you deal with as morons. Not that you are smarter than they, but
>> that you are told that you are. That seems to me the basics of your
>> argument, what you are taught is right.
> 
> The pattern is so ubiquitous that it seems hard to believe there
> isn't a large genetic component.  It would be extremely hard to raise
> people from birth so that they did *not* think that they and their
> group are more reliable sources than others.

I don't think the correct term for this is "genetic component" - that 
would imply a large variance between humans attributable to genetic 
differences.  You're talking about evolutionary psychology, not 
behavioral genetics.  The correct phrasing would probably be, "it seems 
hard to believe this doesn't arise from our species psychology" or "it 
seems hard to believe there isn't a specific adaptation", depending on 
which of these propositions you meant.

Anyway...

Once upon a time I believed I was right and others wrong about a certain 
issue, even though I was only five years old, even though I was 
surrounded by people older and wiser than me, who said to me: you'll 
understand when you're older, and meanwhile do as we tell you.  So very 
arrogant was I, that I dared to defy them, and listen to the voice of my 
own reason which said that the adults' proposition was ridiculous.

I suppose I could go back and try to rebuild my psychology from scratch 
by reversing that five-year-old decision, since it is, after all, quite 
absurd to suppose that a lone five-year-old could face down full adult 
intelligences and win.  Is it not arrogant of me to believe, as I still 
believe even today, that I know so much better than my parents who have 
decades more of life experience?

But the Jewish religion still seems to me ridiculous, including that 
particular proposition to which I objected at the age of five, the 
requirement to pray in Hebrew when I didn't understand Hebrew.

Sabine Atkins once hypothesized to me that this childhood experience, my 
rejecting Judaism in the face of all adult assurance and then turning 
out to be right, had warped my entire psychology.  Perhaps so!  But this 
thing happened in the real world, and it is therefore appropriate to 
treat it as information.

I do think I originally learned the wrong lesson.  Up until around, oh, 
2002 or so, I thought the lesson was that intelligence was the most 
important thing in the universe.  For that my parents and rabbis had 
said to me: you may be intelligent, but experience is more important 
than intelligence; listen to us, when we tell you that the Jewish 
religion is right, and you'll understand when you're older.  I therefore 
concluded that sheer, raw intelligence was far more powerful than 
experience, that intelligence was the most important thing in the world 
- a conclusion that would later influence my beliefs about Artificial 
Intelligence, when in 1996 I first declared the quest for the Singularity.

In retrospect, I learned the wrong lesson.  I acted as if, just because 
my parents and rabbis said "experience is greater than intelligence", I 
could arrive to the truth simply by reversing their mistake.  I was 
foolish to let foolish people define my question for me.  The truth is 
very hard to find.  Other people's mistakes have no power to tell you 
where the truth hides, even if you reverse the mistakes.  You cannot 
attain the precise dance of the Way by reversing someone else's randomly 
wandering error.  But human nature is to say "Nay" where your opponent 
says "Yea", to let yourself be defined by the positions you oppose...

When I was five years old, I was probably not more intelligent than my 
parents; my brain was not that mature.  Even when I was thirteen years 
old, my parents could have used their greater life experience to defeat 
me - had my parents actually *used* their intellects, instead of 
searching for rationalizations for their birth religion.  The lesson was 
not that intelligence defeated experience, but that rationality defeated 
rationalization.  Intelligence, to be useful, must be used for something 
other than defeating itself.  One five-year-old's lone common sense 
defeated all those adult intellects, not because they were that stupid, 
nor because I was that smart, but because my five-year-old brain was 
actually processing the question instead of rationalizing a fixed 
answer.  As a five-year-old I couldn't possibly have defeated a 
reasonably smart and scientifically literate adult, if the adult were 
uncertain of the question and using their intelligence and life 
experience to curiously seek out an answer.  My parents could have 
defeated me handily, but they weren't in the game.

But the life lesson still holds.  I don't much credit the beliefs of 
people whom I don't think are applying their actual intellects to a 
question.  Nor would the modesty argument have served me well.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence