[extropy-chat] Re: Overconfidence and meta-rationality

[Hal Finney]

I have enjoyed the debate between Eliezer and Robin (which I think
took place mostly on the wta-talk list), although I haven't been
able to follow it as closely as I'd like.  These long messages take
a lot of time to get through.  I wanted to make one comment on Eliezer's
posting:

> The modesty argument is important in one respect.  I agree that when two 
> humans disagree and have common knowledge of each other's opinion (or a 
> human approximation of common knowledge which does not require logical 
> omniscience), *at least one* human must be doing something wrong.

I'd put it a little differently.  There's nothing necessarily wrong
when two humans disagree and have common knowledge.  You have to add one
more ingredient.  The people have to both be rational and honest, and,
most importantly, they each have to believe that the other is rational
and honest (and, I think, this has to be common knowledge).

I would imagine that many cases of disagreement can be explained by
each party privately concluding that the other is being irrational.
They're just too polite to say so.  When they say, I guess we'll have
to agree to disagree, they mean, you're being unreasonable and I don't
want to argue with you any more because there's no point.

But actually, we can sharpen Aumann's result.  It doesn't require
assumptions about two people.  It is enough for one person to satisfy
the conditions.

Aumann basically says (neglecting the part about priors) that it is
impossible for a rational person to believe that he has a persistent
disagreement with another person whom he believes to be rational, where
the other person also believes the first person is rational.  Note that
this says nothing about what the second person actually believes.  It all
has to do with what the first person believes about the world.  It says
that a certain combination of beliefs is impossible for a rational person
to hold.

This perspective frees us from the competitive aspect of meta-rationality,
the "mine is bigger than yours" dynamic that sometimes arises in
discussions of this issue.  It's not a matter of one person being wrong
in a disagreement, or one person being more meta-rational than another.

Aumann is giving us a non-obvious piece of logic which we can follow in
our own thought processes, independent of what anyone else does.  I can't
fool myself into believing that I can agree to disagree with another
person, while respecting him as a rational and honest person who offers
the same respect towards me.  For me to hold this set of beliefs is a
logical contradiction.  That's the lesson I draw from this set of results.

In a way, then, Aumann can be read as giving you license to feel
contempt for others.  He's saying that it is mental hypocrisy (if that
means anything!) to try to adopt that generous and polite stance I just
described.  When we try to convince ourselves that we really believe
this noble fiction (that the other person is rational and honest), we are
lying to ourselves.  It's another case of self-deception.  The truth is,
we don't respect the other person as rational and honest.  If we did,
we wouldn't be ignoring his beliefs!  We think he's a fool or a knave.
Probably both.  We're not so damn nice as we try to pretend to be,
as we try to convince ourselves we are.

Hal