[extropy-chat] Re: Overconfidence and meta-rationality

[Hal Finney]

Eliezer writes:
> Thus, I still think that people who disagree should, pragmatically, go 
> on arguing with each other about the matter of interest, instead of 
> immediately compromising based on a belief in the probable rationality 
> of the other.

That seems reasonable, and from what I understand, is consistent with
Robin's results.  I read him as not so much saying that Bayesians "should"
agree; but that they can't help agreeing.  That doesn't change the fact
that the best way to aggregate their information may well be to argue
the situation *as though* they strongly disagreed.

Here's a thought experiment.  Suppose you were duplicated, and you and
your copy went out and independently worked and did research on FAI.
You come back together after a year, and you simultaneously report your
estimates on whether a given FAI project will succeed.  One of you says
the odds are 1/10, and the other says the odds are 9/10.

I would think, in this situation, that at least one of you would be
extremely surprised and shocked.  You would be forced to accept the
fact that your copy had accumulated evidence during that year which led
him to a very different estimate than your own.  And the mere knowledge
of that difference, even before you begin talking about the details of
what you both learned, will be enough to sharply change your internal
estimate of the probability.

You might even be struck sufficiently by the symmetry of the situation
to get past the automatic assumption that you are more likely to be
right than your copy.  You would understand that it was really only an
accident of chance which copy your consciousness was in, that you could
just as easily have been the other one, in which case you would hold
the opposite view about probability.

Suppose your experience was that the strength of the evidence you
accumulated during the year had not been as strong as you hoped and
expected it would be, and therefore it is likely that your copy had
better luck than you did in terms of the strength of his evidence.
Given the symmetry which otherwise exists, you would then prefer to adopt
its position as your new estimate.  OTOH if the year had been unusually
productive and strong in the quality of evidence it gave you for your
beliefs, you would choose not to switch.  Since you know nothing about
what the copy's experiences were along these lines, you can't predict
whether he will switch or not; it will depend solely on whether the
strength and quality of the evidence he accumulated during the year was
above or below average, which is not known to you.

Even if you do follow these predicted behaviors, you would still be right
to argue (or at least vigorously discuss) the reasons why your copy came
to such a dramatically different result than you did.  You want to pool
your information and come up with the best quality estimate based on
everything you two learned during the year.  And it may well be that the
best way to do that is for the one to defend the 0.1 estimate while the
other defends 0.9, each based on what they learned.

While doing this, though, I think mentally you would each be somewhat in
the frame of mind of playing devil's advocate.  You'd be pushing a strong
position while privately believing that the other side may well be right.
As long as you're honest about it, that seems fine.  I think this is
the mental stance that Bayesians would have to hold while arguing and
disagreeing with each other, and such a practice seems plausible to be
the optimal method for consolidating information.

Hal