# [extropy-chat] Probability of identity - solution?

Eliezer S. Yudkowsky sentience at pobox.com
Fri Oct 13 21:33:35 UTC 2006

```Robin Hanson wrote:
>
> At the foundation of decision theory is a key distinction, between
> beliefs and wants (i.e., probabilities and preferences).   You can
> choose what you want anyway you like, but you are *not* free to
> choose your beliefs; beliefs are supposed to be your best estimate of
> the way the world is.   When you ask "what is the chance that ..."
> cannot depend on some value choice you make.

*Nods to Hanson.*

I tried to make the reference class depend on the utility function, and
while it helped clear up some parts of the problem, it didn't answer the
ultimate scientific question:  "What should we expect to *see*?"

> The situation you describe is one that could be repeated again and
> again.   After many repetitions you could compare the frequencies you
> see in your history to the probabilities you had assigned.    Or you
> could make bets based on your probabilities and see whether such bets
> win or lose on average.   These two related methods make clear that
> probabilities are not arbitrary value choices - they can be right or wrong.

The bizarre thing about these situations, as they work in our thought
experiments, is that, on most assumptions you care to make about where
the subjective probability mass goes (or as I sometimes say, where the
realness-fluid flows, bearing in mind that we are almost certainly
talking about some kind of phlogiston that isn't the actual solution) -
anyway, regardless of which assumption you make, after N repetitions of
the experiment, nearly all of your subjective probability mass is in a
set of observers who, by applying induction, would end up with the
assumption you started with - that is, under your assumption, all the
probability mass would end up in observers who, given their experienced
history, would strongly suspect your assumption - which is to say, after
N repetitions of the experiment, then "you", wherever you are, would be

But an outside observer would have no idea where most of the
"probability mass" had gone, so they can't learn anything by performing
the experiment from outside - you would have to be inside it.

> For long enough histories, almost everyone should see statistics where
> frequencies are close to probabilities.

But which subset of observers constitutes "almost everyone" and how can
you tell from outside?

Confusion exists in the mind, not in reality.  All this mess has to be
generated by a bad question - certainly we have to be doing *something*
wrong.  I find it highly suspicious that the central question revolves
around a phlogiston-type substance, subjective (conditional) probability
mass, that cannot be observed from the outside but which we imagine
ending up in different amounts in different observers.

--
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence

```