[extropy-chat] Probability of identity - solution?

[Eliezer S. Yudkowsky]

Robin Hanson wrote:
> 
> At the foundation of decision theory is a key distinction, between 
> beliefs and wants (i.e., probabilities and preferences).   You can 
> choose what you want anyway you like, but you are *not* free to 
> choose your beliefs; beliefs are supposed to be your best estimate of 
> the way the world is.   When you ask "what is the chance that ..." 
> you are asking about probabilities and beliefs, so the answer simply 
> cannot depend on some value choice you make.

*Nods to Hanson.*

I tried to make the reference class depend on the utility function, and 
while it helped clear up some parts of the problem, it didn't answer the 
ultimate scientific question:  "What should we expect to *see*?"

> The situation you describe is one that could be repeated again and 
> again.   After many repetitions you could compare the frequencies you 
> see in your history to the probabilities you had assigned.    Or you 
> could make bets based on your probabilities and see whether such bets 
> win or lose on average.   These two related methods make clear that 
> probabilities are not arbitrary value choices - they can be right or wrong.

The bizarre thing about these situations, as they work in our thought 
experiments, is that, on most assumptions you care to make about where 
the subjective probability mass goes (or as I sometimes say, where the 
realness-fluid flows, bearing in mind that we are almost certainly 
talking about some kind of phlogiston that isn't the actual solution) - 
anyway, regardless of which assumption you make, after N repetitions of 
the experiment, nearly all of your subjective probability mass is in a 
set of observers who, by applying induction, would end up with the 
assumption you started with - that is, under your assumption, all the 
probability mass would end up in observers who, given their experienced 
history, would strongly suspect your assumption - which is to say, after 
N repetitions of the experiment, then "you", wherever you are, would be 
almost certain of the answer.

But an outside observer would have no idea where most of the 
"probability mass" had gone, so they can't learn anything by performing 
the experiment from outside - you would have to be inside it.

> For long enough histories, almost everyone should see statistics where
> frequencies are close to probabilities.

But which subset of observers constitutes "almost everyone" and how can 
you tell from outside?

Confusion exists in the mind, not in reality.  All this mess has to be 
generated by a bad question - certainly we have to be doing *something* 
wrong.  I find it highly suspicious that the central question revolves 
around a phlogiston-type substance, subjective (conditional) probability 
mass, that cannot be observed from the outside but which we imagine 
ending up in different amounts in different observers.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence