[ExI] Again: psi or bad statistics?

[The Avantguardian]

--- Eliezer Yudkowsky <sentience at pobox.com> wrote:

> I don't trust this document
>    http://www.as.utexas.edu/jefferys/slides/berger.pdf
> because its author says some silly things about Bayesian philosophy,
> notably, the idea that "objective Bayes" provides "objective"
> posterior probabilities in experiments.

I agree, Eli. I am not a hair-splitter so to me, it seems kind of pointless to
distinguish between objective and subjective Bayes based on how one chooses
ones priors. Informative priors are better if they are available. If you can't
estimate anything informative, then as a last resort go with the flat prior. If
you have enough data, you'll get the same posteriors no matter what your priors
are.

Objectively, what's going to happen is going to happen. Probability is simply a
quantification of ones ignorance regarding that result. And ignorance is a
subjective phenomenon. Yeah I know the objections: QM, MWI, radioactive decay,
etc. But at the end of the day, and that is what *posteriors* are concerned
with, one future and one future alone will have happened and if you record it
and play it back, that same one future happens again and that is the objective
reality. 

> However, it says that a Dean Radin psi experiment which was
> "statistically significant" at p ~ .0003, subjected to a Bayesian
> re-analysis, ends up with the null hypothesis going to a posterior
> probability of 0.92 if it started with prior probability 0.5.

Well that's just a silly choice of priors. If 50% of everybody was
psychokinetic, nobody would doubt its existence to begin with. Or do they mean
that as the prior distribution of red and green lights?

> I was wondering if Radin had a standard response to this.  It seems
> like a generally useful example of Why To Never Use P-Values, *if*
> true.

It has to do with how the null hypothesis if defined, i.e. perfect randomness.
So his P-value of .0003 is just the probability that the data were completely
random. There are a lot of possible reasons why the data might not be
completely random and psi is just one possible explanation. Not being able to
account for those other possibilities is the real weakness of frequentist
hypothesis testing relative to Bayes.

For example let's say I think the phases of the moon are caused by a giant
invisible mouse that eats and then regurgitates the green cheese of the moon.
By standard frequentist testing I would get a vanishingly small probability for
the null hypothesis that the moon's phases occur randomly. That small P-value
however is not in any way evidence that my pet hypothesis is correct.
With, Bayes on the other hand, you can simultaneously test the null hypothesis
and as many other hypotheses as you so desire.

So yeah P-values work, just not the way people try to use them. 

Stuart LaForge
alt email: stuart"AT"ucla.edu

"In ancient times they had no statistics so that they had to fall back on lies."- Stephen Leacock