[ExI] psi or bad statistics?

[Eliezer S. Yudkowsky]

Damien Broderick wrote:
> 
> Well, here's an extract; the doubtful will need to go to the primary sources:
> 
> <[Dr. Dean] Radin used one of the simplest possible methods, 
> measuring shifts in skin conductivity in fingers or palms when the 
> wired subjects were affronted by computer images of violent or erotic 
> scenes, and those reactions in turn were compared with the physical 
> response elicited by soothing images and no images at all. When the 
> data from many subjects were added together and averaged, in order to 
> remove idiosyncratic responses and the intrusion of random noise, it 
> turned out that the average response (presponse) to neutral or 
> pleasant images followed pretty much the curve one would expect. The 
> image flashes on the screen for three seconds, and while subjects 
> watch the blank screen that follows, their skin conductivity rises 
> slightly, drops away again, flutters along in its normal quietly 
> wandering path.>

It's a remarkable, amazing fact about psychic powers that they seem to 
work as well backward in time as forward; as well for psi-miss as 
psi-hit; as well for manipulation as prediction; as well when the 
telekinesis is exerted after the experiment as before it; and of 
course, the effect size gets smaller and smaller (but still 
statistically significant) as the samples get larger and larger.

Now it seems unlikely that anything a brain can really do would have 
such properties, but they are all naturally expected in association 
with the amazing magical power known as "bad statistics", which works 
as well backward in time as forward, as well for misses as hits, etc.

The particular statistical flaw in Radin's experiments would seem to 
be described here:

http://m0134.fmg.uva.nl/publications/2002/expectationbias_PA2002.pdf

It's amazing what subtle statistical flaws you can uncover when you 
know a priori that the effect is not real.  This problem would almost 
certainly had gone undetected if Radin had been performing a more 
conventional medical experiment.

The psionicists have a legitimate complaint that they are being held 
to higher standards than the rest of science.  Bad statistics are an 
increasingly huge problem for the rest of science, too, but unlike 
psionics we don't know a priori that the detected effects are unreal.

For example, it's now suspected that half of all published medical 
studies in major journals have irreproduceable results. 
(http://www.overcomingbias.com/2007/09/false-findings-.html)  We 
should hold all science to the same strict standards that would be 
required to eliminate ESP.

I've been advocating that p<0.001 should replace p<0.05 as the margin 
of statistical significance.  Physics journals routinely require 
p<0.0001.  It would be better by far to do fewer medical experiments 
with more subjects and have nearly all the published reports be valid. 
  But there's a Nash equilibrium for the bad behavior, where you use 
the lower standard and make sure of a publication.

I myself now put no more than a 50% probability on any published 
finding that is statistically significant at p<0.05 rather than 
p<0.001.  Less, if the finding seems iffy in other ways.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence