[ExI] Statistical tests

[spike]

>... On Behalf Of BillK
Subject: [ExI] Statistical tests

Weak statistical standards implicated in scientific irreproducibility
One-quarter of studies that meet commonly used statistical cutoff may be
false.   11 November 2013

<http://www.nature.com/news/weak-statistical-standards-implicated-in-scienti
fic-irreproducibility-1.14131>

Quote:
>...Johnson then used these uniformly most powerful tests to compare P
values to Bayes factors. When he did so, he found that a P value of
0.05 or less - commonly considered evidence in support of a hypothesis in
fields such as social science, in which non-reproducibility has become a
serious issue - corresponds to Bayes factors of between 3 and 5, which are
considered weak evidence to support a finding...
-----------------
BillK
____________________________________________


Thanks BillK, I have thought this for a long time.  Back when I was learning
about Bayesian statistics I did the math on this and came to the same
startling conclusion: that arbitrary 95% confidence level we are taught in
Statistics 101 is misleading.  That does not mean your conclusion will be
right 95% of the time.  I encourage the math geeks to go over the equations
to see why.  This doesn't mean the 95% criterion is useless, only that it is
an oversimplification that leads often to misinterpretation, and creates
opportunity for all kinds of mischief.  I saw this firsthand in a failed
manufacturing line in 1990 and 1991.

The engineers involved in that are a pathetic poster child for this phenom.
They were absolutely set in stone convinced, if any test was below 95%
confidence, out with it, meaningless.  Above, chisel it in stone tablets, a
universal truth has been discovered.  Nature doesn't really work that way.
It was classic example of a fuzzy line that became chiseled in stone as a
sharp boundary between truth and fiction.  I kept trying to explain why this
was going wrong, but I might as well have been talking to a wall; 95%
confidence is not just an arbitrary mathematical convenience  to them.
Everyone's statistics book said plainly, 95% confidence determines
statistical significance period end of story.

spike