[extropy-chat] IRAQ: Weapons pipeline to Syria

[Eliezer Yudkowsky]

http://image.thelancet.com/extras/04art10342web.pdf

Stephen Van_Sickle wrote:
> 
> But the real clincher was right there in the abstract,
> which I did not read carefully enough until this
> morning.  It seems that the calculation they used for
> the 100,000 estimate had a 95% confidence level range
> of from 8000-194000.  Now, this seems to me a bit like
> saying Bush will will with 51% of the vote, plus or
> minus 40%.  This is enough for me to conclude, in the
> absence of a better explaination, that the whole study
> is meaningless.

The survey methods of Roberts et al. are better than any previous estimates 
provided, which is itself startling.  They had a grand total of seven, 
count 'em, seven investigators with which to interview 33 clusters of 30 
houses apiece.  Why haven't US occupying forces done similar and more 
systematic estimates?  You report the best scientific data you have.  If 
the confidence interval is too wide, you do more surveys.  Meanwhile, it's 
the only data you have, so you grind through your statistical methods 
(hopefully Bayesian ones, though I didn't see any sign of that) and produce 
the maximum likelihood estimate of 100,000.  You can't just throw away that 
estimate because you don't like it, unless you come up with better data. 
If the study was conducted correctly (and if we assume a uniform prior that 
gives us MLE=MAP) then that figure of 100,000 is the most likely to be 
correct compared to any other figure you care to name.  That's what the 
statistical method does.

*However*, that figure of 100,000 is *after* excluding the statistical 
outlier of Falluja, which might be taken as representative of other 
extremely devastated cities not visited by the investigators, and which 
would have implied a figure in the range of 300,000 civilian casualties if 
included into the study.  I'd have included it.  You canna' change your 
experimental design after the fact.  If you want to exclude outlying 
clusters you should write down that procedure in advance.  But then I'm a 
Bayesian and an unusual stickler about such matters.  The Lancet authors, 
concerned with their scientific reputations and media reporting of the 
result and other such irrelevancies, seem to have picked a lower bound that 
couldn't be challenged as the product of an outlying cluster, rather than 
producing the number that was most likely to be actually correct.  As they 
say in the abstract, it's a "conservative" estimate.  Indeed I rate such 
estimates as worthless, if what you need above all is the correct answer to 
the question of simple fact.  So, probably more than 100,000, but with an 
even wider confidence interval, since the exact increase is based on 
extremely war-torn areas not adequately treated in the sample.

> If anyone can explain to me why it is not, I'd love to
> hear it.  I'd hate to think the Lancet would be this
> blatantly, well, dishonest.

The Roberts study overrides any "estimates" based on wishful thinking, 
politics, etc., unless and until someone demonstrates a flaw in the study's 
methods or a deliberate bias / misreporting by the authors, or until 
someone collects better data.  If the US did not wish to be accused by 
independent investigators, they should have collected their own data on 
civilian casualties instead of waiting to be surprised - using active 
methods reliable in cases of this kind, rather than passive reporting as in 
iraqbodycount.net.

I do not find it the least bit implausible that civilian casualties 
reported in two or more independent media stories (IBC.net's criterion) are 
a seventh or less of civilian casualties estimated by active surveying, 
especially given the result that most civilian casualties result from air 
strikes.  The Roberts study did note that the iraqbodycount.net data seemed 
to track trends in their own data fairly well, indicating that reported 
deaths may be a good *sample* of total deaths.

-- 
Eliezer S. Yudkowsky                          http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence