[Paleopsych] CHE: Scholar Sends Sham Papers to Social-Work Journals to Show Weakness of Peer Review
checker at panix.com
Thu Oct 21 20:16:50 UTC 2004
Scholar Sends Sham Papers to Social-Work Journals to Show Weakness of Peer
News bulletin from the Chronicle of Higher Education, 4.10.20
By DAVID GLENN
Most scholars would let out a groan if a journal returned their
submissions with peer-review comments like these: "The author presents
us with a flawed study that is not well analyzed. ... The reviewer has
no idea what statistical tests were conducted, how many of them were
done, and what those findings were."
William M. Epstein, however, was delighted by those comments, and
wishes that he'd gotten many more like them.
Two years ago, Mr. Epstein, a professor of social work at the
University of Nevada at Las Vegas, submitted fictitious, pseudonymous,
deliberately shoddy papers to 33 journals in his field. "The basic
designs were off," he says. "I drew conclusions that were unwarranted
on the basis of the data. I never brought up statistical tests -- I
just simply announced that there was significance."
Even the most quotidian passages were mined with errors. One paper
concluded by saying, "The average cost of care for control clients was
about $576 and for experimental clients about $3,975. The $3,419
difference seems well justified by the outcomes." (The difference
between those costs is actually $3,399.)
In sum, Mr. Epstein says, "a careful reviewer would have butchered
No journal came very close to publishing any of the sham articles, but
Mr. Epstein believes that his experiment nonetheless uncovered some
deep intellectual problems in the world of social-work scholarship. He
explains his deed -- and lays out his accusations against his field --
in an article published this month in the journal Research on Social
Work Practice, which was among the journals hit by his hoax.
Mr. Epstein's targets are not taking this lightly. Some of them say
that his project was needlessly deceptive, and that it was itself
biased and poorly designed.
The first charge in Mr. Epstein's indictment is that social-work
journals suffer from "confirmational response bias." By that he means
that the journals are more inclined to publish papers that support
social workers' typical methods than papers that raise doubts.
To test that hypothesis, Mr. Epstein circulated a pair of articles
about a fictitious program to preserve families whose children are at
risk of being placed in foster care. The "positive" version of the
article said that the program worked well; the "negative" version said
that the program had no effect. Both versions were methodologically
shoddy, and they were identical except for a few paragraphs in the
Half of the journals in Mr. Epstein's experiment were randomly
selected to receive the positive version, and the other half received
the negative version. The positive version got a much warmer welcome.
The negative version received seven flat rejections, but the positive
version received only two. And the positive version received four
acceptances -- in each case, conditioned on various revisions -- while
the negative version received none.
In Mr. Epstein's eyes, social work -- together with other
practice-oriented fields, like counseling, management, and clinical
psychology -- is too reluctant to publish information about techniques
that don't seem to work. There is a bias, he says, toward "happy
Mr. Epstein's second charge is that the general quality of peer review
in social-work journals is feeble. He and three anonymous colleagues
evaluated the thoroughness and sophistication of the peer-review
reports generated by the sham articles. Mr. Epstein's panel deemed
almost three-quarters of the reviews to be inadequate; in his paper,
he describes some of them as "incomplete, short, personalized, and
Very few of the reviews, he says, pointed out that the papers lacked
coherent accounts of their statistical tests.
"When you take a look at the appalling quality of research in the
field," Mr. Epstein says, "it's really quite obvious that
methodological sophistication is not of paramount interest."
This is not Mr. Epstein's first foray into surreptitious research. In
the late 1980s he conducted a very similar project, and it landed him
in hot water. A board of inquiry at the National Association of Social
Workers voted in 1988 to censure him for failing to obtain his
subjects' informed consent, although that decision was later
overturned by the association's National Board of Directors.
This time, Mr. Epstein says, he sought and won approval for the
project from his university's committee on human-subject protection
(he had no academic affiliation when he conducted the 1980s study).
And, he says, he is no longer a member of the social-workers
association, so there are no clubs he could be kicked out of.
But even if he faces no formal disciplinary action in this round, some
of the journal editors unwittingly involved in Mr. Epstein's latest
experiment are deeply displeased.
"I think there are other ways to answer this question without being
deceitful," says Deborah P. Valentine, director of the school of
social work at Colorado State University and editor of the Journal of
Social Work Education, which flatly rejected the two fictitious papers
If she had sat on the human-subject-research committee reviewing Mr.
Epstein's proposal, she says, she would have suggested that, for
example, journal editors could allow outside evaluators to examine a
full year's worth of acceptances and rejections, to see if there was
indeed "confirmational response bias."
Because less-deceptive options were feasible, Ms. Valentine says, Mr.
Epstein's committee acted wrongly when it approved his project.
Beyond such ethical questions, Ms. Valentine and other skeptics have
raised a number of criticisms of Mr. Epstein's study. Those criticisms
and Mr. Epstein's responses to them are as follows:
Some of the peer-reviewers might have written "short, personalized"
commentaries because they knew the papers had no chance of
According to this line of argument, some reviewers might have quickly
spotted the papers' severe weaknesses, and therefore chosen not to
spend time and energy in scrutinizing the papers' statistical models.
"If a manuscript is obviously unacceptable, then it probably wouldn't
get to the point where I would check the math," Ms. Valentine says.
Mr. Epstein replies that this argument accounts for neither the four
tentative acceptances that his sham papers received nor the many
revise-and-resubmit letters that failed to acknowledge the papers'
profound flaws. He regards Ms. Valentine's argument as an evasion. "A
responsible field understands what a threat to its authority is, and
handles it very seriously," he says. "But not here."
The four-person panel of scholars that assessed the peer-review
reports was self-selected -- and it included Mr. Epstein himself.
"It's not hard to find people who share your views, or perhaps your
skepticism," says Michael J. Mahoney, a professor of psychology at the
University of North Texas and the author of Scientist as Subject: The
Psychological Imperative (Percheron Press). A different group of
people, Mr. Mahoney says, might have rated the reports very
Mr. Epstein concedes that his panel might have been skewed, and offers
to send the peer-review reports to any interested scholars who would
like to replicate his analysis. "The inadequacy of the reviews is
patent and overwhelming," he says.
Mr. Epstein's analysis of "confirmational response bias" was
hopelessly contaminated because a number of journal editors had caught
on to what he was doing.
In addition to his family-preservation papers, Mr. Epstein circulated
a pair of papers about a fictitious program to prevent juvenile
pregnancy. In this case, he was caught red-handed by a reviewer who
had been asked to scrutinize the positive version for one journal
(Affilia) and the negative version for another (Social Work). The
papers were submitted under a pseudonym, but Miriam Dinerman, editor
of Affilia, had vivid memories of Mr. Epstein's earlier hoax, and she
told several other editors to be on the lookout for odd-looking papers
about pregnancy prevention.
Mr. Epstein says that he would be happy to toss out the results from
the pregnancy-prevention papers, and limit his analysis to the
family-preservation papers. With the pregnancy papers excluded, he
says, the bias appears even stronger (albeit, he concedes, with a
smaller sample size).
Rachelle D. Hollander, a senior adviser at the National Science
Foundation, says she has mixed feelings about Mr. Epstein's project,
but she hopes that it will prompt journal editors throughout academe
to think carefully about how to improve the quality of their peer
reviews. "The fields and professional societies need to start to think
systematically about these things," she says. "They need to develop
some criteria about what they take their role as gatekeepers to be."
If "confirmational response bias" does indeed exist here, says Ron
Westrum, a professor of science, technology, and society at Eastern
Michigan University, then that is an extremely serious problem.
In the field of juvenile criminology, he says, scholars were much too
slow to take heed of studies that suggested that popular interventions
like "scared straight" programs simply didn't work. Applied fields
like social work, he says, need to devote at least as much ink to
ineffective programs as effective programs. "Social work is going to
need to find a way to respond to this," he says. "People just don't
appreciate how valuable negative information is."
Background article from The Chronicle:
* The Emperor's New Science: French TV Stars Rock the World of
Theoretical Physics (11/15/2002)
45. mailto:david.glenn at chronicle.com
E-mail me if you have problems getting the referenced articles.
More information about the paleopsych