[Paleopsych] CHE: Scholar Sends Sham Papers to Social-Work Journals to Show Weakness of Peer Review

[Premise Checker]

Scholar Sends Sham Papers to Social-Work Journals to Show Weakness of Peer
Review
News bulletin from the Chronicle of Higher Education, 4.10.20
http://chronicle.com/prm/daily/2004/10/2004102003n.htm
[45]By DAVID GLENN

    Most scholars would let out a groan if a journal returned their
    submissions with peer-review comments like these: "The author presents
    us with a flawed study that is not well analyzed. ... The reviewer has
    no idea what statistical tests were conducted, how many of them were
    done, and what those findings were."

    William M. Epstein, however, was delighted by those comments, and
    wishes that he'd gotten many more like them.

    Two years ago, Mr. Epstein, a professor of social work at the
    University of Nevada at Las Vegas, submitted fictitious, pseudonymous,
    deliberately shoddy papers to 33 journals in his field. "The basic
    designs were off," he says. "I drew conclusions that were unwarranted
    on the basis of the data. I never brought up statistical tests -- I
    just simply announced that there was significance."

    Even the most quotidian passages were mined with errors. One paper
    concluded by saying, "The average cost of care for control clients was
    about $576 and for experimental clients about $3,975. The $3,419
    difference seems well justified by the outcomes." (The difference
    between those costs is actually $3,399.)

    In sum, Mr. Epstein says, "a careful reviewer would have butchered
    these papers."

    No journal came very close to publishing any of the sham articles, but
    Mr. Epstein believes that his experiment nonetheless uncovered some
    deep intellectual problems in the world of social-work scholarship. He
    explains his deed -- and lays out his accusations against his field --
    in an article published this month in the journal Research on Social
    Work Practice, which was among the journals hit by his hoax.

    Mr. Epstein's targets are not taking this lightly. Some of them say
    that his project was needlessly deceptive, and that it was itself
    biased and poorly designed.

    The first charge in Mr. Epstein's indictment is that social-work
    journals suffer from "confirmational response bias." By that he means
    that the journals are more inclined to publish papers that support
    social workers' typical methods than papers that raise doubts.

    To test that hypothesis, Mr. Epstein circulated a pair of articles
    about a fictitious program to preserve families whose children are at
    risk of being placed in foster care. The "positive" version of the
    article said that the program worked well; the "negative" version said
    that the program had no effect. Both versions were methodologically
    shoddy, and they were identical except for a few paragraphs in the
    findings section.

    Half of the journals in Mr. Epstein's experiment were randomly
    selected to receive the positive version, and the other half received
    the negative version. The positive version got a much warmer welcome.
    The negative version received seven flat rejections, but the positive
    version received only two. And the positive version received four
    acceptances -- in each case, conditioned on various revisions -- while
    the negative version received none.

    In Mr. Epstein's eyes, social work -- together with other
    practice-oriented fields, like counseling, management, and clinical
    psychology -- is too reluctant to publish information about techniques
    that don't seem to work. There is a bias, he says, toward "happy
    little news."

    Mr. Epstein's second charge is that the general quality of peer review
    in social-work journals is feeble. He and three anonymous colleagues
    evaluated the thoroughness and sophistication of the peer-review
    reports generated by the sham articles. Mr. Epstein's panel deemed
    almost three-quarters of the reviews to be inadequate; in his paper,
    he describes some of them as "incomplete, short, personalized, and
    even impenetrable."

    Very few of the reviews, he says, pointed out that the papers lacked
    coherent accounts of their statistical tests.

    "When you take a look at the appalling quality of research in the
    field," Mr. Epstein says, "it's really quite obvious that
    methodological sophistication is not of paramount interest."

    This is not Mr. Epstein's first foray into surreptitious research. In
    the late 1980s he conducted a very similar project, and it landed him
    in hot water. A board of inquiry at the National Association of Social
    Workers voted in 1988 to censure him for failing to obtain his
    subjects' informed consent, although that decision was later
    overturned by the association's National Board of Directors.

    This time, Mr. Epstein says, he sought and won approval for the
    project from his university's committee on human-subject protection
    (he had no academic affiliation when he conducted the 1980s study).
    And, he says, he is no longer a member of the social-workers
    association, so there are no clubs he could be kicked out of.

    But even if he faces no formal disciplinary action in this round, some
    of the journal editors unwittingly involved in Mr. Epstein's latest
    experiment are deeply displeased.

    "I think there are other ways to answer this question without being
    deceitful," says Deborah P. Valentine, director of the school of
    social work at Colorado State University and editor of the Journal of
    Social Work Education, which flatly rejected the two fictitious papers
    it received.

    If she had sat on the human-subject-research committee reviewing Mr.
    Epstein's proposal, she says, she would have suggested that, for
    example, journal editors could allow outside evaluators to examine a
    full year's worth of acceptances and rejections, to see if there was
    indeed "confirmational response bias."

    Because less-deceptive options were feasible, Ms. Valentine says, Mr.
    Epstein's committee acted wrongly when it approved his project.

    Beyond such ethical questions, Ms. Valentine and other skeptics have
    raised a number of criticisms of Mr. Epstein's study. Those criticisms
    and Mr. Epstein's responses to them are as follows:

    Some of the peer-reviewers might have written "short, personalized"
    commentaries because they knew the papers had no chance of
    publication.

    According to this line of argument, some reviewers might have quickly
    spotted the papers' severe weaknesses, and therefore chosen not to
    spend time and energy in scrutinizing the papers' statistical models.
    "If a manuscript is obviously unacceptable, then it probably wouldn't
    get to the point where I would check the math," Ms. Valentine says.

    Mr. Epstein replies that this argument accounts for neither the four
    tentative acceptances that his sham papers received nor the many
    revise-and-resubmit letters that failed to acknowledge the papers'
    profound flaws. He regards Ms. Valentine's argument as an evasion. "A
    responsible field understands what a threat to its authority is, and
    handles it very seriously," he says. "But not here."

    The four-person panel of scholars that assessed the peer-review
    reports was self-selected -- and it included Mr. Epstein himself.

    "It's not hard to find people who share your views, or perhaps your
    skepticism," says Michael J. Mahoney, a professor of psychology at the
    University of North Texas and the author of Scientist as Subject: The
    Psychological Imperative (Percheron Press). A different group of
    people, Mr. Mahoney says, might have rated the reports very
    differently.

    Mr. Epstein concedes that his panel might have been skewed, and offers
    to send the peer-review reports to any interested scholars who would
    like to replicate his analysis. "The inadequacy of the reviews is
    patent and overwhelming," he says.

    Mr. Epstein's analysis of "confirmational response bias" was
    hopelessly contaminated because a number of journal editors had caught
    on to what he was doing.

    In addition to his family-preservation papers, Mr. Epstein circulated
    a pair of papers about a fictitious program to prevent juvenile
    pregnancy. In this case, he was caught red-handed by a reviewer who
    had been asked to scrutinize the positive version for one journal
    (Affilia) and the negative version for another (Social Work). The
    papers were submitted under a pseudonym, but Miriam Dinerman, editor
    of Affilia, had vivid memories of Mr. Epstein's earlier hoax, and she
    told several other editors to be on the lookout for odd-looking papers
    about pregnancy prevention.

    Mr. Epstein says that he would be happy to toss out the results from
    the pregnancy-prevention papers, and limit his analysis to the
    family-preservation papers. With the pregnancy papers excluded, he
    says, the bias appears even stronger (albeit, he concedes, with a
    smaller sample size).

    Rachelle D. Hollander, a senior adviser at the National Science
    Foundation, says she has mixed feelings about Mr. Epstein's project,
    but she hopes that it will prompt journal editors throughout academe
    to think carefully about how to improve the quality of their peer
    reviews. "The fields and professional societies need to start to think
    systematically about these things," she says. "They need to develop
    some criteria about what they take their role as gatekeepers to be."

    If "confirmational response bias" does indeed exist here, says Ron
    Westrum, a professor of science, technology, and society at Eastern
    Michigan University, then that is an extremely serious problem.

    In the field of juvenile criminology, he says, scholars were much too
    slow to take heed of studies that suggested that popular interventions
    like "scared straight" programs simply didn't work. Applied fields
    like social work, he says, need to devote at least as much ink to
    ineffective programs as effective programs. "Social work is going to
    need to find a way to respond to this," he says. "People just don't
    appreciate how valuable negative information is."
      _________________________________________________________________

    Background article from The Chronicle:
      * [55]The Emperor's New Science: French TV Stars Rock the World of
        Theoretical Physics (11/15/2002)

References

   45. mailto:david.glenn at chronicle.com
   55. http://chronicle.com/weekly/v49/i12/12a01601.htm

E-mail me if you have problems getting the referenced articles.