[Paleopsych] CHE: New Federal Policy Favors Randomized Trials in Education Research

[Premise Checker]

New Federal Policy Favors Randomized Trials in Education Research
The Chronicle of Higher Education, 5.3.11
http://chronicle.com/weekly/v51/i27/27a01602.htm

    By DAVID GLENN

    The U.S. Education Department's controversial efforts to foster what
    it describes as "scientifically based research" in education reached a
    small milestone in late February, when new regulatory language took
    effect. The new policy will permit -- but not require -- all of the
    department's units to give preference to grant applicants who promise
    to use randomized controlled trials or similar quasi-experimental
    methods.

    Known in the jargon of federal regulation as a grant-making
    "priority," the new rule will initially be applied to a small number
    of the department's discretionary grant programs, officials said.

    The department hopes to build a body of experimental data, "so that
    three years from now, we can look across our portfolio and know which
    programs are actually worth replicating and disseminating," said
    Michael J. Petrilli, an associate assistant deputy secretary in the
    department's Office of Innovation and Improvement.

    The growing federal emphasis on randomized trials represents a victory
    for certain critics of education research. The critics have charged
    that most such research is watery and narrowly descriptive, and does
    little to inform the public about what types of classroom practices
    improve student performance. It is much better, the critics say, to
    use medical-style trials, in which, for example, certain classrooms
    are randomly assigned to use a standard mathematics curriculum while
    other classrooms use an experimental technique.

    The department's new National Board for Education Sciences includes
    many of the most prominent critics, including Eric A. Hanushek, of
    Stanford University, and Caroline Minter Hoxby, of Harvard University.
    The board held its inaugural meeting earlier this month.

    Complaints From Researchers

    Other researchers, however, have strenuously opposed the new emphasis
    on experiments. After the Education Department solicited comments on a
    draft of the new regulation in late 2003, nearly 200 scholars sent in
    objections. They argued that randomized trials were expensive and
    difficult to conduct on a meaningful scale, that experiments involving
    children posed ethical problems, and that the draft failed to
    appreciate the value of descriptive, qualitative classroom studies.

    "We had some major concerns," noted Gerald E. Sroufe, director of
    government relations at the American Educational Research Association.
    Some scholars, he said, "are almost giddy about randomization."

    Mr. Sroufe did concede that some degree of emphasis on clinical trials
    was important -- "I know that we've been negligent as a field" -- but
    he added that he believed the department was neglecting many of the
    traditional tools of evaluation.

    Other scholars agreed that the new emphasis could hurt some areas of
    research.

    "People I work with in evaluation are asking questions that truly
    cannot be answered through randomized trials," said Sharon F. Rallis,
    president of the American Evaluation Association.

    For example, she said, some state departments of education are
    studying the effects of moving cognitively disabled students into
    mainstream classrooms. "You can't use randomized trials to study that
    because, first, you just don't have the numbers there" to generate
    statistically valid results, said Ms. Rallis, who is also a professor
    of education at the University of Connecticut.

    Ms. Rallis said she was disappointed that the language contained in
    the regulation was not significantly revised from the initial draft,
    which her organization criticized in a widely distributed statement in
    2003.

    The department's Mr. Petrilli agreed that the new rule would be
    appropriate only in limited areas. "It was never our intention to use
    this for every program," he said. "It's going to be a tiny minority of
    programs for which we use this priority."

    Mr. Sroufe, of the education-research association, said that he was
    skeptical about that. "There are ways of writing regulations to
    suggest that you're only going to use something on occasion," he said.
    "And I just can't see that in this language."

    He added that he was disappointed that the new regulation did not
    insist that third-party evaluators be independent from the grant
    applicants. (It says only that such evaluators are "preferably
    independent.") "Most people would say that that's the sine qua non,"
    he said. "I don't know why they didn't just demand that."

    Frederick M. Hess, director of education-policy studies at the
    American Enterprise Institute, a Washington think tank, said he had
    mixed feelings about the new language. The turn toward randomized
    trials, he said, is "entirely appropriate and sorely overdue" in
    certain areas, such as studies of reading curricula and other
    classroom practices.

    Mr. Hess said that he worried, however, that certain education
    researchers might have embraced a "gold-rush mentality" that led them
    to propose randomized trials for what he regards as inappropriate
    topics, such as incentive-pay plans for teachers.

    Mr. Hess sits on review panels for the department, and in certain
    recent grant applications he has perceived that "some people are
    moving forward almost too enthusiastically," he said. A small
    randomized study of incentive-pay plans, he said, would be unlikely to
    produce generalizable results or to be persuasive to people on either
    side of the incentive-pay debate.