[Paleopsych] CHE: New Federal Policy Favors Randomized Trials in Education Research
checker at panix.com
Wed Apr 13 18:14:01 UTC 2005
New Federal Policy Favors Randomized Trials in Education Research
The Chronicle of Higher Education, 5.3.11
By DAVID GLENN
The U.S. Education Department's controversial efforts to foster what
it describes as "scientifically based research" in education reached a
small milestone in late February, when new regulatory language took
effect. The new policy will permit -- but not require -- all of the
department's units to give preference to grant applicants who promise
to use randomized controlled trials or similar quasi-experimental
Known in the jargon of federal regulation as a grant-making
"priority," the new rule will initially be applied to a small number
of the department's discretionary grant programs, officials said.
The department hopes to build a body of experimental data, "so that
three years from now, we can look across our portfolio and know which
programs are actually worth replicating and disseminating," said
Michael J. Petrilli, an associate assistant deputy secretary in the
department's Office of Innovation and Improvement.
The growing federal emphasis on randomized trials represents a victory
for certain critics of education research. The critics have charged
that most such research is watery and narrowly descriptive, and does
little to inform the public about what types of classroom practices
improve student performance. It is much better, the critics say, to
use medical-style trials, in which, for example, certain classrooms
are randomly assigned to use a standard mathematics curriculum while
other classrooms use an experimental technique.
The department's new National Board for Education Sciences includes
many of the most prominent critics, including Eric A. Hanushek, of
Stanford University, and Caroline Minter Hoxby, of Harvard University.
The board held its inaugural meeting earlier this month.
Complaints From Researchers
Other researchers, however, have strenuously opposed the new emphasis
on experiments. After the Education Department solicited comments on a
draft of the new regulation in late 2003, nearly 200 scholars sent in
objections. They argued that randomized trials were expensive and
difficult to conduct on a meaningful scale, that experiments involving
children posed ethical problems, and that the draft failed to
appreciate the value of descriptive, qualitative classroom studies.
"We had some major concerns," noted Gerald E. Sroufe, director of
government relations at the American Educational Research Association.
Some scholars, he said, "are almost giddy about randomization."
Mr. Sroufe did concede that some degree of emphasis on clinical trials
was important -- "I know that we've been negligent as a field" -- but
he added that he believed the department was neglecting many of the
traditional tools of evaluation.
Other scholars agreed that the new emphasis could hurt some areas of
"People I work with in evaluation are asking questions that truly
cannot be answered through randomized trials," said Sharon F. Rallis,
president of the American Evaluation Association.
For example, she said, some state departments of education are
studying the effects of moving cognitively disabled students into
mainstream classrooms. "You can't use randomized trials to study that
because, first, you just don't have the numbers there" to generate
statistically valid results, said Ms. Rallis, who is also a professor
of education at the University of Connecticut.
Ms. Rallis said she was disappointed that the language contained in
the regulation was not significantly revised from the initial draft,
which her organization criticized in a widely distributed statement in
The department's Mr. Petrilli agreed that the new rule would be
appropriate only in limited areas. "It was never our intention to use
this for every program," he said. "It's going to be a tiny minority of
programs for which we use this priority."
Mr. Sroufe, of the education-research association, said that he was
skeptical about that. "There are ways of writing regulations to
suggest that you're only going to use something on occasion," he said.
"And I just can't see that in this language."
He added that he was disappointed that the new regulation did not
insist that third-party evaluators be independent from the grant
applicants. (It says only that such evaluators are "preferably
independent.") "Most people would say that that's the sine qua non,"
he said. "I don't know why they didn't just demand that."
Frederick M. Hess, director of education-policy studies at the
American Enterprise Institute, a Washington think tank, said he had
mixed feelings about the new language. The turn toward randomized
trials, he said, is "entirely appropriate and sorely overdue" in
certain areas, such as studies of reading curricula and other
Mr. Hess said that he worried, however, that certain education
researchers might have embraced a "gold-rush mentality" that led them
to propose randomized trials for what he regards as inappropriate
topics, such as incentive-pay plans for teachers.
Mr. Hess sits on review panels for the department, and in certain
recent grant applications he has perceived that "some people are
moving forward almost too enthusiastically," he said. A small
randomized study of incentive-pay plans, he said, would be unlikely to
produce generalizable results or to be persuasive to people on either
side of the incentive-pay debate.
More information about the paleopsych