[Paleopsych] CHE: Scholars Say College Admissions Offices Misuse Advanced Placement Data

[Premise Checker]

Scholars Say College Admissions Offices Misuse Advanced Placement Data
The Chronicle of Higher Education, 5.1.21
http://chronicle.com/weekly/v51/i20/20a02701.htm

    A study finds that the high-school courses aren't always good
    predictors of college success
    By DAVID GLENN
    Philadelphia

    College-admissions officers should be cautious when weighing Advanced
    Placement courses on applicants' high-school transcripts, according to
    a working paper presented here at the annual meeting of the American
    Economic Association this month.
    The paper asserts that the mere act of taking AP courses in high
    school -- as distinct from scoring well on the official AP tests
    -- does not predict that a student will perform well in college.
    The paper's authors -- Kristin Klopfenstein, an assistant professor of
    economics at Texas Christian University, and M. Kathleen Thomas, an
    assistant professor of economics at Mississippi State University
    -- analyzed the records of more than 28,000 students who graduated
    from Texas high schools in 1999 and who enrolled that fall in the
    state's four-year public universities.
    The researchers posed two central questions: Did the Texas students
    who took Advanced Placement courses in high school have higher grades
    in their first year of college than their non-AP peers? And were they
    less likely to drop out before their sophomore year?
    The answers turned out to be no and no. In Ms. Klopfenstein and Ms.
    Thomas's statistical analysis -- described in a paper titled "The
    Advanced Placement Performance Advantage: Fact or Fiction?" -- all of
    the variance in the students' grade-point averages and dropout rates
    was explained by the familiar predictors of college performance:
    high-school grade-point averages, SAT scores, parents' education and
    income, and the proportion of experienced teachers in the students'
    high schools. After such variables were accounted for, the AP courses
    on the students' transcripts did not have any extra predictive power
    of their own.
    "AP experience may serve as a signal of high ability and motivation,"
    the authors write, "but it does not by itself indicate superior
    academic readiness."
    Quality vs. Quantity
    Most of the best-known research about the Advanced Placement program
    -- including studies conducted under the auspices of the College
    Board, which creates and administers the official AP tests -- has
    tried to answer a different set of questions. Those studies have
    generally found that AP tests are a reliable equivalent to first-year
    college examinations, and that institutions should feel comfortable in
    awarding college credit to students who score well on the tests.
    The new Texas study does not challenge such classic findings about the
    AP program. Instead, the authors express concern that some school
    districts and college-admissions offices behave as if simply taking an
    AP course, regardless of one's performance on the AP test, will help a
    student do better in college.
    In an interview, Ms. Thomas said that she generally supports the
    recent expansion of the AP program and similar efforts to make
    high-school curricula more rigorous. She worries, however, that some
    school districts may be paying too little attention to the quality of
    their AP courses, and that high schools and colleges may have policies
    that are not warranted by solid evidence.
    In particular, Ms. Thomas cited the University of California's policy
    of giving extra weight to AP courses when calculating applicants'
    high-school grade-point averages. The university considers an A in an
    AP course to be worth 5.0 points, as opposed to 4.0 for an A in a
    standard course. The university grants such extra weight whether or
    not the student chose to take the official AP test at the end of the
    course.
    Ms. Thomas fears that admissions policies like California's
    -- together with various incentives offered by federal and state
    governments -- might be prompting some high schools to slap together
    ill-designed AP programs with poorly trained teachers.
    The AP program has expanded very rapidly. In 1980-81, according to the
    College Board, 133,702 high-school students took Advanced Placement
    exams. In 2003-4, the number was 1,017,396.
    "We suspect that there's a lot of variance in the quality of AP
    programs," Ms. Thomas said. "As schools move to increase access,
    unless their principals and administrators really understand what goes
    into developing that program, we can just envision cases where
    suddenly a teacher hears, 'Guess what? You're the new AP teacher.'"
    Many districts have moved to add AP courses, Ms. Thomas said, in the
    wake of a 1999 study by Clifford Adelman, a senior research analyst at
    the U.S. Department of Education, who found that "intensity of
    high-school curriculum" is a strong predictor of students' college
    performance.
    Ms. Thomas does not dispute Mr. Adelman's general insight, but she
    argues that for certain districts with large numbers of struggling
    students, AP courses might not be the best way to make the curriculum
    more rigorous. AP courses often require teachers to move rapidly
    through a huge amount of material, she said, and "sometimes covering a
    topic in depth is sacrificed for covering breadth."
    Predicting Success
    The Texas study comes on the heels of a similar paper that was
    released last month by scholars at the University of California. Saul
    Geiser, a visiting scholar at the University of California at
    Berkeley's Center for Studies in Higher Education, and Veronica
    Santelices, a Ph.D. candidate at the University of California at
    Berkeley's School of Education, analyzed the records of 81,445
    students who enrolled as freshmen in the university system from 1998
    to 2001.
    Mr. Geiser and Ms. Santelices found that the admissions program's
    "bonus point" for high-school AP courses "bears little or no
    relationship to college performance." Like Ms. Klopfenstein and Ms.
    Thomas, the California scholars found that simply taking AP courses is
    not a valid predictor of college grades or persistence.
    The California authors also found, however, that the subgroup of
    students who took the official AP tests, and did well on them,
    performed very well in college. Strong scores on AP tests, in fact,
    were a better predictor of students' sophomore-year grade-point
    averages than any other variable except their high-school grade-point
    averages. (Ms. Klopfenstein and Ms. Thomas were unable to perform a
    similar analysis because they did not have access to the Texas
    students' AP-test records. The College Board might soon provide them
    with those data, however.)
    At the conference panel here, Ronald G. Ehrenberg, a professor of
    industrial and labor relations and of economics at Cornell University,
    and director of the Cornell Higher Education Research Institute, cited
    the importance of Ms. Klopfenstein and Ms. Thomas's line of research.
    He also said, however, that the study's design might have missed some
    important potential effects of taking Advanced Placement courses. For
    example, he asked, "What's the impact of having taken an AP class, or
    classes, on where students go to college?"
    In a telephone interview, Trevor Packer, executive director of the
    College Board's Advanced Placement program, said that while he has
    serious reservations about the Texas paper's methodology, he welcomes
    both the Texas and California studies.
    "More and more people have felt that AP classes are something on a
    student's transcript that would be meaningful in the admissions
    process," he said.
    That belief, he continued, should be scrutinized carefully because the
    College Board has established only the much narrower claim that
    students who score well on AP tests should be awarded college credit
    in those subjects. The Texas and California studies, he said, might be
    useful steps toward assessing whether taking AP courses, per se, is an
    accurate predictor of college performance.
    Mr. Packer said he wished, however, that the Texas paper had
    distinguished between students who did well on AP tests and those who
    did not. "Until Klopfenstein and Thomas separate their sample into at
    least two different groups," he said, "it's impossible for them to
    make this claim that they're making that AP doesn't impact overall
    college success."
    Further Analysis
    Ms. Thomas said that she and Ms. Klopfenstein are eager to conduct
    such a study, and hope to receive data from the College Board that
    would allow them to do so. But she also said that it was valid for
    them to look at AP students as a single group, without regard to their
    test scores, because that is exactly what some important institutional
    actors, including the University of California admissions offices,
    have done.
    Mr. Packer said he also worries that the Texas study's assessment of
    dropout rates might be flawed because Ms. Klopfenstein and Ms. Thomas
    considered all students who disappeared from the Texas records in 2000
    as having dropped out. It is possible, Mr. Packer said, that some of
    those students might have transferred to private colleges or
    transferred out of state.
    Ms. Thomas replied that national statistics suggest that very few
    Texas students were mislabeled in that way. In any case, she added,
    such a flaw would not affect her analysis of students' first-year
    grade-point averages.
    As for Ms. Klopfenstein and Ms. Thomas's concern that the quality of
    AP courses is sometimes weak, Mr. Packer conceded that it is a
    legitimate worry, but said that more-detailed studies were needed.
    "What would really help administrators," he said, "would be to
    understand what sort of AP classrooms do impact students' college
    success."
    "Administrators need research-based help in understanding what
    configuration of the AP classroom is helpful for expanding access and
    providing increased opportunity and inclusivity, without watering down
    the course," he said.
    Such studies are indeed planned by Ms. Klopfenstein and Ms. Thomas.
    They would like to use their Texas data to identify particular AP
    programs that perform improbably well, given the school district's
    resources or demographics. Ms. Thomas said that studies of such
    programs could explain how they succeed, providing a lesson for other
    districts.
    The College Board, meanwhile, plans to publish new guidelines this
    year about the training that AP teachers should receive, Mr. Packer
    said.
    Ms. Klopfenstein and Ms. Thomas's paper has been submitted to a
    journal but has not yet completed the peer-review process.