Washington
The National Research Council will "shortly" release a revised version of its enormous database on American doctoral education, the project's leaders said here Friday during a conference on the past and future of doctoral-program assessment. In at least a few cases, the officials said, the revisions will significantly change certain programs' rankings.
After the NRC's long-delayed report was released last September, scholars in some fields—notably computer science —said they saw serious errors in the data.
In response to such concerns, the NRC invited programs to formally submit any problems or mistakes they had detected. Of the nearly 5,000 programs in the study, 453 submitted such letters before the November deadline, said the project's staff director, Charlotte V. Kuh, during Friday's meeting.
Many of those letters had to do with errors that universities themselves may have made in filling out the project's surveys in 2006 and 2007—for example, listing incorrect rosters of faculty members for a program. In those cases, Ms. Kuh said, the NRC would not make changes at this late date. But there were three areas, she said, in which the project committee was persuaded that it had made errors—and it is those areas that will be corrected in the forthcoming revision.
First, she said, the committee had wrongly excluded some faculty honors and awards. Second, there was an error in tabulating citations for papers published in 2002. Finally, the committee has determined that some of the report's figures for programs' academic job-placement rates were inaccurate and misleading. For those placement figures, the NRC has relied on data from the National Science Foundation's Survey of Earned Doctorates. That survey has a generally high response rate, but for some specific programs, the response rate was low. In its revised report, the NRC will use the total number of survey respondents for each program, not the total number of doctoral graduates, as the denominator in its placement-rate calculations. For example, a program might now be credited with placing 30 out of 35 students, instead of 30 out of 60.
"What will the effect of those changes be? I don't know yet," Ms. Kuh said during her conference talk. "If they're systematic and across the board," she said, they might not affect programs' relative rankings, because the revisions will affect all programs more or less equally. But in at least "a few cases that we know of," she added, the revisions will, in fact, cause "substantial" changes in programs' relative rankings.
In an interview after the conference, Ms. Kuh said that she could not say exactly when the revisions will be released but that it will be soon.
Flawed but Helpful
For all its imperfections, the NRC report has provided badly needed tools for improving doctoral education, several university officials said Friday.
Shelly Conner, an assistant dean at the University of Michigan at Ann Arbor's Rackham Graduate School, said that she and her colleagues have not put much weight on the rankings, but they have been interested in the NRC report's revelations about how programs perform in specific domains. Until the NRC report appeared last fall, for example, no one at Michigan had been aware that the astronomy department's average time to degree was a year and a half longer than those of its peers.
Officials from Howard University, the Johns Hopkins University, and the Massachusetts Institute of Technology also spoke about plans to use the NRC data to improve individual programs and to make strategic decisions about graduate education on their campuses.
Will there ever be another iteration of the NRC study? If so, who would pay for it, and how should it be designed?
No one seemed eager to replicate the last round, especially given the extraordinary length of time it took to complete. But several scholars said they believed the NRC could still play a constructive role by facilitating a database that could be updated every two or three years, even if no ratings or rankings were involved.
A few people said they had reluctantly concluded that it was a mistake for the NRC to move so far away from direct "reputational" measures based on faculty surveys. "I could never have imagined myself saying this until very recently," said Richard P. Wheeler, the interim vice chancellor of academic affairs at the University of Illinois at Urbana-Champaign, who was a member of the NRC project committee. "But I think we should reintroduce some kind of direct reputational measure."
It is true that reputational surveys suffer from "halo effects" and other pathologies, Mr. Wheeler said. But at the same time, reputation offers information and nuance that "people just know things about other departments that are never going to show up in the data, no matter how much data you have," he said. For example, the faculty rumor mill might accurately convey how relaxed the research-and-publication standards are for students in a particular program.
Two speakers associated with the Association of American Universities Data Exchange offered a strong critique of the NRC's data models. Lou McClelland, director of institutional analysis at the University of Colorado at Boulder, and Julie Carpenter-Hubin, director of institutional research and planning at Ohio State University, criticized each of the NRC's 21 major variables.
Among other things, Ms. McClelland and Ms. Carpenter-Hubin said that several of the variables probably have a curvilinear relationship to program quality rather than a simple linear relationship. In other words, the optimal value is probably somewhere in the middle. An average time-to-degree of nine years might reflect serious problems, but a time-to-degree of four years might mean that a program is too easy. But the NRC's analyses, they noted, simply assumed that one end of the scale was best.
A full Webcast of Friday's conference is available.
