A New Standard for Measuring Doctoral Programs

Annual index ranks departments according to their scholarly output — but its methods rankle some college officials

Related materials

Database: The 2005 Faculty Scholarly Productivity Index

Table: Difference of opinion

Table: Top departments in 104 fields, as ranked by the 2005 Faculty Scholarly Productivity Index

Table: Top research universities in the 2005 Faculty Scholarly Productivity Index

Table: The top ten institutions in six aggregated fields, as ranked by the 2005 Faculty Scholarly Productivity Index

Colloquy: Read the transcript of a live discussion with Lawrence B. Martin, graduate dean at the State University of New York at Stony Brook, about the new system he devised for ranking graduate programs.

Article tools

Printer friendly	E-mail article	Subscribe	Order reprints
	Discuss any Chronicle article in our forums
Latest Headlines San Diego State U. Defends Its Role in Federal Drug Sting The university's decision to use federal undercover drug-enforcement agents in a sting operation has brought the institution both heat and praise. Colleges Foresee Shortfall in Perkins Loan Funds This Year Student-Loan Market Shows Small Signs of Revival U.S. House Approves New GI Benefits, but Measure's Fate in Senate Is Uncertain Rutgers U. Earmark Becomes Issue, and Irony, on N.J. Campaign Trail Forum Focuses on Private Role in Expanding Global Access to Higher Education Where Students Compete 'Against the Wood' Slide Show: Photos From the Spring Meet Commentary Tell TIAA-CREF You Do Not Want Your Retirement Money to Support Genocide

By PIPER FOGG

For years when the dean of the graduate school at Washington University in St. Louis wanted to know how its programs stacked up to their peers, he was at a loss. He resorted to using U.S. News & World Report's annual rankings of graduate programs, even though many in academe pooh-pooh them as unscientific. The National Research Council's rankings of doctoral programs, considered the gold standard in academe, have not been updated since 1995.

The council's most popular rankings, which are based on reputation alone, also made him uneasy. "We really wanted objective measures," says the dean, Robert E. Thach. "We really didn't want projections, opinions to enter into it."

Mr. Thach, and other administrators who have long shared his frustration, have gotten their wish. A new annual index of graduate programs has burst onto the scene.

The Faculty Scholarly Productivity Index, partly financed by the State University of New York at Stony Brook and produced by Academic Analytics, a for-profit company, rates faculty members' scholarly output at nearly 7,300 doctoral programs around the country. It examines the number of book and journal articles published by each program's faculty, as well as journal citations, awards, honors, and grants received. The company has given The Chronicle exclusive access to some of its data, including rankings of the top 10 programs in 104 disciplines.

The most recent index, based on data from 2005, contains plenty of surprises. Some relatively unknown programs rank higher than Ivy League and other institutions with sterling reputations. Take English. The index ranks the University of Georgia at No. 2, while Columbia, Cornell, Duke, Harvard, and Yale Universities, and the Universities of Pennsylvania and of Virginia don't even crack the top 10.

Administrators are always interested in the productivity of their faculties. Harvard's former president, Lawrence H. Summers, famously alienated one of the institution's most prominent faculty members, Cornel West, in 2001 when he questioned whether Mr. West, then a professor of Afro-American studies, was producing the quality of research befitting a Harvard professor.

The new index, boasts Academic Analytics, is the first completely objective measure of productivity. And while it does not reveal data it collects on individual faculty members, its information on departments could give officials hard facts to back up sometimes unpopular decisions to reorganize programs or invest in specific fields.

The index is already attracting attention, both positive and negative, for the way it ranks programs, its occasionally curious results, and its for-profit status. Client universities, which pay up to $30,000 a year, welcome the service. They praise its objective benchmarks, its transparent methodology, and the fact that it is scheduled to come out regularly and predictably.

Mr. Thach, who signed on as a client for three years, has distributed the data to his departments and the Board of Trustees. And he has already used the data to track the effects of recent faculty hires on productivity. "If you make five senior hires," he says, "you can see the impact."

Not everyone in academe is singing the index's praises. Some graduate-school officials complain the data are flawed. Since the company compiles names of faculty members from university Web sites, which can be incomplete or outdated, these critics say the whole ranking system is skewed. Others say the index costs enough that it ought to include data on individuals. The surprisingly high rankings of some programs have made some academics skeptical, while many others turn up their noses because they view the index as too commercial.

But satisfied customers say otherwise. "Academic Analytics is so valuable," says Howard Jackson, former dean of the graduate school at the University of Cincinnati, because "assessment is in the air, and it isn't going away."

An Index Is Born

The brainchild of Lawrence B. Martin, graduate dean at the State University of New York at Stony Brook, the project is a culmination of more than a decade of studying, speaking, and writing about the productivity of faculty members. As a former member of the methodology committee for the National Research Council's next study, Mr. Martin wanted an annual, updatable index that would complement the council's data. He serves as chief scientific consultant at Academic Analytics, which was founded to commercialize his ideas.

Mr. Martin knows a thing or two about assessment. As Stony Brook's graduate dean for 13 years, he is a seasoned administrator. He has served on university committees that deal with merit pay and faculty productivity. And he is a member of a New York State Department of Education panel that focuses on assessment and institutional effectiveness. An anthropologist who studies ape and human evolution, the British-born professor also has the academic chops that his colleagues respect. That he is not seen as someone out to make a quick buck — despite being a paid consultant and owning 15 percent of the company — has helped sell the index.

Mr. Martin spent years analyzing faculty-productivity data from the 1995 National Research Council study, "Research-Doctorate Programs in the United States: Continuity and Change" which rates the quality, effectiveness, and reputation of more than 3,600 doctoral programs at 274 universities. He used those data to determine which variables would be the best predictors of a program's reputation, then created algorithms to produce his own set of rankings.

With financial backing from Stony Brook and Educational Directories Unlimited, which owns the Web site GradSchools.com, Academic Analytics was born. It produced its first index last year based on data from 2004, and covered 6,389 programs at 352 institutions. Its most recent index, which examines faculty productivity for the year 2005, has expanded to rate a total of 7,294 programs and 177,816 faculty members at 354 institutions.

Universities pay $10,000 to $30,000 a year for a three-year subscription to the service, depending on how many graduate programs they have, and get to pick 10 peer institutions with which they would like to be compared. So far, about 30 institutions have signed up.

The index relies on Scopus, a database that compiles journal publication and citation data from more than 15,000 journals, while it counts books using Amazon.com, whose database matches the Library of Congress catalog.

The index incorporates grant data collected either from federal agencies directly or from information on their Web sites, including from the National Institutes of Health, the National Science Foundation, and the National Endowment for the Humanities, among others, though notably not the Department of Defense, since it does not release data on grants to individuals.

The company collects information on honors and awards from the Web sites of 55 organizations, including those that grant Nobel Prizes and MacArthur fellowships.

The variables are weighted differently depending on the program. For example, in fields such as chemical engineering, where books are not considered a crucial measure of productivity, they are not calculated as part of the rankings. And in some fields where honors and awards are not considered important, such as animal science, those are not included.

Filling the Breach

While the Faculty Scholarly Productivity Index is not meant expressly to compete with the National Research Council study, it is hard to separate some university officials' dissatisfaction with the council from their interest in the new index.

Harvey Waterman, associate dean for academic affairs at Rutgers University at New Brunswick, serves on an advisory panel for the council's study, and is also an adviser to Academic Analytics. He says the scope of the research council study is so great — it measures more variables and collects more data on programs than Academic Analytics — that it is naturally limited in what it can do. And in preparing its updated rankings, the council ran into roadblocks before it could collect data because officials there had trouble raising enough money. "Everyone was very unhappy about the delay," says Mr. Waterman.

Once the study got enough financing, partly from government agencies and private foundations, and partly by charging participating institutions up to $20,000 each, he says, the data-collection process was rushed. "This was causing a certain amount of trouble," he adds.

"It's been frustrating," says Karen P. DePauw, dean of the graduate school at Virginia Tech. "The amount of data needed is sometimes overwhelming with the timelines."

The other problem, say graduate officials, is that some of the agencies that pay for the study wanted it to emphasize programs' reputations, as did the 1995 study, which devoted one set of rankings entirely to faculty perceptions of programs. But universities have largely objected to that emphasis, and have been anxious about which direction the new report will take.

Charlotte V. Kuh, director of the council's study, says its latest version relies less on reputations. "The measures of perceived quality are actually grounded in things that can be measured," she says, such as publications and citations. Other data will round out the picture, including information on graduation rates and how long it takes students to get their degrees.

That is a notable difference between the council's study and Academic Analytics' index. The former takes into account factors that directly influence graduate students, such as whether doctoral students get university-paid health-care benefits, whether teaching assistants have a collective-bargaining agreement, and the graduate student body's racial makeup. Both faculty members and graduate students were questioned for the council's latest study.

As for the financing delay, Ms. Kuh says, it couldn't be helped. The study was originally scheduled to be finished in 2005, but the data will not be available until the end of this year, with analytical essays and follow-up reports coming out in 2008. And now officials "are complaining because they are having to collect data that they should have had already." The process is continuing, she says, because the National Research Council is letting institutions themselves weigh in on how the productivity of their own scholars should be measured. "That takes work and that takes time," she says.

Academic Analytics' index, Ms. Kuh says, is a very different animal. In the end, she predicts, the two indexes will most likely complement each other.

The Data

Academic Analytics' clients say they know what they are getting: a focused, specific set of rankings. "It's both credible and transparent," says Cincinnati's Mr. Jackson. He notes that the company explains its methodology clearly, and gives colleges the ability to reweight rankings according to their preferences, which departments at Cincinnati have done informally. For example, if an institution wants to discount grant awards and see how its programs stack up, it can do that. Mr. Jackson says the Cincinnati graduate school has also been using the data as part of the its internal review process. That will help administrators there follow departments over a defined period of time, he hopes.

Mr. Thach, of Washington University, is particularly interested in the new index because it allows high-performing programs to be recognized before their reputations have caught up. And he has been pleased with how Washington has performed so far.

For example, in the 2005 index the university ranks seventh over all in the category of large research universities, behind only Harvard, the California Institute of Technology, the University of California at San Francisco, the Massachusetts Institute of Technology, Yale, and Carnegie Mellon University. It also ranks sixth over all in the humanities. Its political-science program ranks No. 1, not 24, where it placed in the National Research Council's 1995 study's reputational rankings. Academic Analytics' index ranks Washington's English program at fourth place, compared with 50th place in the 1995 council study. Its French program comes in fourth according to the 2005 index, compared with 17th in the 1995 study.

"Reputations take decades to build," notes Mr. Thach, while Academic Analytics' rankings recognize programs that are up and coming.

But a close look at other data in the index has some college officials raising their eyebrows. The University of Georgia's No. 2 ranking in English, for instance, has caused some scoffing. Say you have an exceptionally bright undergraduate poised to enter graduate school in English, says one university administrator who did not want to be named: "Would you really recommend the person attend the University of Georgia? It's where this unidimensional figure gets out of touch."

There are anomalies in other fields too. In economics the 1995 council study put Stanford, the University of California at Berkeley, and Northwestern University in the top 10 of its reputational ranking. That was compiled after asking nearly 8,000 graduate -school professors their opinions of the scholarly quality of programs' faculties and their effectiveness in educating students. Academic Analytics' index omits all three institutions from its economics top 10, which include the University of Wisconsin at Madison and the University of California at San Diego.

In political science the 1995 council study included Berkeley, Stanford, the University of Chicago, and the University of California at Los Angeles in its top 10. None of those appears in the 2005 index's top-10 list, though the University of Kansas and the University of Maryland at College Park do. And in physics, the council put Princeton at No. 2 and MIT at No. 3. Neither make Academic Analytics' top 10.

Lydia S. Snover, director of institutional research at MIT, explains that Academic Analytics' methodology may be part of the problem.

Because the company counts only faculty members listed under specific departments, it missed some scientists connected to MIT's numerous interdisciplinary research centers. Some of those scholars are the principal investigators on multimillion-dollar grants that other physics faculty members also work on, says Ms. Snover. But those grants and the scholars' publications did not get counted in the physics rankings.

Mr. Martin counters that faculty members who are purposely not listed on a program's Web site should not be counted, because if they were heavily involved with graduate students, an important criterion for being part of a graduate program, they would be listed.

Imperfect Methods?

Other critics agree that Academic Analytics has compiled flawed faculty lists. The company uses a Web-crawling service to pull names from department Web sites and then sends each university the list, hoping administrators will send back a corrected version. Last year 133 institutions complied, but the other 141 institutions in the index did not. Some universities refused to do work for a service they were not paying for, especially one that others were profiting from.

Carol B. Lynch, a senior scholar in residence at the Council of Graduate Schools and a former graduate dean at the University of Colorado at Boulder who is on the National Research Council's oversight committee, finds the company's approach misguided. "I'm skeptical of a competitor that feels they can just sort of pull things out of the air," she says. "That's probably a little bit naïve."

Alison Power, graduate dean at Cornell, is likewise wary of Academic Analytics' data-collection methods. "They don't have very good checks," she says. A lot of potential customers saw the early mistakes in the faculty lists, and the time commitment that would be involved in correcting them, and balked.

"People just kind of said, No, forget it," Ms. Power says. "And so people aren't taking it seriously."

Other universities would consider signing up, though, if the methods changed slightly. Julie Carpenter-Hubin, director of institutional research and planning at Ohio State University, was originally intrigued by the index. But she did not want to shell out tens of thousands of dollars unless she could get the names of individual faculty members connected to their scholarly output.

Information the company collects on individual faculty members is aggregated before being published, Mr. Martin says, because such raw data are vulnerable to misinterpretation and misuse.

Ms. Snover at MIT adds that there is still the question of how to count publications. Academic Analytics puts great weight on faculty members' publication records, but those counts don't necessarily translate to the highest-quality work.

Mr. Martin says citations and grants do indicate quality, so the results should balance out. Some officials counter that no adequate citation index exists for the humanities, and Ms. Snover, for one, remains unconvinced. "I think reputational surveys do have a function in these kinds of rankings," she says.

Mr. Martin says that Academic Analytics' data cannot present a perfect reflection of every faculty member's scholarly impact. By taking a particular view of productivity, though, the data can indicate important information about the quality of programs as a whole.

"We think it provides significant contextual information," he says. "Are there some counterintuitive results? Absolutely," he says. But that just means it might be worth taking a second look at the University of Georgia's English department, for instance, to see what's going on there.

As for the complaints he has heard about the for-profit nature of the company, Mr. Martin says, Academic Analytics is aiming for only a 10- to 15-percent profit margin, if it can get about 50 clients total. "I'm a dean. I didn't go into academe to make a lot of money," he says, but "somebody's got to pay the bills."

Ben Ware, dean of the graduate school at Syracuse University, who has subscribed to the index, says that mistakes are a natural part of such rankings. "It's very difficult to make measurements with no errors," he says.

But as long as one keeps the magnitude of error in mind, he adds, universities can still learn valuable things from the information.

For example, deans can learn whether new hires have improved the publication or grant records of specific programs. They can predict how some departments' productivity might change with impending retirements. The data could also identify general areas where there might be deadwood worth culling.

Virginia Tech's Ms. DePauw agrees. "We can criticize a process," she says, but she prefers to work with Academic Analytics to get the best from it. "I'm not into it for precise rankings or precise comparisons," she says. Besides, her faculty members know which departments are the top in their fields anyway. She's more concerned with tracking internal trends.

Once professors and administrators see some of the 2005 data, they will have a chance to reconsider the index's worth, and some naysayers may be won over. But only after the National Research Council comes out with the next installment of its rankings will it become clear how seriously academe will take Academic Analytics. For now, it's the only game in town.

Richard Wheeler, dean of the Graduate College at the University of Illinois at Urbana-Champaign and a member of the National Research Council's oversight committee, at first denigrated Academic Analytics. "It just doesn't inspire confidence in me," he said.

But after being told how his university performed in the 2005 rankings, his tone shifts slightly: "If it turns out we do very well in them, I'll probably think they're great."