NIH Mulls Ways to Lure Back Veteran Peer Reviewers

Many scientists now decline what they say has become a dreary job, putting the quality of U.S.-backed research at risk

Related materials

Table: More NIH Applications, More Reviewers

Article tools

Printer friendly	E-mail article	Subscribe	Order reprints
	Discuss any Chronicle article in our forums
Latest Headlines Universities Must Disclose More Data on Animal Research A court settlement requires the U.S. Department of Agriculture to make public more information about research on animals, prompting some academic researchers to worry about further attacks by animal-rights extremists. 2-Year Colleges Streamline Student Aid and Focus on Counseling For Provost Who Fled Lebanon, U. of Dayton Is His 'Village' Letters Home From World War II Soldiers Are Found in College Basement

By JEFFREY BRAINARD

Bethesda, Md.

Not long ago, academic scientists welcomed calls from the National Institutes of Health asking them to volunteer as peer reviewers.

Many were glad for the opportunity to help distribute billions of dollars in federal biomedical-research grants even though the service required a big time commitment — the equivalent of one month a year to evaluate grant applications and travel three times annually to the NIH's headquarters here for meetings of peer-review committees.

After all, the scientists reaped personal benefits. The invitation offered recognition that they were near or at the top of their fields. At the meetings, the reviewers got a stimulating, front-row view of their discipline's cutting edge. Afterward, they enjoyed shop talk and camaraderie over drinks and dinners. And as important gatekeepers in the NIH's system for selecting the best proposals, they felt they were serving their profession and their country.

While those features of service remain, some scientists say it seems more and more like a frustrating, even dreary chore. They argue that the agency is struggling to recruit the most-talented reviewers in some fields — and that as a result, the quality of science supported by the NIH, the largest single source of funds for academic research, is at risk.

One culprit for that shift in attitude is a decline in the agency's budget since 2003, which in turn has reduced the percentage of grant applications awarded money. That makes many reviewers feel that the long hours of work they donate are for naught.

"You had a feeling you were helping people," says Gregory A. Petsko, a professor of biochemistry and chemistry at Brandeis University who continues to serve on the review panels. "The fun is totally gone when your job is to turn down seven out of every eight grant applications."

NIH officials acknowledge that they have at times strained to recruit enough qualified reviewers to keep up with a soaring increase in grant applications. But they also say they have taken steps to make serving more appealing. For example, the agency is holding more of the peer-review meetings on the West Coast and in Chicago to make it easier for panel members who live there to attend.

"We had a problem two or three years ago," but "I'm very satisfied with the quality now," says Antonio Scarpa, director of the NIH's Center for Scientific Review, which manages the peer reviews for 70 percent of the agency's grant applications.

Help Wanted

Last year the NIH began a major self-study of its peer-review processes, and it received an earful of comments — more than 2,300 in all — from scientists. Complaints about the experience and quality of peer reviewers were among the most frequent.

The malaise concerning peer reviewers has resulted in part from one of the agency's triumphs, the doubling of its budget from 1998 to 2003. That led to a similar rise in grant applications starting in 2002 that leveled off around 2005.

To deal with that volume, the NIH also increased the ranks of peer reviewers. In 2005 nearly 18,000 scientists, most of them from academe, evaluated applications for the Center for Scientific Review, up from about 12,000 three years before.

Peer reviewers serve on 200 committees, called study sections, corresponding to subspecialties in biomedical science. One section reviews only proposals related to tumor genetics, for example; another, only the pathology of kidney disease.

The NIH's need for reviewers expanded so quickly that levels of experience among reviewers did dip somewhat by 2006, says Dr. Scarpa, who came to the NIH in 2005.

To fill seats on the review committees, the Center for Scientific Review recruited more untenured faculty members. The percentage of all reviewers who were assistant professors reached 10 percent in 2005, up from 8 percent in 2002 and 5 percent in 1998.

Most of the newly recruited academics signed on as temporary or ad hoc panel members, which required attendance at as few as one meeting. The NIH wanted to accommodate researchers who preferred a short stint instead of a commitment to attending three meetings a year for four years. (Reviewers get an honorarium of $200 per meeting day.)

The temporary members have also answered a growing need at the NIH. More applications than ever are interdisciplinary and complex, and not all of the regular members have the breadth of expertise to rigorously review all applications presented at a given meeting. So NIH administrators use temporary reviewers to plug the gaps.

The agency hoped the addition of temporary reviewers would not only reduce workloads but also encourage more volunteers. Scientists had long complained that the workload was too heavy. In 1996 the majority of reviewers prepared detailed critiques for 12 applications per meeting, on average. By 2006 the number decreased to six. (Even so, the workload remains high: Each application is reviewed by at least three committee members, and at least two of them prepare written critiques in advance, spending an average of seven hours on each, the NIH says.)

Workload vs. Quality

But like many medical treatments, the NIH's efforts to improve an ailing body have caused some unintended, negative side effects.

The expansion in reviewers has made for more-crowded meetings — some involve 40 people in one room. In one day, they try to reach consensus on the scientific merits of some 30 applications, leaving only about 15 minutes to discuss each one.

Then, for each application, each committee member offers a numerical score, and all scores are averaged. But the temporary members may be relatively green as reviewers. That, and the limited time for discussion, has reinforced a tendency for committee members to defer to the judgments of the three designated reviewers, observers say. All members are encouraged to read all grant applications in advance, but the volume is daunting.

The resulting dynamics of the meetings often seem straight out of the most-popular show on television, American Idol, wrote Michele Pagano, a professor of oncology and pathology at New York University, in a semisatirical commentary in the journal Cell.

Like the peer-review meetings, the show features three judges, one of whom, Simon Cowell, seems to delight in bullying contestants. At peer-review meetings, Dr. Pagano wrote, "the necessarily inexpert or distracted panelist often sides more easily with the Cowellesque reviewer, who is trashing the application, especially when there is not enough money to go around."

Other observers say that is hardly an accurate description of all committees. Experienced members can and often do persuade fellow panelists that an application's flaws are outweighed by its promise to deliver important results.

However, many scientists remain unpersuaded that the committees have sufficient numbers of those seasoned veterans.

One of those skeptics, who is also an accomplished scientist, is Steven L. Teitelbaum, a professor of pathology and immunology at Washington University in St. Louis. Despite his concern, he says he doesn't plan to be part of the solution, because of the time commitment involved. He previously served as a regular member of a study section but says if he were asked to do so again, "I would probably say no."

"I did my rotation, learned what I had to learn" about the peer-review process, "and then took on other responsibilities," he says, including a stint as president of the Federation of American Societies for Experimental Biology.

Dr. Scarpa, of the NIH, says that worries about too many inexperienced reviewers are out of date. The Center for Scientific Review has scaled back the total number of reviewers by more than 2,000 since 2005, and all of those cut were temporary reviewers. What's more, the center has reduced the percentage of all reviewers who are assistant professors, by about a third to about 7 percent.

The average workload per reviewer has crept up slightly. But, Dr. Scarpa says, it was worth the trade-off to shrink the size of the committees, an effort to improve the quality of discussions.

The Center for Scientific Review also called last year on scientific societies to offer the names of qualified scientists willing to volunteer as reviewers, and gathered 3,000 names.

But Mr. Petsko, of Brandeis, who is president-elect of the American Society for Biochemistry and Molecular Biology, says those changes have not yet brought the review committees enough experienced reviewers to ensure quality. Academic rank is a meaningless proxy for technical expertise and good judgment, he says.

"I've been on enough panels," he says, "to know that we're not getting the best people."

Making Service Easier

The NIH is experimenting with several other changes to make service on its study sections more palatable to those experts.

One is to shorten the length of research-grant applications. A test this year by one the agency's largest divisions — the National Heart, Lung, and Blood Institute — found that the shortened application did not reduce reviewers' workloads. However, the Center for Scientific Review has seven additional tests under way.

The NIH is also experimenting with eliminating travel time by holding some meetings of study sections electronically, through conference calls, videoconferences, and the use of secure Web sites.

The Web approach is the most controversial of the three. The sites function like electronic bulletin boards or blogs, allowing committee members to post and respond to written comments at their convenience.

Fans of the approach say it allows them the time to craft more-thoughtful comments than they could in a face-to-face meeting. Members also feel more free to buck opinionated reviewers who dominate the in-person meetings. Skeptics, however, say the Web-based approach is no replacement for face-to-face meetings where reviewers must defend their critiques in person.

Some outside scientists have wanted the NIH to go further and offer a stick and an additional carrot: All scientists who received the agency's grants would be required to serve on a review panel if asked, and the agency would automatically extend the duration of grants for those who did. Other biomedical researchers have proposed an even more radical change: that the NIH replace all of its regular reviewers with temporary ones, a model used by the National Science Foundation. But neither approach is likely to gain traction among NIH officials.

As a follow-up to the comments from scientists and recommendations from an internal policy committee, later this month NIH leaders are expected to prescribe additional changes for maintaining a healthy peer-review process.