NIH Casts Critical Eye on How It Gives Grants

Nearly $15-billion for colleges is at stake in a review of the process for evaluating proposals

Related materials

Tables: NIH Peer Review: A System Under Pressure

Colloquy: Read the transcript of an online discussion with Elias A. Zerhouni, director of the National Institutes of Health, about proposals to reform how the NIH awards grants, part of efforts to overhaul a system regarded by some scientists as broken.

Audio: Elias A. Zerhouni, director of the National Institutes of Health, discusses the agency's wide-ranging review, now under way, of how it awards grants.

Article tools

Printer friendly	E-mail article	Subscribe	Order reprints
	Discuss any Chronicle article in our forums
Latest Headlines Higher-Education Price Index Rises 3.6 Percent The measure of colleges' inflation costs grew only slightly more than last year, but it doesn't include recent jumps in some expenses, like utilities. Higher-Ed Bill Slowed by Two Disputes Campus Planners Discuss How to Attain Sustainability Ruling Says Colorado Board Discriminated Against University Universities in Beijing Tighten Security for Olympics U.S. Universities Negotiate Tricky Terrain in the Middle East Newly Found Chameleon Lives Fast, Dies Young Commentary Richard H. Hersh and Richard P. Keeling: On a 'Liberal Education'

By JEFFREY BRAINARD

Washington

The National Institutes of Health's methods for reviewing and financing academic research proposals are often praised as the gold standard. Some American scientists, though, have recently offered less flattering descriptions, like "broken" and "arbitrary."

NIH officials have heard both arguments, and plenty in between, in recent months. They have begun their broadest-ever self-examination of how the agency chooses grant proposals for biomedical research. That is important for universities because the NIH is the country's single largest source of money for academic research. Just over half of its budget, $29.2-billion this year, goes to institutions of higher education.

The NIH's director, Elias A. Zerhouni, says he is prepared to advocate shaking up the status quo to make improvements. "All possible ideas are on the table," he says. This week a special advisory committee, which has been sifting through reform proposals over the past six months, should suggest a few to try.

Many of the proposals focus on improving the peer-review process which, many scientists complain, evaluates grant applications too conservatively. The panels of volunteer reviewers from academe that make up the heart of the system, the scientists say, tend to favor research projects that only slightly advance existing knowledge instead of testing innovative ideas that could transform medicine and health care.

That tendency also makes winning grants particularly difficult for young researchers, who face unprecedented problems finding the money to start independent careers.

Those problems have been aggravated by the NIH's tight budget, which has fallen behind inflation since 2003. Mindful of the need to stretch dollars, peer reviewers are less willing to gamble on risky ideas and unproven scientists.

"It seems fairly clear that the system has become so ponderous and creaky that it's going to need major change one way or the other," said David Korn, a senior vice president of the Association of American Medical Colleges.

But he and others worry that the budget squeeze and stubborn attitudes will make such changes hard to pull off.

A Doubling of Applications

NIH leaders began the review of grant-making in June. A panel of outside academics held six meetings around the country and solicited written comments. More than 2,000 people and organizations responded with complaints and suggestions.

The last time the process came under such scrutiny was in 1997. Since then, peer review of grant applications has become more complicated because the cutting edge of biomedical research is becoming more interdisciplinary, requiring reviewers to know more about more fields. And because of the stagnant budget, more investigators have been chasing a declining number of new grants.

The annual number of grant applications to the NIH almost doubled, to nearly 80,000, after 1999, when Congress began a five-year effort to double the agency's budget. The budget doubling spurred universities to increase their laboratory space and expand their scientific work forces, which helped drive the increase in applications.

But Congress followed that buildup with a series of more-modest increases that fell below inflation. That has left the NIH's buying power for research about 8 percent lower today than in 2004, one year after the doubling ended. As a result, the percentage of applications financed by the NIH, known as the "success rate," has fallen from one in three in 2001 to the current one in five.

Worse for investigators, they are spending more time trying to win NIH grants. The success rate for applicants on their first attempt has fallen to about 12 percent, from 28 percent in 1998. The NIH allows applicants to revise and resubmit an application twice after an initial rejection. Although success rates are higher in those subsequent rounds, not everyone reapplies. Those who do find the process frustrating and time-consuming.

"An investigator might be better advised to gamble at a casino, where the rate of return is much better," wrote one scientist to the NIH. (The agency told scientists who submitted written comments that it would keep their names confidential, to encourage candor.)

For peer reviewers, the effect of the squeeze is to encourage nitpicking. Academics who volunteer for the NIH's review panels say the low approval rates force them to make fine distinctions among excellent applications that formerly would have made the cut in the first round.

"We need to reduce the amount of what I call wasted energy in the system," Dr. Zerhouni said. "It's not right for you to apply six times to get a grant from the NIH. You're good or you're not good, and let's just cut the rigmarole out."

With the nitpicking comes conservatism, Dr. Zerhouni and other observers say. They worry that the trend might throttle research ideas that seem like long shots but could lead to major advances in the treatment of health conditions like cancer, obesity, and other chronic conditions that afflict America's aging population.

Older, Not Bolder

"Increasingly, reviewers are taking an adversarial stance — they think their role is to try to figure out how the applicant was trying to trick the government into giving them money," said Keith R. Yamamoto, executive vice dean of the University of California at San Francisco School of Medicine.

"That's not exactly what we're looking for in a healthy review endeavor," said Mr. Yamamoto at an October meeting. He is a co-chairman of a working group overseeing the NIH's evaluation of its peer-review process and a long-term member of a peer review panel.

The NIH is already considering several ideas to streamline its application and review procedures, reduce the paperwork burden on applicants, and increase the ability of peer reviewers to conduct meaningful reviews. For example, the NIH's advisory committee recommended this year that the agency reduce the maximum length of its research-grant application from 25 pages to 15. The National Science Foundation's application is 15 pages, and the Howard Hughes Medical Institute's form is only five.

Some observers see that and other proposed procedural changes as desirable but incremental. Such minor alterations will not, they say, alleviate another serious problem that has been highlighted by the budget crunch: the graying of NIH grantees.

Scientists over 50 make up 44 percent of the recipients of the NIH's principal research grants, up from 31 percent in 1998. Meanwhile, the proportion of recipients 40 and younger has fallen.

Aggravated younger scientists charge that peer reviewers tend to be tenured faculty members who, the critics say, are merely protecting their colleagues. (The NIH enforces conflict-of-interest rules, but critics suggest that back scratching goes on nevertheless.)

Those complaints emerged from this year's public-comment process and from an NIH-financed survey in 2002 of attitudes among scientists who had recently received their first research grants from the agency. The survey also included postdoctoral researchers, who are typically not permitted by universities to apply for NIH grants because they are not faculty members. Seventy-two percent of the postdocs and 59 percent of the grant recipients agreed with this statement: "The 'peer review' system of evaluating proposals for research grants is, by and large, unfair; it greatly favors members of the 'old boy network.'"

And agreement with this statement was almost unanimous: "Eminent scientists and scholars are more likely to receive research grants than others who submit proposals of about the same quality."

Bias against younger scientists is also aggravated by reviewers' reluctance to take chances, complained many who wrote to the NIH about peer review. The glut of applications has heightened a tendency among reviewers to expect "preliminary data" related to the hypotheses in grant applications. That means applicants are required to have already conducted some of the experiments. But scientists just beginning their careers have not had the chance to do that.

Faced with such pressures, talented young scientists are simply giving up and leaving biomedical research, says Brian C. Martinson, a demographer with the HealthPartners Research Foundation, a nonprofit organization in Minneapolis. He was the lead author of the 2002 study of new grantees.

More Help for Young Scientists

To relieve pressure on both younger and more-established researchers, some experts are calling for the NIH to award more grants to scientists based on their brilliance and boldness, with less emphasis on the specifics of a particular experiment and whether it might achieve expected results.

The agency has already embraced that approach on a relatively small scale. One program, called the NIH Director's Pioneer Awards, provides $500,000 annually for five years. But only about a dozen researchers win one each year, out of several hundred applicants.

Another new grant program, called Pathway to Independence, helps postdoctoral researchers land faculty jobs by giving them a "dowry," as Dr. Zerhouni calls it, of research money. Here again the program is modest: It handed out about 200 awards this year, while U.S. universities awarded 6,631 doctorates in the biological sciences in 2006.

And beginning in 2006, the NIH for the first time set an explicit goal for raising the number of investigators it supported who had never before won NIH grants. The number of those grantees had dipped to about 1,300 that year, which the agency's leaders pledged to raise to about 1,500, the average of the preceding five years. The NIH gave those applicants extra credit on the numerical scoring system that it uses to award grants. As a result, the number rose to more than 1,600 for the 2007 fiscal year, which ended in September.

A more controversial idea to help younger investigators is to cap the number of NIH grants that any one researcher can hold at once. Approximately 200 principal investigators now hold four or more grants. In the written comments to the NIH, several scientists questioned whether those researchers could productively manage their "superlabs." Capping the number of grants, the writers said, would extend the dollars to a wider pool.

That proposal faces opposition from Dr. Zerhouni and others who fear it might stymie the best scientific proposals. The director favors an expansion of "positive inducements" for universities, like the Pathway grants for postdocs. But the idea of capping grants is not limited to a fringe element. In its written comment to the NIH, the Association of American Medical Colleges supported a step in that direction: limiting applicants to one grant application at a time for any particular type of grant.

A Zero-Sum Game?

Some scientists who are skeptical of the reform proposals argue that it's impossible to define innovation precisely or predict who will turn out to be an innovative scientist. In written comments, they voiced worries that steering more money to new programs for young and innovative researchers while the NIH's budget is flat might hurt applicants for the agency's traditional grants.

Several scientists who wrote the NIH said there was nothing broken in the agency's peer review. Researchers could avoid scrapping with each other for money, they said, if academics and patient advocates lobbied Congress for a larger budget.

The lawmakers' response is hardly a sure thing, though. Although the NIH budget for 2008 remains incomplete, the new Democratic-led Congress has so far shown only a little more generosity than its Republican predecessor. Many legislators see the NIH budget as having received its fair share when it was doubled. The budget is now among the largest of all nondefense federal agencies.

And a financial expansion alone is not a cure-all for the maladies afflicting NIH grant making. In the late 1990s, officials set a goal of doing more to finance innovative research and help scientists begin their careers. The agency's budget doubling provided an opportunity, but progress was limited, as evidenced by the current problems.

More money for the NIH could even add to the problems of younger scientists, considering that biomedical research in America depends heavily on their relatively cheap labor, Mr. Martinson says. Without other changes, more money would simply extend what he sees as a system that resembles a Ponzi scheme.

More cash, he said, would only encourage principal investigators to hire more postdocs and seek graduate students to carry out their projects — work that does not improve career prospects for the younger scientists.

"There's this unquestioned assumption that more is always better, that having more scientists out there will increase the number of 'eureka moments,'" he says. "It comes down to, What is a sustainable scale for this enterprise?"

Going forward, Dr. Zerhouni says he expects the NIH will only incrementally change how it reviews and makes grants, partly because of the tight budget. To get the most impact from that approach, he wants the agency to collect data on which changes work and which do not. Those experiments should begin by next spring.

That kind of methodical follow-up will require a long-term effort and leadership. As a nominee of President Bush, Dr. Zerhouni's remaining time in his job is short. So these issues may end up on the plate of his successor.

NIH PEER REVIEW: A SYSTEM UNDER PRESSURE The NIH's budget and the number of grant applications for research both rose substantially during the last decade. But the number of grants awarded didn't keep pace ... NIH Budget, billions 1998 $13.6 1999 $15.7 2000 $17.9 2001 $20.5 2002 $23.7 2003 $27.2 2004 $28.7 2005 $28.5 2006 $28.5 Research-project grant application and awards Applications Awards 1998 24,151 7,518 1999 26,408 8,556 2000 27,798 8,765 2001 28,368 9,098 2002 30,068 9,396 2003 34,710 10,393 2004 40,861 10,052 2005 43,069 9,599 2006 45,688 9,128 NOTE: Grants awarded represent new awards and competitively reviewed renewals. The NIH typically awards research grants to run for four years. ... so researchers have found it increasingly difficult to win a grant, especially on the first try ... Percentage of grant applications Awarded on first attempt Awarded over all 1998 28% 31% 2003 24% 30% 2006 12% 20% ... and young scientists are getting squeezed out. Percentage of grant recipients Who are 40 years or younger Who are over 50* 1998 24% 31% 2003 18% 39% 2006 15% 44% * The actual number of grants to researchers older than 50 fell from 2003 to 2006, but the decrease for grantees 40 and younger was larger. SOURCE: National Institutes of Health