Search The Site
 
More options | Back issues
Home
News
Opinion & Forums
Careers
Multimedia
Chronicle/Gallup
Leadership Forum
Technology Forum
Resource Center
Campus Viewpoints
Services
The Chronicle of Higher Education
The Chronicle Review
A weekly special section
Brainstorm: Lives of the Mind Dan Greenberg

Peer Review at NIH: A Lottery Would Be Better

For awarding government money for basic biomedical research, would a lottery out perform the traditional system of peer review? A lottery? Ridiculous, when peer review brings knowledge and experience, rather than chance, to the selection process. Maybe not so ridiculous. Read on.

We’re talking mainly about the National Institutes of Health, which presides over the world’s biggest bankroll for health-related research, nearly $30-billion this year. (See a Chronicle story on the matter here.) About half is for grants to individual researchers. As plump as that sum is, there isn’t enough to satisfy all reasonably competent supplicants for grants.

The insufficiency is so great that it’s generally acknowledged that many winners and losers are indistinguishable in terms of capability and the scientific importance of their proposed projects. Moreover, the process for deciding who gets money and who doesn’t is cumbersome, extremely costly, and often meaningless for making wise choices.

The NIH peer review system is one of the wonders of the scientific world. To award that money, NIH relies on some 18,000 reviewers holding small, face-to-face meetings at which they discuss and rate grant applications. Though a cautious, slow transition to electronic communication is under way, the great majority in this army of reviewers attend three meetings per year, usually of two days’ duration, at NIH headquarters, in Bethesda, Md. Travel time for many reviewers adds two days to meeting time. My request for the costs of travel and lodging for this population movement has been languishing at NIH for several months, but I understand that it’s at least a couple of hundred million dollars per year.

At NIH, nine months is the generally reported time for rendering a verdict on a grant application. Efforts are being made to speed up the process, but optimism is not warranted. Off and on, for a long time, such efforts have been made. The National Science Foundation, a sister federal agency, gets its grants out the door in about six months. NSF mainly relies on telephone and e-mail for obtaining evaluations of grant applications. NSF is also overwhelmed by grant seekers. There’s no reason to believe NSF’s decision making suffers from its relatively speedy process.

The disparity between the volume of worthy applications and the availability of NIH grant money is dispiriting for scientists who take peer-review duty seriously. An unknown but apparently rising number are said to decline peer review service as meaningless, given the similarity of winners and losers in the grant derby.

So why bear the logistical burden and expense of a system that’s plummeting into irrelevance and undermining morale in the biomedical research community?

More money for grants is the obvious solution, but that’s not to be in these hard times. The need, then, is to make the most of the available money.

A quick scan by appropriate specialists could easily sift the null applications from the potentially fruitful proposals for using NIH’s scarce money. A lottery drawing could then disburse the available funds. Reliance on chance wouldn’t be inferior to what’s happening now, which, as it turns out, is a game of chance in the guise of informed selection. Moreover, the savings from a lottery could be recirculated to research, providing many millions of dollars for projects that would otherwise go unsupported.

Project selection by lottery? The concept surely grates on the scientific ethos. But, in the spirit of experimentation, it’s surely worth at least a pilot project.

Posted at 04:11:58 PM on February 11, 2008 | All postings by Dan Greenberg

Comments

  1. Dan Greenberg has identified a significant problem. I have participated in the NIH grant review process, both as a reviewer and as an applicant. I have applied for funding many times and have won a few grants. Sometimes I have come close, receiving good priority scores, that were not quite good enough to be funded during that grant cycle; other times, I have not even been close. I have served on a number of site visits for major program projects or facilities, as well as on study sections. The site visits were really helpful in understanding the projects, programs, and facilities. The study sections were worse than useless. The study sections I participated in were a serious waste of time and money, and, I do not think they provided fair evaluation nor consideration of the proposals. Would triage “by appropriate specialists” and a lottery be better? It might be better, but it would not be good. In fact, I think it is a terrible idea. What WOULD work, is a system in which the appropriate specialists assigned qualified peer reviewers and sent them proposals electronically, to review and return their comments, also electronically. The cost of all that transportation and lodging could be saved for inclusion in grants—perhaps with a small portion of it given as a stipend to the reviewers. The NIH grant review process should be more like that of a good peer-reviewed journal.

    — Joe Erwin · Feb 12, 03:30 AM · #

  2. I would prefer a block grant system for distributing funds. Grants of Principal Investigators that have sufficient merit get a threshold amount of funding, say $50,000 per year. Any amount over this threshold would be subject to the current competition style. This system would fund more labs and more breadth of science. The current system is broken and giving rise to sloppy, shallow science.

    — B. Martin · Feb 12, 10:00 AM · #

  3. Sorry, but a lottery is nonsense. Yes, it might be hard to distinguish between the top 20% of grants but if nothing else, the comments one receives if a grant gets turned down are very helpful. I’ve had my NIH proposal rejected once and have seen rejected proposals of multiple colleagues. In each case the revised grant was more focused, with better aims. Also, rejected grants that get revised typically have better odds of being funded. I don’t see how a lottery would take this into account. Yes, money is tight and there a lots of good grants that do not get funded. But especially for the proposals that get rejected the comments are invaluable for improving the proposal for the next round.

    P.S. If the travelling of the reviewers costs 200 million so be it. In light of the 500 billion dollar defense budget proposal, that sum is nothing.

    — Michael · Feb 12, 10:36 AM · #

  4. But, Michael, aside from site visits for program projects, the travel expenditures really do not benefit anyone or anything—they are pure waste. Careful, thoughtful reviews, of the kind demanded by conscientious editors of scientific journals, are, indeed, very helpful for the revision of manuscripts and research proposals. Personally, I think the review process could be better managed by extramural consultants with appropriate expertise than by the intramural grant review management staff, at a fraction of the current cost. A process could be designed that would be far more efficient and effective than the current system. And, by-the-way, NIH and science is not about to get any of the billions that are currently being stupidly squandered on the war.

    — Joe Erwin · Feb 12, 12:19 PM · #

  5. I do not really see an alternative to a number of experts getting into a room and judging the merits of a grant. Fairly often the two primary reviewers of the grant will differ dramatically in their opinion and the issues they differ on have to be discussed and resolved so that everyone can give a score. I guess it’s possible to do this via a videoconference or over the phone, but that brings up issues of confidentiality etc.

    — Michael · Feb 12, 03:44 PM · #

  6. The point that is being missed in the comments is that the top 10% that get funded do not differ in quality or potential for success from the next 15 -20 percent that score below them. Maybe a two-tiered system of review; those that score in the top 30% go into a pool from which the winning tickets are drawn at random!

    — Don · Feb 12, 05:41 PM · #

  7. #5 comment, Michael, seems to have participated in the process. He is aware that there is a primary and a secondary reviewer. In my experience, there has seldom been much input from anyone but the primary reviewer. There has not been much disagreement by the secondary, and there has been very little substantive discussion by others. Maybe I have just gotten in bad study sections, or something. As a journal editor, I have requested and gotten very substantive reviews of manuscripts—of superior quality, I think to the kinds of reviews I have seen in study sections, and, I think, from better qualified reviewers. The concept of random selection is actually way too close to what is happening now. I don’t see how it would help.

    — Joe Erwin · Feb 12, 07:45 PM · #

  8. A block grant of $50,000 sounds good. That is what large scale investors do—put out lots of investments in the hope that 1 in 10 will pay off big.
    I have served on NIH panels, and well it is mostly “good old boy” networks that win the $$

    — mea · Feb 12, 11:08 PM · #

  9. I like the “block grant” notion of relatively small start-up grants, especially for “high-risk, high-yield” projects. The SBIR model could be more widely applied, which involves Phase I ($50-$100K) short-term funding, followed by Phase II ($200-500K) longerterm (e.g., 2yr) if the Phase I produces tangible progress. And, there is a Phase III involving non-NIH funding. It is a nice progression. I kind of like the idea of a $50K Phase I for 6 mo, followed by $200K for 2 yr, with the potential of up to $300K for an additional 3 yrs. That could help get people and projects off the ground. Of course, if indirect costs are too high, it all gets eaten up by them, so maybe these amounts have to be doubled to even begin to make sense. But seriously, folks, 18,000 reviewers X 3 X $1000 is a chunk of change ($54,000,000). I’d be willing to set up a review system for $18,000,000/yr that would do the job more effectively and save $36,000,000 that could be spent on actual research (well, maybe $18 million on directs and $18 million on indirects). Maybe this is the time to also suggest that such a program could cap indirects at 20-50%.

    — Joe Erwin · Feb 13, 04:18 AM · #

  10. I like the idea of saving travel costs by using web-based review, but have heard persuasive arguments about the face-to-face peer review process. Specifically, I’m told that advocacy by one reviewer can overcome misperceptions by a negative reviewer, resulting in a change of score. Yet comments above suggest that the primary/secondary assignments don’t facilitate that. Can others who have participated in review confirm or deny that?

    — Rick Francis · Feb 13, 10:34 AM · #

  11. Web-based review is alive and well, at least with some study sections at NIH. Over the last three decades I have participated in dozens of study section meetings and chaired quite a few. Over the last three years or so, I have served on an IAR committee (Internet Assisted Review) with meets electronically twice a year and face-to-face once annually. The system involves the usual six weeks or so to review the proposals, after which the reviews are posted on the secure web site. Over the two or three days of the actual review meeting, the primary reviewer posts comments on the reviews on a secure “chat room”. The other reviewers respond, then all members are free to join in the discussion. It seems to work about as well as any study section meeting in terms of coming to agreement (or “voting your conscience) on priority scores) although the nuances that may result from direct contact are sometimes lost. In any case, it saves a bunch of money, and it beats the hell out of a day or more of travel and two or three nights in a hotel. Is the review process an “old boy network”? Sometimes yes, more often no. Can a negative review be overcome by advocacy of another reviewer? Occasionally, if the negative reviewer is really off base, but more often, a negative review will result in an unfundable score, often with a recommendation to resubmit. Finally, as mentioned above, recommendations for change often result in a fundable resubmission. Can you imagine the chaos if grant submitters learn that only chance determined their fate, leaving them completely in the dark as to how or if to revise the proposal?

    — Art Broom · Feb 13, 10:58 AM · #

  12. Art, it is great to have such input from someone who is so experienced with the process. What you describe seems like a real step forward. In my experience, I have never seen a negative review presented in a study section by a primary reviewer overturned. I’m wondering if it might be practical to have an annual study section meet in association with an appropriate professional meeting, like FASEB or AAAS. Then people would get some meeting travel costs paid for by NIH as a review expense…. This is a small idea, but a combination of many such innovations could really help, couldn’t it? And, by-the-way, I do not think all reviewers need to be in the face-to-face study section. It would be wise to get three or four reviewers who have very current and deep expertise serve as reviewers—as one would expect in the case of a good manuscript review process. The study section committee that meets face-to-face could be composed of wise and objective people who based their priority scores on the comments of the reviewers. Again, thank you, Art Broom.

    — Joe Erwin · Feb 13, 01:35 PM · #

  13. I would certainly agree that the entire process could be web-based, however, I don’t think the chatroom format is ideal. I have seen (and heard of) too many cases where a grant was rescued (or dropped) by the strong intervention of one of the reviewers or following long discussion and I just don’t think this can be done efficiently over email. A better format would be to have some type of videoconference. Perhaps the NIH could reserve multiple locations around the US that are within driving distance for most people, if there are any issues of confidentiality. Again, the reason why a lottery won’t work is that the people whose proposal is rejected (which is the large majority) need a clear path of what to do next. In particular, the ones “in the next 10%” need to have some type of assurance that once they address the concerns raised by the reviewers they have a high chance of getting funded. Regarding small high risk, high yield grants, they already exist, called R21. In theory they require smaller preparation time and fewer preliminary data and are meant to fund pilot studies. Regarding the old-boy network: It is true that competitive renewals have a higher chance of getting funded compared to new proposals. But (and this is coming from a new investigator) this seems only fair. If you are up for a renewal, chances are you have a productive lab with lots of personell and therefore can ill afford a loss of funding, compared to somebody who is just starting out and who might be able to stretch out his startup, apply for smaller private foundation grants or wait with hiring students and postdocs.

    — Michael · Feb 13, 07:41 PM · #

  14. The present system does not work when 1 out of 8 or 10 applications are funded. The amount of labor to produce a proposal that has a chance of being funded is ridiculous. Mine have gone from a simple application to a Page Maker color printed publication which with all mandated additions is close to 100 pages. And among all the things I write, it is a waste, because its readership is limited to some 20 people. If it is funded it goes in the waste basket, if it is not funded it goes into the same basket! Furthermore, looking back on 30 years in the business and a few NIH grants, my best ideas were never funded (they may have been “borrowed” though). So here are my suggestions. Each PhD who aspires a scientific career, e.g., an assistant professor is assured a block grant $50-100k each year as long as sufficient publication output is maintained. Proof of that is the only thing needed to submit to NIH each year. The initial selection is based on PhD, thesis grades and post doctoral performance. At each 7 year interval, he/she submits list of publication, publications and other evidence showing an active research program over the last 7 years to NIH. If approved, another sequence of block grants will begin. The system would include the possibility of applying for more money along the lines of the present system. Furthermore the selection process of reviewers needs to be influenced by other factors than the Program Directors. How about letting all the professional associations that exist in each specialty select their representatives at NIH through votes by the membership?

    — goran Hellekant · Feb 14, 11:38 AM · #

  15. Goran Hellekant has it right. The emphasis should be on productivity. I know far too many investigators who publish little yet have big labs with lots of funding based on what they propose to do next and their “status” as senior investigators. Meanwhile, lots of innovative work fails to get funded by the current system. The American public needs to know that a great deal of tax-payers’ money is waisted. I have 7 US patents issued to date and never got NIH funding.

    — Bruce Gold · Mar 3, 05:27 PM · #

Commenting is closed for this article.
Copyright © 2008 by The Chronicle of Higher Education | Contact us
User agreement | Privacy policy | About The Chronicle | Site map | Help
Subscribe | Advertise with us | Press inquiries | RSS | Today's most e-mailed
Home | Chronicle Careers | The Chronicle Review