October 3, 2008

NIH Tinkers With Its Peer-Review Process

The National Institutes of Health is the largest single source of money for academic research, so faculty members and administrators have been closely following the agency’s plans to overhaul its peer-review process for grant proposals.

Elias A. Zerhouni, director of the NIH, described some of the expected changes this past summer, but this week the agency unveiled the timeline for the first set of alterations, which can more accurately be called tweaks, not major shifts.

One of the clearest changes will be to cut the maximum length of applications for the major investigator-led grants, known as R01’s, from 25 pages to 12 pages. The agency says that change will ease the burden on scientists who serve on peer-review panels. But the new limit will not kick in until January 2010.

The agency announced it would also seek to help reviewers by making the process of assessing proposals more flexible, although few details were given out this week. One option might be to use online “virtual” review sessions, rather than requiring most reviewers to attend the sessions in person. The NIH will test that option next year. —Richard Monastersky

Comments

1. I have served for the past two years on the CDRMP review panel for prostate cancer research and I find the review process to be tainted by the fact that the reviews are not blind. All reviewers see both the institution and the PI’s names for each proposal. Given that information and the comments and discussions made by panelists, I have occasionally seen scores upped or lowered by the perceived halo effect (or Merton’s Matthew Effect) also called ascription theory to how we scored the proposals. Following Merton’s Matthew Effect, those institutions already prestigous and well funded got higher scores and maybe more money based less on the merits of the proposal and more on the more distant merits of the instituional prestige and the prestige of the PIs. We only assume that a prestigous lab in a prestigous institution would not submit a voodoo science proposal, but could a really good idea not get funded because it came from a less than famous cancer center or an unknown PI—the answer is a scary yes.

   For decades, journals have been using blind reviews on publishing, and even though this is not a sure guarantee of only publishing work of scientific merit, it does help reduce the impact of the Matthew Effect on accepting papers that may have less merit in topic but more prestige in affiliation.

   The key point is that a lot of money is being allocated to major research into catastrophic diseases, how do we know which proposal to fund that will provide society with the best opportunity for a cure or substantially improved early diagnosis and definitive treatment? It is possible that a cure for cancer lies buried within an unfunded proposal.

   — Michael W. Popejoy, Ph.D.    Oct 3, 03:29 PM    #

2. Although the problems of prestigious institutions and investigators getting more than their “fair share” are thorny (& were highlighted in Nature), practical considerations limit the feasibility of double-blind grant application peer review. Reviewers must know the applicant has the appropriate training, resources, facilities, collaborators, patient populations, and commitment to publish. How are preliminary data reported? Who strips identifiers from the applications? Indeed, can all identifiers (straightforwad & subtle) truly be stripped?

   Adding another phase to the funding process in which the science is reviewed blindly and then the environment and qualifications assessed burdens the system further (think of doing this for more than 50,000 applications a year) and with little gain. The possibility of blind reviews was in fact discussed during town hall and advisory committee meetings as part of the Enhancing Peer Review initiative. Further, one early recommendation was to require PIs to invest at least 20% effort per grant application to avoid clustering of grants among the have’s at the expense of the have not’s.

   Separately, Nature has also sought commentary on some of these same issues related to double-blind review of journal manuscripts, which is not a universal practice.

   — writedit    Oct 3, 06:12 PM    #

3. I have served on NIH, NSF, CDMRP review panels. The problem with Dr. Popejoy’s proposal is that blinded reviews do not allow reviewers to use an applicant’s track record as a tool to predict the likelihood that a proposed line of inquiry will be productive. The real problem is that at most granting agencies less than 12% of the proposals are selected for funding. Most reviewers would agree that many more proposals merit funding than can be funded.

   — DJR    Oct 3, 06:19 PM    #

Previous: Illinois Says Ban on Political Activity Could Apply to Students
Next: Champagne and Tears for Major Physics Experiment