Vancouver, British Columbia—Peer reviewers are the dragons guarding the cave where the academic treasures of individual promotion and institutional rankings lie. But the dragons are getting weary as armies of researchers from all over the planet try to get by them.
In a popular session at the annual meeting of the American Association for the Advancement of Science here, one audience member complained he was “bombarded by requests to do reviews.” The task had begun to seem irrelevant to him, he said, because even when he criticized manuscripts, they still got published. “There is so much mediocre work in the journals, I just don’t know what to do,” he said.
Panelists at the session—a university dean, a journal editor, and a director of publishing for a scientific society—offered some insights on peer review and suggested ways to give it fresh strength.
Emilie Marcus, chief executive of Cell Press, in Cambridge, Mass., which has grown from four journals in 1998 to 29 peer-reviewed journals today, explained how 11 editors at Cell, the press’s flagship journal, wade through manuscripts. Of 2,300 submissions last year, 1,500 were sent out for peer review, she said. Prospective authors at Cell can suggest reviewers and are allowed to rule out up to three reviewers. Cell publishes 364 papers each year, or about 15 percent of the papers submitted.
The journal strives for a two-week review time. Every two days that a review is late after a deadline, the journal contacts the reviewers by phone or e-mail, in a system that escalates from contact by an editorial assistant to the editor herself. “Speed is becoming more important,” said Ms. Marcus. For one thing, scientists are often striving to beat other researchers to publication.
Cell‘s editors, along with looking at specifics such as experimental design, also look for broad impact. A paper, Ms. Marcus said, needs to “change the way we think about a biological problem.” Some authors, she said, could ease the burden on peer reviewers if they would stop pitching the most prominent journals with every manuscript. Authors, she said, should be “more honest with themselves about the most appropriate home for their paper.”
Ms. Marcus said peer review has to become more efficient at a time when emerging countries are ramping up their publication rates. Editors will also need to extend their professional networks. Journals “need a global pool of reviewers,” she said.
Linda Miller, the associate dean for basic science at New York University’s Langone Medical Center and a former editor at both Nature and Science, said that scientists’ expectations of privacy have eased along with the rest of society’s. “In science, that means sharing before publication,” she said. Scientists who are in computer-oriented disciplines, like some geneticists, are more apt to share data before publication, she said, while scientists who do “wet lab” work are less likely to do so.
Lowered privacy expectations have put pressure on peer review, with some scientists expecting it to become more public. (The usual rule is that reviewers know who a paper’s author is, but authors do not know who their reviewers are. The readers of a paper usually have no idea who the reviewers were.)
One suggestion at the session was that reviewers’ comments be released when a paper is published, so that journalists, at least, could get an idea of the limitations on the paper’s scientific conclusions. Some panelists also encouraged the sharing of reviews among the reviewers themselves. After reviewers critiqued each others’ comments, the marked-up reviews would then go on to the journal’s editors, who would then get a more intensive discussion of a paper’s strengths and weaknesses.
In an argument for more privacy, not less, Ms. Miller suggested that journals should experiment with “double blind” peer review to see if it would reduce bias against unknown scientists or those from developing countries. Creating a double-blind process would mean removing identifying information about the authors from papers sent out for review, a step regarded as difficult, because writers often reference their earlier work.
In the view of the panelists, two alternatives to peer review, the “crowdsourcing” of papers before they are published or the additional public review of them after publication have not caught on.
Scientists do discuss recently published papers in labs or faculty lounges, but they have little incentive to post extensive online comments, panelists said, unless the work is widely publicized or highly controversial. One scientist told Ms. Miller that if he was going to take the time to think about how the experiments in a published paper should have been done, he would rather just go ahead and do them himself. Ms. Marcus said she believes that post-publication review is possible, but “we don’t have the technology to make it feel natural.”
The director of publishing for the American Astronomical Society, Chris Biemesderfer, said that the society’s two journals got 4,864 manuscripts in 2011 and accepted 3,125. The society published 46 errata but had zero retractions. He asked, rhetorically, why retractions seem to be increasing in other publications, if not in astronomy journals. His conclusion: “It’s easier to do naughty things with digital tools and easier to detect naughty things.”
Ms. Miller said that many scientific journals had pooled articles, including those that require a subscription, for plagiarism detection. But she said the most frequent kind of plagiarism was among review-article authors, who take a broad look at research in a particular field. Those authors, she said, tend to plagiarize the introductions in their own work.
Along with being the guardian of academic quality, peer review also has some popular appeal. A London-based nonprofit organization, Sense about Science, organized the AAAS session and has put out a pamphlet, titled “I Don’t Know What to Believe.” The first print run was 10,000 copies. Ten printings later, the group has distributed 500,000 copies. Now the group is working on a U.S. version.
