• September 30, 2014

Leading Humanities Journal Debuts 'Open' Peer Review, and Likes It

Getting published in a humanities journal usually works like this: Submit an article, then hope for the best as the editors send it to a few hand-picked specialists for critique. The reviewers and the authors are not supposed to know one another's identity.

But now scholars are asking whether this double-blind peer-review system is still the best way to pass judgment. The Internet makes it possible to share work with many people openly and simultaneously, so why not tap the public wisdom of a crowd? One of the top journals in literary studies, Shakespeare Quarterly, decided to put that question to the test.

For this year's fall issue, a special publication devoted to Shakespeare and new media, the journal offered contributors the chance to take part in a partially open peer-review process. Authors could opt to post drafts of their articles online, open them up for anyone to comment on, and then revise accordingly. The editors would make the final call about what to publish (hence the "partially open" label). As far as the editors know, it's the first time a traditional humanities journal has tried out a version of crowd-sourcing in lieu of double-blind review.

The verdict from several scholars who took part: mostly a thumbs up, with a few cautionary notes and a dollop of "It's about time" mixed in.

"It was on the whole a successful experiment," said Martin Mueller, a professor of English and classics at Northwestern University, who took part as a reviewer.

Michael Witmore, a professor of English at the University of Wisconsin at Madison, co-authored an article on the use of statistical analysis and a text-tagging database to reveal linguistic patterns in Shakespeare. He and his co-author "got some terrific ideas and some citations" from the comments of the six or so people who actively responded to the article, he said. "It produced a more interesting paper."

Scholars and editors in the sciences have been trying out open peer review for some time, with not entirely rosy results. The journal Nature did a test run in 2006. In a published overview, the editors concluded the venture had not been a success. Many authors expressed interest but few participated, and the quantity and quality of the comments were disappointing. 

Interviews with participants in Shakespeare Quarterly's open peer-review trial, however, suggest this attempt went much better than Nature's did. At least one participant pointed out that the humanities' subjective, conversational tendencies may make them well suited to open review—better suited, perhaps, than the sciences.

Katherine Rowe, chair of the English department at Bryn Mawr College, guest-edited the special issue. She and the editorial board decided that the issue's new-media theme offered a chance to investigate how scholarly authority works in a networked environment.

"This was genuinely an experiment," Ms. Rowe said. "We didn't know what would result."

The journal's publisher, the Johns Hopkins University Press, supported the idea. MediaCommons, a digital scholarly network set up to encourage such experiments and conversations, agreed to host the project. So Ms. Rowe and other editors put out a call for papers, culled submissions, then offered authors still in the running a chance to post drafts online. All accepted.

Rounding Up Experts

Ms. Rowe invited about 90 scholars, including Northwestern's Mr. Mueller, to comment. Anybody willing to publish thoughts under his or her own name could join in, but the guest editor wanted recognized authorities as part of the field.

"'What's the nature of expertise?' is one of the questions that really gets opened up by an open process," Ms. Rowe said. "Everybody wanted to be sure that experts would be involved." By her count, about 40 commenters, invited and self-selected, finally participated.

All told, four articles and three review essays were posted on MediaCommons during a two-month review period this past spring. (The articles and comments remain archived on the MediaCommons site.) As it turned out, all seven submissions will appear in the issue.

Some of the authors acknowledged doubts going into the experiment. Would open review be rigorous enough? Was it risky to post work in progress? But "the results were terrific," said Mr. Witmore, who wrote his paper on Shakespearean linguistic analysis with Jonathan Hope, a reader in the English department of the University of Strathclyde, in Glasgow. "It's very different from getting a two-paragraph reader's report from a journal," Mr. Witmore said. "In this case, what you get is individual readers from a wide range of subspecialties zooming in on a particular paragraph, saying 'Tell me more about this' or 'Why did you do this?' It seemed more like a dialogue."

Another scholar, Alan Galey, submitted an article about Shakespeare and the history of information. An assistant professor on the Faculty of Information at the University of Toronto, he worried that an article vetted this way might carry less professional weight—a matter of particular concern to a junior professor going for tenure. "It was very much going on faith in a way," he said.

Mr. Galey's dean told him to make sure the process would be rigorous and fair. The stature of the journal also helped reassure him on that point. So did Ms. Rowe's willingness to answer questions and her decision to invite established scholars to join in. Many crowd-sourcing experiments depend on scale, Mr. Galey pointed out, but this relied "on relationships among scholars where you know you can trust somebody. It wasn't a Wild West by any means. It was as controlled a process as traditional peer review. It was just controlled in a different way."

Mr. Galey wound up feeling that the experiment paid off. "I got better feedback from this process than I've had from any other peer-review process," he said.

Another participant, Ayanna Thompson, an associate professor of English at Arizona State University, wrote an article focused on race in performance-based teaching of Shakespeare. She called the experiment "a fascinating process." But she also found it stressful, and saw evidence that some old ways die hard.

Better Feedback

On the positive side, she got lots of feedback—"the equivalent of eight single-spaced pages of comments," she told The Chronicle via e-mail. She felt that the lack of anonymity encouraged reviewers to engage with the material "in a much more thoughtful and thorough way than in blind reviews because their names were attached to the comments." That helped her clarify and revise sections of her argument. But going through a public critique "was a stressful experience, and I kept saying to myself, 'I'm glad I have tenure,'" she said.

She also discovered that academic status still played a role. "The people who were commenting online were very established, senior scholars. In fact, several junior scholars notified me offline that they had comments, but that they did not want to post them in case they contradicted the senior scholars," she said. "So while the process was supposed to democratize the review process, it did keep some of the old hierarchies in place."

Still, Ms. Thompson is glad she took part. "The revised article is much stronger for the feedback, and I feel lucky to have received so much deep engagement with my work," she said.

David Schalkwyk is editor of the journal and director of research at the Folger Shakespeare Library. He called the experiment a success and said that Shakespeare Quarterly plans to try it again for its future special issues, whose focused topics are most likely to attract a critical mass of knowledgeable reviewers.

The system did, however, come at a cost from an editor's perspective, he said. For instance, editors and writers had to keep tabs on the evolving discussion and build in extra time for revisions; authors also tended to make their articles longer and longer in response to multiple comments. "I think we underestimated the amount of time that was required," Mr. Schalkwyk said.

That's what keeps him from adopting the approach for every issue. He said, "My reservation about applying open peer review to normal issues is entirely practical. It's not philosophical. If I could do it, I would."

For Mr. Mueller, the reviewer from Northwestern, the journal's experiment was welcome—and past due. "I think it will take a decade for scholarly communication to find ways of fitting itself into this new technological environment," he said. "It takes time. The human problems always take more time to manage and solve than the technological problems."

Comments

1. 11159995 - July 29, 2010 at 02:19 pm

An interesting experiment, but not so radical as it first appears since it was a highly "controlled" process in which 90 "recognized authorities" were invited to participate. This doesn't seem all that different from asking two or three such authorities to do a review and having them feel obligated to do so, instead of depending on some percentage of the 90 to take up the challenge. This likely is not a scalable approach as the participation rate would probably be minute if this were done for every issue of a journal--and the bias factor would go up since--as we all know from the responses to Chronicle stories--those who have an ax to grind are generally the ones who respond.---Sandy Thatcher

2. lfathe1 - August 25, 2010 at 08:52 am

This may be radical for the Humanities, but it's old news for the sciences. The high energy physics community has been doing open commenting on pre-publications for more than a decade. It works very well.

3. timewaster123 - August 25, 2010 at 08:55 am

There is also a huge issue of how to get articles consistently reviewed if this process will keep being implemented - you might get a large number of reviewers on the first few passes and then reviewer participation would be likely to fall off due to our numerous time constraints.

Also, this process sounds kind of like the pre-publication steps that people should be taking - passing around their work to friends and people they know for comments, to make revisions prior to submission. If authors start to skimp on that step, no wonder it takes a long time for open review.

4. avalongod - August 25, 2010 at 12:40 pm

I agree it's an interesting experiment. I think, however, the authors in this case all claim it's a "success" since all papers got accepted in the end.

The weakness I could see is that this approach could unwittingly further the status quo and discourage novel ways or thinking or critiques of popular (and dogmatic) views. For instance, lets say position X is held by 90% of scholars in a given field. Researcher Y finds evidence against position X. During an "open" peer review, scholars who hold position X could essentially "pig pile" on Researcher Y's paper, commenting negatively en masse. Even where such comments are ideologically motivated, they may frighten the editor from publishing what otherwise might be a groundbreaking paper.

Granted anonymous peer review isn't immune to similar influences, but I could see some potential for "mob rule" breaking out with this process.

5. gerrymck - August 27, 2010 at 09:24 am

As Some May Know I've Long Had An Interest In Alternative Peer Review

"Alternative Peer Review: Quality Management for 21st Century Scholarship"

[ http://www.public.iastate.edu/~gerrymck/APR-1.ppt ]

> >> It's A Large PPT (200+ Slides) >>>

>>> But IMHO ... Well Worth The Experience [:-)]

"Peer Review in the Internet Age: Five (5) Easy Pieces," Against the Grain 16, no. 3 (June 2004): 50, 52-55.

Self-archived at

[ http://www.public.iastate.edu/~gerrymck/FiveEasyPieces.pdf ]

See Also

> Invisible Hand(s): Quality Assurance in the Age of Author Self-Archiving

[ http://scholarship20.blogspot.com/2007/11/invisible-hands-quality-assurance-in.html ]

>> Quality Assurance in the Age of Author Self-Archiving

[ http://www.public.iastate.edu/~gerrymck/ACRL2005.ppt ] / 110+ Slides

[ http://www.public.iastate.edu/~gerrymck/ACRL2005.pdf ]

See Also My Profile Of The Scientist Special Report > Peer Review Rejected? At

[ http://scholarship20.blogspot.com/2010/08/scientist-august-2010-peer-review.html ]

/Gerry

Gerry McKiernan
Associate Professor
Science and Technology Librarian
Iowa State University Library
Ames IA 50011

Follow Me On Twitter > http://twitter.com/GMcKBlogs

"It's Not About Publication; It's About Ideas!"

[ http://bit.ly/bb6uW7 ]

Add Your Comment

Commenting is closed.

subscribe today

Get the insight you need for success in academe.