= Premium Content

  • Monday, May 28, 2012

Previous

Next

Keeping Blind Review Blind by Anonymizing Documents

April 25, 2011, 8:00 am

By Mark Sample

Blindfolded ManEvery once in a while I hear horror stories about about blind peer review for journal submissions or book manuscripts not being, well, blind.

I first heard about this particular failing of peer review in 2006, when a friend of mine opened a reader’s report in Microsoft Word and discovered, quite by accident, that the reviewer’s name was embedded in the document’s properties. The incident made a splash at the time, and was even covered by The Chronicle.

Since 2006, Microsoft Word—the word processor many of us use, for better or worse—has come out with newer versions that replaced the old menu and toolbar interface with the so-called “ribbon.” One consequence of the ribbon is that seldom-used tools are harder to find. And the removal of author information and other metadata counts as one of these hard-to-find features. The process is slightly different for Word 2007 and Word 2010, so I encourage you to find out exactly what you need to do in order anonymize documents before you send them out, as either an author or a reviewer.

But let me offer a provocation. Double blind reviews are supposed to protect both the author and the reviewer, creating a free exchange of ideas and critique of those ideas without subjecting either party to damaging social or professional consequences (beyond the possible rejection of a single piece of scholarship). But is it necessary? Is blind peer review worth it? Does it do what we assume it does? A number of recent scholarly projects have experimented with the opposite of blind review: open review. Learning through Digital Media: Experiments in Technology and Pedagogy was published by the New School using a completely open peer review process. As Jason mentioned earlier this month, several issues of Shakespeare Quarterly have used open peer review. And ProfHacker’s own Kathleen Fitzpatrick is heading a team that was recently awarded a $50,000 grant by the Mellon Foundation to study open review. Are these examples outliers, or are they heralds of the future of scholarly publishing?

[Red Blindfold photograph of Flickr user Stuart Richards / Creative Commons Licensed]

This entry was posted in Profession and tagged , , . Bookmark the permalink.

  • Print
  • Comment
  • redfood

    For many scholars in many fields blind review is a fiction. Once your research is sufficiently well established, it is often obvious who the author of an article is from its content alone.

  • sselisker

    I agree with @redfood, although I’m glad there are still top journals that do blind review—Critical Inquiry (not blind) and PMLA (double blind) serve very different purposes as journals in my view, and I find PMLA’s commitment to giving a very wide audience to excellent work that’s not necessarily by established or high-profile scholars commendable.

    Would open review similarly draw most attention (and the most thorough reviewing) to more established and well-known authors?

    I’ve heard tell of a “Debates in Digital Humanities” collected volume, in which the contributors to the collection are peer-reviewing each other’s work. This sounds like a very exciting format, and it seems to me more generally that the particulars of individual open-review processes—who reviews, what incentives there are to review, whose work receives attention and why—will be major factors in the success or failure of open review.

  • mkgold

    I’m editing the _Debates in the Digital Humanities_ volume that you reference here, sselisker — thanks for the mentioning it, though there are a few differences from the examples cited in Mark’s piece. Instead of having the review site open to the general public, as was the case for Learning through Digital Media, the DHDebates peer-to-peer review site is open only to contributors to the volume. And while the site is private, all comments made within it are non-blind, so that the names of reviewers and authors are visible to each .

    The site has seen a pretty impressive amount of activity, with contributors leaving over 500 substantive comments for one another in less than two weeks. From what I’ve seen, the fact that names are visible hasn’t blunted the ability of reviewers to be critical, though it’s probably true that those criticisms are phrased more constructively than they might otherwise be.

    All contributors were assigned specific essays to review, and those assignments were listed on the site — thus adding a bit of peer-pressure as an added incentive to complete reviews.

    So far, I’ve found the private, non-blind peer-to-peer review model to work well, and I’d say that it has helped build a sense of community among the contributors to this edited volume. I’ll definitely use it again in the future based on these experiences. I should say, too, that the university press publishing the book will be conducting its own external reviews of the volume, which in part enabled us to use a more experimental model for this round of review.

  • gavin_moodie

    I agree with Dr Gold. Blind review is sometimes a shield for tactless if not insulting comments which can be just as critical but more constructively so with modest redrafting, which is encouraged by publishing reviewers’ names.

  • richpa

    I sign my reviews when the journal allows it. I have my biases, which I do my best to keep in perspective, but I think signed reviews can encourage diplomacy without preventing honesty.

  • iris411

    I still think Double-Blind reviews give the junior researchers a much fairer chance to publish. For established researchers, it might be a different story. As for redfood’s comment, I agree that in physics, especially experimental physics, it’s fairly easy to guess who is who just from the content of the paper (it’s not easy to set up a lab). But for disciplines like mathematics and asian studies, double-blind review mostly remain double-blind. And it helps!

  • sselisker

    thanks for the report on the process—sounds like a wonderful project, and a cool model for other edited collections to follow. best of luck getting the final draft together!

  • cwinton

    If the publisher provides a reviewing site, reviews composed using it or pasted to it will not contain the kind of identifying information embedded in documents produced by a personal software environment such as Word or Adobe Acrobat. Almost all reviews I’ve written in recent years have been of this sort, so why is this an issue? Or are some publishers still that far behind technologically? My use of such sites has been helpful because after the fact the publisher usually provides access to the blind reviews written by others for the same submission so I can compare them to my own.

  • anonytrans

    True. And even for those of us who are not yet established, doing research on a unique topic can give anyone’s identity away. As far as I know, I’m the only person doing work on a very niche topic in my field, and anyone in this area who attends our discipline’s professional meetings has a good chance of knowing that.

  • lizgloyn

    Speaking as an early career researcher, I personally like knowing I’m not going to be judged on my previous research record when submitting to a blind reviewed journal. I also like knowing that my work is not going to be judged on the basis of my gender.

Copyright 2012. All rights reserved.

  • The Chronicle of Higher Education
  • 1255 Twenty-Third St, N.W.
  • Washington, D.C. 20037