Experimental Use of Blog-Based Peer Review Gives Mixed Results

By JEFFREY R. YOUNG

Article tools

Printer friendly	E-mail article	Subscribe	Order reprints
	Discuss any Chronicle article in our forums
Latest Headlines San Diego State U. Defends Its Role in Federal Drug Sting The university's decision to use federal undercover drug-enforcement agents in a sting operation has brought the institution both heat and praise. Colleges Foresee Shortfall in Perkins Loan Funds This Year Student-Loan Market Shows Small Signs of Revival U.S. House Approves New GI Benefits, but Measure's Fate in Senate Is Uncertain Rutgers U. Earmark Becomes Issue, and Irony, on N.J. Campaign Trail Forum Focuses on Private Role in Expanding Global Access to Higher Education Where Students Compete 'Against the Wood' Slide Show: Photos From the Spring Meet Commentary Tell TIAA-CREF You Do Not Want Your Retirement Money to Support Genocide

An experiment in using an academic blog to peer-review a scholarly book showed promise, but the approach is time-consuming, and it will not replace traditional blind peer review anytime soon.

That was the assessment of those involved in an effort to post an academic book online, piece by piece over a number of weeks, and let anyone critique it (The Chronicle, February 1).

The self-selected guinea pig for the experiment was Noah Wardrip-Fruin, an assistant professor of communication at the University of California at San Diego. He is one of six bloggers who regularly contribute to Grand Text Auto, which offers an academic take on interactive fiction and video games, and he thought members of the blog's audience would make ideal peer reviewers for his book. The book, Expressive Processing: Digital Fictions, Computer Games, and Software Studies, examines the importance of using both software design and traditional media-studies methods in the study of video games.

MIT Press was willing to try the blog-based peer review but only if it could conduct a traditional peer review as well. Doug Sery, Mr. Wardrip-Fruin's editor at MIT Press, was skeptical of the value of bloggers' comments, and he worried that flame wars would erupt and distract the author from revising the book.

Feedback From All Corners

Now the results are in. So which kind of review was most valuable?

Certainly more feedback came on the blog than from the three official peer reviewers. Mr. Wardrip-Fruin said he got about 400 comments about his book on Grand Text Auto over the eight weeks that he posted segments of it. His initial goal was to answer every comment so that the peer review would become a conversation rather than a dictation from on high. But he admits that he fell behind in responding to comments at one point during the experiment because he was too busy traveling to conferences.

Even though blog readers could post their comments anonymously, most of them signed their names, said Mr. Wardrip-Fruin. It was clear that a wider variety of people commented on the blog than participated in the traditional review. To Mr. Wardrip-Fruin's pleasant surprise, some of the designers of the video games he wrote about have commented on his book.

Such feedback was one of the biggest benefits of the approach, he said, noting that it could help "avoid a problem that I think we sometimes have when the academy is talking about a subject where a lot of the expertise resides outside the academy." Like video games.

One concern about the blog-based approach to peer review is that the comments will become a love fest rather than an honest critique.

"Because of the self-selection, I doubt you are getting any real naysayers here," wrote Ian S. Bogost, a game designer and assistant professor of literature, communication, and culture at the Georgia Institute of Technology, in a comment on the blog. "I think I can say without controversy that most are fans of your work and of this project in general. I think that's actually a shame because one of the hypothetical benefits of peer review is gaining unexpected perspectives."

Bloggers More Critical

But Mr. Wardrip-Fruin said that the comments from blog readers—often people he knew well—were far more critical than the comments from the anonymous reviewers.

Mr. Bogost, for one, offered some frank assessments on the public blog (though overall he said the book was valuable). "Noah, you don't carry the main argument through the examples satisfactorily," he wrote. "It feels like a set of examples that evoke the main argument, but that do not carry it through."

One official reviewer essentially said the same thing, though more gently. "I also suggest that Wardrip-Fruin make some additional attempts to step back from his specific analyses and recapitulate more frequently the broad themes that the introduction raises," said reviewer No. 3.

The blog commentators did a better job giving detailed advice on individual paragraphs and sections, while the traditional peer reviewers gave better advice about the overall structure of the book, said Mr. Wardrip-Fruin.

Mr. Bogost and other commentators noted that because the book had been posted in short sections, it was hard to step back and get a sense of the overall argument when reading it on the blog. "Perhaps what is going on here is not really online peer review at all, but something more like online peer revision," wrote Mr. Bogost.

Mr. Wardrip-Fruin said it was worth the time and effort to get the blog feedback. "Undoubtedly it's going to be a better book," he said. "I learned a lot from the community in a bunch of different parts of the book that will be useful in the rewriting."

No Flame Wars

Mr. Sery, of MIT Press, is still not sold on the idea, though he said he was happy that no flame wars emerged.

He said he checked on the blog about once a day to see the comments but found himself running out of time to read them all. And if the practice catches on widely, he said, it might become even more difficult for professors to find time to participate. "If every one of my games-studies books pursued this model, I think you'd see far less activity," he said.

Mr. Sery's main complaint is that people who read bits of the book on a blog might have difficulty addressing larger points about the book's value as a whole. "You have to have the entire work in front of you if you're going to spend time thinking about it and giving real substantive comments," he said.

Still, he said, he'd be willing to try other, similar experiments in the future.

The traditional peer reviewers liked the book enough to recommend it for publication with only minor changes, by the way. It is expected to be published next spring.