Previous
Next

Does popularity make it good? How do academics feel about reactions to their work?

August 10, 2011, 8:06 pm

 

 

 

 

There is an interesting essay by Chris Colin in the current issue of Wired about the culture of online criticism. It opens with this:

You don’t have to read this essay to know whether you’ll like it. Just go online and assess how provocative it is by the number of comments at the bottom of the web version.

The piece touches on the growing ratings phenomenon. Perhaps Amazon played a big part in this by establishing a system in which every product can be rated and commented upon. When I was looking to purchase a new toaster, the user feedback factored into my selection. I wanted to see what others said about the “toast boost” capability. It’s not enough to just read the manufacturer’s list of features; I wanted data for real people.

Netflix lets subscribers rate films. Wikipedia lets readers rate entries. Apartment Ratings lets tenants review properties. This is nothing new. You see it everywhere online. User feedback sways our opinions. Perhaps we’re even timid to trust things that have not undergone (or received little attention from) this form of peer review. Ah—this song has a low rating it probably sucks. This vacuum cleaner only has one review there must be something wrong with it. Only two people “liked” my last column so it must not be any good. No one retweeted my last blog post so it must have been too abstract.

I noticed when I look at an article or blog post that I instinctively check to see how many “likes” it has, how many times it has been tweeted, or the number of comments. The quantity doesn’t necessarily persuade or dissuade me from reading, but I think it can heighten my interest or raise my expectation. If I come across a popular article then I am more excited about the content.

What’s disheartening is when thought-provoking posts receive just a few comments and minimal social media action, whereas librarian fashion posts receive tons of comments and numerous tweets and likes. Nothing against fashion but we’re a profession in crisis here—let’s talk about the big issues. And I’m not talking about shoes!

Obviously TMZ will beat out TED in terms of mass appeal—but I don’t want to make this about pop culture vs. intellectual content.

What struck me reading the essay is the larger concept of peer review and how it has seeped into the web experience. People want to be liked. Every time I visit Klout I’m told that my popularity and influence is declining – that my content is less engaging. What are my peers telling me by their non-reaction to my posts?

In academia we predominantly think of peer review as part of the publishing process—reviewing articles to ensure they are creditable. But to some degree this can also be measured after publication by how many times the item is cited. What impact did article X have on the discipline? Was this useful? Did it help shape opinions?


In pop culture terms—the new Jay Z and Kanye record has received favorable reviews (pre-release reaction) and has seen strong sales this week (post-release reaction.) You can see from iTunes that it is rated quite favorably. So is it a success? How do the artists feel about it after seeing the reaction? Validated? Encouraged? Happier?

In terms of my book — the comment that stuck was “surprised by how much of this book is about library as place”– I feel like I let the reader down. Should I feel that way? It’s not possible to please everyone, but did I miss a major chapter that this person was expecting?

This got me thinking about academics and how they react to reactions of their work. They labor over a book or article. Get it published. What happens next? Obviously it’s on to the next idea- the next project. But is there an emotional attachment? Do they care what the reaction is? Do they even pay attention to it? We all want our work to get noticed and for it to be appreciated, and better yet cited, but if it doesn’t receive much attention or less attention than anticipated, is it an unsuccessful piece of scholarship?

I decided to ask three professors two questions via email. Perhaps the most amazing thing is that they all responded within eight hours. Of course three responses does not make this generalizable, but it starts to paint a picture of their perspective. Other factors could play into the mix with a larger sample such as career status, personality, the nature of their discipline, institutional preferences, publishing history, and so forth.

I offer their responses without any editing or commentary:

1. Is it enough to publish an article in a “good” journal or a book with a “good” publisher or does the number of people who cite the work factor into how you measure impact?

I think that in the humanities it’s enough to be published in a good journal or with a good publisher.  Usually translations or reprints (say in a collected reader or something) also add to the pot but in terms of status/personnel the first time it runs is when it’s counted.

You sort of get additional kudos when it reappears but it doesn’t count quite as much.  Of course in terms of exposure it’s fantastic, as you get more play and more people start to think of it as a really important contribution, and in fact can become a standard reference.
Associate Professor, Humanities & Fine Arts

AT LEAST FOR ME, THE WHOLE POINT OF PUBLISHING IS TO SHARE ONE’S WORK WITH A WIDER AUDIENCE, AND SO CITATIONS–WHICH SUGGEST THAT ONE’S PUBLICATION WAS SOMEHOW USEFUL–ARE OFTEN MORE IMPORTANT THAN THE PUBLISHER ITSELF.  IN ACADEMIC LIFE, HOWEVER, PUBLISHING IN A ‘GOOD JOURNAL’ IS MUCH MORE LIKELY TO LEAD TO EXPOSURE, TO THAT WIDER AUDIENCE, AND THUS TO SUBSEQUENT CITATIONS.  IT’S ALL QUITE CIRCULAR: TO PUBLISH IN A GOOD JOURNAL, ONE HAS TO SURVIVE A MORE STRINGENT PEER-REVIEW PROCESS; READERS WILL BELIEVE THAT AN ARTICLE IN SUCH A JOURNAL MET THE CUT FOR GOOD REASONS; AND SO SUBSEQUENT AUTHORS WILL RELY ON THAT WORK WITH MUCH GREATER CONFIDENCE WHEN THEY WRITE UP THEIR OWN WORK.  IT’S NOT PERFECT, OF COURSE, BUT AN ARTICLE IN A TOP JOURNAL IS MUCH MORE LIKELY TO BE CITED BECAUSE IT’S IN A TOP JOURNAL THAN BECAUSE IT’S A GOOD ARTICLE PER SE.  LOTS OF GOOD ARTICLES ARE RARELY CITED BECAUSE THEY’RE NOT IN THE BEST JOURNAL.  AND A LOT OF MEDIOCRE ARTICLES ARE CITED OFTEN BECAUSE THEY’RE IN A GOOD JOURNAL, AND BECAUSE PEER REVIEW IS IMPERFECT.  THAT DOPEY ADVICE THAT CREATIVE PEOPLE OFTEN GET–DO WHAT PLEASES YOU–TURNS OUT TO BE PRETTY GOOD ADVICE.
Associate Professor, Social Sciences

I think I would say that I am happy to have papers published in good journals (journals that are well respected, are widely read, have good impact factors). Papers in journals like Science are exciting because of the high visibility, but I don’t think that the best work is always published in such venues. I think journals like Science tend to choose to publish the flashiest papers. “Impact” is really hard to measure, and there is tons of debate about how it should be measured. You have no doubt heard people talking about H factors.
Associate Professor, Sciences

2. When you publish something do you personally check to see the number of cites or other feedback/reactions/reviews, such as web activity?

For books, I don’t know anyone who doesn’t track their reviews.  It’s always done with fear and anticipation, since when you get a bad one it can be pretty devastating. Usually the reviews divide between positive with a few “critical” points but usually nothing that is that useful.  Certainly if there are letters after a review those are of interest too. If you get reviewed in a big time publication (say the NY Review of Books or the NYT Book Review) with high profile, and serious popular exposure, that’s a huge deal.  That would certainly go on a cv because it would count in relation to the stature of the book.
Associate Professor, Humanities & Fine Arts

NOPE, NOT ON PURPOSE.  HOWEVER, BECAUSE I’M NOT ALWAYS AT MY COMPUTER WHEN I’M TRYING TO FIND SOMETHING I’VE WRITTEN, I’VE GOOGLED THINGS I’VE WRITTEN ON OTHER COMPUTERS, AND THROUGH THAT HAPHAZARD WAY, I CAN SEE WHERE OTHER PEOPLE HAVE USED MY WORK. MY WORK ENDS UP IN THE WEIRDEST PLACES.  MY ARTICLES GOT CITED ENOUGH TO GET TENURE, BUT I HAVE A LONG WAY TO GO.

I’M MOSTLY A BOOK PERSON, I’VE BEEN LUCKY TO GET POSITIVE REVIEWS FOR MY BOOKS, AND I DO CHECK FOR THESE ONCE IN A WHILE, BUT BECAUSE I’VE NEVER GOTTEN A REALLY NEGATIVE REVIEW, I’VE NEVER HAD TO STRUGGLE WITH BAD REVIEWS.  THAT WOULD BE A TOTAL BUMMER.  PUBLISHING IS SUCH PAIN IN THE ASS–YOU’VE INVESTED SO MUCH IN THAT PROCESS, IT’S NERVE-WRACKING, AND ACADEMICS CAN HAVE VERY THIN SKIN EVEN THOUGH THEY CAN BE SO DAMN CRITICAL AND MEAN TO OTHER PEOPLE.  I KNOW THIS BECAUSE I AM ONE OF THESE PEOPLE.  AFTER MY FIRST BOOK, I WAS JUST DREADING THE UP AND COMING, SUPER MEAN GRAD STUDENT WHO’D WRITE SOMETHING NASTY IN A TOP JOURNAL ABOUT MY BABY.  I GUESS I’D WORRIED ABOUT NOTHING.  NOW THAT I HAVE TENURE, I SIMPLY DON’T WORRY ABOUT THIS MUCH AT ALL, AND TENURE IS LIBERATING IN THAT WAY.  AND YET I FEEL EVEN MORE COMPELLED TO PUBLISH REALLY GOOD WORK RATHER THAN BECOME CARELESS AND FANCY-FREE.  I TRY TO ANTICIPATE ALL OF MY CRITICS, AND AS YOU KEEP WRITING, YOU GET BETTER AT DOING THAT.
Associate Professor, Social Sciences

I don’t check the number of times my papers are cited. I guess I really should, but I haven’t made a habit of it. For merit reviews it is often recommended that you check the citations and, especially if they are high, report them. When I read papers that are in my research area, I do sometimes look to see whether those other papers have cited my work. When the paper is directly and specifically related, I think that it should include citation of my work. Similarly, I try to be thorough in citing others’ papers. You are right about the long process of getting something published. By the time it finally appears in print (and sometimes it never does because publishing is now electronic only for many journals), it’s sort of a let down.
Associate Professor, Sciences

 

I think I’ll leave it at that for now, but I could see this becoming an occasional theme. I want to dig deeper into their emotions and the attachment to their work. Is it all just business or is it creating art? Do high citation rates just lead to more publishing possibilities or is there some type of validation of intellectual effort?  Perhaps I’ll select a few items from each of their CVs and ask them to tell me about the creation process and the general reaction from peers. It’s one thing to talk abstractly about your work, but another thing when you mention a particular book or article and the effort that went into writing it.

Is scholarship a commodity?
yes

 

 

 

 

 

 

@brianmathews

 

This entry was posted in ScholarlyCommunications. Bookmark the permalink.