next »

[terpsichore]

I'm interested in what people think of this article, "We Must Stop the Avalanche of Low-Quality Research"  http://chronicle.com/article/We-Must-Stop-the-Avalanche-of/65890/

The authors argue somewhat convincingly that there are too many papers being published, putting a host of burdens on the the people and institutions involved in research, and diluting the impact of research. But some of their suggested solutions seem ineffective:

- use impact factors (numbers of citations) instead of numbers of publications. But if the number of publications is reduced, who is going to cite them?

- restrict manuscripts to a very small number of pages, with longer online supplements. This may be a good idea for many papers, but I would miss papers that develops a complex argument over multiple pages.

One of their suggestions is to use only a small number of publications (5 or 6) to evaluate scientists, with the idea of emphasizing quality over quantity. Some funding agencies already do this by restricting biographical information to include only a limited number of publications.

This is relevant to a lot of the discussions here on the fora, from library costs to increasing research expectations for tenure.  What do forumites think?

[biomancer]

As soon as my university is willing to grant me tenure based solely on my teaching and service, and not my publication record, I'd be happy to quit the publishing game.  However, the situation is still publish-or-perish, and as long as that is the case, I must continue to write as many papers as I can to save my own job.

I think the authors of the article completely overlooked that problem.

[aprilmay]

I thought the article was interesting, but I did not agree with much of it. The argument to use impact factors and citations seemed odd. Some people use these, whereas others do not, but surely everybody is aware of their existence and will use, or not use, them according to their views. Judging someone based on their 4-5 top papers does not make sense to me. What if Jack and Jill have 5 papers in top journals, with Jill's #6 to 25 papers also in top journals, but Jack only having 5 papers.  They would not be judged fairly by ignoring everything other than their top five. Quality matters, but so does quantity as a great promotion packet will have top quality research and lots of it. The author didn't prove, to me, that more research means lesser quality. If I have four postdocs that does not mean I have lower quality research than when I have two postdocs, although more research is being produced. The main question I have is who would judge what research is worthwhile and why does the author finds this to be something they can judge. That should be the role of editors and reviewers.

The bottom line: As long as there is a "publish or perish" nature to academia, people will continue to publish as much as they can.

[post_functional]

That papers tend not to be cited is just evidence that there are so many of them, not that they are poor quality.  It's just the age old problem of trying to sip from a fire hose.  The article does not do a convincing job of presenting a metric to determine that so many papers are poor.  

I suspect the root of the problem is too many people get accepted to college in the first place --> too many people become academics --> too many people write papers, of whatever quality.  The gatekeepers at all levels aren't doing the job of gatekeeping, because there's so much money to be made letting everybody in.  Oh, and we're under all kinds of pressure to pass them along, too, for fear of lawsuits and falling retention numbers.  So lots of people stay in the system and that means lots of redundant (though not necessarily poor) papers.

I will add my completely worthless anecdotal experience that trying to publish something truly unique and innovative in my field has already been met with (pure sense of the term) conservative resistance.  Let's not overlook the degree to which human nature is also at fault for the redundancy of papers.  Innovations threaten the status quo, and there is a lot of comfort in the status quo.

[charlesr]

Tenure and promotion incentives at my institution encourage quantity of research over quality of research.  When the incentives change I will be more than happy to change.

[post_functional]

Are you outright admitting that you submit lower quality research than you could be?  Serious question, not meant to be snarky at all.

[charlesr]

Under existing policies I am better rewarded if I write two lower-quality articles than one higher-quality article.  The Dean understands the perverse incentives and with luck the policy will change.

[lost_angeleno]

I'm old enough to remember boolean algebra.  Talk about a useless field!  Did nothing to help us understand the real world we worked in, was totally self-referential, was pathetically trivial.  Talk about low-quality research: the mathematical equivalent of pulp science fiction!
Whatever happened to boolean algebra, anyway?

[polly_mer]

Whatever happened to boolean algebra, anyway?

Well, Boolean Algebra is both a formalization of the algebraic aspects of logic, and the customary language of logic used by the designers of computers

Trivial and useless?  Um, yeah, if you think getting better computers that can do different kinds of operations other than just pure arithmetic and even do pure arithmetic faster is trivial and useless.

This is my problem with the idea that we have too much research.  We don't have too much research; we have inadequate tools for dealing with the mass of research that is produced to find the stuff we want so that we reinvent the wheel.

Yes, some shoddy research is published that shouldn't have been.  That's a failure of the peer-review system that is also collapsing under the weight because not enough qualified people do their part so that "peers" for some journals is pretty loosely defined.

[terpsichore]

Polly, you beat me to it.

Back to the question of quantity vs. quality of publications. At some point more than a decade ago, NSF changed its requirements to allow only 10 publications listed on CVs accompanying grant proposals. Whether the investigator has 10 pubs or 100, he or she must choose up to 5 related to the proposal and 5 additional publications to list. I find this makes it much easier to review the proposals. There's no reason universities couldn't do the same with promotions, but they are unlikely to do so. There's no incentive.

Instead, we do need tools to cope with the flood of research, including the load on the peer-review system. I would like to see some better tools developed. Possibly some of them will require Boolean algebra.

[t_r_b]

Um, I think g_a's riff on Boolean algebra was intended to be sarcastic.

[mathofsorts]

I am curious about the effect of fewer publications on the citation count. My understanding is that highly cited scientists tend to have many papers that are not cited at all.  There is some research into this (Simonton, Great Psychologists and Their Times). The problem is that often the community needs to be the judge of what is 'great research' and what is not. But that implies that you have to publish stuff to see how great it really is.  This book sort of tempered my outlook about publish-or-perish, since it argues in a pretty reasonable way (and backed up with a variety of data) that one almost necessary component to becoming a leader in a field is that you publish a lot more than the rest. 

Limiting to 5-6 pages seems pointless, more work for me as a researcher since it is easier to shove stuff under the rug in 5 pages so I have to dig through two different versions (what in the world do you cite anyway? The short or long version?). More work for me as an author since I have to write two manuscripts in one. More work for the journals since they have to track double versions.

Yes, there is a lot of stuff in the world. Yes, some of it is garbage. I would hazard that this has been true for a long time. I don't think massive restructuring of the academic world is needed.

[daniel_von_flanagan]

There is very little in this article that wasn't already said by Arthur Schopenhauer 120 years ago.  For example, from The Art of Literature:

What an inestimable boon it would be, if in every branch of literature there were only a few books, but those excellent!

For that reason and one other(*) I think that this Chronicle article is a waste of space, and should not have been published. - DvF

(*)The other is that any article worth publishing contains at least one proof of a theorem.  This is such a self-evident criterion for publication quality that, like Impact Factor, it requires no justification.

[mignon]

My research is on an obscure topic.  Does this make it "low-quality," since it gets few citations?  Is research a popularity contest?

[post_functional]

Good point.