next »

[totoro]

This article seems to be behind the curve on what is happening:

1. Here in Australia for example government financial incentives to universities used to favor quantity explicitly over quality but that has changed. Depending on the field either journal quality or number of citations will be the way things are measured quantitatively.

2. Being able only to submit your best publications over a given period is routine here at least in research evaluation and grant applications.

3. Comments about print are silly as no-one reads print journals anymore. I only print out a very few key articles myself. Mostly I read them on a laptop. I do 90% of my reading of all sorts on my laptop (maybe it being a Mac helps :)).

4. There is no need to read everything in the area of ones interest if one knows how to do research properly. You need to read the most cited articles and articles in top journals. The rest can be safely ignored.

5. At top universities around the world quantity of publications has never been favored over quality.

There are, however, huge numbers of new open access journals popping up all over the place right now.

The flip side of the coin is that suppression of research results is bad for science. This results in "publication bias" which can lead to a misleading estimate of the size of a given effect. Meta-analysis of course can help overcome this to some degree.

[totoro]

Citation metrics need to be adjusted for field. There are huge differences across fields.

My research is on an obscure topic.  Does this make it "low-quality," since it gets few citations?  Is research a popularity contest?

[totoro]

I strongly doubt that top scientists have many papers that are not cited at all. Personally, less than 10% of my journal articles fall into the not cited by others in 5 years after publication category. So I doubt that people with more citations have more uncited papers. But 80% of my citations are to 20% of my papers and that is likely typical.

I am curious about the effect of fewer publications on the citation count. My understanding is that highly cited scientists tend to have many papers that are not cited at all.  There is some research into this (Simonton, Great Psychologists and Their Times). The problem is that often the community needs to be the judge of what is 'great research' and what is not. But that implies that you have to publish stuff to see how great it really is.  This book sort of tempered my outlook about publish-or-perish, since it argues in a pretty reasonable way (and backed up with a variety of data) that one almost necessary component to becoming a leader in a field is that you publish a lot more than the rest. 

Limiting to 5-6 pages seems pointless, more work for me as a researcher since it is easier to shove stuff under the rug in 5 pages so I have to dig through two different versions (what in the world do you cite anyway? The short or long version?). More work for me as an author since I have to write two manuscripts in one. More work for the journals since they have to track double versions.

Yes, there is a lot of stuff in the world. Yes, some of it is garbage. I would hazard that this has been true for a long time. I don't think massive restructuring of the academic world is needed.

[quasihumanist]

Citation metrics need to be adjusted for field. There are huge differences across fields.

My research is on an obscure topic.  Does this make it "low-quality," since it gets few citations?  Is research a popularity contest?

There are huge differences between different subfields within a field also.  One of the subfields I work in (incidentally the one that generally has LESS prestige within the mathematical community) probably easily generates two to three times the citations the other one does.

Are you going to look at my citation metrics based on some weighted average for the two subfields?  How are you going to decide what the weights are?  If you weight by the number of papers in each field, then for papers using ideas from both subfields, how are you going to decide what subfield it is in?  If someone has enough expertise to be able to read those papers and make an informed judgment (perhaps by saying X% in one field and the rest in the other), then he or she must be one of the one or two dozen people in the world qualified to understand my research and make a judgment based on actual understanding.  If you can find and trust an expert, what do you need the metrics for?.

[oldfullprof]

It's easy.  My research is "high quality" and yours is not "high quality." 

Seriously, the impact factor idea is manure.  Suppose your research was on white collar crime committed by doctors and hospitals.  I guarentee you that you'd have a low impact factor, even if this type of crime wastes billions of dollars a year.  The sheep follow grants, and grants usually reward non-threatening research.  This article is the usual "Colonal Blimp" treatment that comes out about every six months.  It's show-off arm waving.

[terpsichore]

I find it useful to see which of my papers are cited more than others. I can't always predict what will get picked up and what will languish, and it's entertaining to watch the trends.  Still, citations are only one tool to look at the value of research, and I hate to see that number overemphasized. One of my papers has been cited only a modest number of times, but it resulted in a small change in public policy. I'm especially proud of that work and its impact.

The h-index, on the other hand, is just silly.

[totoro]

Small differences in citation numbers and h-indices aren't meaningful because of statistical noise. Big differences though do tend to mean something from my experience. At least within a discipline. Citation records for departments even out some of the individual variability and can be useful in comparing productivity across universities I think. I'm dubious of "expert opinion" in some of these areas. For example, the Australian Research Council recently brought out a ranking of more than 20,000 journals into A*, A, B, and C categories. There are clear signs of the effects of lobbying. Typically, Australian journals are ranked in higher categories than I or my colleagues think is reasonable. Citation analysis is at least more objective.

[polly_mer]

I strongly doubt that top scientists have many papers that are not cited at all. Personally, less than 10% of my journal articles fall into the not cited by others in 5 years after publication category. So I doubt that people with more citations have more uncited papers. But 80% of my citations are to 20% of my papers and that is likely typical.

What do you mean by top scientists?  One of the giants in my field has more than 500 published papers.  He has a handful of papers that have thousands of citations and he has many more papers that have a couple dozen citations.  I haven't checked, but even if he falls into your characterization of 10% of journal articles not cited by other people, that's 50 papers!

On the other hand, one of the most interesting tribal lore stories that is passed around in polymers has to do with a guy called Huggins.  He published a little throwaway paper in the 1940's and then went on to do other things in his field, which is not polymers.  Coincidentally about the same time, a guy called Flory published a paper on a similar topic specifically applied to polymers.  One of the most influential theories in polymer solubility is called Flory-Huggins (in the 1970's, Flory won the Nobel Prize in Chemistry for his substantial work in polymers) and that throwaway paper by Huggins is very heavily cited.  However, if one had been checking on that Huggins paper just a few years within publication, the number of citations would be very small.  Indeed, the story that is told to all of us junior polymer people is that Huggins didn't know how important that paper was until a casual discussion one day where someone said something like "You're the Huggins who published this paper in the forties?  Wow, it's an honor to meet you, sir!" and explained.

Thus, while low-quality work may be published, immediate citation measures are not the way to judge.  In fact, in a joking mood one day, one of my mentors pointed out that publishing something that has an obvious error is likely to lead to higher citations as everyone and the dog cite you to mention that you did this thing wrong that they did right.

Limiting to 5-6 pages seems pointless, more work for me as a researcher since it is easier to shove stuff under the rug in 5 pages so I have to dig through two different versions (what in the world do you cite anyway? The short or long version?). More work for me as an author since I have to write two manuscripts in one. More work for the journals since they have to track double versions.

Chimety chime chime chime to this one.  Physical Review Letters has a four page limit.  About all one can say in four pages is "My current work has had a breakthrough.  Keep an eye out for the paper with the meat that will appear somewhere else in a couple of months".  While those rapid, short communications served a purpose in pointing out what people were working on that might change the field, the rise of the web, easy up-to-date indexing, and the proliferation of conferences with ease of attending them has made those four page papers much less useful.  At this point, I prefer that people just publish the full 12 page article in some other journal and put the data up as supplemental material instead of making me track down both the short announcement and the full article with supplemental data to make sure that I don't miss anything.

[totoro]

Sure that there will be many cases like that of papers that at first are not seen to be so important. Though I'd bet that today there were less cases with all the search tools now available. The argument is that they'll be in the tail of the distribution.

On the other hand, one of the most interesting tribal lore stories that is passed around in polymers has to do with a guy called Huggins.  He published a little throwaway paper in the 1940's and then went on to do other things in his field, which is not polymers.  Coincidentally about the same time, a guy called Flory published a paper on a similar topic specifically applied to polymers.  One of the most influential theories in polymer solubility is called Flory-Huggins (in the 1970's, Flory won the Nobel Prize in Chemistry for his substantial work in polymers) and that throwaway paper by Huggins is very heavily cited.  However, if one had been checking on that Huggins paper just a few years within publication, the number of citations would be very small.  Indeed, the story that is told to all of us junior polymer people is that Huggins didn't know how important that paper was until a casual discussion one day where someone said something like "You're the Huggins who published this paper in the forties?  Wow, it's an honor to meet you, sir!" and explained.

Thus, while low-quality work may be published, immediate citation measures are not the way to judge.  In fact, in a joking mood one day, one of my mentors pointed out that publishing something that has an obvious error is likely to lead to higher citations as everyone and the dog cite you to mention that you did this thing wrong that they did right.

Limiting to 5-6 pages seems pointless, more work for me as a researcher since it is easier to shove stuff under the rug in 5 pages so I have to dig through two different versions (what in the world do you cite anyway? The short or long version?). More work for me as an author since I have to write two manuscripts in one. More work for the journals since they have to track double versions.

Chimety chime chime chime to this one.  Physical Review Letters has a four page limit.  About all one can say in four pages is "My current work has had a breakthrough.  Keep an eye out for the paper with the meat that will appear somewhere else in a couple of months".  While those rapid, short communications served a purpose in pointing out what people were working on that might change the field, the rise of the web, easy up-to-date indexing, and the proliferation of conferences with ease of attending them has made those four page papers much less useful.  At this point, I prefer that people just publish the full 12 page article in some other journal and put the data up as supplemental material instead of making me track down both the short announcement and the full article with supplemental data to make sure that I don't miss anything.

[newbie]

Tenure and promotion incentives at my institution encourage quantity of research over quality of research.  When the incentives change I will be more than happy to change.

There are huge differences between different subfields within a field also.  One of the subfields I work in (incidentally the one that generally has LESS prestige within the mathematical community) probably easily generates two to three times the citations the other one does.

Building on these points, citations also take time to build up for any given article. In fields with long lag times for publishing, it seems like it's pretty easy for a junior faculty member to NOT have that many citations for published work when coming up for tenure. I like the idea of focusing on quality of articles instead of quantity for promotion, but if the measure for quality is how frequently an article is cited, then junior faculty in fields with long lag times for review and publication are in trouble.

[totoro]

That's where journal quality comes in. In the Excellence in Research for Australia initiative I mentioned above all humanities and most social science disciplines with the exception of psychology I think use journal quality instead of citations. Journal quality was assessed by "Expert Opinion" rather than by impact factors. In reality by a lot of bizarre decisions and lobbying apparently.

Building on these points, citations also take time to build up for any given article. In fields with long lag times for publishing, it seems like it's pretty easy for a junior faculty member to NOT have that many citations for published work when coming up for tenure. I like the idea of focusing on quality of articles instead of quantity for promotion, but if the measure for quality is how frequently an article is cited, then junior faculty in fields with long lag times for review and publication are in trouble.

[menotti]

Cardiology papers get cited more than ob/gyn papers.  Does that make them more important?

[anovan]

What factors cause articles to be cited?  I know of a case of 2 very much alike   articles. It were  case studies. Two  authors were involved in both, with some others in the respective articles. The journal the same, the title much the same, just a different entity from a bigger whole was used. The first article published in 2007 received 53 citations and the other one in 2008 nil so far. Fifty three citations in a 3 year span are quite a lot. Why  the other one is still at nil is a mystery. It could be that the highly cited one was seen by somebody, then cited, then retrieved by many  secondary readers and also cited. The other one may not have been spotted at all by anyone.

It is not matter of poor authorship.  One  scientist involved  has an overall good citation rate   (though he is not in HighlyCited.com).  He managed to publish in Nature, that article has 27 citations since 1999.

[polly_mer]

Totoro,

Point 1:  In case no one has said it, please pay attention to the fact that most of us around here put the quotation and then the response.  It's easier to follow the overlapping discussions on threads that way because context for your comments has been established.

Point 2:

Sure that there will be many cases like that of papers that at first are not seen to be so important. Though I'd bet that today there were less cases with all the search tools now available. The argument is that they'll be in the tail of the distribution.

I argue the reverse with the likelihood that a paper published today in a non-hot topic will languish because people depend heavily on the search engines instead of leafing through journals or randomly encountering things at the non-specialized conferences.  There's just too much for a person to deal with and specialization has taken priority over being a generalist.

I know that I've sat in several talks and read dozens of papers where people in my highly interdisciplinary dissertation topic claim to be the first to do something when it's just a case that they didn't search the standard indexes using the terms standard in a different field where quite a lot of work has been done in that area.  Because of that situation, when I've had literature searches pull up little on a topic that I think should have been studied, I check with colleagues in related fields to describe the research I'm doing and ask if they can suggest some terms for their field that I should be using.

[johnr]

The irony here is that this must be the 20th article I've read in the past five years about the proliferation of low-citation, mediocre research papers.