June 20, 2008
3 Mathematics Associations Caution Against Overreliance on Impact Factor
Three mathematics associations — the International Council for Industrial and Applied Mathematics, the International Mathematical Union, and the Institute of Mathematical Statistics — have examined citation-based statistics such as the impact factor and the h-index, and concluded that the measures are often misunderstood and misused.
The use of the impact factor, developed as a way of ranking scientific journals, as the main tool to evaluate the quality of research has boomed during the last decade, and this measure has become crucial in hiring and tenure decisions, as well as in the awarding of grants.
In a report released this month, the associations say that the impact factor and other citation-based statistics should not be dismissed as tools for assessing research quality, but they warn against using such metrics as the only evaluation method and not taking into consideration other factors, such as peer review. —Maria José Viñas
Posted on Friday June 20, 2008 | Permalink |Comments
Commenting is closed for this article.
Previous: Law Professors File Brief in Support of Piracy Defendant
Next: Indiana U. Is Host for Online Collection of Sheet Music
I remember early in my career, as I was entering into the tenure track, an older and wiser colleague noted that “your colleagues can count, but they can’t read.” We keep finding cleverer and cleverer ways to count.
And the final irony: Mathematicians reminding us to read! It doesn’t get any better than this! (-:
— Ray Jun 20, 08:33 AM #
Maybe I misunderstood this, but it seems to say that the impact factor and counting citations are used as a measure of quality. Shouldn’t that be “quantity”? Surely no one would actually evaluate (estimate the value of) something merely by counting the hits. Unless the subject is baseball or a website.
— dan Jun 20, 11:10 AM #
Dan, it isn’t just counting hits. It is an interesting, but flawed means of looking at quality based on how many times the each work published in a journal is cited in other works. The idea is that the journal must be prestigious if the individual articles it publishes are cited many, many times. So, journals like Science and Nature are really high up on the list, which is not such a bad thing, given that they are important and prestigious. But there are numerous studies that have shown there are major flaws in the formulae used. In a nutshell, it seems that the author of the article that gets cited is the holder of the prestige, not the journal. Do a little googling and you will find all you need to know.
— Robert Jun 20, 12:12 PM #
Any formula that tries to assess “quality” by numbers is necessarily flawed. Quality is necessarily (are we ready for this?…) a qualitative, not a quantitative, judgment. Quality scores for faculty based on grants and publications are bunk because they ignore teaching, and they also ignore the content of the grants and publications. I see no difference in this case.
— Al Jun 20, 01:00 PM #
I apologize. My attempt at irony was heavy-handed. I was trying, albeit clumsily, to make the same point as #4.
— dan Jun 20, 03:37 PM #
The IMU’s cautions are welcome: Research metrics do need to be validated; they need to be multiple, rather than a single, unidimensional index; they need to be separately validated for each discipline, and the weights on the multiple metrics need to be calibrated and adjusted both for the discipline being assessed and for the properties on which it is being ranked. The RAE 2008 database provides the ideal opportunity to do all this discipline-specific validation and calibration, because it is providing parallel data from both peer panel rankings and metrics. The metrics, however, should be as rich and diverse as possible, to capitalize on this unique opportunity for joint validation. Open Access not only provides more metrics, but it increases them, as well as providing open safeguards against manipulation and misuse.
http://openaccess.eprints.org/index.php?/archives/417-guid.html
Stevan Harnad
— Stevan Harnad Jun 20, 03:56 PM #
I fully agree with #6 that we need broad analysis of impact. Impact factors are a poor measure in fields with journals that are slow to review or publish because the journal impact factor is based on articles cited within two years of publication—a short time in many social sciences. Also, the h-index has value but it’s hard to say that a young scholar who has an h of 20 because they have 20 articles cited 20 times (400 citations) is having more impact that a peer who has 10 articles cited 40 times (h of 10). These metrics are certainly more robust the farther into a career you are looking for productivity indicators
— Rick Barth Jun 21, 04:19 PM #
Impact factors are inferior to Google’s pagerank algorithm. They also don’t normalize by the duration of the publishing career or self-referential citations.
— Mark Kantrowitz Jun 23, 05:09 PM #