 |
|
 |
|
|
|
 |
|
Is the Impact Factor Having Too Much Impact?
Wednesday, October 12, at 1 p.m., U.S. Eastern time
Since the "impact factor" was developed, in the 1960s, scientists have increasingly relied on it to judge the quality of a journal -- and, by extension, the work published in that journal. The process of calculating the impact factor is highly automated but, critics say, seriously flawed. And in an effort to raise the profile of their journals, some editors may be rejecting papers that are unlikely to be widely cited, or even manipulating the system.
Is the emphasis on impact factors hurting smaller journals? Are scientists avoiding certain areas of research because their findings will not be published in high-impact journals? Is the practice having an effect on hiring and promotions as well? How useful are alternative ways of measuring the value of journals and the research they publish?
» The Number That's Devouring Science (10/14/2005)
» Impact Factors Run Into Competition (10/14/2005)
Anurag A. Agrawal, an assistant professor of ecology and evolutionary biology at Cornell University, has served on the editorial boards of five journals. He recently published a letter in Trends in Ecology and Evolution decrying some common editorial practices designed to raise citations and journal-impact factors.
Richard Monastersky (Moderator):
Welcome to The Chronicle's live discussion on journal impact factors. My name is Rich Monastersky and I wrote a story in this week's issue about the influence of impact factors and ways that people are abusing the system. Our guest today, Anurag A. Agrawal is an assistant professor of ecology and evolutionary biology at Cornell University. We invite you to send in your questions or comments about your own experiences with impact factors.
Question from Richard Monastersky:
In a letter to Trends in Ecology and Evolution this year, you criticized editors at some journals for asking authors to add citations to previous articles in the same journal--a practice that would boost the impact factor of the journal. It seems to be fairly common, at least in some disciplines. So don't scrupulous editors put their journals at a disadvantage by opting out of this practice.
Anurag A. Agrawal:
Good point... as in life more generally, it is a balance between costs and benefits of the action. I guess I would argue that if we, as the scientific community, reject the practice than editors will be embarrassed into not doing it any more. We must make the cost of stopping the practice (potentially a reduced impact factor) be less than the cost of continuing (disapproval by authors, reduced manuscript submissions, reduced prestige).
Question from Philip Davis, Cornell University:
The criticisms brought up in the Chronicle article (The Number That’s Devouring Science) about possible manipulation of Impact Factors are fair and warranted.
The proposed solutions in the follow-up article (Impact Factors Run into Competition) are not without their own problems, and we shouldn’t be mislead to implement new measurements that are just as ripe for misuse and abuse as the Impact Factor.
Using a small group of faculty to rate the merits of articles (implemented in the Faculty of 1000 Factor) is no less protected from favoritism, cronyism, and sexism than the process of blind peer review – in fact, it is even more open to such abuse. A cursory view of F1000 top ranked articles in Biology demonstrates another obvious point: highly-ranked articles are published in highly-ranked journals.
In addition, BioMed Central’s labeling of "highly accessed" articles in their Open Access journals feeds the argument that downloads equals prestige, and is open to systematic downloading from unethical authors who want to self-promote. While self-citation is a current problem, it is transparent and open to criticism and scorn, as evidenced in the Chronicle article.
Self-promotion is a natural desire for those engaged in research and publication. Still, any new metric to evaluate research should be demonstrated to increase the openness and transparency of the process -- anything else will be to invite more misuse and abuse.
Anurag A. Agrawal:
Right, I welcome comments from others like this... all I can say is that I totally agree that Faculty of 1000 is another 'funny' measure, and one that I find with quite a bit of bias.
Question from Diane Sullenberger, Proceedings of the National Academy of Sciences:
Do you think that the calculations and categorizations of papers that lead to impact factors should be disclosed annually so that the effect of reviews, news articles, and editorials can be transparent?
Anurag A. Agrawal:
I like the spirit of this question.... especially given the some of the unbelievable statistics provided in the article. Though I think this would be useful and leave at least some of the relevant information on the table, we still would not know how many self citations are due to editors' bullying requests.
Really I think that journals should be embarrassed into being more honest. Which is hopefully what we are doing. Perhaps this means that we should all be a bit more vocal and also try to publish more in not-for-profit journals (which often have page charges, but also do not have businesses as the bosses).
Question from David C. Montgomery, Dartmouth:
Has anyone tested retroactively what the "impact factor" schemes would yield for the publication records of great achievements of the past? For example, pivotal physics papers on quantum mechanics, relativity, nuclear physics, chemical physics, and particle physics published between about 1900 and 1960, say, are arguably the most fundamental contributions in the history of any science since journal publication has been more or less standardized. Agreement on which ones they are is now essentially universal among those who know what the contributions are about. It would be interesting to know how these papers would fare in the ISI evaluation schemes. Of particular interest would be those weighting factors that greatly reward citations that occur within the next couple of years after publication, wouldn't it?
Anurag A. Agrawal:
You are right on the money... Impact factors about impact in the past couple of years... I was just discussing this with my colleagues at lunch... for papers that are ten years old, we can pretty easily (by talking to people, etc) get a sense for whether a paper has made an impact on the field or not. But, many of the decisions that are being made using impact factors have to do with 1) papers are have not yet had the time to stand the test of time, and 2) studies not directly in our own research areas. Hiring new faculty at the department level often involves interviewing candidates that are within 5-10 years of getting their PhDs (and even younger papers) and with varied research interests. ISI (the company the produces the impact factor) has a good explanation on their web site for how to interpret impact factors… they also present other indices such as half life, immediacy, etc.
Question from Richard Monastersky:
What kind of changes would you suggest that could either improve impact factors or substitute for them?
Anurag A. Agrawal:
I don't have a clear answer... the most obvious is that we should actually "read" the papers and decide for ourselves whether they are important or not... several issues preclude this... namely time and knowledge of specific subdisciplines.
Comment from E. Roy Weintraub, Duke University:
I'm an Associate Editor of a journal which was the subject of a "Hot Type" note in the Chronicle of September 2, 2005. Our subdiscipline has around ten journals, but only one (ours) was counted in the SSCI, and then it was dropped for "low impact". But with only one "counted" journal, there were very few citations, hence no impact! If the top three journals were counted, the cross citations are large, as is "impact". Thompson ISI, as a private entity, is in fact structuring resource allocation to subdisciplines in my field but takes no responsibility for the outcomes. In the UK, and Italy and the Netherlands, the impact scores are immensely important for the allocation of public monies, but the process is quite flawed. Comments?
Question from Diane Sullenberger, Proceedings of the National Academy of Sciences:
How is online access to research papers, in particular, free online access, affecting research impact?
Anurag A. Agrawal:
Not clear... it does seem that the free availability of articles has got to make things a bit more fair. Nonetheless, there is always going to be a prestige factor.... those articles in high impact journals or by authors from powerhouse universities will likely get more attention simply because of the source. Free on-line access should also further bias what is being cited however... some journals do not have the archives available as PDF files (for various reasons) ... given the current climate of scientists going to the physical library less and less, the are more likely to cite papers from journals with free on-line access.
Comment from Jim Testa, Thomson Scientific:
In response to the posting from Dartmouth, Impact Factors are applicable at the journal level. Thomson does provide data about citation patterns at the article level.
Question from Richard Byrne, Chronicle of Higher Education:
Does the growing cynicism surrounding impact factors -- both in their use as a single yardstick for widely-varying scientific disciplines and in the abuses that the article detailed in boosting these factors -- have the potential to choke off promising careers or areas of research? Are there specific disciplines that may be more harmfully "impacted" by the factor?
Anurag A. Agrawal:
Yes, I think there is this potential... there was an earlier question from an editor of a journal in a "low impact" field... he argued that some fields, by virtue of them being small and, in his case, ISI only tabulating citations from one or a few of the journals in that subdiscipline, are made to seem lower impact than they should be. Impact factors seem especially dangerous when comparing across fields.
Richard Monastersky (Moderator):
We are more than half way through the discussion, so if you are thinking of asking a question, please do so now.
Question from Jennifer K. Ruark, Chronicle of Higher Education:
Even without impact factors, scholars in the humanities routinely judge job candidates or tenure cases based on the publications listed on their c.v.s-- not by reading the work but by recognizing the prestige of the journals (or publishing houses) where the work appeared. Given how pressed for time academics are, would getting rid of impact factors really make a difference in how scientists were evaluated, or would people continue to be rated by where they published more than by what they published?
Anurag A. Agrawal:
I don't advocate getting rid of the impact factor... I simply think we should keep an open mind and at least use several indices... (including our own personal judgment of the prestige of the publication venue). There is no way that this process is going to be made simpler, with a silver bullet, and there is also no possibility to make it completely objective. Our only hope is to not be locked into a self-imposed monopoly where impact factor is god. In addition, we should try to close the door on abuses ... I think the clear path here is to embarrass the editors or publishers that make compromising requests!
Question from Rebecca Minnillo, Soc. for Invest. Derm.:
What do you see as the role of organizations that own and rely financially on these peer-reviewed journals in addressing the issues outlined in the article? What(if any) effect will open access have?
Anurag A. Agrawal:
I am not sure if I understand your question... If you are referring to the publishers (that own the journals) I strongly believe that their role should be to make an honest living... that is to produce a high quality product without attempting to artificially inflate its value. As scientists we have to take part in the processes. As libraries and scientific institutions, we need to take a strong position on the role of science, and at least make the attempt to keep it somewhat objective. Open access journals have the flavor of "science for the sake of science" which I applaud. However, I do not think they are immune to the potential abuses.
Richard Monastersky (Moderator):
Editors and publishers of new open-access journals say they are hurt by the way impact factors are calculated because you need several years of citation data before an impact factor can be determined. So a start-up journal faces a tough time getting submissions from scientists who are concerned about impact factors.
One interesting fact is that the Public Library of Science Biology journal saw its submissions double after it received its first (and quite high) impact factor this summer.
Question from Christopher Surridge, Public Library of Science:
Do you think that the problems we see are specific to the Impact Factor and so could be solved by finding a better metric? Or is it more the case that the importance of a piece of work is inherently unquantifiable?
Anurag A. Agrawal:
It is true that the real impact is determined down the road, depending on the scientific field, maybe 10-20 years later. Impact factors, in particular, have specific properties that are useful and also limiting. Perhaps we should just keep a broader perspective on the many possible measures of impact.
Question from Andrew Mytelka, Chronicle of Higher Ed:
The editors of several prominent journals are quoted in the article as lamenting the influence of impact factors. One editor says "we are slaves to the impact factor." Another says that the impact factor "has taken on a life of its own." My question is: If those leading editors say there's nothing to be done about the power in science of the impact factor, what hope is there that change will come? And from whom?
Anurag A. Agrawal:
Perhaps more mathy people, maybe sociologists, should take a look at this issue. It is quite possible that we have left the issue to businesses or scientists, neither of which are really trained to think about this issue. There must be alternatives. Clearly, part of the onus is on the user of these factors to use more than one, and to evaluate what they really mean. For example, ISI has several indices, but we rarely see reference to them (we mainly see reference to the impact factor). Many other indices should be possible, we just need creative and interested people on the job. I imagine that there are some great PhD thesis possibilities here for students interested in the process, sociology, ethics, and history of science.
Question from Jennifer K. Ruark, The Chronicle of Higher Education:
A question readers who are not scientists might have is, Is it typical for journal editors to point an author to research he or she should cite in the interest of thoroughness, and if so, won't the leading journals inevitably be asking authors to cite articles that happened to appear in those journals?
Anurag A. Agrawal:
Yes, this is an issue. However, it is rarely the highest impact journals that are asking their authors to cite papers from their journal ... it is more so a problem with medium impact journals. The point here is that editors should occasionally ask authors to cite particular papers for completeness. But it is rarely the case that all of these papers are from ‘that’ journal. Also, many of the cases of abuse are much more general... Editors saying something like "can you look through the past several issues of our journal and cite relevant papers... we see this as important to maintain continuity and themes in the journal" to my eyes, this is blatant abuse. There will be other cases where a single paper from ‘that’ journal will be important to cite. But it smells bad when the request is so general or when it just so happens to be several recommendations from the past two years of that journal.
Question from John Drake, National Center for Ecological Analysis and Synthesis:
It appears that another possible way to intentionally inflate impact factors is to release an article online months before it is issued. Then, the article has a longer window during which it can be publicly circulated, incorporated into manuscripts, and cited in subsequent articles. Thus, for instance, if an article is distributed 4 months before it is issued, it has 28 months to be cited, rather than 24, to be included in impact factor, an increase of more than 15%. My questions: Are you aware if this is done intentionally for this purpose? Is it known if articles which publish online prior to print have higher impact factors?
My opinion is that the solution is to reduce our overall, discipline-wide reliance on impacts factors and similar metrics.
Anurag A. Agrawal:
Interesting point. I am unaware if these pre-press on-line publication of articles impacts impact factors. In general, my sense is that paper publication takes time, and there is no reason for publishers to wait. So, in general, I am a big fan of early on-line publication. Science is about getting the information period. Getting it out their and circulating seems only good. It does seem that this should be addressed, at some level by ISI, as to whether some papers are getting a bigger window than others.
For that matter, ISI has done a study on the impact of self citation. I have read
the full report (available freely on-line as a PDF), and there are three reasons I think some journals are continuing to promote self citations, despite the fact that ISI’s study found little effect of self citation: 1) the journal administrators may be unaware of the generally minimal impact of self citation, but are blindly going forward in a fruitless pursuit of higher impact; 2) they are often the lower impact journals that encourage self-citation… ISI’s study found the least effect for high impact journals… It is precisely these lower journals which can most benefit from self citation, 3) the ISI study was based on past correlations... in other words, yes, it is true that currently there is only a weak relationship between self citation and impact factor... but I believe (from my own experience and talking with many others) that the practice of ramping up self-citation by journals is just getting going and is on the rise. Thus, the past correlations may not be reflected in the current shifts introduced by some journals. This is especially the case
when you think that, on average, there should be more self citations in upwards of 25% of the articles published by some journals (this was indicated to me in confidence by a managing editor of one journal-- that is, that they suggest specific papers to be citied in 25% of accepted papers).
Additionally, in an era of shorter papers, where authors are being asked to cite *fewer* papers overall… increasing self citation will have a bigger effect than initially imagined.
Question from Andrew Mytelka, Chronicle of Higher Ed:
The power of the impact factor in the scientific enterprise, and the complaints about its influence, are similar to the role of the U.S. News & World Report ranking of colleges and universities, and the complaints about its influence. College officials have griped for years about the power of the U.S. News rankings, what they see as its questionable methodology, what they criticize as efforts to "game" the system through admissions practices, and so on. Yet the officials also are quick to send out press releases hailing their rise in the rankings. Do you agree with this analogy? And if you do, won't any efforts to defy the power of the impact factor, like efforts to defy the rankings (by refusing to cooperate with U.S. News, for example), be likely to fail?
Anurag A. Agrawal:
I agree with your point... I hope I am not beginning to sound like a completely broken record :-) What I feel are the critical issues are 1) the use of several measures to estimate the impact (of articles, researchers, journals, whatever) and 2) to keep the measures as honest and transparent as possible. We need measures of impact. All will have some limitations... perhaps the best we can do is to be clear about what the limitations are. I do not think ISI has driven the impact factor to the amazing status is has... Scientists have done this!!! That is OK, but we should also consider other measures. The second point is that there is absolutely no defensible position for attempts to inflate a publication’s own impact factor by using unethical bullying.
Question from Ursula Zyzik, Saint Xavier University:
Just a comment on the pedagogical implications of the concept of journal selectivity as expressed by the impact factor. We often advise students to pay attention to the number of cited references by the author advocating, in good faith, that cited references are there because they are related to the issue discussed in the paper. The instances of self-citation should make us more skeptical.It is rather ironic, I think, that librarians and educators direct so much attention now to inculcate in students the serious problems with authority and credibility of web resources with the almost universal acceptance of authority of high impact journals.
Anurag A. Agrawal:
That is an interesting point!
Question from Andrew Mytelka, Chronicle of Higher Ed:
Since the federal government is a huge source of funds for scientific research, do you think it should have an interest in the impact factor's role in setting research priorities? If so, what should it do?
Anurag A. Agrawal:
It may be worth the effort of some governmental agencies, especially those that fund scientific research, to investigate impact factors and journal practices.
Question from Philip Davis, Cornell University:
So what you are advising is to keep the system as transparent as possible and publically scorn those who attempt to game the system?
Anurag A. Agrawal:
Yes. "Scorn" seems strong... I would at least say "expose" ... scientists can then make up their own mind about submitting manuscripts, etc. We are all under some amount of pressure to publish in good places... but most scientists also value the attempt to make science somewhat objective.
Richard Monastersky (Moderator):
We've reached the end of our colloquy on impact factors. I'd like to thank Anurag for taking the time to participate. Another thank you goes out to those who sent in questions. If you have any thoughts about the issue or comments on the article, you can send me an e-mail at rich.monastersky@chronicle.com