next »

[terpsichore]

These are great tips and I appreciate your help.  Some have already provided good leads.  I'll try to explain in more detail what I am looking for.

Although it is important and I may need it later, I am not looking for information about a journal's impact.  There are good metrics available and there's one specific to my discipline that works.  Rather, I am trying to measure the impact of individual articles within a journal.  So, for example, volume 21 of a journal contains five articles: A,B,C,D, and E.  How can I efficiently and accurately measure the impact of each journal article in relation to the other and in relation to the broader articles in the discipline.

My overall goal might clarify things too.  I am the editorial board member of a journal.  I would like to write a brief retrospective of the journal and discuss the influence of individual articles within that journal.  For example, "articles in the Journal of Fungus have changed the discipline.  For example, Frank's article in volume 9 has been cited thirty times and this and that."  I want to capture the scope and influence of both the journal itself and the articles within it.  Efficient and accurate metrics to help me do that would be much appreciated.

Thanks for suggestions so far and more are welcome.

Untenured

Couldn't you do this by picking a few key articles and following them on Web of Science or
another indexing service?  It seems to me you'd want to use anecdotes for such a retrospective.

Impact factor of the journal is good, but I can see that the standard deviation would also
be helpful. Does anyone report that? I've never seen it.

[svenc]

These are great tips and I appreciate your help.  Some have already provided good leads.  I'll try to explain in more detail what I am looking for.

Although it is important and I may need it later, I am not looking for information about a journal's impact.  There are good metrics available and there's one specific to my discipline that works.  Rather, I am trying to measure the impact of individual articles within a journal.  So, for example, volume 21 of a journal contains five articles: A,B,C,D, and E.  How can I efficiently and accurately measure the impact of each journal article in relation to the other and in relation to the broader articles in the discipline.

Web of Science and Google citation counts will be the easiest to access.  I'm going to PM you an example of a recent piece in my subfield that might have some helpful methodological options for you.

Also, if your journal is published online with one of the major publishing houses, they can probably provide access statistics for each article.  The number of reads for each article is an interesting measure to compare to what has been cited the most (and if you're in a field with a slow turn-around in citations, can help provide a glimpse into what will be cited down the road).

[august_leo]

My overall goal might clarify things too.  I am the editorial board member of a journal.  I would like to write a brief retrospective of the journal and discuss the influence of individual articles within that journal.  For example, "articles in the Journal of Fungus have changed the discipline.  For example, Frank's article in volume 9 has been cited thirty times and this and that."  I want to capture the scope and influence of both the journal itself and the articles within it.  Efficient and accurate metrics to help me do that would be much appreciated.

I would try to get a work study student or journal editorial assistant to look up the citations of each article in the past x years. Make an excel file with article; year published; citations 1 year later; citations 2 years later; etc as columns. Include both "total citations 1 year later" and "other citations 1 year later" (i.e., citations from articles where none of the authors of the original are authors. So, if we are looking at Untenured & August-Leo (2006) in 2007 August-Leo, LarryC and Spork (2007) would only count in total). Then you can run interesting statistics and say cool things like:

"within 5 years of being published, most of the papers on Truffles were cited by 6 others in the field"
"most articles on Button Mushrooms are cited within 1 year of being published (M = 89%) and continue to be cited as many as 10 years later (M = 75%)"

[oatmeal]

I have read this thread with interest. It seems to me, and perhaps I am wrong (probably), that the measurement of influence is linked to how many times an article is cited? Now, is this indeed a good way to measure influence? I am not sure. And, how does one actually find out how many times an article has been cited? Is there a fool-proof and universally accepted way to do this? I am intrigued. In the Humanities and Social Sciences, I would say it is a little more tricky. If anyone has ideas, I would be interested to hear them. Certainly a fascinating thread.

[cogscientist]

Once you have the citation information for each article in the volume, you can see how they compare to the citation rates in your field, using WoS.

Go to "Additional resources", then "Essential Science Indicators", then "Citation analysis: Baselines", then "View the percentiles table".

Now look for your field in the first column (maybe "Social Sciences, general").

You can then find a wealth of information such as: 20% of the papers in this field published in 2005 have been cited 7 times or more, only 1% have been cited more than 30 times, and so on, for various percentiles and years of publication. That should give you some useful benchmarks for what you want to do?

[carebearstare]

I have read this thread with interest. It seems to me, and perhaps I am wrong (probably), that the measurement of influence is linked to how many times an article is cited? Now, is this indeed a good way to measure influence? I am not sure. And, how does one actually find out how many times an article has been cited? Is there a fool-proof and universally accepted way to do this? I am intrigued. In the Humanities and Social Sciences, I would say it is a little more tricky. If anyone has ideas, I would be interested to hear them. Certainly a fascinating thread.

I would agree with Oatmeal here. I have found citation indexes to only be of limited use, especially in the humanities. Another metric might be how many times an article is taught in a foundational graduate level seminar in the discipline, or whether it created a subspecialty around itself. These things are harder to measure but possibly more useful. Could you conduct survey research?

[hyperbole]

I had the same doubts about citations as an index of influence for humanities articles. But then again...what article started a subspecialty without being cited a gazillion times?

[carebearstare]

I had the same doubts about citations as an index of influence for humanities articles. But then again...what article started a subspecialty without being cited a gazillion times?

That's true, but I guess it would depend somewhat on the historical timeframe. There are lots of people, for instance, who write about postmodernism without citing Jameson at this point. For some influence, you might have to go through a few "generations" to get to the subfield, at which point its original catalyst might be lost in the citations. In my own dealings, I've also been skeptical about how well books show up in those citation indexes, so if an article inspired a very foundational edited collection or monograph, that might not be as apparent. Not to mention conference proceedings, etc.

Just thinking here. I am not suggesting abandoning citation indexes, but I wouldn't use them as the only metric.

[collegekidsmom]

I see what you are trying to do. You are trying to find out what the most influential articles are in a volume, issue, year, or over time in a particular journal.
I would not use Google Scholar data in trying to make a statement that will be published. Google Scholar data might be better used in an informal way, or in a discipline where citation data cannot be otherwise reported by Thomson WOS or by Scopus. The problem with relying on Google scholar for any more than informal analysis is that Google does not publish its coverage lists-it claims to cover what it deems "scholarly." So, it is fine for an overview, but between that and the unpublished algorithm, and the general reported inaccuracies, it is hard to stand behind the results for a formal study. Same is true for all metrics that rely on GS for data-such as Harzing Publish or Perish.
"H-index" is used most often for studying the impact of an individual researcher.
I do think Eigenfactor may shed some light.

If your field is covered in WOS and Scopus, those would be more authoritative sources of the data then GS. You may be able to use one of the Thomson products such as Journal Performance Indicators, or have them run the numbers. Scopus also has various tools, such as Scopus Journal Analyzer might be of interest. If you know someone who works with scientometrics or a librarian involved in this kind of work, that would be helpful. Alternately you could contact either company for best approach.  Tell them what you are trying to do and ask what product and approach would give you exactly what you want. As long as you aren't in humanities and some of the social sciences(even with newer expanded coverage by Scopus), there will be a product to meet your needs. The two companies that have the data and the assistance would be Thomson Reuters and Elsevier-if you need to be authoritative reportable numbers.