next »

[blue_up_north]

Hi there, I'm new here.

I'm wondering if there's a way of converting journal impact factors into REF stars? The top-rated journal in my field has an IF of around 8, and the rest of the top ten come in around 5ish. Would it be reasonable to say  that top-ten journals might be rated 4*, the next 20 or so 3*, and so on? I have no idea if 4* is being reserved for the top general journals (Nature, et al.), or if star rankings will be domain-specific in terms of IF.

Any advice much appreciated.

[drspouse]

My impression is it is completely random, and very subject-specific - even within those disciplines that would publishin in Nature etc.

Sorry to be no help at all!

[expatinuk]

Any how the REF is supposed to rate the article not the journal. So concievably you can have a 4* article in a lesser jounal and a low rated article in a top journal.

[drspouse]

Any how the REF is supposed to rate the article not the journal. So concievably you can have a 4* article in a lesser jounal and a low rated article in a top journal.

Which is all very nice but I don't quite get how they are supposed to tell what the importance of an article published this year is, already.

[scotia]

Any how the REF is supposed to rate the article not the journal. So concievably you can have a 4* article in a lesser jounal and a low rated article in a top journal.

I know this is the theory, but I am not sure how they can rank articles that they haven't read (I have been told - by reliable sources - that in the UofA I am returned under only about 25% of materials were read in the last RAE. They therefore must have some proxy of quality, and some method of ranking journals seems the most likely).

[mingus]

I had a chat with a REF/RAE-panel type for my UoA, and here's what I was told: Just as has been stated above for another UoA, less than 25% of papers are actually read.  For the rest, the journal is a proxy for 1* to 3*.  Rank the journals (IF), divide into three groups, and then rank the papers therein.

4* is expected to be "ground-breaking" work.  Given that the panel membership covers all main aspects of the UoA (my UoA), at least one person is presumed to be aware of ground-breaking work in a given area.  There is also the expectation that is a person has indeed produced ground-breaking work, they will not be shy about it and will say so; the panel will then take a closer look.   A key point to note from what this chap told me is that it implies the acceptance that 4* work can be found almost anywhere.

[science_expat]

My sub-panel allegedly read all outputs last time and hence they explicitly excluded journal impact factors. That said, a Nature paper is likely to start with a higher expectation of being good quality than one published in the regional journal of not very interesting science.

We're also told that paper titles and abstracts are very important and need to emphasize the wider implications of the work.

[blue_up_north]

Thanks, everyone. My panel will read every output - I guess I'm thinking in terms of using journal IFs as a proxy for "quality" when assessing the returns of other people in my Department  (which I will have to do), but perhaps it's not that straightforward.

[mingus]

My sub-panel allegedly read all outputs last time and hence they explicitly excluded journal impact factors. That said, a Nature paper is likely to start with a higher expectation of being good quality than one published in the regional journal of not very interesting science.

We're also told that paper titles and abstracts are very important and need to emphasize the wider implications of the work.

It would be interesting to see if there is any correspondence between what they condidered good quality and the IFs of the journals where the papers were published.  People might say they "ignore" the IFs, and they might not even know the IFs,  but if you ask them what they consider good journals, then lo and behold.

Titles and abstracts are very important---especially if not every paper gets read :-)

[mingus]

Thanks, everyone. My panel will read every output - I guess I'm thinking in terms of using journal IFs as a proxy for "quality" when assessing the returns of other people in my Department  (which I will have to do), but perhaps it's not that straightforward.

We are doing this.  It helps deal with people who have been busy publishing rubbish but claim they are very REF-able.  Rather than spend time in endless, useless debates, we simply tell them that if their work is so great, it should have gone into a proper journal and not the International Journal of The Northern Sub-County of Outer Mongolia. Unless, of course, they have other indicators of greatness, e.g. Best Paper in X for 2009.  

[science_expat]

My sub-panel allegedly read all outputs last time and hence they explicitly excluded journal impact factors. That said, a Nature paper is likely to start with a higher expectation of being good quality than one published in the regional journal of not very interesting science.

We're also told that paper titles and abstracts are very important and need to emphasize the wider implications of the work.

It would be interesting to see if there is any correspondence between what they condidered good quality and the IFs of the journals where the papers were published.  People might say they "ignore" the IFs, and they might not even know the IFs,  but if you ask them what they consider good journals, then lo and behold.

Titles and abstracts are very important---especially if not every paper gets read :-)

The problem, of course, is that we don't know the scores that were assigned to each output - or indeed each individual.

I guess you could look at the distribution of publications in top class general journals* (e.g. Science and Nature), top class discipline specific journals, good quality international journals etc... and assume this is a proxy for quality. Then compare this distribution with % 4*, 3*, 2* etc obtained in RAE 2008 ... It would be interesting but I suspect that the potential for spurious correlations would be quite high.

*Clearly this post is STEM specific

[blue_up_north]

Thanks, everyone. My panel will read every output - I guess I'm thinking in terms of using journal IFs as a proxy for "quality" when assessing the returns of other people in my Department  (which I will have to do), but perhaps it's not that straightforward.

We are doing this.  It helps deal with people who have been busy publishing rubbish but claim they are very REF-able.  Rather than spend time in endless, useless debates, we simply tell them that if their work is so great, it should have gone into a proper journal and not the International Journal of The Northern Sub-County of Outer Mongolia. Unless, of course, they have other indicators of greatness, e.g. Best Paper in X for 2009.  

I'm glad to hear this, but the problem I am having is convincing certain people of the definition of a "proper journal"... They are often genuinely convinced that a journal with an IF of (say) 0.45 is the leading journal in their area. As I don't know enough about their particular field, I can't construct a strong argument against this, but my instinct is that such a journal cannot claim to be leading, regardless of the size of the field, with such a relatively weak IF. I worry about the navel-gazing effect, whereby a single journal may well be the first port of call in a particular sub-field, but if papers it publishes are generally ignored, then it cannot be "leading" in a more general sense.

[science_expat]

Thanks, everyone. My panel will read every output - I guess I'm thinking in terms of using journal IFs as a proxy for "quality" when assessing the returns of other people in my Department  (which I will have to do), but perhaps it's not that straightforward.

We are doing this.  It helps deal with people who have been busy publishing rubbish but claim they are very REF-able.  Rather than spend time in endless, useless debates, we simply tell them that if their work is so great, it should have gone into a proper journal and not the International Journal of The Northern Sub-County of Outer Mongolia. Unless, of course, they have other indicators of greatness, e.g. Best Paper in X for 2009.  

I'm glad to hear this, but the problem I am having is convincing certain people of the definition of a "proper journal"... They are often genuinely convinced that a journal with an IF of (say) 0.45 is the leading journal in their area. As I don't know enough about their particular field, I can't construct a strong argument against this, but my instinct is that such a journal cannot claim to be leading, regardless of the size of the field, with such a relatively weak IF.

Are you in STEM? If so, there are data available from both ISI and Scopus that rank journals by sub-field and also give useful information on how to normalize paper citations by discipline and year of publication.

Alternatively, do a comparison within the department. For example, the distribution of impact factors in which you publish is such and such and then show where each person's pubs are with respect to the distribution.

[blue_up_north]

Thanks again. Several of the journals people are publishing in don't even appear in the ISI/Scopus rankings...

[the_walrus]

You could check the ESF's journal rankings.  In my field, at least, they're fairly rational.