next »

[cogdoc]

My non-Australian university has proposed to use the ERA 2010 journal rankings as a means to assess publication quality (only A* journals count).  These rankings are new to me. 

What is the impression of them for Australian-based folks?

It seems they are no longer being used by the ERA (2012 at least), which makes it appear there is something problematic about the rankings!

Thanks.

[totoro]

They were abolished because they were so controversial and they decided they couldn't be fixed. How accurate the rankings were depended a lot on discipline. In economics for example they were mostly good but there was still quite a lot of noise - for example History of Political Economy - a journal with almost no citations was ranked A*. But in geography for example the rankings made no sense at all. The other major issue was how to rank Australian journals. My perception was that they tended to be ranked higher than they deserved but the feeling among most people here was the reverse and that was the main driver in sinking the rankings. The problem was people were stopping sending papers to Australian journals that got low rankings (as if we didn't know they weren't that good before). I think they should have had two separate indicators - journal quality, and contribution to Australian policy issues - to deal with this.

In practice we are still using the ranking. For example, I am on a panel compiling the submission in my discipline from my university for ERA 2012 and one of the key things we are looking at is the ranks of the journals in the ERA 2010 list....

[cogdoc]

They were abolished because they were so controversial and they decided they couldn't be fixed. How accurate the rankings were depended a lot on discipline. In economics for example they were mostly good but there was still quite a lot of noise - for example History of Political Economy - a journal with almost no citations was ranked A*.

That sums it up nicely--thanks! I was very surprised some journals in my field with very high impact factors were ranked lower than lower impact specialized journals, and that the percentage of journals ranked A* amounted to less than 5% of the total in the field.

[highway61]

The whole thing was so so so ridiculous that it had to be abandoned. They began with the premise that 50% of journals are worse than horse manure. 50%! They issued a draft ranking according to that assumption--no explanation of how it came about. Then asked for feedback. As I said to the poor public servant who dealt with complaints, "what if the Federal Government announced that they knew half of all public servants were useless and awful, ranked you, and then asked you to comment?" That's what they did to academics.

My sense is kind of the opposite of totoro's re Aussie journals. First round I think it's true that local journals didn't rank as highly. But in my field, scandalously, the local journal (which is perfectly respectable, but really where we tell grad students to submit) jumped from C to A (!!) after lobbying, whereas one of the very top journals from the UK went from A to B as if to make room. That did away with any possible respect I could have had for the project.

I also asked the public servant: so, after you determine that half of the journals are crap, and lots of them shut up shop because you have destroyed their reputations, does that mean that some Bs have to become Cs and so forth? The obvious answer would be "yes," leading to a continual cycle of attrition of Cs and filling redistribution of rankings, always downwards. But of course no one thought through that or anything else. The Minister of Education, in announcing the dropping of rankings, blamed the universities for "misusing" the rankings--ie, using them in things like promotion etc. (How on earth else are they to be used?). The whole episode was really sorry, and still is, given that totoro is no doubt right that the rankings from 2010 will still be secretly used.

[totoro]

I think it's your interpretation that being ranked C is "worse than horse manure". My interpretation is that it is all about how hard it is to get published in these journals, how selective they are. I'm sure there is lots of good research in C journals. If you have a ranking system someone has to be bottom. What I don't understand is why this was news to anyone. We didn't need the ARC to tell us how selective and prestigious journals are. You can just use ISI's (Web of Science) impact factors, and if it's not in the ISI index then it is probably a C journal. So the outrage is hard to understand...

I agree that lobbying pushed up the ranks of Australian journals in many cases. I am working on my university's submission for ERA 2012 in my field at the moment. One of the journals I was just looking at is not even in Scopus but it was ranked B in ERA 2010.

We are using ERA 2010, the number of citations in Scopus and Google Scholar and other journal rankings in working out what should be peer-reviewed etc.

[highway61]

Perhaps it's my interpretation, but it's my interpretation of this (from http://www.arc.gov.au/era/tiers_ranking.htm):

"Generally, in a Tier B journal, one would expect only a few papers of very high quality. They are often important outlets for the work of PhD students and early career researchers.  Typical examples would be regional journals with high acceptance rates, and editorial boards that have few leading researchers from top international institutions. Tier C includes quality, peer reviewed, journals that do not meet the criteria of the higher tiers."

(And, I'm in the humanities, where "impact" and ISI are meaningless.)

[totoro]

"Generally, in a Tier B journal, one would expect only a few papers of very high quality. They are often important outlets for the work of PhD students and early career researchers.  Typical examples would be regional journals with high acceptance rates, and editorial boards that have few leading researchers from top international institutions. Tier C includes quality, peer reviewed, journals that do not meet the criteria of the higher tiers."

(And, I'm in the humanities, where "impact" and ISI are meaningless.)

Yes, I saw that. The statement about PhD students and ERAs is silly. I'm in economics. There were some serious mistakes as I mentioned upthread. But generally the ranking made sense. Maybe it didn't make sense to rank journals in some humanities fields. In econ people were upset that this would stop people from working on Australian policy questions as no Australian journals were ranked A*. But we already knew that and getting into an A* journals is hard - most Australian economists will have very few A* publications. So this outrage didn't make any sense to me. But maybe a lot of people had never thought about journal ranking before if they got Aus. PhD's and never competed in the international job market.

[highway61]

Oh, I see--we're talking at different ends of the scale. My outrage (and that of every humanities person) was directed at the C rankings: I can see that 10-15% of journals might fit that description--but half? I read the great majority of journals in my field and very few are even Bs. Those below just don't last. The percentages were just way way off. As for A*, I do agree with you that they are, and should be, very hard to get into.

[sockdolager]

I think it's your interpretation that being ranked C is "worse than horse manure".

One of the problems with the ERA 2010 rankings, as I understand it, is that many people - including university admins - took this view, which resulted in some researchers being told that it wasn't worth publishing in a C journal (or sometimes in anything below A).

OP, it's true there aren't many A* journals, but at the universities that I'm familiar with it was accepted that A and A* both represent a suitably impressive journal.

[expatphd]

OP, it's true there aren't many A* journals, but at the universities that I'm familiar with it was accepted that A and A* both represent a suitably impressive journal.

I am glad there are sensible universities out there -- however mine just seemed to find it convenient to use the A* list only as a means for restructuring, and thus removing, staff even though previous targets based on impact factor included many journals ranked A, and excluded some A* journals. Not to mention the fact these are now outdated, particularly in terms of impact factor considerations.  I have now found a few news articles noting the controversy surrounding these rankings.

Thanks for all of the information.

[mingus]

We use them, to find who's publishing in crappy journals and who's publishing in proper journals.  Their value is independent of the manner in which they were used by management in some  Australian universities.   Misuse does not change the nature of the journals or the rankings.

[highway61]

Right, their value is independent of their misuse by administrators. But ... their value is extraordinarily limited. "Crappy"--are half the journals published (Half!!) really crappy? In all disciplines?

Really?

No, of course not. So mingus on what grounds do you say a journal is "crappy"? In my field, one journal that got a C first time around then went to A. Was it crappy then but not later? Other journals, with the word "philology" in the title, were C though they are extraordinarily prestigious. The only possible logic was that linguists didn't know them--thus, must be crappy. Problem is, they're not linguistic journals. (I think they might have risen to B in the next round.) And on and on. So, again, on what grounds can you say any journal is "crappy" on the basis of the ERA rankings?

Yes, I agree there are crappy journals. As I said before, I'd put the percentage at roughly 10-15% of journals in my field. At the least I'd want to see the evidence behind the assumption that 50% of journals are worse than manure before I ever used these rankings for anything. Any use is misuse.

[totoro]

I just now have picked the articles going forward for peer review in one of the FOR codes for ERA 2012. I used the 2010 ERA as a guideline. In this discipline it's not bad. In others (e.g. human geography) it's nonsense. And maybe in the humanities it makes no sense whatsoever. And I nominated some articles in C journals if they had decent citations. It's really a mistake to say that research in C journals is "worse than horse manure". What is true is that usually it's not hard to get published in them. But I did nominate all A* articles. All book chapters in Cambridge, Oxford, Chicago etc books. most but not all A articles and some B's with high citations. I've also been tasked with designing a financial incentive scheme for our school.

[helpful]

Do these rankings even take into account whether people read the articles after they are published?

I am so glad in my discipline there is no such ranking system. It sounds so so absurd.

In most publishing, the main criteria is whether the stuff is being read by the people who need to read them. For example, if I were in a policy area (I am not), wouldn't it be more important that decision-makers read the articles on policy relevant to their area? That would add another criteria to a ranking system that woud be more appropriate to a policy field.

[totoro]

ERA 2010 had a ranking for all disciplines... the ranks were decided by the "Learned Academies" here in Australia. Mainly they are based on impact factor (citation rate), acceptance rate, and prestige of the editorial board. The ARC has abolished these ranks but we are still using them in practice.

In psychology and natural sciences the main assessment instrument is citation analysis. In other social sciences and humanities it is peer review of a sample of publications. The journal rankings (now profile) is also looked at in every field. Conferences were ranked too in some fields, but not presses strangely.