Thanks, everyone. My panel will read every output - I guess I'm thinking in terms of using journal IFs as a proxy for "quality" when assessing the returns of other people in my Department (which I will have to do), but perhaps it's not that straightforward.
We are doing this. It helps deal with people who have been busy publishing rubbish but claim they are very REF-able. Rather than spend time in endless, useless debates, we simply tell them that if their work is so great, it should have gone into a proper journal and not the International Journal of The Northern Sub-County of Outer Mongolia. Unless, of course, they have other indicators of greatness, e.g. Best Paper in X for 2009.
I'm glad to hear this, but the problem I am having is convincing certain people of the definition of a "proper journal"... They are often genuinely convinced that a journal with an IF of (say) 0.45 is the leading journal in their area. As I don't know enough about their particular field, I can't construct a strong argument against this, but my instinct is that such a journal cannot claim to be leading, regardless of the size of the field, with such a relatively weak IF. I worry about the navel-gazing effect, whereby a single journal may well be the first port of call in a particular sub-field, but if papers it publishes are generally ignored, then it cannot be "leading" in a more general sense.
If it is your job to make the call, then just do so; being "timid" or "sensitive" will only make your job harder. Impact factors are a good start. New journals might be problematic, but a rough guide would be to start by considering the reputation of the publisher, the editorial board, the quality of papers published so far, etc. If in doubt, then you might consider asking the opinions of externals in that particular sub-field. As for the authors, ask them to make a case such as they would make for a promotion, i.e. no "proof by assertion".
Nobody will cheerfully admit that they have been publishing crap, in dodgy journals. So, inevitably, there will be some wailing and gnashing of teeth. The owners of said teeth, fully knowing the reality, even if they will not confess it to you, will quickly get over it once a clear line is drawn.
