next »

[phalaenopsis]

Dear all,

I agreed to review a manuscript for a nice isi journal two days ago and so received the full manuscript overnight along with the reviewer guidelines. However the problem now is this: I don't quite know how to do it and am afraid I'll make a hopelessly incompetent reviewer who's only able to give generalised comments due to lack of experience and lack of a phd, although the paper is closely my area in which I have 2 publications of my own (my only claim to expertise).

Compounded to that effect is that since manuscripts are supposed to be confidential, so I can't exactly take my comments and the manuscript that isn't mine to my supervisors to have them check it to see if I've been fair enough and thorough in my evaluation.

Now what do I do? The journal gave me a week to do it and it's taken me hours these past couple of days re-reading and looking up each and every reference. I feel compelled to adapt some of the past constructive comments made by previous reviewers on my own papers, but do not know if I can live with the dishonour of doing that because it's still fundamentally plagiarism masked by paraphrasing.

I want to be able to write a review that is honest yet diplomatic and kind. What should I do?

Thank you in advance..

[drscot]

Dear all,

I agreed to review a manuscript for a nice isi journal two days ago and so received the full manuscript overnight along with the reviewer guidelines. However the problem now is this: I don't quite know how to do it and am afraid I'll make a hopelessly incompetent reviewer who's only able to give generalised comments due to lack of experience and lack of a phd, although the paper is closely my area in which I have 2 publications of my own (my only claim to expertise).

Compounded to that effect is that since manuscripts are supposed to be confidential, so I can't exactly take my comments and the manuscript that isn't mine to my supervisors to have them check it to see if I've been fair enough and thorough in my evaluation.

Now what do I do? The journal gave me a week to do it and it's taken me hours these past couple of days re-reading and looking up each and every reference. I feel compelled to adapt some of the past constructive comments made by previous reviewers on my own papers, but do not know if I can live with the dishonour of doing that because it's still fundamentally plagiarism masked by paraphrasing.

I want to be able to write a review that is honest yet diplomatic and kind. What should I do?

Thank you in advance..

This will be very field specific, but I would suggest:

1) Being guided by the structure, style and approach of reviews from experienced colleagues is a good idea, copying comments is not.

2) Your review should be critical but constructive. Good reviews don't have to be "kind", but they do have to offer the author actionable advice - even if that advice relates to how to modify the paper for some other (specified or unspecified) outlet. It's the editor that has requested your help - they need you to help polish "diamonds in the rough", but also to filter out inappropriate submissions.

3) I admire your diligence, but if you have to follow up every reference before you can comment on the paper you are not ready to review. I would bet, though, that this is just a lack of confidence rather than an absence of knowledge in your case. Every reviewer will have to look at the occasional reference that implies something unexpected, but no-one has the time to audit all the sources... You need to develop, use and apply your reading, experience and professional judgement in the field. The good news is: reviewing is a great way to develop these qualities.

4) Just do it, and see what the other reviewers' comments are (in my field it's normal for editors - like me - to copy their decision letters and anonymised reviews to the reviewers). Look at their comments; what do you learn from comparing their observations with yours? If you are off the mark the editor will spot it, and weight their comments towards (one of) the other reviewer(s).

This reply is already too long, but two final caveats. First, how important will this journal / this editor be to you in the future? You could moderate your effort, somewhat, depending on your answer to that. Second - and I know journal submission, revision and rejection cycle can be awful for authors (like me) - it's a review, not a surgical procedure. Doing your best - and learning from the experience - is OK in this context.

I hope some of that helps, and that wiser folk than I also chip in...

DrS

[hegemony]

It is not really that hard.  The only problem is that you haven't seen any models.

Here's a rough template (adapt as appropriate):

---'s study of --- is a ---- study of a question that is central/peripheral to the field of ---.  His focus is --- and his analysis is ---, but before publication I would recommend that the argument be --- and the details of --- and --- be changed to be ----.  Thus I would recommend acceptance with minor changes/revision and resubmission/rejection.

The argument is --- but requires ---. (Expand.)

The details are --- but require ---. (Expand.)

The --- is --- but ----. (Expand.)

Summarize.  Repeat recommendation.

[busyslinky]

When reviewing, I look for things that the paper does that are good and informative (new). Then,  I would look for ways to improve the paper, if it could be improved.  If my list of improvements gets overly long when compared to the strengths, I stop the detailed review and stick with generalities since I will probably recommend rejection. 

I am not there to rewrite the whole paper for the authors.  If it has a good foundation, clear contribution, good (robust) methodology that is clearly defined, interesting results, a good insightful discussion, with a conclusion identifying weaknesses and strengths with identification of future research directions, then it is a good paper.  If there are ways to improve any of these areas, I mention them.

Of course, it needs to read well too (e.g. grammar and English should be very good to excellent).

[science_expat]

Don't be afraid to state that you don't understand things - the for instance, the choice of methodology or the interpretation of the results. Since you've published in the field, if you don't understand it's probably because the authors either i) didn't explain things clearly or ii) have made mistakes. Similarly, I find the use of questions is often helpful.

[busyslinky]

Don't be afraid to state that you don't understand things - the for instance, the choice of methodology or the interpretation of the results. Since you've published in the field, if you don't understand it's probably because the authors either i) didn't explain things clearly or ii) have made mistakes. Similarly, I find the use of questions is often helpful.

Yes, I agree with this.  You should inform the authors if there are ambiguities or gaps in any portions of the manuscript, especially if you are familiar with the field.

[hungry_ghost]

This book http://www.wendybelcher.com/pages/Writing_Your_Journal_Article_in_Twelve_Weeks.htm
has a chapter on "Giving, Getting, and Using others' feedback", and pp. 222-225 are about writing up useful feedback. It is aimed at humanities and social sciences, and at giving feedback to a peer, not blind peer-review, but may still be helpful to you, as it discusses how to focus on and talk about the big picture, which is what you seem to be missing. The book is not available digitally (as far as I can determine) but your library may have a copy.
Good luck!

[phalaenopsis]

wow. you all managed to capture my concerns and gauge what the trouble was exactly even though I didn't even manage to explain what the issue was quite so clearly or succinctly!

it's a review, not a surgical procedure.

drscot: I'm in medicine! Thank you so much for the sound advice. I think the basics of research are the same and hence your suggestions most certaintly applies. Unfotunately though I did manage to search for and read 24 of the 29 reference articles in my initial zeal in truly wanting to do justice to the authors- and you're right in that it boils down more to a lack of confidence.

hegemony: thank you so much for your template, I find the framework most helpful and I believe I will be returning to it again and again in doing future reviews :D

busyslinky & science_expat: actually one of my dilemmas with the manuscript is precisely this; there is a substantial gap in the methodology and I'm wondering if I ought to point it out, but was worried that if by doing so it would be dealing a fatal blow.

hungry ghost: Hmm I've noted the title and pages in my organiser, must check the library on monday!

But really, thank you all so much! I'm still very ignorant and inexperienced, and so am extremely grateful for the guidance and wisdom from all of you who took the trouble to provide the feedback and suggestions :)

[scampster]

busyslinky & science_expat: actually one of my dilemmas with the manuscript is precisely this; there is a substantial gap in the methodology and I'm wondering if I ought to point it out, but was worried that if by doing so it would be dealing a fatal blow.

If you think it is a fatal flaw in the paper, you need to say so. If you think it is a fixable flaw, then you point it out, make a suggestion to remedy it, and give it an "accept with major revisions" if you think everything else is good except for a fixable gap. Remember that you are a gatekeeper. I think, especially with methods, the editor may have asked you to review specifically because you know the methods and no one else might catch the gap.

[sagit]

Good luck, OP.  I remember the first review I did when I was a grad student.  It was very challenging!  Part of the problem was that the article I was given to review was actually really well constructed and well written.  It was hard to find anything to provide critical feedback on but I felt as though that was what I should be doing.  Poorly written or conceptualized articles are so much easier to review.

I'm in social science not medicine so I am not sure what kind of key points people may look for in those articles.  But I think in general you should look for things that are not clear and suggest to the authors that they be more clear in their meaning.  Also ask yourself if you feel that the claims they make are sufficiently backed up by the evidence presented.  If they are, you might make a note of that in your review (you can put positive things in!). 

[anon99]

Here is an article from Nature called Rookie Review.  It is pretty good.  http://www.nature.com/naturejobs/science/articles/10.1038/nj7368-275a

[kron3007]

Just be honest and give your opinions.  If you think the error is a deal breaker say so, if you think it just needs to be discussed as a limitation of the study, say that.  Your role is to provide expert opinion to help the editor make an informed decision on what to do with the paper, not go through line be line and fact check every detail and reference. 

[wanna_writemore]

Here is an article from Nature called Rookie Review.  It is pretty good.  http://www.nature.com/naturejobs/science/articles/10.1038/nj7368-275a

Thanks for this.  I'm in the humanities, but much of the same process applies.

[terpsichore]

busyslinky & science_expat: actually one of my dilemmas with the manuscript is precisely this; there is a substantial gap in the methodology and I'm wondering if I ought to point it out, but was worried that if by doing so it would be dealing a fatal blow.

It's your job as reviewer to point out this flaw. Remember, you're not accepting or rejecting the paper; that's the editor's decision.  The editor can decide whether to give the authors an opportunity to revise their paper to address the flaws found by the reviewers.

Here's how I organize reviews:

- Summarize what I understood to be the topic of the paper and its main contribution (in a short paragraph). 

- Summarize briefly its main strengths and its main weaknesses.

- Provide a list of points the authors should address to improve the manuscript. I typically break this into two categories:  significant issues, and minor points. The significant issues would be the flaw in methodology, major gaps or flaws in the prior work section, sections that are unclear, conclusions that are not supported by the data, etc. Minor points are are things like a missed reference, typos, or wording that obscures the meaning.

I number these lists, to make it easier for the editor and the author to refer to them later.

Finally, I make a summary recommendation (one sentence).

[drscot]

Terpsichore's advice is great and brilliantly concise.

One other point I thought might be worth mentioning, for those who feel bad about offering strong critique, is to offer the author (if you will excuse my crudeness) "sh!t in a sugar sandwich".

What I mean is this kind of approach:

1) start with one nice thing you can honestly say about the paper
2) continue with review content a la Terpsichore
3) conclude with a genuine hope that the advice in (2) will help them improve their paper(s) in the future.

For example -

1) You clearly have a passion for this area of research and have addressed an important topic by...
2) continue with review content a la Terpsichore
3) ... I hope that this advice will help you to develop your thinking in this area and wish you success in the refinement of your work.

It doesn't really change the substance of the review, but it does at least suggest to the author that you recognise that they are a human being...