Reviewing a scientific paper

For scientists, the review process is personal, almost intimate: everyone does it (maybe first as tit for that when submitting your first papers, then for other reasons discussed thereafter), but researchers don't talk about it so much. One probable cause for this is the anonymity principle that is at the core of the review process: authors are not told who their reviewers are, and in double-blind submissions reviewers are not told either who the authors are.

What is the review process

The goal of a review is to assess the contribution of a submitted manuscript, i.e. what's new compared to existing works. Peers of the authors are asked for their evaluation, and on the way give their opinion and other comments to help improve the manuscript. For authors and reviewers alike, the review process is a way of interacting between peers.

Surviving the review process

There is an excellent and short piece of writing by Seth Hutchinson, Surviving the review process, that lifts the curtain on a number of important points about the review process. If you haven't read it yet, you can forget about what's written below and click this link immediately ;)

What are the qualities of a good review?

A good review is constructive in its criticism, and does not diminish the authors' efforts (even when the review has to be negative, or sometimes, very negative). To make sure you achieve this, there is a simple strategy: try to be as nice as possible; remember you are talking to one of your peers. When pointing out that something is wrong, make sure you (1) explain why it is wrong, and (2) point out how it can be improved, whenever possible.

A simple review method

Reading someone else's work is an effort. Your own curiosity in the topic is one way to balance it. Another way is to make your reading "interactive" as follows:

  • Read the paper linearly, taking notes in a text file as you go.
  • Complete your notes following e.g. the template below.
  • Double-check by reading the introduction or conclusion of the manuscript again.
  • Go to the review website and polish your review as you submit it.

For me, notes usually start as a bullet list of oral comments: what I would like to say to the authors in response to specific parts of the manuscript. As I keep on reading, I usually shuffle this list, putting most important comments first and minor ones at the end. This is the "low level" part of the review. The "high level" part of the review (general comments and assessment of the manuscript) usually comes as a set of paragraphs that open your review, before going into detailed (bullet-list like) comments. You can use the following structure.

Summary

Start with a quick summary, using your own words (not the authors'), briefly describing the steps taken in the paper. The summary can simply list out what the paper does without going into constructive criticism (yet). This step is also useful for the editor (and authors when there is a rebuttal phase) to check how you understood the paper.

Contribution

Now is the time for constructive criticism: single out what's new in the manuscript compared to existing works, and discuss it. Is it meaningful? Technically sound? Also, if the authors' contribution statement is incorrect (e.g. they listed as contribution something that was already published), it is the time to gently point them to relevant references.

Here is for example the list of questions that the IEEE Robotics and Automation Society suggests you evaluate at this stage of your review:

  • Is the contribution of the paper relevant to its field?
  • Does it properly cite related existing works?
  • Does it provide all technical material needed to understand and reproduce the contribution?
  • How are its experimental results?
  • Is the writing clear and comprehensible?

It is also common practice to group minor suggestions and typos in a short list at the end of your review.

Strengths and weaknesses

Discussing both pros and cons of the submission is usually a good strategy: if you liked the paper, it reminds you to try to look for its downsides, and conversely. Include suggestions for improvement as appropriate. Be thorough, fair, and constructive.

Final checks

Finally, if you consider that the paper can be accepted for publication, you can give a quick check to the authors' webpage to verify that the work has not been submitted/published elsewhere. (This can happen: when it does, the authors should mention it explicitly in their manuscript, otherwise this is a deontological red flag.)

Q & A

What is the point of the anonymity principle?

It tries to remove bias. Of course it is not a panacea so don't expect it to eliminate all kinds of biases encountered in the scientific community, but think of the following situation: a PhD student reviews a technically flawed manuscript from the lab of a very famous professor in his field (one he may e.g. like to join for post-doc...). Without anonymity, the student faces two conflicting objectives: writing a truthful review, and avoiding potential harm to his/her future career. Anonymity helps avoid such situations.

What about an opposite situation: a famous professor reviews a paper on a topic someone in his/her lab is currently working on?

This is one of the conflicting situations that is not solved by rules of the current system. (Be careful about people who want to put rules everywhere.) It therefore becomes a question of ethics. You will encounter both ethical and unethical tempers among people in your community. Rewarding and interacting with ethical people while avoiding interactions with unethical ones is a simple strategy that everyone can apply at one's own level.

To go further

Here are a couple of references on this topic:

You can also check out the Solicited Reviews section of this page for examples of reviews written by senior researchers.

Pages of this website are under the CC-BY 4.0 license.