Reviewing a scientific paper

For scientists, the review process is sometimes a private matter: everyone does it, yet we don’t talk about it much. This may be related to the confidentiality and anonymity principles at the core of this process, which sound a bit like Swiss banking: reviewers should not tell others about the work until it is published, and authors are not told who their reviewers are (in double-blind reviewers are not told who the authors are either). There are reasons behind these principles, but it doesn't mean we can't talk about reviews altogether. So here is some advice and pointers on things I learned along the way.

What is the review process

The goal of a review is to assess the contribution of a submitted manuscript. A meaningful contribution can come from novelty, new ideas that have not been already proposed, but it may also come from fresh observations, e.g. an extensive benchmark that sheds new light on how existing methods perform. Anyway, peers of the authors are asked for their evaluation, and on the way give their opinion and other comments to help improve the manuscript. For authors and reviewers alike, the review process is a way of interacting between peers and exchanging constructive feedback.

Surviving the review process

There is a great overview by Seth Hutchinson which I warmly advise you to read for starters: Surviving the review process.

What are the qualities of a good review?

A good review is constructive in its criticism: it tells the authors how the work can be improved, and does not diminish their efforts (regardless of whether the review vouches for acceptance or rejection). To make sure you achieve this, there is a simple strategy: try to be as nice as possible; remember you are talking to one of your peers. When pointing out that something is wrong, make sure you (1) explain why it is wrong, and (2) point out how it can be improved, whenever possible.

A simple review method

Reading someone else’s work is an effort. It takes hours, especially when you are doing it for the first time. Your own curiosity in the topic is one way to balance it. Another way is to make your reading “interactive” as follows:

  • Read the paper linearly, taking notes in the margin or in a file as you go.
  • Complete your notes following e.g. the template below.
  • Double-check by reading the introduction or conclusion of the manuscript again.
  • Go to the review website and polish your review as you submit it.

For me, notes usually start as a bullet list of oral comments: what I would like to say to the authors in response to specific parts of the manuscript. As I keep on reading, I usually shuffle this list, putting most important comments first and minor ones at the end. This is the “low level” part of the review. The “high level” part of the review (general comments and assessment of the manuscript) usually comes as a set of paragraphs that open your review, before going into detailed (bullet-list like) comments. You can use the following structure.

1. Summary

Start with a quick summary, using your own words (not the authors’), briefly describing the steps taken in the paper. The summary can simply list out what the paper does without commenting (yet). This step is also useful for the editor who rely on these alternative abstracts to piece together their understanding of the paper.

2. Contribution

Now is the time for constructive criticism: single out what’s new in the manuscript compared to existing works, and discuss it. Sometimes, the contribution you identify will differ from the one the authors state, for example because they missed some prior art. In that case, a "We introduce a novel way of doing X" becomes "This work proposes a variant of [1, 2] where the foo term in the blah process is modified in Y, Z ways."

Here is for example the list of questions that the IEEE Robotics and Automation Society suggests you evaluate at this stage of your review:

  • Is the contribution of the paper relevant to its field?
  • Does it properly cite related existing works?
  • Does it provide all technical material needed to understand and reproduce the contribution?
  • How are its experimental results?
  • Is the writing clear and comprehensible?

4. Strengths and weaknesses

To wrap up the first paragraphs of your review, list both the strengths and weaknesses of the manuscript as a whole. Systematically discussing both pros and cons is usually a good strategy: if you liked the paper, it reminds you to try to look for its downsides, and conversely.

5. Numbered list of comments

Now that the first overview paragraphs are written down, we can jump into less structured and more specific comments on the work. This part is a more direct address to the authors, as the area chairs or editors are more likely to focus on the overview.

If the venue's review process allows for a rebuttal phase (which, if it is a good venue, it does), number all your questions and comments. This bit of structure will facilitate the back and forth in the ensuing discussions.

Good practices

Separate facts and judgements

Strive to separate facts from judgements clearly, in your reviews, and in general ;) If there is a calculation mistake, or a missing technical detail, a claim of novelty for something that was already proposed, etc., that's a fact and you can state it bluntly. If on the other hand you don't see the benefit of the approach, or you disagree with the authors on some foreseen consequences, etc., that's a judgement (a fact about your beliefs, not a fact about the work) and it's best to own it by writing it down as "I/this reviewer believes..." or "I/this reviewer disagrees with the authors regarding...".

Judgements in reviews are totally fine. They are not the ground upon which a work should be evaluated, yet they are part of the scientific conversation between peers (and can be useful feedback for the authors). Don't discard your opinions, especially if you are early in your career and unsure about them: your input is valuable, that's why you were asked for a review! Keep the tone friendly and constructive, and also keep in mind that some things like novelty are delicate to judge because of hindsight bias.

Language and typos

It is common practice to group minor suggestions and typos in a short list at the end of your review. These points should not count in the overall evaluation of the work, unless the language is so problematic that it harms understanding.

Q & A

What is the point of the anonymity principle?

It tries to remove bias. Of course it is not a panacea so don’t expect it to eliminate all kinds of biases encountered in the scientific community, but think of the following situation: a PhD student reviews a technically flawed manuscript from the lab of a famous professor in their field (one they may e.g. like to join for post-doc…). Without anonymity, the student faces two conflicting objectives: writing an honest review, and avoiding potential harm to their future career. Anonymity helps unbias such situations.

What about a symmetric situation: a famous professor reviews a paper on a topic someone in their lab is currently working on?

This is one of the conflicting situations that is not solved by rules of the current system. (There are not rules for everything.) It then becomes a question of ethics. You will observe both ethical and unethical behaviors from people in your community. Do your best to reward and interact more with ethical people, while avoiding interactions with unethical ones.

I'm reviewing a paper that's not 100% my expertise, how deep should I dive?

Associate editors/area chairs pick reviewers not only for the quality of the feedback they expect, but also to be representative of segments of the community. For instance, younger reviewers tend to provide more detailed and technical reviews, while older reviewers tend to connect to more related works. All of those are meaningful feedback both to the authors (to improve their paper) and editors (to evaluate the work), thus editors tend to sample from their pool of reviewers to get a group as representative as possible of their community. Here is how the IEEE Transactions on Robotics puts it in their Information for Associate Editors:

A good mix of senior and junior Reviewers is desirable as they provide reviews from different perspectives and at different levels of detail. For example, a junior Reviewer may carefully check the various technical details of the paper, but may also lack the depth to assess the significance of the contribution, while a senior Reviewer may be in a position to judge whether the paper brings a real contribution to the community. A mix of Reviewers from different geographical regions is also desirable.

If you are not 100% expert in the paper an editor asked you to review, it is likely the editor already anticipated this. It is not necessary to dive deep into every aspect of the paper you are reading (for example, searching related works for missing details), unless of course you get curious to do so. Your review, using the expertise you had at the time you accepted it, is still valuable work that you provide to your peers. (For free: reviewing is a service we give each other.)

To go further

I like the personal method Nato Lambert recounts in How to review a paper. In particular this bit on the overall attitude rings right:

You’ll give a good review if you read the paper as if you would a close friend or colleague who asked for feedback. There is no reason to put on a tough face for a review because it happens to be in a conference. Also, you should and can be supportive of what authors have done well in a paper review.

The idea of having an open ended section for discussion that is not part of evaluation is also nice, and it would fit well in the review method above.

Here are some other relevant reads on this topic:

You can also check out the Solicited Reviews section of this page for examples of reviews written by senior researchers.

Discussion

Feel free to post a comment by e-mail using the form below. Your e-mail address will not be disclosed.

📝 You can use Markdown with $\LaTeX$ formulas in your comment.

By clicking the button below, you agree to the publication of your comment on this page.

Opens your e-mail client.