For scientists, the review process is sometimes a private matter: everyone does it, yet we don’t talk about it much. This may be related to the confidentiality and anonymity principles at the core of this process, which sound a bit like Swiss banking: reviewers should not tell others about the work until it is published, and authors are not told who their reviewers are (in double-blind reviewers are not told who the authors are either). There are reasons behind these principles, but it doesn't mean we can't talk about reviews altogether. So here is some advice and pointers on things I learned along the way.
What is the review process¶
The goal of a review is to assess the contribution of a submitted manuscript. A meaningful contribution can come from novelty, new ideas that have not been already proposed, but it may also come from fresh observations, e.g. an extensive benchmark that sheds new light on how existing methods perform. Anyway, peers of the authors are asked for their evaluation, and on the way give their opinion and other comments to help improve the manuscript. For authors and reviewers alike, the review process is a way of interacting between peers and exchanging constructive feedback.
Surviving the review process¶
There is a great overview by Seth Hutchinson which I warmly advise you to read for starters: Surviving the review process.
What are the qualities of a good review?¶
A good review is constructive in its criticism: it tells the authors how the work can be improved, and does not diminish their efforts (regardless of whether the review vouches for acceptance or rejection). To make sure you achieve this, there is a simple strategy: try to be as nice as possible; remember you are talking to one of your peers. When pointing out that something is wrong, make sure you (1) explain why it is wrong, and (2) point out how it can be improved, whenever possible.
A simple review method¶
Reading someone else’s work is an effort. It takes hours, especially when you are doing it for the first time. Your own curiosity in the topic is one way to balance it. Another way is to make your reading “interactive” as follows:
- Read the paper linearly, taking notes in the margin or in a file as you go.
- Complete your notes following e.g. the template below.
- Double-check by reading the introduction or conclusion of the manuscript again.
- Go to the review website and polish your review as you submit it.
For me, notes usually start as a bullet list of oral comments: what I would like to say to the authors in response to specific parts of the manuscript. As I keep on reading, I usually shuffle this list, putting most important comments first and minor ones at the end. This is the “low level” part of the review. The “high level” part of the review (general comments and assessment of the manuscript) usually comes as a set of paragraphs that open your review, before going into detailed (bullet-list like) comments. You can use the following structure.
Start with a quick summary, using your own words (not the authors’), briefly describing the steps taken in the paper. The summary can simply list out what the paper does without commenting (yet). This step is also useful for the editor who rely on these alternative abstracts to piece together their understanding of the paper.
Now is the time for constructive criticism: single out what’s new in the manuscript compared to existing works, and discuss it. Sometimes, the contribution you identify will differ from the one the authors state, for example because they missed some prior art. In that case, a "We introduce a novel way of doing X" becomes "This work proposes a variant of [1, 2] where the foo term in the blah process is modified in Y, Z ways."
Here is for example the list of questions that the IEEE Robotics and Automation Society suggests you evaluate at this stage of your review:
- Is the contribution of the paper relevant to its field?
- Does it properly cite related existing works?
- Does it provide all technical material needed to understand and reproduce the contribution?
- How are its experimental results?
- Is the writing clear and comprehensible?
3. Link to prior works¶
Next, discuss how the paper positions its contribution with respect to existing works. A delicate situation that happens sooner or later is reviewing a manuscript related to one's previous works yet that does not mention them. At this stage we can feel biased towards self-promotion or irate at the authors for ignoring our works! Keep in mind Hanlon's razor though: there are many not-so-unlikely ways in which your peers may have missed your works, especially in fast-moving domains like robotics or machine learning where the literature evolves so fast. One technique that can be useful in this situation is to treat your own works as if they had been done by others (which also helps keep the review anonymous), and make points at the conceptual level only.
- Don't prompt for a citation without justification, e.g. "at this point it would be nice to cite [vaguely related works X]".
- Do explain conceptual decisions between papers, for instance "this manuscript chooses to do X while [closely related work Y] started the same but did Z instead".
Avoiding the laundry list syndrom is part of the authors' writing duty, so as a reviewer it is only logical we are held to the same standard when pointing out missing citations to existing works.
4. Strengths and weaknesses¶
To wrap up the first paragraphs of your review, list both the strengths and weaknesses of the manuscript as a whole. Systematically discussing both pros and cons is usually a good strategy: if you liked the paper, it reminds you to try to look for its downsides, and conversely.
5. Numbered list of comments¶
Now that the first overview paragraphs are written down, we can jump into less structured and more specific comments on the work. This part is a more direct address to the authors, as the area chairs or editors are more likely to focus on the overview.
If the venue's review process allows for a rebuttal phase (which, if it is a good venue, it does), number all your questions and comments. This bit of structure will facilitate the back and forth in the ensuing discussions.
Separate facts and judgements¶
Strive to separate facts from judgements clearly, in your reviews, and in general ;) If there is a calculation mistake, or a missing technical detail, a claim of novelty for something that was already proposed, etc., that's a fact and you can state it bluntly. If on the other hand you don't see the benefit of the approach, or you disagree with the authors on some foreseen consequences, etc., that's a judgement (a fact about your beliefs, not a fact about the work) and it's best to own it by writing it down as "I/this reviewer believes..." or "I/this reviewer disagrees with the authors regarding...".
Judgements in reviews are totally fine. They are not the ground upon which a work should be evaluated, yet they are part of the scientific conversation between peers (and can be useful feedback for the authors). Don't discard your opinions, especially if you are early in your career and unsure about them: your input is valuable, that's why you were asked for a review! Keep the tone friendly and constructive, and also keep in mind that some things like novelty are delicate to judge because of hindsight bias.
Language and typos¶
It is common practice to group minor suggestions and typos in a short list at the end of your review. These points should not count in the overall evaluation of the work, unless the language is so problematic that it harms understanding.
Q & A¶
What is the point of the anonymity principle?¶
It tries to remove bias. Of course it is not a panacea so don’t expect it to eliminate all kinds of biases encountered in the scientific community, but think of the following situation: a PhD student reviews a technically flawed manuscript from the lab of a famous professor in their field (one they may e.g. like to join for post-doc…). Without anonymity, the student faces two conflicting objectives: writing an honest review, and avoiding potential harm to their future career. Anonymity helps unbias such situations.
What about a symmetric situation: a famous professor reviews a paper on a topic someone in their lab is currently working on?¶
This is one of the conflicting situations that is not solved by rules of the current system. (There are not rules for everything.) It then becomes a question of ethics. You will encounter both ethical and unethical tempers among people in your community. Do your best to reward and interact more with ethical people, while avoiding interactions with unethical ones.
To go further¶
I like the personal method Nato Lambert recounts in How to review a paper. In particular this bit on the overall attitude rings right:
You’ll give a good review if you read the paper as if you would a close friend or colleague who asked for feedback. There is no reason to put on a tough face for a review because it happens to be in a conference. Also, you should and can be supportive of what authors have done well in a paper review.
The idea of having an open ended section for discussion that is not part of evaluation is also nice, and it would fit well in the review method above.
Here are some other relevant reads on this topic:
- Surviving the review process by Seth Hutchinson.
- A Quick Guide to Writing a Solid Peer Review, proposing a flow chart of the process.
- Pitfalls to mind when judging the novelty of a paper under review by Michael Black.
- Information for Reviewers from the IEEE Transactions on Robotics.
You can also check out the Solicited Reviews section of this page for examples of reviews written by senior researchers.
There are no comments yet. Feel free to leave a reply using the form below.