Reviewing a scientific paper

1. Summary¶

Start with a quick summary, using your own words (not the authors’), briefly describing the steps taken in the paper. The summary can simply list out what the paper does without commenting (yet). This step is also useful for the editors who rely on these alternative abstracts to piece together their understanding of the paper.

2. Contribution¶

Now is the time for constructive criticism: single out what’s new in the manuscript compared to existing works, and discuss it. Sometimes, the contribution you identify will differ from the one the authors state, for example because they missed some prior art. In that case, a "We introduce a novel way of doing X" becomes "This work proposes a variant of [1, 2] where the foo term in the blah process is modified in Y, Z ways."

Here is for example the list of questions that the IEEE Robotics and Automation Society suggests you evaluate at this stage of your review:

• Is the contribution of the paper relevant to its field?
• Does it properly cite related existing works?
• Does it provide all technical material needed to understand and reproduce the contribution?
• How are its experimental results?
• Is the writing clear and comprehensible?

When discussing novelty, be specific. If you believe a contribution has already been proposed, invest the time to find the actual references rather than writing "the method is not novel." This is not actionable for the authors, and if the method is truly as widespread as you think, finding a couple of references should not take long.

3. Link to prior works¶

While the previous point focuses on what is new, this one is about how the paper situates itself among existing works. A delicate situation that happens sooner or later is reviewing a manuscript related to one's previous works yet that does not mention them. At this stage we can feel biased towards self-promotion or irate at the authors for ignoring our works! Keep in mind Hanlon's razor though: there are many not-so-unlikely ways in which your peers may have missed your works, especially in fast-moving domains like robotics or machine learning where the literature evolves so fast. One technique that can be useful in this situation is to treat your own works as if they had been done by others (which also helps keep the review anonymous), and make points at the conceptual level only.

• Don't prompt for a citation without justification, e.g. "at this point it would be nice to cite [vaguely related works X]".
• Do explain conceptual decisions between papers, for instance "this manuscript chooses to do X while [closely related work Y] started the same but did Z instead".

Avoiding the laundry list syndrom is part of the authors' writing duty, so as a reviewer it is only logical we are held to the same standard when pointing out missing citations to existing works.

4. Strengths and weaknesses¶

To wrap up the first paragraphs of your review, list both the strengths and weaknesses of the manuscript as a whole. Systematically discussing both pros and cons is usually a good strategy: if you liked the paper, it reminds you to try to look for its downsides, and conversely. Make sure you don't skip the strengths in the process: spelling out what the paper does well signals to the authors what to preserve, and helps you calibrate your own assessment.

5. Numbered list of comments¶

Now that the first overview paragraphs are written down, we can jump into less structured and more specific comments on the work. This part is a more direct address to the authors, as the area chairs or editors are more likely to focus on the overview. Take individual questions and comments you had on specific parts of the manuscript, and list them sequentially as a numbered list.

Numbering the list is important here. If the venue's review process allows for author responses, such as a journal or a conference with a rebuttal phase, this bit of structure will facilitate the back and forth in the ensuing discussions. Rebuttals are typically short (e.g. one page at conferences), therefore, precise and numbered questions are easier for authors to address one by one.

Separate facts and judgements¶

Strive to separate facts from judgements clearly, in your reviews, and in your work in general 😉 If there is a calculation mistake, or a missing technical detail, a claim of novelty for something that was already proposed, etc., that's a fact and you can state it bluntly. If on the other hand you don't see the benefit of the approach, or you disagree with the authors on some foreseen consequences, etc., that's a judgement (a fact about your beliefs, not a fact about the work) and it's best to own it by writing it down as "I/this reviewer believes..." or "I/this reviewer disagrees with the authors regarding...".

Judgements in reviews are totally fine. They are not the ground upon which a work should be evaluated, yet they are part of the scientific conversation between peers (and can be useful feedback for the authors). Don't discard your opinions, especially if you are early in your career and unsure about them: your input is valuable, that's why you were asked for a review! Keep the tone friendly and constructive, and also keep in mind that some things like novelty are delicate to judge because of hindsight bias.

Be specific and constructive¶

While writing you may notice that you start by stating a negative comment. This is fine as a starting point (you don't want to get stuck with analysis paralysis, especially when you begin drafting your review), but the next stage is to promote it into a constructive criticism. If you cannot turn a negative comment into something actionable, like pinpointing a step that is is wrong, a statement that is contradicted by other work from the literature, etc., then the comment may be too broad or unsubstantiated. Being specific almost automatically makes criticism constructive, because it shows the authors exactly what to fix.

In practice, this often means rephrasing. For instance:

• "The evaluation is obviously lacking" → "I think the evaluation would benefit from experiments in settings X and Y, which would help support the claims made in Section Z."
• "This has been done before" → "The approach is related to [1, 2], and the paper would benefit from a discussion of what differentiates it from these works."
• "Section 4 is incomprehensible" → "I had difficulty following the derivation in Section 4, in particular the step from Eq. 3 to Eq. 4. My questions are: Why is ...? A more detailed explanation would help."

Also watch out for adjectives that convey impression without information, such as "clearly", "obviously", "very", "highly", etc. This is especially true in in negative comments. Such adjectives add nothing to your argument, but they can make it look dismissive. Just as we avoid such adjectives when describing results in our own papers, it is better to avoid them in reviews.

Sleep on it, then re-read as an author¶

Our thinking doesn't stop when we step away from a task. After we switch to other activities, and especially after we sleep, part of our brain keeps processing what we worked on during the day. Letting this happen before you finalize your review can noticeably improve its quality. Some scientists make it a point to always sleep at least one night between drafting a review and submitting it.

When you do come back to your review, double-check it by re-reading the introduction or conclusion of the manuscript, to make sure your high-level assessment is consistent with what the authors actually claim. Then re-read your review itself from the authors' perspective: how would you feel receiving this for one of your own papers? This empathy check catches most issues with tone, particularly when the overall recommendation is a rejection. The goal is not to reassess the recommendation itself, but rather to ensure the review is fair and useful on the receiving end.

Language and typos¶

It is good practice to group minor suggestions and typos in a short list at the end of your review. These points should not count in the overall evaluation of the work, unless the language is so problematic that it harms understanding.

What is the point of the anonymity principle?¶

It tries to remove bias. Of course it is not a panacea so don’t expect it to eliminate all kinds of biases encountered in the scientific community, but think of the following situation: a PhD student reviews a technically flawed manuscript from the lab of a famous professor in their field (one they may e.g. like to join for post-doc…). Without anonymity, the student faces two conflicting objectives: writing an honest review, and avoiding potential harm to their future career. Anonymity helps unbias such situations.

What about a symmetric situation: a famous professor reviews a paper on a topic someone in their lab is currently working on?¶

This is one of the conflicting situations that is not solved by rules of the current system. (There are no rules for everything.) It then becomes a question of ethics. You will observe both ethical and unethical behaviors from people in your community. Do your best to reward and interact more with ethical people, while avoiding interactions with unethical ones.

I'm reviewing a paper that's not 100% my expertise, how deep should I dive?¶

Associate editors/area chairs pick reviewers not only for the quality of the feedback they expect, but also to be representative of segments of the community. For instance, younger reviewers tend to provide more detailed and technical reviews, while older reviewers tend to connect to more related works. All of those are meaningful feedback both to the authors (to improve their paper) and editors (to evaluate the work), thus editors tend to sample from their pool of reviewers to get a group as representative as possible of their community. Here is how the IEEE Transactions on Robotics puts it in their Information for Associate Editors:

A good mix of senior and junior Reviewers is desirable as they provide reviews from different perspectives and at different levels of detail. For example, a junior Reviewer may carefully check the various technical details of the paper, but may also lack the depth to assess the significance of the contribution, while a senior Reviewer may be in a position to judge whether the paper brings a real contribution to the community. A mix of Reviewers from different geographical regions is also desirable.

If you are not 100% expert in the paper an editor asked you to review, it is likely the editor already anticipated this. It is not necessary to dive deep into every aspect of the paper you are reading (for example, searching related works for missing details), unless of course you get curious to do so. Your review, using the expertise you had at the time you accepted it, is still valuable work that you provide to your peers. (For free: reviewing is a service we give each other.)

I can't follow a section of the paper, what should I do?¶

Tell the authors where the manuscript lost you, and write down your best guess for what the section is trying to convey. This will help you clarify your thoughts at this stage, while also giving the authors concrete feedback on where their explanation breaks down. Not being able to follow a section is valid feedback in itself: if a peer in the field cannot follow, the writing likely needs improvement.

Can I use an LLM to help write my review?¶

The conference or journal you are reviewing for may have specific guidelines on the use of LLM-based tools to help with intellectual work. A general principle is that you are responsible for everything in your review: your claims and judgements about the submission, the references you provide, etc. LLM-based tools can help with organizing thoughts, analyzing and summarizing references, or just checking phrasing, but they can also hallucinate references, produce generic criticism, or miss the actual point of the paper. Your final review will represent you, so make sure it does it well: read every sentence before you submit and make sure you stand behind it.

The conference has a rebuttal phase, can I ask for more experiments in my review?¶

Conference rebuttals are typically short, e.g. maximum one page at some conferences, and written under tight deadlines. Their main objective is to make sure reviews are accurate about the submission, which makes them differ slightly from iterations of the review process like they happen at a journal. In a rebuttal, you can ask for explanations of parts of the manuscript that don't seem clear, or point out what you think is an error and ask the authors to confirm, but you shouldn't request original work like new experiments or comparison to more baselines. If you find the evaluation lacking, state it in your review as a weakness of the submission, rather than would affect your recommendation if it were addressed within the short timeframe of the rebuttal phase.