Thoughts on Writing Good Scientific Papers and Doing Paper Reviews

The following is a collection of notes about writing papers that should primarily help students working on collaborative papers with me, but potentially also other paper authors who want to improve their papers and stumble upon this page. These notes are compiled based on my experience with writing and reviewing papers in robotics.

Contents

1. Elements of a Good Paper
2. What Do I Look for When Reviewing a Paper?
3. When Would I Reject a Paper?

Elements of a Good Paper

1. Clear logical structure

The main purpose of a scientific paper is to communicate a research idea; the most effective way to do this is to write the paper so that it has a logical flow. In a way, a good paper tells a story - not in a literary sense, but in the sense of having a well-defined starting point and then ensuring that every statement that is made at a given point follows from what was discussed before. This flow may need to be broken occasionally (for instance, when we include a background section before the section in which a proposed method is described), but it should otherwise be followed, both within sections and between sections, so that the reader has a good reading experience.

2. Proper use of language

Good papers are written well. In technical papers, we don’t necessarily expect perfect language, but bad language is an unwanted distraction from the technical aspects of the work. After all, the main purpose of writing a paper should be to communicate one’s work to the research community, and communication works best if language is used correctly. This includes at least the following aspects:

• Appropriate use of terminology in the research field
• Correct use of grammar, such as correct use of the definite and indefinite articles, and correct punctuation
• No typos (if one is not good at spelling words, there are automatic spell checkers for almost all text editors)
• Precise language (a scientific paper is not a literary piece; flowery language, such as words or phrases that don’t contribute anything concrete to a given statement, should be avoided as much as possible)

3. Clear motivation

The motivation of a paper sets the context of the work and should make it possible to understand both why the authors think this problem is relevant and why the reader should like to read the proposed solution to it. Particularly when I review a paper, I am not necessarily working on the exact same line of research that the paper is following, so it is important that the authors make it clear why they believe the problem they address is relevant, and in which exact context.

4. Clear understanding of the related work

The purpose of a scientific paper is not just showing what one has done, but also how this fits within the bigger picture of the research field. This requires detailed knowledge of similar work done by other research groups, both recently and in the more distant past.

In this context, it is important to talk about the recency bias. Scientific research is incremental, so it is generally the case that one would discuss recent papers more frequently than old ones; however, older techniques can sometimes get out of fashion even though they contain very useful ideas, so simply focusing on recent work is likely to lead to relevant old literature being missed. Having a broad view of one’s field is thus important for writing related work discussions that are as complete as possible.

The thoroughness of the related work discussion obviously depends on the publication venue. Conference papers are usually short (and have a strict page limit), so the related work discussion has to be very focused, while journal papers are considerably longer, so the related work discussion is expected to be more thorough and detailed.

5. Appropriate formalisation

Technical papers are most helpful when they are both precise and concise; this is best achieved by using an appropriate mathematical formalisation. Including a formal description of a problem is necessary due to various reasons:

• If defined well and used consistently, formalisation eliminates ambiguity and minimises the possibility for misinterpreting statements
• Mathematical notation can shorten the written text, as entities can be referred to by their mathematical symbols instead of their full names in natural language

In many cases, a paper would build on a well-known general formalisation, which clearly simplifies the task of working with the formalisation, as it is mainly necessary to ensure that notation is used consistently and that entities are used in a way that is consistent with their definitions.

When formalising, it is also important not to over-formalise, namely there should be as much formalisation as necessary for presenting the proposed method, but not more than that. In other words, the purpose of formalisation should not be to impress the reviewers or other researchers, but to explain the proposed method as well as possible so that others can understand it fully and are able to build on it in their own work.

6. Meaningful evaluation

In fields such as robotics, it is generally important to evaluate techniques experimentally in order to demonstrate that they are indeed useful from a practical point of view. The evaluation should, however, be meaningful, namely it should not evaluate aspects of a proposed method that are trivial or theoretically obvious, but should instead focus on aspects that are subject to practical uncertainties and thus require statistical evaluation. If such uncertainties can be excluded from consideration, a use case-based demonstration instead of a full-fledged experimental evaluation may also make sense.

7. Discussion of limitations and plans for future work

Scientific papers are not, or should not, simply be advertisements for a proposed method. An important part of the scientific process is to be honest about the limitations of one’s own work and discuss those openly. After all, authors are the ones who know their method best - both the positive and the negative aspects thereof - so they are in the best position to talk about this in their paper. A discussion of the limitations also leads naturally to ideas for future work, which can be particularly useful for readers who may want to build on the presented method.

8. Appropriate Use of Graphics

A final element of good papers is the use of diagrams and plots so that they help in understanding important aspects of the proposed method.

Diagrams

• Good diagrams are easily readable and include elements that are either clear from the context or are explicitly explained in the caption or legend
• Overview diagrams can be particularly helpful in papers, as they summarise a method and give the reader a general idea about it before they have even read a single sentence from the paper
• Bad diagrams are filled with content that is not really necessary for understanding the proposed method or whose purpose in the paper is not really clear

Plots

• Plots should also be easy to read and the axes should always be appropriately labelled
• If multiple curves are displayed in a plot, they should be easily distinguishable from each other; for instance, very similar colours should not be used for different curves
• Particularly when displaying geometric data, the axes scales should always be equal so that the data are displayed correctly; a grid should preferably be included as well to improve readability
• For plots that are supposed to display similar effects, the ranges of the dependent variable should be the same so that the plots do not present misleading information

What Do I Look for When Reviewing a Paper?

When reviewing papers, I generally try to see if the above elements are included in the paper, and to what extent. I particularly try to answer the following questions in my reviews:

• Motivation
  • Is it clear what problem is being solved and why this is relevant?
  • Have the authors made it clear what they believe their contributions are?
    • This is important because it gives me an opportunity to evaluate whether the perceived contributions are supported by the paper itself
• Related work
  • Is the relation to the state of the art made clear?
    • An annotated bibliography is needed for this. The relation is unclear if the related work is just a collection of short summaries, without any comments about how the cited papers concretely fit the authors’ work
  • Are the cited papers suitable; in particular, are they from venerable peer-reviewed venues?
  • Does the related work discussion include papers that I know address the same or a similar problem?
  • Related to the previous point, if I am not aware of similar work before reading the paper and I conduct my own short literature search, can I easily find similar papers which the authors should have been able to find as well?
• Formalisation
  • Is there sufficient formalisation to understand the addressed problem?
  • Is the formalisation sound?
  • If the used formalisation is generally well-known, are its interpretation and use within the paper accurate?
  • Is the notation appropriate and correctly used?
• Evaluation
  • Is it clear what the purpose of the conducted evaluation is?
  • Is the experimental design described clearly and fully? Are there any important missing details in the design description?
  • When reporting results of statistical evaluations, are uncertainties reported as well?
  • Is the presentation of results appropriate? Are plots designed according to the aforementioned principles?
  • Does the interpretation of the results actually follow from the results?
  • Is the evaluation suitable and sufficient for the proposed method or are there any additional important aspects that should have been included?
• Conclusions
  • Is there a discussion of the limitations of the work?
• Paper structure and language
  • Does the paper have a clear logical flow?
  • Is the used language appropriate for a research paper?
    • The worst offenders of this are papers that read a bit like blog posts
  • Are diagrams clear enough to facilitate the understanding of the paper or do they actually create confusion?

When Would I Reject a Paper?

When reviewing papers, I always try to look for positive aspects that would make the work publishable; however, as a paper reviewer, my job is to also scrutinise the paper in order to ensure that a high scientific standard is maintained. There are thus a few cases in which I have no other choice but to suggest paper rejection. The following are the primary ones:

• The paper is written so sloppily that there is missing content or missing figures that are being referred to
• The paper does not seem to address a concrete problem
• The related work discussion is not detailed enough, or there are unsubstantiated claims about the limitations of the state of the art
• The description of the proposed method lacks sufficient explanations or is written in a way that makes it difficult to understand what the method is really trying to achieve
• There is either no evaluation or the evaluation is inappropriate for evaluating important aspects of the method
• The writing is so bad (beyond a few typos and grammatical errors) that it makes the paper very difficult to understand
• Obviously, if I discover any plagiarism or questionable citing practices