Reviewing for STBE

STBE has recently published two editorials on papers – what makes an excellent journal paper (Cook 2019) and what leads to papers being declined (Spitler 2018). In this editorial, after briefly looking at the review process, I address what makes an excellent review and some common problems in reviews. Reviewing is a vital part of maintaining and improving the journal quality. A search of the engineering literature on “how to review” will turn up a small number of editorials like this one, most of them in the computer science field. In the built environment field, Meier (1992) provided a thoughtful discussion of paper reviewing that was influential on my thinking when I was much younger. While parts of the editorial are now dated, it is still well worth reading.

When an author submits a paper to STBE, it is first reviewed by the Managing Editor for formatting and plagiarism-checking before being passed to the Editor-in-Chief. I then decide whether I should send the paper out for review or not. If so, I will assign it to one of our Associate Editors, who will recruit reviewers. On the other hand, if I decide the paper is out of scope or needs significant work before sending it out for review, I will send it back to the authors with my comments.

Whether given by the reviewers, the Associate Editor, or me, reviews serve three important purposes:

1. Ensuring scientific quality of the work. This includes identifying errors in the experimental design, methodology, interpretation of results, and drawing of conclusions. Uncertainty analyses and model verification/validation are important for demonstrating the validity of the work, so lack of these should be a red flag for the reviewer in most cases.

2. Ensuring that papers provide a substantive and generally applicable contribution to knowledge. While this is partly a value judgment, a proper literature review and background are needed to establish that the contribution of the paper is new, substantive, and has some general applicability. The reviewer should be looking to see that the author has adequately covered previous work so that the author’s contribution is clear. The reviewer should also be looking for a clear and correct explanation of the author’s contribution. While the contribution should be clear in the main body of the paper, it also must be clear in the conclusions. If the work reported is a case study, the author should clearly spell out how it is generally applicable.

3. Ensuring that the paper communicates the work in a satisfactory manner and helping authors to improve the presentation of their work. Third-party reviews help to detect shortcomings in the presentation, including grammatical errors, missing information, unclear organization and other problems. Very few papers are completely free of items that are “clear to the author but not to the reader.”

Reviews for the journal have two or three parts:

1. Answers to a series of seven questions. These questions ask about specific common shortcomings, such as whether or not an uncertainty analysis is included.

2. A written review report, to be sent to the authors. The bulk of the review will go here, and a suggested format follows below.

3. Optionally, comments to the editor. Occasionally, there may be a reason to communicate something to the editor that you do not wish to put in your written review report. The most common reasons for doing this is to give a more nuanced treatment of your recommendation or suggest an additional reviewer.

The written review report may take on several forms. A common form that covers all the needed information is as follows.

1. Overview – a few sentences describing the topic of the paper with, perhaps, a comment on the level of interest that the paper can be expected to receive by the journal readership.

2. Recommendation. It is nice to have this near the beginning of the review, though it may also appear after the major comments. The usual categories are accept as is; requires minor revisions; requires major revisions or reject. In some cases, the recommendation may be more nuanced or conditional. For example, some major clarifications may be needed before the reviewer can deliver a final recommendation.

3. “Good things about the paper” (Meier 1992) This is optional, but as Meier pointed out, this can be quite useful if you are asking the author to make major revisions.

4. Major comments. In this section, the reviewer describes significant problems with the paper that cannot usually be corrected with simple changes to the text or figures. Obvious possibilities are flaws in the methodology, analysis, and interpretation, or any of the example shortcomings described above in the purposes of reviews. A significant lack of clarity due to poor English or poor organization should be called out as a major comment.

5. Minor comments. These are generally readily correctable items that need to be addressed – grammar, style, misused words, figures that are hard to read, minor items that are unclear, etc.

Different formats are sometimes used, particularly if the recommendation is “reject” due to one or a few major shortcomings. In that case, an overview, followed by an explanation of the major shortcomings, and a recommendation to decline the paper might appear in a few paragraphs.

Speaking of the “reject” recommendation, how should the reviewer choose between “reject” and “major revision”? For many papers, the decision can be made based on the criteria in Spitler (2018), but there are many cases where the paper might be improved to be acceptable. In these cases, I would suggest “reject” if the paper is so unclear as to make it difficult to tell whether or not it can be corrected, or if the level of effort to correct it is so high that it would represent a significant fraction of the effort put in to date.

What not to do? The most common problem that I see are reviews that really don’t engage with the paper and instead give it a quick pass with a few minor comments. For one well-written paper, I received two reviews of total length (together) of 50-words. One review consisted solely of the words “good paper” and the other, more critical, 48-word review had a criticism that was incorrect. My own review of the paper was page length, with a number of comments related to needed clarifications.

It should go without saying that reviews should be constructive and polite – the reviewers are there to help improve the quality of the paper. As a reviewer, much of what you write will be critical of the author’s work, and there is little to be gained by making it harder to swallow than necessary. Reviewers should also be on guard against self-serving practices such as insisting that authors include references to the reviewer’s work. This may be necessary in some case – after all, we look for reviewers that have worked in the same field. However, reviewers should consider whether the references are actually needed or not. More broadly, other ethical issues that might arise are addressed by the ASHRAE Code of Ethics (2020).

Finally, one other aspect to the review process should be addressed. We are all busy and many of us are overcommitted. Yet, if you publish journal papers, there is an implied obligation to also review journal papers. Nevertheless, many of us get far more invitations to review papers than we can possibly handle. So, I exhort the reader to be prompt in replying to review requests even if the answer is no. If you agree to review a paper and can’t finish the review on time, please advise the Associate Editor. We understand if additional time is needed. But reviewers that promise reviews that are not delivered and don’t answer email queries cause significant and unnecessary delays in publishing. Please don’t be that reviewer!

I’d like to express a hearty thank you to all the reviewers who work with STBE to improve our quality. Your work is greatly appreciated.

Jeffrey D. Spitler, PhD, PE
Fellow ASHRAE, Fellow IBPSA
Editor-in-Chief, Science and Technology for the Built Environment
Regents Professor and OG&E Energy Technology Chair
School of Mechanical and Aerospace Engineering
Oklahoma State University
Stillwater, Oklahoma, USA
[email protected]