Peer Review in a Generalist Journal

One of the important parts of the job of a journal editor is finding good peer reviewers.¹ In this editorial I investigate the peer review process in the context of the Journal of the American Planning Association. For professional planners I explain some of what makes a refereed journal such as JAPA different than other APA publications such as Planning magazine. For reviewers new to JAPA I outline how reviewing for a generalist journal sponsored by a professional association compares with other kinds of reviewing. Peer review helps authors refine their arguments and communicate to a wider audience. It also ensures that every JAPA article has a high degree of rigor and relevance (Forsyth, 2019b).

JAPA Articles and Peer Review

JAPA publishes three kinds of peer-reviewed articles: standard articles, Review Essays, and Viewpoints. book reviews are edited by the book review editor. Commentaries, JAPA’s version of letters to the editor, are edited by me.

All the peer-reviewed article types go through the same review process at JAPA (Forsyth, 2019a). As editor I read and then desk reject up to half the submissions for reasons such as not having a clear question; that is, these articles do not go to peer review. About one-fifth more I send for advice from an editorial board member (a pre-review). Most of these pre-reviewed articles are desk rejected too, though a few go to full review. For quite a few desk rejects, I provide advice on how to get the paper into shape for a more successful submission (the “soft” reject). For authors new to JAPA, I try to provide more help. This is a quick process allowing authors to regroup and decide what to do next.

The full review process also involves potential rejection. Of all articles I send to full review, around a quarter are rejected by a full slate of reviewers. That leaves an annual acceptance rate that is about a quarter of submissions, sometimes more and sometimes less. By desk rejecting a fair percentage before full review, either alone or with a pre-reviewer, I hope JAPA reviewers know that they are investing time in a paper that may be viable. By having a second opinion for many submissions, as well as rejections after full review, I make sure I am not the sole gatekeeper.

There are several different formats for full review, and JAPA uses double-blind peer review. In this arrangement both authors and reviewers are anonymous or blinded, an approach that is common in the social sciences. Other types include single-blind review where the authors are revealed but not the reviewers, historically used in the natural sciences. A final major type is open review, where both authors and reviewers are named, and reviews may even be published (Harington, 2018). Although there is some concern that blind reviews are not really blind, in that the author reveals themselves via self-citation, I read all articles and work with the Managing Editor to make this more difficult to guess. (I also find that when authors and reviewers do reveal their guesses to me, in terms of each other’s identities, the guesses are often wrong.)

Although there has been controversy about which kind of peer review is best, double-blind review has benefits, particularly for those who are not already prominent. Some studies show it helps less experienced scholars, those from less prestigious institutions, women, and those otherwise marginalized to become published, and at least it does not hurt such authors (Garvalov, 2015; Kaji-O’Grady, 2017). For example, in computer science, conferences use both single- and double-blind review, meaning the two processes can be compared. One study examined 21,535 published articles in 71 major conferences and found “newcomers,” those who had few prior publications in the venue even if they had substantial publications elsewhere, were less likely to be published in single-blind proceedings than those that were double-blind. There could be many reasons for this, but it appeared that the anonymity of double-blind review helped newcomers break into important venues (Seeber & Bacchelli, 2017).

Compared with open review, blind review saves reviewers some time; for example, they do not need to polish the review beyond what is needed to convey a point. Also, compared with open review, blind review makes reviewers more likely to accept review requests (van Rooyen et al., 2010), keeping the pool diverse and of high quality and saving journal editors time finding reviewers.

Though JAPA uses double-blind review, it is subject to a certain degree of scrutiny. JAPA shares the editor’s decision letter and the complete set of anonymous reviews with all reviewers. This is a tradition I inherited. The logic is to help improve review quality, because reviewers get to see how their unnamed peers analyze the same paper. It also subjects the process to some checks and balances. Although the reviewers do not have all the information I have as editor—for example, information about conflicts of interest or overlapping publications—it provides some transparency. However, JAPA does not allow sharing of the papers under review. Some reviewers argue that they can train students by sharing papers with them to co-review; my perspective is those reviewers should find other ways to train students.

Selecting Reviewers

I am sometimes asked by planning professionals how I choose reviewers. The answer is I look at individual qualifications, try to balance the slate, and find people who will be committed to the joint effort of improving the body of knowledge in planning. I examine each of these issues in turn.

Reviewers need a high level of qualification in the field. I find reviewers who have published relevant research. Such reviewers may be cited in the paper being reviewed, but I find reviewers using various publication databases as well. For JAPA, they have also typically finished their PhD, meaning they have some experience. Peer reviewers are true peers, though due to the criteria used for selecting them, they are all more experienced than the least experienced authors. From time to time, I also have a practitioner act as an additional reviewer. This can be helpful for some papers.

JAPA typically uses three reviewers for each article, and within this panel I need to balance various forms of expertise. At a session providing advice to beginning authors, I remember hearing an early-career researcher complaining about submitting to a journal and not receiving three reviews from experts in his exact subarea with about 10 people in it. Although such a narrow review slate might be appropriate for a specialist journal, a generalist journal such as JAPA needs a wider pool. I need to ensure I cover people with methods, theory, topical, and geographical knowledge, while avoiding making a slate of people who have studied with each other. I try to find at least one reviewer who may stretch the author by, for example, knowing about a similar topic in a different geographical area, while being open to the ideas in the paper. For JAPA, where implications for planning practice are a key part of its mission, I also need to include reviewers who understand that practice. This variety among reviewers also means authors need to write clearly and avoid jargon. This is a long-running practice for JAPA, but recent research argues that articles that can be understood by those not in the narrow specialty are, perhaps unsurprising, likely to receive more citations (Freeling et al., 2019; Martínez & Mammola, 2021).

Peer review is a form of both individual altruism and a community endeavor, “a collective commitment to the community production of knowledge” (Meadows & Wulf, 2019). To find reviewers who will actually accept invitations, I need to figure out who is likely to be committed to this work.² Those in the JAPA family are obvious choices, meaning those who have published in JAPA or are faculty in accredited planning programs in the United States (the geographical focus of the American Planning Association). Those interested in the specific subject are also likely to want to know what is happening in the field, tipping them toward accepting a review. People who publish a lot generally feel they need to give back. Here I want to note that authors need to accept more reviews than articles they submit because each article gets multiple reviews.³ Most authors understand this issue, but some are frustratingly oblivious. Of course, JAPA is also competing with other journals for people’s attention, so even the best altruists may be overstretched.

What Good Reviewers Do

Some of what makes a good reviewer is about the character of the advice they give to authors, but this is not the whole picture. Other issues like timeliness are also important. First and foremost, the best reviews help the authors to make a better article. Sometimes, however, that means saying the current version should be rejected so the authors can go back and do more work (Wallace, 2021).

Most reviewers tend to focus on a standard set of issues, though some may be emphasized differently depending on the article type (Faff, 2018; Forsyth, 2019c). Is the paper well written, in clear and largely jargon-free prose? Is the contribution clear, new, and important? Does it have a research question, clearly justified by the literature, and capable of being answered by the methods used? Was the study conducted proficiently, acknowledging limitations? Are findings supported by evidence, including well-composed figures and tables? Are references appropriate? Some reviewers also assess the fit with JAPA’s mission, particularly if they are regular readers, although that is also a job for the editor.

Good reviewers also have a certain attitude or tone (Meadows & Wulf, 2019). They are prepared to identify the paper’s weaknesses clearly. However, they also explain what needs to be done to improve the paper, even if not with the current submission (i.e., recommending a rejection). The reviewers work with the study the authors have, though of course suggesting improvements.

Reviewers also need to be able to assess how well authors have responded to an earlier round of review. This can be tricky because authors are responding to multiple reviewers and the editor. However, if authors are not responsive in their revisions, and the article remains weak due to this, reviewers say so clearly.

Experienced reviewers provide an appropriate amount of depth in their comments. For example, when good reviewers think a paper should be rejected and basically go back to the drawing board, they tend to have shorter reviews of about half a page or even less. If they think there is a chance for revision, they will say more. However, long reviews of four or more pages can be hard for authors to address.

Good reviewers meet deadlines, helping the editor provide timely advice to authors. Timeliness is increasingly prized, and most journals now publish metrics about review times. Such reviewers respond quickly to requests and if they cannot review, they frequently suggest someone else who could do it. JAPA also keeps the same pool of reviewers for all revisions, so authors receive consistent messages. With the exception of those I’ve written to ahead of time (“You said accept twice so I really don’t expect you to review one more time”), those who drop out or ask for extensions that are months long cause me some bother. Fortunately, this does not happen much.

How I Use Peer Review to Help Authors

Like many other journals run by associations, as editor of JAPA I have a role to really build the field, and that means I also provide guidance for authors on how to interpret reviews. In some journals the editor sends reviews to the article author to make sense of. That is not the case in JAPA.

As well as reading each article submission, I read all of the reviews and help the author interpret them. In the vast majority of cases the reviews are similar or at least complementary. Some reviews may appear contradictory but are looking at different aspects of the paper or have similar concerns at a more abstract level while differing in how to address the problems. However, in some cases reviews really are contradictory, and it is the editor’s role to help the author by suggesting how to navigate the different guidance. In addition, some reviewers tend to be more positive or negative about all articles they review, and part of my job is to figure out what they really mean. I also have some information unavailable to reviewers, such as details about conflicts of interest and related publications, so I can frame my advice taking those issues into account.

Peer review is generally useful in helping authors improve their articles and to make sure that a certain standard of quality has been reached. Ultimately, undertaking peer review demonstrates a commitment to the community of scholars and in JAPA to the field of urban and regional planning.

Additional information

Notes on contributors

Ann Forsyth

Ann Forsyth is the Ruth and Frank Stanton Professor of Urban Planning at Harvard University.

Notes

1 Thanks to YingYing Lyu for comments.

2 Though there are systems for gaining recognition, like Publons, in my experience most reviewers do not check that box. I may have, perhaps, if it was around when I started reviewing, but now having conducted many hundreds of reviews even without my Editorial reviews for JAPA, it seems too late to start.

3 If a single authored article goes through two rounds of review with three reviewers, that’s six reviews on the debt side of the cosmic balance sheet for just one submitted article. Of course, this might go down if a paper has lots of co-authors to share the load of reviews per article, but it would go up if an article has three rounds of review (JAPA’s limit). There might even be more rounds if it is rejected and goes to another journal. Unless you are routinely publishing with large teams who can share the reviewing load, an author needs to accept more reviews than they submit articles.