Quality and Peer Review of Research: An Adjudicating Role for Editors

Abstract

Peer review gives research a stamp of approval, but the reviews themselves can be flawed. This is potentially serious for the writer, the journal, and journal user. This study describes shortcomings of the peer review process and condenses them into an explanatory framework involving situational, personal, social, and ethical factors. Some proposals to improve matters are impractical and may make them worse. Some data is offered which illustrates the problem and suggests a potential solution. Informed editors who avoid mechanical approaches engage cautiously and critically with reviews and guard against bias, even in themselves, could make a significant difference.

Keywords:

• editor engagement
• peer review shortcomings

INTRODUCTION

Academics' work generally has several facets, largely to do with teaching, research, administration, and service (Macfarlane, 2006). Of these, teaching and research have been called “the essential duality of Higher Education” (Tight, 2009, p. 190). An institution's success and an academic's advancement depend on their quality (Harman, 1998). For many academics, success in the publication of research is an expectation; it draws attention, adds prestige and attracts income to the institution (Snodgrass, 2006). Rejected research reports and returned applications for funding do nothing for an institution and less for an academic's prospects (Hopps, 1990). Harman (1998) lists various mechanisms by which the quality of research is widely judged. These include reviews of an institution's research products and processes, reviews of funding applications, and reviews of research papers submitted to journals. Each of these generally involves some form of research-related appraisal by peers. This is a judgment of worth made by one or more members of the relevant scholarly community (see, e.g., Hames, 2007). Given what hangs on such judgments, it is important that they are fair and competent or, at the least, as fair and competent as is possible. This article focuses on the peer review of research articles.

Editors commonly have each article reviewed by at least two people with some knowledge of the field. This procedure began on a casual basis in the nineteenth century but became commonplace after World War II as the number of submissions and specialization increased (Burnham, 1990). Studies of peer review have shown it can be seriously flawed, lead to capricious decisions, suppress original thought, and coerce authors into unnecessary revisions (Persaud, 1995; Weller, 2001; Shatz, 2004; Brower, 2008; Liesegang, 2009). For instance, in the USA, Peters and Ceci (1982) re-submitted twelve psychology articles to the journals that had recently published them. Eight were rejected. Garfunkel et al. (1990) took twenty-five accepted medical articles and re-submitted them for review to two more referees. The new referees wanted the articles to be revised before “acceptance.” Bradley (1982) found that 60% of authors felt that referees had focused on trivia, 40% felt they were careless, and 66% said they were pressed to conform to reviewers' subjective preferences. More recently, fourteen leading stem cell researchers wrote an open letter to editors accusing reviewers of “deliberately vetoing publication” of their work in order to gain an advantage (BBC, 2010).

Editors can be aware of reviewers' weaknesses. For instance, Smith (1999), an editor of the British Medical Journal, believes peer review to be “slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless for detecting fraud.” Eysenck and Eysenck (1992) add that it delays publication and is time-consuming. One editor commented that “all who submit articles for publication realize the Monte Carlo nature of the review process” (in Eysenck and Eysenck, 1992, p. 394). Why is article submission a gamble? Studies have revealed a variety of tendencies which shape reviewers' responses.

INFLUENCES ON REVIEWERS' BEHAVIOR

A reviewer's deliberations are expected to be constrained by the stated aims of the journal but there can be other, less evident matters which shape conclusions about the suitability of a submission. For instance, a reviewer may be instructed or assume that “less important” articles should be rejected in favour of articles which might raise the journals' impact (Hopps, 1990). Similarly, replications which confirm another's findings and studies which propose, test, and reject a theory are rarely published: there is a bias towards positive, new results. Of course, nonpublication distorts the record and threatens the validity of meta-analyses (Moonesinghe et al., 2007).

A second and obvious influence can stem from a reviewer's subject knowledge, understanding, and ability. Bacchetti (2002, p. 1271) showed that reviewers of medical research are guilty of “finding flaws where there are none.” For instance, reviewers often complain of a “small” sample size when the statistical analysis is more than capable of producing meaningful results. Ioannidis (2005) points out that attention should be on the constitution of the sample. A reviewer may also focus on trivia and may be negligent or careless. Perhaps a less expected tendency of reviewers is that of giving higher ratings to hard-to-read articles (Sokal, 1996). Reviewers show other biases, not necessarily tied to subject knowledge or competence. There is a tendency to favour arguments agreeing with the reviewer's beliefs and to be hostile towards new ideas (Mahoney, 1977; Armstrong, 1980). There is also a bias which favors the work of well-known, authority figures (Toulmin, 1972; Armstrong, 1982; Campanario, 1998a).

A further influence on reviewers' behaviour is the cultural climate. Eysenck and Eysenck (1992) suggest that reviewers in the West have learned a “persecution mentality” and perpetuate it themselves. The result is that articles, accepted or rejected, receive five times as many negative comments as positive comments, on average (Bakanic et al., 1987). It seems that reviewers see their role as rejecting submissions, something they generally seek to do by criticising the method (Finke, 1990; Tannen, 1998). Eysenck and Eysenck (1992) also point to inconsiderate, ad hominen remarks in a “climate of abuse” made by anonymous referees (see also Campanario, 1998b, p. 285).

These influences may largely be described as situational (as in the stated or perceived requirements of a journal), personal (for example, ability, personality, and attitudes), social (norms of behaviour in a culture, for instance), and ethical (right and wrong behavior). Although noted in relatively recent years in this context, the general phenomena have been known for some time.

Armstrong (1980), for instance, described the higher rating of hard-to-read articles as an instance of the “Dr. Fox Phenomenon.” Dr. Fox gave a meaningless exposition of a nonsense topic with confidence. His knowledgeable audience was impressed by its “clarity” and “stimulating” content (Natfulin et al., 1973). Similarly, psychologists would point out that the so-called confirmatory bias is an instance of Heider's assimilation–contrast theory in which information concordant with beliefs is accepted and discordant information is rejected (Lindzey and Byrne, 1969). Such behaviour preserves a reviewer's often well-integrated and extensive mental structures which might otherwise have to be abandoned. Favoring eminent researchers is a specific instance of Thorndike's halo effect (Thorndike, 1920). When most agree that someone is important, few dare to disagree. On the other hand, perversely opposing someone's theory or argument is accounted for by Brehm's reactance theory which explains why some people generally respond by taking the opposite view. Such people may be motivated by, perhaps, a need to convince others of their ability (Brehms and Brehms, 1966).

Sociologists, on the other hand, could point to the enormous imbalance of power in the reviewer-researcher relationship. As Giddens (2001, p. 420) puts it, “Power is the ability of individuals or groups to make their own interests or concerns count.” The exercise of this power does not have to be conscious. Foucault (1970, 1980) explained how prevailing ideologies—the unconscious values and beliefs generally held by communities—shape and constrain thought (Giddens, 2001, p. 676). Haralambros and Holborn (2000) describe how these “blind members of a society to alternatives [so they] tend to accept the current situation as normal, natural, right, and proper'. Lipton (2004, p. 151) adds that “background beliefs” shape a world view which some find difficult to step outside. Reviewers reading studies which do not accord with their ideologies and discourses may exercise their power to maintain the status quo by rejecting those studies. Shamoo (1994) describes an instance of a young scientists' work being treated in this way. At a shallower level, it could even account for some reviewers' insistence on having an article rewritten in their preferred styles. Even a culture of finding fault can be seen as stemming from habituated, unquestioned expectations of what is expected of a reviewer (McIntyre, 1985).

Regarding ethical influences, the stated purpose of academic journals is generally to further the general good by making available sound studies which enhance knowledge and understanding. Researchers who submit articles in good faith could claim a right to have them judged by reference to this general good and not the private good of the reviewer or the protection of the general good of a group to which the reviewer belongs. To the extent that a reviewer's bias is conscious and deliberate, it is a breach of trust which subverts a journal's aims in order to further selfish ends. Many would see this behaviour as unethical (e.g., Kitchener (1984) whose moral principles include beneficence, nonmalfeasance, justice, and fairness). Ideologies, discourses, and bias, however, may be unconscious. Such behavior could be described as ethically blame-free although it could also be argued that the reviewer has a duty to think, to develop competence, to avoid negligence, and to question his or her own motives and assumptions (Mabbot, 1966; Foucault, 1970).

Given such influences, adverse effects on peer review can be expected. Such influences may vary from reviewer to reviewer and article to article, and there may be some people who are free of them most of the time. Many of these influences, often operating below the level of conscious thought, may not even be evident to a reviewer. Nevertheless, the peer review system seems to be largely predicated on a belief in the perfection of people, a dubious belief in any context.

INFLUENCES ON EDITORS' BEHAVIOR

Of course, editors are people, too. They can also be influenced by the situation, their personal beliefs and attitudes, society's norms, and ethical considerations in similar ways to reviewers. For instance, the situation can press down strongly on an editor to make him or her comply with the views of others. Shamoo (1994) described how he felt that a reviewer's recommendation was unsound so sought another review elsewhere. As a consequence, this eminent reviewer threatened to resign from the editorial board, an event which could have had unwelcome consequences for the journal. Editors have also been described as unethical, as when sending material which was critical of a researcher's study to that same researcher for comment, and of steering reviewers towards a preferred decision (Rivera, 2009; Shamoo, 1994). Few studies of editors' behaviors have been made and little is known beyond what they reveal themselves. Nevertheless, the possibility of conscious and unconscious bias needs to be acknowledged.

COUNTERING ADVERSE INFLUENCES ON THE QUALITY OF PEER REVIEW

Suggestions for improving peer review include a code of conduct, guidelines, and structured response sheets (Epstein, 1995; Hadjistavropoulos and Bieling, 2000; Hauser and Fehr, 2007; Patterson, 2007; Rojewski and Domenico, 2004). These, at least, indicate what counts as “good” behavior (Austin et al., 1990) although, from a postmodern perspective what is “good” depends on your ideology. Even the expectation that criticism should be of the content and not the person is not universally accepted (Hames, 2007). Nevertheless, a particular community of researchers in a field could agree on a code of conduct for its reviewers even if this means that the code varies with field, location, and time. A code, however, has limitations. While it could help those who would be ethical, it is difficult to know when it has been breached (Austin et al., 1990). Nor can a code address unconscious bias.

Other suggestions may only compound the problem. Jayasinghe et al. (2001, 2006), in a study of the peer review of research funding proposals, believe that increasing the number of reviewers and having them review more often makes their recommendations more reliable (see also Marsh et al., 2008). Neff and Olden (2006) recommend using three or four reviewers and taking a majority vote. Vote counting may make the process reliable, even mechanical, but it does not guarantee its quality: reviewing could be reliably biased. Some suggestions could alter the balance of power. Patterson (2007), for instance, argues for discussion between writers and reviewers, but the balance of power would still be with the reviewers. Open peer review in which the reviewer's identity is revealed has been found to make reviewers more objective and thoughtful (Armstrong, 1982; Hadjistravropoulos and Beiling, 2000), but it also makes them reluctant to participate. How practical these are, given that reviewing is voluntary, is open to question. Formal reviewing could be eliminated altogether in a law-court approach in which a case is presented, cross-examined and defended on the Internet (Armstrong, 1997). It assumes, however, that the research community is willing to engage in the prosecution and defence.

A simple alternative suggested itself in a study by Holbrook et al. (2008). They conducted a survey of the level of agreement between Ph.D. thesis examiners in Australia. Even when recommendations varied wildly from failure to acceptance as submitted, the examiners' written reports were highly consistent and an adequate guide for the overseeing committees' decisions. While examining a thesis is not entirely like appraising a research paper, there are some evident parallels in judging the worth of a written account of a scholarly study. It suggests that it should not be reviewers who decide the fate of an article, but others who consider and weigh their reports.

A purely mechanical process in which editors simply endorse reviewers' recommendations would do nothing to counter unwanted effects or foster confidence in the process. An editor who engages critically with reviews and is sensitive to their weaknesses is well-placed to tilt the balance of power a little more towards the writer. This engagement may, on occasions, entail giving the writer an opportunity to respond to a comment. Traditionally, this is seen as what editors do. Is it a myth?

There follows a study of reviewers' comments and editors' responses to them which contribute to the discussion.

PEER REVIEW AND EDITORS' RESPONSES TO IT

The study of peer review is not easy. Editors and reviewers can be reluctant to participate as it wastes reviewers' time and risks losing their goodwill (Campanario, 1998a). On the other hand, a collection of reviews of a variety of genuine articles can be open to the criticism that like is not compared with like. Occasionally, circumstances present a natural event which provides relevant data. Such an occasion arose in the field of Social Science and, specifically, in Education. Evaluations of the quality of the peer review in Education are rare and largely uncritical (e.g., Baker, 2002).

Eight studies of textbooks, each for a different school subject, were submitted to subject-oriented, academic journals in education. All followed the same pattern: a review of the literature as it related to the subject, a statement of aims, an account of the method, and a discussion of the findings and their implications. The articles had a lot in common, particularly in the method, results, and conclusion. Differences were more evident where subject-specific matters were discussed. The anonymously written reviews and editorial responses provided the data.

Two journals accepted and published the articles without forwarding reviews. Nothing can be said about the peer review process in these journals. The content of the reviews for the others was divided into generic matters (material more or less common to all articles) and subject-specific matters. The latter were infrequent and have been omitted below as they could identify a particular journal. The comments received indicated that the reviewers' attention was most often attracted to generic material in the introductions and methods.

Articles A and B

Immediate acceptance

Reviewers' comments were not seen.

There were no editors' comments.

Article C

Immediate acceptance

Reviewers' comments re:

Introduction: Make relevant for an international audience in ways described by a reviewer. One preferred a philosophical rather than a psychological context. S/he indicated an unfamiliarity with psychology.

Editor's response:

This was a précis of the above comments with the option to address or ignore them.

Article D

Acceptance subject to ‘minor’ revisions

Reviewers' comments re:

Title: Change ‘to reflect content better.’

Introduction: Need for terms to be made clear for an international audience. A clearer definition of terms was required. One wrote that the argument was not presented in the form s/he preferred and was ‘annoying’ and ‘American.’

Method: More explanation and illustrative examples were required by one reviewer; another described the method as ‘clear.’

Discussion: One reviewer wrote about his/her dislike of textbooks and his/her concern that the article might encourage others to use them. S/he wanted it rewritten to support this dislike. One indicated that s/he ‘looked forward to reading a revised version;’ another wrote that the article was ‘very nearly publishable as it stands.’

Editor's response:

This was a selective digest of the reviewers' comments, indicating where s/he agreed and disagreed with each reviewer. For example, regarding textbook use, ‘You might refer to and dismiss this,’ was added.

Article E

Acceptance subject to ‘minor’ revisions

Reviewers' comments re:

Method: More examples of the definitions were required.

Discussion: It was felt that there was a need to acknowledge that there are non-textbook sources of support for a teacher.

Editor's response:

The editor asked for the above points to be addressed.

Article F

Acceptance subject to ‘minor’ revisions

Reviewers' comments re:

Abstract: This was described as needing to be more detailed.

Results: One reviewer wrote that s/he was not happy with ‘facts and figures’ and asked for them to be removed.

Discussion: Another reviewer asserted that training courses did not prepare pre-service teachers to use textbooks and wanted this to be added.

Editor's response:

The editor suggested that the referee's comments about ‘facts and figures’ might be given less weight.

Article G

Acceptance after ‘major’ revisions

Reviewers' comments re:

Title: The title was described as ‘motivating.’

Introduction: A reviewer wrote that it was ‘too long;’ another felt it was too short.

Method: One reviewer described the method as ‘detailed and clearly written.’ Another wanted more explanation, exemplification and clarification. This reviewer confused the method of data collection and analysis with another approach and insisted that a description of and citations for this other method be included.

Conclusion: One reviewer described this as ‘unsatisfying,’ without elaboration.

Editor's response:

The editor listed what must be revised before resubmission. Where there was contradiction, the editor opted for the comment requiring additions. No reference was made to the reviewer's confusion over the method and the criticism was included for attention.

Article H

‘Major’ revisions required. Article re-submitted. Additional ‘major’ revisions required. Article withdrawn and submitted elsewhere (see H′)

Reviewers' comments re:

Abstract: One reviewer found it to be unclear.

Title: This was described as ‘eccentric’ by one reviewer, without elaboration.

Introduction: A reviewer complained that s/he could not see how knowing reasons and understanding are related. S/he wanted the article to be ‘refocused’ with a different title and an account of only the textbooks contents, omitting reference to their use by teachers. A revised version was re-submitted with accompanying responses to these comments. There was no subsequent reference to these by the reviewers or the editor. Referring to a part of the article that was unchanged, a reviewer now believed that the illustrative examples of textbook content were presented as examples of good practice and so required further ‘major revision.’

Editor's response:

There was no editorial comment on either occasion, simply the provision of referees' comments. The time taken for the first review was one year and for the second was six months.

Article H′

H′ was the original, unrevised form of H. It was accepted with ‘minor’ revisions

Reviewers' comments re:

Title: This was described as ‘cumbersome’ by one reviewer.

Introduction: One reviewer found the definitions of terms to be ‘excellent,’ another found them to be ‘unclear.’

Editor's response:

The editor listed what was to be revised.

Most reviewers also acted as proof readers, even to the extent of pointing out a missing bracket. As this said little about the substance of an article, such comments were omitted.

DISCUSSION

The Reviews

The title was described as motivating (G), eccentric (H), in need of change to reflect the content better (D), and cumbersome (H′). It could, of course, be all of these but most reviewers, presumably, found it acceptable.

Three referees (C, D, G) made the point that the introduction needed to recognise an international audience. This was a valid criticism but not one that the majority noted.

There were also wide differences in judgments of the quality of the introduction, (for instance, “too long” versus “too short” (G), “excellent” versus “unclear” (H′)). There was also direct expression of emotion (e.g., “annoying”) and personal, stylistic preferences (D). In D, one reviewer wanted to impose preferences in a major way. Being personally inimical to textbook use, s/he felt the article might encourage it so should be recast in a form which discouraged it.

The method also attracted significant attention, some contradictory and some reflecting incompetence, careless reading or inability. For instance, a reviewer could describe it as “clear” or “detailed and clearly written” while another wanted more explanation (D). One reviewer of G confused the method with another and insisted on the inclusion of inappropriate elaboration and citations. And a reviewer of F disliked “facts and figures” so wanted them (that is, the very data constituting the results) to be omitted.

Some comments undoubtedly pointed to aspects of the articles which needed attention, such as allowing for an international audience. But there is evidence of wide variation in reviewers' responses, even to one article. There was evidence of pressing for personal preferences without justification and of emotive intrusions. There was also evidence of incompetence and of unsubstantiated resistance to ideas not shared by the reviewer. If these were removed from some reviews, there would be little left.

The Editors' Comments

The editors' responses were as varied as those of the reviewers. Some (G, H) appeared to reflect a clerical approach: reviews were collected, forwarded to writers and recommendations accepted. Contradictory statements were ignored, as in G, when one reviewer wanted a shorter introduction and considered the method to be clear, while the other wanted a longer one and more explanation in the method. While an editor may be unfamiliar with particular research methods, a reviewer's embarrassing error in this respect (G) should have been evident in the contrast between the reviews but it was, nevertheless, presented for action. Other editors seemed to be moderately mechanistic (for example, E), but this may be understandable given that the revisions required were “minor.” One (C) decided that the revisions were optional. F had a similar response to one referee's dislike of “facts and figures.” There was evidence of one (D) engaging with the reviews critically. This editor's marginal comments on the reviews indicated that s/he had noted the “American” reference and the instance of confirmatory bias (in this instance, the other side of the coin when discordant information regarding book use was rejected) and both were dismissed. Taken together, there was evidence that the editor's role was perceived in different ways, ranging from the mechanical processing of articles and reviews to a thoughtful and critical engagement with content. A mechanistic approach is problematic; the writer may be faced with contradictions, incompetence, expectations of compliance with stylistic preferences and unsubstantiated beliefs, bias and unjust rejection. Equally, it could lead to the unwarranted acceptance of an article. On the other hand, a critical engagement with reviews demonstrably stands a chance of filtering out inadequate comments to arrive at reasonable expectations and a balanced decision—assuming, of course, that the editor is aware of the weaknesses of peer review and is free of bias him or herself. There was no evidence in the editors' responses to indicate conscious or unconscious influences on them—indeed, it would be difficult for there to be influences on editors who chose to be mere observers, other than through the pressures of the situation. These would be unlikely to show themselves in this study.

A natural event allows research where it might otherwise be unwelcome but it has weaknesses. In particular, it is rare for variables to be well-controlled. It could be argued, for instance, that these journals had different standards. At the same time, the articles had much in common but were not identical, particularly when discussing subject-specific matters. Further, editors may have withheld some comments. And their responses may not tell all: there may be more going on behind the scenes. Even allowing that, the comments listed here point to shortcomings like some of those found elsewhere. Some might say these shortcomings are irrelevant as all the articles were published. Changes were made which probably improved the articles but changes were also made merely to accommodate reviewers' preferences, negligent reading, and incompetence, that is, to make them feel good. Less experienced research writers may have had less success.

CONCLUSION

The reviews offer further evidence that reviewers can still be grossly incompetent and biased. That is not altogether surprising and might not matter if editors countered such tendencies. In practice, some editors showed signs of working in ways which are unlikely to do so. They seem to prefer a mechanical role which gives decision making to reviewers, although journals claim that reviews merely assist the editors' deliberations (Jain, 2009). A mechanical approach misses an opportunity to alter the power relationship between the reviewer and the author and contribute to the quality of the process.

Yet, written evaluations of scholarly work, if considered and reasoned, can support defensible decisions about its quality (Holbrook at al., 2008). A way of improving peer review would be to engage with reviews and reflect reasonably on authors' responses to them. To different degrees, this sometimes happened in the study. To the extent that it did, it strengthened the author's hand and protected the author's (and reader's) right to a quality control process in which effort was made to reduce its shortcomings. In this process, the author should be able to draw an editor's attention to perceived incompetence, negligence, carelessness, unethical behavior, and unconscious bias. Often, an editor would be able to consider such points directly but, on occasions, may wish to seek advice. There is a danger, however, that increasing numbers of submissions to journals and the trend towards online, mechanised submission could tempt more editors to avoid engaging with reviews.

There still leaves the possibility of bias on the part of the editor. Editors need to know of potential influences on reviewers and also on themselves. They also need to know how they should deal with them. In part, this may be a matter of training, experience, and a willingness to apply that knowledge. Shamoo (1994) argues for a code of practice for editors which could help. From time to time, a subgroup of the editorial board could also sample reviews and the editor's responses to them to check on the quality of the process and note unwanted tendencies. Those journals with large editorial boards might consider using a small team to nominate reviewers, evaluate reviews, make justified decisions and communicate with the authors. Such a team's brief would include guarding against bias, something easier to see in another than in oneself.

While these actions could improve the peer review process relatively quickly, they do nothing for the reviewers themselves; they merely compensate for their shortcomings. To take a long term view, reviewing might be improved through the education and training of postgraduate students. Training programs should raise awareness of influences on the reviewer and should include guidance and practice in reviewing. This is doubly useful as it prepares such students for reviewing and for dealing with reviews when they, themselves, submit articles to journals. In the short term, journals may be able to supply succinct accounts of reviewer bias and urge their reviewers to read them. Even if they do, people are fallible, so the problem will never go away entirely, and the editor's engagement and vigilance will always be needed.

Given the specter of author litigation, perhaps on grounds of loss, libel, or bias (Robergs, 2003; Kalles, 2005), it would be wise to allow authors the right to appeal against a decision when they can make a nontrivial, prime facie case. Moreover, given the reviewer-author power imbalance, a right of appeal offers some redress to the author. But authors themselves need not be only passive recipients of whatever empowerment is allowed them. Foucault (1988)famously advised that the unreasonable exercise of power could be resisted by a refusal to comply. In practice, the need to publish is a big stick driving the researcher to accept whatever demands a reviewer makes. Nevertheless, authors may benefit if they stiffened their resolve to make a reasoned objection to an editor.

Seven editors of journals similar to those in the study were asked to comment on some of these suggestions. Four responded. They agreed that editors should make the decision about publication, not reviewers. One was confident in the abilities of the reviewers used. One expressed the view that the decision should rest on reviewers' comments and a personal reading of the article. Another wrote that the process is “evidence-informed.” But it was interesting that three saw this mainly in terms of the need to resolve reviewer disagreement. The possibility that reviews may agree and be inadequate was not mentioned. There was agreement that editing a journal takes a lot of time. Asked if sharing the load with additional editors would ease engagement, two felt it might take more time because of the need to liaise and the possibility of disagreement, one already used this system and found it was very helpful, and the other saw it as potentially helpful. Two pointed out that editorial boards and “assistant” editors are sources of advice. Regarding training for the position, most had experienced a mentoring or apprenticeship system where a more experienced person guided the new editor. That this could be insufficient was not mentioned. Nevertheless, the general feeling was that more formal training could be useful. There was some hesitancy about an appeals system, largely because of the time it might take. One editor's response to an author's protestations is to arrange a further review. Another saw it as something for the editorial board to consider. The others took the view that the editor's decision is final. Editors should take the possibility of flawed reviewing seriously and avoid a blind, mechanistic approach. They also need to be aware that they, too, can be biased. If they do not, they risk being accused of complicity by omission, a loss of faith in the system, and a strengthened belief that “peer review starts only after publication” (Rivera, 2009).

While articles for publication in research journals have been at the focus of the discussion, peer review also enters into other areas of academic work, as in the assessment of applications for research funds, the public grading of a university's research output and, at times, the assessment of an applicant for a particular post. The reader's attention is drawn to these as other areas where what has been discussed here is likely to have relevance.