Training the Reviewer?

Football referees must undergo professional training and even sit examinations before they are qualified. What about the referee/reviewer for a scientific journal? Is any training provided for this? Is it necessary and, if so, how is it delivered and validated?

All current postgraduate training programs in orthodontics provide training in critical appraisal of the literature. This is a core skill—essential for understanding and evaluating previously published work to create an informed knowledge base for examination purposes and also for the literature reviewing process, which is an important component of the Masters or Doctorate research thesis. Some training in evaluating the scientific quality of published work, therefore, is provided through literature discussion seminars, journal club discussion groups and during the writing of a critical review that prefaces a research thesis. These should, when delivered well, help a trainee on qualification to be able to evaluate the merits and demerits of any publication in a dental/orthodontic journal. But practice is needed and further training in statistics and epidemiology is often regarded as important for the refereeing process.

Many of us who have been honored with the request to referee for scientific journals will often have been introduced to it by our supervisors. It is perfectly acceptable for senior referees to ask their trainees to read scientific papers that they have been asked to referee, subject to considerations of confidentiality. The supervisor can then discuss with the trainee how to write a referee’s report, sticking to the journal format requested by journal editors. Who better than an experienced referee to teach trainees how to be a referee?^{1, 2} This ‘mentoring’ from an ‘older’ hand is generally regarded as a very useful aspect of training for an academic appointment. With falling numbers of full-time academics in orthodontics, who will undertake this in the future?

Journal reviewers are usually chosen from a bank of referees with acknowledged expertise in a specified area(s) as demonstrated by relevant publications in the scientific literature or by active research interests in similar or allied fields. Reviewers are expected to undertake a review with the training that has been provided as outlined above, aided by further experience. This is usually gained through the pursuit of a higher research degree and on the basis of having undergone the review process themselves several times. The further higher degree training should enhance previous statistical training and other epidemiological training, both of which are very valuable to the refereeing processes. However, is structured training in refereeing available?

The BMJ offers a training package for peer reviewers, which aims to help referees learn about peer review and to understand what makes a referee’s report/review really useful to editors and authors (http://bmj.bmjjournals.com/advice/peer_review/).

Although developed for use in a randomized controlled trial of peer review training, it can be used for other purposes. Much of the material relates to the general art of peer review and includes presentations and written exercises.

It is divided into 4 sections:

• To inform participants on the state of the peer review research

• To make clear what constitutes a good review

• To help participants understand what matters to editors about reviews

• To give participants help in producing a good review

Has training referees been of any use? A 3-arm trial has been conducted where reviewers were allocated randomly to receive a day’s face-to-face training, plus a package of information, the package alone or nothing.³ The outcome measure was the quality of the review of 3 papers before and after training, and the ability of reviewers to spot deliberate errors inserted into the papers. With this limited sample, generally training did not produce improvements, but the author states that the question remains whether more intensive (but expensive) training might. A new class of professional referees could arise³ and this has happened to some extent with statisticians being increasingly involved with refereeing, the development of systematic reviewers, and increasing training in critical appraisal.

For those involved in Cochrane systematic reviews, training is provided (www.cochrane.org/resources/handbook/). These workshops are invaluable for training in literature evaluation and honing skills in critical assessment.

The section on the BMJ website above entitled ‘What we know about the peer review process’ makes for very interesting reading and will be useful to all prospective authors. An area of debate in the review process is whether the authors and reviewers should know each other’s identities. It has been acknowledged that the only ethically justifiable systems of peer review are either completely closed (with no one but an editorial assistant knowing the identity of the authors and only the editor knowing the identity of the reviewer) or completely open.⁴ However, how easy is it to hide an authors’ identity? It is not easy; in RCTs of blinded peer review, reviewers correctly identified either the author or institution in 24–50% of cases.² From the results of RCTs on open review, it made no difference to the quality of the referee’s report or to the recommendations made where reviewers were asked to sign their reports.^5–7 More evidence is required through RCTs with regard to training programs for journal referees. Training for the journal referee would seem important to maintaining the scientific quality of peer-reviewed journals. The 2 websites identified above may help provide initial ‘training’ for the review process to those asked to perform this important task for the first few times and also refresh the minds of those who have been doing it for some time.