STARDing Again: Revised Guidelines for Reporting Information in Studies of Diagnostic Test Accuracy

As members of the Society for Personality Assessment and readers of the Journal of Personality Assessment (JPA), a central focus of our activities is assessment. This naturally implies that much of our work is devoted to developing or using diagnostic tests.¹ The Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 2014) state that the psychometric properties of tests need to be reported in such a way that users can sufficiently identify whether the test provides evidence of reliability and validity. This naturally implies that the articles about the scales need to give enough information about the way they were developed and evaluated that readers can make this judgment. Whiting et al. (2004) highlighted many potential sources of bias and variation in studies of diagnostic accuracy, including the demographic features of the participants, the severity and prevalence of the disorder, inappropriate choice in the selection of the reference standard, the handling of indeterminate results, and so forth. A later review by the same team found “consistent evidence” for the biasing effects of many of these factors (Whiting, Rutjes, Westwood, Mallet, & the QUADAS-2 Steering Group, 2013). However, there is considerable variability among studies in reporting how they were done or what was found (Honest & Khan, 2002).

In an attempt to overcome the problem of inadequate reporting of information regarding the development of diagnostic tests, the Cochrane Diagnostic and Screening Test Methods Working Group, consisting of 25 experts, primarily from medical and epidemiology departments, developed the Standards for Reporting of Diagnostic Accuracy (STARD; Bossuyt et al., 2003). STARD consists of a checklist of 25 items, tapping such areas as the method of recruiting participants, the order of testing, and the reference standard. This is supplemented with a flow chart providing detailed information about the number of people classified at each stage of the study. The Standards were recommended and published by two dozen of the leading medical journals, including the Journal of the American Medical Association, the British Medical Journal, the Lancet, and others. Meyer (2003) wrote an editorial in this journal strongly recommending that authors of articles about diagnostic tests adhere to the STARD initiative, with some modifications making them more appropriate for psychologists.

The STARD guidelines (see Table 1) have recently been updated, with the aim of making them easier to use, and incorporating recent evidence about sources of bias and the applicability of the results (Bossuyt et al., 2013). Most of the items remain unchanged, but some have been reworded, combined, or split, so that there are now 30 items grouped into four sections: introduction, methods, results, and discussion. The additional or changed items, keyed to Table 1, include the following:

2. A structured abstract.

3. A description of the intended use and clinical role of the test.

4. An explicit statement of the study hypotheses in order to preclude an overly generous interpretation of the results.

18. A statement of the sample size, and whether the targeted number of participants was achieved.

26–27. A structured discussion that includes the study limitations, bearing in mind the targeted application of the results.

28. Because prospective test accuracy studies are clinical trials, they can be included in a clinical trials registry, such as ClinicalTrials.gov or isrctn.com.

29. Where the full protocol may be obtained.

30. The source of funding.

Table 1. The STARD 2015 checklist for reporting information in diagnostic accuracy studies.

Section & topic	No.	Item	Reported on page no.
Title or abstract
	1	Identification as a study of diagnostic accuracy using at least one measure of accuracy (e.g., sensitivity, specificity, predictive values, or AUC)
Abstract
	2	Structured summary of study design, methods, results, and conclusions (for specific guidance, see STARD for Abstracts)
Introduction
	3	Scientific and clinical background, including the intended use and clinical role of the index test
	4	Study objectives and hypotheses
Methods
Study design	5	Whether data collection was planned before the index test and reference standard were performed (prospective study) or after (retrospective study)
Participants	6	Eligibility criteria
	7	On what basis potentially eligible participants were identified (e.g., symptoms, results from previous tests, inclusion in registry)
	8	Where and when potentially eligible participants were identified (setting, location, and dates)
	9	Whether participants formed a consecutive, random, or convenience series
Test methods	10a	Index test, in sufficient detail to allow replication
	10b	Reference standard, in sufficient detail to allow replication
	11	Rationale for choosing the reference standard (if alternatives exist)
	12a	Definition of and rationale for test positivity cutoffs or result categories of the index test, distinguishing prespecified from exploratory
	12b	Definition of and rationale for test positivity cutoffs or result categories of the reference standard, distinguishing prespecified from exploratory
	13a	Whether clinical information and reference standard results were available to the performers or readers of the index test
	13b	Whether clinical information and index test results were available to the assessors of the reference standard
Analysis	14	Methods for estimating or comparing measures of diagnostic accuracy
	15	How indeterminate index test or reference standard results were handled
	16	How missing data on the index test and reference standard were handled
	17	Any analyses of variability in diagnostic accuracy, distinguishing prespecified from exploratory
	18	Intended sample size and how it was determined
Results
Participants	19	Flow of participants, using a diagram
	20	Baseline demographic and clinical characteristics of participants
	21a	Distribution of severity of disease in those with the target condition
	21b	Distribution of alternative diagnoses in those without the target condition
	22	Time interval and any clinical interventions between index test and reference standard
Test results	23	Cross-tabulation of the index test results (or their distribution) by the results of the reference standard
	24	Estimates of diagnostic accuracy and their precision (e.g., 95% confidence intervals)
	25	Any adverse events from performing the index test or the reference standard
Discussion
	26	Study limitations, including sources of potential bias, statistical uncertainty, and generalizability
	27	Implications for practice, including the intended use and clinical role of the index test
Other information
	28	Registration number and name of registry
	29	Where the full study protocol can be accessed
	30	Sources of funding and other support; role of funders

As before, there is also a flow chart (Figure 1), and both are freely available at http://www.equator-network.org/reporting-guidelines/stard/.

Figure 1. Standards for Reporting Diagnostic Accuracy (STARD) prototype flow diagram.

The revision of STARD raises three questions:

1. Because the initiative was developed primarily by and for physicians, is it appropriate for tests used by psychologists and, if not, what needs to be changed?

2. Has STARD had any impact on the field?

3. Are there any alternatives to STARD?

In this first of two editorials, we try to answer these questions. In the second part, we propose our revision to STARD 2015, which we believe will make it more relevant for developers of scales used in psychology and the other social sciences.

With respect to the first question, we believe that some modifications and additions need to be made. To begin with, STARD includes a number of items relating to a description of the target population, but very little about the independent raters who might either interpret the test or determine if the person meets the criteria for group membership: who they were in terms of background, training, special training in the use of the test, or even their number. Interestingly, such information was required in the original version of STARD, but not in the revision. We believe these deletions are unfortunate, as such information is necessary to interpret the study and to determine the usability of the test in other hands. Second, presumably because many medical tests are either derived from laboratory equipment or yield dichotomous results, there is no requirement that the items be described in terms of how they were derived, content relevance and completeness, the scoring format, the number of options, whether any are reverse scored, the existence of subscales, and so forth. In psychology, these are considered crucial pieces of information. Third, if there are subscales, it is important to indicate whether these provide any additional information over and above the full-scale score (Reise, Moore, & Haviland, 2010). Fourth, although STARD discusses how missing data were handled, it does not deal with related statistical issues, such as how outliers are dealt with, or the validity of the statistical assumptions. There are other areas that we feel need to be added, and these will be addressed in the next editorial.

The second issue is whether STARD has had an impact on reporting. Unfortunately, the picture is far from encouraging. Smidt et al. (2005; Smidt et al., 2006) compared 124 articles published before STARD with 141 published afterward. Only 41% reported on more than 50% of the STARD items, and none reported more than 80%. There was an increase in 1.81 items after STARD, but the change was not statistically significant, nor was there any difference between journals that had and had not adopted the initiative.

Regarding the last point, alternatives to STARD, there is good news and bad. The good news is that there are other sets of criteria. The Guidelines for Reporting Reliability and Agreement Studies (GRRAS; Kottner et al., 2011) is similar to STARD, but goes into more detail and is more appropriate for tests used in the behavioral sciences. However, it is limited to the reliability phase of scale development and has not been formally adopted by any journal. Also, Streiner and Kottner (2014) wrote an article outlining in narrative form how scale development and testing articles should be written, but it does not have a checklist or flow chart.

So where does that leave us? STARD has not (yet) had a major impact on the quality of articles of diagnostic tests, and needs to be modified to accommodate the types of studies used in psychology. However, we believe that it is definitely a worthwhile undertaking and the adoption of a modified STARD, more appropriate for psychologists, would definitely improve the reporting of such studies in this journal.

Notes

¹ A diagnostic test is one that is used to classify persons into one of two or more groups, such as normal, borderline, or not normal, or accepted, or not accepted into a graduate program. Scales yielding continuous scores are not usually used as a diagnostic test, unless cut points are used to define diagnostic groups.