Studies on Reliability and Measurement Error of Measurements in Medicine – From Design to Statistics Explained for Medical Researchers

Abstract

Reliability and measurement error are measurement properties that quantify the influence of specific sources of variation, such as raters, type of machine, or time, on the score of the individual measurement. Several designs can be chosen to assess reliability and measurement error of a measurement. Differences in design are due to specific choices about which sources of variation are varied over the repeated measurements in stable patients, which potential sources of variation are kept stable (ie, restricted), and about whether or not the entire measurement instrument (or measurement protocol) was repeated or only part of it. We explain how these choices determine how intraclass correlation coefficients and standard errors of measurement formulas are built for different designs by using Venn diagrams. Strategies for improving the measurement are explained, and recommendations for reporting the essentials of these studies are described. We hope that this paper will facilitate the understanding and improve the design, analysis, and reporting of future studies on reliability and measurement error of measurements.

Keywords:

• reliability
• measurement error
• classical test theory
• generalizability theory
• intraclass correlation coefficient
• standard error of measurement

Abbreviations

CTT, classical test theory; G theory, Generalizability theory; ICC, intraclass correlation coefficients; SEM, standard errors of measurement; CSA, cross-sectional area; MRI, magnetic resonance imaging; PET, positron emission tomography; US, ultra sound; RCT, randomized controlled trial; ANOVA, analysis of variance; D-study, decision study.

Data Sharing Statement

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

Acknowledgments

The authors would like to thank Myrthe Verhallen for her help in the development of the figures of the Venn diagrams.

Author Contributions

All authors made a significant contribution to the work reported. They all took part in the conception, delineation of the content, and discussion of the examples. LM drafted the various versions and all took part in revising and critically reviewing these versions; all gave final approval of the version to be published; have agreed on the journal to which the article has been submitted; and agree to be accountable for all aspects of the work.

Disclosure

Lidwine B Mokkink and Henrica CW de Vet receive royalties from Cambridge University Press for the book “Measurement in Medicine” to which the authors referred to in this paper. The authors declare that they have no other competing interests in this work.

Additional information

Funding

This work is part of the research programme Veni (received by LM) with project number 91617098, funded by ZonMw (The Netherlands Organisation for Health Research and Development). The funding body has no role in the study design, the collection, analysis, and interpretation of data or in the writing of this manuscript.