Hypothesis testing methods for multi-reader multi-case studies

ABSTRACT

Rationale and Objectives:

Multi-reader multi-case (MRMC) studies are widely used to compare the diagnostic accuracies of different imaging modalities. Although Dorfman–Berbaum–Metz (DBM) method is the most popular one among the MRMC methods, the adaptions of ANOVA statistic for linear mixed model (LMM) are not based on solid theory and the assumption of ANOVA that all groups have the same number of samples might not be met in some situations. The purpose of the article is to investigate whether the statistics for testing fixed effect in linear mixed model can yield a closer type I error rate to nominal level.

Materials and Methods:

We proposed to use the statistics such as likelihood ratio test (LRT) and Wald statistic to test the hypothesis of equivalence in several imaging modalities. Extensive simulations were conducted and the application to a clinical example dataset was illustrated.

Results:

The simulation results showed the type I error rates of Wald statistic were closer to the nominal level under many simulated situations, especially when the simulated data was ordinal and the number of diseased and non-diseased were 100.

Conclusion: The Wald statistic whose degrees of freedom ware approximated by Satterthwaite's method showed competitive performance, indicating the potential of the statistic applied in DBM model for MRMC analysis.

KEYWORDS:

• MRMC
• linear mixed model
• hypothesis testing method

Acknowledgments

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript

Disclosure statement

The authors declare no conflicts of interest.

Additional information

Funding

This work was supported by the grants from Natural Science Foundation of China (No. 81773546 funding body: National Natural Science Foundation of China) and Beihang University & Capital Medical University Advanced Innovation Center for Big Data-Based Precision Medicine Plan (BHME-201801).

Notes on contributors

Huan Zhang

Huan Zhang, M.S., is a computational statistician. She has extensive experience in statistical modeling, algorithm development and machine learning.

Yuying Li

Yuying Li, PhD candidate. Research interests include diagnostic medicine, genomic statistics, infectious disease.

Qiushi Lin

Qiushi Lin, PhD candidate, is a Mathematical statistician. He has developed dynamic models on COVID-19, and has extensive experience in diagnostic medicine.

Xiao-Hua Zhou

Xiao-Hua (Andrew) Zhou, Ph.D., is PKU Endowed Chair Professor at Beijing International Center for Mathematical Research and Chair of the Department of Biostatistics at Peking University. Previously he was Professor in the Department of Biostatistics at University of Washington. He has made important contributions to medicine and public health by developing new statistical methods, particularly in diagnostic medicine and causal inference.

Guoshuang Feng

Guoshuang Feng, Ph.D., research professor, is the director of Big Data Center of National Center for Children's Health, Beijing Children's Hospital, Capital Medical University. He specializes in data exploration and mining in medicine.