Calculation of test-retest variability in phase I/IIa clinical trials for Inherited Retinal Degenerations

ABSTRACT

Background: Several novel treatments of inherited retinal degenerations have undergone phase I/IIa clinical trials with limited sample size, yet investigators must still determine if toxicity or an efficacy signal occurred or if the change was due to test–retest variability (TRV) of the measurement tool.

Materials and Methods: Synthetic datasets were used to compare three types of TRV estimators under different sample sizes, mean drift, skewness, and number of baseline measurements.

Results: Mixed effects models underestimated the standard deviation of measurement error (SDEM); the unbiased change score estimator method (UBS) was more accurate. The fixed effect model had less bias and smaller standard deviation than UBS if >2 baseline measurements. The change score estimator had no bias; other estimators introduced bias for lower variability. With sample size <10, all estimators had high variance. With sample size ≥10, the differences between methods were often minimal. The pooled estimator model did not capture drift, whereas a fixed effect regression or mixed effects models accounted for drift while maintaining an accurate measure of variance. With small sample sizes, the bootstrap estimates of SDEM were severe underestimates, while the jackknife estimates were mildly low but much better. The jackknife was more accurate for the unbiased change score method than for the pooled estimator.

Conclusions: The ideal phase I/IIa study has ≥20 subjects and uses UBS or its fixed effect model generalization if >2 baseline measurements. With non-ideal study parameters, investigators should at least quantify the error estimate present in their data analysis.

KEYWORDS:

• Test–retest variability
• clinical trials
• inherited retinal disease
• measurement error

Declaration of interest

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Supplemental data

Supplemental data for this article can be accessed on the publisher’s website.

Additional information

Funding

This research was supported by the National Institute of Health grant K23EY026985.