Abstract
Purpose: This study has two objectives. The first one is to investigate the question whether it is possible to discriminate between eyes with and without a glaucomateous visual field defect based on standard ophthalmologic examinations as well as optic nerve head topographic parameters. The second objective raises the question of the ability of several suggested statistical models to generalize their results to new, previously unseen patients.
Methods: To investigate the above addressed question: (a) independent, two-sided t-tests, (b) a linear discriminant analysis with a forward stepwise variable selection algorithm, (c) four classification tree analyses and (d) three different neural network models with a forward, backward and a genetic variable selection algorithm were applied to 1020 subjects with a normal visual field and 110 subjects with a glaucomateous visual field defect. The Humphrey Visual Field Analyzer was used to test the visual fields and the TopSS® Scanning Laser Tomograph measured the optic nerve topography. A 10-fold cross-validation method was used for the models (b), (c) and (d) to compute approximative 95% confidence intervals for the specificity and sensitivity rates.
A literature study of 18 studies dealt with the question whether/how the generalization error was controlled (control of sample bias, cross-validation procedures, training net size for valid generalization). It was also looked up whether point estimators or 95% confidence intervals were used to report specificity and sensitivity rates.
Results: (a) Only few differences of the means could be found between both groups, including age, existing eye diseases, best corrected visual acuity and visual field parameters. The linear discriminant analysis (b), the classification tree analyses (c) and the neural networks (d) ended up with high specificity rates, but low sensitivity rates.
The literature study showed that three models did not apply a cross-validation procedure to report their results. Two models used a test sample cross-validation and thirteen used a v-fold cross-validation method. Although most authors reported confidence intervals for the area under the ROC, no author reported confidence intervals for the true, but unknown sensitivity and specificity rates.
Conclusions: (a) The results of this study suggest that the combination of standard ophthalmologic eye parameters and optic nerve head topographic parameters of the TopSS® instrument are not sufficient to discriminate properly among normal eyes and eyes with a glaucomateous visual field defect. (b) We follow important suggestions given in statistical learning theory for proper generalization and suggest to apply these methods to the given models or to models in future. At least three conditions should be met: (1) confidence intervals instead of point estimators to assess the classification performance of a model (control of test sample bias); (2) sensitivity and specificity rates should be estimated instead of reporting a point estimator for the area under the ROC and (3) the generalization error should be reported both in a training and a test sample and methods should be applied to select an appropriate training sample size for valid generalization.
Acknowledgements
The authors wish to cordially thank the permanently and extremely ambitious co-workers at the “Glaukom-Vorsorgeambulanz” of the Landesaugenklinik Salzburg, Mrs Anna Konitsch and Mrs Anneliese Prieler and the many other clinical coworkers over the time of this study. We would also like to thank the Health Department of the Government of the County of Salzburg (Director: LH-Stv. Mag. Gabi Burgstaller), the ‘Fond Gesundes Österreich’ and the “Hauptverband der Sozialversicherungsträger” of Austria for their generous support.
Notes
‡Private practice in Salzburg, Austria.
An analysis in the Salzburg-Moorfields-Collaborative-Glaucoma-Study using visual field and optic nerve head topographic parameters.