Do complex models increase prediction of complex behaviours? Predicting driving ability in people with brain disorders

Abstract

Prediction of complex behavioural tasks via relatively simple modelling techniques, such as logistic regression and discriminant analysis, often has limited success. We hypothesized that to more accurately model complex behaviour, more complex models, such as kernel-based methods, would be needed. To test this hypothesis, we assessed the value of six modelling approaches for predicting driving ability based on performance on computerized sensory–motor and cognitive tests (SMCTests™) in 501 people with brain disorders. The models included three models previously used to predict driving ability (discriminant analysis, DA; binary logistic regression, BLR; and nonlinear causal resource analysis, NCRA) and three kernel methods (support vector machine, SVM; product kernel density, PK; and kernel product density, KP). At the classification level, two kernel methods were substantially more accurate at classifying on-road pass or fail (SVM 99.6%, PK 99.8%) than the other models (DA 76%, BLR 78%, NCRA 74%, KP 81%). However, accuracy decreased substantially for all of the kernel models when cross-validation techniques were used to estimate prediction of on-road pass or fail in an independent referral group (SVM 73–76%, PK 72–73%, KP 71–72%) but decreased only slightly for DA (74–75%) and BLR (75–76%). Cross-validation of NCRA was not possible. In conclusion, while kernel-based models are successful at modelling complex data at a classification level, this is likely to be due to overfitting of the data, which does not lead to an improvement in accuracy in independent data over and above the accuracy of other less complex modelling techniques.

Keywords:

• Human performance modelling
• Driving
• Brain disorders
• Prediction
• Cross-validation

Acknowledgments

This work was supported through a Canterbury Medical Research Foundation fellowship and a Road Safety Trust grant. We would like to thank the occupational therapists who undertook the off- and on-road assessments for this study, Julie Severinsen and Jane Gray (Driving and Vehicle Assessment Service, Burwood Hospital, Christchurch), Kevin O'Leary and colleagues (Kevin O'Leary & Associates, Wellington), and Jenny Oxley, Barbara Brook, and colleagues (Organisation of Therapy and Rehabilitation Services, Hamilton).