Assessing fitness to drive—A validation study on patients with mild cognitive impairment

ABSTRACT

Objectives: There is no consensus yet on how to determine which patients with cognitive impairment are able to drive a car safely and which are not. Recently, a strategy was composed for the assessment of fitness to drive, consisting of clinical interviews, a neuropsychological assessment, and driving simulator rides, which was compared with the outcome of an expert evaluation of an on-road driving assessment. A selection of tests and parameters of the new approach revealed a predictive accuracy of 97.4% for the prediction of practical fitness to drive on an initial sample of patients with Alzheimer's dementia. The aim of the present study was to explore whether the selected variables would be equally predictive (i.e., valid) for a closely related group of patients; that is, patients with mild cognitive impairment (MCI).

Methods: Eighteen patients with mild cognitive impairment completed the proposed approach to the measurement of fitness to drive, including clinical interviews, a neuropsychological assessment, and driving simulator rides. The criterion fitness to drive was again assessed by means of an on-road driving evaluation. The predictive validity of the fitness to drive assessment strategy was evaluated by receiver operating characteristic (ROC) analyses.

Results: Twelve patients with MCI (66.7%) passed and 6 patients (33.3%) failed the on-road driving assessment. The previously proposed approach to the measurement of fitness to drive achieved an overall predictive accuracy of 94.4% in these patients. The application of an optimal cutoff resulted in a diagnostic accuracy of 100% sensitivity toward unfit to drive and 83.3% specificity toward fit to drive. Further analyses revealed that the neuropsychological assessment and the driving simulator rides produced rather stable prediction rates, whereas clinical interviews were not significantly predictive for practical fitness to drive in the MCI patient sample.

Conclusions: The selected measures of the previously proposed approach revealed adequate accuracy in identifying fitness to drive in patients with MCI. Furthermore, a combination of neuropsychological test performance and simulated driving behavior proved to be the most valid predictor of practical fitness to drive.

KEYWORDS:

• Fitness to drive
• mild cognitive impairment
• neuropsychological assessment
• driving simulator
• on-road test

Introduction

Cognitive impairment is a risk factor for unsafe driving (Devlin et al. 2012; Dubinsky et al. 2000; Frittelli et al. 2009; Wadley et al. 2009), but advising patients with cognitive impairment on fitness to drive is difficult due to the many factors that influence driving safely (Bacon et al. 2007). Several clinical assessment tools have been composed to evaluate fitness to drive in patients with cognitive impairment, but there is no consensus yet on how fitness to drive should be investigated in clinical practice (Carr and Ott 2010). It has been shown that predictive accuracies of the available methods for the assessment of fitness to drive differ greatly when compared to pass–fail decisions of on-road evaluators and often fail a successful replication on independent samples (Gifford 2013; Hoggarth et al. 2013; Innes et al. 2011; Mathias and Lucas 2009; Vrkljan et al. 2011). Several possible explanations for this variation in predictive accuracy of such approaches can be identified. First, a great heterogeneity of patients exists both within and between studies that becomes particularly obvious with respect to clinical characteristics such as etiologies, symptoms, and impairments. It can be argued that patients suffering from different types of impairments may require a different set of measures for the prediction of fitness to drive (Piersma, de Waard et al. 2016). Second, many studies that explored predictors of fitness to drive employed comprehensive assessment tools and test batteries, including numerous tests and test variables. For these reasons, many studies introduced a large number of candidate variables and explored the validity of these measures for the prediction of fitness to drive on a rather small, heterogeneous clinical sample. The identification of significant predictors among a large number of variables on small samples may result in a problem that is referred to as capitalization on chance (MacCallum et al. 1992), which describes the risk of a large part of the associations found between predictor variables and the outcome fitness to drive having occurred due to random error. Capitalization on chance is a relevant issue for the identification of predictors for fitness to drive, because prediction models tend to perform better on the data set on which the model was estimated (derivation set) than on a new data set (validation set). External validation of a prediction model on an independent data set is therefore a crucial step before clinical application can be suggested (Bleeker et al. 2003; Toll et al. 2008).

In a recent study, an assessment strategy was composed for the prediction of practical fitness to drive in patients with Alzheimer's dementia (AD), consisting of clinical interviews, a neuropsychological assessment, and driving simulator rides (Piersma, Fuermaier et al. 2016). The predictive accuracy of this approach was found to be 97.4% on a sample of 55 patients with AD. Before application in clinical practice can be recommended, the validity of such an assessment strategy needs to be evaluated on an independent sample of patients. The aim of this study was therefore to explore the validity of the assessment approach and to extent this approach to a closely related group of patients with cognitive impairment; that is, mild cognitive impairment (MCI).

Methods

Patients with mild cognitive impairment

Recruitment and assessment of patients with MCI was performed as part of the original study and followed the same study protocol as described by Piersma, Fuermaier and colleagues (2016). MCI was diagnosed by a neurologist, geriatrician, psychiatrist, or general practitioner. The diagnosis was established by following the criteria as described by Petersen (2004) and Albert et al. (2011), which includes (1) cognitive complaints of the patient indicating cognitive decline, usually corroborated by an informant; (2) objective evidence of cognitive impairment that cannot be explained by normal aging; (3) essentially preserved functional abilities; and (4) the absence of a diagnosis of dementia. There was no predefined set of measures that was consistently used in the diagnostic process of all patients in the present study. However, the diagnostic process of all patients was supported by the use of various diagnostic instruments that were mainly used in a qualitative fashion for diagnostic purposes. Data on formal neuropsychological testing were not available for all patients; therefore, a distinction between amnestic and nonamnestic MCI, as well as single or multiple domain impairment, cannot be made for all patients in the present sample. Recruitment and assessment were conducted at 5 locations in The Netherlands, resulting in the inclusion of 18 patients diagnosed with MCI who performed the complete fitness to drive assessment method as described in the previous study on patients with AD. Participants’ ages ranged from 49 to 79 years (mean = 67.5 years; SD = 8.6 years), including 2 females and 16 males. Inclusion criteria were a valid driver's licence, a diagnosis of MCI, a binocular visual acuity of at least 0.5, and a horizontal visual field of at least 120°. Exclusion criteria were neurological conditions unrelated to any etiology of dementia and usage of medications with known severe influence on driving. The study was approved by the Medical Ethical Committee of the University Medical Center Groningen, The Netherlands.

Fitness to drive assessment

The approach to the assessment of fitness to drive recently described (Piersma, Fuermaier et al. 2016), including clinical interviews, a neuropsychological assessment, and driving simulator rides, was administered to all patients. For the purpose of this study, only those measures entered analysis that included the final prediction equations as derived from the original study on the assessment of fitness to drive. Measures of clinical interviews included 2 subscores of the Clinical Dementia Rating (CDR; Morris 1993)—that is, Orientation and Judgment and Problem Solving—as well as additional information about the patients’ judgments about their own driving safety and recent driving experience. The neuropsychological assessment included the Mini-Mental State Examination (MMSE; Folstein et al. 1975; Kok and Verhey 2002), the Reaction Time S2 (Prieler 2008; Schuhfried 2013), the Hazard Perception Test (Vlakveld 2011), and a traffic theory test (for details, see Piersma, Fuermaier et al. 2016). For the driving simulation, fixed-based Jentig50 driving simulators of ST Software (Groningen, the Netherlands) were used. Driving simulator measures included the minimal speed when approaching an intersection with traffic lights, the number of collisions in a ride with intersections, and 2 measures regarding a merging maneuver; that is, the deceleration of the car behind right after merging and the time headway to the car in front directly after merging (for details of simulator rides, see Piersma, Fuermaier et al. 2016).

On-road driving assessment

The on-road driving assessments were performed by approved experts on practical fitness to drive of the CBR Dutch driving test organization. Experts were blind to the patients’ diagnoses and fitness to drive assessment results. The experts made use of the Test Ride Investigating Practical fitness to drive (Tant et al. 2002; Withaar 2000). Finally, one overall score was determined by the expert, resulting in a pass, doubtful, or fail outcome. The overall score was recoded into a dichotomous item; that is, pass outcomes indicating fitness to drive and doubtful or fail outcomes indicating unfit to drive (criterion).

Statistical analysis

Receiver operating characteristic (ROC) analyses were performed to evaluate the predictive accuracies of the obtained measures in determining individuals’ practical fitness to drive. ROC analyses were performed on the basis of clinical interviews, neuropsychological test performances, driving simulator rides, as well as the complete assessment approach including all 3 types of predictors. The area under the curve (AUC) was used as a classification measure with larger areas indicating better predictive accuracy. The predictive accuracy of the complete approach was further evaluated by calculating classification statistics; that is, sensitivity, specificity, positive predictive power, and negative predictive power.

Results

Of the 18 patients with MCI, 12 patients (66.7%) passed the on-road assessment and were regarded fit to drive, and 6 patients (33.3%) with doubtful or fail outcomes were regarded unfit to drive. Of the 12 patients who were regarded fit to drive, 5 patients had no restrictions placed on them, 2 patients were restricted to driving with automatic transmission, and 5 patients had a restriction on the duration that the driver's license was valid, ranging between 1 and 5 years (depending on the severity of impairments found in neuropsychological tests). Descriptive results of patients with MCI who passed and failed the on-road assessment are presented in Table 1.

Table 1. Descriptive results of patients with MCI who pass and patients with MCI who fail the on-road assessment on selected measures of clinical interviews, neuropsychological assessment, and driving simulator rides (M ± SD).

Predictors^a	Pass (n = 12)	Fail (n = 6)
Clinical interviews^b
Clinical Dementia Rating
CDR Orientation	0.333 ± 0.389	0.500 ± 0.316
CDR Judgment and Problem Solving	0.417 ± 0.359	0.417 ± 0.376
Anamnesis
Judgment driving safety^c	1.25 ± 0.45	1.33 ± 0.52
Driving questionnaire
Recent driving experience^d	2.75 ± 1.42	3.33 ± 1.63
Neuropsychological assessment^e
MMSE
Total score	27.25 ± 2.90	26.67 ± 1.86
RT S2
Reaction time (ms)	262.75 ± 53.58	270.50 ± 54.16
Hazard perception
Number of correct trials	16.50 ± 3.34	10.83 ± 4.49
Traffic theory
Response time (s)	6.22 ± 1.16	7.45 ± 1.13
Driving simulator rides^f
Intersection ride
Minimum speed intersection 2 (km/h)^g	5.27 ± 9.53	5.04 ± 12.21
Number of collisions	0.42 ± 0.67	1.00 ± 0.89
Merging ride
Deceleration rear car (km/h)	−0.03 ± 0.09	−2.11 ± 3.26
Time headway (s)	0.91 ± 0.39	1.80 ± 0.64

^aPrediction equation for fitness to drive (complete approach) = Clinical interviews ^* 0.328 + Neuropsychological assessment ^* −0.620 + Driving simulator rides ^* 0.483.

^bPrediction equation for fitness to drive (clinical interview) = CDR Orientation ^* 0.675 + CDR Judgment and Problem Solving ^* 1.036 + Judgment driving safety ^* 1.250 + Recent driving experience ^* −0.576.

^cJudgment about driving safety whether participant is (1) still driving as safely as when the participant was middle-aged, (2) driving less safely compared to when the participant was middle-aged, or (3) drives unsafely.

^dKilometres driven in the previous 12 months: (1) less than 1,000 km, (2) 1,000–5,000 km, (3) 5,000–10,000 km, (4) 10,000–20,000 km, (5) 20,000–30,000 km, (6) 30,000–50,000, (7) more than 50,000 km.

^ePrediction equation for fitness to drive (neuropsychological assessment) = MMSE ^* 0.129 + RT S2 RT ^* −0.003 + Correct trials of Hazard Perception ^* 0.206 + Response time of traffic theory ^* −0.310.

^fPrediction equation for fitness to drive (driving simulator rides) = Minimum speed intersection 2 ^* 0.021 + Number of collisions ^* 0.738 + Deceleration rear car ^* −0.367 + Time headway ^* 0.732.

^gIntersection with need to give right of way; the traffic lights at this intersection turn yellow and subsequently red.

Prediction equations of fitness to drive as described in the previous study on patients with AD were calculated for all types of assessments applied (Piersma, Fuermaier et al. 2016). The coefficients that determine the prediction equations are presented in the footnote in Table 1. Based on these prediction equations, ROC analysis revealed that the clinical interview was not useful for the prediction of practical fitness to drive in the present patient sample, because it was shown by a nonsignificant AUC value close to chance level (AUC = 0.528, SE = 0.151, P =.851). In contrast, ROC analyses demonstrated good predictive accuracies of results from the neuropsychological assessment (AUC = 0.819, SE = 0.102, P =.031) and driving simulator rides (AUC = 0.861, SE = 0.089, P =.015), with significant predictive accuracies of greater than 80%. ROC curves depicting graphical plots of sensitivity versus 1 − specificity of results derived from all 3 types of assessments are presented in Figure 1.

Figure 1. ROC curves presenting diagnostic accuracies of clinical interviews, neuropsychological assessment, and driving simulator rides for the prediction of fitness to drive.

The complete approach combining all 3 types of information (interviews, neuropsychological assessments, and driving simulator rides) achieved a high predictive accuracy of detecting patients who were unfit to drive (AUC = 0.944, SE = 0.052, P =.003), which was close to the predictive accuracy of the approach as derived from the data set of patients with AD (97.4%; Piersma, Fuermaier et al. 2016). Applying previously proposed cutoffs of the final predictor variable (−0.6 and −0.8) on the present sample of patients with MCI, 14 out of 18 patients were classified correctly, with 2 patients incorrectly classified as failing (false negatives) and 2 patients incorrectly classified as passing (false positives). When adapting the cutoff to −1.2, 16 out of 18 patients were classified correctly, with 2 false negatives but no false positives (Table 2).

Table 2. Classification accuracy of the final predictor variable (including clinical interviews, neuropsychological assessment, and driving simulator rides) in detecting patients being unfit to drive (n = 6) relative to patients being fit to drive (n = 12).

Cutoff	Sensitivity	Specificity	Positive predictive value	Negative predictive value
−0.6	66.7	83.3	66.7	83.3
−0.8	66.7	83.3	66.7	83.3
−1.2	100.0	83.3	75.0	100.0

Discussion

Of the 18 patients with MCI who participated in this study, 12 patients passed and 6 patients failed the on-road assessment. This rate of one third of the patients failing the on-road assessment supports the notion that MCI can be a threat to safe driving (Devlin et al. 2012; Frittelli et al. 2009; Wadley et al. 2009). At the same time, results show that for two third of the patients there is no need to stop driving, and they can retain mobility.

Due to the risk of overestimating predictive validity when interpreting classification statistics on the derivation set (also referred to as capitalization on chance; Bleeker et al. 2003; MacCallum et al. 1992; Toll et al. 2008), the purpose of the present study was to evaluate predictive accuracy of the recently proposed approach to the assessment of practical fitness to drive (Piersma, Fuermaier et al. 2016) on a closely related group of patients with cognitive impairment; that is, MCI. High overall predictive accuracy of the proposed approach combining clinical interviews, a neuropsychological assessment, and driving simulator rides was achieved on the present sample of patients with MCI (AUC = 94.4%), which is almost similar to what has been found in the original study on patients with AD (AUC = 97.4%). When applying cutoff values of the final predictor variable as proposed in the study on patients with AD, only mediocre classification accuracies for identifying practical fitness to drive in patients with MCI were reached; that is, 67.7% sensitivity and 83.3% specificity. More adequate diagnostic accuracies were achieved when applying a stricter cutoff (−1.2), yielding 100% sensitivity and 83.3% specificity.

In^{Table 2} correspondence with findings in patients with AD (Piersma, Fuermaier et al. 2016), results of the neuropsychological assessment and driving simulator rides were also found to be valid predictors for practical fitness to drive in patients with MCI. However, contrary to the findings in patients with AD, results of the clinical interview were not found to significantly predict practical fitness to drive in patients with MCI. This lack of predictive validity on the validation set in contrast to predictive validity that has been found on the derivation set might be explained by capitalization on chance (MacCallum et al. 1992), indicating that a considerable proportion of shared variance between predictors and the criterion practical fitness to drive in the derivation set may have occurred by random error. Another explanation might be that the patients with MCI might not be so much aware of their actual deficits because their deficits are mild in nature, potentially leading them to under- or overestimate their deficits. Thus, one may conclude that clinical interviews do not represent valid measures for the prediction of fitness to drive in patients with MCI. However, as an alternative explanation, one may argue that the selected measures of clinical interviews in fact present valid predictors for fitness to drive in AD but not in other types of dementia or milder cognitive impairments. Patients with MCI may or may not develop dementia and, if so, they may develop AD or another type of dementia. This usually remains uncertain until a later stage of their disease (Jungwirth et al. 2012). Patients with MCI therefore represent a more heterogeneous group than patients with AD. For example, problems with orientation (as reflected by CDR Orientation) are very common in AD (Tu et al. 2015) and may therefore be predictive for fitness to drive in samples with AD, whereas these problems may be rare in patients with MCI, resulting in a lack of predictive accuracy in patients with MCI.

As a limitation of this study, it must be noted that some but not all patients with MCI in the present sample may have a subclinical AD-like pathology. Thus, the majority of patients used for the validation of the prediction strategy (patients with MCI) might be characterized by a different etiology than the sample of patients used for the derivation (patients with AD). Before clinical application can be recommended, further validation on another sample of patients with AD is advisable to determine the predictive accuracy of the different types of assessments, in particular measures of clinical interviews that have not been shown to be useful in the present study. The methodology applied in the current study should also be employed in studies on fitness to drive in patients with other types of dementia than AD, because other predictor variables may play a role in the prediction of fitness to drive in patients with other types of dementia (Piersma, de Waard et al. 2016).

It is important to note that validation of the prediction strategy and the extension to patients with MCI as described in the present study is based on a relatively small sample of 18 patients with MCI. A comparison of the different types of measures between patients passing and failing the on-road test as presented in Table 1 must therefore be interpreted with caution. Thus, further validation studies on patients with AD and other types of dementia should preferably be based on larger samples, in order to derive more valid conclusions on the utility of the proposed approach to the practical assessment of fitness to drive. Further studies on patients with MCI should also consider distinguishing between different subtypes of MCI, including amnestic and nonamnestic MCI, as well as single or multiple domain impairment, in order to explore their differential effects on fitness to drive.

In conclusion, the proposed approach to the assessment of fitness to drive revealed adequate accuracy in predicting practical fitness to drive of a group of patients closely related to the original group of patients with AD; that is, patients with MCI. Though a combination of a neuropsychological assessment and driving simulator rides may yield most valid prediction of fitness to drive in patients with cognitive impairment, it cannot be determined on the basis of the present data whether the selected measures of clinical interviews are valid predictors for fitness to drive.

Funding

This work was funded by the Ministry of Infrastructure and the Environment (NL). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the article.