Interrater and Test–Retest Reliability of a Fixed Condition Design Fluency Test

Abstract

Despite its potential as a unique neuropsychological test, the emergence of a psychometrically sound research foundation for Jones-Gotman and Milner's (1977) Design Fluency Test (DFT) has been constrained by the lack of consistent administration and scoring practices and limited information about its reliability. Here we describe an approach to administering and scoring the fixed condition DFT that is modeled on Jones-Gotman and Milner's original method and that clarifies procedural ambiguities. Results include interrater and long-term test-retest reliability analyses using this approach. First, based on five raters who scored 50 DFT protocols, good to excellent intra-class correlation coefficients were obtained for all DFT scores. Second, in a broadly representative sample of 87 healthy adults who were tested twice over an average of 5½ years, the test–retest reliabilities for total and novel design scores ranged from good to excellent. This study demonstrates that the fixed condition DFT can be scored reliably using these procedures and that the reliability coefficients for DFT total and novel designs scores are comparable to those of other commonly used neuropsychological tests.

ACKNOWLEDGEMENT

This study was supported by NIH grants MH60504 and MH43775.

Notes

Note: * = Significant difference between two or more raters. Superscript denotes rater(s) that significantly differ after Bonferroni correction, p < .05, ^a  = rater 1, ^b  = rater 2, ^c  = rater 3, ^d  = rater 4.

Note. SD = standard deviation; r = Pearson correlation coefficient; ICC = intraclass correlation coefficient. Sum rule breaks = nameable designs + line violations; Sum perseverative designs = exact copy/rotation + minimal variations; Sum unacceptable designs = sum rule breaks±sum perseverative designs.

* = p ≤ .05; ** = p ≤ .01; *** = p ≤ .001.

Note. Sdiff = standard error of the difference based on the formula Sdiff = √SEM²1 + SEM²2; SEM = SD√1 − r₁₂; 80% CI = 80% confidence interval (Sdiff*1.28); 90% CI = 90% confidence interval (Sdiff*1.64); NS = no significant predictors entered the regression equation.

A preliminary version of these data was presented at the Annual Meeting of the International Neuropsychological Society in February 2005 in St. Louis, MO.