Cumulative False Positive Rates Given Multiple Performance Validity Tests: Commentary on Davis and Millis (2014) and Larrabee (2014)

Abstract

Controversy has arisen over interpretation of performance validity tests (PVTs) when multiple PVTs are given. Some papers state that more stringent criteria are needed to judge overall performance as invalid, while others argue that concerns about the number of PVTs are overstated and that widely used criteria are appropriate. We examine theoretical models and assumptions, and analyze published data to determine the magnitude of effects implied by theory and observed in practice. Assertions advanced in the primary papers are examined for consistency with the empirical data. Existing theoretical models do not account well for the diverse empirical data, substantial empirical effects remain poorly understood, and the primary papers include assertions that are not empirically supported. The results indicate that: (a) neuropsychology lacks solid theoretical bases for estimating PVT failure rates given various combinations of PVTs, and thus needs to rely on empirical data; (b) existing empirical data fail to support the application of any uniform criteria across the broad range of scenarios involving multiple PVTs; and (c) practice should rely on empirical studies involving combinations of PVTs that have been studied together, in samples clearly appropriate to the individual case, using experimental designs germane to the questions under consideration.

Keywords:

• Performance validity
• Symptom validity
• Forensic
• Medico-legal
• Evidence-based.

This article is referred to by:

Reply to Commentary by Bilder, Sugar, and Helleman (2014 this issue) on Minimizing False Positive Error With Multiple Performance Validity Tests

Notes

¹ Unless otherwise specified, all scenarios presented here assume that the false positive rate for each individual PVT is 10% (or that “specificity” is 90%). Berthelson and colleagues noted that actual FPRs reported for individual tests in practice sometimes exceed 15%, which further increases estimates of the overall FPRs (see Berthelson et al., 2013, Table V).

² Davis and Millis make three different assertions about their findings, but it should be noted that testing the rate of “failing zero PVTs” is the formal complement to the test of rates of “failing one or more PVTs,” so Davis and Millis actually had only two testable hypotheses. Table 1 shows that none of their assertions is supported by their data.

³ It is beyond the scope of this paper to address multiple additional assumptions. For example, all the arguments here take for granted that the false positive rates are valid for each individual PVT.

⁴ The impact of examinee characteristics is seen clearly in the Davis and Millis (2014) negative binomial regression, which highlights how important education level is for estimated overall FPR. Table 2 shows that using 12 rather than 14 years of education shifts the FPR (using the two or more failed PVT criterion) from 7.1% up to 11.4% (if five PVTs are given), and from 15% to 22% (if eight PVTs are given). Using the same parameters in Table 2, but dropping education to only 8 years, an examinee would need to fail four of five PVTs to keep overall FPR < 10%.

Additional information

Funding

Funding. The preparation of this manuscript was supported by the Michael E. Tennenbaum Family Center for the Biology of Creativity and a grant from the National Institute of Mental Health (R01MH101478). We thank Erin Bigler, Lori Holt, Justin B. Miller, and Jeffrey A. Schaeffer for valuable comments on earlier versions of this manuscript.