Assessing the validity of multinomial models using extraneous variables: An application to prospective memory

Abstract

The class of multinomial processing tree (MPT) models has been used extensively in cognitive psychology to model latent cognitive processes. Critical for the usefulness of a MPT model is its psychological validity. Generally, the validity of a MPT model is demonstrated by showing that its parameters are selectively and predictably affected by theoretically meaningful experimental manipulations. Another approach is to test the convergent validity of the model parameters and other extraneous measures intended to measure the same cognitive processes. Here, we advance the concept of construct validity (Cronbach & Meehl, 1955) as a criterion for model validity in MPT modelling and show how this approach can be fruitfully utilized using the example of a MPT model of event-based prospective memory. For that purpose, we investigated the convergent validity of the model parameters and established extraneous measures of prospective memory processes over a range of experimental settings, and we found a lack of convergent validity between the two indices. On a conceptual level, these results illustrate the importance of testing convergent validity. Additionally, they have implications for prospective memory research, because they demonstrate that the MPT model of event-based prospective memory is not able to differentiate between different processes contributing to prospective memory performance.

Keywords:

• Multinomial model
• Construct validity
• Prospective memory

Acknowledgments

This research was supported by a grant from the Deutsche Forschungsgemeinschaft (DFG) to the third author. We thank Rebekah E. Smith for sharing her experimental software of the colour-matching task with us.

Notes

¹ Restrictions on parameters in MPT modelling are often necessary to render a model identifiable. However, restrictions should be theoretically motivated and empirically testable. Therefore, and for comparability reasons, we employed the same restrictions as those of Smith and Bayen (2004) and Horn et al. (2011).

² Smith and Bayen (2004) have argued that reaction times (RTs) on trials following a PM trial are likely to be affected by the processing of the PM cue, and thus longer RTs on these trials might not reflect resource-demanding processes. Therefore, they excluded the four trials following each PM trial. The same exclusion policy resulted in an exclusion of 48% of all trials in the present experiment. However, analyses excluding the PM trials and the first trial following each PM trial, which resulted in exclusion of 17% of all trials, produced the same pattern of results.

³ In all groups, half of the trials of the ongoing task included a red rectangle, but in the nondemanding PM group, the colour red had a special meaning (i.e., it corresponded to the colour of the PM cue). In this group, the red rectangle could have served as a reminder of the PM task. Previous research has shown that reminders can result in a higher level of resource-demanding processing in terms of increased costs to the ongoing task (Scullin, McDaniel, & Einstein, 2010). We therefore compared RTs between those trials with a red rectangle and those without to control for cost differences. However, RTs did not differ between those two trial types, t < 1, and did not interact with groups, F < 1. Importantly, the contrast analysis yielded the same pattern of results when analysing trials with and without red rectangles separately.

⁴ On the suggestion of a reviewer, we also analysed the RTs in a 3 (group: demanding PM vs. nondemanding PM vs. control) × 2 (block: baseline vs. second) ANOVA. This analysis revealed no main effect of block, F < 1, η² = .006, but a significant main effect of group, F(2, 88) = 8.45, p < .001, η² = .162. This main effect was qualified by a significant interaction between group and block, F(2, 88) = 18.09, p < .001, η² = .291. To analyse this interaction further, we compared all possible pairs of groups and found a significant interaction between the demanding PM group and the control group, F(1, 58) = 21.99, p < .001, η² = .275, but not between the nondemanding group and the control group, F(1, 59) = 1.53, p = .220, η² = .025. Importantly, we also found a significant interaction between the demanding and the nondemanding PM group, F(1, 59) = 17.58, p < .001, η² = .230, mirroring the significant differences in costs in these two groups, which was also found with the RT-difference-score analyses. Thus, the 3 × 2 ANOVA led to the same conclusions as the analysis of RT-difference scores.

⁵ The 95% confidence intervals of the P parameter estimates did not overlap with zero for either the demanding PM group [CI: .42, .58] or the nondemanding PM group [CI: .83, .93].