Using a Sharp Instrument to Parse Apart Strategy and Consistency: An Evaluation of PPT and Its Assumptions

ABSTRACT

Potential Performance Theory (PPT) is a general theory for parsing observed performance into the underlying strategy and the consistency with which it is used. Although empirical research has supported that PPT is useful, it is desirable to have more information about the bias and standard errors of PPT findings. It also is beneficial to know the effects of violations of PPT assumptions. The authors present computer simulations that evaluate bias and standard errors at varying levels of strategy, consistency, and number of trials per participant. The simulations show that, when the assumptions are true, there is very little bias and the standard errors are low when there are moderate or large numbers of trials per participant (e.g., N = 50 or N = 100). But when the independence assumption is violated, PPT provides biased findings, although the bias is quite small unless the violations are large.

Keywords:

• consistency
• Potential Performance Theory
• simulations
• strategy

Notes

1. There is the theoretically interesting case where strategy is below chance. In this case, decreasing randomness actually increases task performance. However, this unlikely possibility will be ignored hereafter.

2. We wish to be clear that this statement should not be interpreted as indicating approval of the null hypothesis significance testing procedure for detecting whether or not there is a difference between blocks. In fact, we have argued repeatedly against using the null hypothesis significance testing procedure to detect effects of all kinds (Trafimow, 2003, 2005; Trafimow & Rice, 2009b, 2009c).