The role of training, alternative models, and logical necessity in determining confidence in syllogistic reasoning

Abstract

Prior research shows that reasoners' confidence is poorly calibrated (Shynkaruk & Thompson, 2006). The goal of the current experiment was to increase calibration in syllogistic reasoning by training reasoners on (a) the concept of logical necessity and (b) the idea that more than one representation of the premises may be possible. Training improved accuracy and was also effective in remedying some systematic misunderstandings about the task: those in the training condition were better at estimating their overall performance than those who were untrained. However, training was less successful in helping reasoners to discriminate which items are most likely to cause them difficulties. In addition we explored other variables that may affect confidence and accuracy, such as the number of models required to represent the problem and whether or not the presented conclusion was necessitated by the premises, possible given the premises, or impossible given the premises. These variables had systematically different relationships to confidence and accuracy. Thus, we propose that confidence in reasoning judgements is analogous to confidence in memory retrievals, in that they are inferentially derived from cues that are not diagnostic in terms of accuracy.

Keywords:

• Confidence
• Training
• Metacognition
• Syllogistic reasoning

Acknowledgments

This research was supported by the Natural Sciences and Engineering Research Council of Canada. The authors would like to thank Jonathan St. B. T. Evans, Juan A. Garcia Madruga, Linden J. Ball, Jamie I. D. Campbell, Ron Borowsky, Jody Shynkaruk, Laura Aspenlieder, Nicole Robert, and three anonymous reviewers for the thoughtful comments they provided on earlier versions of this manuscript.

Notes

¹There was potentially some confusion arising from the fact that participants were instructed to answer “impossible” if a conclusion did not follow logically from the premises. However, in practice this did not appear to be a problem, as reasoners correctly chose the “impossible” category close to 100% of the time for conclusions that were Impossible. In addition, reasoners almost never identified Possible Strong conclusions as “impossible”, even though they technically fit the definition of not “following logically from the premises.” Thus, despite some ambiguity in the instructions, reasoners appeared to perform the task as intended.

²Because we were not entirely successful in counterbalancing figure across models, the models variable may need to be interpreted with some caution. However, given that the only difference between the conditions was an extra figure-1 single-model problem and one fewer figure-3 single-model problems, this seems unlikely. Nonetheless, in the analyses that follow, we will contrast the effects of model for each figure type, to make sure that the observed differences could not be explained by differences in figure.

³This pattern held for all figures.

⁴This pattern held for figures 4 (p = .002) and 1 (p = .065, two-tailed), but not figure 2 (p = .91).

⁵In addition to comprehensible diagrams, some participants had also taken notes, however in all cases these notes were just letters, scribbles, or arrows on the page. As our results did not change when these notes were included in the analyses, only correct model representations, in the form of Venn diagrams or their equivalent, were used in the final analyses.