The cognitive reflection test revisited: exploring the ways individuals solve the test

References

• Aczel, B., Bago, B., Szollosi, A., Foldes, A., & Lukacs, B. (2015). Measuring individual differences in decision biases: Methodological considerations. Frontiers in Psychology, 6, 1770.
   PubMed Web of Science ®Google Scholar
• Baron, J. (1993). Why teach thinking? An essay. Applied Psychology, 42(3), 191–214.
   Web of Science ®Google Scholar
• Baron, J., Badgio, P., & Gaskins, I. W. (1986). Cognitive style and its improvement: A normative approach. Advances in the Psychology of Human Intelligence, 3, 173–220.
   Google Scholar
• Baron, J., Scott, S., Fincher, K., & Metz, S. E. (2014). Why does the cognitive reflection test (sometimes) predict utilitarian moral judgment (and other things)? Journal of Applied Research in Memory and Cognition, 4, 265–284
   Web of Science ®Google Scholar
• Barton, S. B., & Sanford, A. J. (1993). A case study of anomaly detection: Shallow semantic processing and cohesion establishment. Memory & Cognition, 21(4), 477–487.
   PubMed Web of Science ®Google Scholar
• Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Lme4: Linear mixed-effects models using Eigen and S4. R Package, version 1.1-8. Retrieved from https://cran.r-project.org/web/packages/lme4/index.html
   Google Scholar
• Böckenholt, U. (2012). The cognitive-miser response model: Testing for intuitive and deliberate reasoning. Psychometrika, 77(2), 388–399.
   Web of Science ®Google Scholar
• Brandstätter, E., & Gussmack, M. (2013). The cognitive processes underlying risky choice. Journal of Behavioral Decision Making, 26(2), 185–197.
   Web of Science ®Google Scholar
• Brosnan, M., Hollinworth, M., Antoniadou, K., & Lewton, M. (2014). Is empathizing intuitive and systemizing deliberative? Personality and Individual Differences, 66, 39–43.
   Web of Science ®Google Scholar
• Campitelli, G., & Gerrans, P. (2014). Does the cognitive reflection test measure cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42(3), 434–447.
   PubMed Web of Science ®Google Scholar
• Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with judgments and choices. Judgment and Decision Making, 5(3), 182–191.
   Web of Science ®Google Scholar
• Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012). Measuring risk literacy: The Berlin numeracy test. Judgment and Decision Making, 7(1), 25–47.
   Web of Science ®Google Scholar
• Cokely, E. T., & Kelley, C. M. (2009). Cognitive abilities and superior decision making under risk: A protocol analysis and process model evaluation. Judgment and Decision Making, 4(1), 20–33.
   Web of Science ®Google Scholar
• Cokely, E. T., Parpart, P., & Schooler, L. J. (2009). On the link between cognitive control and heuristic processes. In N. A. Taatgnen & H. Van Rijn (Eds.), Proceedings of the 31th annual conference of the cognitive science society (pp. 2926–2931). Austin, TX: Cognitive Science Society.
   Google Scholar
• Del Missier, F., Mäntylä, T., & Bruin, W. B. (2012). Decision‐making competence, executive functioning, and general cognitive abilities. Journal of Behavioral Decision Making, 25(4), 331–351.
   Web of Science ®Google Scholar
• De Neys, W., & Bonnefon, J.-F. (2013). The “whys” and “whens” of individual differences in thinking biases. Trends in Cognitive Sciences, 17(4), 172–178.
   PubMed Web of Science ®Google Scholar
• De Neys, W., & Glumicic, T. (2008). Conflict monitoring in dual process theories of thinking. Cognition, 106(3), 1248–1299.
   PubMed Web of Science ®Google Scholar
• De Neys, W., Moyens, E., & Vansteenwegen, D. (2010). Feeling we're biased: Autonomic arousal and reasoning conflict. Cognitive, Affective, & Behavioral Neuroscience, 10(2), 208–216.
   PubMed Web of Science ®Google Scholar
• Dienes, Z. (2008). Understanding psychology as a science: An introduction to scientific and statistical inference. New York, NY: Palgrave Macmillan.
   Google Scholar
• Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290.
   Web of Science ®Google Scholar
• Dienes, Z. (2014). Using Bayes to get the most out of non-significant results. Frontiers in psychology, 5, 781.
   PubMed Web of Science ®Google Scholar
• Elqayam, S., & Evans, J. S. B. (2011). Subtracting “ought” from “is”: Descriptivism versus normativism in the study of human thinking. Behavioral and Brain Sciences, 34(5), 233–248.
   PubMed Web of Science ®Google Scholar
• Erickson, T. D., & Mattson, M. E. (1981). From words to meaning: A semantic illusion. Journal of Verbal Learning and Verbal Behavior, 20(5), 540–551.
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.
   Web of Science ®Google Scholar
• Evans, J. S. B. (1996). Deciding before you think: Relevance and reasoning in the selection task. British Journal of Psychology, 87(2), 223–240.
   Google Scholar
• Evans, J. S. B. (2003). In two minds: Dual-process accounts of reasoning. Trends in Cognitive Sciences, 7(10), 454–459.
   PubMed Web of Science ®Google Scholar
• Evans, J. S. B. (2009). How many dual-process theories do we need? One, two, or many? In In two minds: Dual processes and beyond (pp. 33–54). New York, NY: Oxford University Press.
   Google Scholar
• Evans, J. S. B., & Ball, L. J. (2010). Do people reason on the Wason selection task: A new look at the data of Ball et al. (2003). Quarterly Journal of Experimental Psychology, 63(3), 434–441.
   PubMed Web of Science ®Google Scholar
• Evans, J. S. B., Barston, J. L., & Pollard, P. (1983). On the conflict between logic and belief in syllogistic reasoning. Memory & Cognition, 11(3), 295–306.
   PubMed Web of Science ®Google Scholar
• Evans, J. S. B., & Stanovich, K. E. (2013). Dual-process theories of higher cognition advancing the debate. Perspectives on Psychological Science, 8(3), 223–241.
   Web of Science ®Google Scholar
• Fific, M. (2014). Double jeopardy in inferring cognitive processes. Frontiers in Psychology, 5, 1130.
   PubMed Web of Science ®Google Scholar
• Finucane, M. L., & Gullion, C. M. (2010). Developing a tool for measuring the decision-making competence of older adults. Psychology and Aging, 25(2), 271–288.
   PubMed Web of Science ®Google Scholar
• Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316–344.
   PubMed Web of Science ®Google Scholar
• Frederick, S. (2005). Cognitive reflection and decision making. The Journal of Economic Perspectives, 19(4), 25–42.
   Web of Science ®Google Scholar
• Gervais, W. M., & Norenzayan, A. (2012). Analytic thinking promotes religious disbelief. Science, 336(6080), 493–496.
   PubMed Web of Science ®Google Scholar
• Ghazal, S., Cokely, E. T., & Garcia-Retamero, R. (2014). Predicting biases in very highly educated samples: Numeracy and metacognition. Judgment and Decision Making, 9(1), 15–34.
   Web of Science ®Google Scholar
• Haran, U., Ritov, I., & Mellers, B. A. (2013). The role of actively open-minded thinking in information acquisition, accuracy, and calibration. Judgment and Decision Making, 8(3), 188–201.
   Web of Science ®Google Scholar
• Jacoby, L. L., Kelley, C. M., & McElree, B. D. (1999). The role of cognitive control: Early selection versus late correction. In S. Chaiken & Y. Trope (Eds.), Dual-process theories in social psychology (pp. 383–400). New York, NY: Guilford.
   Google Scholar
• Jacoby, L. L., Shimizu, Y., Daniels, K. A., & Rhodes, M. G. (2005). Modes of cognitive control in recognition and source memory: Depth of retrieval. Psychonomic Bulletin & Review, 12(5), 852–857.
   PubMed Web of Science ®Google Scholar
• Jeffreys, H. (1961). The theory of probability. Oxford: Oxford University Press.
   Google Scholar
• Johnson, E. D., Tubau, E., & De Neys, W. (2016). The doubting system 1: Evidence for automatic substitution sensitivity. Acta Psychologica, 164, 56–64.
   PubMed Web of Science ®Google Scholar
• Kagan, J., Rosman, B. L., Day, D., Albert, J., & Phillips, W. (1964). Information processing in the child: Significance of analytic and reflective attitudes. Psychological Monographs: General and Applied, 78(1), 1–37.
   Google Scholar
• Kahneman, D. (2011). Thinking, fast and slow. New York, NY: Farrar, Straus, and Giroux.
   Google Scholar
• Låg, T., Bauger, L., Lindberg, M., & Friborg, O. (2014). The role of numeracy and intelligence in health‐risk estimation and medical data interpretation. Journal of Behavioral Decision Making, 27(2), 95–108.
   Web of Science ®Google Scholar
• Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012). Individual differences in numeracy and cognitive reflection, with implications for biases and fallacies in probability judgment. Journal of Behavioral Decision Making, 25(4), 361–381.
   PubMed Web of Science ®Google Scholar
• Mata, A., Ferreira, M. B., & Sherman, S. J. (2013). The metacognitive advantage of deliberative thinkers: A dual-process perspective on overconfidence. Journal of Personality and Social Psychology, 105(3), 353–355.
   PubMed Web of Science ®Google Scholar
• Mata, A., Schubert, A.-L., & Ferreira, M. B. (2014). The role of language comprehension in reasoning: How “good-enough” representations induce biases. Cognition, 133(2), 457–463.
   PubMed Web of Science ®Google Scholar
• Meyer, A., Spunt, R., & Frederick, S. (2015). The bat and ball problem. Unpublished manuscript.
   Google Scholar
• Pachur, T., & Spaar, M. (2015). Domain-specific preferences for intuition and deliberation in decision making. Journal of Applied Research in Memory and Cognition, 4(3), 303–311.
   Web of Science ®Google Scholar
• Pacini, R., & Epstein, S. (1999). The relation of rational and experiential information processing styles to personality, basic beliefs, and the ratio-bias phenomenon. Journal of Personality and Social Psychology, 76(6), 972–987.
   PubMed Web of Science ®Google Scholar
• Paulhus, D. L. (1991). Measurement and control of response bias. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). San Diego, CA: Academic Press.
   Google Scholar
• Paxton, J. M., Ungar, L., & Greene, J. D. (2012). Reflection and reasoning in moral judgment. Cognitive Science, 36(1), 163–177.
   PubMed Web of Science ®Google Scholar
• Pennycook, G., Cheyne, J. A., Koehler, D. J., & Fugelsang, J. A. (2015). Is the cognitive reflection test a measure of both reflection and intuition? Behavior Research Methods, 48(1), 341–348.
   Web of Science ®Google Scholar
• Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (2012). Analytic cognitive style predicts religious and paranormal belief. Cognition, 123(3), 335–346.
   PubMed Web of Science ®Google Scholar
• Pennycook, G., Fugelsang, J. A., & Koehler, D. J. (2015). Everyday consequences of analytic thinking. Current Directions in Psychological Science, 24(6), 425–432.
   Web of Science ®Google Scholar
• Pennycook, G., & Ross, M. R. (2016). Commentary: Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 7, 9. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4722428/
   PubMed Web of Science ®Google Scholar
• Perkins, D. (1995). Outsmarting IQ: The emerging science of learnable intelligence. New York, NY: Free Press.
   Google Scholar
• Peters, E. (2012). Beyond comprehension the role of numeracy in judgments and decisions. Current Directions in Psychological Science, 21(1), 31–35.
   Web of Science ®Google Scholar
• Peters, E., Dieckmann, N., Dixon, A., Hibbard, J. H., & Mertz, C. K. (2007). Less is more in presenting quality information to consumers. Medical Care Research and Review, 64(2), 169–190.
   PubMed Web of Science ®Google Scholar
• Piazza, J., & Sousa, P. (2013). Religiosity, political orientation, and consequentialist moral thinking. Social Psychological and Personality Science, 5(3), 334–342.
   Web of Science ®Google Scholar
• Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J. (2015). The development and testing of a new version of the cognitive reflection test applying item response theory (IRT). Journal of Behavioral Decision Making, 29. doi:10.1002/bdm.1883
   Web of Science ®Google Scholar
• Reisen, N., Hoffrage, U., & Mast, F. W. (2008). Identifying decision strategies in a consumer choice situation. Judgment and Decision Making, 3(8), 641–658.
   Web of Science ®Google Scholar
• Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F. (2009). How numeracy influences risk comprehension and medical decision making. Psychological Bulletin, 135(6), 943–973.
   PubMed Web of Science ®Google Scholar
• Simon, H. A. (1956). Rational choice and the structure of the environment. Psychological Review, 63(2), 129–138.
   PubMed Web of Science ®Google Scholar
• Sinayev, A., & Peters, E. (2015). Cognitive reflection vs. calculation in decision making. Frontiers in Psychology, 6, 532. doi:10.3389/fpsyg.2015.00532
   PubMed Web of Science ®Google Scholar
• Stanovich, K. E., Toplak, M. E., & West, R. F. (2008). The development of rational thought: A taxonomy of heuristics and biases. Advances in Child Development and Behavior, 36, 251–285.
   PubMed Web of Science ®Google Scholar
• Stupple, E. J., Ball, L. J., & Ellis, D. (2013). Matching bias in syllogistic reasoning: Evidence for a dual-process account from response times and confidence ratings. Thinking & Reasoning, 19(1), 54–77.
   Web of Science ®Google Scholar
• Stupple, E. J., Gale, M., & Richmond, C. R. (2013). Working memory, cognitive miserliness and logic as predictors of performance on the cognitive reflection test. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th annual conference of the cognitive science society (pp. 1396–1401). Austin, TX: Cognitive Science Society.
   Google Scholar
• Svedholm-Häkkinen, A. M. (2015). Highly reflective reasoners show no signs of belief inhibition. Acta Psychologica, 154, 69–76.
   PubMed Web of Science ®Google Scholar
• Szaszi, B. (2016). The role of expertise and preference behind individuals’ tendency to use intuitive decision style. Journal of Applied Research in Memory and Cognition, 5(3), 329–330.
   Web of Science ®Google Scholar
• Szollosi, A., Bago, B., Szaszi, B., & Aczel, B. (in press). Exploring the determinants of confidence in the bat-and-ball problem.
   Google Scholar
• Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making, 11(1), 99–113.
   Web of Science ®Google Scholar
• Thompson, V. A., & Johnson, S. C. (2014). Conflict, metacognition, and analytic thinking. Thinking & Reasoning, 20(2), 215–244.
   Web of Science ®Google Scholar
• Thompson, V. A., Turner, J. P., & Pennycook, G. (2011). Intuition, reason and metacognition. Cognitive Psychology, 63(3), 107–140.
   PubMed Web of Science ®Google Scholar
• Thompson, V. A., Turner, J. P., Pennycook, G., Ball, L. J., Brack, H., Ophir, Y., & Ackerman, R. (2013). The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition, 128(2), 237–251.
   PubMed Web of Science ®Google Scholar
• Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7), 1275–1289.
   PubMed Web of Science ®Google Scholar
• Toplak, M. E., West, R. F., & Stanovich, K. E. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147–168.
   Web of Science ®Google Scholar
• Tor, A., & Bazerman, M. H. (2003). Focusing failures in competitive environments: Explaining decision errors in the Monty Hall game, the acquiring a company problem, and multiparty ultimatums. Journal of Behavioral Decision Making, 16(5), 353–374.
   Web of Science ®Google Scholar
• Travers, E., Rolison, J. J., & Feeney, A. (2016). The time course of conflict on the cognitive reflection test. Cognition, 150, 109–118.
   PubMed Web of Science ®Google Scholar
• Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.
   PubMed Web of Science ®Google Scholar
• Wason, P. C., & Evans, J. S. B. (1975). Dual processes in reasoning? Cognition, 3(2), 141–154.
   Web of Science ®Google Scholar
• Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2013). Development and testing of an abbreviated numeracy scale: A Rasch analysis approach. Journal of Behavioral Decision Making, 26(2), 198–212.
   PubMed Web of Science ®Google Scholar
• Welsh, M., Burns, N., & Delfabbro, P. (2013). The Cognitive Reflection Test: How much more than numerical ability. In M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Proceedings of the 35th annual conference of the cognitive science society (pp. 1396–1401). Austin, TX: Cognitive Science Society.
   Google Scholar