Reasoning Under Uncertainty: Maximum Likelihood Heuristic in a Problem With a Random Transfer

Abstract

The aim of this study is to explore the judgments and reasoning in probabilistic tasks that require comparing two probabilities either with or without introducing an additional degree of uncertainty. The reasoning associated with the task having an additional condition of uncertainty has not been discussed in previous studies. The 66 undergraduate students, participants in this study, used an analytic process for the task without an additional condition of uncertainty and a heuristic for the task with it. The findings show that they focused on the most likely event and derived a prediction based on this event that, in some cases, led them to answer incorrectly. The educational implications include a gradual method for developing better intuition for the students to help them tackle similar problems in the future.

Keywords:

• Comparing probabilities
• Heuristic judgment
• Intuition
• Probabilistic reasoning

1 Introduction

Urn tasks have been used to represent a chance process and play an important role in probability. Already in the 18th century the French philosopher and mathematician Condorcet concluded that all questions in computing probabilities can be reduced to a single hypothesis regarding a certain quantity of balls of different colors mixed together from which different balls are drawn randomly in a certain order or in a certain proportion (Daston 1980). In this study, two urn tasks involving white and black balls were presented to the participants in a computerized test. The first task required the participants to compare probabilities of randomly drawing a black ball from Urns A and B containing white and black balls each. The second task was to determine how the probability of drawing a random ball from Urn B would change after transferring a random ball from Urn A to Urn B. These two tasks differ in their content but can be solved analytically in the same way by comparing probabilities.

The aim of this study is to explore the judgments and reasoning in probabilistic tasks that require comparing two probabilities either with or without introducing an additional degree of uncertainty. To the best of our knowledge, there is no research exploring the probabilistic reasoning in urn tasks that involve an additional condition of uncertainty as described in this paper. An exploration of the difference between the accuracy of responses in the two tasks and a comparison of reaction times provides an opportunity to understand the reasoning processes associated with these two kinds of tasks.

2 Theoretical Background

2.1 Intuitive Beliefs and Misconceptions in Probability

A large body of research exists about intuitive beliefs and misconceptions in probability for students of different ages, beginning from elementary, middle, and high school to college, university, and prospective teachers (e.g., Batanero and Sanchez 2005; Giuliano et al. 2006; Rubel 2007; Chernoff 2012; English and Watson 2016). Misconceptions in different topics of probability among college students are common (e.g., Giuliano et al. 2006; Khazanov 2008), including the conditional probability topic relevant to this study (Jones, Langrall, and Mooney 2007).

Many studies have documented that probability is a domain known to elicit intuitive probability-based inferences, faulty heuristics, biases, schemas, and misconceptions that often conflict with mathematical theory and lead to incorrect responses (Fischbein and Schnarch 1997; Kahneman Kahneman 2011; Wedell 2011; Meder and Gigerenzer 2014; Kapon et al. 2015).

Probabilistic reasoning consists of drawing conclusions about the likelihood of uncertain events. Of particular relevance to the present article are studies that have suggested different kinds of probabilistic reasoning under uncertainty (Kahneman, Slovic, and Tversky 1982; Konold et al. 1993; Watson and Kelly 2009). Tversky and Kahneman (1983) conducted landmark studies on human understanding of probability; they investigated the ways that humans use, or fail to use, normative probabilistic reasoning when faced with situations involving chance and uncertainty. For example, people use the availability heuristic to estimate the likelihood of events on the basis of how easy it is for them to call to mind particular instances of the event (Kahneman and Tversky 1973).

Konold (1989) first introduced the terminology of outcome approach to describe a model of probability that leads students to make yes or no decisions when faced with uncertain events. He argued that students interpret a statement about the probability of an event as a prediction that the event will occur (i.e., students’ predictions were often based on a deterministic model of the situation). Specifically, a probability of less than 0.5 would imply that the event will not happen, a probability of over 0.5 would imply that the event will happen, and a probability of 0.5 would imply that it is impossible to predict whether the event will happen. The outcome approach derives from a failure to understand that probability depicts a sample space rather than a single outcome. Other studies of students’ reasoning documented that when students had to make predictions from sample data that involve uncertainty they tended to articulate strong deterministic statements. Before intervention students tended to express either extreme confidence in their conclusion or decided that they cannot make any conclusion. They rarely performed an accurate probabilistic analysis (Ben-Zvi et al. 2012).

2.2 Dual-Process Theory

Dual-process theorists (e.g., Kahneman and Frederick 2005; Evans 2006) suggest that reasoning occurs through two distinct types of processing. They characterize the intuitive/heuristic process (System 1) as fast, automatic, with little cognitive effort; the analytical process (System 2) is considered as slower, controlled, deliberate, and effortful. The two general types of reasoning processes tend to be used in making probabilistic judgments (Tversky and Kahneman 1983; Klaczynski 2001; Kahneman and Frederick 2002; Bonner and Newell 2010). On the one hand, people use analytic processes, for example, when calculating the probability of a particular event in a sample space with equally likely outcomes (e.g., the number of outcomes favorable to that event divided by the total number of all outcomes). On the other hand, people develop individual heuristics, that is, intuitive rules for analyzing the probabilities of real-world events. In some cases, judgments based on heuristics match those made with analytic probability theory; in other cases, the decisions are different and inconsistent.

2.3 Intuitive Rules Theory

According to the intuitive rules theory, people of different ages use a small number of intuitive rules that influence their solutions to a wide variety of conceptually non-related tasks that share some common features. Many alternative conceptions related to different domains are inspired by the use of these rules (Stavy and Tirosh 1996). The theoretical framework for developing these rules is Fischbein’s (1987) theoretical model of intuitions. These rules are considered to be intuitive (Stavy et al. 2006) because the learners see such explanations as sufficient in addition to being self-evident and use them with great confidence. Furthermore, responses in line with them are considered absolute to the learners and alternatives are excluded as unacceptable (Fischbein 1987; Stavy and Tirosh 2000). The idea of intuitive rules has been supported by reaction time research and brain imaging research (Stavy et al. 2006; Babai et al. 2010).

2.4 Reaction Time

The reaction time technique is widely used in cognitive science; it measures the time interval between stimulus presentation (e.g., visual or auditory) and response execution (typically, pressing a button). Generally, the longer the reaction time, the more complex the reasoning process (Denes and Pizzamiglio 1999; Spieler 2000). One of the claims made in the dual-process theory is that the heuristic process (System 1) is faster than the analytical process (System 2). Comparing the reaction time between the two tasks helps to identify whether a response is the result of a heuristic or an analytic process.

3 Description of the Computerized Tasks

Urn tasks in probability transform real-life problems into drawing balls of two or more different colors. Drawing balls from urns is a visual mechanism to conceptualize the principles of discrete probability. A famous urn problem is the Pólya urn model where a ball is added to the urn, depending on the color of the previous ball drawn from the urn (e.g., DasGupta 2010).

Inspired by many studies of understanding probability, we used urn tasks of binary comparisons (Babai et al. 2006; Falk, Yudilevich-Assouline, and Elstein 2012; Stavy, Babai, and Kallai 2016). A typical probabilistic task presents the participants with two containers (e.g., urns, boxes, jars, etc.) of elements (e.g., balls, marbles, beads, etc.) in two different colors; participants are then asked to compare the probabilities of drawing randomly an element of a specific color from the two containers. Ansari et al. (2007) found that subjects respond to symbolic number comparisons (e.g., 3 vs. 4) faster and more accurately than to nonsymbolic comparisons (e.g., drawing of dots). This study’s and other studies’ methods were used to devise two tasks; both tasks used symbolic numbers. Each symbolic number represented the number of balls in the corresponding color.

The first task of our computerized test, a comparison of probabilities (CP), asked participants to compare probabilities of randomly drawing a black ball from two different urns (Figure 1).

Fig. 1 Example for a trial in the comparison of probabilities task.

The second task of our computerized test, an additional condition of uncertainty (ACU), asked participants to decide how the probability of randomly drawing a black ball from Urn B would change after transferring a randomly selected ball from Urn A to Urn B (Figure 2). This task involved not only the uncertainty of the color drawn from Urn B at the second stage of the experiment, but also an additional uncertainty condition about the color of the ball that was transferred from Urn A to Urn B on the first stage.

Fig. 2 Example of a trial in the additional condition of uncertainty task.

In our experiment Urn A was displayed on the right side of the screen and Urn B on the left, since the tasks were given in a right-to-left language. These two tasks differ in their content and question but can be solved analytically in the same way as explained below. The ACU task can be solved by comparing the probabilities of drawing a black ball from the two urns. For example, if the probability of randomly drawing a black ball is higher in Urn A than in Urn B, transferring a random ball from Urn A to B will increase the probability of randomly drawing a black ball in Urn B. The intuitive understanding of why this is the solution to the problem can be illustrated by considering an analogous task with two containers that contain sugar solutions. The black balls are analogous to teaspoons of sugar and the white balls are analogous to cups of water. The sweetness of the solution is analogous to the probability of drawing randomly a black ball. Deciding how the sweetness in container B would change after pouring a cup of solution from container A into container B depends on the relative sweetness of the cup of solution being transferred that requires a comparison of the proportions of sugar in the two original solutions. Analogously to the solution task, the selection and transfer of a colored ball require comparing the probability of drawing a black ball in both urns to solve the ACU task. If the probability is higher for Urn A than Urn B, the transfer will increase the probability of Urn B; if the probability is less for Urn A than Urn B, the transfer will decrease the probability of Urn B.

The mathematical logic and reasoning are shown in the following analysis. Let b and w be the number of black and white balls in Urn B, respectively; both b and w represent natural numbers. B₀ represents Urn B before the transfer while B₁ represents Urn B after the transfer. The probability of drawing a black ball from Urn B before the transfer is $P_b(B_0)=\frac{b}{b+w}$. Let us denote by $P_b(A)$ the probability of randomly drawing a black ball from Urn A, which is equivalent to the probability of transferring a black ball from Urn A to Urn B.

According to the total probability formula $P(B)={\sum }_{i=1}^{n}P(B|A_i)P(A_i)$, where A_i .. are pairwise disjoint events whose union is the entire sample space, the probability of randomly drawing a black ball from Urn B after transferring randomly a ball from Urn A to Urn B is$$\begin{array}{c}{P}_b\left(B_1\right)=P_b\left(A\right)·\frac{b+1}{b+w+1}+\left(1-P_b\left(A\right)\right)·\frac{b}{b+w+1}\\ =\frac{P_b\left(A\right)·b+P_b\left(A\right)+b-P_b\left(A\right)·b}{b+w+1}=\frac{P_b\left(A\right)+b}{b+w+1}.\end{array}$$

The probability of drawing a black ball in Urn B after the transfer is calculated by combining the probability of each event (i.e., transferring a black or a white ball from Urn A) with the probability of the prediction given that event. The probability of randomly drawing a black ball will increase if $P_b(B_1)>P_b(B_0)$. This condition is equivalent to $P_b(B_1)-P_b(B_0)>0$.$$\begin{array}{c}\begin{array}{cc}\hfill & P_b\left(B_1\right)-P_b\left(B_0\right)\hfill \\ \hfill & \hspace{1em}=\frac{P_b\left(A\right)+b}{b+w+1}-\frac{b}{b+w}\hfill \\ \hfill & \hspace{1em}=\frac{P_b\left(A\right)·b+P_b\left(A\right)·w+b^2+bw-b^2-bw-b}{\left(b+w+1\right)\left(b+w\right)}\hfill \\ \hfill & \hspace{1em}=\frac{P_b\left(A\right)·b+P_b\left(A\right)·w-b}{\left(b+w+1\right)\left(b+w\right)}>0\hfill \end{array}\\ \begin{array}{cc}\hfill & P_b\left(A\right)·b+P_b\left(A\right)·w-b>0=>P_b\left(A\right)>\frac{b}{b+w}\hfill \\ \hfill & \hspace{1em}=>P_b\left(A\right)>P_b\left(B_0\right).\hfill \end{array}\end{array}$$

Put differently, the probability of randomly drawing a black ball after the transfer will increase if the $P_b(A)$ is higher than the initial probability of drawing a black ball in Urn B. Similarly, the probability of randomly drawing a black ball after the transfer will decrease if $P_b(A)$ is lower than the probability in Urn B. The probability of randomly drawing a black ball after the transfer will not change if $P_b(A)$ is equal to the probability in Urn B.

4 Approach and Aim of the Study

In this study, we focused on the reasoning processes of participants in probabilistic tasks with and without an additional condition of uncertainty. When participants are faced with a problem that includes an occurrence of uncertain events, will their prediction rely on the occurrence of the most likely event? That is, will participants overweight the most likely event and act as if this event will happen with absolute certainty while ignoring the effects of other possible events? Although it might seem strange to ignore other possible events, previous research (Kahneman, Slovic, and Tversky 1982; Eshach and Schwartz 2002) suggests that people of different ages do not always use all available information in making a decision.

The novelty of this research is in exploring the probabilistic reasoning in tasks that involve an additional condition of uncertainty. Specifically, we refer to the context of the additional condition of uncertainty task used in this study in which the color of the ball that will be transferred from Urn A is an uncertain event. In trials where there are more black balls than white balls (i.e., the probability of drawing a black ball is greater than 0.5) in Urn A, the likely event is that a black ball will be transferred from Urn A to Urn B. If participants judge as though this likely event does occur, namely, that a black ball will be transferred from Urn A to Urn B, they will determine that the probability of drawing a black ball from Urn B will increase. Likewise, when there are more white balls than black balls in Urn A, they will likely determine that the probability of drawing a black ball from Urn B will decrease. Since this heuristic does not take into account the less likely event and the ratio between the urns’ proportions, it is not generally applicable; in some tasks, it will lead to incorrect judgments. Unlike the ACU task, in the CP task there is no additional uncertain event involved. Participants were expected to use their analytic reasoning in this task by calculating and comparing two probabilities. This type of reasoning is expected to lead to a high accuracy of judgments.

The following research questions were addressed:

1. Is there any difference in the accuracy of responses for each task between the two groups of participants: the group that started with the CP task and the group that started with the ACU task?

2. Is there any difference in the accuracy of responses between the CP task and the ACU task?

3. How does $P_b(A)$, the probability of transferring a black ball from Urn A to Urn B, influence participant responses?

4. Is there any difference in the response times between the CP and ACU tasks for different values of $P_b(A)$, the probability of transferring a black ball from Urn A to Urn B?

5 Method

The experimental design used in this study documented the performance and reaction time of volunteer college participants. The tasks and sequence of presentation were balanced across the participants.

5.1 Participants

The participants were 66 undergraduate Hebrew-speaking students (29 males and 37 females) who had previously studied a course in probability (probability axioms, conditional probabilities, determining probabilities using tree diagrams, Bayes’ rule) in college in central Israel. All participants participated in classes that aimed to help them develop the required formal knowledge on probability to solve both tasks. During these classes, our students got exposed to a few examples in the context of balls and urns but most of the problem they faced was presented in a different context. Most (>90%) of the participants were business management students. Their mean age was 24.5 years (SD = 2.1). Only students who were not eligible for extended time in tests (or specifically, who had not been diagnosed with attention deficit hyperactivity disorder, dyscalculia, dyslexia, or motor problems) participated. This deselection was made since students with attention or learning disorders are typically less accurate, slower, and have more variable reaction times compared with other (Leth-Steensen, Elbaz, and Douglas 2000; Metin et al. 2013; Peters et al. 2018).

5.2 Materials

The participants were presented with two computerized tasks that can easily and quickly measure their reaction time. Each task included 80 experimental trials. Each trial presented two urns, each containing white and black numbers representing the quantity of balls of the two colors. The trials in both tasks had the same urn composition and the same order as was used in Stavy, Babai, and Kallai (2016). The trials were classified into nine categories based on three ranges of $P_b(A)$: $P_b(A)<0.5,P_b(A)=0.5$ and $P_b(A)>0.5$ and the three possible formal correct judgments (Table 1).

Table 1 Description of the trials classified according to $P_b(A)$ and the formal correct judgment.

	Type of trial
	Type-A		Type-E		Type-B
	Urn B	Urn A	Urn B	Urn A	Urn B	Urn A
$P_b(A)$	black/white	black/white	black/white	black/white	black/white	black/white
$P_b(A)<0.5$	1/2	2/3	4/6	2/3	2/2	2/3
	2/6	1/2	2/4	1/2	2/3	1/2
	2/3	3/4	1/2	2/4	2/2	2/4
	3/5	2/3	2/6	1/3	4/6	2/4
	1/3	3/5	1/3	2/6	3/4	2/3
$P_b(A)=0.5$	2/4	2/2	3/3	2/2	4/2	2/2
	3/4	3/3	2/2	3/3	4/3	3/3
	3/5	3/3	1/1	3/3	5/3	3/3
	2/5	2/2	4/4	3/3	5/2	2/2
	2/3	2/2	4/4	2/2	3/2	2/2
$P_b(A)>0.5$	2/2	3/2	6/4	3/2	2/1	3/2
	3/2	2/1	4/2	2/1	6/2	2/1
	6/4	4/2	2/1	4/2	3/2	4/3
	2/2	4/2	6/2	3/1	5/3	3/2
	4/3	3/2	3/1	6/2	3/1	5/3

NOTE: The numbers in the table represent the ratio of black to white balls in each trial.

The preliminary analysis obtained an equal number of trials across all categories by selecting 45 trials out of the 80 original trials: five trials from each of the nine categories. In the original test, the distribution of trials in categories was such that the minimum number of trials across the nine categories was five. For those specific categories, the entire five trials were chosen. To counter-balance the effect of total numbers of balls, the trials were chosen in a way so that for each trial there is a counter-trial, where in each urn the colors of the balls are exchanged. For example, the trial [2/3 1/2] has a counter-trial [3/2 2/1]. Trials where the number of black and white balls is equal in both urns serve as their own counterbalance. The results and analysis reported in this article are based on these 45 trials distributed across the following types:

• Type-A trials—The formal correct judgment is the probability is higher in Urn A in the CP task, and the formal correct judgment is the probability will increase in the ACU task.

• Type-B trials—The formal correct judgment is the probability is higher in Urn B in the CP task, and the formal correct judgment is the probability will decrease in the ACU task.

• Type-E trials—The formal correct judgment is the probability is equal in the CP task, and the formal correct judgment is the probability will not change in the ACU task.

5.3 Procedure

The experiment was run individually in a computer laboratory where the participants were randomly assigned to two groups that differed in the order of task presentation. The participants in the first group (Group I) were first presented with the CP task. After completing it and taking several minutes of recess, they were presented with the ACU task. The purpose of the recess was to allow participants to rest after completing a relatively lengthy trial. Moreover, as the two trials resembled each other, the recess served as a separator to reduce any confusion that might occur if the two trials were given one after the other. The second group (Group II) responded to the same tasks but started with the ACU task. The trials were presented on a computer screen, using SuperLab 5 Pro© software (http://www.superlab.com). The experiment was divided into three principal parts.

5.3.1 Part 1: Instructions

Instructions were presented to participants on a computer screen and included a description of the task and the way to choose among the possible responses. To encourage participants to respond intuitively rather than making computations, they were asked to respond correctly and as fast as they could; they were not informed of the number of tasks ahead of them. However, they were not limited in time for each task or for the whole session. In the CP task, participants were asked to compare the probabilities of randomly drawing a black ball between the two urns, that is, to judge whether the probability is higher in Urn A or Urn B, or the two probabilities are equal. Each trial was presented on the screen until the participant responded. The orientation of the keys matched the layout of the urns on the screen; In the ACU task, participants were asked to judge whether the probability of drawing a black ball from Urn B will increase, decrease, or not change after transferring a random ball from Urn A to Urn B. Each trial was presented on the screen until the participant responded: will increase, will decrease, or will not change.

5.3.2 Part 2: Practice Set

Participants were presented with a practice set of eight trials (different from the experimental ones) to familiarize them with the task and the experimental setting.

5.3.3 Part 3: Experimental Setting

Participants were presented with the experimental setting of 80 trials. Their responses and reactions time were recorded. Finally, a slide that indicated the end of the test was presented.

6 Results

This section consists of four subsections: comparison of the accuracy of responses between the two different groups of participants; comparison of the accuracy of responses and the typical responses between the two tasks; the influence of $P_b(A)$, (the probability of transferring a black ball from Urn A to Urn B), in each task on the participants’ responses; and the comparison of reaction times between the two tasks.

6.1 Comparing the Accuracy of Responses Between the Two Groups

An independent samples t-test was used to compare the accuracy of responses for each task between the two different groups of participants: Group I (n = 34) that started with the CP task and Group II (n = 32) that started with the ACU task. Results showed no discernible differences in the accuracy of responses between these two groups. Consequently, further statistical analyses were performed for all participants.

6.2 Comparing the Accuracy of Responses Between the Two Tasks

A paired samples t-test was used to explore the difference in the accuracy of responses between the CP and ACU tasks. Table 2 summarizes the accuracy of responses indicating a discernibly (significantly) higher accuracy in the CP task in two of the three types of trials. The only case where there was no discernible difference was the Type-A case in which $P_b(A)>0.5$, where the accuracy of the ACU task was relatively high in comparison to other performances of this task. A possible explanation for this is that the participants’ intuitive reasoning (i.e., more black balls than white balls in Urn A will increase the probability of drawing a black ball in Urn B) matched the formal correct judgment. Surprisingly, in the Type-B case where $P_b(A)<0.5$ and intuitive reasoning matched the correct judgment, there was still a discernible difference between the two tasks favoring the CP task. A possible explanation for this is that the participants were asked about the probability of a black ball while the likely event, in this case, is that a white ball was transferred to Urn B. This might have confused them and decreased the accuracy of their responses in the ACU task.

Table 2 Comparison of the accuracy of responses between the two tasks (N = 66).

		Accuracy in percent (%) of responses: Mean (SD)
$P_b(A)$	Trial	CP task	ACU task	t(65)
$P_b(A)<0.5$	Type-A	69.4 (27.0)	33.9 (33.8)	6.6***
	Type-E	64.2 (37.4)	13.9 (25.5)	8.9***
	Type-B	71.8 (24.0)	59.7 (34.4)	2.9**
$P_b(A)=0.5$	Type-A	82.4 (24.2)	33 (30.1)	9.5***
	Type-E	84.8 (27.6)	57.9 (36.5)	4.9***
	Type-B	75.2 (30.8)	22.7 (26.0)	5.6***
$P_b(A)>0.5$	Type-A	69.4 (25.6)	68.5 (34.6)	0.2
	Type-E	58.8 (35.9)	12.7 (22.0)	9.0***
	Type-B	61.21 (30.8)	22.7 (26.0)	7.8***

NOTE: Paired t-tests were employed to compare the accuracy of responses between the two tasks.

${}^{**}p<$ 0.01.

${}^{***}p<$ 0.001.

In each probability range $(P_b(A)<0.5,P_b(A)=0.5,P_b(A)>0.5)$, there are three possible formal correct judgments; the correct judgment depends on the relation between the probabilities of drawing a black ball from the two urns.

6.3 The Influence of $P_b(A)$ on Participants’ Responses

The main hypothesis was that the probability range for drawing a black ball from Urn A would predict the participant’s response in the ACU task. In other words, those trials where $P_b(A)>0.5$ would be more likely to respond will increase while those trials where $P_b(A)<0.5$ would be more likely to respond will decrease. In contrast, for the CP task, the hypothesis was that the correct judgment of each trial would predict the participant’s response.

Multinomial logit models with random effects (using generalized linear model SPSS software version 22) were employed to predict the participants’ responses. These models are used when subjects respond to several related multiple-choice questions in which the response categories are nominal/ordinal with more than two categories (Hartzel, Agresti, and Caffo 2001). In this study, these models were used to analyze the association of $P_b(A)(P_b(A)<0.5,P_b(A)=0.5,P_b(A)>0.5)$, the formal correct judgment (type of trial), and their interactions with the Group (I, II) and with the participant’s response. In these models, subjects were considered as random effects while the $P_b(A)$, the correct judgment and their interactions with the Groups (I, II) were considered as fixed effects. The models were analyzed separately for each task.

6.3.1 Results for the Comparison of Probabilities Task

The multinomial logit model with random effect revealed a discernible interaction between the correct judgment and the group, F(2, 2950) = 7.91, p < 0.001, and a marginally discernible interaction between $P_b(A)$ and group, F(2,2950) = 2.29, p = 0.06. Therefore, the models were analyzed separately for each group.

The multinomial logit model produces Exp(β), coefficients called odds ratios (OR) that can be interpreted as how a particular independent variable affects the odds of an outcome’s occurrence. The OR evaluates whether the odds of a certain event or outcome is the same for two groups. Specifically, the OR measures the ratio of the odds that an event or result will occur to the odds of the event not happening. An OR = 1 indicates that the event is equally likely to occur in both groups. An OR > 1 indicates that the event is more likely to occur in the comparison group. Table 3 summarizes the ORs of the participants’ responses between two ranges of $P_b(A)$ and two types of trials. For instance, regarding the CP task, Type-A trials are more likely than Type-E trials to be responded Urn A (i.e., the odds for Type-A trials in Group I are 202.92 times the odds for Type-E trials). In other words, when the probability of drawing a black ball is higher in Urn A, the participant’s response Urn A is more likely than the response equal. Unsurprisingly, in this task, there is a strong association between those two variables, which explains the large OR’s in the results.

Table 3 Associations of $P_b(A)$ and the correct judgment with participants’ responses in each task.

		CP task		ACU task
Participants’ responses	Comparison/reference	OR Group I	OR Group II	OR
Urn A/Will increase	$P_b(A)>0.5/P_b(A)=0.5$	1.02	2.28***	10.87***
	$P_b(A)>0.5/P_b(A)<0.5$	1.04	1.47	1.86**
	Type-A/Type-E	202.92***	38.04***	1.89***
	Type-A/Type-B	5.92***	2.8***	0.8
Equal/Will not change	$P_b(A)=0.5/P_b(A)>0.5$	1.02	2.36***	11.00***
	$P_b(A)=0.5/P_b(A)<0.5$	1.30	2.19***	6.35***
	Type-E/Type-A	231.27***	42.026***	1.89***
	Type-E/Type-B	113.19***	36.62***	2.4***
Urn B/Will decrease	$P_b(A)<0.5/P_b(A)>0.5$	1.30	1.32	6.29***
	$P_b(A)<0.5/P_b(A)=0.5$	1.30	2.15***	14.55***
	Type-B/Type-A	19.8***	8.00***	1.96*
	Type-B/Type-E	98.364***	33.20***	1.82**

${}^{*}p<$ 0.05.

${}^{**}p<$ 0.01.

${}^{***}p<$ 0.001.

Monson (1980) suggested an interpretation of the values for OR’s such that 1.2 < OR $\le$ 1.5 indicates a weak association, 1.5 < OR $\le$ 3 indicates a moderate association, and OR > 3 indicates a strong association. Other researchers (e.g., Taubes Taubes 1995; Oleckno 2008) have adopted Monson’s rule as a guideline for judging the strength of the OR effect. Following this rule, Table 3 shows that (a) within the CP task, the OR’s of trial-types greater than 3 in both groups indicates that trial-types have a strong association with participants’ responses and (b) $P_b(A)$ had no association with participants’ responses in group I (i.e., the ORs are not discernibly different). However, $P_b(A)$ had a moderate association with participants’ responses in Group II.

6.3.2 Results for the Preceding Condition of Uncertainty Task

The multinomial logit model with random effects indicated that $P_b(A)$ had the highest effect in predicting the participants’ responses, F(4, 2950) = 95.25, p < 0.001. The correct judgment also had a discernible effect in predicting participants’ responses, F(4, 2958) = 5.95, p < 0.001. This model revealed no discernible interactions between the correct judgment and the group, F(2, 2950) = 0.81, p = 0.52, and between $P_b(A)$ and group, F(2, 2950) = 1.72, p = 0.14.

Table 4 shows that most of the ORs of $P_b(A)$ are greater than 3, which indicates that $P_b(A)$ has a strong to infinite association with participants’ responses. For instance, trials where $P_b(A)>0.5$ are more likely than trials where $P_b(A)=0.5$ to be responded will increase (the odds for trials where $P_b(A)>0.5$ are 10.87 times the odds for trials where $P_b(A)=0.5$). Moreover, Table 3 shows that the correct judgment has a weak to moderate association with the participants’ responses.

Table 4 Distribution of participants’ responses in the additional condition of uncertainty task according to two specific compositions of Urn A, with several different compositions of Urn B (for the same Urn-A composition) and the formal correct judgment.

			Formal correct judgment: Probability of drawing a black ball after the transfer	Participants’ response (%): Probability of drawing a black ball from Urn B after the transfer
Urn A	Urn B			Will	Will	Will
black/white	black/white	$P_b(A)$		increase	not change	decrease
2/3	2/2	$P_b(A)<0.5$	Will decrease	24.8	10.7	54.5
2/3	3/4		Will decrease	25.8	6.1	68.1
2/3	1/2		Will increase	34.9	12.1	53
2/3	3/5		Will increase	31.8	6.1	62.1
2/3	4/6		Will not change	22.7	16.7	60.6
3/2	2/2	$P_b(A)>0.5$	Will increase	68.2	13.6	18.2
3/2	4/3		Will increase	68.2	12.1	19.7
3/2	2/1		Will decrease	71.2	12.1	16.7
3/2	5/3		Will decrease	62.1	7.6	30.3
3/2	6/4		Will not change	59.1	21.2	19.7

NOTE: Most frequent responses in each comparison are marked with bold.

To illustrate the influence of the probability of drawing a black ball in Urn A on the participants’ responses in the ACU task, more detailed results for two representative compositions of Urn A with several different compositions of Urn B are depicted in Table 4. To reach the formal correct judgment, participants have to compare the probabilities of drawing a black ball between the two urns. However, participants’ responses are affected by the composition of Urn A, regardless of the composition of Urn B. When $P_b(A)<0.5$, the probability of transferring a black ball from Urn A to Urn B is less than a half; the most frequent response is that the probability of drawing a black ball from Urn B will decrease regardless of the initial composition of Urn B and the formal correct judgment. Similarly, when $P_b(A)>0.5$, the most frequent response is that the probability of drawing a black ball from Urn B will increase regardless of the initial composition of Urn B and the formal correct judgment. These setups illustrate that not only are most responses according to the probability of drawing a black ball but also most of the participants judge according to the composition of Urn A.

6.4 Comparing the Reaction Times Between the Two Tasks

Another way to learn about the reasoning processes involved in solving these tasks is to compare participants’ reaction times between the two tasks. Table 5 depicts this comparison (N = 66) according to the probability range and the formal correct judgment.

Table 5 Comparison of the reaction times in milliseconds between the two tasks.

	Formal correct judgment: CP task/ACU task	Task reaction time (ms)
$P_b(A)$		CP: Mean (SD)	ACU: Mean (SD)	t(65)
$P_b(A)<0.5$	Urn A/Will increase	7168 (4360)	4366 (5313)	3.4***
	Equal/Will not change	5409 (3206)	3295 (3609)	3.9***
	Urn B/Will decrease	6252 (3796)	4191 (4235)	3.3***
$P_b(A)=0.5$	Urn A/Will increase	5034 (2884)	4818 (5576)	0.3
	Equal/Will not change	3183 (2609)	5905 (4500)	4.7***
	Urn B/Will decrease	5103 (3147)	4708 (5634)	0.6
$P_b(A)>0.5$	Urn A/Will increase	7704 (4722)	4889 (5721)	3.3***
	Equal/Will not change	6847 (4489)	3270 (4927)	5.0***
	Urn B/Will decrease	7087 (5195)	3371 (4116)	5.1***

NOTE: Paired t-tests were employed to compare reaction time between the two tasks. Each participant’s reaction time was calculated as the median of the reaction times (for both correct and incorrect responses) in each type of trial.

${}^{***}p<$ 0.001.

When $P_b(A)\ne 0.5$, the reaction time in the CP task is discernibly longer than in the ACU task. However, when $P_b(A)=0.5$, reaction times revealed different results that can be distinguished as two subcases. When the probability of drawing a black ball is 0.5 in both urns, the reaction time in the CP task is discernibly shorter than in the ACU task. When the probability of drawing a black ball is not equal in the two urns (i.e., $P_b(A)=0.5$ and $P_b(B_0)\ne 0.5$), the results show no discernible difference between the reaction times.

7 Discussion

7.1 General Discussion

The processes associated with responding to two tasks requiring different levels of probabilistic reasoning were explored. These tasks have different content and questions but can be solved in the same analytic way by comparing probabilities. Overall, the study revealed that the reasoning processes used by participants (that who do not represent a random sample of students) in the comparison of probabilities task are different from those used in the additional condition of uncertainty task.

Unsurprisingly, the results show that the accuracy of responses in the CP task is substantially higher than in the ACU task. In the CP task, 70% (n = 46) of the participants responded correctly to above 60% of the trials; in the ACU task, only 3% (n = 2) responded correctly to above 60% of the trials. In the CP task, the typical response of participants was according the formal correct computation; in Table 3, for example, for trials where the correct judgment was Urn A, the odds to respond Urn A for Group I were 5.92 times the odds of those where the correct judgments were Urn B. In this task, the number of balls in each color is known in both urns and there is no additional uncertainty; it seems that the participants used their formal probabilistic knowledge. In the ACU task, the number of balls in each color in Urn B after randomly transferring a ball from Urn A is unknown and the analytical computation is more difficult. As the results show, participants responded based on the likely event that may occur. That is, when the probability of drawing a black ball in Urn A was greater than 0.5—or the likely event was that a black ball will be transferred from Urn A to Urn B—the frequent response was will increase. It was also revealed that the odds in trials where $P_b(A)>0.5$ were 10.87 times those in trials where $P_b(A)=0.5$ and 1.86 times those where $P_b(A)<0.5$ to be responded will increase (Table 3). Similarly, when $P_b(A)<0.5$—or the likely event is that a white ball will be transferred from Urn A to Urn B—the frequent response was will decrease. It was also revealed that the odds in trials where $P_b(A)<0.5$ and the response would be will decrease were 14.55 times those in trials where $P_b(A)=0.5$ and 6.29 times those trials where $P_b(A)>0.5$ (Table 3).

Participants’ typical responses were according to the reasoning described above, even for a set of tasks with an identical composition of Urn A and several different compositions of Urn B. The different compositions of Urn B affected the formal correct judgment but did not change the participants’ responses. When $P_b(A)=0.5$, a typical response cannot be ascertained. In this case, there is not a higher probability of drawing a black ball from Urn A. In this study, participants were faced with an uncertainty that manifested itself in the unknown ball color to be transferred from Urn A to Urn B. When faced with uncertainty, the participants’ judgments were based on the likely event. Specifically, the participants solved the problem as if the more probable ball color in Urn A will be transferred to Urn B.

With regard to reaction times, interesting results were found when $P_b(A)\ne 0.5$ (i.e., when there is a majority color in Urn A). In these cases, the reaction times in the ACU task were discernibly shorter than those in the CP task. As previously mentioned, studies have shown that the longer the reaction time, the more complex the reasoning process is (Denes and Pizzamiglio 1999; Spieler 2000). At first glance, this finding may seem puzzling and inconsistent with the above analysis that suggests the CP task is simpler than the ACU task. However, the shorter reaction time is in line with the erroneous heuristic that participants may use in the ACU task. A strategy that assumes their reasoning depends only on the probability of drawing a black ball from Urn A requires the respondent to compare the number of black and white balls only in Urn A. It is certainly easier than comparing two probabilities as required in the CP task. These findings are congruent with the theory of Kahneman and Frederick (2002) who explain that, sometimes when individuals have to make a judgment that is complex, they substitute instead a more easily calculated heuristic. Kahneman (2000) argued that the difference in processing time—heuristic processing is faster than analytic processing—serves as a basis to identify whether a response is the result of a heuristic or an analytic process (Stupple, Ball, and Ellis 2013).

The present study supports the conclusions of other studies (Evans and Curtis-Holmes 2005; Leron and Hazzan 2009; Evans 2010; Banks and Hope 2014) that documented people’s use of analytic and heuristic processes. This study presents two different levels of tasks that can be solved in the same analytic way; however, participants solved one task analytically (System 2 process) and the other task heuristically (System 1 process). In the simpler task, where the number of balls in each color in both urns is known, people used an analytic process; in the more complex task that was not previously investigated, where the composition of the second urn depends on an additional uncertain event, people solved it heuristically. When people are confronted with uncertainty about the urn’s content, their judgment relies on the majority color instead of a probabilistic outcome. More specifically, this study demonstrates that these participants used a heuristic in which they assumed that the most likely event occurs, thereby greatly reducing the complexity of the task at hand.

7.2 Educational Implications

In probability, Urn problems serve as an instrument of translating realistic problems into mathematical ones; some objects of real interest (e.g., responses, people, characteristics, etc.) are represented as colored balls in an urn or another container. Usually, the goal in these problems is to determine the probability of drawing one color or another. Urn problems are continuously used in probability and statistics courses as didactic tools (Johnson and Kotz 1977; Batanero, Henry, and Parzysz 2005). Understanding school and college students’ reasoning in these kinds of problems, like the ACU task, is essential to building methods to overcome their intuitions, which may lead to incorrect responses.

Specifically, in the ACU task described in this paper, the students tended to rely on the most probable event and seemed to ignore all possible outcomes. To overcome this bias toward a deterministic thinking, we suggest a gradual method that would build better intuition for the students to help them approach similar problems in the future. As a first stage, we suggest that educators present the ACU task to students with emphasis on the counter-intuitive urn configurations (e.g., Type-B trials where $P_b(A)>0.5$) that, when solving it heuristically, leads to incorrect judgment).

Once the students choose the erroneous answer, it will be effective to point out that this answer is incorrect and to promote a discussion to understand their reasoning. Counter-intuitive problems trigger students’ curiosity and raise their motivation to dive deeper into the analytical parts of the problem. One aspect of instruction is to elicit in learners an awareness of the possible conflict between their primary intuition and the analytical structures of probability (Fischbein and Schnarch 1997).

Next, it will be helpful to present an analogous problem with which the students are more familiar, such as the sugar solution task where teaspoons of sugar replace the black balls and cups of water replace the white balls (described in Section 3). This can illustrate the correct answer for the ACU task without the element of probability. The use of a physical analogy is enormously helpful in developing good statistical intuition (Martin 2003). Another interesting analogy is to ask students how the probability of drawing a black ball from Urn B will change after transferring the entire content of Urn A into Urn B—instead of transferring only one random ball as described in the ACU task. Both of these suggested analogies eliminate the additional uncertainty of which ball color is transferred. Yet, these analogies demonstrate that one should consider all possible outcomes (e.g., pouring some of a solution or pouring the entire content is similar to considering all the events at once) and may help students avoid wrong intuition, which is based on the most likely event. Using these analogies in the instruction can provide experiences for building an understanding of probabilistic reasoning and develop more normative intuitions that can reduce students’ limiting primary intuitions. It is essential for learners to develop an ability for recognizing when to ignore a primary intuition or, as Fischbein (1987) described it, to create an “intuition for the non-intuitive” (p. 96).

As the last stage of the gradual method, we suggest solving analytically the ACU task and showing the analogic solutions (using the total probability formula) to the CP task (described in Section 3). Optionally, the educator may start with an analytic solution for a specific counter-intuitive Urn configuration and then generalize it. This follows the approach of Gigerenzer, Todd, and ABC Research Group (1999) that presents a simple solution to this complex problem instead of an expected complex solution.

This method can develop a conceptual understanding needed to construct a qualitative representation of the ACU task before using mathematical calculation. We believe that looking at this problem in several perspectives may enrich and deepen students’ understanding of probability and uncertainty.

Misconceptions of college students in probability do not always disappear after formal learning (Khazanov 2008). According to Vosniadou et al. (2001), “Instructional interventions need to be designed to make students aware of their implicit representations, as well as of the beliefs and presuppositions that constrain them” (p. 392). Effective learning environments aiming specifically to eliminate students’ misconceptions are essential in this situation (e.g., Hirsch and O’Donnell 2001; Pratt 2005).

According to Heyvaert et al. (2018), insights into the specific motivation for a learner’s choice among different alternatives could provide important knowledge for educational practice. In addition, a brain-imaging research study (Babai, Shalev, and Stavy 2015) showed that it is possible to improve students’ problem-solving abilities using effective instruction that includes explicit warning from intuitive solutions.

Future research could benefit from including qualitative and quantitative research approaches (mixed methods). Interviews with subjects where they explain their choice can help pinpoint students’ reasoning. Indeed, according to Clement (2000), interviews enable researchers to collect and analyze data on mental processes at the level of a subject’s ideas and meanings and help to expose the processes in the subject’s thinking. Comparison of the subject’s answers with their grades in the probability course is another possible direction for future research. In addition, it would be interesting to examine the instructional effectiveness of the described gradual method, including each suggested stage. We hope that using these counter-intuitive tasks in class can be very useful tools to motivate and challenge students to think beyond superficial paradigms (Sowey 2001) and to explore statistical concepts at a deeper level.