How Confident are Students in their Misconceptions about Hypothesis Tests?

Figures & data

Table 1. Modus Tollens and the Illusion of Probabilistic Proof by Contradiction

Statements	Modus tollens	Example	Illusion
Premise 1	p ⇒ q	If it is raining there are clouds	If H0 is true there is a high probability that the p-value is large
Premise 2	q^c	There are no clouds	The p-value is small
Conclusion	p^c	It is not raining	H0 is improbable

Table 2. Percentages of selected responses for the definition of hypothesis test (total n=144)

Option	“A hypothesis test is…	Nr. Students	Percentage
H 1	…a proof of H0 (or) or its falseness”	29	20%
	…a proof of Hα (or) or its falseness”	2	1%
H 2	…a proof of the probability or improbability of H0”	27	19%
	…a proof of the probability or improbability of Hα”	5	3%
Correct	…an evaluation of the evidence in favor or against H0 in the data”	89	62%
Confusing H0 and Hα	…an evaluation of the evidence in favor or against Hα in the data”	8	6%
Completely correct answer: Selecting the correct option only		80	56%

Note: For a correct interpretation of the percentages in the table, take into account that, although they were told to select the most correct option, students often picked more than one response and therefore these percentages do not sum up to 100% neither does the number of students sum up to 144.

Table 3. Percentages of selected responses for the definition of the p-value (n=144)

Option	“Given a p-value of 0.01…	Nr. Students	Percentage
p3	… the probability of the null hypothesis is 0.01”	17	12%
p1	… the probability of the null hypothesis assuming the same or more extreme data is 0.01”	13	9%
Correct	… the probability of obtaining the same or more extreme data assuming the null hypothesis is true”	75	52%
p2	… the probability of obtaining the same or more extreme data is 0.01”	10	7%
p4	… the probability of making an error when rejecting the null hypothesis is 0.01”	23	16%
p5	…the difference between the groups is big”	11	8%
p6	…the difference between the groups is small”	15	10%
Completely correct answer: Selecting the correct option only		66	46%

Note: For a correct interpretation of the percentages in the table, take into account that, although they were told to select the correct option, students often picked more than one response and therefore these percentages do not sum up to 100% neither does the number of students sum up to 144.

Table 4. Percentages of selected responses for the definition of the significance level

Option	“If the results of a test are statistically significant for a level α = 0.05 …	Nr. Students	Percentage
α4	… the null hypothesis is proven to be improbable”	14	10%
α5	… the null hypothesis is proven to be false”	9	6%
α2	…the probability of rejecting the null hypothesis is of 95%”	24	17%
Correct	… the probability of rejecting the null hypothesis assuming the null hypothesis is true”	60	42%
α1	…the probability of the null hypothesis assuming that it is rejected is of 5%”	15	10%
α3	…the probability of the null hypothesis is of 5%”	24	17%
Completely correct answer: Selecting the correct option only		57	40%

Note: For a correct interpretation of the percentages in the table, take into account that, although they were told to select the correct option, students often picked more than one response and therefore these percentages do not sum up to 100% neither does the number of students sum up to 144.

Table 5. Test for proportions of completely correct answers by backgrounds

H0	Z	Interval for difference (95%)	p
Freq. Comp. Correct Answer HT(No Background) = Freq. Comp. Correct Answer. HT (Background)	2.81	(0.077, 0.405)	0.0050
Freq. Comp. Correct Answer p-value (No Background) = Freq. Comp. Correct Answer. p-value (Background)	6.39	(0.414, 0.694)	<.0001
Freq. Comp. Correct Answer α (No Background) = Freq. Comp. Correct Answer α (Background)	−1.05	(−0.258, 0.077)	0.2924

Note: Participants who did not answer any of the three questions or did not provide information about their statistical background were not included in this analysis (n=133).

Table 6. Summary of confidence level for each item

Item	n	Mean Confidence	Standard Deviation	Min. Selected	Max. Selected
Hypothesis Test	142	6.21	2.58	0	10
p-value	140	5.66	2.96	0	10
Significance level α	134	4.69	2.79	0	10

Table 7. Summary of students providing a confidence level for all three items

Item	n	Mean Confidence	Standard Deviation	Min. Selected	Max. Selected
Hypothesis Test	132	6.21	2.56	0	10
p-value	132	5.63	2.98	0	10
Significance level α	132	4.70	2.81	0	10

Table 8. Medians for Contrast (Paired) Confidences

Variable	n	Mean	Standard Deviation	Min	Max
Confidence Item 1 — Confidence Item 3	132	1.51	2.64	−6	9
Confidence Item 2 — Confidence Item 3	132	0.92	3.13	−7	9
Confidence Item 1 — Confidence Item 2	132	0.58	2.39	−4	8

Figure 1. Confidence of students with correct answer vs. students with misconception for all three items

Note: Sample sizes of the groups are HT item: Completely correct n=75, Misc n=57; p-value item: Completely correct=62, Misc n=70; α item: Completely correct n=57, Misc n=75. The symbol “α” has been substituted by the word “alpha” in the horizontal-axis.

Table 9. Two-sample t-tests for Confidence classified by type of answer (completely correct vs. misconception)

Item	Perfect Answer	Mean Confidence	Stand. Dev.	DF	t-value	p-value
Hypothesis Test	No	5.33	2.36	130	−3.60	0.0005
	Yes	6.88	2.51
p-value	No	4.47	2.85	130	−5.19	< .0001
	Yes	6.94	2.57
Significance level a	No	4.65	2.65	130	−0.24	0.8112
	Yes	4.77	3.02

Figure 2. Confidence of students selecting each (and only) option for the first item.

Note: The width of the boxes is proportional to the number of students selecting each (and only) option. H1 (for H₀): 19, H1 (for H_a): 1, H2 (for H₀): 18, H2 (for H_a): 2, Conf. Hyp: 3, Correct: 73.

Figure 3. Confidence of students selecting each (and only) option for the second item.

Note: The width of the boxes is proportional to the number of students selecting each (and only) option. p1: 8, p2: 7, p3: 11, p4: 18, p5: 9, p6: 1, Correct: 59.

Figure 4. Confidence of students selecting each (and only) option for the third item.

Note: The width of the boxes is proportional to the number of students selecting each (and only) option. α1: 13, α2: 19, α3: 19, α4: 11, α5: 6, Correct: 53. The symbol α has been substituted by the word “alpha” in the x-axis.

Table

	0	1	2	3	4	5	6	7	8	9	10
No confidence												100% confident

Table

	0	1	2	3	4	5	6	7	8	9	10
No confidence												100% confident

Table

	0	1	2	3	4	5	6	7	8	9	10
No confidence												100% confident