Think-Aloud Interviews: A Tool for Exploring Student Statistical Reasoning

Figures & data

Table 1 Initial questions, with answer choices, on correlation and causation.

clinical-trial (original)
A clinical trial randomly assigned subjects to receive either vitamin C or a placebo as a treatment for a cold. The trial found a statistically significant negative correlation between vitamin C dose and the duration of cold symptoms.
Which of the following can we conclude?
Recovering faster from a cold causes subjects to take more vitamin C. Taking more vitamin C causes subjects to recover faster from a cold. We cannot draw any conclusions because correlation does not imply causation. We cannot draw any conclusions because assignment was random instead of systematic.
books
A survey of Californians found a statistically significant positive correlation between number of books read and nearsightedness.
Which of the following can we conclude about Californians?
Reading books causes an increased risk of being nearsighted. Being nearsighted causes people to read more books. We cannot determine which factor causes the other, because correlation does not imply causation. We cannot draw any conclusions because Californians aren’t a random sample of people.

Table 2 Revised and new questions, with answer choices, about correlation and causation.

clinical-trial (revised)
A clinical trial randomly assigned subjects to either practice mindfulness meditation or a placebo relaxation exercise as a treatment for a cold. The trial found that subjects who practiced mindfulness meditation had a shorter time to recovery than students assigned to the relaxation exercise, and the result was statistically significant.
Which conclusion does this support?
Recovering faster from a cold causes subjects to meditate. Mindfulness meditation causes subjects to recover faster from a cold. We cannot draw any conclusions because correlation does not imply causation. We cannot draw any conclusions because assignment was random instead of systematic.
font-test
Professor Smith wants to know if typing her introductory statistics exams in Comic Sans will improve their exam performance. To answer this question, she randomly gives half of the 200 students in her class an exam with all of the questions typed in Comic Sans, while the other students get the same exam with questions typed in Times New Roman.
After comparing the exam scores across both groups of students, Professor Smith finds that the students who were given the exam typed in Comic Sans had a higher average grade on the exam, compared to the average grade for students who did not receive the exam typed in Comic Sans. Professor Smith repeats this experiment across multiple semesters of her course and always sees the same result.
Which of the following is true?
The result is statistical evidence that giving students exams typed in Comic Sans will lead to higher exam scores across the class. All teachers in every subject should print their exams in Comic Sans to improve their students’ performance. Professor Smith can’t draw any conclusions from these tests because other factors, such as the amount of hours students spent studying, might also affect their exam results. Professor Smith can’t draw any conclusions from these tests because she randomly decided which students would receive the exam typed in Comic Sans instead of choosing students systematically, such as giving only the female students the exam typed in Comic Sans.

Fig. 1 (study-time, original version) To estimate the average number of daily hours that students study at a large public college, a researcher randomly samples some students, then calculates the average number of daily study hours for the sample. Pictured (in scrambled order) are three histograms: One of them represents the population distribution of number of hours studied; the other two are sampling distributions of the average number of hours studied $\overline{X}$, one for sample size n = 5, and one for sample size n = 50. Circle the most likely distribution for each description.

●Population distribution: A B C

●Sampling distribution for n = 5: A B C

●Sampling distribution for n = 50: A B C

Fig. 2 (farm-areas) Farmer Brown collects data on the land area of farms in the United States (in square kilometers). By surveying her farming friends, she collects the area of every farm in the United States, and she makes a histogram of the population distribution of U.S. farm areas. She then takes two random samples from the population, of sizes $n=1000$ and n = 20, and plots histograms of the values in each sample. One of the rows below shows her three histograms. Using the shape of the histograms, choose the correct row.