Content Audit for p-value Principles in Introductory Statistics

Figures & data

Table 1 Curricular component rubric.

The curricular component does not reflect the ASA principle.	The curricular component barely reflects the ASA principle.	The curricular component mostly reflects the ASA principle.	The curricular component fully reflects the ASA principle.
Score = 0.	Score = 0.3	Score = 0.7	Score = 1.0

Table 2 Curriculum audit report card.

		ASA p-value principles
		1	2	3	4	5	6
Curricular component	Instruction: lecture, discussion, video lecture, etc.
	Activity: lab, worksheet, case studies, etc.
	Assessment: homework, quiz, exam, etc.
	Support Materials: readings, tutorials, apps, etc.
	Totals
	Overall p-value principle grades

Table 3 Completed curriculum audit report card.

		ASA p-value principles
		1	2	3	4	5	6
Curricular Component	Instruction: lecture, discussion, video lecture, etc.	0.3	0.7	0.7	0.3	0.7	0.3
	Activity: lab, worksheet, case studies, etc.	0.3	1.0	0.7	0.7	1.0	0.3
	Assessment: homework, quiz, exam, etc.	0.3	1.0	0.7	0.7	1.0	0.7
	Support Materials: readings, tutorials, apps, etc.	0.3	1.0	0.7	0.7	1.0	0.7
	Totals	1.2	3.7	2.8	2.4	3.7	2.0
	Grades	D	A–	B–	C+	A–	C

Table 4 Results of our content audit with a short explanation specific to each principle.

1. p-values can indicate how incompatible the data are with a specified statistical model. Grade: D Reason: Hypothesis tests (HT) are presented and assessed with statements of assumptions, but these assumptions are often made without consideration of why these assumptions are important. May not be until the first regression / modeling class before intuition about the importance of assumptions on validity of HT. Even then, the p-value is more often emphasized as linked to a parameter vs. a model.

2. p-values do not measure the probability that the studied hypothesis is true, or the probability that data were produced by random chance alone. Grade: A– Reason: Conditional probability is strongly emphasized and p-values are consistently discussed and assessed in the context of a particular hypothesis being true. Minor problems remain where we may still see the second interpretation sneak into instructor or graduate assistant descriptions.

3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold. Grade: B– Reason: We regularly and thoroughly challenge the bright line rules (p ¡ 0.05 significant) with examples, readings and discussions; for instance questioning what the difference between p = 0.049 and p = 0.051 really means. However, context factors—design of study, quality of measurements, and data analysis assumptions—are often given little emphasis or done as a topic of discussion separate from the p-value calculation and interpretation.

4. Proper inference requires full reporting and transparency. Grade: C+ Reason: Our curricular components currently discuss p-values in the context of a single study. The class does a lab exercise on data cleaning and ethical / transparent data handling procedures, but largely omits the discussion about transparent reporting of analysis choices. Topics like multiplicity of testing and “p-hacking” are rarely discussed in our introductory classes, which do not tend to emerge until a second course in statistics.

5. p-value does not measure the size of an effect or the importance of a result. Grade: A– Reason: Exercises and class activities address practical significance vs. statistical significance. Emphasize interval estimation and reporting effect sizes across curricular components. Need to be careful that we do not implicitly suggest that larger p-values imply “lack of importance or even lack of effect” or that smaller p-values necessarily imply “larger or more important effects.”

6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. Grade: C Reason: We do not suggest that data analysis ends with the construction of a p-value; but rarely do we discuss or evaluate other evidence in support of our models/hypotheses.

Table 5 Highlights from focus group meeting with colleagues from client departments.

Question prompts	Summary of focus group discussion responses
Teaching in our introductory statistics service courses: Do your students show understanding of p-values and hypothesis testing? What methods should an introductory statistics course include?	p-value understanding: Students do not adequately understand p-values after intro courses Need more theory and intuition about p-values, less procedural mechanics and thresholds, show impacts of assumptions Desired coverage: Want students to be good data consumers—need “soft touch” in intro Considering alternative sources of uncertainty Students do not seem to understand discrete probability distributions More fundamental probability, including conditional probability More exposure and practice with software and packages Emphasize effect sizes and interval estimates May benefit from resampling / randomization—illustrate null model Open question: How can we get more advanced methods from intro?
Teaching in our advanced statistics service courses: What does it do well to prepare your students? Do your students show understanding of p-values and hypothesis testing? What is missing or needs more detail?	Coverage of Methods: p-value good starting point, necessary for reading literature Many students do not understand p-values even after a second course Becoming practitioners, need to know principles and adv. methods What is needed: More modeling vs. HT—focus on fit (R^2) and selection vs. tests Behind in teaching Bayesian methods Challenges to extending beyond HT: May go against adviser’s training and level of expertise Books/literature still have heavy focus on HT “p < 0.05 will continue to be taught b/c it is so much easier to teach” “What do we reinforce? We implicitly support this bright line” Grad students too focused on application, ignoring important nuance Simplicity of “yes/no” type of outcome is attractive
Why is a fundamental understanding of statistics important in your field? What attracted and engaged you personally with statistics?	Big issues for stats in STEM education: Too much focus on “what of statistics”, need more “why of statistics” Motivation gap for learning statistics vs. science specialty Fighting student expectations (and sometimes their adviser’s too) Challenging idea: something could be wrong in a scientific model Individual responses for personal experience with learning statistics: Tools for addressing the idea of significance / expressing confidence Not interested as an undergraduate—took off as a graduate student when encountered Markov models to explain behavior “Hated undergraduate stats class but in graduate school it made sense and I love it now” Interest and passion emerged in grad school but stat classes not directly relevant to their field—90% of statistics background was learned as a researcher. Need to start motivating the learning earlier.