Inferential Statistics: Understanding Expert Knowledge and its Implications for Statistics Education

Figures & data

Table 1. Research Scenario Dyads.

Dyad 1	Participation and achievement among 4^th grade students. Salaries of teachers with a master's degree.
Dyad 2	Salaries of teachers with a master's degree. Relationship between high school and college achievement.
Dyad 3	Relationship between high school and college achievement. Participation and achievement among 4^th grade students.
Dyad 4	Pairing in mathematics class. Preparation time and success in examination.
Dyad 5	Relationship between high school and college achievement. Pairing in mathematics class.
Dyad 6	Relationship between high school and college achievement. Preparation time and success in examination.

Table 2. Statistical Technique Triads.

Triad 1	Two-way ANOVA	X^2 Test	t-test
Triad 2	Test of Pearson's r	X^2 Test	t-test
Triad 3	Two-way ANOVA	Test of Pearson's r	Sign Test
Triad 4	Test of Pearson's r	t-test	Sign Test
Triad 5	Two-way ANOVA	Test of Pearson's r	t-test
Triad 6	Two-way ANOVA	X^2 Test	Sign Test

Table 3. The constructs and the rating matrix elicited from Novice 3 by using the names of statistical tests.

	Two-way ANOVA	Test of Pearson's r	X^2 Test	t-test	Sign Test
1. You can test main effects as well as interaction between factors by this test.	5	1	1	1	1
2. This is a test of proportions.	1	1	5	1	1
3. You can use this test to compare two things, like pre and post test scores.	1	5	3	3	5
4. You use this test when you have two independent variables in a research situation.	5	5	3	1	1
5. This test is used for relatively complicated research situations.	5	1	5	1	1
6. You can test changes in percentage (distributions) over time by using this test.	1	1	5	1	1
7. There is one dependent variable in the situations in which this test is used.	1	1	1	5	5
8. F is your test statistic in this test.	5	1	1	1	1
9. This test is used when you have one independent variable.	1	1	1	5	1
10. Correlation between two things (or factors) can be tested by this technique.	5	5	1	1	1
11. You use this test when you have 2 variables.	1	1	1	5	5
12. This test is covered in Stat II.	1	1	5	1	5
13. The test statistic can take values between 1 and −1 in this test.	1	5	1	1	1
14. You can compare two different populations by using this test.	1	1	5	1	1
15. You can compare one population on two things (e.g. before or after a treatment) in this test.	1	1	4	5	5
16. This test is taught in Stat I.	1	5	1	5	1
17. How different factors combine to affect another variable is what you can test for by using this technique.	5	1	1	1	1
18. You can use this test with repeated-measures situations.	5	1	1	1	1
19. When you have two different groups, this test may be used.	5	1	5	3	5
20. This test can be used with randomized-block-type research designs.	5	1	1	1	1

Figure 1. Extensiveness of the knowledge used by statistical consultants and mathematical statisticians in research scenario tasks.

Figure 2. Connectedness of the knowledge used by statistical consultants and mathematical statisticians in research scenario tasks.

Figure 3. Extensiveness of knowledge used by the two groups of experts in statistical technique tasks.

Figure 4. Connectedness of knowledge used by the two groups of experts in statistical technique tasks.

Figure 5. Task-adaptedness of knowledge used by the two groups of experts in research scenario tasks.

Figure 6. Task-adaptiveness of knowledge used by the two groups of experts in statistical technique tasks.

Figure 7. Extensiveness of knowledge used by experts and novices in research scenario tasks.

Figure 8. Connectedness of knowledge used by experts and novices in statistical technique tasks.

Figure 9. Extensiveness of knowledge used by experts and novices in statistical technique tasks.

Figure 10. Connectedness of knowledge used by experts and novices in statistical technique tasks.

Table 3. Number of research design, theoretical and procedural themes addressed by a majority in expert and novice groups.

	Research Design	Theory	Procedures
Experts	17	3	1
Novices	11	4	1

Table

Participation group	Grade 1		Grade 2		Grade 3
	mean	(s.d.)	mean	(s.d.)	mean	(s.d.)
non-participants	8.50†	(.83)	9.15	(.89)	9.72	(.95)
passive participants	8.76	(.79)	9.50	(.78)	10.07	(.82)
active participants	9.11	(.77)	9.95	(.73)	10.51	(.67)

Table

Subject	1	2	3	……	59	60
High School GPA	2.2	2.6	3.4	……	3.1	2.5
College GPA	1.5	1.7	2.5	……	3.0	2.0

Table

	Pass	Fail	Total
Less than 5 hours	11	15	26
5 to 10 hours	25	7	32
More than 10 hours	35	7	42
Total	71	29	100

Table

	Average annual salary	standard deviation
Teachers with a bachelor's degree	$32,100	$5,100
Teachers with a master's degree	$36,475	$4,400

Table 4. Research design themes observed in the elicited constructs and the number of subjects who addressed the themes.

	Experts (n=6)		Novices (n=6)
Themes	research scenarios	statistical scenarios	research scenarios	statistical scenarios
PURPOSE
1. test of difference*	6	5	6	4
2. test of association	5	6	5	6
3. goodness of fit test	0	2	0	0
4. testing homogeneity	2	2	0	0
NUMBER OF INDEPENDENT, DEPENDENT AND ALL VARIABLES
5. all, one variable	0	1	1	0
6. all, two variables	5	5	2	4
7. all, three variables	1	2	1	1
8. independent, one variable	2	2	0	2
9. independent, two variables	4	4	3	4
10. main effects	3	1	0	2
11. interactions	3	4	5	3
12. test of association	1	3	0	3
MEASUREMENT SCALE OF INDEPENDENT VARIABLE
13. nominal	1	3	1	1
14. ordinal	2	1	2	0
MEASUREMENT SCALE OF DEPENDENT VARIABLE
15. nominal	5	4	6	4
16. ordinal	4	4	1	1
17. interval	6	4	1	1
NUMBER OF CATEGORIES OF INDEPENDENT VARIABLE
18. two	0	1	0	0
19. more than two	0	2	1	0
20. post-hoc comparisons	0	2	0	2
RELATIONS AMONG SAMPLES
21. independent samples	4	2	2	1
22. related samples	2	1	2	3
23. matched-pairs samples	5	4	5	2
24. repeated-measures samples	5	1	3	4
NUMBER OF GROUPS
25. one group	2	4	2	1
26. two groups	6	4	3	5
27. three groups	3	0	3	1
POPULATION PARAMETER
28. means	6	6	6	6
29. correlations	2	5	1	2
30. medians	0	2	0	0
31. regression coefficient	0	1	0	0
32. proportions	4	3	0	2
ISSUES OF VALIDITY
33. internal validity	4	0	2	0
34. external validity	3	0	2	2
TYPES OF DESIGNS
35. experimental research	2	1	3	2
36. research using pre-test, post-test	1	0	4	3
37. trend analysis/timeseries/runs designs	1	2	1	0
38. research in which causality can be inferred	1	1	1	1
39. research with retrospective data	1	0	1	0
40. complex research designs	3	0	0	0

* Themes given in bold indicate the ones that were addressed by a majority of experts (n>=4) in at least one of the tasks, research scenarios or statistical techniques, and the themes in italics were addressed by a majority of the novice group.

Table 5. Themes observed in the constructs related to theoretical aspects and the number of subjects who addressed the themes.

	Experts (n=6)		Novices (n=6)
Themes	research scenarios	statistical scenarios	research scenarios	statistical scenarios
PURPOSE
1. test of difference*	6	5	6	4
2. test of association	5	6	5	6
3. goodness of fit test	0	2	0	0
4. testing homogeneity	2	2	0	0
NUMBER OF INDEPENDENT,
DEPENDENT AND ALL VARIABLES
5. all, one variable	0	1	1	0
6. all, two variables	5	5	2	4
7. all, three variables	1	2	1	1
8. independent, one variable	2	2	0	2
9. independent, two variables	4	4	3	4
10. main effects	3	1	0	2
11. interactions	3	4	5	3
12. test of association	1	3	0	3
MEASUREMENT SCALE OF
INDEPENDENT VARIABLE
13. nominal	1	3	1	1
14. ordinal	2	1	2	0
MEASUREMENT SCALE OF
DEPENDENT VARIABLE
15. nominal	5	4	6	4
16. ordinal	4	4	1	1
17. interval	6	4	1	1
NUMBER OF CATEGORIES OF
INDEPENDENT VARIABLE
18. two	0	1	0	0
19. more than two	0	2	1	0
20. post-hoc comparisons	0	2	0	2
RELATIONS AMONG SAMPLES
21. independent samples	4	2	2	1
22. related samples	2	1	2	3
23. matched-pairs samples	5	4	5	2
24. repeated-measures samples	5	1	3	4
NUMBER OF GROUPS
25. one group	2	4	2	1
26. two groups	6	4	3	5
27. three groups	3	0	3	1
POPULATION PARAMETER
28. means	6	6	6	6
29. correlations	2	5	1	2
30. medians	0	2	0	0
31. regression coefficient	0	1	0	0
32. proportions	4	3	0	2
ISSUES OF VALIDITY
33. internal validity	4	0	2	0
34. external validity	3	0	2	2
TYPES OF DESIGNS
35. experimental research	2	1	3	2
36. research using pre-test, post-test	1	0	4	3
37. trend analysis/timeseries/runs designs	1	2	1	0
38. research in which causality can be inferred	1	1	1	1
39. research with retrospective data	1	0	1	0
40. complex research designs	3	0	0	0

* Themes given in bold indicate the ones that were addressed by a majority of experts (n>=4) in at least one of the tasks, research scenarios or statistical techniques, and the themes in italics were addressed by a majority of the novice group.

Table 6. Procedural themes addressed in the constructs of subjects and the number of subjects who addressed the themes.

	Experts (n=6)		Novices (n=6)
Themes	research scenarios	statistical scenarios	research scenarios	statistical scenarios
TYPE OF INFERENCE PROCEDURE
1. directionality of hypothesis testing (one-tailed, two-tailed	1	1	0	1
2. nature of inference procedure (i.e., hypothesis testing, estimation, fitting models)	1	4	0	1
3. constructing a confidence interval	2	0	2	0
COMPTUTATIONAL ELEMENTS
4. significance level (alpha) given/needed	0	0	3	1
5. kind of test statistic used (e.e,. X^2, x, t, or F	0	1	1	4
6. range of the values test statistic can take (i.e., being between −1 and +1)	2	3	0	3
7. need for a standar table to look up	0	0	0	2
8. method of determining degrees of freedom	0	1	0	1
9. need for transformation of test statistics (i.e., Fisher's z transformation for r)	0	1	0	0
10. consideration of outlier vlaues during computation	2	1	1	1
11. availability of population variance for computations	0	0	0	1
12. data being usually organized in tables	0	0	0	1
13. using row and column totals in calculations	0	0	0	1
14. computing “pooled variances”	0	1	0	0
15. mathematical calculations being relatively complicated	0	1	1	3
MEANING OF COMPUTATIONAL PROCEDURES
16. comparing within and between group variability	0	1	0	1
17. comparing observed and expected values	0	1	0	1
18. relating the order of one distribution with another distribution	0	2	0	1

Table 7. Non-technical themes observed in the constructs of subjects and the number of subjects who addressed the themes.

	Experts (n=6)		Novices (n=6)
NON-TECHNICAL THEMES ABOUT STATISTICAL TECHNIQUES	research scenarios	statistical scenarios	research scenarios	statistical scenarios
REQUIRED ASSUMPTIONS
1. popularity of the use of test	n/a	0	n/a	3
2. class in which the technique is covered (Stat I, II, or III)	n/a	0	n/a	4
3. level of conceptual difficulty of the technique	n/a	0	n/a	1
4. how the test is named (after a person, by the underlying probability distribution, etc.)	n/a	1	n/a	0
5. degree of usefulness of the statistical technique	n/a	0	n/a	4
6. statistical techniques being related to other tests (without specifying reasons)	n/a	1	n/a	3
7. the need for a textbook to remember the specifics of the statistical technique	n/a	0	n/a	2
8. generality of statistical techniques in terms of the situations they can be used for	n/a	0	n/a	1
NON-TECHNICAL THEMES ABOUT RESEARCH SCENARIOS
9. kind of statistical method that the scenario suggests for analysis	6	n/a	5	n/a
10. evaluation of the research scenario for its general quality	3	n/a	3	n/a
11. clarity of the design of a research scenario	0	n/a	3	n/a
12. factual observations from a research scenario (e.g., age range of subjects, context of research, data being based on self-report, etc.	3	n/a	4	n/a