Computer-Automated Approach for Scoring Short Essays in an Introductory Statistics Course

Figures & data

Figure 1. Expert matrix example.

Figure 2. Student matrix example.

Figure 3. Expert and student knowledge structures in the GIKS.

Table 1. Instructor inter-rater polychoric correlations.

	Pre-test (N = 53)				Post-test (N = 44)
	r	χ^2(1)	p-value		r	χ^2(1)	p-value
Prompt 1	0.871	62.093	< 0.001		0.888	60.266	< 0.001
Prompt 2	0.915	69.651	< 0.001		0.956	80.912	< 0.001

Table 2. Frequency distribution of scores on Prompt 1.

	Pre-test			Post-test
Links in common	N	Percent		N	Percent
0	17	44.74		6	15.79
1	12	31.58		8	21.05
2	7	18.42		13	34.21
3	2	5.26		5	13.16
4	0	0		6	15.79
	Pre-test			Post-test
Keyword occurrence	N	Percent		N	Percent
0	7	18.42		0	0
1	9	26.68		3	7.89
2	10	26.32		6	15.79
3	5	13.16		7	18.42
4	4	10.53		9	23.68
5	3	7.89		9	23.68
6	0	0		1	2.63
7	0	0		2	5.26
8	0	0		1	2.63

Table 3. Frequency distribution of scores on Prompt 2.

	Pre-test			Post-test
Links in common	N	Percent		N	Percent
0	24	63.16		2	5.26
1	6	15.79		12	31.58
2	7	18.42		10	26.32
3	1	2.63		7	18.42
4	0	0		5	13.16
5	0	0		1	2.63
6	0	0		1	2.63
	Pre-test			Post-test
Keyword Occurrence	N	Percent		N	Percent
0	0	0		0	0
1	7	18.42		0	0
2	16	42.11		2	5.26
3	5	13.16		8	21.05
4	8	21.05		13	34.21
5	2	5.26		10	26.32
6	0	0		3	7.89
7	0	0		1	2.63
8	0	0		0	0
9	0	0		1	2.63

Table 4. Descriptive statistics for manual scores.

	Pre-test					Post-test
	Mean	SD	Median	Skew.		Mean	SD	Median	Skew.
Prompt 1	2.724	1.830	2.500	1.923		7.171	3.664	6.750	0.314
Prompt 2	2.158	1.311	2.000	0.400		5.776	2.506	5.750	0.142

NOTE: N = 38

Table 5. Change in scores with Wilcoxon signed rank tests.

	Change (post- pre)
	Q1	Median	Q3		S	Unadjusted p-value
Prompt 1
Links in common	0	1	2		142.0	< 0.0001
Keyword occurrence	0	2	3		192.5	<0.0001
Manual score	2	3	7		336.5	<0.0001
Prompt 2
Links in common	1	1	2		263.5	<0.0001
Keyword occurrence	1	2	3		257.5	<0.0001
Manual score	2	3.75	5		352.0	<0.0001

NOTE: N = 38.

Table 6. Polychoric correlations for Prompt 1.

	1.	2.	3.	4.	5.	6.
1. Links in common, pre-test
2. Keyword occurrence, pre-test	0.871^**
3. Manual score, pre-test	0.553^*	0.779^**
4. Links in common, post-test	0.146	0.176	-0.008
5. Keyword occurrence, post-test	0.156	0.135	0.039	0.720^**
6. Manual score, post-test	0.114	0.128	0.107	0.361^*	0.714^**
7. Final exam score	0.044	0.337^*	0.523^*	0.380^*	0.328^*	0.359^*

NOTE: N = 38,

* p-value ≤ .05,

** p-value ≤ .001.

Table 7. Polychoric correlations for Prompt 2.

	1.	2.	3.	4.	5.	6.
1. Links in common, pre-test
2. Keyword occurrence, pre-test	0.984^**
3. Manual score, pre-test	0.862^**	0.749^**
4. Links in common, post-test	0.278	0.299	0.199
5. Keyword occurrence, post-test	0.384	0.433^*	0.160	0.833^**
6. Manual score, post-test	0.146	0.108	0.131	0.795^**	0.753^**
7. Final exam score	0.367	0.362^*	0.255	0.412^*	0.335^*	0.519^*

NOTE: N = 38,

* p-value ≤ 0.05,

** p-value ≤ 0.001.

Table 8. Instructor inter-rater polychoric correlations.

	Pre-test (N = 95)				Post-test (N = 75)
	r	χ^2(1)	p-value		r	χ^2(1)	p-value
Prompt 1	0.983	143.502	< 0.001		0.909	111.938	< 0.001
Prompt 2	0.803	21.902	< 0.001		0.956	110.412	< 0.001

Table 9. Frequency distribution of scores on Prompt 1.

	Pre-test			Post-test
Links in common	N	Percent		N	Percent
0	64	86.49		15	20.27
1	4	5.41		22	29.73
2	2	2.70		18	24.32
3	2	2.70		10	13.51
4	1	1.35		5	6.76
5	1	1.35		2	2.70
6	0	0		1	1.35
7	0	0		1	1.35
	Pre-test			Post-test
Keyword occurrence	N	Percent		N	Percent
0	3	4.05		0	0
1	15	20.27		1	1.35
2	43	58.11		17	22.97
3	7	9.46		10	13.51
4	3	4.05		17	22.97
5	2	2.70		10	13.51
6	1	1.35		10	13.51
7	0	0		5	6.76
8	0	0		1	1.35
9	0	0		1	1.35
10	0	0		2	2.70

Table 10. Frequency distribution of scores on Prompt 2.

	Pre-test		Post-test
Links in common	N	Percent	N	Percent
0	48	64.86	26	35.14
1	18	24.32	24	32.43
2	8	10.81	14	18.92
3	0	0	5	6.76
4	0	0	5	6.76
	Pre-test		Post-test
Keyword occurrence	N	Percent	N	Percent
0	30	40.54	6	8.11
1	17	22.97	12	16.22
2	19	25.68	18	24.32
3	8	10.81	20	27.03
4	0	0	16	21.62
5	0	0	2	2.70

Table 11. Descriptive statistics for manual scores.

	Pre-test				Post-test
	Mean	SD	Median	Skew.	Mean	SD	Median	Skew.
Prompt 1	1.811	1.354	2.000	2.498	6.065	2.927	6.000	0.842
Prompt 2	0.297	0.421	0.000	1.741	1.318	1.318	1.000	0.898

NOTE: N = 74

Table 12. Change in scores with Wilcoxon signed rank tests.

	Change (post-pre)
	Q1	Median	Q3	S	Unadjusted p-value
Prompt 1
Links in common	0	1	2	827.5	<0.0001
Keyword occurrence	1	2	3	901	<0.0001
Manual score	2	4	6	1307.5	<0.0001
Prompt 2
Links in common	0	1	2	431	<0.0001
Keyword occurrence	0	1	2	713.5	<0.0001
Manual score	0	0.5	2	611.5	<0.0001

NOTE: N = 74

Table 13. Polychoric correlations for Prompt 1.

	1.	2.	3.	4.	5.	6.
1. Links in common, pre-test
2. Keyword occurrence, pre-test	0.986^**
3. Manual score, pre-test	0.938^**	0.628^**
4. Links in common, post-test	0.595^**	0.292^*	0.288^*
5. Keyword occurrence, post-test	0.435^*	0.253^*	0.296^*	0.790^**
6. Manual score, post-test	0.309	0.188	0.159	0.665^**	0.787^**
7. Final exam score	0.060	0.185	0.160	0.342^*	0.411^*	0.452^**

NOTE: N = 74,

* p-value ≤ 0.05,

** p-value ≤ 0.001.

Table 14. Polychoric correlations for Prompt 2.

	1.	2.	3.	4.	5.	6.
1. Links in common, pre-test
2. Keyword occurrence, pre-test	0.989^**
3. Manual score, pre-test	0.292	0.435^*
4. Links in common, post-test	0.240	0.195	0.161
5. Keyword occurrence, post-test	0.255	0.182	0.132	0.911^**
6. Manual score, post-test	0.115	0.087	0.092	0.579^**	0.655^**
7. Final exam score	0.060	0.187	0.046	0.276^*	0.226	0.286^*

NOTE: N = 74,

* p-value ≤ 0.05,

** p-value ≤ 0.001.