Benchmark rating procedure, best of both worlds? Comparing procedures to rate text quality in a reliable and valid manner

Figures & data

Table 1. Means, SD and Cronbach’s Alpha Reliability for Holistic, Benchmark, and Analytic Ratings.

		Task Yummy				Task Smurf
Rating procedure	Scale	N	Mean	SD	α	N	Mean	SD	α
Holistic	0–10	4	5.93	1.67	.79	7	4.94	1.86	.91
Benchmark	0 - ∞	7	91.00	20.48	.93	5	82.59	19.88	.82
Analytic	0–15	7	8.76	3.01	.93	6	8.30	2.72	.87

Table 2. Average Reliability Coefficients for One, Two and Three Raters.

	Yummy			Smurf
	Reliability (ρ)			Reliability (ρ)
Rating procedure	1 rater	2 raters	3 raters	1 rater	2 raters	3 raters
Holistic scale	.48	.65	.74	.60	.75	.82
Benchmark scale	.64	.78	.84	.48	.65	.74
Analytic scale	.64	.78	.84	.54	.70	.78

Table 3. Correlations between Holistic, Benchmark, and Analytic Ratings.

	Yummy			Smurf
	Holistic	Benchmark	Analytic	Holistic	Benchmark	Analytic
Holistic	−	.90	.83	−	.83	.63
Benchmark	1.00	−	.86	.96	−	.62
Analytic	.97	.93	−	.86	.73	−

Note. Uncorrected correlations are above the diagonal and attenuated correlations are below the diagonal.

Table 4. Variance Components as Proportions of the Total Variance for each Rating Procedure.

Source	Holistic	Benchmarks	Analytic
Student (s)	.23	.33	.28
Task (t)	.06	.06	.00
Student by task (st)	.10	.18	.27
Rater within task (r:t)	.35	.08	.08
Student by rater within tasks, and error (s(r:t),e)	.26	.35	.37

Figure 1. Estimated generalisability of writing scores for one task (solid lines) and four tasks (dashed lines), with varying number of raters. Lines represent the rating procedure that is used by raters, b: benchmark rating procedure, a: analytic rating procedure, h: holistic rating procedure.

Table 5. Means, SD and Reliability by Benchmark Condition and Text Sample.

Similarity	Yummy benchmark scale				Smurf benchmark scale
	N	Mean	SD	α	N	Mean	SD	α
Topic + genre	5	89.87	18.18	.92	5	94.09	14.53	.91
Genre	5	90.50	17.44	.90	5	88.85	17.81	.92
None	5	90.83	15.95	.87	5	88.43	15.97	.84