Analytic or holistic? A study about how to increase the agreement in teachers’ grading

Figures & data

Table 1. Participants and conditions

	Analytic grading	Holistic grading
EFL	24	18
Mathematics	17	15

Table 2. The assignments

Task no.	EFL	Mathematics
1	Writing a biography of a relative or a friend of the family.	Eight simple tasks in arithmetic.
2	Writing an argumentative text about food waste.	Four short problem-solving tasks.
3	Writing a text about what a friend should be like.	One open-ended task that could be solved in many different ways. (written response)
4	Writing a short text message (SMS) to someone you care (or cared) about.	One open-ended task that could be solved in many different ways. (oral response)

Figure 1. Typical example of justification and categorisation (in square brackets). Translated from Swedish by the authors

Table 3. Comparison of agreement and correlations between the conditions

	EFL		Mathematics
	Analytic	Holistic	Analytic	Holistic
Fleiss’ kappa	.401	.317	.273	.195
Spearman’s rho (Std. deviation)	.929 (.040)	.952 (.057)	.893 (.110)	.919 (.102)
Consensus agreement (MAD)^1	66.7 (.375)	59.7 (1.00)	57.4 (.250)	51.7 (.500)

¹Median absolute deviation (1 = The mean difference is one grade level, e.g. from E to F).

Table 4. Summary of teachers’ references in relation to the criteria for students and conditions (EFL)

	Student 1	Student 2	Student 3	Student 4	Analytic (mean)	Holistic (mean)	Sum
Mechanics	13	16	7	11	19 (0.8)	28 (1.6)	47
Grammar	25	26	23	27	56 (2.3)	45 (2.5)	101
Organisation	24	25	16	31	45 (1.9)	51 (2.8)	96
Content	5	18	3	8	14 (0.6)	20 (1.1)	34
Style	66	96	67	78	149 (6.2)	158 (8.8)	307
Voice	6	20	5	7	22 (0.9)	16 (0.9)	38
Comprehensibility	30	5	28	12	39 (1.6)	36 (2.0)	75
Rigour	4	10	10	13	17 (0.7)	20 (1.1)	37
Student	10	4	10	10	15 (0.6)	19 (1.1)	34
Only grades	4	3	3	8	16 (0.7)	2 (0.1)	18
Total (mean)	187	223	172	205	392 (16.3)	395 (21.9)	787

Table 5. Summary of teachers’ references in relation to the criteria for students and conditions (mathematics)

	Student 1	Student 2	Student 3	Student 4	Analytic (mean)	Holistic (mean)	Sum
Methods	17	15	16	17	37 (2.2)	28 (1.9)	65
Concepts	24	23	23	20	36 (2.1)	54 (3.6)	90
Problem solving	8	5	7	7	11 (0.6)	16 (1.1)	27
Communication	11	10	14	18	21 (1.2)	32 (2.1)	53
Reasoning	6	5	10	7	12 (0.7)	16 (1.1)	28
Student	8	9	9	6	16 (0.9)	16 (1.1)	32
Only grades	18	14	16	15	48 (2.8)	15 (1.0)	63
Total	92	81	95	90	181 (10.6)	177 (11.8)	358

Table 6. Comparison of teachers’ references in relation to the criteria for student 4 (mathematics)

	Analytic	Holistic
Methods	10	11
Concepts	7	13
Problem solving	4	3
Communication	9	9
Reasoning	4	3
Student	2	1
Only grades	10	5