Inter-method reliability of school effectiveness measures: a comparison of value-added and regression discontinuity estimates

Figures & data

Table 1. Year group and number within each cohort by time period.

	Time Period
	T1		T2		T3
	(2007/08)		(2008/09)		(2009/10)
Cohort	Year Group	N	Year Group	N	Year Group	N
A					Yr3	9,831
B			Yr3	13,132	Yr4	10,232
C	Yr3	13,356	Yr4	13,401	Yr5	10,469
D	Yr4	13,895	Yr5	14,031	Yr6	10,584
E	Yr5	13,964	Yr6	13,848	Yr7	10,081
F	Yr6	14,210	Yr7	14,555	Yr8	10,441
G	Yr7	14,673	Yr8	14,305	Yr9	9,738
H	Yr8	14,869	Yr9	14,129
I	Yr9	14,934
Total*		99,901		97,401		71,376

Notes: *There were 141,057 unique pupils with a recorded score in at least one year. Year groups in bold are those for which school effects are estimated as both longitudinal and cross-sectional sources of comparison are available.

Table 2. Summary of Making Good Progress performance data.

Performance by Time Period
MGP Variables Used in this Study	Obs	Mean	Std. Dev.	Min	Max
T1 Teacher-Assessed Mathematics Score^1	99,513	27.03	7.96	9	53
T2 Teacher-Assessed Mathematics Score^1	97,044	27.31	8.06	9	53
T3 Teacher-Assessed Mathematics Score^1	71,144	27.35	7.96	9	53
Key Stage 1 Average Point Score^2	13,5962	15.27	3.61	3	27
Free School Meals Eligible^3	20.1%
Gender Recorded as Male	52.3%

¹Teacher-assessed mathematics point score data from the Making Good Progress data were recorded in sublevels. Possible scores ranged from 9 to 53 in increments of 2.

²The Key Stage 1 average point scores were moderately fine grained, with 1/3 and 1/2 points recorded as well as integers. There was a marked ceiling effect at 21 points, affecting about 8% of pupils, and a marked spike at the expected score of 15 affecting just under 16% of pupils. The use of this variable was minimal.

³Free school meals eligibility is a binary variable used as an indicator of poverty in the English system.

Table 3. Added-year effects by national curriculum year.

	2007/2008			2008/2009			2009/2010
National Curriculum Year	Constant	Age effect (per month)	Annual Progress^1	Constant	Age effect (per month)	Annual Progress^1	Constant	Age effect (per month)	Annual Progress^1
3	17.3	0.16	–	17.3	0.16	–	17.4	0.15	–
4	19.8	0.17	2.50	20.0	0.17	2.71	20.2	0.17	2.76
5	22.8	0.17	3.08	22.8	0.18	2.79	23.1	0.19	2.88
6	26.4	0.14	3.55	26.6	0.15	3.82	26.9	0.16	3.82
7	28.9	0.17	2.53	29.5	0.14	2.89	29.5	0.19	2.60
8	31.6	0.14	2.65	32.1	0.18	2.66	32.4	0.15	2.94
9	34.5	0.18	2.90	35.4	0.14	3.22	35.2	0.18	2.75

¹Gross annual progress, including both the absolute school effect and the age effect.

Figure 1. Mean pupil performance (NC Scores) by pupil relative age in months.

Table 4. Estimated school (cohort) effects on attainment for each measurement design.

Measure	Obs	NC Year	Std. Dev.	Min	Max	Measure	Obs	NC Year	Std. Dev.	Min	Max
LRD_i*	271	4	0.9	–3.6	3.3	CVA_i	271	4	0.7	–2.2	2.2
LRD_ii	271	5	0.9	–3.3	3.2	CVA_ii	271	5	0.7	–2.2	2.6
LRD_iii	260	6	1.1	–3.4	3.2	CVA_iii	260	6	0.9	–2.9	3.1
LRD_iv	69	8	1.5	–3.8	4.7	CVA_iv	69	8	1.3	–3.2	3.0
LRD_v	68	9	1.4	–3.4	3.3	CVA_v	68	9	1.2	–3.3	2.8
LRD_vi	225	4	0.8	–2.2	3.1	CVA_vi	225	4	0.7	–1.9	2.7
LRD_vii	226	5	0.9	–3.6	2.3	CVA_vii	226	5	0.8	–2.8	1.9
LRD_viii	212	6	1.1	–3.0	3.4	CVA_viii	212	6	0.9	–2.7	2.4
LRD_ix	52	8	1.2	–3.8	2.5	CVA_ix	52	8	1.0	–3.3	2.1
LRD_x	49	9	1.3	–2.7	4.6	CVA_x	49	9	1.2	–2.8	4.4
RD1_i	271	4	0.4	–1.3	1.0	RD2_i	271	4	0.3	–1.3	1.1
RD1_ii	271	5	0.7	–1.9	1.7	RD2_ii	271	5	0.6	–1.6	2.0
RD1_iii	260	6	0.6	–1.6	2.4	RD2_iii	260	6	0.6	–1.6	2.5
RD1_iv	69	8	0.8	–1.9	2.2	RD2_iv	69	8	0.7	–1.9	1.7
RD1_v	68	9	0.9	–3.3	1.9	RD2_v	68	9	0.8	–3.5	1.8
RD1_vi	225	4	0.5	–1.4	1.3	RD2_vi	225	4	0.5	–1.2	1.0
RD1_vii	226	5	0.4	–1.2	1.3	RD2_vii	226	5	0.4	–1.3	1.1
RD1_viii	212	6	0.5	–1.4	1.5	RD2_viii	212	6	0.5	–1.4	1.7
RD1_ix	52	8	0.8	–1.6	2.3	RD2_ix	52	8	0.6	–1.3	1.9
RD1_x	50	9	1.1	–3.5	3.9	RD2_x	50	9	1.1	–3.4	4.4

*LRD were mean centred to clearly compare the size of differences in rates of progress by school.

Table 5. The correlation between the value-added measure and four regression discontinuity measures for the corresponding study/national curriculum year.

	Study Year	NC Year	RD1	RD2	LRD
CVAi	2	4	0.41	0.42	0.92
CVAvi	3	4	0.50	0.58	0.94
CVAii	2	5	0.55	0.69	0.93
CVAvii	3	5	0.50	0.45	0.96
CVAiii	2	6	0.39	0.40	0.95
CVAviii	3	6	0.45	0.51	0.96
CVAiv	2	8	0.56	0.53	0.92
CVAix	3	8	0.65	0.54	0.93
CVAv	2	9	0.28	0.28	0.97
CVAx	3	9	0.64	0.63	0.97

Table

	Year 3–4	Year 4–5	Year 5–6	Year 7–8	Year 8–9	Year 3–4	Year 4–5	Year 5–6	Year 7–8	Year 8–9
	(i)	(ii)	(iii)	(iv)	(v)	(vi)	(vii)	(viii)	(ix)	(x)
	T2 Maths Score					T3 Maths Score
Age Within Year in Months	0.170	0.177	0.167	0.155	0.158	0.166	0.185	0.177	0.159	0.161
	(0.006)	(0.007)	(0.007)	(0.010)	(0.011)	(0.007)	(0.008)	(0.008)	(0.012)	(0.013)
Gross Upper Year Effect	2.880	2.982	3.814	2.769	3.289	3.024	3.031	3.795	2.761	2.999
	(0.072)	(0.092)	(0.100)	(0.153)	(0.174)	(0.084)	(0.092)	(0.109)	(0.181)	(0.234)
Gender (is male)	–0.300	–0.244	–0.441	–0.316	–0.169	–0.129	–0.377	–0.257	–0.219	–0.194
	(0.057)	(0.067)	(0.073)	(0.099)	(0.113)	(0.064)	(0.074)	(0.083)	(0.120)	(0.130)
Free School Meals (FSM)	–1.447	–1.782	–2.121	–2.443	–2.733	–1.484	–1.759	–2.231	–2.279	–2.674
	(0.076)	(0.085)	(0.095)	(0.121)	(0.141)	(0.085)	(0.096)	(0.107)	(0.141)	(0.154)
Upper Year*Gender	0.063	–0.200	–0.091	0.172	–0.147	–0.240	0.120	–0.085	0.043	0.007
	(0.080)	(0.093)	(0.103)	(0.139)	(0.159)	(0.090)	(0.104)	(0.117)	(0.167)	(0.187)
Upper Year*FSM	–0.350	–0.297	–0.050	–0.296	–0.350	–0.280	–0.437	0.168	–0.442	–0.186
	(0.103)	(0.119)	(0.133)	(0.172)	(0.201)	(0.117)	(0.132)	(0.148)	(0.196)	(0.223)
Constant	17.62	20.46	23.48	29.89	32.66	17.72	20.65	23.73	30.21	32.97
	(0.078)	(0.086)	(0.110)	(0.203)	(0.268)	(0.089)	(0.102)	(0.114)	(0.223)	(0.299)
Observations	26,253	27,319	27,831	28,842	28,414	19,897	20,628	21,013	20,499	20,168
Number of groups	271	271	276	69	69	226	226	230	52	52

Standard errors in parentheses.

Table

	Year 3–4	Year 4–5	Year 5–6	Year 7–8	Year 8–9	Year 3–4	Year 4–5	Year 5–6	Year 7–8	Year 8–9
	(i)	(ii)	(iii)	(iv)	(v)	(vi)	(vii)	(viii)	(ix)	(x)
	T2 Maths Score					T3 Maths Score
Prior Maths Score (T1/T2)	0.758	0.813	0.732	0.797	0.868	0.743	0.859	0.786	0.856	0.900
	(–0.008)	(–0.007)	(–0.006)	(–0.007)	(–0.006)	(–0.009)	(–0.008)	(–0.007)	(–0.007)	(–0.007)
Key Stage 1 Attainment	0.276	0.301	0.263	0.305	0.341	0.266	0.223	0.263	0.263	0.237
	(–0.007)	(–0.008)	(–0.008)	(–0.011)	(–0.012)	(–0.008)	(–0.008)	(–0.009)	(–0.011)	(–0.013)
Free School Meals (FSM)	–0.181	–0.240	–0.067	–0.360	–0.296	–0.070	–0.103	–0.046	–0.199	–0.072
	(–0.041)	(–0.047)	(–0.050)	(–0.072)	(–0.078)	(–0.044)	(–0.047)	(–0.052)	(–0.074)	(–0.084)
Gender (is male)	–0.205	–0.292	–0.293	–0.272	–0.427	–0.351	–0.179	–0.193	–0.293	–0.200
	(–0.031)	(–0.036)	(–0.038)	(–0.057)	(–0.059)	(–0.032)	(–0.036)	(–0.039)	(–0.062)	(–0.068)
Constant	3.460	2.870	6.586	5.088	3.663	4.255	3.034	5.517	3.979	3.373
	(–0.115)	(–0.129)	(–0.135)	(–0.233)	(–0.235)	(–0.124)	(–0.134)	(–0.146)	(–0.241)	(–0.271)
Observations	12,225	12,833	12,697	13,076	12,934	9,418	9,693	9,785	9,374	8,688
Number of groups	271	271	260	69	68	225	226	212	52	49

Standard errors in parentheses