The Sensitivity of Value-Added Estimates to Specification Adjustments: Evidence From School- and Teacher-Level Models in Missouri

Figures & data

Table 1. Basic descriptive statistics. School-level data

	Full model	One-test model
% Free/reduced lunch	42.5	43.3
% English as a 2nd language	1.8	2.0
% Special education	11.9	12.0
% Disadvantaged minority	19.9	20.5
% Female	49.3	49.1
Schools	348	348
Students (math)	133,773	146,938
Students (com arts)	133,535	146,896

NOTE: Disadvantaged minorities are defined as being American Indian, black or Hispanic. Student characteristics are calculated using all students who are included in either the math or communication-arts model. Some grade-6 students in Missouri attend elementary schools but most attend middle schools. Grade-6 students in middle schools are included in our analysis.

Table 1A. Basic descriptive statistics. Teacher-level data

	Full model	One-test model
% Free/reduced lunch	47.1	47.4
% English as a 2nd language	4.0	4.5
% Special education	13.6	13.5
% Disadvantaged minority	46.6	46.6
% Female	49.4	49.3
Teachers (math)	289	289
Teachers (com arts)	388	390
Students (math)	18,910	20,871
Students (com arts)	19,150	21,129

NOTE: Disadvantaged minorities are defined as being American Indian, black or Hispanic. Student characteristics are calculated using all students who are included in either the math or communication-arts model.

Table 2. Model descriptions

	Model 0	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7
No. of lagged test scores	3	3	3	3	1	1	1	1
Free/reduced lunch status	X		X		X		X
Language status	X	X	X		X	X	X
Special-education status	X	X	X		X	X	X
Race	X	X			X	X
Gender	X	X			X	X

NOTE: The two-step models include unit aggregates that correspond to the set of student characteristic listed for each model (including the test score histories).

Table 3. Correlation Table for school-level VAM estimates. All models are compared to Model 0 using a fixed estimation sample (within subject and schooling-level)

	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7
One-step model
Math	0.997	0.999	0.996	0.922	0.911	0.913	0.903
Communication Arts	0.995	0.990	0.983	0.918	0.898	0.898	0.881
Two-step model
Math	0.992	0.998	0.978	0.974	0.966	0.969	0.947
Communication Arts	0.979	0.991	0.955	0.971	0.954	0.960	0.928

Table 3A. Correlation Table for teacher-level VAM estimates. All models are compared to Model 0 using a fixed estimation sample (within subject and schooling-level)

	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6	Model 7
One-step model
Math	0.999	0.998	0.994	0.934	0.931	0.922	0.910
Communication arts	0.998	0.990	0.979	0.935	0.928	0.909	0.884
Two-step model
Math	0.996	0.995	0.977	0.969	0.965	0.966	0.951
Communication arts	0.993	0.991	0.944	0.932	0.919	0.917	0.863

Table 4. Correlations between school-level VAM estimates and school-level averages for key student characteristics (Math)

	Correlation with	Correlation with	Top 25%	Bottom 25%	Top 25%	Bottom 25%
	avg. FRL	avg. lagged score	avg. free lunch	avg. free lunch	avg. lagged score	avg. lagged score
One-step model
Model 0	−0.14*	0.18**	47.0	56.4	0.04	−0.11
Model 1	−0.20**	0.22**	46.2	57.3	0.05	−0.13
Model 2	−0.16**	0.20**	47.1	58.4	0.04	−0.16
Model 3	−0.21**	0.23**	46.7	58.3	0.04	−0.15
Model 4	−0.17**	0.24**	47.6	57.7	0.06	−0.14
Model 5	−0.27**	0.29**	44.2	58.6	0.09	−0.16
Model 6	−0.25**	0.34**	46.2	59.1	0.08	−0.18
Model 7	−0.33**	0.38**	43.6	60.2	0.10	−0.18
Two-step model
Model 0	0.06	−0.05	51.3	52.4	−0.04	−0.04
Model 1	0.02	−0.07	50.4	52.6	−0.04	−0.00
Model 2	0.06	−0.05	52.1	52.0	−0.05	−0.04
Model 3	0.02	−0.07	51.7	53.0	−0.07	−0.01
Model 4	0.07	−0.05	52.4	52.6	−0.03	−0.03
Model 5	0.02	−0.07	51.9	52.7	−0.05	−0.01
Model 6	0.07	−0.05	52.8	51.7	−0.03	−0.01
Model 7	0.02	−0.07	51.2	52.4	−0.06	−0.02

**Correlation is significant at the 1% level. *Correlation is significant at the 5% level. NOTE: The by-quartile averages are calculated as simple averages across schools. There are strong negative correlations in the data between school size and free-lunch status and test scores (many small, rural schools in Missouri have high free-lunch shares and low scores), which explains why the simple averages for free/reduced-price lunch share presented here are large compared to what we report for the student-level sample in Table 1. Average test scores for the top and bottom quartiles are centered on the school-level sample average for ease of presentation.

Table 4A. Correlations between teacher-level VAM estimates and teacher-level averages for key student characteristics (Math)

	Correlation with	Correlation with	Top 25%	Bottom 25%	Top 25%	Bottom 25%
	avg. FRL	avg. lagged score	avg. free lunch	avg. free lunch	avg. lagged score	avg. lagged score
One-step model
Model 0	−0.28**	0.34**	40.8	65.8	0.21	−0.31
Model 1	−0.32**	0.37**	40.7	66.1	0.20	−0.30
Model 2	−0.32**	0.37**	39.9	65.2	0.24	−0.30
Model 3	−0.36**	0.39**	39.3	67.2	0.23	−0.33
Model 4	−0.32**	0.40**	40.7	67.9	0.25	−0.39
Model 5	−0.38**	0.44**	39.8	70.3	0.27	−0.42
Model 6	−0.41**	0.47**	36.5	71.1	0.33	−0.45
Model 7	−0.48**	0.52**	34.5	71.6	0.36	−0.46
Two-step model
Model 0	0.04	−0.03	51.3	56.3	−0.05	−0.10
Model 1	−0.00	−0.02	50.0	58.5	−0.04	−0.11
Model 2	0.04	−0.04	50.8	57.2	−0.03	−0.10
Model 3	−0.01	−0.01	50.9	59.3	−0.04	−0.11
Model 4	0.06	−0.05	52.4	55.5	−0.06	−0.09
Model 5	0.01	−0.03	51.6	57.3	−0.05	−0.11
Model 6	0.05	−0.05	52.6	55.1	−0.05	−0.07
Model 7	−0.02	−0.01	52.1	58.8	−0.06	−0.09

**Correlation is significant at the 1% level. *Correlation is significant at the 5% level. NOTE: The by-quartile averages are calculated as simple averages across teachers. Average test scores for the top and bottom quartiles are centered on the teacher-level sample average.

Figure 1. Comparisons between output from Models 0 and 7 for the one-step and two-step models. Note: High FRL schools are defined as schools with 80% or more students eligible for free/reduced-price lunch. Low FRL schools are schools with 20% or fewer eligible students.

Table 5. Correlations between VAM estimates and school-level averages for key student characteristics (Communication arts)

	Correlation with	Correlation with avg.	Top 25% avg.	Bottom 25% avg.	Top 25% avg.	Bottom 25% avg.
	avg. FRL	lagged score	free lunch	free lunch	lagged score	lagged score
One-step model
Model 0	−0.22**	0.27**	45.8	57.6	0.07	−0.12
Model 1	−0.30**	0.31**	43.2	59.3	0.09	−0.14
Model 2	−0.28**	0.34**	44.3	58.9	0.09	−0.16
Model 3	−0.34**	0.37**	43.5	59.9	0.09	−0.15
Model 4	−0.31**	0.38**	43.3	57.4	0.11	−0.13
Model 5	−0.41**	0.43**	39.7	59.5	0.14	−0.14
Model 6	−0.40**	0.50**	41.1	60.6	0.15	−0.18
Model 7	−0.50**	0.54**	38.2	61.7	0.17	−0.18
Two-step model
Model 0	0.05	−0.03	51.3	49.9	−0.03	0.00
Model 1	−0.04	−0.07	49.4	52.9	−0.06	0.00
Model 2	0.04	−0.02	50.8	49.6	−0.03	−0.02
Model 3	−0.04	−0.07	49.0	53.3	−0.06	−0.00
Model 4	0.05	−0.02	51.6	50.2	−0.03	−0.00
Model 5	−0.04	−0.06	48.3	54.0	−0.04	−0.02
Model 6	0.04	−0.01	51.0	49.5	−0.02	0.00
Model 7	−0.05	−0.05	49.3	53.4	−0.06	−0.00

**Correlation is significant at the 1% level. *Correlation is significant at the 5% level. NOTE: The by-quartile averages are calculated as simple averages across schools. There are strong negative correlations in the data between school size and free-lunch status and test scores (many small, rural schools in Missouri have high free-lunch shares and low scores), which explains why the simple averages for free/reduced-price lunch share presented here are large compared to what we report for the student-level sample in Table 1. Average test scores for the top and bottom quartiles are centered on the school-level sample average for ease of presentation.

Table 5A. Correlations between teacher-level VAM estimates and teacher-level averages for key student characteristics (Communication arts)

	Correlation with	Correlation with avg.	Top 25% avg.	Bottom 25% avg.	Top 25% avg.	Bottom 25% avg.
	avg. FRL	lagged score	free lunch	free lunch	lagged score	lagged score
One-step model
Model 0	−0.34**	0.34**	38.0	62.6	0.18	−0.24
Model 1	−0.39**	0.38**	34.8	64.1	0.21	−0.28
Model 2	−0.43**	0.41**	33.6	66.1	0.23	−0.30
Model 3	−0.48**	0.44**	29.3	67.7	0.28	−0.33
Model 4	−0.43**	0.45**	34.7	67.2	0.24	−0.32
Model 5	−0.48**	0.49**	31.5	68.3	0.28	−0.34
Model 6	−0.55**	0.53**	27.6	70.6	0.33	−0.37
Model 7	−0.61**	0.59**	24.9	71.0	0.36	−0.38
Two-step model
Model 0	−0.08	0.08	47.1	51.1	−0.01	−0.07
Model 1	−0.12*	0.08	44.4	52.5	0.02	−0.06
Model 2	−0.08	0.10	46.8	52.6	0.01	−0.10
Model 3	−0.22**	0.09	41.6	57.1	0.04	−0.10
Model 4	−0.05	0.06	47.9	52.8	−0.01	−0.08
Model 5	−0.12*	0.06	48.3	54.3	−0.03	−0.07
Model 6	−0.06	0.09	48.4	52.6	−0.01	−0.09
Model 7	−0.27**	0.10*	40.8	60.0	0.00	−0.15

**Correlation is significant at the 1% level. *Correlation is significant at the 5% level. NOTE: The by-quartile averages are calculated as simple averages across teachers. Average test scores for the top and bottom quartiles are centered on the teacher-level sample average.

Table 6. Average free/reduced-lunch percentages in schools that finish in the top value-added quartile based on estimates from either the One-Step or Two-Step versions of Model 0, but not both

	N	F/RL%
One-step only—math	29	41.1
One-step only—communication arts	15	37.4
Two-step only—math	29	53.9
Two-step only—communication arts	15	69.2

NOTE: There are 348 schools included in our analysis (see Table 1).

Table 6A. Average free/reduced-lunch percentages for teachers that finish in the top value-added quartile based on estimates from either the one-step or two-step versions of Model 0, but not both

	N	F/RL%
One-step only—math	21	25.9
One-step only—communication arts	19	21.2
Two-step only—math	21	61.4
Two-step only—communication Arts	19	67.3

Table A.1. Correlation Table for school-level VAM estimates from single-lagged-score models using all available data. All models are compared to Model 4 using a fixed estimation sample

	Model 5	Model 6	Model 7
One-step model
Math	0.992	0.985	0.973
Communication arts	0.989	0.964	0.943
Two-step model
Math	0.991	0.997	0.974
Communication arts	0.981	0.987	0.954

Table A.1A. Correlation Table for teacher-level VAM estimates from single-lagged-score models using all available data. All models are compared to Model 4 using a fixed estimation sample

	Model 5	Model 6	Model 7
One-step model
Math	0.997	0.991	0.979
Communication arts	0.997	0.977	0.959
Two-step model
Math	0.995	0.996	0.974
Communication arts	0.981	0.986	0.909