A best-evidence meta-analysis of the effects of digital monitoring tools for teachers on student achievement

Figures & data

Figure 1. PRISMA flow diagram.

Table 1. Included studies.

	Authors (year)	Title	Subject	Student age (years)	Study duration	Research design	n students experimental group	n students control group	Effect size
1	Carlson et al. (2011)	A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement	Math Reading	8–14	1 school year	Randomized	≈ 135,439 ≈ 138,074	≈ 135,439 ≈ 138,074	0.17 (combined)
2	Cordray et al. (2012)	The impact of the Measures of Academic Progress (MAP) program on student reading achievement	Reading	9–11	2 school years	Randomized	1,149 (Grade 4) 701 (Grade 5)	765 (Grade 4) 1,105 (Grade 5)	−0.10 (combined)
3	Faber et al. (2017)	The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment	Math	8–9	5 months	Randomized	822	986	0.20
4	Faber & Visscher (2018)	The effects of a digital formative assessment tool on spelling achievement: Results of a randomized experiment	Language	8–9	5 months	Randomized	619	986	0.02
5	Förster & Souvignier (2014)	Learning progress assessment and goal setting: Effects on reading achievement, reading motivation and reading self-concept	Reading	9–10	6 months	Randomized	335	285	0.21
6	Konstantopoulos et al. (2013)	The impact of Indiana’s system of interim assessments on mathematics and reading achievement	Math Reading	5–14	1 school year	Randomized	≈ 11,344	≈ 6,587	0.14 (combined)
7	Konstantopoulos et al. (2016)	Effects of interim assessments on student achievement: Evidence from a large-scale experiment	Math Reading	5–14	1 school year	Randomized	≈ 15,559	≈ 15,558	−0.02 (combined)
8	May & Robinson (2007)	A randomized evaluation of Ohio’s Personalized Assessment Reporting System (PARS)	Language Math Reading	15–16	1 school year	Randomized	≈ 26,306	≈ 25,274	−0.92 (combined)
9	Nunnery & Ross (2007)	The effects of the School Renaissance program on student achievement in reading and mathematics	Math Reading	8–14	1 school year	Matching	416–482	448–510	0.12 (combined)
10	Ritzema (2015)	Professional development in data use: The effects of primary school teacher training on teaching practices and students’ mathematical proficiency	Math	7–9	1 school year	Matching	527	527	−0.08
11	Roschelle et al. (2016)	Online mathematics homework increases student achievement	Math	12–13	2 school years	Randomized	1,621	1,229	0.18
12	Slavin et al. (2013)	Effects of a data-driven district reform model on state assessment outcomes	Math Reading	8–14	4 school years	Randomized	≈ 61,505 (Grade 5) ≈ 42,885 (Grade 8)		0.34 (combined)
13	van der Scheer et al. (2018)	Effects of a data-based decision-making intervention for teachers on students’ mathematical achievement	Math	9–10	1 school year	Randomized	269	404	0.05
14	van Kuijk et al. (2016)	Goals, data use, and instruction: The effect of a teacher professional development program on reading achievement	Reading	7–9	1 school year	Matching	420	399	0.37

Note: Study duration = timespan between the pretest/start of the intervention and the posttest; sample size experimental/control group = reported in terms of student numbers if provided; if not, it was estimated (see Appendix 1).

Table 2. The features of the digital monitoring tools (DMTs) and the interventions.

Study	DMT (feedback) features				Intervention features			Group
	Frequency	Content	Class	Predictive	Duration/frequency	Content	Target
	1–2 times a year 3–7 times a year more than once a month	scores and progress and specific contents and instruction advice			1–2 per year 3–5 per year monthly	technical and translating into instruction and class/school support	teachers and principals and school board
1	3–7 times a year	scores	yes	yes	monthly	Translating into instruction	and school board	A
2	3–7 times a year	and specific contents	no	yes	3–5 per year	class/school support	teachers	B
3/4	more than once a month	and specific contents	yes	no	1–2 per year	technical (with a coach)	teachers	C C
5	more than once a month	and specific contents	no	no	no intervention implementation activities	technical	teachers	C
6/7	DMT Kindergarten–Grade 2: more than once a month DMT Grades 3–8: 3–7 times a year	and progress and instruction advice	yes yes	no yes	2–3 sessions (train-the-trainer approach) ^b 2–3 sessions (train-the-trainer approach) ^b	technical technical	teachers teachers	C B
8	1–2 times a year	and specific contents	yes	yes	2 sessions (train-the-trainer approach) ^b	technical	and school board	A
9	more than once a month	and specific contents	no	no	no information	translating into instruction	teachers	C
10	1–2 times a year ^a	and specific contents	yes	yes	monthly	class/school support	teachers and principals	B
11	more than once a month	and specific contents	yes	no	3–5 per year	class/school support	teachers	C
12	3–7 times a year	scores	yes	yes	monthly	class/school support	and school board	A
13	1–2 times a year ^a	and specific contents	yes	yes	monthly	class/school support	teachers	B
14	1–2 times a year ^a	and specific contents	yes	yes	monthly	class/school support	teachers and principals	B

^a Teachers were also stimulated to use students’ daily work and curriculum assignments; however, the focus of the intervention seemed to be on using the results of standardized assessments as provided by the DMT.

^b In each school, teachers were trained to train their colleagues. In Study 7, participants were mainly administrators, coordinators, or guidance counselors.

Figure 2. Forest plot of the effects found in the included studies.

Note: TE = treatment effect; seTE = standard error treatment effect; SMD = standardized mean difference; CI = confidence interval.

Table 3. The effect sizes for the included studies based on multilevel analyses.

			95% CI
Group	k effects	ES	LL	UL
Overall	14	0.12*	0.04	0.19
Primary education	16	0.14*	0.07	0.22
Secondary education	9	0.04	−0.04	0.12
Mathematics	13	0.10*	0.02	0.18
Reading	10	0.17*	0.05	0.28
Language	2	0.02	−0.05	0.09
Group A	8	0.25*	0.12	0.39
Group B	7	0.02	−0.12	0.15
Group C	10	0.13*	0.07	0.19

Note: CI = confidence interval; k effects = the number of effects included; ES = effect size; LL = lower limit; UL = upper limit.

*p < 0.05.

*Carlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398. https://doi.org/10.3102/0162373711412765

 Web of Science ®Google Scholar

Cordray, D., Pion, G., Brandt, C., Molefe, A, & Toby, M. (2012). The Impact of the Measures of Academic Progress (MAP) program on student reading achievement (NCEE 2013–4000). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

 Google Scholar

*Faber, J. M., Luyten, H., & Visscher, A. J. (2017). The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment. Computers & Education, 106, 83–96. https://doi.org/10.1016/j.compedu.2016.12.001

 Web of Science ®Google Scholar

*Faber, J. M., & Visscher, A. J. (2018). The effects of a digital formative assessment tool on spelling achievement: Results of a randomized experiment. Computers & Education, 122, 1–8. https://doi.org/10.1016/j.compedu.2018.03.008

 Web of Science ®Google Scholar

*Förster, N., & Souvignier, E. (2014). Learning progress assessment and goal setting: Effects on reading achievement, reading motivation and reading self-concept. Learning and Instruction, 32, 91–100. https://doi.org/10.1016/j.learninstruc.2014.02.002

 Web of Science ®Google Scholar

*Konstantopoulos, S., Miller, S. R., & van der Ploeg, A. (2013). The impact of Indiana’s system of interim assessments on mathematics and reading achievement. Educational Evaluation and Policy Analysis, 35(4), 481–499. https://doi.org/10.3102/0162373713498930

 Web of Science ®Google Scholar

*Konstantopoulos, S., Miller, S. R., van der Ploeg, A., & Li, W. (2016). Effects of interim assessments on student achievement: Evidence from a large-scale experiment. Journal of Research on Educational Effectiveness, 9(sup1), 188–208. https://doi.org/10.1080/19345747.2015.1116031

 Google Scholar

*May, H., & Robinson, M. A. (2007). A randomized evaluation of Ohio’s Personalized Assessment Reporting System (PARS). Consortium for Policy Research in Education. https://doi.org/10.12698/cpre.2007.pars

 Google Scholar

*Nunnery, J. A., & Ross, S. M. (2007). The effects of the School Renaissance program on student achievement in reading and mathematics. Research in the Schools, 14(1), 40–59.

 Google Scholar

*Ritzema, E. S. (2015). Professional development in data use: The effects of primary school teacher training on teaching practices and students’ mathematical proficiency [Doctoral dissertation, University of Groningen]. https://research.rug.nl/en/publications/professional-development-in-data-use-the-effects-of-primary-schoo

 Google Scholar

*Roschelle, J., Feng, M., Murphy, R. F., & Mason, C. A. (2016). Online mathematics homework increases student achievement. AERA Open, 2(4). https://doi.org/10.1177/2332858416673968

 Google Scholar

*Slavin, R. E., Cheung, A., Holmes, G., Madden, N. A., & Chamberlain, A. (2013). Effects of a data-driven district reform model on state assessment outcomes. American Educational Research Journal, 50(2), 371–396. https://doi.org/10.3102/0002831212466909

 Web of Science ®Google Scholar

*van der Scheer, E. A., & Visscher, A. J. (2018). Effects of a data-based decision-making intervention for teachers on students’ mathematical achievement. Journal of Teacher Education, 69(3), 307–320. https://doi.org/10.1177/0022487117704170

 Web of Science ®Google Scholar

*van Kuijk, M. F., Deunk, M. I., Bosker, R. J., & Ritzema, E. S. (2016). Goals, data use, and instruction: The effect of a teacher professional development program on reading achievement. School Effectiveness and School Improvement, 27(2), 135–156. https://doi.org/10.1080/09243453.2015.1026268

 Web of Science ®Google Scholar