Goals, data use, and instruction: the effect of a teacher professional development program on reading achievement

Abstract

In this paper, we investigated whether student reading comprehension could be improved with help of a teacher Professional Development (PD) program targeting goals, data use, and instruction. The effect of this PD program on 2nd- and 3rd-grade student achievement was examined using a pretest-posttest control group design. Applying propensity score matching, 35 groups in the experimental condition were matched to 35 control groups. Students in the experimental condition (n = 420) scored significantly higher on a standardized assessment than the control condition (n = 399), the effect size being d = .37. No differential effects of the PD program were found in relation to initial reading performance or grade. Different model specifications yielded similar albeit smaller effect sizes (d = .29 and d = .30). At the end of the program, students in the experimental condition were more than half a year ahead of students in the control condition.

Keywords:

• program effectiveness
• reading achievement
• propensity score matching
• hierarchical linear modeling

Acknowledgements

The authors would like to express their gratitude to Henk Guldemond, who prepared the software needed to conduct the propensity score matching.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. Our PD program was the only one explicitly including the school principal and internal support coordinator (in addition to the participating teachers). This is why, for the other PD programs as part of the larger series of intervention studies, we did not expect any “contamination” of the second- and third-grade teachers via the school principal or internal support coordinator.

2. This relatively small number of third-grade classes can be explained by the use of the multisubject test at the end of third grade to obtain an end-of-the-school-year measurement of student reading comprehension: The use of this test is detailed in the instruments section of the current paper. A precondition for inclusion of a third-grade class in the pool of possible controls was that this additional multisubject test had been administered.

3. There are different matching algorithms, and each matching algorithm has its own advantages and disadvantages – for more information, see Caliendo and Kopeinig (2008) and Heinrich, Maffioli, and Vazquez (2010). “It should be clear that there is no ‘winner’ for all situations […]. Pragmatically, it seems sensible to try a number of approaches. Should they give similar results, the choice may be unimportant” (Caliendo & Kopeinig, 2008, pp. 44–45). We tested several options and found robust results in terms of balance among the variables used to estimate the propensity score.

4. Even though the group-level is not the same as the classroom level (as some groups were part of a multigrade class), we refer to the group/teacher level for sake of simplicity as the results were found to be similar when higher levels of nesting were accounted for. More complex hierarchical models, including students nested in groups, nested in classrooms/teachers, nested in schools, were fitted to account for this structure of the data. The results of these models were almost the same as those of the model reported in this paper; the effect of the program remained significant and had a similar size.

5. The total unexplained variance of the start model is 116, and the total unexplained variance of the main effect model is 112.95 (see Table 4). R² = 1 – [112.95/116] = 0.0263.

6. This is an application of Cohen’s (1988) formula of d = to a multilevel setting, for which we are interested in the variation within groups (i.e., Level 1).

7. Cohen (1988) provides the following guideline for the interpretation of effect sizes: d = 0.2 is considered to be a small effect, d = 0.5 a medium effect, and d = 0.8 a large effect.

8. At the student level, outliers were defined as values with standardized scores lower than z = ‒3.29 or larger than z = 3.29 (Tabachnick & Fidell, 2001). Three students could be flagged as having an outlying score after fitting the model, which was caused by an extremely high result on the posttest. For the standardized residuals at the classroom level, we used a stricter z-score criterion, as outlying cases at the classroom level may influence the model more substantially than outlying cases at the pupil level (Rasbash, Steele, Browne, & Goldstein, 2012). In total, four classes with a standardized residual below z = ‒2 or above z = 2 were modeled separately. In addition to using this z-score approach to outlier identification, we also checked the influence of outliers with help of a method proposed by Tukey (1977) which makes use of P25 (the first quartile), P75 (the third quartile), and the Inter-Quartile Range (IQR): Here, outliers have values below P25 – 1.5 x IQR and above P75 + 1.5 x IQR. This approach yielded a larger effect size for the program, with d = .40, 90% CI [.23; .58].

Additional information

Notes on contributors

Mechteld F. van Kuijk

Mechteld van Kuijk is a researcher and lecturer at GION education/research, University of Groningen (The Netherlands). Her research interests include teacher professional development, data use, reading comprehension, goal setting, and educational effectiveness.

Marjolein I. Deunk

Marjolein Deunk is a researcher and lecturer at GION education/research, University of Groningen (The Netherlands). Her research focuses on language, reading, and literacy development in educational contexts, data use, differentiation, and small primary schools.

Roel J. Bosker

Roel Bosker is a full Professor of Education and director of GION education/research, University of Groningen (The Netherlands). His research interests include equity issues in education, educational effectiveness, quality care and school self-evaluation, program evaluation, comparative education, and multilevel modeling.

Evelien S. Ritzema

Evelien, also known as Lieneke, Ritzema is a PhD student at GION education/research, University of Groningen (The Netherlands). Her research focuses on teacher professional development, goal setting, data use, differentiation, and mathematics.