Assessing teacher practice and development: the case of comprehensive literacy instruction

Abstract

In this paper, we report on 2 studies developing, testing, and using an observation tool for measuring primary literacy instruction, the Developing Language and Literacy Teaching (DLLT) rubrics. In Study 1 (an instrumentation study), we show that the DLLT has a high level of internal consistency, that there are high levels of inter-rater reliability when the tool is used by in-school coaches, that the items order consistent with a hypothesized developmental trajectory, and that the DLLT can distinguish between novice and more experienced teachers at one time point. In Study 2 (a 3‐year longitudinal study), we show that the DLLT is sensitive to growth in teaching practice, that this growth is related to exposure to one-on-one coaching, and that teacher practice as measured by the DLLT is related to teachers' value added to student achievement year by year.

Keywords:

• measurement
• teaching quality
• observation rubric

Acknowledgements

The work described in the current article was supported by a Teacher Quality Grant from the Institute for Educational Sciences (IES), R305M040086. We are appreciative of the support provided by IES. All errors of fact, omission, and/or interpretation are solely the authors' responsibility. The research team involved in this study included affiliates of the coaching program being investigated. This collaboration informed the study design and the development of tools to measure teacher practice. The analytical team worked independently from those affiliated with the program to ensure objectivity.

Notes

1. The complete rubric can be found online ( http://www.carnegiefoundation.org/files/DLLT_20120904.pdf).

2. Note that Atteberry and Bryk (2011) reported that teachers received 0.79 one-to-one coaching sessions per eligible month with a standard deviation of 0.63. In the analyses reported on here, we use averages from an HLM model that produced an average estimate for each teacher in each year, adjusted using Empirical Bayes.

3. Field observation by study staff suggested some upward scale drift by coaches toward the end of the 1st year of coaching activity. Consequently, study staff retrained coaches during the summer of 2006, and the inflated scores reported in the spring dropped down to more realistic levels in the fall of 2006. While we include the spring 2006 data in our analyses, we add a fixed effect to control for this one-time measurement artefact.

4. In the analyses presented in this paper, we used Empirical Bayes estimates of average coaching per month in lieu of the raw coach reports. By doing so, we were able to use estimates of each individual's exposure adjusted for possible error in the coach's report.

5. Comprehensive literacy learning theory suggests that students' acquisition of reading skill is influenced by both reading and writing instruction. For this reason, the DLLT includes elements from both reading and writing instruction. The study only used standardized reading measures as outcome variables, since reliable writing tests were not available for early primary grades.