Are teachers consistently effective across subject matters? Revisiting the issue of differential teacher effectiveness

ABSTRACT

Although consistency in teacher effectiveness has attracted scholarly interest since the 1960s, revived interest in differential teacher effectiveness is recently witnessed, with empirical studies more systematically testing this assumption. Most of these studies, however, focus on cognitive outcomes in “core” subject matters. Drawing on 2 subject matters with differences in teaching context and targeted learning objectives, mathematics and physical education, in this study, we examined how consistent 18 elementary schoolteachers were in promoting student cognitive and psychomotor learning outcomes; we also investigated the consistency in instructional quality across their lessons, as measured by a generic observational instrument. The study provides evidence supporting teacher differential effectiveness across curriculum areas. We discuss the practical implications of the study findings for summative and formative teacher evaluation, their theoretical implications for developing differential models of teaching effectiveness, and their methodological implications for exploring teacher effectiveness using value-added models and classroom observation data.

KEYWORDS:

• Classroom observations
• cognitive learning outcomes
• differential teacher effectiveness
• psychomotor learning outcomes
• value-added models

Acknowledgements

The authors would like to thank the teachers who participated in the study.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Charalambos Y. Charalambous is an Assistant Professor in Educational Research and Evaluation at the Department of Education of the University of Cyprus. His research interests include measuring and understanding teaching quality and factors contributing to it, with a particular focus on teachers’ practices (generic and content specific) and teachers’ use of personal and contextual resources.

Ermis Kyriakides holds a PhD in Physical Education from the Department of Education of the University of Cyprus. His research interests lie in the field of teaching effectiveness in Physical Education. He is interested in exploring how teacher behaviors, and especially teaching practices (generic or content specific), can influence student learning in physical education.

Leonidas Kyriakides is Professor of Educational Research and Evaluation at the Department of Education of the University of Cyprus. His main research interests are in the area of school effectiveness and school improvement and especially in modeling the dynamic nature of educational effectiveness and in using research to promote quality and equity in education. Leonidas acted as chair of the AERA SIG on School Effectiveness and Improvement and of the EARLI SIG on Educational Effectiveness.

Niki Tsangaridou is a Professor in Physical Education at the Department of Education of the University of Cyprus. The focus of her research is on teacher reflection and reflective teaching, teacher education and learning to teach, and teaching practices in Physical Education.

ORCID

Charalambos Y. Charalambous http://orcid.org/0000-0002-0051-6926

Notes

1 During the 1990s, a series of studies on consistency across subjects has been conducted, but these focused on school effectiveness rather than on teacher effectiveness (cf. Scheerens & Bosker, 1997, pp. 88−91). For elementary education, consistency reported (largely across mathematics and reading) ranged from r = 0.51 to r = 0.73 for value-added-based school effects. For secondary education, consistency was even lower (around r = 0.40 to r = 0.46; one study also showed that 40% of the school effect was due to the subject being taught).

2 Interpreting the results of this study, Lipowsky (2005), however, counter-argued that they provide more evidence of consistency than inconsistency, since the ratio of consistent to inconsistent teachers was 2:1.

3 This hypothesis resonates with the distinction between generic and content-specific practices (see Charalambous & Kyriakides, 2017). Because the first type of practices (e.g., classroom management, motivational support) is assumed to cut across different subject matters, one would expect to find more consistent results when examining these practices. Content-specific practices (e.g., cognitive activation, content explicitness during instruction), on the other hand, are thought to be more pertinent to particular subject matters or have particular manifestations in these subject matters; from this respect, more inconsistencies could be expected to emerge when studying these practices.

4 Although one could raise concerns as to why we are focusing on PE since it is often considered a “peripheral” subject matter, recently there has been increased emphasis on school PE worldwide (European Commission/EACEA/Eurydice, 2013; UNESCO, 2014).

5 The sample was limited to these four grades due to difficulties in recruiting first- and second-grade teachers who taught the two subject matters under consideration.

6 At the time of the study, the allocated time for PE was two 40-min periods for all grade levels. However, during the next school year (2015–2016), a decision was made to increase the allocated PE time for the fifth and sixth graders to three 40-min periods per week.

7 This design decision was informed by prior research findings both in mathematics (Hill, Charalambous, & Kraft, 2012) and in PE (Charalambous, Kyriakides, Tsangaridou, & Kyriakides, 2017) suggesting that at least three lessons are needed per teacher for obtaining reliable estimates of instructional quality.

8 The use of the low- rather than the high-inference DMEE observation instrument was informed by prior research (e.g., Azigwe, Kyriakides, Panayiotou, & Creemers, 2016) suggesting the promise of the data yielded from this instrument for detecting the effects of the DMEE factors on student learning as compared to data derived from the high-observation instrument. Because of this decision, we were not able to examine two of the DMEE factors (time management and classroom as a learning environment), which are tapped by the high-inference instrument. The eighth factor, assessment, was not examined, because its examination requires observing more (sequences of) lessons than those considered.

9 To increase power, we utilized the database from the entire project which included 944 students (the additional 612 students were taught by additional 50 teachers teaching only either mathematics or PE as well as six teachers teaching both subject matters yet to different students).

10 Reliabilities over r = 0.75, infit and outfit mean squares close to one, and infit and outfit t’s close to zero show acceptable model fit (cf. Bond & Fox, 2012).

11 An alternative (and more preferable approach) would be to run a multivariate-multilevel model with students’ scores in mathematics and PE at Level 1, students at Level 2, and teachers at Level 3. The correlation of the variances at Level 3 could be used as an indicator of the consistency between the two subject matters. When we ran this model, we had problems of convergence, apparently due to power issues.

12 Low: standardized residual < −2SE; average: −2SE ≤ standardized residual ≤2SE; and high: standardized residual > 2SE.

13 An exploratory factor analysis run with all the data of both subject matters provided strong evidence of unidimensionality, given that the ratio of the eigenvalue of the first factor (54.56) to the eigenvalue of the second factor (0.39) was way beyond 2 – which is often taken as a threshold for testing the tenability of the unidimensionality assumption of IRT models (cf. Bond & Fox, 2012; Hambleton, Swaminathan, & Rogers, 1991, p. 56).

14 Similar to our VAM approach, for power reasons, the scale was developed by considering all project teachers: the 18 teachers teaching both mathematics and PE discussed herein, 25 teachers who were teaching only mathematics, 25 teachers who were teaching only PE, and six teachers who were teaching both subject matters, yet to different students. This resulted in using 147 PE lessons and 147 mathematics lessons.

15 Admittedly, the instructional practices explored in this study and those considered in prior work are different (e.g., we did not examine classroom management and motivational support as in Praetorius et al., 2016). However, because both this study and Praetorius et al.’s (2016) work drew on practices thought to be generic, one could expect the results of the two studies to be more comparable.

Additional information

Funding

The research reported in this manuscript was supported by the University of Cyprus/Leventis foundation under Grant [number 8037P-12083].