What Can a Cognitive Coding Framework Reveal About the Effects of Professional Development on Classroom Teaching and Learning?

Abstract

We investigated what a cognitive framework for measuring classroom teaching could reveal about (a) the impact of a professional development (PD) program on teaching practices and (b) the relationship between teaching practices and student science learning. To conduct this study, we leveraged a collaboration between two entirely independent projects. The first project had developed a framework, the Teacher Tasks and Questions (TTQ) coding framework, for measuring teaching. The second project, the Science Teachers Learning From Lesson Analysis (STeLLA-1) project, possessed a rich body of teaching and student learning data. The two projects’ independence from each other allowed us to examine the generalizability of the TTQ framework and to investigate its validity. The TTQ measures were applied to the data set from the STeLLA-1 project. We examined whether the TTQ measures detected meaningful changes in teaching practice from pre-PD to post-PD and whether they predicted student learning. We found that the TTQ measures did detect changes in teaching practices that were consistent with STeLLA-1’s aims; furthermore, some TTQ measures also predicted student learning. We discuss the value of having generalizable measurement frameworks that can be used across different studies. We also glean lessons about data sharing across independent projects.

Acknowledgments

We thank researchers at the Biological Sciences Curriculum Study for sharing the Science Teachers Learning from Lesson Analysis data set. We thank research assistants Dr. Matthew Cunningham, Jonna Crocker, Jacqueline Shaib, Jonathan Van Luven, Mackenzie Whitfield, and Katelyn Wirtz for their work in coding lesson videos. We are also deeply grateful to Dr. Karie Huchting and the Journal of the Learning Sciences anonymous reviewers for their feedback on our manuscript.

Funding

This work was supported by Grant No. 201400019 from the Spencer Foundation.

Notes

¹ There have been two STeLLA studies. The current article drew on data from the first study; the second study was under way when this article was in preparation. To distinguish between the two studies, we refer to the STeLLA program described in the current article as STeLLA-1.

² The STeLLA-1 study was conducted by LessonLab Research Institute and California State Polytechnic University, Pomona. The data moved to the BSCS when LessonLab Research Institute was closed and Kathleen J. Roth (the principal investigator of the STeLLA-1 study) moved to BSCS.

³ By Vandana Thadani, this article’s first author.

⁴ For readers familiar with cultural-historical activity theory (CHAT), the CHAT analytical framework (Norton, McRobbie, & Ginns, 2007; Yamagata-Lynch, 2010) can be used to frame many of the ideas in this paragraph and to understand how TTQs might operate during lessons. An extensive overview of CHAT is beyond the scope of this article. However, in CHAT language, the object of the lesson refers to the goals and purposes for which lesson activities are completed. The object here would include the teachers’ major learning objectives and students’ interpretation of those objectives. Given that teachers had taken part in PD with very specific aims, the aims of the PD (described in “the section titled The STeLLA-1 PD Program”) might have been part of the object, as perceived by teachers. TTQs, through the lens of CHAT, would comprise discourse-based tools (i.e., language and physical tools that mediate how participants engage in an activity). TTQs can be used to clarify the object, set rules (e.g., expectations for how students engage with lesson material), and establish a division of labor (roles and responsibilities of the subjects, in this case teachers/students/student groups). Yamagata-Lynch (2010) stated that there is “no guarantee that the activity will become robust” (p. 17). Indeed, “an activity [can] put the subject in contradictory situations that can preclude achieving the object” (p. 23). In other words, whatever the initial or stated object of lesson activities is, complex interactions among subjects’ goals, existing skills, knowledge, and prior history might alter the effects of the activity. We argue that TTQs are measurable language-based tools through which teachers can communicate goals, expectations, and rules, and as such they provide a means for understanding how and why the larger stated object of lesson activities was/was not achieved successfully.

⁵ See http://www.bscs.org/science-teacher-learning-through-lesson-analysis-stella.

⁶ This terminology follows Bakeman and Gottman (1986), who classified kappas from .40 to .60 as fair, from .60 to .75 as good, and greater than .75 as excellent.

⁷ The proportion of variance explained is estimated as the difference between the total parameter variance and the residual parameter variance relative to the total parameter variance. The ES for growth rates is calculated as the ratio of the parameter multiplied by time to the standard deviation of the measure of interest.

⁸ Going back to CHAT theory (Norton et al., 2007; Yamagata-Lynch, 2010), which we footnoted earlier in this article, we can frame this question as follows: Do TTQs serve as tools for clarifying the object of lessons, setting the rules for engaging with lesson material, and establishing division of labor? Do teachers use TTQs to establish classroom culture? Do they use them to set expectations for students’ cognitive/learning jobs or responsibilities?

Additional information

Funding

This work was supported by Grant No. 201400019 from the Spencer Foundation.