Exploring the impact of pre-service science teachers’ reflection skills on the development of professional knowledge during a field experience

ABSTRACT

The process of reflection is assumed to be important for developing professional knowledge through practical experience in science teaching. However, this claim requires more evidence, based on a clear definition of reflection. The main goal of the present study is to explore how reflection skills influence the development of professional knowledge gained through teaching experience. Before and after a five-month field experience, we have measured pre-service physics teachers’ professional knowledge and reflection skills (N = 94; 133 cases pre and post from four German universities). We also collected data for learning opportunities during the field experience (e.g. the number of taught lessons). The present study uses a novel standardized digital simulation of collaborative oral reflections to measure reflection skills (performance assessment), as a way of increasing validity compared to self-reports. The data have been analyzed using path analysis. The main results show that reflection skills before a field experience impact the development of content knowledge (β = 0.231*) and pedagogical knowledge (β = 0.354**) during the field experience. Regarding the learning opportunities during the field experiences, we develop the following evidence-based post-hoc hypothesis: the more pre-service science teachers are enculturated into a community of practical teachers, the less (academic) content knowledge and pedagogical content knowledge they acquire during a field experience. Consequences for science teacher education will be discussed.

KEYWORDS:

• Reflection
• PCK
• Professional Knowledge

Introduction

Reflection on and in science teaching constitutes an important research topic in science education, in part because the process of reflection is assumed to be essential for developing professional knowledge through practical experience in science teaching: ‘Reflection is a mechanism for turning experience into knowledge.’ (McAlpine et al., 1999, p. 116). That would make reflection skills very valuable for science teachers.

Indeed, many studies and models have discussed the role of reflection. Also, a recent model of science teachers’ pedagogical content knowledge (PCK)—the refined consensus mode of PCK (RCM) (Hume et al., 2019)—argues that reflection is a mechanism that further develops an individual’s personal PCK, based on enacted PCK. Personal PCK is the PCK of a particular teacher, which can be verbalized. Enacted PCK exists in action only and determines how a teacher behaves while teaching science. Of course, enacted PCK is more than just experience—it is knowledge in action. But the idea that underpins the mechanism of reflection in the RCM is quite similar to McAlpine et al.’s (1999). However, this idea requires more empirical evidence, based on a clear definition of reflection. Empirical evidence for an impact of reflection skills on the development of professional knowledge would both serve as an argument for the importance of reflection skills for science teachers and support the RCM.

In academic science-teacher education, field experiences (sometimes referred to as practicums or school internships) constitute important learning opportunities, enabling participants to develop–among other things–professional knowledge through practical experience (Cohen et al., 2013). The present paper contributes to an understanding of the role of reflection skills in the development of professional knowledge during a one-semester field experience. The main research question is as follows: Do reflection skills impact the development of professional knowledge during a field experience?

Measuring reflection skills is challenging. Although many studies have focused on rich data collection (e.g. Lotter & Miller, 2017; Borko et al., 2008) research on reflection skills has generally been based on self-reports, raising questions about the validity of such studies (Holtz & Gnambs, 2017). In the present paper, reflection skills are measured using a performance assessment, which simulate authentic reflection situations. The validity of this measurement should be higher than that of pre-service teachers’ self-reports because comparatively, it is a more proximal measurement that requires the actual application of reflection skills. (Blömeke et al., 2014; Holtz & Gnambs, 2017; Kulgemeyer & Riese, 2018; Bartels et al., 2019) (cf. section ‘Performance Assessment of Reflection skills’).

Theoretical background

Reflecting in and on science teaching

Reflecting in and on teaching has been a research topic for several decades. Numerous studies have been published in this field. In the present paper, we only discuss a few of them. The goal of this section is not to conduct a systematic literature review but rather to develop a notion of reflection for the following empirical study. Reflection is described as beneficial for a range of actions in teaching situations (Aufschnaiter, Fraij, et al., 2019; Combe & Kolbe, 2008); it can also improve teaching in general (Korthagen, 2001). A vast number of studies and frameworks have suggested that reflection supports the professional development of teachers (e.g. Aufschnaiter, Fraij, et al., 2019; Hashweh, 2005; Justi & van Driel, 2005; Magnusson et al., 1999; Nilsson, 2008; Schön, 1983), helping to bridge the gap between theory and practice (Aufschnaiter, Fraij, et al., 2019; Gruber & Rehrl, 2005; Korthagen, 2001; Park & Oliver, 2008; Schön, 1983).

There seems to be a consensus among scholars in science education and pedagogy that reflection skills are particularly important for teachers. Reflection skills are mentioned in teacher-education standards (e.g. the National Board for Professional Teaching Standards, 2016) and they represent an important goal in general teacher education (Hatton & Smith, 1995). Interestingly, while researchers agree on the importance of reflection, there is no agreement on what reflection pertains. Hatton and Smith (1995) described reflection as ‘ill-defined’ (p. 33) and that judgment has not changed over the years (Korthagen, 2014). More recently, Aufschnaiter, Fraij, et al. (2019) have criticized an inflationary use of the term. Three very different definitions of reflection in the literature are provided below as examples:

• ‘[R]eflection can be defined as a thinking process which gives coherence to a situation which is initially incoherent and unclear.’ (Clarà, 2015, p. 263)

• ‘Reflection is a mechanism for turning experience into knowledge.’ (McAlpine et al., 1999, p. 116)

• ‘Reflection is a thought process that analyzes experiences (e.g. observations, emotional impressions) with the goal being an individual’s further development as a teacher.’ (translated from Aufschnaiter, Hofman, et al., 2019, p. 51)

While the first definition focuses on problem-solving, the second highlights the possibility of transforming experience into a cognitive resource that can be used to deal with future problems. The third definition focuses on the professional development of teachers. It excludes any analysis of instruction provided by other individuals that could help teachers develop their teaching skills. The lack of any consensual definition makes it hard to compare research results on this very important topic, even though some characteristics of the reflection process are present in most concepts.

The term ‘reflection’ was introduced into pedagogy by Dewey (1910, 1933) and then later by Schön (1983, 1987). Dewey (1933) saw reflection as a specific kind of thinking with a specific goal: to become aware of why we act in the way we do. Schön (1983, 1987) connected reflection and action by distinguishing between two different kinds of reflection: reflection-in-action and reflection-on-action. Reflection-in-action occurs during action, while reflection-on-action—either alone or with colleagues—reviews the action afterwards and helps to generate decision-making alternatives to improve further actions.

Dewey’s (1933) concept of reflection as a specific kind of thinking has been widely accepted (e.g. Aufschnaiter, Fraij, et al., 2019; Hatton & Smith, 1995; Mohlman Sparks-Langer & Bernstein Colton, (1988); Nguyen et al., 2014). It implies that reflection is an analytic process that can relate to any kind of action. This concept establishes important common ground, influencing the definition used in the present study. As this study is focused on teaching, the concept of reflection will represent this focus.

The present study has thus adopted the following understanding of reflection:

Reflection is the theory-based analysis of teaching with the goal of improving the quality of instruction and/or leading to further development as a science teacher.

This definition is close to that of Aufschnaiter, Hofman, et al. (2019, p. 51) but explicitly includes the possibility of reflecting on instruction provided by other teachers, as well as one’s own. Van Manen (1977) described three levels of reflectivity that have been used by Zeichner and Liston (1987) to design a teaching-student program. In our study, reflection encompasses all three levels of reflectivity briefly: (1) reflection without questioning the given ends; (2) reflection on educational consequences; and (3) reflection including moral/ethical questions.

Several models of reflection have been developed. For example, Korthagen (1985) introduced the ALACT model of reflection, which describes five phases: (1) the action, (2) a look back on the action, (3) becoming aware of essential aspects, (4) developing alternative methods of action, and (5) attempting these alternative methods. Step 5 begins a new reflection cycle. However, even teacher training aimed at these phases does not necessarily lead to meaning-oriented reflections (Korthagen, 2014). Hoekstra (2007) differentiates between action-oriented and meaning-oriented reflection, the former being superficial and oriented on how to improve the action, and the latter on understanding the reasons for an event. Hoekstra (2007) highlights that meaning-oriented reflection contributes more to professional development than action-oriented reflection. This also emphasises the significance of employing theory to understand classroom practice. Korthagen (2004) developed core reflections to enable teachers to reflect more deeply. Core reflections strive for a more holistic reflection by integrating, for example, feelings, beliefs and internal obstacles in order to become aware of how to use the core potential. Studies indicate that this is a promising approach for teacher education (e.g, Adams & Greene, 2013). The notion of reflection as a theory-based practice used in this paper includes the primacy of deep reflections as demonstrated by these studies. Furthermore, similar to core reflections, it includes meaning-oriented reflections as it highlights the importance of continued professional development.

Professional knowledge and the RCM of PCK

Over the last few decades, a large body of research in science education has focused on the professional knowledge of science teachers (e.g. Abell, 2007; Fischer et al., 2014; Hume et al., 2019; Van Driel et al., 1998). It should be noted that the extensive research conducted on professional knowledge cannot be covered comprehensively in this section of the study. The goal of this section is to discuss the relationship between professional knowledge and teachers’ actions in the classroom. The majority of research on professional knowledge is based on Shulman’s (1987) fundamental considerations and three areas of professional knowledge in particular: content knowledge (CK), pedagogical content knowledge (PCK), and pedagogical knowledge (PK) (e.g. Baumert et al., 2010; Hill et al., 2005). PCK, in particular, has been the focus of many recent studies, resulting in the Refined Consensus Model (RCM) of PCK (Hume et al., 2019), which considers both CK and PK to be ‘foundational to teacher PCK in science’ (Carlson & Daehler, 2019, p. 82).

Professional knowledge has often been treated as a disposition for the actions of teachers and, thus, an important influence on instructional quality (e.g. Hill et al., 2005). However, there is no detailed understanding of how CK, PCK, and PK impact teachers’ actions in physics (Cauet et al., 2015; Kulgemeyer & Riese, 2018; Keller et al., 2016), even though promising results come from other disciplines like mathematics (e.g. Baumert et al., 2010). Since PCK has often been described as topic-specific (e.g. Hashweh, 2005) the results from mathematics might not be directly applicable to physics. However, even the evidence in mathematics is ambiguous. Hill et al. (2005) suggest that a joint knowledge of CK and PCK together influence the quality of instruction. Delaney (2012) found no connection between knowledge and the quality of instruction – even though the same test instrument has been used. Cauet et al. (2015) and Keller et al. (2016) did not find a connection between physics teachers CK and PCK and instructional quality. Kulgemeyer et al. (2020) and Kulgemeyer and Riese (2018) present evidence for the crucial role of PCK for the quality of physics teachers’ instructional explanations.

The RCM provides a framework for understanding the relationship between professional knowledge and teachers’ actions (Carlson & Daehler, 2019). The RCM proposes three different realms of PCK: collective PCK (held by multiple specialized educators and represented in textbooks and other materials), personal PCK (the knowledge held by an individual), and enacted PCK (knowledge that only exists in action) (Carlson & Daehler, 2019).

Collective PCK influences personal PCK and vice versa. Although personal PCK is impacted by collective PCK, an individual teacher can choose to communicate his or her personal PCK to influence the collective PCK. This ‘two-way knowledge exchange’ (Carlson & Daehler, 2019, p. 82) also characterizes the relationship between personal PCK and enacted PCK. Personal PCK is the main factor influencing enacted PCK. However, through reflection in and/or on action, enacted PCK can also impact personal PCK. A well-reflected-on practical experience can lead to improved personal PCK. While evidence supports the impact of personal PCK on enacted PCK (e.g. Vogelsang et al., 2019; Kulgemeyer & Riese, 2018), the mechanism by which enacted PCK impacts personal PCK through successful reflection has not yet been researched. Generally speaking, there is no detailed understanding of how reflection helps to develop professional knowledge through practical experience. However, there are promising results regarding the development of PCK in the context of research on ‘content representations (CoRe)’ (e.g. Loughran et al., 2006; Hume & Berry, 2011). This tool has been proposed to capture teachers’ pedagogical content knowledge (PCK) and to develop pre-service teachers’ PCK through a reflection process. Pre-service teachers develop content representations based on questions that guide their pedagogical thinking about teaching specific content. These questions are particularly valuable during the planning process (for example, ‘What do you expect the students to learn about this concept?’; Tuithof et al., 2021). Research suggests that reflecting on such questions (and the big ideas a class should have understood after being instructed on a specific topic) aids in the development of PCK (Juhler, 2016).

Field experiences

Field experiences are widely considered a key part of teacher education. According to Holtz and Gnambs (2017), they are thought to bridge the theory-practice gap between the professional knowledge acquired in universities and teachers’ actions in the classroom. The more or less naïve assumption that ‘more practical experience leads to better practical skills based on theoretical knowledge’ has been questioned many times (e.g. Tabachnick & Zeichner, 1984). However, most prior studies that investigated the effects of field experiences in teacher education were descriptive (Wilson & Floden, 2003). Cohen et al. (2013) reviewed 113 studies of field experiences in teacher education. They found mainly positive effects for various traits, including reflection and action-related skills. Indeed, pre-service teachers often see themselves as more competent after a field experience (Besa & Büdcher, 2014). However, the overwhelming majority of these studies rely on self-reports, raising many questions about their validity (Holtz & Gnambs, 2017). Self-reports, e.g. often lead to an overestimation of skills (Oeberst et al., 2015). Therefore, Kulgemeyer et al. (2020) have relied on a more proximal measure, using performance assessments to determine skills before and after a field experience. They have shown that professional knowledge has an impact on the development of explaining skills during a field experience. However, as mentioned above, it is not yet clear whether professional knowledge benefits from practical experience—or how reflection skills contribute to this process. Based on the considerations raised by Holtz and Gnambs (2017), reflection-skill self-reports are not a suitable measure for tackling this problem. We have, therefore, used a direct measurement of reflection skills.

Methods

Research goal

The present study sets out to test a crucial assumption about the function of reflection: that reflection should help to develop professional knowledge through practical experience. This research focuses on a one-semester field experience in academic teacher education.

While we cannot control how the actual reflection process works during the field experience, we can assume that the higher the level of reflection skills at the beginning of the field experience, the better the process will work. This leads to our first hypothesis:

Hypothesis 1: The better an individual’s reflection skills are at the beginning of a field experience, the more professional knowledge he or she will develop during the field experience.

However, the level of reflection skills before the field experience is not the only important factor; there must also be opportunities to use reflection skills during the field experience. The pre-service physics teachers were mandated to both teach and observe several lessons. This paper argues that both teaching lessons and observing lessons taught by other teachers are opportunities for applying reflection skills through which participants can develop professional knowledge. This leads to the second hypothesis:

Hypothesis 2: The number of lessons taught and the number of lessons observed impact the development of professional knowledge during a field experience.

Two other important situations, both of which require the application of reflection skills, are collaborative oral reflections about taught lessons led by teacher educators, in this case mentoring teachers and university lecturers. The potential positive effects of such professional exchanges are documented in the literature (e.g. Min et al. (2020)). Mentoring teachers are often considered very important to the success of a field experience (e.g. Kreis & Staub, 2011). In Germany, pre-service teachers are mandated to hold collaborative oral reflections during their field experience. We would argue that the more intense and comprehensive these collaborative oral reflections are, the more likely it is that teachers will develop professional knowledge. This leads to the third hypothesis:

Hypothesis 3: The intensity of collaborative oral reflections with (a) mentoring teachers and (b) university lecturers influences the development of professional knowledge during a field experience.

The research goal can also help to understand an inherent assumption of the RCM of PCK: that the development of personal PCK from enacted PCK depends on the process of reflection. We are aware that enacted PCK is more than just teaching experience, however, there is no enacted PCK without teaching experience as well. If Hypothesis 1 can be confirmed, it should provide first evidence in support of the RCM of PCK.

Study context and design

Study context

This study was conducted in Germany. In the German teacher-education system, most German federal states implement a so-called ‘practical semester’ of long-term field experience as part of their academic teacher-education programs. During this semester, pre-service teachers are given exposure to one-semester (approximately five-month) field experience in local schools. They are supervised by both a university lecturer and at least one teacher from the school, who serves as a mentor. Although their main place of work during the practical semester is the school, pre-service teachers generally have some courses (usually one day a week) at the university, mainly to support them in their practice. The pre-service teachers are obliged to teach physics (usually six to ten lessons over the semester, although most teach more lessons voluntarily) and to sit in on lectures given by in-service teachers (around 30 h, although most pre-service teachers attend more lectures voluntarily). Both the university lecturers and the mentoring teachers sit in on the teaching of pre-service teachers and reflect with them on their instruction. There is one important difference, however. The mentoring teachers sit in on six to ten lessons, while the university lecturers sit in on one or two. It is important to note that teaching quality is not graded or required to pass the course; instead, these visits are designed to be consultative and reflective.

Study design

Professional knowledge and reflection skills were measured before and after a one-semester field experience. Before the field experience, participants completed a questionnaire related to personal variables, such as study progress and school education. This questionnaire made it possible to describe the sample in more detail. After the field experience, participants completed an additional questionnaire on the field experience, covering the number of lessons they taught and observed, which topics they discussed in collaborative oral reflections with their mentoring teachers and university lecturers, and how they developed lesson plans. The questionnaire provided important insights into the field experience, related to learning opportunities for professional knowledge and reflection skills.

Sample

The sample consisted of N = 94 (pre and post: N = 133 cases; 55 cases with missing data) pre-service teachers from four German universities. The universities have similar teacher education programs with a strong notion on CK, PK, and PCK in the bachelor’s program and a one-semester field experience at the beginning of the master’s program. All participants held a bachelor's degree in physics education (a six-semester bachelor's program). The average age was 25.8 (21–45). The sample represented a census of participants in a one-semester field experience in physics education at four German universities.

During the field experience, participants taught an average of 20 (5–50) physics lessons and observed an average of 61.7 (10–220) physics lessons taught by other teachers. Before the field experience, participants had taught an average of 8.4 (0–165) physics lessons. For most of the pre-service teachers, the field experience was a major influence on their overall practical experience because it more than doubled the overall teaching experience in terms of lessons taught. All the pre-service teachers had a second teaching subject, in many cases math. In their second teaching subject, they had taught 21.1 (4–54) lessons, on average, and observed 66.7 (2–200) lessons. All participants were visited by a university lecturer at least once and had multiple opportunities to talk with mentoring teachers, both to plan lessons and to reflect on their teaching.

Instruments

As noted above, a questionnaire was used to gain insights into learning opportunities during the field experience. Of importance in the present study were four key learning opportunities: the number of lessons taught in physics, the number of lessons observed in physics, the intensity of collaborative oral reflections with mentoring teachers, and the intensity of collaborative oral reflections with university lecturers. The numbers of lessons taught and observed were self-reported by the pre-service teachers.

The intensity of collaborative oral reflections was measured, based on the number of topics dealt with and the degree to which those topics were part of the reflection sessions. For each mentoring by a teacher and a university lecturer, the questionnaire included 14 possible topics, including student misconceptions, teaching methods, and dealing with heterogeneity. These topics resulted from expert interviews with university science educators; the pre-service teachers had to rate the degree to which the topics were discussed, using a 4-point Likert scale. The intensity was the sum of all ratings for the 14 topics. Thus, two topics with average ratings had the same weight as one topic with a perfect rating. Of course, we are aware that good collaborative oral reflection can focus on one topic in depth. However, given that a field experience can incorporate various opportunities to hold collaborative oral reflections, the number of topics addressed during such talks overall, combined with information on how deeply those topics were explored, arguably provides a good indication of how intense the collaborative oral reflections have been overall.

Established instruments, applied in numerous other studies, were used to measure professional knowledge. Like Kulgemeyer et al. (2020), we used three different instruments to measure the complex area of professional knowledge: the ‘ProfiLe-P-tests’ for PCK and CK, and Seifert and Schaper’s (2012) test for PK.

Pedagogical-content knowledge

The ‘Profile-P-test on PCK’ is based on the PCK model developed by Gramzow et al. (2013), which itself relies on various other concepts, including Lee and Luft (2008), Magnusson et al. (1999), and Park and Oliver (2008). Gramzow et al.’s (2013) model also relies on a curricular analysis of German academic teacher education related to PCK. The instrument has been used in numerous other studies, including Kulgemeyer and Riese (2018) and Kulgemeyer et al. (2020).

The paper-and-pencil test included 43 items, which were either open situational-judgment items or multiple-choice items and required 60 min to complete. The test was analyzed for content validity (expert ratings), construct validity (an analysis of a nomological network), and cognitive validity (using a think-aloud study). It was reliable overall (EAP/PV reliability: 0.84), reaching an appropriate level of interrater reliability (κ = .87). In addition, the four sub-scales provided reliable measures: (1) instructional strategies (EAP/PV reliability: 0.62); (2) students’ misconceptions and how to deal with them (EAP/PV reliability: 0.69); (3) experiments and teaching science adequately (EAP/PV reliability: 0.74); and (4) PCK-related theoretical concepts (EAP/PV reliability: 0.76).

Pedagogical knowledge

To assess PK, we used a short-scale of the instrument developed by Seifert and Schaper (2012), who provided arguments for constructing validity and content validity. It has been used in other studies (e.g. Mertens & Gräsel, 2018). The short-scale addressed two aspects of PK: general teaching methods and classroom management. The test took 15 min, and the instrument was reliable (α = .76). We expected the two aspects to have high practical relevance in the collaborative oral reflections. In recent years, PK has been conceptualized using the three-dimensional instructional process, student learning, and assessment (Sonmark et al., 2017). The three-dimensional instructional process includes both general teaching methods and classroom management (König et al., 2020); our instrument focused on the three-dimensional instructional process.

Content knowledge

The underlying model for the test for physics teachers’ CK consisted of three dimensions that were reliable subscales of the test instrument: (1) school knowledge (physics knowledge derived from school textbooks) (EAP/PV reliability: 0.82); (2), a deeper understanding of physics knowledge than could be derived from school textbooks (e.g. knowledge of different solution procedures; deeper school knowledge) (EAP/PV reliability: 0.81); and (3) university knowledge (physics knowledge from a university textbook) (EAP/PV reliability: 0.82). The test consisted of 48 multiple-choice questions and required 60 min to complete. Content validity was established through a curricular analysis and an analysis of the most frequently used textbooks. Construct validity was tested using a Rasch analysis. Cognitive validity was researched using a think-aloud study. Vogelsang et al. (2019) have presented the instrument in more detail.

Performance Assessment of Reflection skills

Various empirical approaches may be used to measure reflection skills. As previously stated, many studies on pre-service teachers’ reflections rely on self-reports, limiting their validity (Holtz & Gnambs, 2017). Some studies on teaching performance videotape and analyze lessons. For instance, in the QuIP project, 69 lessons from 92 physics teaches were videotaped and analyzed for cognitive activation as a key measure of teaching quality (Ergönenc et al., 2014). This type of measurement is unquestionably authentic, closely related to actual teaching performance, and may also be applicable to measuring reflection skills. Reflection talks with mentoring teaching, for example, could be videotaped and analyzed. However, as Kulgemeyer and Riese (2018) note, the high effort required to analyze videotapes limits the sample size; even an international research project involving various researchers, such as QuIP, was unable to analyze a sample size with medium or small effects. Blömeke et al. (2014) argue that while tests with video vignettes may be more useful and have a larger sample size, they still test cognition rather than actual action quality in core situations of a science teachers’ practice. Miller (1990) proposed alternative assessment methods including testing for so-called performance that simulates core situations in a specific profession. Performance assessments are simulated situations in which simulated students play a specific role. As such, they must be authentic and genuine (Bartels et al., 2019). Kulgemeyer and Riese (2018) and Kulgemeyer et al. (2020) have used performance assessments to measure the practical skills of teachers explaining physics phenomena. ‘[Performance assessments] rely on a direct measurement of action, but this is under standardized conditions’ (Kulgemeyer et al., 2020, p. 13). According to research in medicine and psychology, high standards, particularly regarding reliability and validity, can be fulfilled (Hodges et al., 1998; Walters et al., 2005). The goal of our study was to simulate an authentic situation in which reflection skills are required for pre-service teachers participating in a field experience. Therefore, in the present study, we simulated collaborative oral reflections. We relied on a digital test instrument, featuring a peer intern in the field experience (‘Robert’). Overall, the test time was 70 min. Figure 1 presents the scenario structure.

Figure 1. Structure of the reflection-skills instrument

The scenario structure consisted of three phases:

1. Contextual information was provided. Here, the peer intern asked for help. The goal of the lesson and a lessons plan was presented, both could be viewed at any time during the test. Information about the group of students was shared (e.g. prior knowledge).

2. Reflection of 13 parts of the lessons. Here, 13 video vignettes (approx. five minutes each) were presented as parts of a 90-minute lesson on introducing the conservation of momentum, based on Newton’s third law. Before and after each vignette, Robert was in the picture (pre-prompt and post-prompt). In the pre-prompt, Robert explained the context of the following teaching sequence; in the post prompt, Robert expressed his need for reflection (e.g. ‘Can you understand why the students answered that way?’) During this phase, the participants gave verbal answers, which were recorded. The participants were not told that the video vignettes were staged and scripted to highlight 48 typical problems (the test items) relying on CK, PCK, or PK, such as dealing with student misconceptions, classroom management, and errors in the underlying physics concepts (e.g. students being unable to distinguish between momentum and kinetic energy).

3. Reflection on the lesson as a whole. The last part of the instrument consisted of six further items, which dealt with the lesson as a whole (e.g. how it fit prior knowledge and learning goals).

The recorded answers were analyzed using a set of categories developed from a qualitative content analysis (Mayring, 2000). The categories were based on a model of reflection skills (Figure 2) (Nowak et al., 2019) which concurs with our definition of reflection (see above) because it highlights that reflection should be based on theory and aim for further professional development and/or improved teaching quality. Therefore, the concept of reflection is posited by our method of assessing reflection skills.

Figure 2. Underlying model of reflection skills

The model differentiates between elements of reflection and the knowledge bases that reflection is based on. Each statement about a science teacher's action can be matched with an element of reflection (a description of the observed action, evaluation of the action, alternatives to the observed action, and consequences for further actions or personal development). These elements of reflection are based on the Plöger and Scholl (2014) model. Reflection is theory-based, as expressed by the knowledge bases and reasoning. In particular, each statement can be matched with a knowledge base used to evaluate, present alternatives, or present consequences. These knowledge bases connect each statement with the theory. The model confirms the understanding of reflection discussed above (cf. section ‘reflection in and on science teaching’). The knowledge base confirms the fact that reflection is theory-based. The goal of reflection—to improve the quality of instruction and/or to lead to further development as a teacher—is expressed through the elements of reflection, in particular, ‘consequences.’ ‘Consequences’ may also include the third level of reflectivity as described by Van Manen (1977).

Thus, the three knowledge bases, CK, PCK, and PK provided the theories needed to provide a rationale for each statement.

The rating process worked as follows. Every statement made by the pre-service teachers was matched with an element of reflection and a knowledge base. It was also coded, reflecting whether an evaluation, alternative, or consequence was presented with a supporting reason (category ‘reasoning yes/no’). Of the statements, 17% were triple-coded (average Gwet’s AC₁=.904, which implies very good agreement). After the coding, scores were awarded to rate the quality of each statement. The scores showed which elements of reflection were used and whether a reason was presented (Table 1).

Table 1. Sample statements and levels of reflections

Points	Level of Reflection	Sample Statement
1	Description of the teaching situation or lesson context	‘Two students held up their hands and you called one of them up immediately.’
2	Evaluation of the teaching situation	‘It was not good to call this one student up so fast.’
3	Reasoned evaluation of the teaching situation	‘It was not good to call this student up so fast because the other students did not have a chance to really think about the task.’
4	Alternatives to the teaching situation	‘You should wait longer before you call someone up or let the students talk to each other before you call them up.’
5	Reasoned alternatives to the teaching situation	‘You might consider letting the students discuss the question with their peers before you call them up because talking about one’s own misconceptions requires a lot of courage. It is important to make it part of the experience to say that other students have the same misconceptions.’
6	Consequences for future instruction or personal development	‘You might consider working more with cooperative learning techniques in the future. For the next lesson, that would be worth trying explicitly.’
7	Reasoned consequences for future instruction or personal development	‘You might consider working more with cooperative learning techniques in the future. For the next lesson, that would be worth trying explicitly. It might help to let the students discuss the question with their peers before calling them up. Cooperative learning explicitly suggests following a ‘think-pair-share’ approach, which includes a phase of peer discussion.’

A simple description received one point while a reasoned consequence for future instruction or personal development received seven points. In the end, all of the points for each participant were added together. The arguments for validity included the results of a study that compared 65 pre-service and 10 beginning teachers after one year of service; teachers undergoing the first two years of in-service teacher training received slightly higher scores (T(73) = 4,709, p=.000, d=1,6 (CFI (95%): 0,89 < d < 2,32). They also included interview studies confirming that participants perceived the situation as authentic. The instrument is presented more in detail in Vogelsang et al. (2019). The final scale was reliable (α = .741).

Methods of data analysis

To analyze the data, path analysis was used. The standardized regression coefficients between the analyzed variables were reported. Path analysis is a frequently used method to estimate the values of explained variance of a variable by other variables (Vinzi et al., 2010).

The rather low sample size of just N = 133 cases (combined pre and post) needs to be considered. First of all, additional 55 additional cases contribute data to either the pre – or the post-test but not to both. As a consequence, robust maximum-likelihood estimation (MLR) (Sass et al., 2014) and full information maximum likelihood (FIML) have been used to deal with missing data (Little & Rubin, 2014). Also, the models rely on manifest values, which are the better choice to deal with the rather low sample sizes. Since manifest models tend to underestimate the relationship, this was a rather cautious approach. The sample size as a prerequisite for path analysis has been discussed in the literature (e.g. Wolf et al., 2013; Sideridis et al., 2014). Nunnally (1967) proposed at least ten cases per included variable, which would allow estimating a model with 13 variables based on our data. For our model, we require eleven variables (Figure 3). This is a limitation of our study, but still, our study is the first one to use a proximal measurement of reflection skills that takes a lot of effort to analyze due to the verbal data. However, path analysis should still be treated with care. The R-package lavaan has been used for the path analysis (Rosseel, 2015).

Figure 3. Path model for the development of professional knowledge (CK: content knowledge, PCK: pedagogical content knowledge, PK: pedagogical knowledge) over the course of the field experience. RMSEA=0.065; CFI=0.986. n.s.: non-significant, * p < .05., **p < .01

In a nutshell, the sample size was rather small but appropriate for an observation of large effects, expected given the long duration of the 5-month intervention.

As a measure for model fit, we reported CFI (comparative fit index; very good fit CFI > .95; cf. Fan et al. (1999)) and RMSEA (root mean square error of approximation; required RMSEA < .05).

Findings

Descriptive analysis of the development of study variables during the field experience

Table 2 shows the intercorrelations between the study variables. Before the field experience, the data showed a correlation between PCK and reflection skills (r = 0.48, p < 0.01). However, there was no correlation between other domains of professional knowledge and reflection skills. After the field experience, that picture fundamentally changed: there was no correlation between PCK and reflection skills—but there was a correlation between PK and reflection skills (r = 0.34, p < 0.05).

Table 2. Manifest intercorrelations of measured variables (Pearson’s r)

	R1	PCK1	CK1	PK1	R2	PCK2	CK2	PK2
R1	1				0.49**	0.14	0.07	0.20
PCK1	0.48**	1			0.25	0.76**	0.15	0.61**
CK1	−0.06	0.12	1		−0.19	0.35	0.53**	0.17
PK1	0.11	0.63**	0.09	1	0.23	0.59**	−0.07	0.64**
R2					1
PCK2					0.12	1
CK2					−0.21	0.28	1
PK2					0.34*	0.57**	0.21	1
L_t	−0.11	0.18	0.03	0.06	0.02	−0.04	−0.42*	0.15
L_o	−0.00	−0.01	0.08	0.12	0.38**	0.14	0.09	0.29
I_u	−0.03	0.07	0.15	0.15	0.20	0.22	0.38*	0.26
I_t	−0.09	−0.02	−0.14	−0.11	0.00	0.08	0.09	−0.22

* p < .05. **p < .01

Note. CK = content knowledge, PCK = pedagogical content knowledge, PK = pedagogical knowledge, R = reflection skills, L_t: number of lessons taught, L_o: number of lessons observed, I_t: intensity of collaborative oral reflections with mentoring teachers, I_u: intensity of collaborative oral reflections with university lecturers, 1: pre-measure, 2: post-measure

The correlations between pre- and post-test measures show that the pre-measure on reflection skills correlates with reflection skills after the field experience (r = 0.49, p < 0.01). This tendency is true for all of the subscales of professional knowledge. However, there was no correlation between reflection skills (pre) and any of the subscales of professional knowledge (post).

Table 3 shows the test results before and after the field experience, demonstrating that all sub-scales of professional knowledge developed over the course of the field experience. However, it is important to note that we cannot observe a development of reflection skills.

Table 3. Descriptive statistics of study variables

	Pre				Post
	M	SD	Minimum	Maximum	M	SD	Minimum	Maximum
Reflection	0.14	0.048	0.02	0.29	0.15	0.059	0.03	0.29
CK	0.61	0.133	0.33	0.96	0.65	0.104	0.42	0.84
PCK	0.46	0.128	0.17	0.73	0.55	0.112	0.29	0.81
PK	0.33	0.120	0.10	0.61	0.43	0.104	0.21	0.66

Note. CK = content knowledge, PCK = pedagogical content knowledge, PK = pedagogical knowledge, Reflection = reflection skills, M = means, SD = standard deviation. The score of all test instruments ranges between 0 and 1.

To investigate the development of reflection skills in detail, we analyzed the average level of reflection attained by the participating pre-service teachers. Based on our model, the levels of reflection range from Level 1 to Level 7 (Table 1). Prior to the field experience, the pre-service teachers acquired an average level of M = 0.9 (SD = 0.67, range: 1-4). Level 1 corresponds to a description of the teaching situation or lesson context (Table 1). They failed to identify 71% of the 48 typical problems found in the test items. In the post-test, we were unable to detect any substantial change in the average level of reflection (M = 0.9, SD = 0.62, range: 1-4) or in the amount of identified typical reflection problems (72% of 48 typical problems). Subsequently, we focusses the analysis on situations on which they noticed the problem to reflect on in order to gain insights into their possible reflection depth. Here, the level of reflection is higher. Pre-service teachers achieved an average level of M = 3.1 (SD = 0.44, range: 1-4) prior to the field experience, and a similar value with a slightly different range after the field experience M = 3.1 (SD = 0.42, range: 2-5). Level 3 corresponds to a reasoned evaluation of the teaching situation (Table 1).

Do high-level reflection skills impact the development of professional knowledge?

We analyzed the influence of reflection skills before the field experience on the development of professional knowledge during the field experience (Hypothesis 1) using path analysis (Figure 3). In the same model, we analyzed the influence of lessons observed, lessons taught (H2), and intensity of collaborative oral reflections with (a) mentors, (b) university lecturers (H3).

The most important factor for professional knowledge after a field experience was professional knowledge before. For all three domains of professional knowledge, the data showed a significant influence, with a moderate to large effect size. Reflection skills before the field experience had an impact on CK and PK. Also, collaborative oral reflections with university lecturers had an impact on the development of professional knowledge. The collaborative oral reflections with mentoring teachers had no impact. Taught lessons had an influence on the development of PK—and a negative influence on the development of CK. Observed lessons had a negative influence on the development of all three aspects of professional knowledge.

Discussion

First, we could not observe a general development of the reflection skills during the field experience. Neither the score for reflection skills changes nor the average level of reflection the pre-service teachers reached. However, our data points to the fact that this might, first of all, be a problem of (a) noticing the situations that require a reflection and (b) professional vision (Roth et al., 2011; Sherin, 2001). In most cases, the pre-service teachers did not notice the problems and did not reflect on them at all. Whenever they identify the situations to reflect on, however, on average they connect their judgements with theory and reach level 3 of 7.

The goal was to discover whether reflection skills helped to develop professional knowledge through practical experience. Three related hypotheses were tested.

Overall, we found evidence to support Hypothesis 1 (The better an individual’s reflection skills are at the beginning of a field experience, the more professional knowledge he or she will develop during the field experience). Evidence was observed concerning CK (β=0.231*) and PK (β=0.354**); we did not observe a significant effect on the development of PCK (Figure 3). Since two out of three scales supported the hypothesis and the third did not contradict it (in the absence of a negative effect), we would argue that hypothesis 1 should be accepted as a tendency. This result suggests an impact of reflection skills on the development of professional knowledge (at least for CK and PK) based on practical experience and supports McAlpine et al.’s (1999) notion of reflection. In tendency, this also constitutes evidence to support the relationship between enacted PCK and personal PCK in the sense of the RCM. CK and PK are treated as foundational for PCK in the RCM. We are well aware that our data include only the opportunity to use enacted PCK in teaching situations, without rating how well participants taught or how high their enacted PCK was. That is a limitation of the present study. Due to limited resources, we were unable to analyze certain data, such as the lessons the pre-service teachers held for teaching quality. Nevertheless, we would argue that the opportunity to use enacted PCK is the prerequisite for using enacted PCK. Thus, the impact of practical experience in general on the development of CK and PK in tendency also supports the claim that reflection on enacted PCK leads to a higher personal PCK.

Hypothesis 2 (The number of lessons taught and the number of lessons observed impact the development of professional knowledge during a field experience) must be rejected, based on our data. While the number of lessons taught was found to influence the development of PK, it had a negative impact on the development of CK. The number of lessons observed had a negative impact on the development of CK, PCK, and PK. Potential reasons for this are discussed below. We looked for a critical point in the relationship between lessons taught/ lessons observed and the development of professional knowledge, acknowledging the possibility that too many lessons may leave pre-service teachers with insufficient time for detailed reflections. In this case, we anticipated an increase in knowledge up to a certain number of lessons taught/observed followed by a decrease. However, we were unable to identify such a point, which could be attributed to the small sample size. Future studies should examine this in more detail.

Hypothesis 3 (The intensity of collaborative oral reflections with (a) mentoring teachers and (b) university lecturers influence the development of professional knowledge during a field experience) can be accepted only in part. The collaborative oral reflections with mentoring teachers had no influence at all. Prior studies have shown that support from mentoring teachers influences the development of PK, although the effects have been small (König et al., 2020). The present study could not find small effects, due to the limited sample size. The collaborative oral reflections with university lecturers, however, were shown to be a core element of the field experience when it came to developing professional knowledge. They contributed to an increase in all measured parts of professional knowledge (CK: β = 0.356**, PCK: β = 0.295**, PK: β = 0.269**). If the goal of a field experience was to develop professional knowledge, this data would suggest that field experiences should make room for university lecturers, who should visit the lessons of pre-service teachers to engage in collaborative oral reflections.

The role of the possible learning opportunities for the development of professional knowledge in field experiences is very interesting for further research. In our study, (1) the more lessons the pre-service teachers taught, the less they increased their CK (Figure 3; β = −0.283**), but the more they developed their PK (Figure 3; β = 0.345**).

A possible explanation would be that the more the pre-service teachers taught, the more they became aware of PK-related issues (such as classroom management, an important part of the PK test instrument) and the more they used the field experience as a learning opportunity for this part of professional knowledge. It sounds plausible that beginning teachers initially struggle to establish a sound learning environment. Also, the test for PK used in our study focuses on classroom management and general teaching methods which might be of importance in such situations. The choice of this test instrument might, therefore, overestimate the general development of PK. That, however, does not explain the negative effect on CK.

Another explanation might be rooted in the nature of PK compared to CK and PCK. Pre-service teachers who teach many lessons might learn less CK because they are on average confronted with a broader set of lesson content (‘A mile wide an inch deep’, cf. Schwartz et al., 2009) as opposed to pre-service teachers who teach just a few lessons but take more time to prepare them and reflect more deeply on the content. PK, on the other hand, might be more transferable from lesson to lesson, meaning that the pedagogical challenges related to a specific knowledge element (e.g. classroom management) occur more often in practical teaching knowledge elements of CK (e.g. Conservation of Energy) would. In that case, it would also make sense that the effect of ‘lessons taught’ on PCK lies between the two (PK and CK) as one could argue that a specific knowledge element of this category (e.g. strategies on dealing with misconceptions) is more generic—and therefore more frequently occurring—across different lessons than an element of CK but less generic than an element of PK. This also supports the claim that PCK is topic-specific rather than a more general PK (Hashweh, 2005, p. 290).

Also, our data shows (2) that the more lessons they observed the less the pre-service physics teachers developed all subscales of their professional knowledge (Figure 3; CK: β = −0.283**, PCK: β = −0.295**, PK: β = −0.345**).

Previous studies found that (pre-service) teachers hold the belief that knowledge gained from their university courses is not helpful in practical teaching (e.g. Bleck & Lipowsky, 2020; Bromme & Tillema, 1995). The number of lessons taught and lessons observed might be first measures for the intensity of a field experience because the pre-service teachers are free to observe and teach as many lessons as they like. The intensity of a field experience may increase this belief that scientific knowledge—and perhaps CK and PCK more than the tested aspects of PK—was irrelevant to school teaching. Also, the more the pre-service teachers kept contact with the academic part of their teacher education (represented by the intensity of the collaborative oral reflections with their university mentors), the more they develop their professional knowledge. That leads to the following post-hoc hypothesis that might be interesting for future studies: The more intense a practical experience is (and, maybe, the more pre-service teachers are enculturated into a community of practical teachers), the less CK and PCK they acquire during a field experience. The closer the pre-service teachers keep contact with academic teacher education, the more professional knowledge they acquire. This might, however, not even be a problem; the pre-service teachers might just focus on other issues such as the acquisition of practical skills (Kulgemeyer et al., 2020). In particular, future studies should examine the ambiguous evidence for the development of PK. As discussed above, it seems possible that PK-related issues are simply more important for beginning teachers and that might be due to the aspects of PK we tested in particular.

Limitations

As discussed above, the measurement of reflection skills in this study relied on the analysis of instruction provided by other individuals. Reflection in the sense of Aufschnaiter, Fraij, et al. (2019) aims at professional development based on one’s own experiences, which might have even more impact on the development of professional knowledge. Future studies might cover this part of reflection.

Our data certainly have many limitations and there is a need for future studies that use a proximal measure for reflection skills to focus in more detail on how such skills develop during field experiences. Important limitations of the present study include the sample size, which prevented us from analyzing small enough effects to reveal further insights, mediation effects of the learning opportunities, and also limits the path analysis. In addition, the study design did not allow for causal interpretations, as an experimental study would. Also, there is plenty of room for future research that also focuses not only on the development of professional knowledge but also on the development of reflection skills and the effects of field experiences using a proximal measure.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the German Federal Ministry of Education and Research under Grant 01PK15005B.