The impact of an assessment for learning teacher professional development program on students’ metacognition

ABSTRACT

Assessment for learning (AfL) can facilitate students’ development of metacognition. However, teachers often struggle with the implementation of AfL in their classroom. The dynamic approach can help teachers develop professionally in the complex competency that is AfL. The dynamic approach is based on four principles: (a) addressing the professional needs of teachers, (b) integrating skills, (c) explaining underlying mechanisms, and (d) implementing support and feedback. In this experimental study, we investigated the effect on students’ metacognition of an AfL teacher professional development program for secondary mathematics teachers based on the dynamic approach (n_teachers = 47, n_students = 803). We found a statistically significant positive effect on students’ ability to predict (d = 0.24) and evaluate (d = 0.22), but not on students’ ability to plan.

KEYWORDS:

• Assessment for learning
• teacher professional development
• dynamic approach
• student metacognition

Introduction

Metacognition is an important predictor of student achievement and student motivation (e.g., Desoete & De Craene, 2019). It literally means “thinking about thinking” (Flavell, 1976) and is commonly defined as “one’s knowledge concerning one’s own cognitive processes and products or anything related to them … [and] the active monitoring and consequent regulation and orchestration of these processes” (Flavell, 1976, p. 232). However, without instructional support for students, it is unlikely that they will develop their metacognition (Baars et al., 2020). One promising approach to help students to develop their metacognition is the implementation of assessment for learning (AfL) in the classroom (Nicol & MacFarlane-Dick, 2006). AfL is:

assessment to the extent that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited. (Wiliam & Leahy, 2015, p. 8)

When AfL is being implemented in the classroom, students and their teachers are encouraged to frequently assess students’ learning progress (i.e., self- and peer assessment; Gulikers et al., 2021). These assessment results may give students more (accurate) input for monitoring their own cognitive processes and their progress towards certain learning goals. This, in turn, can enable students to make better decisions regarding future learning efforts, such as time investment and choice of strategies (Nicol & MacFarlane-Dick, 2006). Students’ regulation of cognitive processes can thereby be stimulated when teachers implement AfL, as the AfL activities can enhance these regulatory processes (Allal, 2020). This is in line with the student-centered focus of feedback in AfL: “a focus on active student involvement in self-regulation … where students actively seek, generate and use feedback to improve their learning outcomes” (Brooks et al., 2021, p. 2).

However, the use of AfL in the classroom is not simple (Heitink et al., 2016; Kippers et al., 2018). Teachers are not only supposed to develop and share the learning goals with their students, to elicit, analyze, and interpret collected data, and to make decisions based on those data, but also to include students as active agents in their own and their peers’ learning processes. This complexity can make it difficult for teachers to implement AfL, and because of that, the potential benefits of AfL for students’ metacognitive, self-regulatory skills may not be realized.

Therefore, a teacher professional development program for teachers to facilitate their implementation of assessment for learning (Heitink et al., 2016) may be beneficial. The dynamic approach to teacher professional development could be appropriate for such a complex teacher skill. The dynamic approach takes the interrelatedness of the different teacher skills involved and the varying professional needs regarding AfL skills into account (e.g., Antoniou et al., 2015). Recognizing and addressing the different (levels of these) professional needs (a common feature of AfL) in a teacher professional development program that uses the dynamic approach makes it more likely that teachers can improve the skills necessary for AfL (Christoforidou et al., 2014).

In the present study, we assessed the impact on students’ metacognition of a teacher professional development program for AfL based on the dynamic approach. We expected that if teachers are supported in the implementation of AfL in their classrooms, their students’ use of metacognition may be facilitated, and through this, the students may show better metacognitive development compared to students of teachers without participation in a teacher professional development program, as we will further explain in the theoretical framework. The research question for this study was: “What is the effect on student metacognition of a teacher professional development program for assessment for learning based on the dynamic approach?”

Theoretical framework

Metacognition

Metacognition has two parts (Kyriakides et al., 2020): (a) metacognitive knowledge and (b) metacognitive skills. Metacognitive knowledge is “declarative knowledge stored in memory and comprises models of cognitive processes, such as language, memory, and so forth” (Efklides, 2008, p. 278). Students depend on metacognitive skills in order to use their metacognitive knowledge effectively (Desoete & De Craene, 2019). Metacognitive knowledge is updated continuously as new information coming from observing one’s actions and the consequences of these actions while working on a task is integrated. Moreover, the integration of new information can also result from communication with others, such as through feedback (Efklides, 2008). Metacognitive skills refers to “the deliberate use of strategies (i.e., procedural knowledge) in order to control cognition” (Efklides, 2008, p. 280). Students’ metacognitive skills typically develop as students age (Veenman et al., 2006); however, this development may also be influenced by external factors, such as opportunities to use feedback, and dialogue about feedback (Nicol & MacFarlane-Dick, 2006). Important metacognitive skills are (a) planning, (b) monitoring, and (c) evaluating (Desoete & De Craene, 2019; Kyriakides et al., 2020).

Planning

Prior to starting a task, students may begin by planning their learning strategies (how, what, and when to act) in order to accomplish the learning goals (Desoete & De Craene, 2019). This planning can involve activities such as determining the kind of problem. Moreover, planning requires making connections between prior and current knowledge and selecting appropriate strategies to solve the problem (Shilo & Kramarski, 2019).

Monitoring

After students have planned their learning strategies and started the assignment, they may engage in monitoring their progress towards achieving the learning goals. Students can ask themselves questions during tasks such as “Does the solution make sense?” (Desoete & De Craene, 2019). If the answer is no, this might mean asking extra, clarifying, questions or investigating models of how to solve the problem.

Monitoring their results can later help students to predict the difficulty of a task and their capacity to address a task successfully (Baars et al., 2020). For example, students may decide to spend more time on tasks with larger learning gaps. For students to be able to predict the difficulty of tasks correctly, it is necessary that they have some prior knowledge of the topic, or familiarity with the type of task, so that they can distinguish better between apparent and real difficulties (Desoete & De Craene, 2019).

Evaluating

When the task has been completed, students may evaluate the effectiveness of their learning strategies. Students can use information gained from the completed task to improve their understanding, which, in turn, can improve their explanations and their approach to their next task (Shilo & Kramarski, 2019). However, when external feedback is provided, the student might also diagnose an error in their work, which, in turn, can lead to a re-do, or an adjustment of their work and the use of alternate learning strategies.

Assessment for learning

Teachers can stimulate students to use metacognitive skills by making the formative AfL cycle of “Where is the learner going?”, “Where is the learner now?”, and “How is the learner going there?” part of the classroom routine. For example, having students formulate and share learning goals can influence their knowledge of course expectations, which in turn can influence their effective reception of feedback. Below we will further describe the three formative questions, and how these can support students’ metacognitive skills (Nicol & MacFarlane-Dick, 2006; Wiliam & Leahy, 2015).

Where is the learner going?

Answering this question involves clarifying, sharing, and understanding learning goals and success criteria. Learning goals are what a teacher wants the learner to learn. Success criteria are more concrete parameters that can identify precisely where students are in their learning process compared to the learning goals (Wiliam & Leahy, 2015). The teacher can share learning goals and success criteria in the form of dialogues in which both the teacher and student play a role. For example, students can compare and assess exemplars: “carefully chosen samples of student work which are used to illustrate dimensions of quality and clarify assessment expectations” (Carless & Chan, 2017, p. 930). If learning goals and success criteria are clarified for both students and teachers, teachers and students can elicit evidence in a more targeted manner. In addition, clarification of learning goals and success criteria can prompt students to better plan and further manage their learning strategies, as they may have a better sense of what is expected of them (Carless & Winstone, 2020; Winne, 2018).

Where is the learner now?

Answering this question involves engineering classroom discussions and other learning tasks in order to continually elicit evidence of student progress compared to the learning goals. There is a wide variety of qualitative and quantitative assessment techniques that can give both teacher and students insight into where the students are in the learning process (van der Kleij et al., 2015). For example, teachers can pose hinge questions, carefully designed multiple-choice questions that can indicate student misconceptions (Wiliam & Leahy, 2015). The evidence that results from such assessment techniques can serve as continual feedback for students, which they can use, in turn, to monitor and evaluate their learning progress (Nicol & MacFarlane-Dick, 2006). Related to this, students’ progress can also be self-assessed and peer assessed, for example, with the use of co-constructed success criteria. In this way, students can become more accurate at self-reflection, and their improvement efforts can be better directed (Carless & Winstone, 2020). When students are not involved in assessment, they usually find the assessment process incomprehensible, which can make it difficult for them to effectively predict, plan, monitor, and evaluate their learning (Carless & Winstone, 2020).

How is the learner going there?

Feedback based on the collected information, which can be given by both teacher and students, can be an effective way of improving students’ understanding and learning (Hattie & Timperley, 2007). The use of feedback may be the most complex strategy of all, as the effects of feedback on student achievement are inconsistent and not always positive (Brooks et al., 2021). Recent studies (Gulikers et al., 2021; Panadero et al., 2018) have suggested that student involvement in the other AfL(-related) strategies may be beneficial for students’ feedback use. Students can be stimulated to improve their own learning process based on feedback only when that feedback gives information on where students are compared to the learning goals and success criteria. Moreover, feedback must give enough information about how specific strategies can help students to improve (Nicol & MacFarlane-Dick, 2006). For teachers to be able to carry out such improvements, they can best give feedback during the learning process. Teachers often decide to give feedback only at the end of the task completion process, but it can be more effective when provided during the process (Carless & Winstone, 2020).

Not only can the elicited evidence be used as feedback for students, it can also give teachers insight into how to continue with their teaching efforts. Teachers may, for example, diagnose a misconception in some or all students’ knowledge of a topic, and decide to use an alternate instructional strategy, such as instructional methods that require active student thinking rather than passive listening to repeated explanations of the subject matter. The use of such adaptations to match students’ level of understanding is similar to the concept of differentiated instruction, when “teachers deliberately plan such adaptations to facilitate students’ learning and execute these adaptations during their lessons” (Smale-Jacobse et al., 2019, p. 1).

Professional development for assessment for learning

AfL moves teachers away from a teaching approach focused more on “transmission” (i.e., telling students what to do) to a way of teaching in which students take an active role (Brooks et al., 2021). When students take this active role, they have more ownership over their own learning progress, which may instigate the development of metacognition. Competencies for AfL include, for example, the ability of teachers to formulate appropriate learning goals and success criteria for their subject domain as well as the ability to share these meaningfully with their students. As every step of AfL depends on the others, AfL can be defined as a complex teacher competence.

Teacher professional development may be helpful to equip teachers for the implementation of AfL in their own classroom. The dynamic approach can be used for teacher professional development in AfL (Antoniou et al., 2015). The dynamic approach is characterized by the following core principles:

1. Addressing the professional needs of teachers. Teachers may vary greatly in their ability and their motivation to implement AfL in the classroom. Where some teachers may already coherently implement all strategies belonging to AfL, other teachers may still struggle with the first AfL strategy of formulating suitable learning goals for their target groups of students (Kippers et al., 2018). Teacher development is more likely to occur when teachers’ specific professional needs are taken into account and addressed in a teacher professional development (TPD) program (Christoforidou et al., 2014). In addition, when teachers’ needs are analyzed prior to (but also during) the TPD program, it is possible for the TPD program to better connect to teachers’ practices (Darling-Hammond et al., 2017). Such coherence between TPD and teaching practice increases relevance for teachers, which, in turn, can help make change happen (“coherence”; Desimone, 2009).

2. Integrating skills. As stated, each AfL strategy is dependent on the others (Wiliam & Leahy, 2015). An integrated approach to addressing these skills in teacher professional development is necessary in order to help teachers tackle the complexity of AfL and transfer the complex skills to their own classroom. This means, for example, that the TPD program should include not only discussing different forms of assessment techniques but also explaining the relation of these assessment techniques with success criteria, and the timing of using assessment (results). In addition, not only do the AfL skills depend on each other during their development, the development of these skills also depends on teachers’ development of knowledge and beliefs regarding AfL (Heitink et al., 2016). For example, when teachers have a more positive attitude towards AfL, they can also feel greater appreciation when they are given feedback on their implementation of AfL.

3. Explaining underlying mechanisms. For teachers to recognize the potential benefits of AfL for student learning in their own classrooms, it is helpful that they understand why AfL can improve students’ metacognition. For example, an AfL mechanism might be that sharing learning goals can create transparency on what is expected from students, which can lead to increased student motivation (Carless & Winstone, 2020). Not only will it be helpful for teachers to have such mechanisms explained and demonstrated, ideally, teachers can also experience these mechanisms themselves. Through teachers’ use of action plans, they can create their own path for experimentation with respect to the implementation of AfL in their own classrooms (“active learning”; Desimone, 2009).

4. Implementation support and feedback. Teachers need help not only in the form of explanations and demonstrations but also in the form of real-time, individual, feedback from experts on the topic of AfL. Such feedback, as well as the explanations and demonstrations, can be especially supportive if the expert is also knowledgeable in the teacher’s specific subject area (“content”; Desimone, 2009). When teachers are experimenting with AfL in their classrooms, these experts can monitor and elicit evidence on whether teachers are able to transformatively apply TPD content to their lessons, and are not just layering new techniques on top of their current practice (Darling-Hammond et al., 2017). A potential way to monitor this effectively is through reflective discussion during the teacher professional development program (Korthagen, 2017). In addition, the reflective discussions, facilitated by knowledgeable experts, may also provide the needed scaffolding for teachers to take actual steps to transform their lessons (Darling-Hammond et al., 2017; Desimone, 2009). Experts in AfL can help to identify teachers’ underlying beliefs and misconceptions regarding AfL that may restrict their professional development (Timperley, 2008).

Method

Context

This study, which took place in the Netherlands, was part of the Erasmus project “Promoting Formative Assessment: From Theory to Practice to Policy” (FORMAS). One aim of this project was to investigate the impact on student achievement and metacognition of a teacher professional development program on AfL for secondary math teachers in four countries: Belgium, Cyprus, Greece, and the Netherlands.

Participants

The participants in this study were recruited via convenience sampling. Approximately all 462 secondary school locations in the Netherlands with the necessary school levels (i.e., general higher education and pre-university levels) of the lower grades (i.e., ages 12–14), were approached for participation in this study via email. In addition, calls for participation were published in professional magazines and a mathematics newsletter. A total of 33 schools decided to participate, with one to four mathematics teachers per school (N = 75). These teachers were asked to participate in this study with only one class of students, and were randomly assigned to either the experimental (n = 36) or the control group (n = 39). As we randomized at the school level, this could have caused the slightly uneven initial group sizes. Unfortunately, 27 teachers dropped out during our study (i.e., they did not participate in both measurements). The main reason for dropout was a shift in priorities, which was caused by extra workload or, especially during the posttest, the COVID-19 pandemic. Ultimately, 26 teachers in the experimental group and 22 teachers in the control group participated in both the pre- and posttests (see Table 1).

Table 1. Characteristics of the teachers in the experimental and the control groups.

Characteristics		Experimental group (n = 26)	Control group (n = 22)
Gender	Male	30.8%	18.2%
	Female	69.2%	81.8%
Teaching assignment (L = lower grades, H = higher grades, D = department leader)	L	53.8%	40.9%
	L + H	26.9%	50.0%
	L + D	11.5%	4.5%
	L + H + D	7.7%	4.5%
School urbanization	Small municipality (<50,000)	38.5%	22.7%
	Medium-sized municipality (50,000–120,000)	34.6%	31.8%
	Large municipality (120,000–300,000)	23.1%	40.9%
	Very large municipality (>300,000)	3.8%	4.5%
Years of experience	<5	3.8%	13.6%
	5–9	19.2%	18.2%
	10–14	15.4%	31.8%
	15–19	26.9%	13.6%
	>20	34.6%	22.7%
Experience with AfL	Missing	3.8%	4.5%
	No experience	46.2%	22.7%
	Informed	30.8%	45.5%
	Some practice (once or twice)	11.5%	27.3%
	Much practice	7.7%	0.0%

Note: The distribution of the teachers’ characteristics, assessed with Fisher’s Exact Test for Independence, did not show statistically significant differences between the experimental and the control groups. AfL = assessment for learning.

We therefore ended up with a total data set of 803 students (43.1% in the experimental group and 70.2% in the control group) who completed both the pre- and the posttest, out of the original 1,443 students. The gender distribution of the participating students did not differ significantly between the experimental and the control groups: χ²(1, N = 803) = 0.854, p = .36. Unfortunately, the dropout caused by COVID-19 led to a statistically significant difference for grade between these groups: χ²(2, N = 803) = 228.40, p = .00 (see Table 2 for the characteristics of the participating students).

Table 2. Characteristics of the students who participated in both the pretest and the posttest.

	Experimental group			Control group
Characteristics	Pretest n = 745	Posttest n = 498	Pre- and posttest n = 332	Pretest n = 655	Posttest n = 348	Pre- and posttest n = 471
Male	363	239	227	329	178	171
Female	382	258	244	326	170	161
Grade 7	160	85	83	416	241	231
Grade 8	256	183	173	166	68	64
Grade 9	329	230	215	73	39	37

Instruments

The metacognitive test

We developed four metacognitive mathematical tests, originally in English and for this study translated into and validated in Dutch. These four tests served as pre- and posttests for the different grade levels (see Table 3), which included tasks in which students could show their metacognitive knowledge and skills (cf. Magno, 2009).

Table 3. Order of administration of the metacognitive tests.

	Grade 7	Grade 8	Grade 9
Pretest	Metacognitive Test 1	Metacognitive Test 2	Metacognitive Test 3
Posttest	Metacognitive Test 2	Metacognitive Test 3	Metacognitive Test 4

Each test consisted of five exercises with a similar format, addressing declarative and procedural knowledge, planning, predicting, and evaluating (see the scales below). When we developed the exercises, it proved to be impossible to develop exercises for the monitoring and evaluating scales that were distinctive enough to measure these constructs separately. We therefore had to exclude the monitoring scale from the instrument. In addition, we had to combine declarative and procedural knowledge into one scale.

1. Declarative and procedural knowledge (Exercises 2 and 3): Students were asked to evaluate several solutions to a proposed problem and to indicate which solution was best. An example item can be found in Figure 1.

2. Planning (Exercise 4): Students were asked to evaluate and indicate which strategies were needed to solve a specific problem. An example can be found in Figure 2.

3. Predicting (Exercise 1): Students were asked to predict their ability to solve a problem on a 4-point Likert scale (4 = sure I cannot to 1 = sure I can). An example is shown in Figure 3.

4. Evaluating. (Exercise 5): Students were asked to solve the five problems posed in Exercise 1, and to indicate how sure they were about the correctness of their solution on a 4-point Likert scale (4 = sure it is wrong to 1 = sure it is correct). An example can be found in Figure 4.

All tests had been validated prior to the start of our study. However, in the current study, we found that we unfortunately had to exclude the declarative and procedural knowledge scale from the analysis, as the items on this scale were found not to fit the data anymore. This meant that we could only measure the effects of the program on metacognitive skills and not on metacognitive knowledge. Three valid scales remained that were used in this study: (a) predicting, (b) planning, and (c) evaluating. Scores for all subscales were based on calculating the Rasch person estimates of each scale (cf. Kyriakides et al., 2020). For more details on the analysis and the results of the validation study, the project website can be consulted (https://www.ucy.ac.cy/formas/en/).

Figure 1. Exercise 2 of Metacognitive Test 1.

Figure 2. Exercise 4 of Metacognitive Test 1.

Figure 3. Exercise 1 of Metacognitive Test 1.

Figure 4. Exercise 5 of Metacognitive Test 1.

The teacher questionnaire

The self-perception questionnaire on teacher assessment skills, developed by Christoforidou et al. (2014), was used to evaluate teachers’ professional needs at the start (and end) of the teacher professional development program. This teacher questionnaire, translated into Dutch for our study, included 115 items focusing on the phase of assessment (i.e., constructing, administering, recording, analyzing, and reporting), assessment techniques (i.e., oral, written, or performance assessment), and the assessor (i.e., teacher, student, peer). In addition, these items were measured in relation to various aspects of assessment: (a) frequency: the number of assessment moments in a classroom, (b) focus: use of multiple assessment moments to inform decision(s), (c) stage: the timing of the assessment moments, (d) quality: construct validity of assessment, and (e) differentiation: adapting assessment and/or feedback to student. Most items were measured on a 5-point Likert scale (1 = never to 5 = every lesson).

Rasch’s extended logistic model (Andrich, 1988) was used to identify a hierarchy of difficulty in items, which were consequently clustered into levels of difficulty that may be taken to stand for types of teacher behavior that move from relatively easy to more difficult (cf. Christoforidou et al., 2014). Three teacher developmental stages were distinguished prior to the professional development program:

• Group A (“beginner”): seven teachers, who mainly used written assessments to measure student achievement in mathematics for summative purposes.

• Group B (“advanced”): 13 teachers, who used different assessment techniques to measure achievement in mathematics, but without defining appropriate success criteria or providing constructive feedback.

• Group C (“expert”): six teachers, who used assessment techniques to measure more complex educational objectives to provide constructive feedback, but yet without involving students in the assessment process and without differentiating their assessment practice.

Procedure

The teacher professional development program

The teachers in the experimental group participated in the teacher professional development program that was scheduled to last 1 school year and consisted of five sessions, each session emphasizing one or more of the formative AfL questions. In the first, introductory session, the teacher questionnaire was also administered.

We had planned different activities for each group of teachers, based on their developmental stage. However, after the first session we noted that teachers in Group A felt that their topics were less related to their specific subject or not advanced enough, and therefore less relevant to them. We changed our approach from differentiation between groups (different topics and activities per group) to differentiation within groups (same topics and activities, emphasis on developmental stage within individual feedback) on the basis of these reactions. The teacher professional development program included several recurring activities, in the following order:

• Experimentation and reflective inquiry. Between sessions, teachers were asked to try to implement the content learned, and/or apply the developed tools in their own lessons. In this way, we could start the next session with reflective inquiry regarding the successfulness of this implementation, with questions such as “Why did you implement the AfL strategy in this way in your classroom?” and “What effect of using the AfL strategy did you notice?” Teachers’ reflection on their own teaching practice could be stimulated through such inquiry. In addition, the reflection gave the AfL trainers the opportunity to fight remaining misconceptions about AfL, such as the idea that the implementation of AfL is too time intensive, when teachers see it as an “extra” element on top of their current teaching.

• Explanation and demonstration. Teachers were given an explanation of the definition of each AfL strategy and why the AfL strategy could be used to improve AfL in their classrooms. These explanations were often additionally substantiated by video examples of teachers using the AfL strategy in their classroom.

• Hands-on activity. The largest part of each session was spent on a hands-on activity related to the AfL strategies discussed in that session. For example, the session on “where is the learner going” was mostly spent on developing and formulating learning goals and criteria for success for the next chapter or marking period together. Moreover, teachers were asked to think about ways to share the learning goals and criteria for success meaningfully with their students. Other examples of hands-on activities included developing rubrics (i.e., a collection of learning goals and success criteria, ranked from less to more advanced) and developing hinge questions.

• Developing a lesson plan. Teachers were asked to develop a plan for a lesson in which they could integrate all of the discussed content from this session and previous sessions to answer all formative questions: “Where is the learner going?”, Where is the learner now?”, and “How is the learner going there?” They were supported in this process by each other and the AfL expert. After the session, teachers were also given the opportunity to send the lesson plan to the AfL expert for extra feedback.

The data collection

The metacognitive test was scheduled to be administered to students between the first and the second session (pre), and after the fifth session (post). This was done so that teachers could administer the test themselves. Unfortunately, due to the outbreak of COVID-19, schools closed after our fourth session. We rescheduled the fifth session to September of the next school year. The post-measurement was done before the summer break and thus also before the fifth session. Students in the Netherlands often change teachers after the summer holiday, which could have impeded our administration of the posttest. After schools changed to online lessons because of COVID-19, we changed from in-school administration during the pretest to digital administration during the posttest. Teachers were all asked to administer the tests themselves during an online lesson.

Analysis

Most of the items were rated on a 4-point Likert scale. Other items were scored based on an answer key and added to the data file. We checked whether the groups were comparable by comparing the students’ metacognitive pretest scores. There was no significant difference for predicting [t(795) = 2.806, p = .25] and evaluating [t(799) = 1.919, p = .55] across the experimental and the control groups. This was not the case for planning [t(801) = 2.806, p = 0.01]. However, the mean difference (ΔM = 0.19) in pretest scores for planning between these two groups, in favor of the control group, was within the boundaries for comparison (What Works Clearinghouse, 2020). Moreover, there was no interaction effect between dropout and group (experimental-control) on the pretest scores for any of the metacognitive variables, which indicates that there was no relation between a student in a particular group dropping out and their pretest score. Due to the COVID-19 pandemic, this study had a high dropout level, which decreased its statistical power. For this reason, we decided to equate the scores of the three grades. This was done by using item response theory techniques that are similar to the techniques used by the Programme for International Student Assessment (Organisation for Economic Co-operation and Development, 2017).

As the data were nested (Level 1: students and Level 2: teachers), we decided to conduct multilevel analyses to assess the impact of the teacher professional development program on the students’ metacognitive skills. The high dropout rate meant that the statistical power was insufficient to conduct a multivariate multilevel analysis. Multilevel analyses were conducted per metacognitive skill: planning, predicting, and evaluating. These analyses were carried out with MLwiN software (Charlton et al., 2020). We analyzed an empty model (i.e., without any explanatory variables added) to investigate the variance in the scores across teachers and students: $\begin{array}{rl}& Y_{ij}(scoreposttestsforstudentiofteacherj)\\ & ={\beta }_{0j}(generalmean)+U_{0j}(randomeffectteacherlevel)+R_{ij}(randomeffectstudentlevel)\end{array}$

In Model 1, the explanatory variables of prior achievement and gender were added to the empty model: $\begin{array}{rl}{Y}_{ij}(& scoreposttestsforstudentiofteacherj)\\ & ={\beta }_{0j}(generalmean)+{\beta }_1{X}_{1ij}(priorachievement)+{\beta }_2{X}_{2ij}(gender)\\ & +U_{0j}(randomeffectteacherlevel)+R_{ij}(randomeffectstudentlevel)\end{array}$

To be able to assess the effectiveness of the teacher professional development program, we added the exploratory variable of group (0 = control, 1 = experimental) in Model 2: $\begin{array}{rl}& Y_{ij}(scoreposttestsforstudentiofteacherj)\\ & ={\beta }_{0j}(generalmean)+{\beta }_1{X}_{1ij}(priorachievement)+{\beta }_2{X}_{2ij}(gender)+{\beta }_3{X}_{3ij}(group)\\ & +U_{oj}(randomeffectteacherlevel)+R_{ij}(randomeffectstudentlevel)\end{array}$

We also evaluated the improvement of (self-reported) assessment skills by teachers, in order to validate that the changes at the student level were indeed caused by changes at the teacher level. We found an effect size of 0.88 (see Table 4).

Table 4. Standardized means and SD of pre- and posttest of teacher assessment skills per group, and the corresponding t test values.

	Experimental group		Control group		t test
	M	SD	M	SD	t value	df	p value
Pretest score	−0.49	0.39	−0.52	0.40	−0.38	65	0.71
Posttest score	−0.06	0.41	−0.49	0.42	−3.34	40	0.00

Results

Descriptive statistics

The pre- and posttest scores on the metacognitive tests per construct can be found in Table 5. As can be observed, the scores of the students in the control group are almost the same for both measurement points, whereas students in the experimental group scored higher on the posttest.

Table 5. Standardized pretest and posttest scores per group.

		Control		Experimental
		M	SD	M	SD
Pretest scores	Predicting (n = 1,400)	0.10	1.04	−0.09	0.95
	Planning (n = 1,390)	−0.05	0.95	0.08	1.02
	Evaluating (n = 1,395)	0.08	1.02	0.02	0.96
Posttest scores	Predicting (n = 844)	−0.03	0.91	0.13	0.82
	Planning (n = 811)	0.01	0.99	0.14	0.95
	Evaluating (n = 773)	−0.01	1.06	0.15	0.88

Planning

The results of the multilevel analysis for planning can be found in Table 6. Contrary to our expectations, the second model showed that the teacher professional development program did not have a statistically significant effect on the planning ability of students, β = 0.12, p < .10.

Table 6. Parameter estimates and standard errors for planning (students within teachers).

Factors	Model 0	Model 1	Model 2
Fixed part
Intercept	0.097 (.051)	0.114 (.060)	0.030 (.072)
Student level
Prior achievement		0.174 (.035)	0.174 (.035)
Gender (0 = boy, 1 = girl)		−0.026 (.068)
Group (0 = control, 1 = experimental)			0.123 (.095)
Variance components
Teacher	0.059 (.024)	0.050 (.022)	0.044 (.021)
Student	0.888 (.047)	0.864 (.045)	0.865 (.046)
Explained		0.033	0.038
Significance test
χ^2	2109.878	2085.878	2084.445
Reduction		24	1.433
Degrees of freedom		1	1
p value		0.001	0.05

Predicting

The results of the multilevel analysis for predicting can be found in Table 7. The second model revealed that the teacher professional development program had a statistically significant positive effect on students’ ability to predict their own mathematical skills, β = 0.23, p < .05. The standardized effect size was d = 0.24, which is considerable (Lipsey et al., 2012).

Table 7. Parameter estimates and standard errors for predicting (students within teachers).

Factors	Model 0	Model 1	Model 2
Fixed part
Intercept	0.054 (.048)	−0.003 (.056)	−0.125 (.074)
Student level
Prior achievement		0.190 (.029)	0.195 (.029)
Gender (0 = boy, 1 = girl)		0.110 (.056)	0.104 (.056)
Group (0 = control, 1 = experimental)			0.228 (.093)
Variance components
Teacher	0.065 (.022)	0.066 (.022)	0.058 (.020)
Student	0.644 (.033)	0.606 (.031)	0.605 (.031)
Explained		0.037	0.046
Significance test
χ^2	1965.412	1918.942	1913.108
Reduction		46.47	5.834
Degrees of freedom		2	1
p value		0.001	0.001

Evaluating

The results of the multilevel analysis for evaluating can be found in Table 8. The second model revealed that the teacher professional development program had a statistically significant positive effect on students’ ability to predict their own mathematical skills, β = 0.22, p < .05. The standardized effect size was d = 0.22, which is considerable.

Table 8. Parameter estimates and standard errors for evaluating (students within teachers).

Factors	Model 0	Model 1	Model 2
Fixed part
Intercept	0.094 (.044)	0.011 (.059)	−0.112 (.077)
Student level
Prior achievement		0.273 (.035)	0.276 (.035)
Gender (0 = boy, 1 = girl)		0.156 (.067)	0.151 (.067)
Group (0 = control, 1 = experimental)			0.220 (.091)
Variance components
Teacher	0.030 (.018)	0.048 (.021)	0.037 (.018)
Student	0.875 (.047)	0.792 (.043)	0.792 (.043)
Explained		0.065	0.076
Significance test
χ^2	1998.719	1936.344	1930.852
Reduction		62.375	5.492
Degrees of freedom		2	1
p value		0.001	0.001

Discussion

The aim of this study was to investigate whether teacher professional development for AfL based on the dynamic approach would be beneficial for students’ development of metacognition. The results showed that predicting and evaluating, two of the three measured metacognitive skills, improved considerably for students whose teacher participated in our teacher professional development program. The four dynamic-approach principles likely helped teachers to integrate AfL in their practice in various ways.

First, we addressed teachers’ professional needs in a differentiated manner, which may have allowed us to adapt to teachers’ zones of proximal development and connect more closely to their daily practice (Darling-Hammond et al., 2017; Timperley, 2008). We did not offer the initially intended between-group form of differentiation because many of the teachers in the “beginner” group were not content to be in that group. In their opinion, that would not benefit them as much, and they were more interested in the topics of the other groups, which included more applied activities for AfL such as developing rubrics and hinge questions.

Second, AfL was taught and practiced as a whole task, meaning that the focus throughout the program was on the coherent implementation of the set of AfL strategies. In the first session, for example, teachers were asked to formulate appropriate learning goals for their class. In the second session, teachers were asked to discuss what questions could help them elicit evidence as to where students were in their progress towards the previously formulated learning goals. Still, during each session, we had to put more emphasis on one particular AfL strategy to be able to discuss that one more extensively. The notion of the importance of coherence between all AfL strategies, especially the more difficult AfL strategies, such as the involvement of students in assessment (Christoforidou et al., 2014), could have been lost for some of the teachers.

Third, as a way to incorporate active learning (e.g., Desimone, 2009), teachers were not only “told” that AfL works but were also stimulated to ask and explain how AfL could benefit their teaching practice. For example, teachers were shown video examples of other teachers implementing AfL with varying degrees of success, and also asked to compare the video examples with their own standards for AfL. In this way, teachers could better understand why AfL can help to improve teaching quality.

Fourth, teachers were supported by each other and by knowledgeable AfL experts in their endeavor to integrate AfL in their teaching practice. As most teachers were new to the concept of AfL and some teachers had misconceptions regarding AfL (“I do not have time to take extra follow-up steps after having assessed my student’s knowledge”), it may have been beneficial that they received external feedback. This support is in line with guidelines by Timperley (2008) and Darling-Hammond et al. (2017), who mentioned that an external expert not only challenges assumptions but can also scaffold teachers to implement practices that will make a difference for student learning.

Our use of these principles as guides meant that teachers appeared to have been supported effectively to implement AfL in their own classroom, and this, in turn, appeared to have facilitated students’ development of two components of metacognition. It is interesting that this effect occurred without specific attention to the concept of metacognition during the intervention. It may indeed be possible that just the structure that AfL provides, such as clear expectations, frequent assessments, and feedback, helped to develop students’ metacognitive skills (Nicol & MacFarlane-Dick, 2006). Our findings could also demonstrate the proposed overlap between activities related to AfL and metacognition, such as working towards specific goals and monitoring the progress towards these goals (Gulikers et al., 2021; Panadero et al., 2018).

Limitations

It is difficult for us to explain why the metacognitive skills of predicting and evaluating showed statistically significant improvement, whereas planning did not. It is likely that this difference in effects for the different metacognitive skills was due to the way metacognition was measured. A limitation is that we used a proxy for planning that referred to being able to recognize the necessary solutions for problems. Other elements of planning, such as personal preferences regarding goal setting, as well as higher executive skills such as foresight and impulse control (Magno, 2009), need to be measured to improve the construct validity of planning. However, these elements are difficult to measure quantitatively, as they are often heavily based on recognition, like rank ordering of goals and strategy selection (Shilo & Kramarski, 2019).

The difficulty of measuring metacognition by means of tests alone was also apparent in the fact that we were unable to measure metacognitive knowledge validly. It is likely that instead of measuring metacognitive knowledge, we actually measured mathematical knowledge. Still, the operationalization of metacognitive knowledge used in this study is worth exploring further, as there are indications that such an assessment can be more accurate than a self-report questionnaire on metacognition (e.g., Craig et al., 2020).

Another limitation of this study is that it was carried out partly during the COVID-19 pandemic. Due to this, teachers had to switch from in-school lessons to online lessons approximately 2 months prior to the posttests. It is unlikely that teachers implemented AfL in their lessons with the same effectiveness as would have been the case if the lessons had been in person, as the focus of the teacher professional development was not on online lessons. This could have negatively influenced the outcome of this study.

The COVID-19 pandemic also caused differential attrition in our sample. This resulted in some unevenness between the experimental and control groups in their background characteristics. For example, we found that more teachers with more years of experience participated in the experimental group compared to the control group (not statistically significant). Fortunately, our analytical sample remained comparable for the purposes of this study, as students’ pretest scores on the metacognitive skills did not differ statistically significantly between the experimental and control groups. Still, the decrease in the number of students caused a decrease in the power, which made it impossible to use multivariate multilevel analyses, which could have revealed more about the relationship between teacher and student characteristics and the improvement of students’ metacognition.

Recommendations

This relatively large-scale study has shown the potential of teacher professional development for AfL based on the dynamic approach, as improvement in two of students’ metacognitive skills were observed. Still, further investigation into the potential of the dynamic approach for this purpose is necessary, as the metacognitive skill of planning did not improve and two metacognitive components could not be measured. A possibility would be to include open-ended questions with more options in order to capture more detailed elements of metacognition, such as goal setting and effort allocation (Baars et al., 2020). As this would, however, result in large amounts of data (e.g., 803 students in our sample), it would be necessary to code all data using artificial intelligence, which has already proven useful in other fields (e.g., peer feedback; Gardner et al., 2021).

Moreover, it will be worthwhile to repeat the study in times without a pandemic outbreak to ensure the reliability of the effectiveness of this TPD program. A larger data set could also allow for multivariate multilevel analyses, revealing more about the links between all background variables and the student outcomes with greater certainty. In addition, if this study were implemented and evaluated at a larger scale, this would also make it possible to have a more representative sampling procedure, as opposed to the convenience sampling used in this study. More and stronger variation between participating teachers, such as teachers from different subjects and with and without a preference for AfL, would also make it possible to investigate in greater depth the link between teachers’ attitude, knowledge, and skills on the one hand, and student outcomes on the other.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by Erasmus+ Project of the European Union under Grant number 2017-3118/001-001.

Notes on contributors

Jitske A. de Vries

Jitske A. de Vries, MSc, is a PhD candidate at ELAN Teacher Development, University of Twente, the Netherlands. She is involved in projects on teacher professional development for assessment for learning and has conducted multiple impact studies of teacher professional development programs.

Andria Dimosthenous

Andria Dimosthenous was a postdoctoral researcher at the Department of Education at the University of Cyprus. Currently, she is Assistant Professor of Educational Evaluation at the Department of Preschool Education at the University of Crete. She has participated in several international projects on educational effectiveness and school improvement. Her research interests involve parameters such as the home learning environment that affect student achievement gains.

Kim Schildkamp

Kim Schildkamp is a full professor at ELAN Teacher Development, University of Twente, the Netherlands. Her research focuses on formative assessment (data-based decision making, assessment for learning, and diagnostic testing), the use of research in schools, and big data in education. One of her central questions is how to support schools in the use of data to improve education. She is also the initiator and project leader of the data team® procedure project.

Adrie J. Visscher

Adrie J. Visscher is a full professor at ELAN Teacher Development, University of Twente, the Netherlands. In his research he investigates how the provision of various types of feedback to students, teachers, and schools can support the optimization of the quality of classroom teaching and student learning. His research also focuses on the characteristics of effective teacher professionalization.