Evaluation of a State-Wide Mathematics Support Program for at-Risk Students in Grade 1 and 2 in Germany

Abstract

Supporting students with difficulties in learning mathematics is a challenge for teachers and educational administrators. Formative assessment is considered to play a successful role in supporting at-risk students as well as students without difficulties in mathematics. There is a need for intervention programs, including formative assessment techniques, that (a) are easy to implement in the regular classroom without requiring radical changes in teachers’ individual teaching style, and (b) are effective in supporting at-risk students at the earliest stage possible in their school careers. This article analyzes an effectiveness trial of a formative assessment program developed to meet these goals and conducted in the first two years of elementary school. The examination of the longitudinal dataset from Grades 1–3 (N = 2,330) revealed an effect after the implementation, which was maintained at nearly the same effect size one year after completion of the program. The findings imply that formative assessment can foster the arithmetic achievement of students at risk as well as that of the entire class without changing the curriculum or teachers’ individual teaching style.

Keywords:

• Formative assessment
• mathematical difficulties
• mathematics
• evaluation
• intervention

1. Introduction

It is well known that mathematical skills before school entrance are predictive of mathematical achievement at the end of primary school (e.g., Aunola et al., 2004; MacDonald & Carmichael, 2018). Therefore, early interventions at school entrance aiming at improving the mathematical knowledge and skills of students have the potential to significantly improve mathematical outcomes (e.g., Clements et al., 2013; for an overview see Clements & Sarama, 2011).

Interventions following a formative assessment approach have been described as effective in improving student performance, in particular the achievement of low-performing students (Dunn & Mulvenon, 2009). Formative assessment can be described as a method eliciting, interpreting, and using evidence about student achievement to adapt instruction to the needs of students (Black & Wiliam, 2009). Despite the potential of formative assessment for improving the mathematical achievement of students, research has shown that both formative assessment practice and the development of formative assessment materials are challenging (e.g., Black & Wiliam, 1998; Lee et al., 2012, Quyen & Khairani, 2016; Yin et al., 2008) and that theoretically effective elements are not always followed by the expected effects (Maier et al., 2016, Förster & Souvignier, 2014). Therefore, studies that test the effectiveness of theoretically developed interventions in practice are needed (e.g., Schütze et al., 2018).

In this context, the mathematics support program “Mathe macht stark” (MMS; Mathematics Makes You Strong) for at-risk students in the first and second grades of elementary school (age 6–8 years) was developed, coordinated, and financed by the German federal state of Schleswig-Holstein. MMS is based on formative assessment and was developed in line with recommendations made on the basis of previous implementation studies. These recommendations concern, for example, the frequent use of assessment tasks and the subsequent adaptation of teaching based on the obtained information, the easy administration of the program in the regular classroom, a clear alignment with the educational standards and the prescribed curriculum being taught, as well as the recommendation that the program not be time-consuming and that it accompany professional development (Frohbieter et al., 2011; Herman & Gribbons, 2001; Karuza, 2004; Lembke & Stecker, 2007; Nicol & Macfarlane-Dick, 2006; Sharkey & Murnane, 2006; Vendlinski et al., 2009). The MMS program provides material for teachers that can be used during and in addition to regular mathematics teaching (i.e., it does not involve either additional curriculum elements or prestructured lessons) to diagnose and support at-risk students. The program implementation includes short in-service teacher training sessions with information about the program and how to use the assessment tasks and the information obtained from these tasks to support students’ learning. The aim of the program is to recognize students’ learning difficulties in arithmetic at the earliest stage possible and to improve students’ understanding of the key concepts of the arithmetic curriculum in Grades 1 and 2. The effectiveness of the MMS program was examined in a three-year longitudinal study, which covered the two-year implementation phase in Grades 1 and 2 and a follow-up assessment one year after the intervention at the end of Grade 3. Schools that did not participate in the MMS program served as the comparison group. The evaluation also included an investigation into whether the program’s effect can be increased by providing the schools with additional teacher working hours. By presenting the findings of this study, which are based on 135 school classes, this article answers the call for research results to back up formative assessment programs by conducting ecologically valid implementation studies.

1.1. Students with Mathematical Difficulties—The Need for Early Intervention

Students already have different levels of cognitive prerequisites (Magnuson & Duncan, 2006) and prior knowledge (Hoff & Tian, 2005; Verdine et al., 2013) before entering elementary school. The children’s socioeconomic status (SES) has been associated with those differences in early childhood (e.g., Klein et al., 2008).

Even though growth trajectories are highly variable across students (Dumas et al., 2019), studies show that early mathematics skills are related to later mathematics learning (the average effect size of school-entry mathematics skills for later mathematics achievement in school was .34; Duncan et al., 2007). Morgan et al. (2009) showed that children with mathematical difficulties had the lowest growth rates from first grade through fifth grade. Furthermore, there is evidence of an achievement gap, in which under-resourced students systematically underperform their more privileged peers in math (Bohrnstedt et al., 2015; Lee, 2002; Reardon & Galindo, 2009). Those early achievement gaps in mathematics tend to widen throughout schooling (Burchinal et al., 2011; Cameron et al., 2015; Klibanoff et al., 2006).

Therefore, students whose performance in early mathematics is considerably lower than that of their peers are often identified as being at risk or disadvantaged regarding their future development in mathematics (the 25% to 35% lowest performing students;¹ Hanich et al., 2001; Jordan et al., 2003; McLean & Hitch, 1999; Morgan et al., 2009). Those students are often called low-achieving or low-performing students (e.g., Baker et al., 2002; Smith et al., 2013), students with mathematical difficulties (MD; e.g. Jordan et al., 2003), or students at risk for MD (e.g. Gersten et al., 2015; Morgan et al., 2009). In this article, we use the latter term.

The results regarding the association between early and later mathematics achievement as well as the persistence of differences in early mathematical knowledge and skills provide a compelling rationale for schools to support students who are at risk for MD. This is then linked to the necessity of providing schools with effective methods for supporting students early in their school careers (Smith et al., 2013). A wide variety of early interventions aiming at improving the mathematical achievement of students have been developed, some of which have demonstrated the capability to significantly improve mathematical outcomes (e.g., early numeracy preventative Tier 2 intervention, Bryant et al. 2011; Building Blocks, Clements & Sarama, 2007; Clements et al., 2013; cognitively guided instruction, Carpenter et al., 2015; preventive 1st-grade tutoring, Fuchs et al., 2005). Although preschool intervention can significantly steepen learning trajectories (Dumas et al., 2019), some studies indicate that positive intervention effects decrease or fade out over time (Bailey et al., 2018; Leak et al., 2012; Li et al., 2020; Preschool Curriculum Evaluation Research Consortium, 2008; Turner et al., 2006).

1.2. Formative Assessment

Using information obtained by formative assessment techniques has been described as an effective means to improve student performance, in particular that of lower-performing students (Dunn & Mulvenon, 2009). Fuchs et al. (2008) even conclude that ongoing progress monitoring is the most essential element in any type of intervention with low-performing students, and this needs to occur in the form of formative assessments. Definitions of formative assessment differ in the literature. Sometimes formative assessment is defined as testing material, sometimes as a process of assessment, but it is always described with a formative purpose (Dunn & Mulvenon, 2009). In this article, we follow the description of Black and Wiliam (2009), who consider it to be a characteristic element of formative assessment “that evidence about student achievement is elicited, interpreted, and used by teachers, learners, or their peers, to make decisions about the next steps in instruction that are likely to be better, or better founded, than the decisions they would have taken in the absence of the evidence that was elicited.” (Black & Wiliam, 2009, p. 9). There are several criteria for classifying different types of formative assessment. A fairly common distinction regarding the degree of formalization is made by Shavelson (2006). The author distinguishes between (1) on-the-fly formative assessment (lesson sequences in which teachers spontaneously identify misconceptions and gaps in understanding, provide feedback, and adapt the lesson content accordingly), (2) planned-for-interaction assessment (planned lesson sequences, such as question-and-answer sequences that provide information about the students’ learning level), and (3) curriculum-embedded assessment (relatively strongly formalized; it can be part of the conventional teaching activity, e.g., homework or presentations, but it can also be embedded in the curriculum at key points through additional assessment instruments such as individual tasks or whole test procedures and batteries; for a description of possible implementations, see Schütze et al. [2018]).

The basic idea of formative assessment—that is, using assessment to obtain information about student thinking in order to compare it with the intended learning goal and, in the case of inadequate fit, to modify teaching and learning activities to help students achieving the learning goal—has gained increasing attention since the mid-1990s (Souvignier & Hasselhorn, 2018). An increasing number of studies deal with the impact and design of formative assessment in mathematics instruction. Based on their critical analysis of the work of Black and Wiliam (1998) and on their own review of nine studies from 1999 to 2007, Dunn and Mulvenon (2009) argue that there is some support for the impact of formative assessment on student achievement, but there is still a need for studies with more efficient methodologies and designs to facilitate conclusive results about its effectiveness. There is evidence that formative assessment has a positive impact on the whole group of students (e.g., Decristan et al., 2015). However, Phelan et al. (2011) report findings that students with higher pretest scores benefited especially from the formative assessment. Other studies show that formative assessment is an effective means for low-achieving students (Bottge et al., 2021, King, 2016).

Kingston and Nash (2011, 2015a, 2015b) report a small-to-medium effect of formative assessment on performance in their meta-analysis based on 13 studies from 1990 to 2010 using a control-group or comparison-group design. The results of their moderator analysis suggest that the effectiveness depends on the school subject (e.g., they found a higher effect on English language arts, d = .32, than on mathematics, d = .17, or science, d = .09) as well as on features related to the specific implementation of the formative assessment (e.g., professional development, d = .30, and the use of computer-based formative systems, d = .28, were more effective than e.g., curriculum-embedded assessments, d = −.05, and the specific use of student feedback, d = .03). Yet studies on long-term effects (≥ 1 year) are scarce. Those studies that included a follow-up measure found contradicting outcomes regarding long-term effects (Andersson & Palm, 2017).

Despite the supportive evidence regarding formative assessment, research has identified factors that hinder an effective implementation of formative assessment in everyday school practice. The implementation of formative assessment is still very demanding, time-consuming, and frequently rather unsystematic (e.g., Black & Wiliam, 1998; Quyen & Khairani, 2016; Yin et al., 2008). Other studies describe a lack of teaching materials suitable for systematic and sensitive assessments. Furthermore, schools and teachers often lack the background and capacity to engage in assessment (Herman et al., 2006; Herman & Gribbons, 2001; Quyen & Khairani, 2016; Stiggins, 2002; Yin et al., 2008).

In addition, the precise implementation of effective formative assessment programs is content- and competency-specific and there is some question whether it can be described in general terms. Such content-related principles can be found in the subject-specific didactical literature and often have no explicit reference to formative assessment (Schütze et al., 2018). But there are promising general factors for achieving successful implementation based on previous studies, that are discussed in the literature (Phelan et al., 2011). In the following, we outline and discuss some of these promising factors but also barriers for successful curriculum-embedded formative assessment implementation based on previous studies, without any claim to completeness. For example, curriculum-embedded formative assessment is recommended, including frequent assessments to generate information about students’ learning progress, so that teachers can continuously make instructional decisions (Lembke & Stecker, 2007; Nicol & Macfarlane-Dick, 2006). However, research suggests that teachers are often unable to conduct the assessments and use the information from assessments to make instructional decisions (Karuza 2004; Vendlinski et al., 2009). Formative assessment practice including a dynamic process of evidence elicitation, analysis, and action is a very complex and demanding task for the teacher and makes demands on teachers’ content and pedagogical knowledge. In this context, Phelan et al. note critically that “there is a great deal of rhetoric surrounding formative assessment and doing something with the results, but in reality, teachers do not always have the wherewithal to do anything except repeat what they have already done” (2011, p. 331). Hence, professional development is required for teachers to learn how to conduct such assessments, and especially how to analyze the data and derive instructional decisions to help close the gaps between a student’s current performance and the stated learning goal (Herman et al., 2006; Karuza, 2004; Sharkey & Murnane, 2006). Other barriers for realizing formative assessment in the classroom are the preparation, application time, and effort needed (e.g., Lee et al., 2012). Despite the advantages of curriculum-embedded formative assessment tools created for teachers, interventions seem to offer benefits when teachers are involved in the practical implementation. Therefore, teachers need time for this extra work, ideally during the school day (Sharkey & Murnane, 2006). In particular, teachers need effective strategies with which they can realize formative assessment within the given time constraints of school lessons (Hondrich et al., 2016). Therefore, the assessment should be easy to administer, clearly aligned with the prescribed educational standards and curriculum being taught, user-friendly, and not time-consuming (Lembke & Stecker, 2007; Sharkey & Murnane, 2006). Finally, a specific formative assessment program has to be implemented over a long period of time in order to obtain a sustained integration of formative assessment into teachers’ pedagogical practices (Vendlinski et al., 2009).

In summary, there is a need to provide schools with effective methods for supporting at-risk students at the earliest stage possible in their school careers. Formative assessment can potentially be an effective method in conjunction with a successful implementation in everyday school practice. But even with a consensus on the effectiveness of formative assessment in facilitating student learning, a need for further research still exists on the implementation of this approach as well as for materials developed specifically for practical use (Dunn & Mulvenon, 2009; Schütze et al., 2018). Furthermore, there is a need for intervention studies in which specific measures to implement the concept of formative assessments are tested in practice and are scientifically evaluated (Souvignier & Hasselhorn, 2018); there is also a need for follow-up studies (Andersson & Palm, 2017). Therefore, the purpose of this study was to evaluate the implementation of a formative assessment program that aims to support students at risk for MD in the first two grades of elementary education.

1.3. The MMS Support Program—Overview of the Study

The MMS program is a state-driven mathematics support program in Germany that aims to support elementary school students at risk for MD. The program is based on formative assessment and takes into account the above-mentioned factors that previous research has identified as being promising for an effective implementation. MMS focuses on students at the beginning of their school career in Grades 1 and 2, where mathematics learning starts in a formal context (Germany does not have a mathematics curriculum for kindergarten and kindergarten teachers are usually not trained in mathematics education). The idea of MMS is to support teachers in their individual everyday mathematics teaching rather than to prescribe how to teach ready-made teaching material. This approach includes a sequence of assessments as well as systematic information to support teachers’ use of those assessments to improve student learning. In this manner, assessment material and guidance are provided to teachers aiming to diagnose and—if necessary—improve students’ understanding of the key arithmetic concepts and principles in Grades 1 and 2 that are central prerequisites for further mathematical development. The focus on arithmetic is motivated by the crucial role of arithmetic in students’ mathematics development (e.g., Jordan et al., 2003). Arithmetic makes up a large part of the German elementary school standards for mathematics, which is reflected in the mathematics curriculum in each federal state² (KMK—Sekretariat der Ständigen Konferenz der Kultusminister der Länder in der Bundesrepublik Deutschland [the Standing Conference of the Ministers of Education and Cultural Affairs of the Federal States in the Federal Republic of Germany], 2005).

1.3.1. Content of the MMS Support Program

The two-year intervention targets central arithmetic knowledge and skills. The intervention is structured into seven “milestones” of the arithmetic curriculum. A milestone encompasses 2–5 mathematical topics to be taught within a curriculum unit, and for each topic learning goals for the students are defined. Together, all seven milestones comprise 23 basic arithmetic topics from the Grade 1 and Grade 2 curriculum, for example, ordering and comparing (sets, lengths, numbers), counting (forward, backward, counting on, counting by 2s), number relations (successor, predecessor), part-whole relations, elementary strategies for addition and subtraction (counting on, doubling, halving, commutativity, bridging through 10), the place-value system, etc. These topics were chosen because of research findings on their importance for students’ arithmetic achievement. The milestones are arranged in accordance with the curriculum of Grade 1 and Grade 2. Materials were developed to enable teachers to identify students at risk for MD, to detect the specific difficulties of those students in reaching the learning goals, and to select appropriate teaching material and strategies to support those students in learning the specific arithmetic topics. The materials comprised a teacher booklet and a student booklet, structured in line with the arithmetic topics of the seven milestones. The MMS material was developed by expert teachers from the Institute for Quality Development in Schools in Schleswig-Holstein and reviewed by mathematics education researchers. The material was professionally designed after completion of a pilot phase in which small adjustments were made.

The focus on professional development was another pillar of the MMS intervention. When teachers use MMS for the first time, the implementation is accompanied by short teacher training sessions organized by the Institute for Quality Development in Schools in Schleswig-Holstein. The professional development focus has two goals. First, the teachers should receive information about the design, materials, and implementation of MMS and should receive guidance on how to integrate the program into their daily teaching practice. Second, the training sessions provide opportunities for the teachers to acquire relevant content knowledge, pedagogical content knowledge, and assessment knowledge (Timperley et al., 2007). This specific professional knowledge is relevant for the teachers in interpreting the diagnostic information obtained through the MMS assessment material and for connecting this information with effective teaching material and strategies to support students with specific difficulties. For the second goal, the material is organized in a flexible manner so that it can easily be adapted to the needs of the participating teachers (e.g., if specific gaps in teachers’ knowledge become apparent during a session). The framework of the training sessions was designed according to criteria for effective teacher professional development, as described in meta-studies (e.g., Timperley et al., 2007; Yoon et al., 2007). More specifically, this means that the sessions encompass training on five occasions (an initial session of seven hours and four sessions of four hours in the first school year) and are coherent and content-focused (each session is directly associated with the MMS content topics), interactive (teachers can reflect on their experiences between sessions), based on the teachers’ needs (learning opportunities are provided if gaps in professional knowledge become apparent), and practical (the teachers can directly integrate the presented material into their next lessons). In addition, the teachers are given an easily accessible teacher booklet with guidelines on how to use the MMS material, which provides relevant information for each content topic at a glance.

1.3.2. Intervention Sequence

The MMS materials for the different arithmetic topics were designed to complement the different kinds of existing curricular material in such a way that the MMS materials can fit into the mathematics teachers’ individual teaching frameworks and timelines during regular mathematics lessons. Hence, the idea of MMS is not to present new or innovative content to the teachers. Instead, the main feature of MMS is to combine well-proven assessments, formats, and learning material in such a way that it can easily be understood and used by teachers. Because the topics treated in MMS are the core ideas of the first- and second-grade mathematics curriculum, the assessments can easily be incorporated into the ongoing curriculum at appropriate points. MMS assists teachers in: (1) identifying students’ individual learning difficulties in mathematics at the earliest stage possible, and (2) offering suitable learning opportunities and support for students at risk for MD. The procedure for the MMS intervention is as follows:

1. For each of the 2–5 topics belonging to a milestone, the teacher administers 3–4 short tasks in the form of a class screening (approx. 10 minutes) after a mathematical topic has been taught. The students work individually on the screening tasks in their student booklet (Figure 1). The tasks cover the learning goals for the topic.³ The instructions are included in the teacher booklet and read aloud by the teacher. Furthermore, the teacher booklet contains short didactical information regarding the learning goals, tasks, and the domain. Based on the students’ results on these tasks, the teacher can identify students in the class who potentially did not reach the learning goals related to the assessed topic.

2. The teacher conducts a short interview with the identified students, comprising 3–4 additional tasks, to get insights into the mistakes made by the students and their possible misconceptions (Figure 2). The interview can be done individually or in small groups. Precise instructions are presented in the teacher booklet and in the student booklet. Based on the interview the teacher can determine where the students are in relation to the learning goal.⁴

3. The teacher can choose suitable learning opportunities based on the information gained through the solutions reached or mistakes made by the students in the class tasks and through the students’ answers in the task-based interview. The teacher booklet contains a brief description of suggested learning activities and corresponding material that is directly linked to each interview task and the observations. These suggested learning activities are briefly explained with the help of didactical recommendations.⁵ The MMS program refers only to standard mathematics learning material available in every classroom in Germany.

4. Steps 1–3 can be implemented in the mathematics classroom when addressing the corresponding mathematical topic so that teachers still have full flexibility. When all topics of a milestone are finished, MMS offers a short summative assessment (up to 10 items) for the milestone content as a test for the entire class. The test covers the learning goals from all mathematical topics in the respective milestone (2–5 topics) to evaluate to what extent the students reached those learning goals.

Figure 1. Example of two tasks from Milestone 2, Part-Whole Relationships. Instructions are read aloud by the teacher. (1. “You see 2 boxes. In the first box there are 5 dots in total. How many of them are hidden? In the second box there are 7 dots in total. How many of them are hidden?” 2. “Write down all ways in which the number 6 can be broken down into smaller numbers in the decomposition house. And now write down all ways in which the number 8 can be broken down into smaller numbers).”.

Figure 2. Example of an interview task from Milestone 2, Part-Whole Relationships. Instructions are given by the teacher (1. “How could you divide the apples among the plates? Is there another possibility?” 2. “Take a good look at the picture. What number is divided here? How many circles are lying left of the pencil?” The child is allowed to reenact the task with two-colour counters, a teaching material available in all German elementary schools).

The program started in the 2013/2014 school year in 100 elementary schools in the German federal state of Schleswig-Holstein. All schools received free material for each student. The program is financed by the Ministry of Education, Science and Cultural Affairs of the German federal state of Schleswig-Holstein.

1.4. The Present Study

The goal of this study was to investigate the effectiveness of the two-year MMS program after completion as well as the sustained effect one year after completion. The study was conducted with the support of IPN—Leibniz Institute for Science and Mathematics Education (Germany). The IPN researchers, however, were not involved in the implementation of the MMS program. We analyzed the effect of MMS on both students at risk for MD and the entire group of students, as the aim of MMS is to support students at risk for MD (see Section 1.3) and because formative assessment is known to also have a positive effect on average-achieving students (see Section 1.2).

The evaluation was conducted in the form of an effectiveness trial to take into account the naturally varying implementation of the real-world situation. Classroom-based learning is strongly contextualized and influential factors, which often cannot be controlled by the teachers and which prevent, restrict, support, or foster student learning, are likely to influence not only student achievement but also interventions designed to improve achievement (e.g., at the individual level: social contexts, community influences, norms, values, and cultural capital; at the school level: beliefs, commitments, education, experience, roles, professionalization, and autonomy of teachers; McDonald et al., 2006). Therefore, the goal of this effectiveness trial was to analyze whether the intervention was effective in the real world of typical school settings. We used a quasi-experimental design with two intervention groups and one comparison group. We implemented two intervention groups because time turned out to be an essential factor for teachers to be able to implement the formative assessment methods (see Section 1.2) and had not been an initial component of the MMS program; hence, one group of schools received the originally planned MMS intervention, and a second group of schools received the MMS intervention with two extra teacher working hours per week to support the implementation of the MMS program.

We analyzed the data of a three-year longitudinal study with 135 elementary school classes comprising information on teachers as well as information on students concerning basic cognitive abilities as well as basic numerical and language skills at the beginning of Grade 1 and mathematics skills at the end of Grades 2 and 3. The sample was selected in one federal state of Germany (Schleswig-Holstein). As a result, the prescribed mathematics curriculum in this sample is the same (i.e., the state-wide curriculum). Therefore, this sample allowed us to examine the effects of the formative assessment intervention (MMS) on students’ mathematics achievement with a longitudinal design and a sound sample size while controlling for relevant covariates. Accordingly, our study addressed the following research question:

Does the MMS intervention have an effect on students’ mathematics achievement at the end of Grade 2 as well as at the end of Grade 3?

We developed three hypotheses based on this research question:

1. Given that the target group of the MMS program was low-performing students, our first hypothesis was that the MMS intervention would have an effect on the achievement of students at risk for MD.

2. As research suggests that formative assessment raises not only the achievement of low-performing students, but also that of average-achieving and high-achieving students (e.g., Black & Wiliam, 1998; Decristan et al., 2015; Phelan et al., 2011), our second hypothesis was that the MMS intervention would also have an effect on the achievement of the entire group of students.

3. Studies show that one barrier for realizing formative assessment strategies is the time and effort needed for preparation and application (Lee et al., 2012) and that teachers need time to do this extra work, ideally during the school day (Sharkey & Murnane, 2006). This is not addressed by the MMS program. Therefore, our third hypothesis was that the intervention would have a greater effect if the schools received additional teacher working hours per week.

2. Methods

2.1. Design and Participants

The study was based on a quasi-experimental design with two intervention groups and one comparison group. One intervention group (MMS+) consisted of 20 elementary schools that received the MMS material (student material, teacher booklet) for Grades 1 and 2 and the in-service teacher training sessions as well as two extra teacher working hours per week.⁶ The second intervention group (MMS) consisted of 10 schools and received only the MMS material and the in-service teacher training sessions. The comparison group consisted of 10 elementary schools. Teachers in the comparison group did not get any specific support and were asked to give their regular mathematics instruction. It should be mentioned that, due to a directive of the Ministry of Education, all teachers are required to provide specific learning opportunities for low-achieving students. Although there is no supervision of whether teachers follow this directive, we assumed that the teachers in the comparison group also used some kind of specific learning material. In total, the longitudinal sample consists of 2,330 students nested in 135 classes from 40 elementary schools (about 10% of the student cohort in this federal state).

The 30 schools of the two intervention groups were selected from the 100 schools in the federal state of Schleswig-Holstein that had volunteered for the first wave of the MMS implementation in 2013.⁷ The 30 schools were chosen so as to ensure a broad geographical distribution (e.g., urban and rural regions in different parts of the federal state) as well as an adequate distribution of the socio-economic and cultural background. From the 30 schools, 20 schools were randomly chosen for the MMS + group, and we checked that the two groups were comparable with respect to geographical distribution and socio-economic and cultural background. Finally, the 10 schools of the comparison group were chosen so that they were comparable to the intervention group schools in terms of geographical distribution and the socio-economic and cultural background. This means that the sample of our study was not selected strictly randomly. However, since the material was offered for free—which was appealing for all schools—a positive selection was not necessarily the case. Finally, the intervention and comparison groups were comparable concerning the student composition with respect to basic cognitive abilities (see Section 3.1). As a result of sampling all schools were from the same federal state and therefore followed the same state-wide curriculum.

2.2. Measures

2.2.1. Mathematics Achievement

The assessment of students’ learning progress in arithmetic at the end of Grades 2 and 3 was based on federal state curriculum standards. The items for these tests were adapted from approved tests of the longitudinal “personality and learning development of elementary school children” study (Persönlichkeits- und Lernentwicklung von Grundschulkindern, PERLE; Lipowsky et al., 2011). The items cover arithmetic knowledge that should be acquired in the Grade 2 and Grade 3 curriculum. The test at the end of Grade 2 (59 items, 40 min) for example covered addition, subtraction, multiplication, division in the 0–1,000 number range, doubling, halving, and factors of a given number. The test at the end of Grade 3 (40 items, 40 min) covered, for example, addition, subtraction, multiplication, division in the 0–10,000 number range, and factors of a given number. The items were constructed and formatted in such a way that they were similar to typical items that students encounter in examples in textbooks. However, the format of these test items differed from that of the items in the MMS assessment material. The short-constructed response format was used (response in the form of a number). For the data analysis some of the items were combined and, in those cases, partial credit coding was used. The Grade 2 and Grade 3 tests had 14 common items so that the same scale could be used for both tests (see Section 2.4). The reliability of the tests was very good (EAP/PV reliability = .90 to .94).

2.2.2. Control Measures

Since this study is a quasi-experiment, the intervention and control groups are not strictly comparable. To analyze the effect of the intervention on the arithmetic achievement, we control for several covariates that might have an impact on students’ arithmetic learning. As mentioned in Section 1.2, teachers’ professional knowledge might influence the quality of formative assessment. Because in the federal state of Schleswig-Holstein about 40% of the mathematics elementary school teachers do not have a formal qualification to teach mathematics (i.e., they are out-of-field teachers), we included the covariate “earned certificate to teach mathematics” (studied mathematics or not) as an indicator of teacher knowledge in our analyses. We also controlled for the covariate “mathematics textbook” as the mathematics textbooks used in this sample had shown an effect on student achievement in a previous study. There were seven textbook categories in our sample (four categories for the frequently used textbooks A–D, the category “other books” for textbooks that were used only sporadically in our sample, “no books” for classes that did not use a textbook, and “missing book” for missing information). Data on earned certificate to teach mathematics and mathematics textbook were collected using a teacher questionnaire.

In order to control for individual differences between individual students as well as between classes, students’ basic numerical and language skills were measured at the beginning of Grade 1 using approved standardized tests (basic numerical skills: Hamburger Rechentest, HaReT, [Lorenz, 2007], Cronbach’s α =.74; German language skills: Münsteraner Screening, MÜSC, [Mannhaupt, 2012], Cronbach’s α = .72). Students’ basic cognitive abilities were measured using the Culture Fair Intelligence Test (CFT 1–R, Weiß & Osterland, 2013, Cronbach’s α = .91).

To account for differences between schools, we control for the average of school-level achievement scores from the German national mathematics comparison test at the end of Grade 3 (Vergleichsarbeiten, VERA) from 2010 to 2014 in the multilevel regression model.

2.3. Procedure

Data collection started in the school year 2013/2014. At the beginning of Grade 1, basic numerical and language skills were assessed. This was followed by an assessment of basic cognitive abilities halfway through Grade 1. After completion of the MMS intervention at the end of Grade 2 and one year after completion at the end of Grade 3, the mathematics achievement was assessed. All tests were administered by trained test administrators in accordance with the respective manuals.

2.4. Data Analyses

The tests of Grade 2 and Grade 3 had 14 common items and 45 and 26 unique items, respectively. This anchor-item design of the mathematics tests allowed us to estimate the student mathematics scores in Grades 2 and 3 on the same scale using Item Response Theory (IRT). Since the test contained partial credit items, an extension of the Rasch model, the IRT partial credit model (PCM; Masters, 1982), was used. The student scores were estimated in a two-step procedure. First, the Grade 2 test was calibrated using the data of all students who completed the test to monitor the quality of the items and estimate the item parameters. Second, the same procedure was done for the Grade 3 test, but the item parameters of the 14 common items were fixed to the parameters of the Grade 2 scaling (fixed item parameter calibration, FIPC). In this manner the two tests are linked and the item and person parameters of the Grade 3 test can be interpreted on the Grade 2 scale.

Next, five plausible values (PV) for each of the 2,230 students were generated for Grade 2 and Grade 3 each, by fitting a latent regression model and using item parameters anchored at their estimated values from the previous calibration. A detailed review of the PV methodology is given in Mislevy (1991). This method was chosen because it allows for unbiased population-level analyses of competence distributions and context variables (Davier et al., 2009).

To analyze the effect on students at risk for MD, we first identified the low-achieving students in the sample. Therefore, in line with the cited literature, students who scored in the lowest 25th percentile of the basic numerical skill assessment at the beginning of Grade 1 were categorized as being at risk for MD. Second, we estimated multiple regression models to estimate the effects of the covariates and independent variables on the arithmetic achievement. Because of the nested structure of the data (students nested within classes), we used a sandwich estimator to adjust standard deviations for non-independence (with “type = complex” and “cluster = group” in Mplus).⁸ To analyze the effect of the intervention on mathematics achievement, we tested four regression models. In the first model we included the basic cognitive abilities as well as basic numerical and language skills (measured by the HaReT, MÜSC, and CFT 1–R), because these variables are known to have an effect on mathematics achievement. In the second model, we included the mathematics textbook (the mathematics textbook used) and the earned certificate to teach mathematics (studied mathematics or not) as dummy-coded covariates, because according to previous studies we expect those variables to have an effect on students’ mathematics achievement (Mullis et al., 2012). In the third model, we included the mean of the school VERA scores from 2010 to 2014 to account for differences between schools. In the last model, we included the intervention group variables as dummy-coded variables. In this way, we tested if the coefficients of the intervention groups have any significant predictive effect in addition to the coefficients of the basic cognitive abilities as well as basic numerical and language skills, mean of the school-level VERA sores, textbooks and earned certificate to teach mathematics. Or stated differently, we analyzed the effect of coefficients of the intervention groups while holding the covariates constant.⁹

To analyze the effects of the intervention on the entire class, the whole sample was analyzed with multilevel random intercept models to control for the nested structure of the data. First, models without predictors (null models) were run to estimate the partition of variance between and within classes. Second, models were estimated including individual characteristics (at individual level) and class composition (at class level). The basic cognitive abilities as well as basic numerical and language skills were included on the individual level to account for individual differences between students. For all three variables, the individual scores were aggregated to class means and included on the class level to control for class composition. In a third step, we added mathematics textbook and earned certificate to teach mathematics and in a fourth step the mean of the school-level VERA sores to the model. In a final model, we included the intervention group variables on the class level.

To test for significant differences between the regression coefficients of the two intervention groups, we used the Wald chi-square test.¹⁰

Scores for the basic cognitive abilities and basic numerical and language skills as well as the mathematics achievement scores were standardized so that the corresponding ß-coefficients can be interpreted as effect sizes similar to Cohen’s d (see Tymms, 2004). For missing data on the independent variables for the basic cognitive abilities and basic numerical and language skills, we applied a full-information-maximum-likelihood (FIML) approach in Mplus 7.0 (Muthén and Muthén 1998–2012). FIML combines missing data and parameter estimation in a single step and uses all the available information (Enders, 2010). The percentage of missing data in the basic cognitive abilities and basic numerical and language skills was 6.1% for the CFT 1–R, 8.1% for the HaReT, and 8.6% for the MÜSC. These tests were a nonobligatory part of the regular mathematics lessons and we did not detect specific patterns in the missing data. Due to sample selection, there were no missing data for the predictor support program and the school-level mathematics achievement scores (VERA).

3. Results

First, we present a short summary of the descriptive information obtained about the earned certificate to teach mathematics, the mathematics textbooks, and the students’ basic cognitive abilities as well as basic numerical and language skills.

3.1. Descriptive Results

About 39% of the teachers in our sample had a formal qualification to teach mathematics, whereas 51% teachers were qualified for other subjects and taught mathematics as out-of-field teachers. The remaining 10% of the teachers did not provide information about their qualification.

Of the 2,330 students, 562 were identified as being students at risk for MD. The descriptive results for the cognitive abilities as well as basic numerical and language skills and the dependent variables (mathematics achievement) in the two intervention groups and the comparison group are reported in Table 1 (entire sample) and Table 2 (students at risk for MD). To analyze baseline equivalence, we tested whether the intervention groups differed from the comparison group in the basic cognitive abilities and numerical and language skills. For the sample of the whole class we found only small differences (Cohens d between −0.06 and 0.15). The differences for the sample of students at risk were a bit higher. For the basic numerical and language skills Cohens d was between −0.19 and 0.19, and for the basic cognitive abilities the comparison group showed higher scores than the two intervention groups (Cohens d: 0.28 and 0.33). All our analyses presented below are adjusted by these measures. We also tested whether earned certificate to teach mathematics was independent of the group. The number of teachers having a formal qualification to teach mathematics did not differ significantly between the three groups (χ²[2, N = 135] = 6.65, p = .16).

Table 1. Mean values and standard deviations for individual basic cognitive abilities as well as basic numerical and language skills and mathematics achievement in Grades 2 and 3 (dependent variables) in the intervention groups for the whole sample.

	Basic numerical skills	Basic cognitive abilities	Language skills	Arithmetic achievement Grade 2	Arithmetic achievement Grade 3
MMS+	31.06 (7.43)	27.00 (9.08)	33.06 (6.39)	1.09 (1.74)	1.73 (1.35)
MMS	32.31 (6.81)	26.96 (8.91)	33.87 (5.76)	1.23 (1.73)	1.93 (1.34)
Comparison group	31.19 (7.44)	27.25 (8.66)	33.45 (5.88)	0.80 (1.73)	1.57 (1.29)

Note. Arithmetic achievement at the end of Grades 2 and 3 are PVs from the IRT model whereas all other results are sum scores from the tests.

The maximum score for cognitive abilities was 44, for basic numerical skills 43, and for language skills 40.

Table 2. Mean values and standard deviations for individual basic cognitive abilities as well as basic numerical and language skills and mathematics achievement in Grades 2 and 3 (dependent variables) in the intervention groups for students at risk for MD.

	Basic numerical skills	Basic cognitive abilities	Language skills	Arithmetic achievement Grade 2	Arithmetic achievement Grade 3
MMS+	20.99 (5.02)	19.62 (8.06)	28.36 (7.64)	–0.08 (1.65)	0.84 (1.27)
MMS	21.97 (4.35)	19.27 (7.85)	29.67 (6.91)	–0.09 (1.58)	1.07 (1.19)
Comparison group	21.00 (5.55)	21.85 (7.94)	29.79 (6.57)	–0.21 (1.64)	0.92 (1.26)

Note. Arithmetic achievement at the end of Grades 2 and 3 are PVs from the IRT model whereas all other results are sum scores from the tests.

The maximum score for cognitive abilities was 44, for basic numerical skills 43, and for language skills 40.

Null models were run in the multilevel analyses for the entire sample to determine the share of between-class variance in students’ mathematics achievement (given by the intra-class coefficient, ICC). At the end of Grade 2, there was 19.2% variance between the classes; at the end of Grade 3, the variance between the classes was 17.1%. Hence, the mathematics achievement differed depending on the class attended by the student.

3.2. Results for Students at Risk for MD

Regarding the individual characteristics at school entrance (see Model 1 in Tables 3 and 4), basic cognitive abilities and basic numerical skills showed the expected significant association with mathematics achievement at the end of Grades 2 and 3. Controlling for those individual characteristics at school entrance, the mean of the school-level VERA scores showed no significant association with mathematics achievement at the end of Grades 2 and 3. Three textbooks showed a significant association with mathematics achievement at the end of Grade 2 (β = .37 − .47) and two textbooks showed a significant association with mathematics achievement at the end of Grade 3 (β = .32 − .48). In order to avoid putting too much information into the tables, these covariates are not displayed in Tables 3 and 4. These outcomes are in line with previous research on textbooks.

Table 3. Regression for individual and classroom covariates and intervention groups on arithmetic achievement for students at risk for MD at the end of Grade 2.

	Model 1	Model 2	Model 3	Model 4
Basic cognitive abilities	0.40** (0.05)	0.38** (0.05)	0.38** (0.05)	0.39** (0.05)
Basic numerical skills	0.22** (0.05)	0.22** (0.05)	0.22** (0.05)	0.21** (0.04)
Language skills	0.09* (0.04)	0.10* (0.04)	0.10* (0.04)	0.10* (0.04)
Earned certificate to teach mathematics		–0.14 (0.10)	–0.14 (0.10)	–0.18 (0.10)
VERA mathematics school average 2010–2014			0.08 (0.04)	0.06 (0.04)
MMS+				0.27* (0.12)
MMS				0.35* (0.14)
Intercept	0.00 (0.05)	–0.17 (0.12)	–0.16 (0.12)	–0.38** (0.14)
R²	0.33	0.35	0.35	0.37

Note. **p < .01; *p < .05.

standard errors are in parentheses, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group, the dependent variable and non-categorical independent variables are standardized;.

Models 2–4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

Table 4. Regression for individual and classroom covariates and intervention groups on arithmetic achievement for students at risk for MD at the end of Grade 3.

	Model 1	Model 2	Model 3	Model 4
Basic cognitive abilities	0.40** (0.05)	0.37** (0.05)	0.38** (0.05)	0.38** (0.05)
Basic numerical skills	0.18** (0.05)	0.18** (0.06)	0.17** (0.05)	0.17** (0.05)
Language skills	0.06 (0.05)	0.06 (0.05)	0.06 (0.05)	0.06 (0.05)
Earned certificate to teach mathematics		–0.17 (0.12)	–0.16 (0.12)	–0.19 (0.12)
VERA mathematics school average 2010–2014			0.03 (0.03)	0.02 (0.03)
MMS+				0.12 (0.13)
MMS				0.29* (0.14)
Intercept	0.00 (0.05)	–0.12 (0.12)	–0.12 (0.12)	–0.27 (0.17)
R²	0.27	0.29	0.29	0.30

Note. **p < .01; *p < .05;

Standard errors are in parentheses, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group, the dependent variable and non-categorical independent variables are standardized;

Models 2–4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

When including the dummy variables for the intervention groups in the regression models, the explained variance increased slightly for Grades 2 and 3 (Grade 2: ΔR² = 1.6%, Grade 3: ΔR² = 1%; Model in Tables 3 and 4; the comparison group is the reference category). The regression coefficients of the two intervention groups in Grade 2 were of considerable size and showed the significant average difference in the standardized student achievement between the respective intervention group and the comparison group (Model 4 in Table 3: MMS: β = .35 and MMS+: β = .27). Therefore, we can conclude that, compared to the comparison group, the intervention had a positive effect on the mathematics achievement of students at risk for MD after a two-year implementation period. There was no significant difference between the effects of the two intervention groups, which means that the additional teacher working hours per week did not have an effect, X² (1, N = 2,330) = 0.41, p = 0.52. Regarding the end of Grade 3, only the MMS group showed a significant effect on the achievement of students at risk for MD (Model 4 in Table 4: β = .29) whereas the effect of MMS+ (β = .12) was not significant. Hence, one year after completion of the intervention, after teaching without intervention material, the MMS intervention still affected the achievement of students at risk for MD. The finding that the MMS + intervention with additional teacher working hours did not have a significant effect one year after the intervention ended is rather interesting and is discussed in Section 4.1.

3.3. Results for the Entire Class

Similar to the results found for students at risk for MD, cognitive abilities and basic numerical skills had a significant positive association with the mathematics achievement of the entire class at the end of Grades 2 and 3, whereas language skills had a significant positive but small association with mathematics achievement at the end of Grade 2 and did not have a significant association at the end of Grade 3 (see Model 1 in Tables 5 and 6).

Table 5. Multilevel regression for individual and classroom covariates and intervention groups on students’ arithmetic achievement at the end of Grade 2.

	Model 1	Model 2	Model 3	Model 4
Individual level
Basic cognitive abilities	0.38** (0.03)	0.38** (0.03)	0.38** (0.03)	0.38** (0.03)
Basic numerical skills	0.22** (0.03)	0.22** (0.03)	0.22** (0.03)	0.22** (0.03)
Language skills	0.06** (0.03)	0.06** (0.03)	0.06** (0.02)	0.06** (0.02)
Class level
Basic cognitive abilities (mean)	0.34** (0.11)	0.28* (0.12)	0.27* (0.11)	0.31** (0.11)
Basic numerical skills (mean)	0.07 (0.09)	0.09 (0.09)	0.07 (0.09)	0.03 (0.09)
Language skills (mean)	–0.05 (0.11)	–0.02 (0.11)	–0.04 (0.11)	–0.06 (0.11)
Earned certificate to teach mathematics		–0.05 (0.07)	–0.05 (0.06)	–0.09 (0.06)
VERA mathematics school average 2010–2014			0.05 (0.03)	0.05 (0.03)
MMS+				0.21** (0.07)
MMS				0.30** (0.07)
Intercept	0.01 (0.03)	–0.12 (0.07)	–0.12(0.07)	–0.30** (0.07)
Explained within-class variance	0.40	0.40	0.40	0.40
Explained between-class variance	0.26	0.39	0.39	0.49

Note. **p < .01; *p < .05;

Standard errors are in parentheses, variables on individual level, the VERA school-level mathematics score, and the output variable are standardized, basic cognitive abilities, basic numerical skills, and language skills on class level are arithmetic means, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group;

Models 2–4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

Table 6. Multilevel regression for individual and classroom covariates and intervention groups on students’ arithmetic achievement at the end of Grade 3.

	Model 1	Model 2	Model 3	Model 4
Individual level
Basic cognitive abilities	0.37** (0.03)	0.37** (0.03)	0.37** (0.03)	0.37** (0.03)
Basic numerical skills	0.22** (0.03)	0.22** (0.03)	0.22** (0.03)	0.22** (0.03)
Language skills	0.04 (0.02)	0.04 (0.02)	0.04 (0.02)	0.04 (0.02)
Class level
Basic cognitive abilities (mean)	0.39** (0.12)	0.32* (0.12)	0.32** (0.12)	0.36** (0.13)
Basic numerical skills (mean)	0.05 (0.12)	0.08 (0.13)	0.07 (0.13)	0.03 (0.12)
Language skills (mean)	0.02 (0.13)	0.02 (0.12)	0.02 (0.12)	–0.01 (0.12)
Earned certificate to teach mathematics		–0.06 (0.08)	–0.07 (0.08)	–0.10 (0.07)
VERA mathematics school average 2010–2014			–0.01 (0.02)	–0.01 (0.02)
MMS+				0.16* (0.07)
MMS				0.29** (0.08)
Intercept	0.01 (0.03)	0.04 (0.08)	–0.04 (0.08)	–0.19* (0.08)
Explained within-class variance	0.37	0.37	0.37	0.37
Explained between-class variance	0.22	0.29	0.29	0.37

Note. **p < .01; *p < .05;

Standard errors are in parentheses, variables on individual level, the VERA school-level mathematics score, and the output variable are standardized, basic cognitive abilities, basic numerical skills, and language skills on class level are arithmetic means, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group;

Models 2–4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

The association between class composition regarding the basic numerical and language skills and the mathematics achievement at the end of Grades 2 and 3 was not significant, whereas the composition regarding the basic cognitive abilities yielded substantial regression coefficients and had a significant association with the mathematics achievement at the end of Grades 2 and 3. Controlling for those class-level characteristics at school entrance, the mean of the school-level VERA mathematics scores showed no significant association with mathematics achievement at the end of Grades 2 and 3.

When including the dummy variables for the intervention groups in the multilevel models (Model 4 in Tables 5 and 6), the explained variance increased noticeably (Grade 2: ΔR² = 9.9%, Grade 3: ΔR² = 8.4%). At the end of Grade 2, the regression coefficients of the two intervention groups reached the level of significance (MMS+: β = .21 and MMS: β = .30) and showed a considerable difference in average student achievement relative to the comparison group. The intervention groups did not differ significantly, X² (1, N = 2,330) = 1.27, p = 0.26. We can conclude that, after a two-year intervention period, the MMS program had a positive effect on student achievement for the entire group of students relative to the comparison group. At the end of Grade 3—one year after completion of the intervention—the MMS intervention still had a significant effect on student achievement (MMS+: β = .16 and MMS: β = .29). Again, the intervention groups did not differ significantly, X² (1, N = 2,330) = 2.74, p = 0.10.

4. Discussion

4.1. Summary and Implications for Educational Research

Supporting students with difficulties in learning mathematics is challenging for teachers. Since previous research has shown that early mathematics skills are related to later mathematics development (e.g., Duncan et al., 2007, kindergarten through third grade; Morgan et al., 2009, kindergarten through fifth grade), low-achieving students should be supported at the earliest stage possible in their school careers. Although there is evidence that formative assessment approaches can play a successful role in supporting at-risk students, there is a need for ecologically valid intervention studies in which formative assessment techniques are implemented in everyday school practice and are evaluated scientifically (Souvignier & Hasselhorn, 2018). The goal is to provide schools with effective methods for supporting low-achieving students that can be used from the time students enter school and that are easy to implement in the regular classroom without radical changes having to be made in teachers’ individual teaching style. Using an effectiveness trial with a quasi-experimental design and a sound sample size, our study addresses this need and contributes to the existing research by analyzing the effect of the MMS program. This program aims to support students at risk for MD in their first two years in elementary school and is based on a formative assessment approach. It incorporates characteristics considered effective in the literature, for example, the frequent use of assessment tasks, the use of existing assessments that are easy to administer, a clear alignment with the learning goals of the prescribed educational standards and curriculum being taught, assessments that are user-friendly and not time-consuming, and guidance for teachers to interpret assessment data and to select appropriate teaching material (Frohbieter et al., 2011; Herman et al., 2006; Karuza, 2004; Lembke & Stecker, 2007; Nicol & Macfarlane-Dick, 2006; Sharkey & Murnane, 2006). By analyzing a large longitudinal dataset, we were able to show that students at risk for MD in the intervention groups on average reached higher mathematics achievement than students at risk for MD in the comparison group. Therefore, we can conclude that students at risk for MD benefited from the two-year MMS implementation at the end of Grade 2 and that this effect was still maintained one year after the intervention and without providing specific Grade 3 formative assessment material. It turned out that not only the mathematics achievement of students at risk for MD was fostered but also the achievement of the whole group of students. Hence, the formative assessment elements the teachers used in the mathematics classrooms for students at risk for MD were also beneficial for the other students. This outcome is in line with previous research (Black & Wiliam, 1998; Decristian et al., 2015). Yet in the light of previous follow-up studies showing that intervention effects often fade out after the completion of the intervention (Bailey et al., 2018; Li et al., 2020), our results stand out. One reasonable explanation could be that the teachers may have learned effective methods with which they can carry out formative assessments in their ongoing classroom practice and they were thus able to maintain some of the practices even after completion of the intervention.

A second version of the MMS program (the MMS+) was implemented because previous research has indicated that the time available to teachers during the school day is an important factor for a successful implementation of formative assessment programs (Lee et al., 2012; Sharkey & Murnane, 2006). MMS + was enriched by two additional teacher working hours per week and per school. Interestingly, these extra teacher working hours did not have any added positive effect on students at risk for MD in the follow-up test (i.e. at the end of Grade 3). More specifically, the MMS + group’s positive effect on students’ achievement in the whole class at the end of Grade 2 (i.e., for the intervention period) and Grade 3 was similar to the positive effect of the MMS group (without extra teacher working hours). For students at risk for MD, the MMS group showed a significant effect at the end of both grades (i.e., Grades 2 and 3), whereas the effect of the MMS + group reached significance at the end of Grade 2 and was not significant at the end of Grade 3. A possible explanation for these unexpected findings on the effectiveness of the MMS and MMS + groups is that, in the MMS + group, schools used the additional teacher working hours in different ways, for example to do team-teaching (some mathematics lessons were taught by two teachers) or to separate students after the diagnosis (students who needed extra support were taught by another teacher in a different room or even got extra lessons). In such cases the regular mathematics teacher is teaching as usual and identifies students who experience difficulties. Those students then get extra support with the MMS material from the team-teacher or co-teacher. Hence, the availability of additional teacher working hours led to the involvement of a second teacher for parts of the lessons, which possibly resulted in an outsourcing of the formative assessment elements from the regular mathematics teacher to another teacher. Students benefited from this formative assessment treatment similarly to the students in the MMS group without extra teacher working hours. However, with the additional teacher working hours falling away in Grade 3, it is possible that the mathematics teachers of the MMS + group were not able to continue the formative assessment practices because they had not learned how to implement these methods within their regular teaching hours. In the German education system, frequent teacher changes are avoided in elementary schools. Therefore, it is common for one mathematics teacher to stay with one class for several years (we unfortunately have no data on whether the mathematics teacher changed during our study). This would be in line with research findings that teachers need to implement the formative assessment over a long period of time to learn those methods and how to use them in their classroom practice (e.g., Herman et al., 2006; Karuza, 2004; Sharkey & Murnane, 2006; Vendlinski et al., 2009) and that interventions effects often fade out after the intervention is completed (Bailey et al., 2018; Li et al., 2020). The result that the effect of the MMS+ intervention vanished one year after the intervention was completed highlights the importance of ongoing progress monitoring, especially for supporting students at risk for MD (e.g., Fuchs et al., 2008).

In summary, our findings on the positive effect of the MMS implementation are in line with previous formative assessment research, which has shown a positive effect of formative assessment on low-achieving students as well as on average-achieving students (e.g. Black & Wiliam, 1998; Decristan et al., 2015; Phelan et al., 2011). With the results obtained from our effectiveness trial, our study contributes to the existing research by answering the call to test the implementation of formative assessments in everyday school practice and evaluating this implementation scientifically. Our study suggests that the positive effects of formative assessment remain after the intervention if the mathematics teachers themselves learned how to implement the formative assessments during a longer period (here: the 2 years of the intervention program). In order for teachers to achieve this, it is important that the formative assessment program be easy to implement and not require major changes in the individual teaching style, the teaching content, or the methods.

4.2. Limitations

There are several limitations to this study. This study was designed as an effectiveness trial and therefore did not analyze whether the separate components of the program were in fact effective under laboratory conditions. We only analyzed the program as a whole in everyday classroom practice and we do not know whether, and if so how much, the teachers departed from the MMS framework when teaching. Therefore, we cannot draw conclusions about which aspects of the program are effective or which are irrelevant. As noted in the Introduction, all teachers are required by the Ministry of Education to provide specific learning opportunities to low-achieving students. Because of the study design, we do not know which procedures the teachers in the comparison group used. We only can conclude that the intervention groups using MMS on average reached a higher mathematics achievement than the comparison groups that used a variety of procedures.

Furthermore, we assume that, through the MMS program, the quality of teaching increases (in addition to the formative assessment elements implemented) because the MMS program provides teachers with opportunities to improve their professional knowledge. We therefore would assume that the effect of the program on student achievement—at least to some degree—is mediated by the quality of teaching. Due to time and practical restrictions, it was not possible to assess the quality of teaching in this study. Therefore, we cannot make any statements about whether teachers’ professional knowledge possibly moderates the MMS effects. But it could be worthwhile for future research to analyze how teachers implement the program and the extent to which it enhances the quality of teaching.

Another limitation of this study is that we were not able to model the school level in the multilevel analyses because there were not enough groups per school. We also did not have any additional information about the achievement of previous cohorts, which would have enabled us to control for school effects. Furthermore, the schools in the MMS intervention groups where chosen from a pool of schools that volunteered to implement MMS, and the possibility of selection bias can thus not be precluded (Shadish et al., 2002). It is possible that mainly schools with certain characteristics or schools that were more eager to improve applied for the MMS program. To account for differences between schools, we included the mean of the school-level achievement mathematics scores (VERA) from 2010 to 2014 in the regression models. We argue that schools which are more eager to improve or have a more advantaged catchment area are more likely to have higher school averages on the VERA test. We did not find significant effects of the VERA scores (Tables 3–6). This means that the schools’ average performance on the national VERA test over the last five years before the intervention has no added explanatory effect of the arithmetic achievement. Because the material was offered for free and about one quarter of all elementary schools in the state volunteered for the intervention, we presume that the intervention was appealing to all kinds of schools. Furthermore, the comparison group was chosen in such a way that at least the socioeconomic and cultural background was distributed evenly across the intervention and comparison groups. Our study should therefore be considered as providing first evidence of the effectiveness of the MMS program, which can be further strengthened by conducting replications and varying implementation conditions. Especially the question of long-term effects, that is, whether the MMS program still shows effects at the end of Grade 4, which is the end of elementary school in Germany, is relevant for future research. It would be most interesting to see whether MMS students at risk for MD reach at least the minimum standards at the end of Grade 4 and whether more MMS students transfer to the academic school track than their peers from comparison classes.

One further limitation of our study is the identification of students at risk based on the 25th percentile of the sample regarding the basic mathematical skills. This criterion is highly confounded with the sample used and may not accurately identify those students most at risk. The identification of students at risk for MD at one point in time can be the result of natural fluctuations in the growth of mathematical knowledge or can be due to measurement error. However, as studies have shown a high persistence of MD and a strong relation between early mathematics achievement and later mathematics learning (Duncan et al., 2007; Jordan et al., 2003; Morgan et al., 2009), we do not expect there to be great variation in the identification of students at risk for MD. Another vulnerability of this classification stems from the specific situation in Germany. Kindergarten students in Germany do not have any structural exposure to mathematical learning opportunities. Therefore, one could argue that, on the one hand, differences in the basic mathematical skills are based on developmental differences. On the other hand, children in Germany do encounter mathematical learning opportunities in kindergarten (Krajewski and Schneider, 2009). Furthermore, Krajewski and Schneider (2006) were able to show that, while controlling for basic cognitive abilities and basic literacy skills, the basic mathematical skills of students in kindergartens in Germany still correlated rather highly with the mathematics achievement at the end of Grade 4. Therefore, we assume that also in Germany, students with low basic mathematical skills at the beginning of Grade 1 are likely to have disadvantages regarding their mathematical development in comparison to their peers, based on more than just developmental differences.

4.3. Educational Implications

The evaluation showed that the implementation of an early formative assessment program had a positive effect on the achievement of students at risk for MD as well as on the entire class. The results of this study emphasize the potential of the formative assessment method. Furthermore, the MMS program answers the call to provide schools with methods, material, and professional teacher development to support students at risk for MD already at an early stage in their school career. The program could be an example of how to successfully incorporate the potential effective factors for implementing formative assessments discussed in previous studies. Moreover, assessment programs such as MMS could be an instrument to enhance educational effectiveness without the need for additional teacher working hours.

The MMS program takes into account the relevant factors required to successfully implement the formative assessment method in everyday school practice. The program is relatively easy to implement because there are no changes in the curriculum, no new content to teach, no new methods to use, and no need for a radical change in teachers’ individual teaching style. The comprehensive program incorporates professional development, assessment tasks, and specific feedback in the form of a targeted intervention for the milestones of arithmetic development. Additional teacher working hours are not necessary for the successful implementation of the program. This result is of major interest from an economic perspective, because the continuous supply of additional teacher working hours would be the most expensive element of the program.

4.4. Conclusion

This study contributes to research on the effectiveness of early formative assessment on mathematics achievement and answers the call to provide schools with methods and material to close the achievement gap as early as possible. The longitudinal data indicate the stability of MMS effects from the end of Grade 2 up to the end of Grade 3. The findings imply that formative assessment, as implemented in the MMS program, can foster the development of arithmetic achievement for students at risk for MD as well as for the entire class, without changing the curriculum, and without requiring teachers to learn new methods or teach new content.

Notes

1 As mentioned before the developmental trajectory of children in this age varies considerably. Thus, it must be noted that the identification of students at risk for MD at one point in time by a relative performance criterion has its weaknesses. We discuss this problem in the limitations section.

2 In Germany the national educational standards are a framework for the whole country. Each federal state has a state-specific curriculum that is aligned to these standards and all schools in a federal state have to follow the state-specific curriculum. In this manuscript we use the term “curriculum” to refer to this level of obligatory learning content and objectives, which is the most concrete and binding.

3 Example of learning goals from Milestone 2, Part-Whole Relationships: Students have developed a basic idea of the decomposition of numbers. They are able to decompose numbers with concrete material, they are able to decompose numbers mentally by imagining concrete material, they achieve automatization of decomposition.

4 Possible observations for the interview task regarding the topic Part-Whole Relationships: (1) Decompositions based on material can be made; (2) Decompositions can be made (without material); (3) All decompositions of a given number can be found; (4) Decompositions can be identified without counting; (5) Decompositions can be made with high speed.

5 For example, for observation (1) from Footnote 4, learning activities regarding decompositions based on material are suggested, for observation (4), learning tasks regarding the automatization of the decomposition are suggested.

6 In these schools, the teacher participating in the MMS intervention was relieved from other responsibilities for these two hours.

7 The 100 schools account for a quarter of all elementary schools in this state.

8 Because the Level 2 cluster sizes in this subsample were too small, we used linear regression models with the “type = complex” procedure instead of multilevel models to correct for standard errors.

9 As suggested in the review process, we conducted alternative regression analyses by including the predictors in a different order (first the intervention groups, then the individual school entry measures, the textbooks and teacher variable, and finally the school-level achievement scores). The findings are presented in the Appendix.

10 Based on a suggestion in the review process, we also conducted alternative multi-level analyses for the whole sample. Again, the findings are presented in the Appendix.

Appendix

Tables 7 to 10 show an alternative structure of the regression models. In the first model, only the intervention groups are included. The second model includes the basic cognitive abilities, basic numerical skills and language skills. The third model additionally includes the earned certificate to teach mathematics and the textbooks. Finally, in Model 4 the VERA school-level mathematics score (average from 2010 to 2014) is included in the model.

Table 7. Alternative regression modeling for individual and classroom covariates and intervention groups on arithmetic achievement for students at risk for MD at the end of Grade 2.

	Model 1	Model 2	Model 3	Model 4
MMS+	0.08 (0.18)	0.21 (0.12)	0.24# (0.12)	0.27* (0.12)
MMS	0.18 (0.23)	0.27 (0.15)	0.36* (0.14)	0.35* (0.14)
Basic cognitive abilities		0.42** (0.05)	0.39** (0.05)	0.39** (0.05)
Basic numerical skills		0.21** (0.05)	0.21** (0.04)	0.21** (0.04)
Language skills		0.09* (0.04)	0.10* (0.04)	0.10* (0.04)
Earned certificate to teach mathematics			–0.18 (0.10)	–0.18 (0.10)
VERA mathematics school average 2010–2014				0.06 (0.04)
Intercept	–0.08 (0.16)	–0.16 (0.05)	–0.47** (0.16)	–0.38** (0.14)
R²	0.00	0.34	0.36	0.37

Note. **p < .01; *p < .05; #p = .05;

Standard errors are in parentheses, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group, the dependent variable and noncategorical independent variables are standardized;

Model 3 and 4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display covariates.

The first model in Table 7 shows the effect of the intervention on the arithmetic achievement of students at risk at the end of Grade 2 without controlling for covariates. We see no significant effect of the intervention groups. This could be due to differences in covariates. For example, the control group has a higher mean value on the basic cognitive abilities than the intervention groups. Basic cognitive abilities are known to have an effect on the arithmetic achievement. Therefore, in the second model we included the basic cognitive abilities, basic numerical skills and language skills at the beginning of first grade. We see that those covariates have a significant effect on the arithmetic achievement and explain 34% percent of the variance between students. The intervention groups have no significant effect when holding those covariates constant. In the third model we include the earned certificate to teach mathematics (studied mathematics or not) and the textbooks used by the groups. The textbooks have a significant effect on arithmetic achievement. When holding the textbooks constant, we see a significant effect of the intervention group MMS on arithmetic achievement at the end of Grade 2. When inserting the average of the school-level mathematics scores from the years before the intervention, both intervention groups MMS + and MMS show a significant effect in arithmetic achievement. We can conclude that students at risk in the intervention groups, who have comparable scores on the covariates to those in the comparison groups, reach higher scores on the arithmetic achievement test at the end of Grade 2. Similar results can be seen for students at risk at the end of Grade 3 (Table 8). Here, the MMS + group does not show a significant effect on students’ arithmetic achievement after controlling for the earned certificate to teach mathematics, the textbooks used, and the school-level mathematics score (Models 3 and 4).

Table 8. Alternative regression modeling for individual and classroom covariates and intervention groups on arithmetic achievement for students at risk for MD at the end of Grade 3.

	Model 1	Model 2	Model 3	Model 4
MMS+	–0.06 (0.18)	0.06 (0.14)	0.12 (0.13)	0.12 (0.13)
MMS	0.12 (0.21)	0.21 (0.15)	0.29* (0.14)	0.29* (0.14)
Basic cognitive abilities		0.41** (0.05)	0.38** (0.05)	0.38** (0.05)
Basic numerical skills		0.17* (0.05)	0.17** (0.05)	0.17** (0.05)
Language skills		0.06 (0.05)	0.06 (0.05)	0.06 (0.05)
Earned certificate to teach mathematics			0.19 (0.12)	–0.19 (0.12)
VERA mathematics school average 2010–2014				0.02 (0.03)
Intercept	0.01 (0.15)	–0.07 (0.11)	–0.27 (0.17)	–0.27 (0.17)
R²	0.01	0.28	0.30	0.30

Note. **p < .01; *p < .05; #p = .05;

Standard errors are in parentheses, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group, the dependent variable and non-categorical independent variables are standardized;

Model 3 and 4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

In Table 9 the results for the whole group at the end of Grade 2 can be seen. Including only the intervention groups, no effect on the arithmetic achievement at the end of Grade 2 can be seen. This could be due to differences in covariates. For example, the control group has a higher mean value on the basic cognitive abilities than the intervention groups. Basic cognitive abilities are known to have an effect on the arithmetic achievement. Therefore, in the second model we included the basic cognitive abilities, basic numerical skills and language skills at the beginning of first grade. We see that those covariates have a significant effect on the arithmetic achievement and explain 33% percent of the variance between classes. The intervention does have a significant effect when holding those covariates constant. In the third model we additionally include the Earned certificate to teach mathematics (studied mathematics or not) and the textbooks used by the groups. The textbooks have a significant effect on arithmetic achievement. When holding the textbooks constant, we see a higher value of the regression coefficient for the MMS intervention group. Furthermore, the explained variance between classes increases (48%). In the fourth model, we included the average school-level mathematics achievement from 2010 to 2014 to account for school achievement characteristics. This covariate hardly influences the effect of the intervention. We can conclude that classes in the intervention groups, who have comparable scores on the covariates to those in the comparison groups, reach higher scores on the arithmetic achievement test at the end of Grade 2.

Table 9. Alternative multilevel regression modeling for individual and classroom covariates and intervention groups on students’ arithmetic achievement at the end of Grade 2.

	Model 1	Model 2	Model 3	Model 4
Individual level
Basic cognitive abilities		0.38** (0.03)	0.38** (0.03)	0.38** (0.03)
Basic numerical skills		0.22** (0.03)	0.22** (0.03)	0.22** (0.03)
Language skills		0.06** (0.02)	0.06** (0.02)	0.06** (0.02)
Class level
MMS+	0.18 (0.10)	0.18** (0.07)	0.18** (0.07)	0.21** (0.07)
MMS	0.23 (0.12)	0.23** (0.08)	0.31** (0.07)	0.30** (0.07)
Cognitive abilities (mean)		0.36** (0.11)	0.33** (0.11)	0.31** (0.11)
Basic numerical skills (mean)		0.04 (0.09)	0.05 (0.09)	0.03 (0.09)
Language skills (mean)		–0.04 (0.11)	–0.03 (0.10)	–0.06 (0.11)
Earned certificate to teach mathematics			–0.10 (0.06)	–0.09 (0.06)
VERA mathematics school average 2010–2014				0.05 (0.03)
Intercept	–0.16 (0.08)	–0.13* (0.05)	–0.29** (0.07)	–0.30** (0.07)
Explained within-class variance		0.40	0.40	0.40
Explained between-class variance	0.04	0.33	0.48	0.49

Note. **p < .01; *p < .05;

Standard errors are in parentheses, variables on individual level, the VERA school-level mathematics score, and the output variable are standardized, basic cognitive abilities, basic numerical skills, and language skills on class level are arithmetic means, earned certificate to teach mathematics is coded dichotomously, MMS+ and MMS are dummy variables with reference category comparison group;.

Models 3 and 4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.

At the end of Grade 3 (Table 10) we see a significant effect of the intervention group MMS in the first model. When including the cognitive abilities, basic numerical skills and language skills at the beginning of the first grade on individual and on the class level, we see a significant effect of the first two on individual level and only of cognitive abilities on class level. In the third model the covariates earned certificate to teach mathematics and textbook are included. The textbooks have a significant effect on arithmetic achievement at the end of Grade 3. When holding those covariates constant, the regression coefficient of the MMS+ intervention group becomes significant. In the fourth model, we included the average school-level mathematics achievement from 2010 to 2014, which did not influence the effect of the intervention. We can conclude that classes in the intervention groups, who have comparable scores on the covariates to those in the comparison groups, reach higher scores on the arithmetic achievement test at the end of Grade 3.

Table 10. Alternative multilevel regression modeling for individual and classroom covariates and intervention groups on students’ arithmetic achievement at the end of Grade 3.

	Model 1	Model 2	Model 3	Model 4
Individual level
Cognitive abilities		0.37** (0.03)	0.37** (0.03)	0.37** (0.03)
Basic numerical skills		0.22** (0.03)	0.22** (0.03)	0.22** (0.03)
Language skills		0.04 (0.04)	0.04 (0.02)	0.04 (0.02)
Class level
MMS+	0.49 (0.10)	0.12 (0.07)	0.16* (0.07)	0.16* (0.07)
MMS	0.25* (0.12)	0.24** (0.08)	0.29** (0.08)	0.29** (0.08)
Cognitive abilities (mean)		0.43** (0.12)	0.36* (0.13)	0.36** (0.13)
Basic numerical skills (mean)		0.01 (0.12)	0.04 (0.12)	0.03 (0.12)
Language skills (mean)		0.01 (0.13)	–0.01 (0.12)	–0.01 (0.12)
Earned certificate to teach mathematics			–0.10 (0.07)	–0. 10 (0.07)
VERA mathematics school average 2010–2014				–0.01 (0.02)
Intercept	–0.13* (0.02)	–0.11 (0.05)	–0.19* (0.08)	–0.19* (0.08)
Explained within-class variance		0.37	0.37	0.37
Explained between-class variance	0.04	0.29	0.37	0.37

Note. **p < .01; *p < .05;

Standard errors are in parentheses, variables on individual level, the VERA school-level mathematics score, and the output variable are standardized, cognitive abilities, basic numerical skills, and language skills on class level are arithmetic means, earned certificate to teach mathematics is coded dichotomously, MMS + and MMS are dummy variables with reference category comparison group;

Model 3 and 4 also include the textbooks used as dummy-coded variables, to avoid putting too much information into the tables we do not display these covariates.