Factors affecting intervention fidelity of differentiated instruction in kindergarten

Abstract

This paper reports on the findings in the first phase of a design-based research project as part of a large-scale intervention study in Dutch kindergartens. The project aims at enhancing differentiated instruction and evaluating its effects on children’s development, in particular high-ability children. This study investigates relevant intervention fidelity factors based on [Fullan, M. (2007). The New Meaning of Educational Change. New York: Teachers College Press]. A one-year intervention in 18 K-6 schools was conducted to implement the screening of children’s entry characteristics, differentiation of (preparatory) mathematics and language curricula, and a policy for the differentiation and teaching high-ability children. The intervention fidelity and implementation process were scored for each school using data from observations, field notes and log books. Self-report questionnaires measured participants’ perceptions of the intervention (n = 35 teachers, 18 principals). Quantitative results showed that intervention fidelity differed between schools. Qualitative analyses of perceptions and cross-case analyses of three kindergartens showed that a strong need, pressure from parents, an involved principal, and teacher time and motivation contributed to successful implementation. Implementation barriers were the innovation’s complexity, teacher beliefs, an absent principal and low teacher motivation (which was partly due to communication problems). Implications for interventions in general and differentiated instruction for high-ability children in particular are discussed.

Keywords:

• Differentiation
• instruction
• intervention fidelity
• implementation
• kindergarten
• teacher

Introduction

Kindergarten, and its classroom quality in particular, is assumed to positively affect children’s future academic and social behaviours (Cabell et al. 2013; Weiland et al. 2013). Recent research indicates, however, that instructional quality may be non-optimal (e.g. instructional interactions are insufficiently adapted to children’s needs; Cabell et al. 2013). Many initiatives for improving teacher practice, usually during in-service training, are being undertaken to improve the instructional quality (Pianta et al. 2014). However, it has been shown that in-service training often does not achieve a change in teachers’ practices (Bitan-Friedlander, Dreyfus, and Milgrom 2004), thus the quality remains sub-optimal.

Changing teachers’ practices requires urgent attention since, of all school factors influencing learner outcomes, the teacher is possibly the most important (Hattie 2009). According to Fullan (2007), the effects of interventions are to a large extent determined by the degree to which schools and teachers both accept the proposed changes and implement them. In this respect, intervention fidelity refers to the implementation of an intervention as intended or designed (O’Donnell 2008). It is important to elucidate the degree of intervention fidelity and which factors affect it (Bywater and Sharples 2012; McKenna, Flower, and Ciullo 2014; O’Donnell 2008; Swanson et al. 2013); only then can the relative strength of an intervention for outcomes be assessed and the information used – if necessary – to create a more successful intervention and, finally, to upscale the intervention (Darrow 2013; Swanson et al. 2013). Further, as evidence-based interventions that ‘can be implemented with fidelity’ (Swanson et al. 2013, 3) gain attention (Onderwijsraad 2006), it is important that intervention fidelity and factors affecting it are investigated.

One of the areas in which evidence-informed interventions are top priority is the teaching of high-ability or gifted children (e.g. Callahan et al. 2015; Koshy, Pinheiro-Torres, and Portman-Smith 2012; Robinson, Shore, and Enersen 2007; Segers and Hoogeveen 2012). In mainstream education, teaching is often not adapted to the specific learning processes of these children, which involve larger learning steps, longer periods of concentration, less repetition, and the willingness and ability to work rather independently (Colangelo, Assouline, and Gross 2004; Mooij 1992). Two problems teachers face are that they often do not have a clear picture of the (1) levels at which the children function, especially high-ability children (cf. Al Otaiba et al. 2011; Doolaard and Harms 2013; Mooij 2000) and (2) cognitive levels required for carrying out specific curricular learning activities (cf. Mooij et al. 2014; Tomlinson et al. 2003). The resulting lack of fit between children’s needs and learning activities may have negative consequences for high-ability children’s achievement and behaviour (cf. Colangelo, Assouline, and Gross 2004; Gross 1999; Mulder, Roeleveld, and Vierke 2007).

With the above in mind, an intervention was designed to improve the match between levels and activities, particularly for high-ability children. The ultimate goal was to enhance the learning and development of all children, including those of high ability. Although matching appropriate learning activities to individual children is relevant for all children, this method seems to be particularly suitable for high-ability children as these children, in the Netherlands, usually carry out learning activities that are (far) below their cognitive levels (Dutch Inspectorate of Education 2015; Mooij 2013). Recent Dutch research shows for example that the offering of appropriate learning activities for these pupils is strongly teacher-dependent, the learning activities for high-ability pupils are not integrated in a curriculum and are only offered at an ad hoc basis (Dutch Inspectorate of Education 2015). Implementing the intervention presented in this study includes changes in teachers’ behaviour, such as screening children’s entry characteristics, and offering children different instruction and/or materials that match their (entry) levels.

This study focuses on the implementation process of the intervention in Dutch kindergarten classrooms and hopes to provide insight into crucial success factors and barriers for its implementation. As said, the teacher is a very important factor in influencing learner outcomes, and therefore teacher’s attitudes towards the intervention could be a crucial factor for its implementation. Hargreaves (2005) states that teachers respond differently towards educational change based on age, career stage and colleagues. Sustainable educational change needs a mixture of teacher age groups, mentoring (across the generations) and memory (conscious collective learning from wisdom and experience). Furthermore, the renewal of educational programmes should be a collective action between teachers and managers (Peck et al. 2009). Changing educational practices is not easy for teachers, as they exhibit a variety of concerns while dealing with novel educational practices, negotiating their professional identities and determining their response to the change (Vähäsantanen 2015). This individual professional development however, can’t take place without school development: it is not enough to change individuals; there must also be changes in systems (Fullan 2006). Understanding the dynamics at work when changing education is crucial to help individuals and organisations to achieve change.

In the case of our intervention, the change goes beyond the individual teachers, but affects the whole school and perhaps community. Therefore, we need a framework that looks at the change process from various perspectives, and Fullan’s (2007) framework of nine implementation factors for educational change matches those requirements. The research questions guiding this study are: (1) What is the intervention fidelity of the intervention? and (2) Which factors (i.e. characteristics of the intervention, local and external factors) influence the intervention fidelity? Following a design-based research (DBR) approach (Anderson and Shattuck 2012), lessons learned for future interventions for high-ability students are uncovered and discussed. First, the question of how student differences can be anticipated in teaching and what factors may be of influence when teachers try to do this are discussed. Then, the intervention and the current study is described in detail. For more information about the theoretical assumptions underlying the intervention, see Mooij et al. (2014). Hereafter, the research method describes the participants, procedure, measures including the intervention fidelity, participants’ perceptions of the intervention, and qualitative data of the implementation process, data collection and analyses. The results section starts with an overview of the intervention fidelity, then examines participants’ perceptions of the intervention, and ends with an extensive cross-case analysis of Fullan’s implementation factors in three schools. Conclusions, major study limitations and implications are discussed.

Anticipating student differences

Teachers can improve the match between learner levels and learning activities by differentiating their instruction according to the relevant characteristics of children in the classroom. This may involve differentiation in content (i.e. curricula), process, product, and/or learning environments, based, for example, on children’s ability levels and interests (Tomlinson 2005; Tomlinson et al. 2003). By doing so, teachers ‘maximise the potential of all learners by proactively designing learning experiences in response to individual needs’ (Santangelo and Tomlinson 2012, 310). For successful differentiation practices, several organisational aspects are required, as previously discussed in Mooij, Dijkstra, Walraven, and Kirschner (2014).

First, learning must start with some type of diagnostics with respect to the level of competence of the individual child within a specific subject area. Regular monitoring of levels and progress is necessary so that teachers may continually modify free play and instruction and vary grouping patterns to meet changing characteristics and needs (Deunk et al. 2015; Purcell et al. 2002).

Second, to make appropriate decisions about each child’s instruction, kindergarten teachers need to become comfortable with and gain proficiency in the curriculum they are teaching. A solid understanding of the learning goals and developmental progression in each subject or skill is required, as are methods for achieving and using a differentially implemented curriculum including learning materials (Deunk et al. 2015; Mooij et al. 2014; Tomlinson et al. 2003). Learning processes can be evaluated and managed in order to provide for further learning steps.

Third, beliefs and practices related to systemic issues in kindergarten and elementary school need to fit differentiated instruction in a flexible system, as several systemic factors (e.g. school climate, resources, etc.) and teacher attitudes can impede the use of differentiated instruction (Kingore 2004; Maier, Greenfield, and Bulotsky-Shearer 2013; Roy, Guay, and Valois 2013; Tomlinson et al. 2003).

Implementation factors for differentiated instruction

Adequate implementation of differentiated instruction in kindergarten is not self-evident (Darrow 2013; Fullan 2007; O’Donnell 2008). The problem may be that the typical situation of teachers is one of perseverance with many factors, tending to keep things they always have been. There is often little room nor inclination for change, especially when it is imposed from the outside. The result may be that teachers adjust to proposed changes by doing as little as possible. According to Fullan (2007), characteristics of the intervention, local characteristics and external factors collectively influence an intervention’s implementation.

Characteristics of an intervention include the need for change, clarity of goals, complexity of the change and quality and practicality of the programme. The need for an intervention to improve the teacher’s ability to differentiate instruction is clear. A number of researchers have shown that the skills needed for differentiated instruction are not in the repertoire of all teachers (Deunk and Doolaard 2013; Doolaard and Harms 2013; Van de Grift 2010). Though assessment data are widely present in schools, teachers often lack knowledge of how to acquire and use that data for monitoring progress, and may feel uncertain about how to differentiate in the curriculum (i.e. what to teach, such as more advanced content for high-ability children) (Doolaard and Harms 2013; Santangelo and Tomlinson 2012). Thus, in practice, there is a need to improve the differentiated instruction skills of kindergarten teachers, which may positively influence the intervention fidelity.

Local characteristics include the school district, community, school principal and teachers. According to Fullan and Stiegelbauer (1991), ‘the local school system represents one major set of situational constraints or opportunities for effective change’ (73). A risk to implementation is that schools maintain a ‘culture of caution’ (Le Fevre 2014). For example, when children are allowed more initiative in their own learning activities, as is the case in differentiated instruction, teachers may fear a loss of control. Their perceived risk in the face of uncertainty may result in their demonstrating a conservative impulse and being motivated by a need to protect their current practice. Also, strong beliefs can persist, both at the individual teacher level and collectively at the school level, and can function as obstacles to improving teacher practice (Le Fevre 2014). For example, Dutch kindergarten teachers often show a resistance towards academic activities, because they believe that kindergarten is a play space and not a learning space (Oberon 2013). Also, the belief exists that high-ability children do not need to be challenged in class (Doolaard and Harms 2013; Tomlinson et al. 2003). Thus, local factors such as school culture and teacher perceptions may hamper the implementation of differentiated instruction.

Finally, external factors include the government and other agencies. These external parties often present conflicting demands on education (Bergen and Van Veen 2004; Luttenberg, van Veen, and Imants 2013). For example, adjusting the needs of all children in the classroom, as is the case in differentiated instruction, conflicts with a strong focus on covering prescribed curricula (Engel, Claessens, and Finch 2013).When confronted with multiple, conflicting agendas, teachers feel vulnerable and uncertain (Le Fevre 2014), which may negatively affect the implementation.

Method

This article reports on DBR findings in a larger study to implement differentiated instruction in Dutch kindergartens and to evaluate its effects on the learning and development of kindergarten children, in particular high-ability children (Mooij et al. 2014). The current study focuses on how Fullan’s (2007) factors influence intervention fidelity. Using a DBR approach, this information can be taken into account to enhance the quality and implementation process for subsequent years. This research used a mixed-methods approach, as recommended in DBR (Anderson and Shattuck 2012), employing qualitative and quantitative methods in the same study. The research included a cross-case analysis that aimed to identify relevant factors for intervention fidelity.

Intervention

DBR is situated in the educational context and happens in collaboration between researchers and practitioners (Anderson and Shattuck 2012; Barab and Squire 2004). A DBR approach was used to design and evaluate the intervention. The intervention was based on an extensive literature review and earlier pilots (Mooij 2007, 2008). It aims at improving teachers’ differentiation practices, with particular focus on how teachers can support high-ability children. In this intervention, schools use or develop three ‘critical components’ (see O’Donnell 2008), as discussed in Dijkstra (2015).

The first component is the screening of entry characteristics of all incoming 4-year-old children (Mooij 2000). This is based on the assumption that teachers need knowledge of individual children’s cognitive and social needs and levels to appropriately differentiate in the curriculum, and that a screening would help teachers achieve this. The screening took the form of a 29-item questionnaire for parents and teachers to estimate the child’s developmental levels in seven areas: social interaction/communication (Cronbach’s alpha .92 for parents/.88 for teachers), general cognition (.65/.80), language proficiency (.80/.87), preliminary arithmetic (.76/.90), sensorimotor level (.65/.75), emotional-expressive level (.68/.78) and expected educational behaviour/motivation (.84/.91) of the child. The response options estimate whether a child is less (1), slightly less (2), about the same (3), slightly more (4) or more (5) developed compared with his or her peers. Except for sensorimotor level, Mooij (2000) confirmed the predictive validity of the screening concepts by relating the concepts with criterion instruments while checking on co-variables at both pupil and class levels. With this screening, parents and teachers can cooperate to inform each other about the levels and needs of the children when they begin kindergarten and help teachers make informed decisions about appropriate learning activities.

Second, teachers were asked to develop a framework of the learning materials available in their classrooms that matched the curriculum learning goals. This component was based on the assumption that teachers need knowledge of the structure and levels of the curriculum in order to match children’s levels and needs to appropriate curricular learning activities (see Mooij et al. 2014). This framework for (preparatory) mathematics and language separately consisted of a table in which centrally defined learning goals (Stichting Leerplan Ontwikkeling 2010a, 2010b) were matched with the levels, assessment methods and learning materials corresponding to these goals. Teachers were asked to arrange the materials in the classroom cabinets according to the levels in the framework. Generally, Dutch kindergartens keep most playing and learning materials on shelves distributed throughout the classroom, along the walls or in the corridors. Children may access these materials on their own initiative, at specific times or because of an assignment by the teacher who may want to promote some pedagogical or instructional criterion. The typical ordering of these materials is not according to the specific domain or ability level. In the intervention, the materials should be ordered with respect to domain and difficulty level. Teachers can use different colours and icons to indicate the ordering of both content and levels. Such prepared playing-learning situations enable small groups of children or individual children to access and use materials and instructions independent of their age. Of course, children will always need the teacher, but for different children this support is needed for different types of activities and to varying degrees.

The third component was the development of a school policy protocol with information on how assessment and differentiated instruction takes place, with a special focus on high-ability children. Currently, the Dutch Inspectorate of Education (2015) stimulates schools to develop teaching for high-ability pupils that is less teacher-dependent, and one of the features they stimulate is the development of clear policy how to support high-ability pupils similar to one of the components of Excel Kwadraat. This component was based on the assumption that a structured, preventive approach provides guidance within the team and for parents. In this way, it was clear for both teachers and parents how the school handles assessing children’s levels and what the school’s differentiation approach was.

Participants

Eighteen elementary schools, in which kindergarten is an integrated part, were recruited from the school network of the researchers and participated voluntary in the intervention. Participating schools received financial compensation of about €3.000 for purchasing learning materials and for allotting kindergarten teachers time for intervention activities. Requirements for receiving this compensation were the participation of all principals (n = 18) and kindergarten teachers (n = 35), carrying out intervention activities, and responding to the questionnaires. With one exception, all schools had a combined first and second year of kindergarten.

Procedure

A one-year pilot intervention was carried out, as previously described. Support sessions were organised to help teachers implement the components in each kindergarten classroom (see Mooij et al. 2014). Schools participated in three to five meetings per year. All school principals received a manual with information on the intervention and background information about necessary curriculum changes and how to realise them. The principals were also asked to distribute the information to their staff and teachers and to discuss the intervention regularly with the team. The manual or other documents and the implementation characteristics and changes were discussed in regional sessions. The researchers clarified the meaning of the successive components and promoted the correct development or implementation of the components in each school. Kindergarten teachers within each school were expected to collaborate and develop or implement the components stepwise in their own practice.

Measures

O’Donnell (2008) recommends measuring the fidelity of critical components and processes present in the intervention and including the extent to which participants are engaged by and involved in the activities and content of the programme. Therefore, the variables investigated in this study were intervention fidelity, participants’ perceptions of the intervention and the implementation process in the schools.

Intervention fidelity

The extent to which the critical components of the intervention were implemented in each school was scored using a fidelity rubric checklist (see Table 1), as discussed in Dijkstra (2015). Each component was scored separately, that is: (1) screening entry characteristics, (2) using a differentiation framework for the math (a) and language (b) curriculum and the corresponding organisation of materials in the cabinets and (3) using a policy protocol for differentiation practices and high-ability children in the school. The first three authors developed the rubrics collaboratively based on 10 randomly chosen schools, and then applied them to all schools. In both rounds, two researchers separately studied the written documents and oral presentations per school, and rated the components. Hereafter, they discussed their ratings until agreement was reached.

Table 1. Intervention fidelity rubric checklist.

Component and scoring categories	Rubric explanation	Information source
1. Screening of school-entry characteristics
Screening not implemented	<10% of new four-year-old pupils are screened	Digitally tracked in NetQ
Screening partially implemented	10–90% of new four-year old pupils are screened
Screening fully implemented	>90% of new four-year-old pupils are screened
2a. Differentiation mathematics curriculum
Framework not developed	No framework of ordered learning materials per (sub)domain and difficulty level	• Face-to-face presentations of cabinets with learning materials in the schools • Frameworks (sent by e-mail) • Photographs of cabinets with explanations (sent by e-mail)
Started structuring in framework and materials	A first version of the framework was developed, but the school lacked the ordering of learning materials in cabinets according to the framework
Advanced structuring and organising framework	The framework was developed and learning materials were ordered, but the school needed to fine-tune the correspondence between framework and materials in the cabinets
Framework and organisation of materials developed	The framework corresponds with the materials in the cabinets
2b. Differentiation language curriculum
Framework not developed	No framework of ordered learning materials per (sub)domain and difficulty level	• Face-to-face presentations of cabinets with learning materials in the schools • Frameworks (sent by e-mail) • Photographs of cabinets with explanations (sent by email)
Started structuring in framework and materials	A first version of the framework was developed, but the school lacked the ordering of learning materials in cabinets according to the framework
Advanced structuring and organising framework	The framework was developed and learning materials were ordered, but the school needed to fine-tune the correspondence between framework and materials in the cabinets
Framework and organisation of materials developed	The framework corresponds with the materials in the cabinets
3.Differentiation policy protocol
Policy not developed	No protocol	• Protocols sent by email or given during a meeting
Policy partially developed	Protocol with limited information on pupil assessment or subsequent learning activities
Policy fully developed	Complete protocol with information on pupil assessment and subsequent learning activities

For the screening, the scoring categories were ‘fully implemented’, ‘partially implemented’, or ‘not implemented’. The differentiation frameworks for math and language and the corresponding organisation of materials were scored using categories ‘framework and organisation of materials developed’, ‘advanced structuring and organising framework’, ‘started structuring in framework and materials’ and ‘framework not developed’. Categories for the policy protocol were ‘policy fully developed’, ‘policy partially developed’ and ‘policy not developed’.

Perceptions of the intervention

A seven-item questionnaire with two multiple choice items and five open-ended questions was developed to measure participants’ (i.e. teachers, principals) expectations about intervention activities (three components) and perceptions of outcomes/effects. The questions (see Table 3) related to the expected work needed for the intervention and expected benefits and effects. Further, participants were asked about the extent to which the expectations were fulfilled and if they had any suggestions for improving the support given by the researchers. By using mostly open-ended questions, participants had the opportunity to formulate and reflect on their expectations and outcomes as free as possible. The two multiple choice questions were used to structure the participants’ thinking and subsequent answers (see Table 3).

Implementation process

The implementation process in each school was tracked via the researchers’ field notes and logs. During the support sessions, the main discussion points about the intervention and a summary of agreements were recorded in field notes by the first author. In addition, communication between schools and researchers was recorded in a log.

Data collection

The perceptions questionnaire was administered via a web portal using NetQ®. Response rates were 100% for both teachers and principals. Intervention fidelity data were collected in several ways. The extent to which the screening of entry characteristics was implemented was tracked for each school in NetQ®. All teachers presented their cabinets with materials to the researchers during visits to their classrooms. Further, between and after the support sessions, teachers sent written frameworks and/or policy protocols to the researchers. The intervention fidelity data were scored at the end of the school year using the fidelity rubric.

Data analyses

First, the intervention fidelity data were analysed for all schools using descriptive statistics. The open-ended data of the participants’ perceptions of the intervention questionnaire were analysed using a grounded theory approach (Creswell 2007). Data were imported into a Microsoft Excel® file and anonymised. First, data were coded using ‘open coding’. For each question, all respondents’ answers were analysed for content, and new codes were created when previous codes did not apply or were insufficient. After coding the entire data-set in this way, ‘axial coding’ was used in which codes were merged to form new, overlapping codes. To this end, a codebook was created in which each question, corresponding categories with a short explanation of when an answer matched the particular category, and an example answer from the database were included. For example, the first row of the codebook consisted of the question: ‘What expectations did you have about the intervention activities before participation in the intervention?’, category: ‘Little/no expectations’, explanation: ‘The participant specifically states that he or she had little or no expectations’ and example answer: ‘We did not have any expectations … (teacher 187)’. The process of axial coding resulted in four codes or categories per question.

The implementation process data were analysed using Fullan’s framework (2007). For each school, relevant statements were recorded per factor. Then, three schools were selected for more thorough discussion, serving as exploratory multiple case studies (Baxter and Jack 2008). These schools were the three for which the most information on the implementation process was available. A cross-case analysis was used to present the results, as this facilitated comparing commonalities and differences between the cases (Miles and Huberman 1994).

Results

Intervention fidelity

The level of implementation of the intervention components in schools, as assessed by the researchers, is shown in Table 2. The screening of new 4-year-old children was partly or fully implemented by 14 schools; four schools did not implement screening at all. The screening was more widely implemented than the differentiated curricula for mathematics and language. A majority preferred beginning with mathematics; eight schools started the implementation, six were advanced and two completed their design of a differentiated mathematics curriculum so that children could work at their own level and pace. The numbers are lower for the language domain, with only eight schools developing or having completed a differentiated curriculum. Finally, only two schools worked on an explicit differentiation policy in kindergarten. In the majority of the support sessions, this component was not discussed, because schools needed a relatively long time to implement the first two. In sum, there was variation in the level of implementation for each component.

Table 2. Intervention fidelity (n = 18 schools).

Intervention component and scoring categories	Schools
	n	%
Screening entry characteristics
Not implemented	4	22
Partially implemented	5	28
Fully implemented	9	50
Differentiation math curriculum
Not developed	2	11
Started structuring framework and materials	8	44
Advanced structuring and organising framework	6	33
Framework and organisation of materials developed	2	11
Differentiation language curriculum
Not developed	10	56
Started structuring framework and materials	3	17
Advanced structuring and organising framework	3	17
Framework and organisation of materials developed	2	11
Differentiation policy protocol
Not developed	16	89
Partially developed	2	11
Fully developed	–	–

Perceptions of the intervention

In a first attempt to understand this variety, participants’ perceptions were studied (see Table 3). The results show that a large number of respondents (30%) had little or no expectations about the intervention activities. Some respondents mentioned that they were not properly informed about the intervention beforehand. The difference between teachers and principals in having little or no expectations was striking: 40% vs. 11%, respectively. Another 30% of respondents expected to learn an educational approach for high-ability children, such as ‘taking up giftedness of children and then adjusting the didactic and pedagogical learning needs’ (Teacher 56).

Table 3. Participants’ perceptions of the intervention.

Perceptions	Teachers (n = 35)	School principals (n = 18)	Total (n = 53)
	%	%	%
Expectations about activities
Little/no expectations	40	11	30
Educational approach	29	33	30
Identification	6	6	6
Research activities	9	11	9
Coherence of activities with expected activities
Yes	23	33	26
No, I had to do more things	46	33	42
No, I had to do other things	23	22	23
No, I had to do fewer things	6	11	8
Expectations about outcomes/effects
Little/no expectations	26	17	23
Educational approach	46	56	49
Identification	23	33	26
Student results	6	11	8
Team/teachers/policy	3	17	8
Coherence of outcomes with expected outcomes
Yes	37	39	38
No, more outcomes	23	44	30
No, fewer outcomes	37	17	30
Points of improvement
Support sessions	34	56	42
Planning	6	17	9
Communication	17	11	15
Continuation	–	11	4

Note:

As some respondents did not answer particular questions, percentages may not add up to 100%.

About a quarter of respondents expected the actual intervention activities, while for 73%, the actual activities differed from expectations. Most respondents reported having to do more than they had expected (e.g. attending meetings, arranging learning materials and completing screening questionnaires), which was negatively perceived. One teacher (110) puts it aptly, ‘The workgroups, completing the questionnaires requires more time than expected … and we already “have to do” so much!’ Principal 1018 wrote, ‘Structuring the learning materials in the classrooms was a new element for me. It’s also a big job that cannot be realised in a year’. Some respondents indicated that although the educational approach was more extensive than expected, they appreciated it. This was evident in Principal 1024, ‘Beforehand, I didn’t estimate such an impact of the project on our teaching. It takes more time than expected and the changes are quite large as well. The latter I think is positive’. Again, some respondents were insufficiently informed prior to the intervention, as evidenced by Teacher 239, ‘Beforehand we knew about completing the questionnaires twice a year. I did not know about attending several meetings’.

In addition, about half of respondents (49%) expected an educational approach for high-ability children to be the main outcome of the intervention. Again, 23% had few or no expectations, and Teacher 116 said, ‘Whether to participate in the intervention was not decided in consultation with teachers’. Some principals also mentioned outcomes at the team level, ‘A policy that guides the school that you can make fixed rules in your team about high-ability children’ (1011).

More than a third of respondents expected the actual outcomes, such as Teacher 121 noted, ‘I gained insights on my view of teaching high-ability children which are required to give them what is necessary’ and Teacher 247, ‘I see children flourish indeed and I am very happy with that’. Some respondents stated that it was still too early to look at the outcomes, because the intervention required more time than one year to be fully implemented. In contrast, some respondents were more negative and felt they had not received any new information in the intervention. Differences were also observed between teachers and principals regarding outcomes; teachers more often perceived fewer actual outcomes than expected, and principals more often perceived more actual outcomes than expected.

A large number of respondents mentioned points of improvement. They focused mainly on the support sessions that needed to be more practical and to convey the purpose and method of the intervention more clearly. Principal 1011 summarised this as follows, ‘More practical. More time for discussions. Maybe occasionally do an assignment in the classroom. Make more use of images’. Some respondents also mentioned better planning and better communication, as Principal 1038 suggested, ‘There should be a route + timetable how your school could implement the intervention’.

Implementation process in three case studies: a cross-case comparison

A number of Fullan’s implementation factors, such as the clarity and practicality of the intervention (i.e. the purpose and method of the intervention being unclear and support sessions that should be more hands-on and practical) and the role of the teacher (i.e. some teachers being assigned to participate without prior information; high workload in the intervention), were mentioned in participants’ perceptions of the intervention. Thus, participants’ perceptions gave the first indication that certain intervention characteristics and local factors were relevant for implementing the intervention. However, the perceptions questionnaire was short, did not cover all of Fullan’s implementation factors, and was completed only at the end of the year. For a complete picture of the relevant implementation factors, it is important to study the complete implementation process during the school year. Therefore, the implementation process in three case studies (i.e. Schools A, B and C) are compared using Fullan’s (2007) factors. These schools differ in size, the use of multi-grade classes and their intervention fidelity (see Table 4). Roughly, the intervention fidelity was highest at School C and lowest at School A. In the following description, quotes are from teachers, internal counsellors or principals.

Table 4. Case characteristics including intervention fidelity of school cases.

Case characteristics	Schools
	A	B	C
School characteristics
Number of students	198	151	268
Number of kindergarten classes	3	2	2
Class type	Multi-grade	Single-grade	Multi-grade
Part of the Netherlands	Middle	East	West
Number of support sessions	5	5	5
Intervention fidelity
Screening of entry characteristics	Absent	Completed	Completed
Differentiation mathematics curriculum	Absent	Started	Advanced
Differentiation language curriculum	Absent	Absent	Advanced
Differentiation policy protocol	Absent	Absent	Partial

Need

School A did not specify a need during the meetings. School B mentioned a specific need, namely receiving information about ‘learning materials for the best students’. In School C, there was a strong need to improve teaching for high-ability children, ‘we need to do something with the children who are more able’. This was partly caused by their experiences with no intervention or too late intervention for high-ability children in upper elementary school, according to the principal, ‘because we run [sic.] into this problem for years’. In this school, many initially high-ability children became underachievers.

Clarity

During the sessions, participants from School A regularly mentioned that the intervention had to be more specified, in particular the second component. This school wanted more clarity concerning ‘what learning materials were suitable for which learning objectives, and how do you determine what each student is going to do’. School B wanted more clarity on the practical side, ‘I just want to know how to do it. Give us the real practical advice about how we have to do it, not that we have to reinvent the wheel’. School C experienced difficulties understanding the practical aspects. As the principal put it, ‘I think everyone is motivated to let each student work at his/her own level, but we don’t know what the end point actually is’. This remained difficult during the implementation, as a teacher once said at the end of a session, ‘I’ve got a lot of answers, I’m very happy. But I still have many questions as well’.

Complexity

The intervention was quite complex for School A. Participants wondered how level-based education could be combined with group plans and age-based curricula. ‘We’re used to age-based classes. How is it feasible to have education at one’s own level?’ In addition, questions arose about the consequences of this intervention for higher grades: ‘What is going to happen later in grade 1?’ Further, classification of learning materials was experienced as complex, because ‘the material is often suitable for different levels, but then the instructions are very different’.

School B experienced problems with the complexity for classroom management. As the internal counsellor put it, ‘What does a day look like for such a student? How to organise it? A day is all organisation’. This school had difficulty with leaving the age group approach behind and were not keen on doing so. ‘It needs to be adapted, everyone on their own level, but within our current system’.

School C acknowledged this complexity, too, but this school was more open to change and was better able to deal with it. The principal said in a meeting, ‘It will be difficult with personnel formation. Now we work in a year group system, and we will slowly bend it’. And later, ‘We have to put a lot aside. Is it going to be like we expect?’ Subsequently, the school continued the intervention and a teacher emailed, ‘You will understand that this is very exciting for us and we would therefore like to hear from you how you perceive it’. Thus, the intervention was complex for this school as well, but the school tried and wanted feedback from the researchers on their progress.

Quality and practicality of the intervention

The intervention’s quality and usefulness were negatively perceived by School A. First, this concerns the use of the screening, as the principal indicated, ‘We have been using more lists, may we decide that we do not use this screening too? What is the added value of this list?’ In addition, the importance of organising the materials based on student levels was not recognised, ‘I would not like to structure it separately for each class, because now all children can use it’. Finally, this school had few materials to completely fill the level-based curriculum.

School B also experienced this situation. Moreover, classroom space was an issue, a threat to the quality and usefulness of the intervention. There was no space available in the classrooms for the additional cabinets that were required to classify the learning materials. At that moment, the materials were not structured at all, and the content of the cabinets was unclear for both teachers and children. This school did not see any relevance of children working at an accelerated pace, as ‘nowadays, methods are usually enriched, and in three levels’. This school believed that the current methods provided sufficient challenge for high-ability children.

School C wondered about space as well, but they estimated the quality and usefulness of the intervention very highly. ‘Building blocks, cabinets, organising between groups, are beautiful ideas. It feels safe too’. However, they wondered how to avoid ‘that gifted children have completed the regular curriculum of the school in a very short amount of time’. Moreover, higher level testing of children was not always useful for this school, because it took more time than expected and not all tests were suitable for young children.

School board/district

School A appeared, without being aware, to have applied to take part in the intervention as a result of the school board’s actions. During meetings, it became clear that the school was obliged to participate. The board’s influence was large, since board representatives were present during sessions as well, and they interfered with the intervention’s content. School B, on the other hand, wanted more involvement from their board. As the internal counsellor put it, ‘There have been issues going which are school-wide policy. We actually want to achieve this at board level. I think we need to develop our policy school board-wide’. This school had a wait and see attitude, and wanted prior board approval for the intervention. A clear role for the board was not shown. In School C, the board played no role.

Community

Competition between schools played a role for School A, as ‘there is an elementary school nearby in XX, where gifted children receive extra support three times per week’. High-ability children moved there, and the school was afraid of more children leaving. The community, and particularly parents, served a different role in School B. ‘The are just a bit programmed, in YY, like you have to be normal’. This school mentioned that parents prefer not to accelerate their children. School C has ‘relatively many smart children here at school’. This school felt pressure from parents to do something for high-ability children, because parents would otherwise move their children to a neighbouring ‘Leonardo’ school (a school for gifted children only). The parents at this school were characterised as highly educated, articulate and very demanding of their children.

Principal

School A’s principal was present once, and soon transferred responsibility to a teacher. Thereafter, he played no role in the intervention. In School B, the principal was never present and played no role. The principal had an important role in School C, was present at every meeting and also founded an internal project group in the school that worked on the intervention between official sessions. The principal also participated in this group, primarily intervening in discussions and democratically reaching decisions. For example, he asked, ‘How can we make it easy for each other? Maybe we can implement school wide times for instruction, and try this next year’. He stressed the importance of success experiences, ‘We need a success experience right now. Otherwise it remains vague and unstructured, and then we won’t make it to the finish’. He also intervened when teachers were overloaded, ‘The enthusiasm is still high, but as principal I think: Isn’t this intervention too ambitious? I also want everyone to make it to the finish. We have to keep it manageable’.

Teachers

Teachers from School A were not always present during support sessions. One had negative experiences with accelerating her daughter, who was first accelerated and later repeated a grade because of her socio-emotional development. This experience was a barrier for this teacher and school. In addition, she said, ‘We think that using material only is detrimental for the children, because playful learning is very important’. According to the internal supervisor, the culture in this school was strongly focused on playful learning, also for high-ability children. Colleagues were not informed about the intervention. Teachers from School B also were not present all the time. One teacher, who was always present, did not value the intervention, as ‘the good children will make it anyway’. The other teachers at the school were not informed about the intervention. In addition, making time for intervention activities was a problem for this teacher. In School C, all teachers of kindergarten through second grade were present at each meeting and realised that they colleagues need to be informed. ‘This requires a decision of the whole team’. There was a constructive atmosphere. One teacher was still a little uneasy about the intervention, saying, ‘I’d like to continue with those kids, but I don’t think they can continue to second grade at the end of the year. I don’t dare accelerate them’. Another teacher replied, ‘Trust yourself. We all think it is exciting’. The tension between keeping the children with them and releasing them came back regularly: ‘I think it’s difficult if you do not always see your children. I’m afraid I lose the overview’.

Government and other agencies

The government and other agencies were not mentioned in Schools A and B. The Educational Inspectorate played a small role in School C. According to the principal, it would be difficult to justify higher level assessments and subject acceleration to the inspectorate, as proposed in the intervention.

Findings

The success of an educational intervention – and thus the future determination thereof – depends largely on its implementation in everyday school practices (Fullan 2007). It is, therefore, essential to study the implementation in detail. This study examined the fidelity of an intervention to improve teachers’ differentiation practices in Dutch kindergartens with a particular focus on how this would support high-ability students, and what factors influenced the intervention fidelity.

Results showed that the level of intervention fidelity differed considerably between schools. In most schools, the implementation of the screening and differentiated curricula took more time than expected, which led to less or no time for the policy protocol. Based on participants’ perceptions and cross-case analyses of the implementation in three schools, the following interacting factors appeared to contribute to successful implementation of this intervention: the experiencing of a strong need for educational change for high-ability children, pressure from parents, an actively involved principal who facilitates teachers in the intervention and the trust and support of a team in which everyone is aware of the intervention. Barriers for implementation were ambiguity and complexity of the intervention, which was partly due to lack of information and lack of communication between individuals; an interfering or reluctant school board; an absent principal; and certain beliefs, low motivation and little support and time among teachers. However, the case studies showed that factors can vary greatly per school, and that schools differ in how they deal with these factors.

Study limitations

A number of methodological considerations needs to be taken into account when interpreting the findings. First, the sample of schools was taken from the researchers’ network of schools, all with an interest in better supporting high-ability pupils. The sample may thus be not completely representative of Dutch primary schools and may be somewhat biased. Moreover, the schools received financial support for implementing the components, which may have influenced their (extrinsic) motivation to sign-up for participation. It seems plausible that realising comparable changes in elementary schools involved in a representative, non-self-selected sample will be more difficult (cf. Shadish, Cook, and Campbell 2002).

Second, the perceptions of the intervention were measured only at the end of the intervention. As a consequence, the validity of the answers, particularly those of the initial expectations of the intervention, may be lower than when the questionnaire was administered at the start of the intervention. To compensate for this, we extended the questionnaire with a number of questions asking for the coherence between expectations and outcomes, and asking for points of improvement for the intervention. By using open-ended questions in which the participants could exemplify their answers, participants had the opportunity to reflect on their intervention perceptions in a open, non-directed way. Further, the outcomes of the perception questionnaire were in line with the outcomes of the cross-case analysis of Fullan’s implementation factors in three schools. This indicated that the validity of the perception questionnaire was acceptable. Third, the limited time to implement the three components caused a stronger focus on the first two components and a smaller focus on the third component, i.e. the development of a policy protocol. Although it was the intention to implement all three components, this was not possible in the busy everyday school practice. However, as policy is a strong instrument in changing teacher practice, it is important that schools develop a protocol in which the vision is shared by all teachers (Dutch Inspectorate of Education 2015). Schools require more time to develop this component. In this respect, the current stimulation of the Inspectorate may be helpful.

Finally, in some schools, teachers were not informed beforehand about participating in the intervention. Even though it was the researchers’ intention to have schools voluntarily participating, some schools were surprised by their school boards, which were responsible for the participation but had refused to communicate this to the principals. A similar thing was noted by Kirschner and Prins (2008) who studied the nationwide implementation of three different ‘innovations’ in Dutch secondary schools. In their study of parents’, teachers’ and administrators’ experiencing of the innovations, they found (1) a large degree of agreement between parents’ and teachers’ experience, who found the innovation to be top-down, poorly planned and poorly funded in terms of materials, time and money; and (2) a large discrepancy between these two groups and the administrators, who were much more positive about the innovations and their implementation. They note, ‘there is a chasm between school administrators (i.e. principals) and teachers primarily over the goals and the achievement thereof’ (121). Problems in relationships and communication between school principals and teachers may result in weak relational trust (cf. Le Fevre 2014), which may have negatively affected implementation processes. As shown in this study, a lack of ownership when interventions are imposed on schools can hamper implementations (see also Bergen and Van Veen 2004; Swann and Brown 1997).

Discussion

This study indicated that implementation variation between schools is the result of a complex interaction between different factors (Fullan 2007). As some schools showed a ‘culture of caution’ (Le Fevre 2014), others, even while demonstrating uncertainty, deliberately took the challenge. In this respect, the interaction between factors is visible in such a way that teachers and principals support each other, as there is a strong will to change and support high-ability children at their own level (cf. Deunk and Doolaard 2013). According to Le Fevre (2014), a school culture that is non-punitive but which embraces inquiry as a model of professional learning might reduce the perceived degree of risk in trying something new. Research has shown that successful implementation is enhanced in school cultures with safe contexts for conversations and sharing, when teacher teams collaborate and learn from each other and when teams have positive attitudes towards educational change (Bergen and Van Veen 2004). An example of such a school culture can be seen in School C. In schools like School C, with strong organisational learning cultures, teachers are more likely to report higher levels of psychological safety, experimentation and leadership, which reinforces learning (Le Fevre 2014).

Furthermore, teacher attitudes towards high-ability children and their tuition and the complexity of differentiation affected the implementation. Some teachers thought that high-ability children had no need for difficult subject matter, which confirmed earlier research (Doolaard and Harms 2013). This is often referred to as a ‘non-competitive mentality’ and has traditionally long been present in the Dutch society (Hofstede 1997). In this respect, Mooij and Fettelaar (2010) clarify that equal opportunities in education are often confused with equal teaching. In the latter sense, equality is interpreted as equal education for every student, regardless of level, thereby neglecting the learning needs of high-ability children. Finally, teachers experienced difficulties in integrating the concepts of the intervention into their teaching. This is not surprising, as earlier research showed that the skill to adequately differentiate for all children in a class, including high-ability children, has not been mastered by a large percentage of Dutch teachers (Van de Grift 2010). It is, thus, important that there is training to support teachers in adopting differentiation in their teaching practices. From the case studies presented here, it became clear that the intervention partly succeeded in this challenge, but that specific intervention improvements and more support are necessary.

Implications

Differentiation in an age-related school system requires a lot from teachers (cf. Segers and Hoogeveen 2012). As this study showed, it is not easy to implement an intervention in kindergarten to enhance differentiation for high-ability children. It is thus important to take the following learned lessons into account in future interventions (Swanson et al. 2013).

This study stresses the importance of taking teacher beliefs into account when schools want to implement differentiated curricula in their teaching. The belief that high-ability pupils do not need to receive education at their own level still existed (cf. De Boer, Minnaert, and Kamphof 2013), even though most participating schools voluntarily participated in this research and were willing to better support their high-ability pupils. It is important that these beliefs are identified before participation and are anticipated by the deliverers of a training or intervention as much as possible (Le Fevre 2014), because if the teachers do not want to change, the change will not succeed. Clear communication from the start – which also includes insuring that the message as given is the same as the message as received – can ensure that teachers feel ownership and remain motivated and perceive that they and their problems are taken seriously (Castro-Villarreal, Rodriguez, and Moore 2014; Gorozidis and Papaioannou 2014). Here, an important task is identified for the principal. By providing support where teachers need it, and facilitating time and collaboration among teachers, the principal is crucial in taking care of a culture of progress and development.

A next research step could focus on how teachers could best be taught (i.e. in pre-service) and trained (i.e. in in-service) in improving their differentiation skills. Here, pre-service and in-service teacher education can play an important role by providing teachers with knowledge about the characteristics and needs of high-ability pupils and by more intensive training in the acquisition of differentiation skills. Currently, the Ministry of Education, Culture and Science stimulates research that aims at helping pre-service and in-service teachers with more optimally differentiating between their pupils (Dekker 2015). By integrating a practical training method for teachers’ differentiation skills in Excel Kwadraat, the intervention can improve both the requirements for and the skill of differentiation between pupils, to more optimally serve the pupils on their own level.

For continuing support of high-ability pupils’ learning, it is crucial that Excel Kwadraat is not limited to kindergarten. Further research should focus on how higher grades could also be included in Excel Kwadraat and can differentiate their curricula in order to make individual learning paths possible. To provide continuing, uninterrupted learning paths for high-ability pupils, it is important to study ways to integrate the differentiated curricula into these higher grades with their stronger structured organisation. Here, ICT could be relevant for structuring and monitoring pupils’ learning activities and progress (Mooij et al. 2014).

In addition, it seems that long-term professional development is required to fully work according the components of Excel Kwadraat. To this end, collaboration between schools is encouraged. After the intervention ended, some schools that participated in this study decided to collectively continue improving their differentiation practices in early primary school and higher grades. Ongoing support and profiting from the experiences of other schools would give these schools the opportunity to further develop their teaching practices for diverse types of pupils. In this respect, the use of a community of practice approach (Lave and Wenger 1991; Sheridan et al. 2009) is recommended, wherein participants can share and discuss ideas with colleagues.

Based upon the DBR-approach used here (Anderson and Shattuck 2012), some adjustments in the intervention delivery can be recommended. Specifically, more attention should be given to the clarity and complexity of the intervention. This could be done by using more representations (i.e. images of cabinets from other schools, etc.), completed examples and process worksheets (i.e. a plan of a level-based curriculum, protocols, etc.) and assignments during the sessions. Sustainability can be enhanced by better communication of the intervention activities beforehand and noting agreements at each meeting to implement a specific part of the intervention before the next meeting. By doing this, the intervention is expected to better fit the needs of the participants and be implemented with more fidelity.

Notes on Contributors

Elma M. Dijkstra (1987) is a PhD candidate at the Welten Institute, Research Centre for Learning, Teaching and Technology at the Open University of the Netherlands. Her research is centred around the optimisation of education for gifted pupils, especially in Kindergarten. Her PhD project involves the training of teachers to anticipate pupil differences and apply adequate differentiation practices. She studies the effect of teachers’ insight into children’s levels of cognitive and social developments, teacher subject and curriculum knowledge, and school-wide policy for integrating differentiation on the cognitive and socio-emotional development of (initially) excellent and non-excellent kindergarten children.

Amber Walraven (1980) is a senior researcher at the Institute for Applied Social Sciences (ITS), Radboud University. She is an expert in the field of ICT in education. Integrating ICT in education, and what that means for teachers, students and organisation are the main focus of her research. She is also the project leader of two projects around optimal education for excellent students. Both projects involve the implementation of new approaches, materials and procedures in everyday education. Very similar to ICT in education this affects teachers, students and organisation. The way teachers go about these (ICT and education for excellent students) changes, effects on student learning and what others can learn from it, are discussed in various publications.

Ton Mooij holds a position as a professor by special appointment for educational technology at Welten Institute (Research Centre for Learning, Teaching and Technology), Open Universiteit in the Netherlands. He also served as a manager and senior researcher at ITS (Institute for Applied Social Sciences) of Radboud University Nijmegen, the Netherlands. His research combines ICT-supported assessment and improvement of school processes including cognitive and social learning effects with pupils in primary and secondary education. His main topics of interest concern the high ability of pupils, development of differentiated education and support for optimal cognitive and social learning in school practice, school safety and prevention of bullying and violence for both pupils and teachers in and around schools. He has been awarded several prizes for his publications focusing on theory and experimental research on cognitive excellence in elementary school practice. He has conducted six national surveys into pupils’ and teachers’ social safety, bullying and violence, in both primary and secondary education. Ton Mooij also investigated various school-based intervention practices to effectively prevent cognitive and social problem behaviour of pupils.

Paul A. Kirschner (1951) is a professor of Educational Psychology and Programme Director of the Fostering Effective, Efficient and Enjoyable Learning environments (FEEEL) programme at the Welten Institute, Research Centre for Learning, Teaching and Technology at the Open University of the Netherlands as well as visiting professor of Education with a special emphasis on Learning and Interaction in Teacher Education at the University of Oulu, Finland. He is an internationally recognised expert in his field. He is a past President (2010–2011) of the International Society for the Learning Sciences, a member of the Scientific Technical Council of the Foundation for University Computing Facilities (SURF WTR), is chief editor of Journal of Computer Assisted Learning and associate editor of Computers in Human Behaviour and chief editor of the Journal of Computer Assisted Learning, is author of the highly successful book Ten steps to complex learning, and editor of two other recent books (Visualising Argumentation and What we know about CSCL).

Funding

This research was supported by Agentschap NL [grant number ODB 10029].

Disclosure statement

No potential conflict of interest was reported by the authors.