Challenges for implementation in diverse settings: reflections on two randomised controlled trials of educational interventions in South American communities

ABSTRACT

Research in the UK suggests that multi-componential interventions focusing on language and pre-literacy skills can improve children’s reading and language skills. However, simple translations of such programmes may not produce equivalent effects in diverse communities. The reasons for this are multi-faceted and include factors beyond the rationale and content of the intervention programmes themselves. Understanding these factors is critical for creating programmes that will generalise across settings. In this review, we reflect upon challenges encountered in two reading and language intervention programmes in South America to identify community and cultural contextual factors that can influence the implementation and scalability of educational programmes. We use our findings to develop an education-specific framework to guide the development and implementation of high-quality evidence-based approaches to language and literacy intervention. Our model guides implementation practices in diverse contexts and stresses the importance of the evidence-base and communication.

KEYWORDS:

• Reading
• language
• intervention
• implementation
• cross-cultural abbreviations: CLARA - chilean language and reading alliance intervention trial
• PROLIN - educational programme for promoting child language (programa educacional para a promoção da linguagem infantil)
• CFIR - consolidated framework for implementation research
• RCT – randomised controlled trial

Introduction

One in four students globally fail to reach baseline proficiency in reading. The proportion is higher for children from developing countries and low-income and rural communities (Organisation for Economic Co-operation and Development (OECD), 2017). Research demonstrates that structured, multi-componential intervention programmes that promote a strong foundation in language can ameliorate reading difficulties (Whitehurst and Lonigan 1998). However, it is also noted that outcomes and effect sizes vary between implementations (Lortie-Forgues and Inglis 2019). This sometimes reflects differences in study design or application (Lortie-Forgues and Inglis 2019; Lemons et al. 2014; Carroll et al. 2007) or may result from contextual influences which are known to have particularly strong effects in educational research (Glass 2016; Durlak and DuPre 2008). Such differences mean that intervention programmes that are successful in one setting may not necessarily port across to other cultures, even when the societal structure is similar (Slavin, Sheard, and Hanley 2014). These findings take on particular significance when we want to extend existing programmes across diverse settings. There are few evaluations of such interventions in low-income contexts (Piedra 2006; Mount-Cors 2010) and even fewer appraisals of factors such as community and resource complexities that may affect the sustainability of such programmes (Nag et al. 2014; Spier et al. 2014). In short, while the evidence from intervention studies conducted in high-income countries (primarily with English-speaking children) has been informative (What Works Clearing House 2017), there are few implementation frameworks that relate specifically to educational settings and little discussion of the challenges associated with sustainable implementation in disadvantaged communities (e.g. low-income, isolated, rural and minority communities).

Information regarding the context of an intervention (including quality tracking, evaluation, outreach, monitoring and local processes) has proven critical to the real-world implementation of clinical interventions (Knowler et al. 2002; Katula et al. 2011; Ackermann et al. 2008) and allows the generalisation of successful programmes across a range of settings (Glasgow et al. 2012). Implementation studies can inform policy and feed into a two-way system, where previous experience informs future programmes, maximising benefit. Thus, they have the potential to make a significant contribution to social science, education, child psychology and psychiatry (Williams and Beidas 2019), aiding researchers to move beyond ‘What works’ to questions regarding ‘For Whom?’ and ‘Under what conditions?’.

Greenhalgh et al. (2004) conducted a systematic review of the assimilation of health innovations into service delivery, asking why innovations are adopted at different rates within and between networks (Greenhalgh et al. 2004). They make it clear that adoption of an innovation should not be viewed as an event but, rather, a process in which there is ongoing interaction between the innovation and its adopters’ values, norms and perceived needs. Assimilation is a dynamic process and vital for sustainability. In the pre-adoption phase, it is important to ensure that there is awareness, information and understanding of the effects of innovation on users as well as the outcomes. During the early phases of its use, access to information, training and support needs to be available. Following training, established users will wish to adapt and refine the innovation to improve fitness of purpose (Greenhalgh et al. 2004). Key attributes which affect adoption rates include the complexity of the innovation and its perceived relative advantage (in terms of effectiveness or cost-effectiveness). Innovations are adopted faster if they can be broken into manageable parts and adopted and trialled incrementally. A high degree of uncertainty about outcomes should be avoided as innovations are more readily adopted if the benefits are visible. Finally, flexible innovations that allow adaptation are more readily adopted (Greenhalgh et al. 2004).

Greenhalgh et al. (2004) acted as a starting point for the development of the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al. 2009), a multidisciplinary framework that catalogues contextual factors that affect sustainable implementation (Damschroder et al. 2009). Within this framework, factors are split into five domains (intervention characteristics, outer setting, inner setting, individual characteristics and process). Damschroder et al. (2009) suggest researchers can select CFIR constructs that are most relevant for their particular setting and use these to guide assessments of implementation success. It is relevant to note that, although the CFIR is stated to be multidisciplinary, it is still mostly medical in orientation and specific educational frameworks have yet to be developed.

Here we reflect on the challenges of implementing evidence-based language interventions in diverse settings. These reflections can provide a window on issues which need to be addressed if new interventions are to be assimilated and sustained across communities. With the aim of proposing an education-specific implementation framework, the main objective of the present study was to evaluate the relevance of the CFIR to educational interventions and to implementation across contexts. Process documentation from two randomised trials in South America was analysed to identify implementation challenges.

Materials and methods

Intervention programmes and settings

This paper examines the implementation of two different evidence-based reading and language intervention programmes in South American communities, one rural and isolated, one urban. Both municipalities in this study were classified as developing communities in Upper Middle Income Countries (per capita Gross National Income (GNI) $4126-$12,745 in 2013) at the time of intervention (Organisation for Economic Co-operation and Development (OECD) 2017). The research teams in each case were multi-national and implemented a school-based randomised controlled trial design built upon evidence from earlier effective interventions delivered in the UK (Fricke et al. 2017; Hatcher et al. 2006; Clarke et al. 2010).

The Chilean Language And Reading Alliance intervention trial (CLARA)

The first intervention targeted reading and language development. It included guidance in phoneme awareness and letter sound knowledge in the context of graded books (Hatcher, Hulme, and Ellis 1994) and oral language training through active listening, vocabulary and narrative skills (Fricke et al. 2013) supplemented in the later stages by support with reading comprehension using metacognitive strategies and reciprocal teaching (Clarke et al. 2010).

The programme was delivered over 27-weeks by specially-trained tutors to an Island community west of Chile. The Islanders are culturally distinct and geographically isolated from the mainland. In summer, there is weekly airplane access, but during winter there is only a twice-monthly cargo ship. This setting brings with it many educational challenges; Islanders often spend protracted periods of time on the mainland affecting school attendance (for example, for medical appointments), resources (for example, books) are in limited supply and it can be hard to recruit and keep teachers at the Island school. This population was chosen for the intervention study because of a reported high incidence of language and learning disorders (Villanueva, De Barbieri, and Palomino 2008). CLARA included 68 children between 4 and 14 years of age and was delivered within the single local school, in which Spanish is the language of instruction. The intervention was effective: the children who had received the programme made gains in language, reading and reading comprehension compared with the waiting control group (effect sizes d > .25). Gains in reading and in word knowledge were maintained 9 months later but gains in language and reading comprehension were not.

Ethical approval for the CLARA study was granted by Oxford University Research Ethics Committee (R42391/RE001).

The Educational Programme for Promoting Child Language (Programa Educacional para a Promoção da Linguagem Infantil – PROLIN) Intervention Trial

The second intervention, PROLIN, (Puglisi, Hugo Cogo-Moreira, and Polanczyk 2019; Puglisi et al. 2016, 2018) was inspired by the Nuffield Early Language Intervention (NELI: Silke et al. 2013, 2017), a language intervention for preschool children struggling to develop oral language skills; it applied principles of active listening, vocabulary and narrative skills. PROLIN consisted of two separate studies; a universal intervention delivered by teachers to 568 children, aged 4–5 years, across 27 schools in São Caetano do Sul, a metropolitan area of São Paulo (Puglisi, Hugo Cogo-Moreira, and Polanczyk 2019), and a smaller study with teachers in Rio Claro, an urban city in the countryside of the state of São Paulo (Puglisi et al. 2016, 2018). The Rio Claro arm targeted 124 children, aged 4–5 years, with poor language performance attending the first year of Early Education (equivalent to Reception in the UK). There was significant heterogeneity between the schools participating in PROLIN, some were in low-SES areas and others in areas of medium SES, where parent- school associations are stronger.

In contrast to the CLARA trial, the PROLIN trials produced significant effects only in some settings. The research team proposed that the variability in effects could be explained by differences in fidelity across schools. There were other differences between CLARA and PROLIN, however. While the source of the intervention was similar between the projects, the delivery, settings and stakeholders differed. All of these factors could have affected efficacy and ultimately would affect sustainability.

Analysis framework

It is not our purpose here to focus on programme efficacy; this has been documented elsewhere (Mesa et al. 2020; Puglisi et al. 2016). Rather, we consider the implementation issues that surrounded the two programmes in order to identify factors that should be taken into account when planning RCTs in order to promote sustainable interventions. These contextual factors were not immediately assessed or accounted for in terms of trial impact within the RCTs but may have affected programme outcomes and sustainability. We use the CFIR framework (intervention characteristics, inner setting, outer setting, characteristics of individuals and process) to reflect on aspects of implementation and identify elements that are critical to interventions in diverse contexts. We explore the utility of this framework for considering the roll-out of educational interventions and use our findings to develop an educational research framework and a theory of change.

Results

Evaluation of implementation

As a starting point for considering issues of implementation, team members discussed the relevance of each construct of each CFIR domain for the programme they had implemented. Not all constructs had been directly measured in the participant RCTs and none had been included as variables within the intervention studies. Tables 1–4 summarise the findings for CLARA and PROLIN and the inferences that could be made.

Table 1. Evaluation of CFIR intervention characteristics domain in CLARA and PROLIN.

Domain	Construct	CLARA	PROLIN	Notes relating to adoption in low-income settings
Intervention characteristics	Source: perceived as internally or externally developed by stakeholders.	External.	External but introduced by a professional researcher from same region.	Development outside of the community likely to have affected adoption.
	Evidence strength & quality: whether stakeholders believe that the intervention will lead to the expected outcome.	No direct evidence that this combined intervention works in this environment.	Evidence of an effective programme in the UK. Believed to be high quality by participating teachers.	May not be perceived as relevant given lack of a tradition for adopting evidence-based practices.
	Relative Advantage: belief that the expected outcomes of the intervention will be advantageous.	Not seen as relevant to the community.	High uptake implies stakeholders saw programme as relevant.	Highly relevant to engagement and recruitment.
	Adaptability: can be adapted to meet local needs and customs and flexibility is desirable.	Programme elements made contextually relevant by researchers with local knowledge.	Adaptations were made by a team with good knowledge of the community.	Protocols for RCTs tend to allow limited adaptability to ensure robust fidelity.
	Trialability: opportunities to try out the intervention in small scale pilots.	Some protocol modifications made as trial proceeded.	All material piloted within two classrooms.	Piloting is vital to acceptability and ideally needs to be completed before recruitment.
	Complexity: aspects of the length of the programme, the disruptiveness, and novelty in relation to usual activities and practices.	Withdrawal from the classroom posed logistical challenges Several components unfamiliar to professionals delivering programme.	Delivery only required small changes to the curriculum. Some components unfamiliar to professionals delivering programme.	Less complex interventions more feasible; training an important consideration when introducing novel education programmes.
	Design quality and packing	Local resources limited use of worksheets. Stakeholders (teachers and municipal staff) did not become familiar with materials, limiting sustainability.	Printing worksheets required additional teacher time. Teachers said they found the amount of material confusing and would like better signposting.	User-friendly resources are an important design feature (see above Complexity).
	Cost: in monetary terms.	One-to-one delivery is expensive and limits sustainability.	Costs were cut by using books already in the school library.	Cost effectiveness is important for sustainability.

Table 2. Evaluation of CFIR outer settings domain in CLARA and PROLIN.

Domain	Construct	CLARA	PROLIN	Notes relating to adoption in low-income settings
Outer settings	Participant needs and resources: are they taken into account?	Communication was in person and required significant effort and time; programmes were individualised; tutors were selected for post of delivering programme.	Programmes were designed with knowledge of participant needs in a wide range of schools and communities. In larger classrooms, it can be difficult to address individual needs.	Important to separate needs of beneficiaries from organisations. The latter make up the outer settings and it is difficult to get any more than a superficial impression of these without living in the community or co-opting locals.
	Organization networks (cosmopolitanism): how institutions delivering interventions are connected with other external organisations.	School personnel did not feel ownership of the programme; its philosophy was unfamiliar to them.	Working with volunteer schools allowed ownership of programmes and agents for sustainable change. Principals were trained alongside teachers providing potential to multiply knowledge.	A full appreciation of these networks and more importantly, how they operate, can be difficult for an ‘outsider’ to gain.
	Peer pressure (competition between organisations).	Competition between organisations less obvious in a remote community; however, school teachers may have felt a competitive advantage over speech therapists delivering the intervention.	Cooperative climate between teachers and research team was beneficial to delivery.	Typically peer pressure is reduced in low-income settings and remote communities.
	External policies and incentives: policy recommendations and guidelines.	It was difficult to take account of the views of external agencies in pre-implementation stage because team resided outside of community.	Access to programme was centrally controlled through government agencies. Teachers were given certificates of participation recognised by the authorities.	Where possible, intervention should draw on existing resources and will have little chance of success if in conflict with prevailing educational policy.

Table 3. Evaluation of CFIR inner settings and characteristics of individuals domains in CLARA and PROLIN.

Domain	Construct	CLARA	PROLIN	Notes relating to adoption in low-income settings
Inner Setting	Structural characteristics: social architecture of organisation in which intervention is implemented.	Organization localised.	Organization of schools and local authorities must be considered.	If possible, consider connections with external organisations (cosmopolitanism).
	Networks and communication: Connections between individual stakeholders and research team.	Rapport with tutors varied over time; tutor support was irregular owing to physical distance.	Continuing and regular interactions engendered ‘team spirit’.	Vital but difficulties in achieving it are easy to under-estimate.
	Culture: how values and norms of a given organisation or population can affect implementation.	Wider issues of values placed on language and literacy in the community were not taken into account [See Table 1].	Working with the teachers, schools and governmental agencies ensured the programme fitted the norms of the society.	Cultural appropriacy is vital.
	Implementation climate: the way in which the goals of the programme are communicated. Input of stakeholders into change process.	Remoteness of community made consultation difficult. Stakeholders did not provide feedback until after intervention.	Good engagement of leaders and a sense of community; practitioners were involved in the development of the programme.	Interactions between internal and external stakeholders need to be regular to maintain mutual understanding.
Characteristics of Individuals	Knowledge and beliefs: An individual’s background affects their approach the intervention in terms of delivery (schools and teachers) and receipt, (students and their families).	Tutors were selected because of relevant background; they were informed of (but some underestimated) the challenges associated with Island life.	Delivered by trained teachers who were fully informed of the aims of the study and motivated to deliver it.	Important to understand the community’s values especially if there is a gap between the skills/experience of the development team and the setting in which it will be delivered.
Characteristics of Individuals	Self-efficacy: central to behavioural change theories. The more self-belief an individual has, the higher the chances of success.	Tutors needed to complete tasks outside of the typical remit and make decisions about project management (e.g. re-scheduling).	Teachers were confident in classroom delivery but not material and/or rationale. Individual differences observed.	Self-efficacy can be difficult to assess prior to intervention delivery.
	Individual identification: an individual’s perception of the management of organisations involved in the intervention.	Islanders raised concerns that new teachers often brought practices that were rarely integrated into practice.	Positive rapport between researchers and teachers aided design and delivery.	Relates to knowledge and beliefs and the extent to which the ‘source’ of intervention is ‘external’ or ‘internal’ to the organisation.
	Individual state of change: attitude towards the intervention can change across time	As the programme became established, a change was seen; more families requested to be involved and teachers were open to additional work.	Some teachers became more confident as programme progressed. Others felt ‘lost’ in the amount of information provided.	May be related to self-efficacy. Change is likely to be slow when new interventions are introduced. Important to signpost materials and to monitor progress.
	Other personal attributes: traits such as motivation, values, competence, capacity, tenure and learning style	Short tenure of teachers had negative effect upon motivation and belief.	Differences in personal attributes such as proactive behaviour affected delivery competence.	These traits affect fidelity of delivery. In research trials, fixed term contracts present a risk to successful completion.

Table 4. Evaluation of CFIR process domain in CLARA and PROLIN.

Domain	Construct	CLARA	PROLIN	Notes relating to adoption in low-income settings
Process	Planning: Building of local capacities to allow delivery. The use of ‘dry-runs’ to allow incremental introduction.	Incremental introduction not feasible given timeline, financial constraints and fixed-term contracts of tutors.	Local capacity was promised in planning phase. Incremental introduction was not feasible because of the project timeline and endpoint for researcher contracts.	School management systems and schedules can affect implementation. On-the-ground needs have to be balanced with maintaining integrity and fidelity.
	Engaging: Engagement needs to be at multiple levels in both inner and outer settings and may include formal appointments, informal ‘champions’ and external change agents.	A few strong-minded individuals were observed to drive changes in attitude, either negatively or positively.	Advocates at educational or municipal levels (or both) with existing links to the community and an understanding of local governance was helpful.	Engagement is important to successful innovation. Local champions are important influencers. Identifying these takes care and is not always possible before implementation begins.
	Executing: A high level of fidelity is critical to build an evidence base. However, successful implementation also requires some level of flexibility to allow an organic programme that maximises sustainability	Adherence and fidelity was assessed by diaries and tutor observations. Flexibility in design allowed the addition of components to stretch able readers.	Pilot phase allowed trial and training. Variability in delivery was seen between individuals and quality of implementation was not measured	High fidelity is needed to maintain intended outcomes. ‘Drift’ is more likely if the originators of the intervention do not remain involved.
	Reflecting and Evaluating: Programme objectives should be specific, measurable, attainable, relevant and timely (SMART) allowing accurate evaluation. Reflection can promote shared learning	Primary and secondary outcomes were agreed in planning phases and provided robust evaluation of efficacy. Fidelity was monitored. Feedback was solicited at end of trial and provided important information regarding sustainability.	Primary and secondary outcomes were agreed in planning phases and provided robust evaluation of efficacy. Bespoke assessment measures were used. After the intervention, feedback and reflection led to constructive proposals for a second funded trial	It is self-evident that evaluation is important. However, while it is possible to set up SMART targets for some processes, not all are measurable. Ongoing monitoring with adjustments over time to ensure efficacy is desirable for all educational interventions.

Intervention characteristics

The first CFIR domain focuses upon the characteristics of the intervention, i.e. programme-specific features. The CLARA programme was adapted by an interdisciplinary group of professionals, keeping close to the structure of the UK programmes on which it was based. In contrast, the PROLIN authors designed the programme (including book selection and activities) with local teachers in a pilot study, making sure the core structure of the NELI programme and its principles were preserved.

Both research teams faced similar challenges – how to develop lesson-plans, activities, and materials to reflect the language and culture of communities and schools while maintaining the integrity of evidence-based practices. The absence of a pilot study for CLARA meant that some difficulties were unanticipated. For example, the children were unfamiliar with the interactions and questions typical of evidence-based practices; they were not used to adults reading to them or posing questions to promote complex language skills (e.g., inference making); they were also not typically encouraged to express themselves (Nag et al. 2014; Opel, Ameer, and Aboud 2009; Nag, Snowling, and Asfaha 2016).

Both teams reported that the strong evidence base for their approaches helped to build confidence and facilitated a shared belief between stakeholders, some of whom were initially sceptical. CLARA took place on a remote Island and was delivered by speech and language professionals who moved there to deliver it. Before the start of the programme, Islanders raised concerns that practices brought by new teachers were often disruptive to routines and rarely altered outcomes beyond the typical syllabus. Such concerns had affected recruitment and highlight the need to form a relationship with stakeholders before the start of intervention. The lead Chilean researcher had previously spent time living and working in the community and the research team visited as often as possible, actively seeking opportunities to listen to and cooperate. The team held meetings with teachers to understand the content of the intervention.

Through interviews and contact with caregivers, research teams can gain insight into parents’ values and expectations. Many Island parents expressed concerns about the importance of the CLARA programme and questioned the need for an alternative approach to reading instruction. Those families that did enrol had stronger links with the mainland; only 30% of child participants had a parent who had been born on the Island. Following conversations with residents, researchers reported that non-consenting families felt that their child did not need extra help learning to read and write as this skill was already taught within school or that reading and writing were not central life skills on an Island where fishing is the main source of income. In contrast, 93% of consenting caregivers indicated that being able to read was an important life-skill and 95% of consenting children had at least one parent who had completed high school. These examples demonstrate the importance of perceived relative advantage in the acceptance and sustainability of evidence-based programmes in education. Furthermore, these factors inevitably lead to ascertainment bias with consequent skewing of intervention outcomes and effect sizes; sadly it is often the participants most in need to intervention that are hardest to reach. The CLARA study found that children with the weakest skills at pre-test experienced the larger gains from their intervention programme but no formal assessment of attrition biases were made by the CLARA or PROLIN study (Mesa et al. 2020; Puglisi et al. 2018; Puglisi, Hugo Cogo-Moreira, and Polanczyk 2019). To avoid attrition of participants from a typically ‘hard-to-reach group, a UK study co-opted staff in local Children’s Centres to interact with parents and to deliver an adapted version of NELI (Burgoyne et al. 2018). Such an approach, wherein familiar local mediators rather than the research team are supported during implementation can increase the efficacy of a treatment.

In contrast to CLARA, PROLIN was delivered by teachers within their current classrooms (mainstream or special education), possibly fostering a greater level of acceptance from the outset. The PROLIN programme was first approved by the Secretary of Education, then presented to Principals and educational managers of all eligible schools. Twenty-seven of the 30 invited preschools in São Caetano do Sul; and all eight invited schools in Rio Claro volunteered to take part. One teacher in each of the schools showed immediate interest in the study, demonstrating their belief in the programmes quality and rationale. Teachers and Principals further helped to explain the purpose and aims of the study to parents, enabling the participation of almost all students in each classroom.

Logistic complexity was highlighted as a factor that had a strong effect upon implementation. Both interventions had similar structures – daily sessions centred around sound- and language-related strategies through shared activities, but these were delivered in different ways. In PROLIN, the sessions were taught by teachers in their normal classroom minimising the impact upon children’s routines. However, this approach placed additional stress upon already stretched teachers. In contrast, CLARA was delivered in small group or individual sessions outside of the classroom during the school day. This inevitably led to disruption of routines not only for the children but also for their classmates, teachers and administrators, especially when the weather was bad. The CLARA team listened to teachers’ advice on how to incorporate the programme into the daily schedule to avoid fatigue or scheduling conflicts. Over time, children’s engagement helped to reassure parents’ and teachers’. However these disruptions meant that not all participants received the complete programme.

Educational interventions usually need to span 20 or 30 weeks before significant gains are observed; when school holidays are factored in, this can extend to a calendar year before efficacy can be evaluated. Thus, time and funding constraints can limit the possibilities of collecting pilot data. PROLIN used pilot trials as training and feedback opportunities from both participant and tutor points of view. Piloting of assessments may also be needed, particularly in an RCT, where reliable measures are required for accurate baseline and outcome testing. Both teams noted the need to develop assessment tests that are both contextually appropriate and psychometrically robust, echoing a finding of studies available across low- and middle-income countries (Nag 2017). Most tests are developed in and normed on monolingual populations in the United States or UK and may not be validated for longitudinal research. CLARA found that even tasks that had been devised for use in mainland Chile were not always suitable for Islanders. Existing tasks required adaptation in both projects, by adding or modifying stimuli, possible responses, and instructions, requiring input from local professionals and piloting to assess reliability and validity. The fundamental need for the development of appropriate measures for use in intervention research has been noted by researchers (e.g. Restrepo and Silverman 2001; Langdon 1992) and is particularly important when interventions are implemented in diverse communities (Nag et al. 2014).

Turning to costs and resources, schools in rural communities typically have fewer resources (e.g. teaching learning materials, classroom infrastructure, libraries) and tend to have larger class sizes, meaning that resources are stretched (Sailors et al. 2010; Peña-López 2016). Where possible, PROLIN used books that already existed in school libraries. CLARA was given permission to use books donated by the publisher, Global Education Systems. Worksheets, which are often used in UK interventions, require paper and printing facilities and so in CLARA these were often replaced with group discussions and interactive activities. Schools in PROLIN had print facilities but the need to print worksheets added to teacher burden. Children in CLARA showed a strong preference for books over printed or electronic materials, even when the loose-leafed printed material was context-appropriate and included bright, attractive illustrations. Arguably, this may reflect the scarcity of books on the Island, a factor that should be considered when replacing paper books with loose-leafed and online resources. Teachers in PROLIN appreciated quality features such as clear signposting and organisation of material as this reduced the time required to select activities and enabled better fidelity.

Ultimately, any successful intervention programme must have costs that are sustainable. CLARA was a ‘proof of principle’ intervention study, but the use of full-time resident tutors mean that the costs would always be unsustainable in this community. PROLIN reduced project costs through the engagement of local teachers and was based in an urban setting where resources are more readily available.

To summarise for this domain, resources required, and the relative advantage associated with the aim of the programme as well as its complexity and ease in mode of delivery were highlighted as being particularly important. These programme features were largely influenced by social perceptions rather than the intervention itself.

Outer setting

The second CFIR domain, outer setting, considers factors external to the agency of delivery (Table 2).

Both research teams indicated that communication with participants was key to understanding participant needs. Participant needs can be met at a group level by careful study design but individual needs are sometimes harder to anticipate particularly within larger group activities and in programmes that include participants with special needs, such as the Rio Claro arm of PROLIN. Ultimately, the teachers, administrators and external governmental agencies have governance over the school curriculum and their support is crucial. External policies and incentives will affect this support.

According to Opel, Saadia Ameer, and Aboud (2009), non-contact recruitment strategies such as letters through the school, are not necessarily effective in remote areas; this situation is compounded if illiteracy is common. Since CLARA was delivered within a small community, personalised communication with community leaders (health care, municipality and school leaders) was possible and face-to-face meetings could be held at places where people usually congregate (e.g., health care centre and school). Goodwill was developed further by support outside of the study, for example by providing advice to parents and teachers with diagnostic assessments. Although this extraneous support fostered participant alliance and nurtured two-way communication, it arguably posed a threat to the integrity of the randomised controlled trial through the introduction of unassessed factors relevant to outcomes.

PROLIN also required careful planning to work closely with the Secretaries of Education from both cities to make sure they would provide full support to teachers and Principals. Here, the role of external networks and competition between them were regarded as helpful to implementation. With this in mind, teachers and municipal agencies were given some autonomy over the programme content during delivery. The direct involvement of teachers aided their a priori understanding of the human and physical resources available and, generated a support network. Incentives such as certificates of participation for teachers can provide a powerful means of engagement but are only meaningful when they are recognised by schools and educational authorities.

Overall, within the outer setting, positive effects can be mediated by understanding participant needs and external policies and incentives.

Inner setting and characteristics of individuals

The third and fourth CFIR domains consider the context and organisation within which the intervention will be implemented and the characteristics of the individuals involved in the intervention (Table 3).

The structural characteristics of an institution can influence networking and communication opportunities and the implementation climate, both of which were highlighted as particularly strong influencers by the two intervention teams. Research efforts involving isolated populations are very likely to bring outsiders (e.g. Banerjee et al. 2010; Opel, Saadia Ameer, and Aboud 2009) who may undervalue community beliefs and apply practices without attention to context (Nag, Snowling, and Mesfun Asfaha 2016) . To circumvent this, evidence-based practices need to build on strong relationships. Nag et al. (2014) recommend that initiatives to enhance school literacy practices should emerge from the community and their efforts should be documented (Nag et al. 2014).

CLARA developed a multi-pronged communication strategy and this likely offset some effects of bringing evidence-based practices to a community that was not necessarily familiar with those practices. A lead tutor was specifically assigned the role of communicating with stakeholders but, in reality, the entire team needed to take on this role. Face-to-face meetings were scheduled on request to provide administrators, teachers and parents with additional information. The local radio was found to be an effective channel for communication with guardians. The CLARA team held an open house and invited direct stakeholders to visit if they had any questions. Every effort was made to support delivery; however, internet connectivity is poor on the Island and lead researchers could not access the Island because of weather conditions. Tutors needed to respond to questions about project design, implementation, materials and timeline beyond a level usually expected in this role. The challenges posed by living in an isolated community with limited resources led to declining levels of motivation for some tutors, posing a risk to the quality of implementation. The situation was compounded by the short tenure of mainstream teachers posted to the Island. In the context of robust project design, changing attitudes can be problematic. However, the increased support for the project aims and team during the course of the intervention had a positive effect.

Turning to individuals, the CFIR highlights the importance of stakeholder knowledge and belief since an individual’s background affects the way they approach the intervention both in terms of delivery (by schools and teachers) and the way it is received by students and their families. In educational settings, knowledge, belief and self-efficacy allows autonomy and enables flexibility of interventionists in responding to the needs of individuals and communities. Some PROLIN teachers had more years of experience or specific training that helped them to apply a purpose-focused approach, and to improvise or adapt activities when students demonstrated difficulties. Less experienced teachers applied an example-focused approach and were less likely to adapt their practices. Besides knowledge and experience, personal characteristics (such as proactive behaviour) may also influence the quality of implementation. Previous studies have shown that teachers’ characteristics, such as the ability to engage in sensitive and stimulating interactions (Burchinal et al. 2008), the frequency of high quality teacher–child interactions (Downer et al. 2012), socioemotional competence and wellbeing (Jennings and Greenberg 2009), as well as skills, competencies and beliefs that are aligned with curriculum (Costin and Pontual 2020) are crucial to the successful delivery of educational programmes. These findings also highlight the importance of taking account of school level data in future cluster randomised trials so that results can be appropriately contextualised.

In summary, although knowledge, beliefs and individual self-efficacy for programme execution can promote the success of interventions, individuals will be limited by organisational networks and by the engagement of the stakeholders. The structural characteristics of networks together with culture and implementation climate can limit implementation. Readiness can be encouraged through effective communication but this can be complex and time-demanding.

Process

The final CFIR domain considers the process of change (Table 4).

At the point of funding, the commitment to delivery lies with the research team. As the intervention progresses, it is important to transfer ownership to local stakeholders in order to allow integration into schools or communities and ongoing delivery. This requires local training, lasting resources, continuing support and local champions who will act as agents of change. The CLARA team left materials on the Island and trained local champions but the intervention was not integrated into the school day. PROLIN underlined the importance of training material in the promotion of high-fidelity implementation within limited time frames. This consideration highlights the balance between scientific rigour and responsive real-world application (e.g. increasing exposure and providing explanations). CLARA was implemented by full-time researchers, resulting in high fidelity and (perhaps) more consistent gains but these features also made it less sustainable. PROLIN applied a more pragmatic approach to delivery but this, by definition, decreased the consistency of intervention effects and highlighted a need to accurately monitor (and, if needs be, adjust) fidelity across the programme.

In summary, aspects of process are perhaps the hardest to control in an educational setting – project designs must be understood by all stakeholders and allow for the influences of existing processes and regulations. Scientific evaluation is built into a research application but there is usually scarce time for reflection and evaluation by the team within the time-frame of funding severely compromising the growth of protocols essential to build up the knowledge for implementation science.

Discussion

In this paper we have considered how two educational interventions were implemented in diverse settings. We first draw out strengths and difficulties encountered in implementing the trials (irrespective of their efficacy) which are likely to affect sustainability. We then turn to focus on issues of implementation as defined by the Consolidated Framework for Implementation Research with the broad aim of assessing its utility in education. Examining each of the domains in turn, we identify those elements that are critical for successful and sustainable implementation of interventions in new contexts. Since we found some overlap between domains, redundancies across constructs, and some omissions, we propose an alternative structure for educational research in which we map CFIR domains on to constructs which are more common in education for considering a theory of change.

Factors important for sustainability: consideration of CLARA and PROLIN

As we have discussed, there were similarities and differences between the programmes. In each case, the programmes to be introduced had an external source and were evidence-based, the programmes were complex and training was required before delivery; however in terms of relative advantage, PROLIN had the benefit of being introduced by a credible professional from within the community. Since the views here were based on a trial and not roll-out per se, the issue of adaptability was not applicable but there were clear benefits in terms of acceptability for PROLIN in which piloting of materials and input to their design was possible. Together these factors (including cost) suggest PROLIN is the more sustainable programme.

Both programmes took careful account of participant needs and resources and were able to implement training and support; they were also designed to be culturally appropriate by including professionals and practitioners in their development, though this was less easy for CLARA given it was to be delivered in a remote region. While PROLIN was delivered in a cooperative climate in volunteer schools and local educational policies were taken into account, there was some professional tension surrounding the CLARA programme and it took time to gain the support of the community. Together these factors mean that PROLIN was at an advantage in terms of acceptability from the outset and, although CLARA was effective, sustainability could not be guaranteed.

As already noted, the efficacy of PROLIN varied between settings. Plausibly this was because of differences in the knowledge, beliefs and self-efficacy of those delivering it but these attributes were not measured systematically. Fewer individuals were involved in CLARA, however, observations of teaching highlighted the role of personality traits such as proactive behaviour and self-efficacy as important factors determining quality of delivery. Notwithstanding this, the engagement of those receiving the CLARA intervention and of other stakeholders increased over time. Together these observations highlight the need for sufficient time to be devoted to establishing an intervention in a new setting and for communication with stakeholders.

Although the CFIR was found to be useful for directing attention to aspects of intervention, there was no one domain that distinguished the CLARA and PROLIN trials or that could explain the level of efficacy or sustainability, although clearly some domains were more important than others in an educational setting. A need for consideration of the locus of control and interactions between domains was highlighted by our use of the CFIR and is integrated into the educational framework described below.

Summary and implications

To summarise findings within the CIFR, the characteristics of the intervention, the source (internal versus external) and perceived relative advantage were strongly related to each other particularly in the remote setting and communication about the intervention engaging stakeholders was fundamental at every stage of the process. We group these factors under community engagement and ownership.

For communication and engagement to be effective, attention needs to be paid to both outer and inner settings, the boundaries of which are not fixed in education systems. Knowledge of the local education structure and social hierarchies and of educational policies will ease implementation. It is clear from our reflections that if programme developers are distanced from the community, there is a risk of superficial understanding of participant needs; if individuals are to identify with the intervention, then they must see that the relevant organisations are prioritising the programme (this is also related to the source of the intervention). If this is not the case, then elements may be dropped, changed or distorted by cultural filters. Together, these considerations might be summarised as background knowledge and policy guidance.

A fundamental tenet is that educational interventions must be contextually appropriate and adaptable, allowing for locally produced quality resources with realistic costs. Co-opting internal players can ensure better design and flexibility and provide opportunities for trialling, ensuring sufficient quality resources with realistic costs. Less easy to control for is the self-efficacy, motivations and competence of those involved in implementation and their readiness for change. The importance of training, ongoing support and monitoring cannot be under-estimated and should be factored into intervention design and allow for differences in individual beliefs and knowledge. Knowledge exchange and coproduction of the intervention are time-intensive activities but can ensure intervention uptake, and sustainability.

Although the cases considered here related to the implementation of research trials, many of the principles apply to scaling-up educational interventions. Two constructs which do not form part of the CFIR are fidelity of delivery, to ensure that delivery as intended, and safeguard content against ‘drift’ over time. Programmes need to be integrated into existing systems if they are to be sustained with fidelity.

Towards a framework for implementation of educational interventions

Consideration of implementation in these two programmes allowed us to identify factors of general relevance to educational practitioners and led to the development of an education-specific framework as shown in Figure 1. This framework is proposed as a guide to the development and implementation of evidence-based approaches to language and literacy intervention in diverse settings and draws heavily on the framework of Williams and Beidas (2019) for implementing interventions in clinical psychology. Guided by our findings, this framework allows for different levels of implementation and considers interactions between domains.

Figure 1. Suggested educational framework.

The framework is split into two layers; core factors extrinsic to the intervention sit in the external layer and intrinsic factors in the internal layer. Factors in each layer can be classified according to the locus of control; outer level factors, which are entirely under the control of the research team, are shown as black boxes. Intermediate level factors are shown as dark grey boxes and represent agents over which researchers have less control. Inner level factors are represented as light grey boxes and act as agents of acceptability, fidelity and sustainability. Figure created in Lucidchart (www.lucidchart.com).

Our framework is split into two layers; the upper layer reflects factors that are external to the intervention. The lower layer summarises the main features internal to the intervention that are determinants of success. Each layer can be split into different levels which reflect the locus of control (indicated by different shade levels in Figure 1). The outer level (shown as black boxes in Figure 1) includes the factors evidence-base and communications. These are directed by the research team and represent the pillars upon which successful development and implementation must be built. The intermediate level includes 5 factors (shown as dark grey boxes in Figure 1), over which researchers arguably have less control but should endeavour to understand and, when appropriate or necessary, aim to influence or integrate into the intervention. These consist of three external agents; (1) cultural context and norms; (2) community engagement; and (3) background knowledge and policy guidance, and two internal agents (4) the development of quality teaching resources and (5) availability of robust assessment tools. Ensuring that agents within the intermediate level are in place before the start of the intervention will support factors at the inner level (shown as light grey in Figure 1); support from local networks, intervention climate, teacher training and support, confidence in evaluation, and ownership entailing motivation to accept. These factors are all internal and, as such are of upmost importance to the end goals sustainable, reliable and accepted intervention programmes (shown in white in Figure 1). However, these inner agents are also complex in nature, interacting with and depending upon many factors that are often beyond the control of the research group. Our framework suggests that, as researchers, we should focus upon measuring and understanding the outer and intermediate levels as these agents are easier to influence and will support the development of the essential inner agents.

There are two sine qua non in our framework; evidence-base and communications. A strong evidence-base fosters belief in the programme and will facilitate strong internal agents. While the content of the selected programme will need adaption according to local needs, a solid evidence-base provides a useful starting point for research in early childhood development in diverse settings including low- and middle-income countries and other disadvantaged communities. Communications should take account of the needs of different audiences with different experiences which lead to appreciation of external factors including policy guidance and the implementation climate.

Of the intermediate agents, community engagement might be considered an over-arching construct necessary for an intervention to be implemented successfully and sustained. The design of programmes should always take into account the cultural context which will ultimately influence the complexity of delivery, development of affordable quality resources and the availability of valid assessment tools. A well-designed intervention with a strong evidence base can fail if the required resources are unobtainable or irrelevant to local context. Flexibility allows input from local stakeholders and can influence identification with the programme, motivation to implement the intervention and ownership, fostering sustainability. In addition, the needs of individuals delivering interventions and how best to support them before, during and after training is universal to ongoing success.

To conclude, the framework in Figure 1 is simplified to highlight areas of significant relevance to practitioners in order to maximise sustainability. It makes explicit that community engagement is the bedrock of the successful roll-out of an intervention and high quality, culturally normed, resources built upon a strong evidence-base, are the cornerstone of the approach. Only when these are in place should a researcher embark on a trial. In addition, care must be taken to ensure that interventionists charged with delivering the intervention are properly trained and supported during its delivery. A two-way communication during early phases is essential if the intervention is to remain true to its protocols to safeguard effectiveness. Ideally, to ensure that fidelity is maintained and the intervention sustained, the intervention should align with the aims of the overarching curriculum and the authors need to remain open to feedback and agile in the need to make changes that will not threaten efficacy.

Data Availability

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Acknowledgments

We are grateful to the Municipality, school, teachers, children and families who participated in the CLARA and PROLIN studies and to Trisha Greenhalgh for her comments on the draft of this manuscript.

This work was funded by the British Academy Early Childhood Education Programme Grant (ECE190048, PI Dianne Newbury). The CLARA project was funded by an ESRC:CONICYT Newton Research Partnership: “A language and reading intervention programme for Chile, piloted in the Robinson Crusoe Island” (ES/N01913X/1, PI Dianne Newbury). The PROLIN intervention was funded by a Newton International Fellowship awarded to Marina Puglisi - “Reducing the risk of language and literacy disorders in pre-schoolers from disadvantaged backgrounds” (NF141010).

The funding bodies had no role in the data collection, analysis or interpretation of the data or writing of the manuscript.

DFN, CM, CH and MJS conceptualised the framework and wrote the manuscript. CM, and MP provided reflections on their intervention studies for this manuscript. CM, MP, MN, SN and CH provided reflections on the conceptualised framework and manuscript material. All authors read and approved the final manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the British Academy [ECE190048]; Economic and Social Research Council [ES/N01913X/1]; Newton International Fellowship [NF141010]. The funding source for the Newton International Fellowhips was the Newton Fund, under the responsibility of the Department for Business, Energy and Industrial Strategy (BEIS).

Notes on contributors

DF Newbury

Dianne Newbury is a molecular geneticist and Reader at Oxford Brookes University. Her investigations specifically focus around speech and language impairment and its relationship to disorders such as dyslexia. Dianne has a lab at Oxford Brookes University. Her work is currently funded by the Rosetrees Trust, the Leverhulme Trust and the British Academy.

C Mesa

Carol Mesa is an Early Career Fellow in the Department of Speech, Language, and Hearing Sciences at the University of Texas at Austin. She primarily studies cross-sectional and longitudinal associations among home, school, and child factors that shape Latino children’s language and reading skills in the context of the U.S.

M Puglisi

Marina L. Puglisi is a Speech-Language Therapist, Adjunct Professor at the Department of Communication Sciences at the Federal University of São Paulo (UNIFESP). PhD in Communication Sciences at the University of São Paulo (USP); Post-doctorate at USP and University of Oxford and former Newton International Fellow. Expert in child language, with particular emphasis on early intervention programmes.

M Nash

Marysia Nash practised as a speech and language therapist for 35 years becoming increasing specialised in, and clinical lead for developmental disorders of language and literacy. Her particular interest In vocabulary motivated her doctoral studies on vocabulary acquisition in children . She has developed intervention for children's communication development in SA and also now teaches English to adults across the world.

S Nag

Sonali Nag is Professor of Psychology and Education and Education Fellow of Brasenose College. Sonali investigates child learning within diverse settings. Her research is comparative with a focus on languages, writing systems, cultural settings and levels of socio-economic status. She uses a wide variety of methods including surveys in schools and home settings, child assessments, corpora analysis, secondary data analysis, and narrative reviews. Her work seeks to develop a nuanced, contextually grounded understanding of child development. Her research can broadly be categorised into the study of child level factors and contextual factors.

C Hulme

Charles Hulme is a Professor of Psychology and Education at the University of Oxford. Recipient of the Feitelson Research Award from the International Reading Association (1998), the Marion Welchman International Award for Contributions to the study of Dyslexia from the British Dyslexia Association (2016) and the Society for the Scientific Studies of Reading Distinguished Scientific Contributions Award (2019). Fellow of the British Academy and of the Academy of Social Sciences and member of Academia Europea.

MJ Snowling

Margaret J. Snowling is the President of St. John’s College and Professor of Psychology at the University of Oxford. Awarded the British Psychological Society Presidents’ Award (2003) and the Samuel T Orton Award of the International Dyslexia Association (2005). Holds honorary doctorates from Goldsmiths London, University College London, Warwick and Bristol Universities for contributions to the science of reading and dyslexia. Fellow of the British Academy, the Academy of Medical Sciences and the Academy of Social Sciences. Appointed CBE for services to science and the understanding of dyslexia in 2016.