Developing, monitoring, and reporting of fidelity in aphasia trials: core recommendations from the collaboration of aphasia trialists (CATs) trials for aphasia panel

ABSTRACT

Background

Developing, monitoring, and reporting of fidelity are essential and integral components to the design of randomised controlled trials (RCTs) in stroke and aphasia. Treatment fidelity refers to the degree to which an intervention is delivered as intended and is directly related to the quality of the evidence generated by RCTs. Clear documentation of treatment fidelity in trials assists in the evaluation of the clinical implications of potential benefits attributed to the intervention. Consideration of the implementation requirements of a research-based intervention as intended in a clinical context is necessary to achieve similar outcomes for a clinical population. Despite this, treatment fidelity is rarely reported in RCTs of aphasia intervention.

Aim

To describe fidelity strategies and develop core recommendations for developing, monitoring, and reporting of fidelity in aphasia intervention RCTs.

Scope

Relevant conceptual frameworks were considered. The Behaviour Change Consortium comprehensive framework of fidelity was adopted. It includes five areas: study design, training providers, delivery of treatment, treatment receipt, and treatment enactment. We explored fidelity in RCTs with a range of complex aphasia interventions (e.g., ASK, Big CACTUS, COMPARE, FCET2EC, POLAR, SUPERB, and VERSE) and described how different trial design factors (e.g., phase of trial, explanatory vs. pragmatic, number and location of sites, and number and type of treatment providers) influenced the fidelity strategies chosen. Strategies were mapped onto the five areas of the fidelity framework with a detailed exploration of how fidelity criteria were developed, measured, and monitored throughout each trial. This information was synthesised into a set of core recommendations to guide aphasia researchers towards the adequate measurement, capture, and reporting of fidelity within future aphasia intervention studies.

Conclusions/Recommendations

Treatment fidelity should be a core consideration in planning an intervention trial, a concept that goes beyond treatment adherence alone. A range of strategies should be selected depending on the phase and design of the trial being undertaken and appropriate investment of time and costs should be considered.

KEYWORDS:

• Aphasia
• trials
• fidelity
• adherence
• recommendations

Introduction

This paper is part of a special series of papers in Aphasiology on Methodological Issues in Aphasia Trials. The series comprises tutorial-type papers with core recommendations for aphasia intervention studies and RCTs. The series is guest edited by Professor Katerina Hilari, Dr Caterina Breitenstein, Dr Erin Godecke, Dr Helen Kelly, and Professor Miranda Rose on behalf of the Trials for Aphasia Panel of the Collaboration of Aphasia Trialists https://www.aphasiatrials.org/

Treatment fidelity refers to the “methodological strategies used to monitor and enhance the reliability and validity of behavioural interventions” (Bellg et al., 2004, p. 443) and is becoming increasingly recognised as an important quality marker for clinical trials (Bellg et al., 2004; Mowbray et al., 2003; Walker et al., 2017). Reporting fidelity can assist researchers to identify the essential elements that help make a treatment work (Carroll et al., 2007). Demonstrating fidelity is an important component in determining an intervention’s replicability, efficacy, and effectiveness.

Fidelity is a multifaceted concept that has evolved over time (Bellg et al., 2004; Carroll et al., 2007). Early definitions of fidelity focused on treatment delivery, which Moncher and Prinz (1991) separated into issues related to treatment integrity and differentiation. Treatment integrity relates to whether a treatment is carried out as intended with adequate levels of adherence and competence (Perepletchikova et al., 2007); treatment differentiation refers to whether one treatment differs from another (Kazdin, 1986). Treatment integrity and, in particular, adherence to the protocol have historically been considered as the main factors demonstrating sufficient fidelity (Carroll et al., 2007; Mowbray et al., 2003; Orwin, 2000; Walker et al., 2017). Adherence to a treatment protocol is the minimum standard for fidelity, referring to the degree to which providers deliver the intervention as specified in the manual (Carroll et al., 2007; Perepletchikova et al., 2007). Treatment receipt and enactment were later added to expand further the definition of fidelity (Lichstein et al., 1994). Treatment receipt refers to a participant’s ability to understand and perform the treatment skills (demonstrated, e.g., in pre-post tests, homework, and self-practice logs), while treatment enactment refers to a participant’s ability to independently use learnt treatment skills in relevant daily-life settings (demonstrated, e.g., in direct observation, self-report, or follow-up discussion).

The Treatment Fidelity Workgroup of the National Institutes of Health Behaviour Change Consortium (BCC) reviewed existing definitions and treatment fidelity strategies and developed recommendations for consistent reporting (Bellg et al., 2004). The group identified two other areas related to fidelity: study design, which refers to methods that ensure a study can adequately test the hypotheses under investigation; and training providers, which are methods that ensure treatment providers are satisfactorily trained to deliver the treatment. These five areas of fidelity (study design, training providers, delivery of treatment, receipt of treatment, and enactment of treatment skills) will be used and referred to throughout this manuscript. The BCC workgroup has further described and refined methods for assessing, monitoring, and enhancing treatment fidelity for all five areas (Borelli et al., 2005; Borrelli, 2011).

Monitoring fidelity is important for maintaining the internal and external validity of a study (Moncher & Prinz, 1991). Importantly, in terms of internal validity, fidelity results may help explain negative, null, or ambiguous findings (Hohmann & Shear, 2002; Resnick et al., 2005). In the absence of fidelity, it is difficult to determine whether a null trial was driven by poor adherence to a specified protocol or ineffectiveness of the treatment itself (Carroll et al., 2007; Perepletchikova et al., 2007). Moreover, monitoring fidelity can help decrease deviations from the treatment protocol, including any instances of therapist or programme drift (Mowbray et al., 2003). Measuring drift is important to show provider skills are maintained consistently across the trial with no change or decay from the beginning to the end of treatment.

In terms of external validity, the benefits of considering and evaluating fidelity include an opportunity to identify or confirm the active ingredients of treatment that contribute to efficacy findings (Bellg et al., 2004; Moncher & Prinz, 1991). Active ingredients refer to intervention-specific components serving as key levers of change (Abry et al., 2015). Fidelity monitoring enables researchers to replicate the findings of the study and make comparisons across other similar studies, which is important for maintaining external validity. Finally, the lack of standardisation of provider training and monitoring of adherence to treatment protocols will impede study replication (Mowbray et al., 2003; Resnick et al., 2005).

When designing a fidelity evaluation, it is essential to consider the phase and purpose of the trial. In 2000, The Medical Research Council (Campbell et al., 2000) described a series of phases in the evaluation of complex interventions ranging from pre-clinical (i.e., laboratory animal studies) and early studies on the safety and modelling components (phase I) to preliminary evaluation needed to identify key components and test the feasibility of an intervention (phase II), to comparing the intervention to an appropriate alternative (phase III) and, lastly, testing whether an intervention works in everyday practice and gains are maintained in the long term (phase IV).

The purpose of the trial—explanatory or pragmatic—should also be considered. An explanatory trial determines the efficacy of an intervention in tightly controlled conditions to establish whether the intervention “can” work under ideal conditions (Zwarenstein et al., 2008). In contrast, pragmatic trials determine the effectiveness of an intervention if it were implemented in routine clinical practice, by asking “does it work in the ‘real world’?”; the latter can often result in a more flexible application of the intervention with a more inclusive sample and multiple providers as well as individual adjustments of the therapeutic approach (Zwarenstein et al., 2008). Where a trial sits on the explanatory–pragmatic continuum can be determined using the PRagmatic Explanatory Continuum Indicator Summary 2 (PRECIS-2; Loudon et al., 2015).

The recommended practices for ensuring and monitoring fidelity proposed by the BCC are more applicable to explanatory than to pragmatic trial designs. Detailed recommendations for ensuring fidelity in pragmatic trials are yet to be published. However, as more pragmatic trials focus on interventions closer in nature to routine care evaluation, fidelity frameworks from the implementation literature, such as the Conceptual Framework for Implementation Fidelity (Carroll et al., 2007), have been utilised (Masterson‐Algar et al., 2014; J. Palmer et al., 2019). In pragmatic, routine care trials, it is essential that strategies to enhance fidelity, such as provider training and relaying fidelity evaluation findings back to providers, do not go beyond that which can be delivered in routine clinical practice outside of a trial (Miller & Rollnick, 2014). Implementation fidelity models place greater focus on measuring the fidelity, with strategies to ensure fidelity needing to be built into the intervention itself, rather than relying on external trial processes (Carroll et al., 2007).

From development to reporting, fidelity should be an integral component of intervention trial design (Walker et al., 2017). Despite this, treatment fidelity procedures were not included within international clinical trial reporting guidelines such as the Consolidated Standards of Reporting Trials (CONSORT 2010, Eldridge et al., 2016) until the Template for Intervention Description and Replication (TIDieR) extension (Hoffman et al., 2014).

In stroke rehabilitation and aphasia studies, fidelity is rarely reported. For example, in published stroke rehabilitation trials during 2015, fidelity was reported for fewer than 10% of trials, with most studies addressing a single aspect of fidelity such as adherence (Walker et al., 2017). In post-stroke aphasia intervention studies, the measurement and reporting of fidelity is still in its infancy. A review of 149 studies published between 2002 and 2011 revealed that only 14% of studies reported on fidelity (Hinckley & Douglas, 2013). Systematic reviews of specific interventions for people with post-stroke aphasia have highlighted that fewer than 27% of studies reported fidelity (Cherney et al., 2008, 2013; Faroqi-Shah et al., 2010; Purdy et al., 2019; Rose et al., 2013; Simmons-Mackie et al., 2010, 2016). More recently, 21% of aphasia RCTs from 2012 to 2017 explicitly reported fidelity (Brogan et al., 2019). In these 42 aphasia RCTs, only one study addressed all five fidelity areas (Marshall et al., 2016). Approximately half the studies addressed either training providers or delivery of treatment (Brogan et al., 2019).

In designing a fidelity evaluation plan, criteria for fidelity assessment need to be clearly identified and specified a priori (Mowbray et al., 2003). This process involves specifying the active ingredients of a treatment and defining “standard operating procedures” to ensure the treatment is delivered as planned (Moncher & Prinz, 1991). A fidelity protocol should include the training and ongoing supervision of providers and development of a treatment manual to help minimise provider or therapist drift (Hildebrand et al., 2012; Moncher & Prinz, 1991). Further criteria to measure fidelity should also be established; these criteria may be structural (e.g., the planned frequency and intensity of contact), which require less subjective judgements, or process-related (e.g., staff–client or client–client interactions), which are more subjective and often based on observation (Mowbray et al., 2003). Direct observation by trained and reliable raters of treatment sessions using a prescribed list of criteria (or behaviours) is considered the gold standard of fidelity measurement (Hart, 2009; Kaderavek & Justice, 2010; Markowitz et al., 2000; Moncher & Prinz, 1991; Mowbray et al., 2003). In their review of aphasia studies, Hinckley and Douglas (2013) report that about half of the studies (13/21) checked adherence by having raters review 10–20% of videotapes of sessions for the presence of target behaviours.

The TIDieR extension to the CONSORT 2010 statement is recommended to encourage more transparent reporting of treatments. Not only does this template advocate for a complete and detailed description of an intervention (e.g., materials, procedure, how, where, when, and how much), but fidelity is also a core part. Two items (11 and 12) explicitly refer to planned and actual fidelity strategies undertaken in an intervention trial. These items have since been matched to the BCC’s five areas of fidelity (Item 11: study design, training providers, delivery of treatment; Item 12: receipt of treatment, enactment of treatment skills). The actual outcome of planned fidelity strategies is less commonly reported in aphasia intervention trials (Brogan et al., 2019).

While fidelity monitoring and reporting may not have been historically common practice in aphasiology, more recently, the field has become increasingly aware of the importance of the development, implementation, evaluation, and reporting of fidelity strategies. This paper aims to provide examples of how fidelity was addressed across recent aphasia RCTs of different types and scales and build on existing frameworks to develop core recommendations for developing, monitoring, and reporting of fidelity.

Scope/Methods

To explore the measurement of fidelity and strategies used, seven RCTs for people with post-stroke aphasia from four countries (Australia, Germany, UK, USA) were examined. Considering the latest review of fidelity in aphasia RCTs covering trials up to 2017 (Brogan et al., 2019), we specifically included RCTs that were published or ongoing (with a published protocol) in the period from 2017 to the present. The trials needed to have a detailed description of planned fidelity as a minimum to be used as exemplars of current practice in fidelity monitoring and reporting. The selected studies comprised four explanatory (phases II and III) and three pragmatic (phases III and IV) trials. Explanatory trials included one feasibility and three definitive trials. The feasibility trial was of a peer-befriending intervention for people with aphasia and low levels of distress when discharged from hospital and active interventions cease (typically <6 months post-stroke) (SUPERB, Hilari et al., 2019). The three definitive trials were intensive aphasia therapy in the very early weeks post-stroke (VERSE, Godecke et al., 2021); multi-modal and constraint-induced aphasia therapy in the chronic phase (>6 months) post-stroke (COMPARE, Rose, Copland et al., 2019); and aphasia language therapy more than 12-months post-stroke (POLAR, Spell et al., 2020). Pragmatic trials included a psychosocial intervention to prevent depression in the early months post-stroke (ASK, Worrall et al., 2016), self-managed computerised speech and language therapy more than 4-months post-stroke (Big CACTUS, Palmer et al., 2020; R.2019), and intensive “agreed best practise” aphasia therapy starting at least 6-months post-stroke (FCET2EC, Breitenstein et al., 2017). Further information about trial design, sample size, and intervention characteristics is provided for each trial in Table 1.

Table 1. Study and intervention characteristics of aphasia intervention trials.

	POLAR	COMPARE	ASK	SUPERB	VERSE	FCET2EC	Big CACTUS
Study reports	Spell et al. (2020)	Rose et al. (2019); Rose et al. (2019)	Worrall et al. (2016); Carragher et al. (2019)	Hilari et al. (2019); Hilari et al. (2021); Behn et al. (2021)	Godecke et al. (2016); Godecke et al. (2021); under submission)	Baumgaertner et al. (2013); Breitenstein et al. (2017)	Palmer et al. (2020) Harrison (2019)
RCT phase and type*	Phase II, explanatory, multicentre, open-label, blinded endpoint, crossover RCT	Phase III, explanatory, multicentre, open-label, blinded endpoint, parallel group RCT	Phase III, pragmatic, superiority multicentre, cluster, double-blind, parallel group RCT	Phase II, explanatory, feasibility, multicentre, single-blind (intervention arm)/ double-blind (control arm), parallel group RCT	Phase III, explanatory, multicentre, open-label, three arm, blinded endpoint, parallel group RCT	Phase III/IV, pragmatic multicentre, open-label, blinded end point, parallel group RCT	Phase IV, pragmatic, superiority, multicentre, three arm, single-blind, parallel group RCT
Actual/target sample size	120 (target)	216 (actual)	344 (target)	56 (actual)	246 (actual)	158 (actual)	278 (actual)
Number of sites/setting	2 university clinics/community	Urban community facilities and University clinics	20 clusters (health regions)	5 Hospitals + linked community services; 13 GP practices	17 acute care hospital; 45 subacute hospital and home	19 inpatient and outpatient centres; day-care treatment facilities; hospitals; independent SLTs	21 community sites
Aphasia phase	≥12 months	>6 months	<6 months and study participation continued to 12-months post-onset	<6 months	<15 days	≥6 months	>4 months
Intervention and control arms	2 arms: semantically focused therapy using 3 approaches vs.phonologicallyfocused therapy using 3 approaches	3 arms: usual care CIAT Plus M-MAT	2 arms: ASK psychosocial intervention vs. attention control (secondary stroke prevention information)	2 arms: usual care + peer-befriending vs. usual care control	3 arms: usual Care + 15–20 hours of standard ward-based aphasia therapy vs. usual care + 15–20 hours prescribed therapy (targeting spoken output) vs. usual care control	2 arms: intensive agreed best practice + self-managed (computerised) language exercises vs. waiting list control	3 arms: usual care + self-managed computerised speech and language therapy vs. usual care + attention control (puzzle books and phone calls) vs. usual care control
Provider	Speech and language therapist	Speech and language therapist	Speech and language therapist	Person with aphasia	Speech and language therapist	Speech and language therapist	Self-managed with support from speech and language therapist and volunteer/ assistant

^*See glossary of terms

Note. POLAR = Predicting Outcomes of Language Rehabilitation; COMPARE = Constraint-induced Or Multi-modal Personalised Aphasia Rehabilitation; ASK = Action Success Knowledge; SUPERB = SUpporting wellbeing through PEeR Befriending; VERSE = Very Early Rehabilitation in SpEech; FCET2EC = From Controlled Experimental Trial to Everyday Communication; CACTUS = Clinical and cost-effectiveness of Aphasia Computer Treatment vs. Usual Stimulation or attention control long-term post-stroke; CIAT-Plus = Constraint-Induced Aphasia Therapy Plus; M-MAT = Multi-modality Aphasia Therapy.

To select studies for inclusion and describe the fidelity strategies of each trial, we considered peer-reviewed academic publications, information in the public domain including trial protocols, fidelity monitoring plans, and treatment manuals, and sought additional information from study authors. First, studies needed to have provided information on fidelity in either peer-reviewed publications and/or other forums in the public domain as above. Then, the study authors of each trial rated their own study and whether they had “met”, “partially met”, or “not met” the fidelity practice requirements according to treatment fidelity strategies within the five areas of fidelity proposed by Bellg et al. (2004): study design, training providers, delivery of treatment, receipt of treatment, and enactment of treatment skills. The study authors’ ratings were reviewed for inclusion of each study in this review by the first and last authors of the present paper to ensure fidelity strategies were used across at least four of the five areas.

Each trial was then rated by two different raters, who were experienced aphasia trialists and co-authors of this paper but who were independent of the RCT study they were rating. Raters reviewed the published documents and rated whether fidelity strategies were met in each RCT. The independent raters were not aware of the ratings provided by the study authors. Planned (if ongoing) and actual (if completed) fidelity strategies were rated. Each rater was required to independently review the published documents for fidelity strategies relevant to the specific area and make a judgement as to whether the trial had “met”, “partially met”, or “not met” the requirements. Resources to rate fidelity, that is, relevant papers (Bellg et al., 2004; Borelli et al., 2005; Borrelli, 2011) were given to the raters; but no training was provided.

To examine the inter-rater reliability of the independent raters, Gwet’s AC1 was used (Gwet, 2002, 2014). Unlike Cohen’s Kappa, Gwet’s AC1 can be calculated even when one rater’s scores are a constant/do not vary; Gwet’s AC1 is less susceptible to skew due to prevalence (Wongpakaran et al., 2013). A reliability coefficient of 0.81–1.00 is considered very good; 0.61–0.80, good; 0.41–0.60, moderate; 0.21–0.40, fair; and below 0.20, poor (Gwet, 2014). Statistical analyses were carried out using STATA version 15.

As disagreements in some of the ratings emerged, definitions of fidelity areas and fidelity strategies were discussed among the raters assigned to the same study to explore differences. Raters then met to reach an agreement on the rating by consensus. Based on the review of published documents for each trial, fidelity ratings, and comments on the seven trials by authors and raters, key fidelity strategies used across trials were described. Through a review of these strategies (planned or actual), the author team then agreed on a set of recommendations for which areas of fidelity should be considered essential or desirable across trial phases and the explanatory to pragmatic continuum.

Results

For inter-rater reliability of fidelity ratings, there was a fair to very good level of agreement between raters. There was fair agreement for the ASK (Gwets AC1 = 0.33 [95% CI, −0.10, 0.75], p = 0.12) and VERSE (Gwets AC1 = 0.37 [95% CI, −0.04,0.79], p = 0.07) trials. There was moderate agreement for COMPARE (Gwets AC1 = 0.42 [95% CI, 0.20, 0.82], p = 0.04), Big CACTUS (Gwets AC1 = 0.51 [95% CI, 0.13, 0.89], p = 0.01), and SUPERB trial (Gwets AC1 = 0.57 [95% CI 0.20, 0.95], p = 0.005). There was very good agreement for the POLAR (Gwets AC1 = 0.84 [95% CI, 0.60, 1.00] p < 0.001) and FCET2EC (Gwets AC1 = 0.86 [95% CI, 0.63, 1.00], p < 0.001) trials. Overall, across all raters and trials, there was 63% agreement (ranging from 43.8% to 87.5% agreement). However, when the categories “partially met” and “met” were collapsed into one, the agreement increased to 84% (ranging from 75% to 100%).

Table 2 shows the final ratings agreed for each area of fidelity and the extent to which each component was “met”, “partially met”, or “not met”. Training providers and delivery of treatment were addressed more completely than other areas of fidelity. Treatment receipt and enactment were addressed less well.

Table 2. Description of fidelity strategies and number of strategies rated as met, partially met, or not met in each trial.

	Planned fidelity			Actual fidelity
	POLAR	ASK	COMPARE	SUPERB	VERSE	FCET2EC	Big CACTUS
Fidelity report	Spell et al.d(2020)	Carragher et al. (2019)	Rose et al. (2019)	Behn et al.(2021)	Godecke et al. (under submission)	Breitenstein et al. (2017)	Palmer et al. (2020);Harrison (2019)
Study design
Ensure same treatment dose within conditions	Met	Partially met	Met	Met	Met	Partially met*	Not met
Ensure equivalent dose across conditions	Met	Partially met	Partially met	Partially met	Not met	Met	Not met
Plan for implementation setbacks	Partially met	Partially met	Partially met	Met	Met	Partially met	Partially met
Training providers
Standardise training	Met	Met	Met	Met	Partially met	Met	Partially met
Ensure provider skill acquisition	Met	Met	Met	Met	Partially met	Met	Partially met
Minimise therapist drift	Met	Partially met	Partially met	Met	Met	Met	Partially met
Accommodate provider differences	Partially met	Met	Partially met	Met	Partially met	Met	Partially met
Delivery of treatment
Control for provider differences	Met	Met	Met	Met	Met	Partially met	Partially met
Reduce differences within treatment	Met	Met	Met	Met	Met	Met	Partially met
Ensure adherence to protocol	Met	Met	Met	Met	Met	Met	Partially met
Minimise contamination between conditions	Met	Met	Met	Met	Met	Met	Partially met
Receipt of treatment
Ensure participant comprehension	Met	Met	Partially met	Met	Met	Met	Partially met
Ensure participant ability to use cognitive skills	Met	Not met	Not met	Met	Partially met	Met	Partially met
Ensure participants ability to perform behavioural skills	Met	Not met	Met	Partially met	Partially met	Met	Partially met
Treatment enactment
Ensure participant use of cognitive skills	Not met	Not met	Partially met	Partially met	Not met	Partially met	Met
Ensure participant use of behavioural skills	Not met	Not met	Partially met	Partially met	Not met	Met	Met

* In FCET2EC, the same treatment dose across participants was not an option as it was an ethical requirement to provide the maximum amount of therapy funded by the individual’s health insurance plan. For the first (and most relevant) follow-up assessment after three weeks though, dose was highly comparable across participants within the intervention condition (at least 80% of the prespecified dose).

POLAR = Predicting Outcomes of Language Rehabilitation; COMPARE = Constraint-induced Or Multi-modal Personalised Aphasia Rehabilitation; ASK = Action Success Knowledge; SUPERB = SUpporting wellbeing through PEeR Befriending; VERSE = Very Early Rehabilitation in SpEech; FCET2EC = From Controlled Experimental Trial 2 Everyday Communication; CACTUS = Clinical and cost-effectiveness of Aphasia Computer Treatment vs. Usual Stimulation or attention control long-term post-stroke.

There were subtle and distinct differences across all trials as to the strategies employed for each fidelity area according to the type and phase of the trial. Below is a summary of key strategies used across the seven trials (Table 3). It should be noted that fidelity strategies were either planned for ongoing trials at the time of writing this manuscript (i.e., POLAR, ASK, and COMPARE) or actual reported for completed trials (i.e., SUPERB, VERSE, FCET2EC, and Big CACTUS). It is noted that as Big CACTUS was a pragmatic trial, which aimed to monitor rather than ensure fidelity, many areas are marked as partially met, given the focus of the Bellg framework on ensuring fidelity. Instances of extended practice are identified to highlight examples where a trial went beyond the expectations of fidelity monitoring detailed by Bellg et al. (2004).

Table 3. Example strategies for each area of fidelity across recent aphasia RCTs.

	Rationale	Example strategies
Study Design	Ensures a study can adequately test its hypotheses in relation to underlyingtheory and clinical processes	Specified theory underpinning treatment and essential components identified. Record the length, number, and frequency of all contacts including practice and homework sessions (in all trial arms) Videorecorded sessions to independently assess for adherence Study-specific record sheet or daily therapy logs Supervision and training of providers Train more providers than required Allow additional time for recruitment of participants and intervention to manage setbacks
Training providers	Ensures treatment providers have been satisfactorily trained to deliverintervention	Same or certified trainers with set qualifications and/or experience levels Training material described Standardised training Length of training recorded Videorecorded training and/or treatment sessions to independently assess providers skills and competence Feedback to providers Knowledge quiz to providers at regular intervals Regular provider face-to-face supervision, site visits, or telephone calls Booster or refresher training or support Predetermined acceptable level of protocol adherence for reported treatment sessions
Delivery of treatment	Monitor and improve the delivery of the intervention so that that the protocol is delivered as intended	Daily therapy logs or visit records Monitor information recorded by computer software Standardisation of intervention elements to participants Recording adverse events, complaints, problems, or protocol deviations Videorecorded intervention sessions and independently assess for adherence with fidelity checklist Audio or videorecorded sessions Qualitative participant interviews Participant questionnaires of their experiences Supervision and feedback to providers at regular intervals or when required Standardised intervention manual and/or scripted intervention protocol Providers deliver a single intervention Providers do not liaise with other conditions or trial arm Different intervention conditions occur in different locations Close monitoring of intervention or services received by all trial arms Study participants in control arm explicitly not aware of intervention (if feasible)
Treatment receipt	Monitor and improve the ability of participants to understand the intervention and perform the skills being taught in intervention sessions (behavioural and cognitive)	Providers have skills or been trained to facilitate communication Inclusion and exclusion criteria ensure participants have skills to participate Incremental or stage structure to learning skills Review of videotaped intervention sessions, daily therapy logs, or homework exercises Ratings of goal achievement Monitoring functional tasks, e.g., conversation and group performance
Treatmentenactment	Monitor and improve the ability of participants to perform the skills being taught in real-life settings	Qualitative participant interviews Opportunities to practice trained items/behaviours in functional communication tasks, e.g., conversation and self-reported outcomes or behavioural measures

Strategies to address study design

In planning a treatment trial, the theory underpinning the treatment should be described and essential components or active ingredients identified. While this was completed across all trials, some trials did this more explicitly in publications focused on treatment fidelity (e.g., ASK, SUPERB, and VERSE). A number of trials also did pilot studies to explore the active ingredients in depth (e.g., ASK, Big CACTUS, COMPARE, SUPERB, and VERSE).

To ensure fidelity, key issues to consider when planning a study are dose within and across conditions and strategies to address implementation setbacks (Bellg et al., 2004). To ensure and monitor the same treatment dose within conditions, all trials planned to or recorded the length and number of all contacts. For trials with an independent practice component in the intervention arm, information was planned or recorded on the number and length of practice sessions (e.g., Big CACTUS, COMPARE, and FCET2EC). An example of extended practice in monitoring dose was in the VERSE trial, where participants in all trial arms had a proportion of treatment sessions videorecorded and independently assessed for overall treatment protocol adherence.

The data recorded on length and number of contacts were also used to ensure equivalence of dose across conditions. Other strategies were also employed, such as recording any speech and language therapy and other services received by participants including those in the control arm (e.g., ASK, Big CACTUS, COMPARE, FCET2EC, SUPERB, and VERSE); and having participants document therapy activity using a study-specific diary in the control arm or have providers keep a daily therapy log for those in the intervention arm (e.g., COMPARE and FCET2EC). In some cases, ensuring the same treatment dose within and equivalence of dose across conditions is not possible or desirable. For example, pragmatic trials in routine care, where there may be variability in how much therapy provision is funded by healthcare providers (e.g., FCET2EC) or for self-managed interventions where the aim is to increase the dose of therapy in the intervention arm only (e.g., Big CACTUS, and VERSE).

To manage setbacks, trials described careful ongoing monitoring through supervision and training (e.g., ASK, COMPARE, FCET2EC, POLAR, and SUPERB), including training more providers than were required (e.g., FCET2EC and VERSE) and allowing additional time to stagger recruitment of participants, training, and/or completion of intervention (e.g., Big CACTUS, COMPARE, SUPERB, and VERSE).

Strategies to address training providers

Training providers refers to the training of treatment providers to ensure that they can adequately and consistently deliver the intervention. Standardisation of training is a strategy described in all seven trials and includes the use of the same and/or certified trainers, set training materials, and/or a clear description of the length of training. An example of extended practice was having the training sessions videotaped and independently rated against a fidelity checklist (e.g., SUPERB).

To demonstrate that researchers ensured or monitored the acquisition of skills by providers and accommodated for differences between them, provider training included opportunities for discussion and questions. Specific strategies included video recording to ascertain a provider’s understanding of the training content followed by monitoring of therapy sessions, which were also recorded (e.g., COMPARE), review of video recording for each provider’s first intervention sessions (e.g., ASK) or a selection of treatment sessions (e.g., VERSE), feedback given after a session delivered in the first week of treatment (e.g., POLAR), independent rating of videotaped training sessions for skill acquisition (e.g., SUPERB), or a quiz to evaluate the knowledge of providers (e.g., Big CACTUS).

To minimise provider/therapist drift, a range of strategies was used, including regular, scheduled supervision (e.g., FCET2EC, POLAR, and SUPERB) or refresher supervision and training as necessary (e.g., ASK, COMPARE, and VERSE). To monitor drift, strategies included monitoring written therapy documentation, which underwent quality assurance checks and at least weekly telephone conversations with providers (e.g., FCET2EC), or giving providers a quiz at 5, 10, and 15 months post-training to check provider knowledge and what they had understood, internalised, and were applying during the trial (e.g., Big CACTUS). Further examples involved a review of videotaped intervention sessions to give feedback to providers to ensure skill acquisition (e.g., ASK, COMPARE, POLAR, SUPERB, and VERSE) and regular site visits (e.g., VERSE).

To accommodate for provider differences, training was given to all providers regardless of existing qualifications and/or experience levels (e.g., Big CACTUS, COMPARE, FCET2EC, and VERSE). Small group training was reported to ensure the individual needs of providers, who were people with aphasia, were met (e.g., SUPERB). In addition, a predetermined clinically acceptable level of protocol adherence for treatment sessions was reported to ensure all providers were consistent in their delivery (e.g., ASK, COMPARE, VERSE, and FCET2EC).

Strategies to address delivery of treatment

Delivery of treatment refers to strategies that determine if the intervention is delivered as intended. To control for provider differences, how the treatment was delivered by individual providers was monitored in different ways, including daily therapy logs or visit records (e.g., COMPARE, SUPERB, ASK, FCET2EC, and VERSE), documentation of therapy choice (e.g., Big CACTUS), standardisation of training for key therapeutic elements such as the rate of presentation of materials and therapeutic techniques (e.g., COMPARE, POLAR, and FCET2EC), and recording adverse events, complaints, and/or problems reported (e.g., COMPARE, FCET2EC, SUPERB, VERSE, and ASK). Ratings of session adherence help to control for differences between treatment providers (e.g., ASK, COMPARE, POLAR, SUPERB, and VERSE). In SUPERB, where the providers of the peer-befriending intervention were people with aphasia, criteria for matching a provider with a participant were monitored and recorded. Examples of extended practice included qualitative interviews for a selection of participants at the end of the trial to assess provider perceptions and experiences of the intervention (e.g., SUPERB, VERSE, and ASK) or completion of a questionnaire by providers to describe differences in the therapeutic bond, and goal and task agreement (e.g., Big CACTUS).

To reduce differences within treatment, strategies included training on key therapeutic elements (e.g., COMPARE and VERSE), self or independent ratings of session adherence (e.g., ASK, COMPARE, POLAR, SUPERB, VERSE), supervision and feedback to providers (e.g., ASK, COMPARE, FCET2EC, POLAR, SUPERB, and VERSE), use of a standardised treatment manual (e.g., Big CACTUS, FCET2EC, SUPERB, and VERSE), and use of a scripted and continuously monitored intervention protocol (e.g., FCET2EC).

To ensure and monitor adherence to protocols, intervention sessions were monitored using videotaped sessions rated against fidelity checklists that contained key criteria (e.g., ASK, COMPARE, SUPERB, and VERSE) or using a combination of face-to-face and videotaped sessions (e.g., POLAR). These ratings were completed by an independent assessor in some studies (e.g., SUPERB, VERSE, and COMPARE). The proportion of face-to-face or videotaped intervention sessions rated overall ranged from 10% to 25% (e.g., ASK, COMPARE, POLAR, SUPERB, and VERSE) with two trials intending to rate all first treatment sessions (e.g., ASK and COMPARE). Information about the intervention dose was also recorded and monitored, including the number and length of intervention sessions with protocol deviations (e.g., ASK, COMPARE, FCET2EC, SUPERB, and VERSE); daily therapy logs or visit records (e.g., COMPARE, FCET2EC, SUPERB, VERSE, and ASK); and monitoring of information recorded by therapy software (e.g., Big CACTUS). In some trials, feedback was given to the providers about adherence during the course of the trial (e.g., ASK, COMPARE, FCET2EC, SUPERB, and VERSE).

A range of strategies were employed to minimise contamination between conditions. Providers were trained to not liaise with providers of other conditions (e.g., VERSE) and to give a single intervention (e.g., ASK and SUPERB). Each healthcare/community location was responsible for a specific intervention condition (e.g., COMPARE and ASK); study participants in the control arm were not explicitly aware of the intervention tested (e.g., SUPERB and ASK). In two trials, contamination was not expected as no treatment was given by the provider during the waiting period (e.g., FCET2EC) or was limited due to the requirements of the computer software used in the treatment (e.g., Big CACTUS). Extended practice included monitoring of all trial conditions to determine what intervention they received (e.g., Big CACTUS, COMPARE, FCET2EC, and SUPERB). In one trial, a planned review of videotaped sessions from all conditions to monitor the risk of contamination was planned but partially achieved (e.g., VERSE). Should contamination be found, a protocol deviation would be logged and provider retraining commenced (e.g., COMPARE).

Strategies to address treatment receipt

Treatment receipt assesses whether the participant can comprehend the intervention and use the skills being taught. Strategies in this area included more subtle methods, such as training the providers or having professionals (i.e., speech and language therapists) who already have skills in supporting communication to deliver the intervention (e.g., ASK, Big CACTUS, FCET2EC, POLAR, SUPERB, and VERSE) or specifying minimum comprehension or cognitive ability as a study inclusion criterion to ensure sufficient skills to take part (e.g., Big CACTUS, FCET2EC, and VERSE). The design of some interventions included incremental (or adaptive) levels or a stage structure whereby a participant needed to comprehend the intervention and use the cognitive skills being taught to be able to progress (e.g., Big CACTUS, COMPARE, FCET2EC, and VERSE). An example of extended practice was the review of videotaped intervention visits or treatment logs to monitor the degree with which a provider ensured a participant’s comprehension and had sufficient cognitive skills to carry out and benefit from the intervention (e.g., POLAR and SUPERB). In addition, providers reviewed completed homework exercises and monitored participant performance daily to ensure the participants had understood training instructions (e.g., FCET2EC).

To ensure a participant could perform the behavioural skills being taught, a range of strategies were used, including goal achievement (e.g., VERSE), functional communication ability measured by video recorded conversations (e.g., Big CACTUS), performance in group therapy sessions as monitored by therapy supervisors (e.g., FCET2EC), and an activity log that demonstrated the use of therapy skills completed between sessions (e.g., COMPARE).

Strategies to address treatment enactment

Treatment enactment evaluates whether the participant demonstrates the ability to “use the intervention” in real-life settings. “Using the intervention” means employing the cognitive and behavioural skills that were taught during the intervention. Compared to other fidelity areas, treatment enactment may be harder to capture and has received less attention in aphasia RCTs. Examples of strategies to use here include qualitative interviews of a purposive sample of participants to explore their experiences and the impact of the intervention on their performance (e.g., Big CACTUS and SUPERB), examining the extent with which the language learned was then used in everyday conversation with volunteers (e.g., Big CACTUS) or the use of self-reported outcomes or behavioural measures that probed the participants’ use of the taught skills (e.g., simulated everyday life communication scenarios such as role-plays in FCET2EC and functional communication measures in COMPARE).

Recommendations

Having reviewed the literature on fidelity standards (Bellg et al., 2004; Borelli et al., 2005; Borrelli, 2011; Carroll et al., 2007) and fidelity strategies employed in a sample of ongoing and recently completed aphasia RCTs, we propose a set of recommendations for aphasia researchers to appropriately address fidelity when planning, conducting, and reporting aphasia intervention trials.

• Fidelity monitoring should be seen as an integral part of all intervention trials (whether explanatory or pragmatic).

• Intervention trials should have a fidelity monitoring plan/protocol at the outset of the trial (Feely et al., 2018).

• Explanatory trials need to ensure fidelity; pragmatic trials need to monitor fidelity and statistically control for any effects of variable fidelity. Depending on the phase of a trial and the question of interest, some areas of fidelity need to be addressed to a greater or lesser extent. For example, treatment enactment may be a lower priority in early feasibility trials that do not evaluate efficacy and effectiveness.

• Study design: The intervention dose within and between all trial arms should be ensured/monitored including the control group. Active ingredients or essential components of the intervention should be clearly defined, described, and operationalised in a fidelity checklist (or list of protocol behaviours to be observed).

• Fidelity checklists should be created systematically, including developing an intervention framework, obtaining stakeholder feedback, and piloting the checklist to improve inter-rater reliability (Walton et al., 2020).

• Training providers: a detailed description is required regarding how providers are trained before the commencement of intervention. Providers should be clear about the active ingredients of the intervention (e.g., fidelity checklists should be shared with providers). Supervision and feedback to providers during the intervention should be specified to monitor adherence to the protocol.

• Delivery of treatment: adherence to intervention should be ensured or monitored using a range of data collection strategies including written data (e.g., daily records and activity logs) and/or observation of sessions (e.g., videorecorded sessions). Sessions should be rated for adherence. Researchers should aim to rate adherence for a high proportion of sessions, as feasible within their trial design. Rating 20% of sessions is considered optimal (Borrelli, 2011); when this is not feasible, we would recommend rating a minimum of 10% of sessions. A predetermined acceptable level of session adherence should be set (e.g., 100% of a specific set of behaviours; 80% of the general target behaviours). Failure to reach an acceptable level should be recorded as a protocol deviation and a management plan to mitigate future instances should be described.

• Treatment receipt: participants’ grasp of the information provided and ability to apply the trained behaviours should be assessed.

• Treatment enactment: participants’ application of the behaviours/skills to real-life settings should be assessed. Assessment methods may include questionnaires, self-reports, follow-up interviews, and telephone calls (Bellg et al., 2004; Borrelli, 2011) as well as conversation samples or assessment of target performances in non-therapeutic environments.

Discussion

The aim of this paper was to describe how fidelity has been addressed across recent aphasia RCTs and to develop core recommendations for fidelity in aphasia intervention trials. Historically, fidelity monitoring was not consistently embedded within aphasia intervention studies. Previous reviews of fidelity monitoring in aphasia intervention studies have found that less than half explicitly assessed adherence via video recordings of intervention sessions (Hinckley & Douglas, 2013), less than a quarter reported supervision of the intervention (Hinckley & Douglas, 2013) and in aphasia RCTs published prior to 2017 only around 50% reported on training providers and delivery of treatment (Brogan et al., 2019). The current paper demonstrates a shift in the field of aphasiology towards more consistent and comprehensive fidelity monitoring and reporting. Seven aphasia RCTs were reviewed, including trials of different phases (phases II to IV) and both explanatory and pragmatic trials. Encouragingly, four trials at least partially met all five areas of fidelity, with the remaining three trials at least partially meeting four areas. Such a finding indicates that fidelity is increasingly being considered in the planning and reporting of aphasia intervention trials. Furthermore, the interpretation of fidelity in aphasia RCTs has moved beyond treatment adherence, indicating progress from the historic focus of fidelity in stroke rehabilitation research (Walker et al., 2017). Based on the literature and the reviewed RCTs, we provided a detailed but not exhaustive description of potential fidelity strategies to use and developed initial recommendations to help guide researchers as to what areas of treatment fidelity to consider for a range of trial phases.

Pragmatic trials present a challenge to the Bellg et al. (2004) framework as they monitor rather than ensure fidelity. In such trials, firstly, any modifications to the intervention should be systematically documented, using, e.g., FRAME (Stirman et al., 2019), to understand their impact. Secondly, alternative frameworks of fidelity may also be appropriate such as that proposed by Carroll et al. (2007), which focuses on treatment adherence (i.e., content and dose) and the factors that may influence the degree with which the intervention is implemented in practice (i.e., comprehensiveness of policy description, strategies to facilitate implementation, quality of delivery, and participant responsiveness). However, the Carroll et al. framework appears to place less emphasis on the receipt and enactment of the treatment and conceptualises these areas as more related to participant responsiveness. To guide researchers in such instances, the fidelity strategies employed by two pragmatic trials included in this paper, i.e., FCET2EC and Big CACTUS, may be helpful.

Treatment receipt and enactment have been addressed in less than 12% of aphasia intervention trials (Brogan et al., 2019). All seven trials reviewed here at least partially reported on an element of treatment receipt and four reported on an element of treatment enactment. Both these areas of fidelity extend the focus beyond the provider to the participant, underlining that an intervention should not merely be provided, but also received and consecutively lead to a change in everyday life. These areas partially overlap with the assessment of treatment effectiveness, which is likely assessed beyond a trial’s fidelity plan. The fidelity domain of enactment continues to present challenges for definition, assessment, and reporting. Enactment may be difficult to assess because of the complex nature of communication (not a single behaviour to observe), the heterogeneity of aphasia, and because of limited or absent carer support to facilitate the use of taught skills in the home and community settings. Expert consensus is needed to determine whether other assessments completed during a trial (e.g., activity logs, conversation samples, and talk time) may be suitable strategies to measure enactment.

A challenge within this paper related to achieving agreement between raters in whether a requirement was met, partially met, or not met; with reliability coefficients ranging from fair to very good. A large proportion of disagreement related to whether a requirement was met or partially met. One difficulty may lie in the strategies proposed by Borelli et al. (2005) and Borrelli (2011), which were designed for health behaviour change trials where one specific behaviour is targeted (e.g., smoking). In aphasia, typically a range of behaviours are addressed as part of the intervention. However, there was also inconsistency in the raters’ interpretation of the criteria and strategies. Pilot rating of a study with discussion between assessors on criteria and rating discrepancies may help ensure a common understanding of criteria and higher agreement in the future.

Moreover, the recommendations proposed in the current paper were generated through discussion and consensus agreement within a group of experienced aphasia trialists interested in treatment fidelity. A systematic methodology such as an e-Delphi exercise may be considered in future research in this area to determine the most important aspects of fidelity assessment in aphasia rehabilitation. The focus of this paper has been on treatment fidelity, which is only one aspect of fidelity. Researchers may also want to consider fidelity of assessment, which has been reported for two of the trials comprised in the current paper (i.e., COMPARE, POLAR).

Conclusions

This paper describes how the field of aphasiology has progressed in its monitoring and reporting of treatment fidelity. Referring to a selection of ongoing or completed aphasia trials as exemplars, we highlight examples of fidelity strategies that represent good practice to guide planning of future stroke and aphasia intervention trials. Treatment fidelity should be considered as integral to trial designs in the evaluation of interventions. A range of strategies should be selected depending on the phase and design of the trial. Enhanced fidelity monitoring and reporting will support transparency of trial reporting, increase our confidence in the trial results, and ultimately improve clinical decision-making for the benefit of people living with aphasia.

Glossary of terms

https://www.aphasiatrials.org/design-planning/

Trial design

Parallel group: where participants are randomised to one of two or more treatments, which are compared

Crossover: main alternative design to the parallel group designs where all participants receive the same treatment(s), but the order in which they receive them depends on the arm to which they are randomly assigned.

Superiority: trials designed to detect the superiority of a treatment compared to other treatment(s), in contrast to trials that are designed to assess non-inferiority or equivalence.

Blinding/masking (safeguard strategy against bias)

Open label: where researchers, providers, and participants are all aware of the treatment assigned because blinding to treatment assignment is difficult or impossible. Frequently applied in combination with blinded endpoint evaluation (see below).

Single-blind: where either the researcher collecting data, the healthcare provider, or the participant is blind to the treatment assigned.

Double-blind: where both the researcher collecting the data and the participant are blind to the treatment assigned.

Blinded endpoint: An independent investigator or preferably an endpoint committee who was blinded to the group assignment (and preferably time of assessment in designs involving a baseline assessment) did the assessment of clinical recovery in the study (blind end point assessment).

Acknowledgments

ASK was funded by a grant from the National Health and Medical Research Council Grant (APP1060673); Big CACTUS was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme (12/21/01) and Tavistock Trust for Aphasia; COMPARE was funded by a National Health and Medical Research Council Grant (APP1083010); FCET2EC was funded by the German Federal Ministry of Education and Research (BMBF; identifier, 01GY1144) and the German Society for Aphasia Research and Treatment (GAB); POLAR was funded by National Institute on Deafness and Other Communication Disorders Grants P50 DC014664; SUPERB was funded by The Stroke Association, Psychological Consequences of Stroke – Priority Programme Award (PPA2015-03); VERSE was funded by the National Health and Medical Research Council Grant (APP1044973), The Tavistock Trust for Aphasia, and Edith Cowan University, Australia.

Disclosure statement

No potential conflict of interest was reported by the author(s).