The feasibility of measuring fidelity of implementation in parent-child interaction therapy: A clinician and parent fidelity study

Abstract

Purpose

Measuring fidelity of implementation in parent-child interaction therapy (PCIT) involves assessing the training delivered by clinicians and how parents implement the techniques with their children. The aim of this study was to determine the feasibility of measuring fidelity of implementation for a PCIT intervention designed for young children with Down syndrome.

Method

We applied a framework to measure dosage, adherence, quality, and participant responsiveness using a mixed methods approach with observational and interview data.

Result

Our results showed that clinicians delivered 94% of the planned dosage; they adhered to the goals of program and reached the quality criterion in 4/6 rated sessions. Parents described their ability to engage with the program and perceived that it changed how they interacted and communicated with their children. Parents were unable to collect dosage data, but did adhere to 7/9 of the targeted techniques and met the quality criterion on 6/9 of these. It was also possible to measure the children’s responsiveness scores when interacting with parents during the intervention.

Conclusion

This study revealed the opportunities and challenges that occur when measuring fidelity of implementation. There is a need to refine definitions of fidelity measures and to develop appropriate measurement tools so that a more consistent and useful framework can be used by speech-language pathologists (SLPs) to measure fidelity.

Keywords:

• Parent education
• intervention
• Down syndrome
• fidelity
• implementation science

Introduction

Parent-child interaction therapy

Children learn language through their interactions with parents and caregivers. It is now well established that how children learn language is shaped by: (a) the number of these interactions that take place daily; (b) parental responsiveness to the child’s communication; (c) the quantity and quality of language input; and (d) the use of language support techniques (Hart & Risley, 1995; Huttenlocher et al., 1991; Roberts & Kaiser, 2011). Parent-child interaction therapy (PCIT) refers to a group of language techniques that involve upskilling and training parents in how to maximise communication in their everyday interactions with their child, in order to promote speech, language, and communication development (O'Toole et al., 2021). The intervention is now recognised internationally as a valuable approach to remediating communication difficulties in young children (Law et al., 2019). Systematic reviews have found that parental responsiveness and children’s language skills, particularly expressive language, can be improved (Roberts et al., 2019; Roberts & Kaiser, 2011), however, the effect of PCIT is smaller for children with intellectual disability and receptive language difficulties compared to those with solely expressive language difficulties (Roberts & Kaiser, 2011; Te Kaat-Van Den Os et al., 2017). For example, a systematic review of PCIT for children with Down syndrome concluded that it improved the way parents interacted with their children, but had inconsistent and often limited effects on the children’s language (O'Toole et al., 2018). As the review was based on only three small studies of relatively low quality, the authors stated that there was insufficient evidence showing the effect of PCIT on the language skills of children with Down syndrome. Moreover, the review noted that the studies did not clearly describe the intervention procedures used or how the fidelity was monitored, which are factors that can moderate and mediate the effects of intervention (O'Toole et al., 2018).

Fidelity of implementation in parent-child interaction therapy

Fidelity of implementation refers to the degree with which an intervention is delivered as intended (Haring Biel et al., 2020; Lieberman-Betz, 2015) and measuring this helps to determine the relationship between the intervention procedures and the outcomes achieved. Documenting intervention dosage and how accurately it was implemented can also help to improve reliability and validity, so that others can adopt the same approach or extend the intervention to another context (Bellg et al., 2004; Haring Biel et al., 2020). The terminology proposed to describe fidelity of implementation in the field of interventions such as PCIT can vary (Barton & Fettig, 2013; Lieberman-Betz, 2015), and is further complicated by the argument, by some authors, that it needs to be measured on multiple levels. For example, Barton and Fettig (2013) use the term implementation fidelity to describe the degree to which the clinician implements practices to coach and support parents to implement the intervention and the term intervention fidelity to describe the parents’ implementation of the intervention techniques with their children. They argue that although both terms are used interchangeably, it is important to make distinctions between them. This is because indirect interventions such as PCIT are thought to have a cascading effect (Roberts et al., 2014) meaning that effective parental teaching and coaching by the clinician on how to use specific techniques (high implementation fidelity), leads to accurate and consistent parent implementation of the techniques (high intervention fidelity) in their daily interactions. This in turn leads to positive changes in the child’s communication development (Barton & Fettig, 2013; Lieberman-Betz, 2015; Roberts et al., 2014). Because of this cascading effect, measuring clinician fidelity is essential to ensure that training procedures are carried out as planned, as they will impact on parental capacity to use the techniques in the intervention. Similarly, measuring parent fidelity will indicate whether the parents are implementing the intervention as intended, which can in turn mediate child outcomes. On the other hand, in her review of fidelity in PCIT interventions, Lieberman-Betz (2015) uses the term fidelity of implementation to capture both clinician and parent fidelity. In this paper we use the term fidelity of implementation to cover both implementation and intervention fidelity, but will specify clinician fidelity when we refer to how clinicians implement the intervention with parents and parent fidelity when we discuss how parents implement the intervention with their children.

Lieberman-Betz (2015) argues that fidelity of implementation should be measured in terms of four factors at the level of the clinician and parent, which she defines as follows:

1. Dosage: The amount of intervention provided.

2. Adherence: Whether all prescribed elements of an intervention are delivered accurately.

3. Quality: How well or how frequently the intervention techniques are delivered.

4. Participant responsiveness: How participants respond to, or are engaged in, the intervention.

These factors are based on Dane and Schneider’s (1998) conceptualisation of fidelity of implementation. They argue that measuring fidelity of implementation needs to go beyond measuring adherence and include wider factors such as dosage, quality, and participant responsiveness in order to promote a broader examination of some of the key ingredients that effect change or could be improved for future iterations of an intervention (Lieberman-Betz, 2015).

Studies of fidelity of implementation in parent-child interaction therapy

In the field of PCIT interventions, few studies have included a description of fidelity of implementation. A systematic review of the effectiveness of parent-implemented interventions by Roberts and Kaiser (2011) included 18 studies, but observed that only five (28%) of these reported some aspect of fidelity of implementation and half did not provide a detailed description of the parent training practices. In addition, most studies failed to describe the techniques used to teach parents or how much training occurred. They argued that poor description of these aspects of fidelity meant that it was difficult to determine specific characteristics of the training that resulted in changes in the parent and child, and the factors that may have moderated these changes. A more recent systematic review of 73 PCIT interventions by Roberts et al. (2019) noted that only 36 (47%) reported any element of fidelity. As these systematic reviews were not specifically focused on fidelity of implementation, neither review investigated fidelity measures for clinicians and parents separately. However, a review of 24 PCIT intervention studies for children with disabilities completed by Barton and Fettig (2013) investigated clinician and parent fidelity features in more detail and found that while 19 studies (79%) reported intervention fidelity for parents, only seven (29%) described implementation fidelity for clinicians.

Most of the existing research in the area of fidelity of implementation in PCIT, has focused on adherence to the intervention (Carroll et al., 2007), but few studies have investigated dosage, quality of delivery, or participant responsiveness. In her review of 35 studies of PCIT for early communication difficulties, Lieberman-Betz (2015) noted low reporting practices across these elements. She found that 71% of studies (n = 25) reported dosage at the clinician level by documenting how many sessions of parent training were offered, as well as the treatment frequency and duration. However, very few (14%; n = 5) reported dosage at the parent level. Regarding adherence, only 12 studies (34%) reported whether the clinicians implemented the intervention in accordance with self-reported checklists or independent observer assessment. This was in contrast to 21 studies (60%) that measured whether parents accurately adhered to the intervention, mostly collected through recorded observations of parents implementing techniques. For measures of quality, seven studies (20%) reported this for clinicians, generally measured by a pre-determined criterion level to which they were expected to implement a technique or through an independent observer rating the quality of their technique use. For parents, 10 studies (29%) reported quality, which Lieberman Betz defined as a measure of the frequency with which parents used a technique or whether it was carried out to a pre-specified criterion level. Finally, participant responsiveness was measured by asking participants to make judgements about the acceptance and relevance of an intervention (Carroll et al., 2007) or by measuring their engagement with an intervention. At the clinician level, participant responsiveness measures how parents (as the recipients of the clinician training) respond to the intervention, while at the parent level it measures the children’s response to the intervention. Lieberman-Betz noted that 54% of studies (n = 19) reviewed provided information on parent responsiveness as measured through attendance data, satisfaction questionnaires, or interviews. Similarly, 43% of studies (n = 15) reviewed how the children responded to the treatment, either using parental report of child engagement or by measuring targeted child behaviours during parent-child interactions.

Based on the findings of these systematic reviews, it is evident that many interventions involving PCIT do not report detailed fidelity of implementation practices, particularly for parental dosage, clinician adherence, and quality at both parent and clinician level. It is important that intervention studies attempt to measure all aspects of fidelity in order to capture a comprehensive view of the moderators and mediators that may affect outcomes as well as any potential problems with the intervention delivery model (Carroll et al., 2007).

Aims of the current study

The current study set out to determine the feasibility of measuring fidelity of implementation at the clinician and parent level (for dosage, adherence, quality, and participant responsiveness as defined by Lieberman-Betz [2015]) in a community based PCIT program called the Little Explorers Early Intervention Programme (LEEIP) that was developed specifically for very young children with Down syndrome and their families. The following research questions were addressed:

1. Can clinicians’ fidelity of implementation in terms of dosage, adherence, quality, and participant responsiveness be measured when implementing the LEEIP program?

2. Can parents’ fidelity of implementation in terms of dosage, adherence, quality, and participant responsiveness be measured when implementing the LEEIP program?

Method

This study was completed in two phases. The first phase involved a consultation with the speech-language pathologist (SLP) interventionists in which the goals of the program were identified for the parent, child, and SLP. The second phase aimed to determine if it was possible to measure fidelity of implementation as described below.

Phase 1: Elaboration of the program session plans

The first phase involved capturing the goals of the Little Explorers Early Intervention Programme (LEEIP). The program was developed by SLPs who provide services for children with Down syndrome, and was targeted at those aged between 10 and 36 months and their families in order to respond to the collective and individual needs of this group. Similar to other PCIT programs, LEEIP strives to promote the prelinguistic and early language skills of children with Down syndrome while coaching their parents in communication and interaction techniques. The intervention goals aim to address weaknesses evinced by children with Down syndrome, such as joint attention and auditory memory, while capitalising on their strengths, such as visual learning through the use of gestures and key word signing (called Lámh in Ireland). Following a number of years of implementing and adapting the program, the SLPs who developed LEEIP wanted to create a manual for the program, so that other clinicians working with these children could implement it reliably. To accomplish this, they approached the first and last authors to discuss a researcher-clinician collaboration. Discussions centred on describing how the intervention was delivered in a transparent way in order to measure the aspects of fidelity of implementation that needed to be reported in the manual. For example, their initial session goals (Table I) needed to be elaborated on in order to make the goals more explicit. This involved detailing the techniques that the clinicians were aiming to use, such as explaining, modelling, and coaching, as well as specifying the goals for the parents and children involved (the triad in PCIT interventions; Table II). These plans could then be used as the gold standard against which adherence and quality aspects of fidelity of implementation could be measured (Kaderavek & Justice, 2010) and later facilitate the creation of a treatment manual, which was a desired outcome of the collaboration.

Table I. Pre-collaboration session plan.

Greetings	Goal	Techniques used and equipment needed
1. Hello song	Child to attend to Hello Bob song.	Bob puppet Lámh Modelling
2. Photo	Child to attend to their photo and photos of peers.	Individual photos
3. Where are you? song	Child to tolerate hand over hand cue for hand to chest action (here I am). Child to tolerate physical cue to shake Bob’s hand.	Bob puppet Lámh Modelling
4. Goodbye song	Child to attend to Goodbye song.	Bob puppet Lámh Modelling

Table II. Revised session plan for measuring adherence.

1. Greetings	Equipment Needed	Therapist Therapist will:	Parent goal: To facilitate child’s attention and participation in greetings routine Parent will:	Child goal: To attend and participate in greetings routine Child will:
1. Hello song	None	Explain parent and child goals of the activity Model slowed singing and leaving gaps (waiting) in the song for children to attend and participate, with one child from group Coach parents on how to use slowed signing with pauses with their child Provide feedback on parental use of the technique	Be positioned at child’s level (face-to-face with eye contact) Sing Hello song slowly Use key word signs while singing Wait for their child’s response Imitate any gesture/action/vocalisation/word (attempt)	Attend to parent singing Hello song Participate in singing Attempt/ produce Lámh signs Attempt/produce vocalisations/words
2. Photo	Large A4 individual photos of children Bags for each parent	Explain parent and child goals of the activity Model pulling Bob’s photo from bag, naming/pointing to Bob, and singing Where are you? song to Bob Model slowed singing and leaving gaps (waiting) in the song for children to attend and participate, with one child from group Coach parents on how to pause and wait for their child to respond Provide feedback on parental use of the technique	Encourage child to pull their photo from bag (use hand over hand if necessary) Point, name, and draw attention to picture Sing song slowly Use key word signs while singing (name, where, you) Wait for their child’s response Imitate any gesture/action/vocalisation/word (attempt)	Attend to their own photo Point to photo of self Participate in singing Attempt/produce Lámh signs Attempt/produce vocalisations/words
3. Goodbye song		Explain parent and child goals of the activity Model Goodbye song with slowed singing, leaving gaps (waiting) in the song for children to attend and participate Coach parents on how to use slowed signing with pauses with their child Provide feedback on parental behaviour	Be positioned at child’s level (face-to-face with eye contact) Sing Goodbye song slowly Use key word signs Wait for their child’s response Imitate any gesture/action/vocalisation/word (attempt)	Attend to parent singing Goodbye song Participate in singing Attempt/produce Lámh signs Attempt/produce vocalisations/words

In order to elaborate on the original goals of the program, we used the Teach-Model-Coach-Review (TMCR) procedure described by Roberts et al. (2014) to clearly describe the teaching and coaching techniques for clinicians. The four stages are: 1. Teach (explain the goal to the parent); 2. Model (model the technique with child); 3. Coach (give parent an opportunity to practice the technique); and 4. Review (provide constructive feedback). Following our discussion, the SLPs adapted the session plans, explicitly stating the goals for each member of the triad (Table II).

Phase 2: Measuring feasibility of implementation

The next phase involved determining whether it was feasible to measure fidelity of implementation with a group of children and their parents who were taking part in LEEIP.

Intervention program

The LEEIP intervention program runs over three terms in a 10 month period (September–June) and is partially subsidised by Down Syndrome Ireland. Each intervention term consisted of bimonthly group sessions (x4) and individual sessions (x2). During the group sessions, parents are coached in areas of early communication and language development, including: following the child’s lead, turn taking, waiting, and imitation; how to promote Lámh key word signing for vocabulary development; and speech sound play and practise. These techniques are practiced in functional activities that can be incorporated into everyday routines of the family. The targeted themes included bath time, mealtimes, morning routine and dressing, playtime and books, outdoors, home and family life. Each session is run twice as the repetition facilitates learning for parents and children.

Participants

Following ethical approval, parents who enrolled in the LEEIP program during 2018–2019 were invited to take part in the evaluation. All families were in receipt of private speech-language pathology services provided by a charitable organisation for people with Down syndrome. To be included, the children had to be between 10–18 months old, have English as their first language, and had not previously have taken part in the LEEIP program. Seven families enrolled, all agreed to take part in the evaluation, and their details are summarised in Table III. Parents provided written consent to take part in the research. Each child was allocated a pseudonym to uphold confidentiality. The parent participants were all mothers with university level education, from a white-Irish background. Six were aged between 30–40 years and one parent was between 40–50 years. At the time of the intervention, three mothers were on parental leave, three worked part-time, and one was on an extended maternity leave. The participating children were three boys and four girls and were aged between 10 and 17 months at baseline. Three children had two older siblings, two had one older sibling, one was an only child, and one had a younger infant sibling (born during the program). Most children had fluctuating mild-moderate hearing loss associated with middle ear infections at the time of testing. All children were also attending publicly available speech-language pathology services outside of the private service, although this was minimal for most. Intelligence Quotients ranged between 65 and 80 on the Bayley Cognitive Assessment (categorised as Extremely Low and Low Average respectively). All children had experienced varying degrees of hospitalisations ranging from 5 days to 4 months.

Table III. Summary of demographic information for participants and their mothers.

	Sophie^a	Jack	Conor	Daniel	Ellie	Grace	Luke
Age (months; days)	13m 15d	11m 28d	13m 15d	11m 2d	10m 25d	17m 7d	12m 24d
Gender	Female	Male	Male	Male	Female	Female	Male
Hearing	Moderate	Mild	Moderate	Mild	Mild/moderate	Mild	Within normal limits
Bayley composite score	65	80	75	65	75	60	65
Hospital duration	9 days	16 days	23 days	42 days	4 months	7 days	5 days
Mother’s working status	Full-time to part-time	Parental leave	Extended maternity leave	Part-time	Parental leave	Parental leave	Part-time

Note. ^aChildren’s pseudonyms.

Research team

The interventionist clinicians were two SLPs. One was a senior clinician with nine years’ experience as an SLP and three years’ experience running the LEEIP program. The other clinician had three years’ experience working as an SLP and had been implementing the LEEIP program for one year. This was their first experience in researching and measuring fidelity of implementation of the LEEIP program. The interventionist researcher was a qualified female SLP undertaking a postgraduate degree and she was supervised by two female academic SLPs with over 20 years’ experience in delivering and researching PCIT interventions (first and last author on this paper).

Procedure

This study was part of a multiple-baseline single case series that measured parent and child outcomes from the LEEIP programs (Cronin, 2020). There were six assessment points completed: two pre-intervention baseline assessments, one assessment at the end of each term of LEEIP, and one three-month follow up assessment. These assessment sessions were completed by a researcher who was blind to the purpose, aims, goals, and content of the LEEIP intervention. Assessments took place in the family home of the participants. As this was a community based project that parents had to pay for, we could not insist that families attend all three terms of LEEIP in order to take part in the research. However, we collected data from all participants where possible regardless of how many intervention sessions they took part in. These assessments included parent-reported vocabulary items, standardised tests of language, and analysis of recorded parent-child interactions. While data from these assessments is not the main focus of the current paper, some of the data collected was used in our measure of fidelity of implementation as outlined below.

Research question 1: Clinician fidelity of implementation

Dosage

Dose was a measure of the length and frequency of the sessions, in line with the definition of Lieberman-Betz (2015). Before the first term commenced, parents were invited to attend a one-hour workshop where the goals and techniques of the LEEIP program were explained. Group sessions were fortnightly and 45 minutes in duration. Individual sessions were 30 minutes and scheduled halfway through an intervention term (after two group sessions) and at the end of a term. Therefore the total number of intervention sessions offered to each family over the three terms was 18, which included nine hours of group intervention and three hours of individual sessions.

Adherence

Clinician adherence was reviewed by the first and last authors by observing a recording of the first group session of each theme and comparing the accuracy of the clinicians’ behaviour (e.g. explain, model, coach, etc.) to that outlined on the intended session plans (Table II). We provided immediate feedback to the clinicians in order to increase subsequent adherence for the second iteration of this session. We did not seek consent to record the individual sessions as the SLPs did not wish for this as they felt that these discussions were often of a sensitive nature.

Quality

We measured quality by quantifying the number of times the clinicians used the parent-training techniques in the recordings compared to that specified on the session plans (Table II). We set a pre-defined criterion level of 80% to indicate mastery of technique delivery, which was in line with that specified in other studies (Kaiser & Roberts, 2013; Roberts et al., 2014).

Participant responsiveness

At the clinician level of fidelity in PCIT interventions, the participants are the parents who take part (Lieberman-Betz, 2015). We assessed whether fidelity of implementation for parent responsiveness could be measured by documenting their attendance and through semi-structured interviews completed at the post-assessment and follow up visits (see Supplemental Material 1 & 2). Their interview responses were transcribed and this data was analysed qualitatively using content analysis in NVivo 12 (QSR International Pty Ltd., 2020), which involved sorting their responses into different categories (Vaismoradi et al., 2013). The first author initially read the interviews and developed a set of categories for coding. These initial codes were reviewed with the last author and refined following discussions. The newly developed codes were then used to reanalyse the data by the first author before we summarised the codes and drew conclusions.

Research question 2: Parent fidelity of implementation

Dosage

Dosage at the parent level was measured by asking parents to keep a weekly record of how often they implemented the techniques at home with their child each day so that we could calculate the average rate of teaching episode per day and per week. We provided them with a basic form to document this, and records were to be collected at the individual sessions with the SLPs.

Adherence

Due to the difficulty measuring individual responses during group sessions, parental adherence to the intervention was measured during the individual home assessments visits. This involved recording parents interacting with their child at home for ten minutes at baseline, following each of the three terms of the intervention, and three months post-intervention. Parents were instructed to play with their child as they typically would using a standard set of preselected toys. Five minutes (minutes 2 to 7) from each recording were analysed, disregarding the first two minutes to ensure the participants were comfortably engaged in play. The recordings were rated using The Parent Interaction and Language Rating Scale (PILRS; adapted from the Teacher Interaction and Language Rating Scale by Girolametto et al., [2000]) for adherence. We used this scale as it measures most of the behaviours that were targeted in the LEEIP program. Parents were rated on their accuracy for nine techniques that were targeted in the LEEIP program, namely:

1. Wait and listen: The ability to incorporate a pause into interactions while using a slow pace to allow the child to initiate.

2. Follow the child’s lead: The ability to respond verbally or nonverbally to their child’s initiations and interests.

3. Join in and play: The number of times they actively joined in the child’s play as a partner without dominating.

4. Face-to-face interaction: The number of times they adjusted their physical level by sitting on the floor or leaning towards their child to be at eye level.

5. Turn taking: Balancing the number of child to adults turns.

6. Use of gesture: Accompanying language with a showing, giving, reaching, or pointing gesture.

7. Use of manual signs (Lámh): How often they signed or helped their child to sign by using hand over hand to promote communication.

8. Imitation: The number of times they mimicked their child’s nonverbal or verbal communication.

9. Variety of labels: The use of a range of word types (nouns, verbs, adjectives), repetition, and emphasis of key words.

A score of 1–3 indicates that the accuracy of the technique ‘needs improvement’, 4 that ‘fine tuning is required’, and 5–7 that the use of a technique ‘achieves expectations’.

Quality

We also used the PILRS as a measure of skill mastery by considering how frequently a parent implements a technique, which Lieberman-Betz (2015) uses as a definition of quality of intervention implementation. The PILRS rates each skill on a scale of 1–7, where scores of 1–2 means ‘almost never used’, 3–4 is ‘sometimes used’, 5–6 is ‘frequently used’ and 7 is ‘consistently used’. Our criterion for mastery of quality was 80% or a score of 6 or higher.

Participant responsiveness

According to the framework by Lieberman-Betz (2015) the participants at the parent level of fidelity are the children who take part in the intervention. Using the same recording as outlined above, we assessed this aspect of fidelity through analysis of the children’s ability to engage in joint attention and initiation when responding to parents using the Pivotal Behaviour Rating Scale (PBRS; Mahoney & Wheeden, 1998). The PBRS measures Attention by rating: (1) the child’s general ability to attend to an activity; (2) their level of persistence within an activity; (3) their involvement in the activity; and (4) their compliance with an activity. It also measures Initiation by rating: (1) the child’s ability to initiate activities, and (2) their affect/emotional state during the interaction, and their ability to initiate interaction with adults. All behaviours are measured on a scale of 1 (very low) to 5 (very high) and an average score for Attention and Initiation is calculated.

Reliability of observational measures

In order to determine reliability for the observational measures using the PILRS and the PBRS, 20% of the parent-child interaction recordings (n = 6) were double coded by another researcher who was also blind to the goals of the intervention. The researchers watched each recording twice before rating each measure and achieved a Cohen’s kappa coefficient ranging from 0.6- 1.0 on the PILRS and 0.5–1.0 on the PBRS.

Result

Research question 1: Clinician fidelity of implementation

Dosage

The clinicians delivered 17 of the planned 18 intervention sessions, or a total of 11.25 out of 12 (94%) of the planned intervention hours. One of the group sessions was cancelled due to clinician illness.

Adherence

As one researcher was unavailable to rate two of the recordings, in order to provide the immediate feedback required before the second iteration of the session, two sessions were rated for adherence by the first author only. For the other four sessions, the first and last authors watched the recordings together and individually rated them for adherence to the treatment protocol outlined in each session plan. Following this, inter-rater reliability was calculated and a consensus reached on the final agreement score (see Figure 1). Agreement for each of these sessions was >93%. Scores for adherence for session 1 was low at only 59.6% but increased to over 80% (ranging from 82%- 88%) for the subsequent four sessions before dropping to just over 70% for the final session (see Figure 1).

Figure 1. Clinician adherence and quality measures (80% criterion for mastery) and agreement for the same. Sessions 10 and 13 were coded by one coder.

Quality

We set our quality criterion level at 80% for clinician implementation of the planned techniques for parent training. As can be seen in Figure 1, the clinicians reached this criterion in 4/6 of the observed sessions.

Participant responsiveness

We categorised the themes from the parent interviews to measure participant responsiveness as outlined in Table IV.

Table IV. Themes from parental interviews (children’s pseudonyms used alongside parent quotes).

Theme	Subtheme	Illustrative quotes
1. Engagement with the program	Age appropriateness	“Felt the age of 1 was a bit early for my child; 18 months is better” (Conor’s mother)
	Capacity to take part	“Taking part required some planning due to home location and three other children” (Grace’s mother)
	Useful techniques	“Everything—being face-to-face, working at his level—the skills learnt in LEEIP benefit him more when he is at home as he’s not distracted by people in the new environment” (Conor’s mother)
	Group benefits	“It was helpful being with parents of babies with similar ages” (Luke’s mother); “Hearing other parent’s questions was very useful” (Grace’s mother)
2. Perceived outcomes	Changes in parent	“We have simplified everything; learned to wait for child cues or responses and are waiting” (Jack’s mother)
	Changes in child	“She’s using Lámh signs and is more vocal using more babbling” (Sophie's mother); “More of my child’s character is coming out” (Jack’s mother)

In relation to parental engagement with the program, there was a degree of variation regarding parents perspectives on the age appropriateness of the program. Three parents believed that, at 12 months, their child was too young to take part in and benefit from the program, one parent felt that the age was appropriate, while another parent noted that even if the children were too young, it benefitted the parent who took part. All seven parents stated the schedule of every second week was manageable as “weekly would have been too much for my child to see change or practice what was learnt” (Conor’s mother). They all found the program to be acceptable and the most useful techniques mentioned were wait and listen, followed by using Lámh signs, and singing with their children. Five parents mentioned the benefits of the group sessions from a social and supportive perspective and how “there was never a silly question” (Ellie’s mother). Only one parent felt that the individual session was better for their child, while most favoured the mix of group and individual sessions. Some parents highlighted the challenges of taking part, such as the burden around commuting to attend the sessions, having “a lot on” at the time of the program and having to juggle caring for other children. All seven parents perceived changes in how they communicate with their child and have “slowed down” and are waiting more for their child’s cues. Four parents stated that their children were now using Lámh signs and three stated that their children are more interested in communicating. At the three-month follow up, four parents said that they had taught the techniques to family members and day-care staff.

The number of sessions attended by parents was also used as a measure of responsiveness (See Table V). Two parents attended all three terms, four attended two terms, and one parent attended just one term, as she gave birth to a new baby after this term and was therefore unable to attend. Attendance rates were higher at the individual sessions (at 90%) than the group (at 76% on average).

Table V. Attendance data for each participant.

	Sophie	Jack	Conor	Daniel	Ellie	Grace	Luke
Terms	3	2	1	2	3	2	2
Group	8/11	5/7	3/4	5/7	10/11	5/7	5/7
Individual	5/6	3/4	2/2	4/4	5/6	4/4	4/4

Research question 2: Parent fidelity of implementation

Dosage

The parents reported that they found it too much of a burden to collect data on how often they were using the techniques at home. Anecdotally they reported to the clinicians that it was enough just remembering to do the techniques without also having to record this on paper. Therefore we were unable to collect data on how often the techniques were implemented at home.

Adherence

The PILRS was used to rate how accurately parents used the techniques during the naturalistic recordings taken at home. Figure 2 combines results for all parents for each technique during the six time points taken from baseline to follow up. At baseline, seven out of the nine techniques were rated as ‘needing improvement’, with face-to-face interaction and gesture requiring ‘fine tuning’. At follow up, parental use of seven of the nine techniques were rated to have ‘achieved expectations’, with imitation and use of Lámh continuing to need ‘fine tuning’.

Figure 2. Parent interaction and language rating scale rated for adherence and quality.

Quality

We also used the PILRS to determine if we could measure the quality of parental use of the techniques based on frequency data, where a frequency score of 6 or higher (80% criterion) indicated mastery of the technique (see Figure 2). At baseline, three techniques were rated as ‘almost never’ used by parents, namely imitation, use of Lámh, and variety of words. Although the frequency scores increased by follow up to a score of 4 (‘sometimes used’) for imitation and use of Lámh and 5 (‘frequently used’) for variety of words, they failed to reach a 6/7 or the 80% criterion level. Four other techniques had an average rating of 4, ‘sometimes used’, at baseline. These were wait and listen, follow the child’s lead, join in, and play and turn-taking. The rating score on all of these techniques increased to an average of 6 or higher by follow up, indicating that they were mastered by parents in their interactions with their children. Finally use of gesture and being face-to-face were rated as ‘sometimes used’ (score of 3–4) at baseline, but increased to being ‘consistently used’ (score of 7) by follow up, indicating that they had reached the required quality criterion.

Participant responsiveness

Participant responsiveness for parent fidelity related to the behaviours seen in the children when interacting with their parents. We used the PBRS (Mahoney & Wheeden, 1998) to examine if it was feasible to measure child responsiveness by assessing seven behaviours related to attention and initiation. These two variables are presented across all time points in Figure 3.

Figure 3. Attention and initiation scores for each participant as rated on the Pivotal Behavior Rating Scale (PBRS).

As the focus of this paper was not on effectiveness or change in behaviours, we have not completed statistical analysis on these scores, but will describe the trends in relation to the feasibility of measuring responsiveness. Reviewing the data, it can be seen that most children demonstrated an increase in attention scores, which was largely maintained at follow up. However individual differences were noted. It was also feasible to measure initiation of joint attention, although once again individual differences were noted.

Finally, the feasibility of completing this research in terms of time commitment should be considered. In addition to the time it took clinicians to adapt the session plans in phase 1 so that fidelity could be measured (approx. 12 hours), the coding of the recordings to measure adherence took a further 4.5 hours of researcher time, and the coding of parent-child interaction recordings to measure quality and responsiveness took about 20 hours. This time commitment should be considered part of the feasibility of measuring fidelity of implementation.

Discussion

The present study adds to the literature on fidelity of implementation in PCIT interventions, by reviewing how feasible it is to measure in a community-based intervention, designed and delivered by SLPs. With regard to clinician fidelity, using the categories recommended by Lieberman-Betz (2015), we were able to take a basic measure of dosage, and noted that 93% of the planned intervention hours were delivered. While this measure of dosage (specifying only session frequency and length) reflected that which was reported in 71% of studies reviewed by Lieberman-Betz (2015), it does not reflect the complexity of dosage as a construct, which includes both quantitative and qualitative elements (see reviews by Frizelle et al., 2021). Warren et al. (2007) suggested five dosage characteristics that should be included to describe intervention intensity, namely: dose form (which aims to describe the tasks or activities within which teaching episodes are delivered); dose (to include the number and average rate of teaching episodes per unit of time, the length of the intervention session, and the density of the episodes within the session); dose frequency (the number of intervention sessions per unit of time); total intervention duration (the time period for which the intervention is delivered); and cumulative intervention intensity (the product of the previous three components). Furthermore, the LEEIP intervention involved four levels of teach, model, coach, and review (Roberts et al., 2014). While teach and model were delivered at the group level, coach and review were implemented individually in the parent-child dyads, and so these teaching episodes would need to be defined and specified in order to measure dose in group settings. A further complication regarding the measurement of dosage was that this intervention was delivered fortnightly, and so it did not align with the calculation for cumulative intervention intensity as outlined by Warren et al. (2007), which assumes a minimum of once weekly intervention frequency. Although the concept of dosage requires further revision and agreement in the literature, we would argue that intervention studies should aim to measure dosage in this more detailed way for both group and individually-delivered interventions, in order to truly capture the active ingredients of an intervention and how they might mediate the outcomes.

In relation to clinician adherence, we noted that the accuracy of the technique delivery was lowest in the first session, which was likely due to the fact that the clinicians were getting used to the new protocol, were being recorded, and were getting to know the families. The Hawthorne effect, where people change their behaviour due to being observed, could also have been an influencing factor. Had we been able to record all of the intervention sessions, instead of just the first one in each theme, we could have allowed for this. However it is also important to remember that real-world clinical practice does not always involve perfect adherence and therefore a range of accuracy should be expected. We found that feedback and discussions about adherence after the first session, and increased familiarisation with the process, resulted in increased adherence levels that were maintained for the subsequent four sessions. This was largely achieved through the open and honest relationship that existed between the clinical and research teams. Breitenstein et al. (2012) also found that regular discussion and feedback on clinician adherence improved fidelity of implementation in their study of a group-delivered parenting program. We note that in our study adherence decreased at the final session, which is likely due to the fact that the lead SLP was absent for that session and consequently it was implemented by the SLP who was usually in a more supportive role. This again reflects clinical work in practice, where interventions are not always implemented under perfect laboratory conditions. In her review of PCIT interventions, Lieberman-Betz (2015) noted that clinician adherence was only reported in 35% of studies, and was usually measured through a self- or observer-completed checklist. Given our experience, this low level of reporting could be explained by the large amount of work involved in creating detailed session plans to describe the techniques used and differentiating the goals for all participants, as well as the need to record and analyse the sessions in order to provide feedback. The need to measure adherence however did lead to the creation of a clear description of the activities to be implemented, which will later be used to devise a manual for the LEEIP program and help improve subsequent program fidelity (Dane & Schneider, 1998).

Quality was measured for clinician fidelity by reviewing the session goals and comparing the frequency of technique implementation to a pre-defined criterion mastery level of 80%. As with adherence, the clinicians met this quality level in 4/6 sessions, but failed to achieve this in the first and last sessions (with frequency ratings of 60% and 70% respectively). Although Lieberman-Betz (2015) maintains that frequency measures indicate that a technique is being implemented more often, and therefore is at a higher quality, we would again argue that this is problematic as it overlaps with the construct of dosage outlined above. In addition, it does not capture how well or to what degree a technique is being implemented. Moreover, much like the difficulty in measuring dosage at a group level, it became apparent when viewing the recordings that it would have been too difficult to measure quality in a group setting. This was because parents could become distracted by their children’s behaviour during the sessions and the SLPs had to spend a lot of time ensuring that each family got some individual attention as well as keeping the attention of the group on task. Interestingly, Lieberman-Betz (2015) also noted that only 20% of studies reviewed measured quality for clinician fidelity of implementation. Additionally, Kaderavek and Justice (2010) note that quality is more difficult to measure than adherence as it may be linked to the sensitivity of the interventionist to the different individual needs of participants receiving treatment. For example, one parent may need a lot of coaching and support to acquire a technique that they find challenging, while another may need more modelling. Carroll et al. (2007) also maintain that as quality aims to measure whether the clinician implements a technique well or otherwise, it is not as easy to define, is more open to subjectivity, and therefore, is not often evaluated in fidelity of implementation studies. Nonetheless, they argue that it is still important to capture, where possible, as an intervention can be implemented badly even when adherence to the procedure is high. One way in which quality could be measured is through a self-completed checklist on how well clinicians consider they implemented the technique (Kaiser & Roberts, 2013). Measures of quality in group sessions would also need to be different to individual sessions, as aspects such as managing group dynamics as a component of quality, would need to be considered. These aspects would benefit from future consideration in research.

Finally, participant responsiveness in relation to clinician fidelity was measured through interview data collected from parents and through a review of their attendance data. The analysis of interviews captured how integral parental engagement is to the success of PCIT (Melvin et al., 2020) and was in line with other reviews of parental experiences of PCIT interventions (O'Toole et al., 2021). The current study noted benefits such as the emotional and practical support provided in group sessions, and how parents found certain techniques to be more useful than others, such as the use of singing and signing with their young babies. Parents also perceived an improvement in their child’s communication and in their own ability to facilitate communication development following the intervention. This reflects parental empowerment and overall wellbeing that can be achieved as a result of PCIT interventions (O'Toole et al., 2021). Furthermore, their high attendance rates at the individual and group sessions also reflected their engagement with, and responsiveness to, the program.

Our study found that parental dosage was not easy to measure, as asking parents to keep a written record of how much and how often they used the techniques was too burdensome. Lieberman-Betz (2015) noted that only 14% of their studies included a measure of dosage at the parent level, which was collected through parental report and diary keeping. There are alternative ways to measure parental dosage, including recording devices such as the LENA system to measure the number of words used by adults, although this system cannot measure quality of input and has been criticised for a lack of reliability due to coding errors (for a full review of the system see Ganek & Eriks-Brophy, 2018). Measuring dosage at the parent level therefore remains difficult for PCIT research. For adherence and quality, the use of individual recordings captured during parent-child interaction in the home was relatively quick and easy to collect, and could be used to monitor these factors during and after the intervention. Using the adapted PILRS (Girolametto et al., 2000), we noted that parents adhered to seven out of nine techniques to our pre-defined criterion of 80% following the intervention, and we could also determine that the quality of six techniques increased to being ‘frequently or consistently used’ (>80% criterion level) by parents. Lieberman-Betz (2015) found that most studies (60%) reviewed included a measure of parental adherence, however only 29% reported on quality. As we have previously outlined, this conceptualisation of quality, as measured through frequency or rate of technique delivery by parents, overlaps with the concept of dosage and is therefore problematic. Although the PILRS allows for dual scoring of adherence and quality based on a frequency rating, a more detailed and nuanced measure of quality in particular is needed. For example, in one study, clinicians rated how well parents use the techniques and linked the scores to a parent self-completed confidence rating (Kasari et al., 2010). This method could be explored for the measurement of quality, once a clear and transparent definition can be achieved. Finally, we were able to measure responsiveness of the child participants at the level of parental fidelity based on reviewing the children’s ability to respond to and initiate joint attention when interacting with parents though the PRBS. Almost half (43%) of the studies reviewed by Lieberman-Betz (2015) also measured participant responsiveness for children, either through direct assessment or parent report.

Limitations of the current study

There were a number of limitations to the current study. Firstly, the PILRS as a measure of adherence was problematic as some of the techniques are measured in terms of the frequency with which they are implemented (e.g. join in and play) while others only for accuracy (e.g. follow the child’s lead). Furthermore, scoring on the PILRS is not evenly distributed across the scale (i.e. a score of 1–3 indicates needs improvement, 4 that fine tuning is required, and 5–7 achieves expectations). Although the tool was not designed to be a measure of adherence or quality, there is potential for it to be improved to measure both constructs more effectively. In addition, as the focus of this paper was not on effectiveness, we did not complete statistical analysis of the outcome measures. In order to achieve this we would have needed to collect more data points than the 1–2 baseline conditions and 1–3 treatment phases, as some researchers suggest five baseline datapoints to interpret trends in single case designs (Ledford et al., 2018).

Summary and conclusions

Monitoring and assuring fidelity is a critical aspect of implementation research (Hsueh et al., 2020), particularly in effectiveness research where clinicians try to replicate beneficial results of an intervention in real world settings and with different clinical populations (Breitenstein et al., 2012). It is likely that many PCIT intervention studies fail to report fidelity of intervention measures either as interventionists are not aware of the various factors that need to be considered, because they are not well defined, or because it is burdensome for SLPs and parents to include these measures. This study showed that it is feasible to measure adherence and participant responsiveness at the level of the clinician and parent, if there is open and honest collaboration between clinicians and researchers who have the time to review these factors. However, it is essential for PCIT interventions that the triad of goals (for the clinician, parent, and child) are defined and described in a clear and transparent manner if fidelity is to measured. We found that dosage and quality measures were harder to capture. Moreover, dosage and in particular quality need to be further defined and agreed on in the literature so that we can improve our measurements of these. We strongly recommend removing frequency from the definition of quality in the Lieberman-Betz framework as it overlaps with the concept of dosage. We would also recommend removing attendance as a measure of participant responsiveness, as this also relates to dosage. Rather the construct of responsiveness could be broadened to include a measure of acceptability as outlined by Sekhon et al. (2017). Although both concepts are somewhat interconnected, acceptability captures a broader range of elements that are likely to impact on the success of an intervention. Intervention research in the field of speech-language pathology in general would benefit from greater consensus on the active ingredients of therapy and how to define and measure these (see Frizelle et al., 2022). Fidelity of implementation aspects of dosage, adherence, quality, and responsiveness should also be included in this work.

PCIT for communication disorders is a complex intervention that requires a skilled clinician to deliver effective coaching to a parent, who in turn has to implement techniques with sufficient frequency and quality to their child, in order to effect change (Roberts et al., 2014). To do this effectively, the clinician is required to respond to each individual family’s situation as well as their ability to engage with the intervention and this can only be achieved through collaborative practice (Klatte et al., 2020), mutual understanding, and a trusting relationship (O'Toole et al., 2021). The parents in turn are required to implement the techniques with sufficient accuracy and frequency in their everyday interactions with their children, in order to improve the child’s chances of maximising their language and communication skills. The fidelity achieved during PCIT interventions is therefore affected by numerous factors, which may go some way towards explaining why an intervention effect is often reported for participating parents but not always for child language (Roberts & Kaiser, 2011). Although we don’t yet know the minimum level of treatment fidelity that needs to be achieved in order to effect change in children, it is likely that higher levels will result in the correct dosage, adherence, and quality in order to achieve the intended responses in the children (Justice et al., 2008; Kaderavek & Justice, 2010).

For future studies, we would recommend that once a clear definition of quality is achieved, more structures and supports are put in place in order to reduce the burden on parents and allow us to measure dosage and quality of PCIT interventions in the home setting. This should be developed in conjunction with parents themselves, and may include the use of technology, simple text reminders, and self-completed, criterion-related checklists. We also need mechanisms in place to make it easier for clinicians to measure dosage and quality in real world practice, as analysis is time consuming and is not always practically achievable. Where two clinicians are involved in delivering a group program, it may be possible for one to document dosage through the use of a simple checklist, while the other leads the intervention. However, this is not a solution for those delivering a program alone. Where recording is being used, it would be preferable to record all sessions, even if all are not analysed, in order to reduce the Hawthorne effect. It is only by measuring all aspects of fidelity of implementation that we can advance our understanding of how and why PCIT interventions may or may not affect change (Carroll et al., 2007) thereby allowing us to identify required adaptations to enhance their effects.

Disclosure statement

No potential conflict of interest was reported by the author(s). Mide Kearney and Deirdre Flynn work for the organisation Down Syndrome Ireland which runs the LEEIP program for parents.

Additional information

Funding

This work was supported by a MSc scholarship awarded to Sarah Cronin by Down Syndrome Ireland.