Training Teachers to Implement Classroom Pivotal Response Teaching during Small-group Instruction: A Pilot Study

ABSTRACT

Background: Providing effective education to students with autism spectrum disorder (ASD) poses a significant challenge to educators. Although several evidence-based practices (EBPs) have been developed, few have been systematically implemented in educational settings. Pivotal response treatment (PRT) is a naturalistic behavioral intervention that has been adapted for implementation in the school context.

Methods: This pilot study used a concurrent multiple baseline design across seven teachers and students with ASD to examine the effectiveness of teacher training in classroom pivotal response teaching (CPRT) on teacher fidelity of implementation during small-group instruction and students’ communication skills and maladaptive behaviors in schools for special education in the Netherlands.

Results: Results indicated no replicated effect of CPRT training on teachers’ fidelity of implementation or children’s communications skills and maladaptive behavior, although teachers reported high satisfaction with the CPRT training.

Discussion: Implications for clinical practice and directions for future research are discussed.

KEYWORDS:

• Autism spectrum disorder
• classroom pivotal response teaching
• fidelity of implementation
• special education
• staff training

Introduction

Providing effective education to students with autism spectrum disorder (ASD) poses a significant challenge to educators. Students with ASD have impairments in social communication and interaction, exhibit restricted, repetitive, and stereotyped behaviors, and often engage in challenging behaviors, such as disruptive behavior, elopement, aggression, and self-injurious behavior.^1,2,3 In addition, comorbid psychiatric diagnoses are common in students with ASD, particularly attention deficit hyperactivity disorder (ADHD), oppositional defiant disorder (ODD), and anxiety disorders.^4,5,6 These impairments may hinder their success in educational settings. For example, students’ challenging behaviors negatively impact teachers’ instructional efforts,⁷ thereby reducing opportunities for learning functional skills. As the prevalence of ASD continues to rise and thus more students with ASD are enrolled in schools,^8,9 teachers in both regular and specialized education are increasingly challenged to meet the complex educational needs of this population to improve their social, behavioral, and academic outcomes.

Although several evidence-based practices (EBPs) have been developed to teach functional skills to students with ASD and to reduce challenging behaviors,^10,11 few have been systematically implemented in educational settings.^12,13 Barriers to the implementation of EBPs in schools include the complexity of these practices and limited training of teachers in the use of EBPs.^14,15,16 Furthermore, most EBPs have been designed as one-to-one interventions, and therefore may poorly fit the classroom context, where students usually receive instruction in a group setting.¹³ As a result, teachers may adapt or combine EBPs to fit their own teaching preferences and their students’ educational needs.¹⁵ Although adaptations are inevitable, it is often unclear whether these modified or combined interventions are as effective as the original EBP.^17,18,19

Since several years there is increased interest in how to address this research-to-practice gap in autism and to translate EBPs for students with ASD to community settings, including schools.^12,13 The implementation science has developed numerous models and frameworks that describe the phases of implementation and factors across multiple levels that affect the implementation process.^12,20,21 The implementation of EBPs in community settings used to be a unidirectional process, simply moving information from research into practice. Recently, this process shifted toward a more bidirectional and interactive process, characterized by collaborations between researchers and community stakeholders during all phases of implementation and often referred to as community-based participatory research.^22,23 Pivotal Response Treatment (PRT) provides an example of an EBP that has been translated to an educational setting using a community-academic partnership.

PRT is a naturalistic intervention based on the principles of applied behavior analysis.²⁴ PRT targets pivotal skills (e.g. motivation, initiations, and responsivity to multiple cues) in children with ASD to produce generalized improvements in other – often untargeted – areas of functioning by providing opportunities for learning in natural contexts using antecedent techniques (i.e. incorporating child choice, presenting clear opportunities, varying tasks, and interspersing maintenance and acquisition tasks) and consequent techniques (i.e., contingent and natural reinforcement and reinforcement of attempts). Positive outcomes for children with ASD have been reported as result of PRT, including increases in initiations and collateral improvements in a variety of language and communication skills, play, affect, and challenging behavior.²⁵ Most research supporting the effectiveness of PRT has been conducted with pre-schoolers with ASD and cognitive impairments, although an increasing number of studies indicate that school-aged children and adolescents or young adults with average cognitive abilities may also benefit from PRT.^{26,27,28,29,30} As PRT is designed to be implemented in natural contexts and students with ASD spend several hours per day at school once they enter primary education, PRT has been considered an appropriate EBP to be translated into an educational setting.

To systematically adapt PRT for implementation in the school context Stahmer, Suhrheinrich, and colleagues conducted a series of studies in collaboration with special education teachers and school administrators using both qualitative and quantitative methods. Focus groups were conducted to obtain information on the importance and the feasibility of each PRT technique in the classroom.³¹ Teachers’ use of PRT was observed in another study to determine the extent to which teachers implemented each PRT technique with fidelity.³² Fidelity of implementation refers to the degree to which an intervention is implemented as intended by its developers.³³ The results of these studies indicated that teachers implemented some PRT techniques with fidelity in their classrooms (e.g. gaining the child’s attention, providing clear opportunities, and incorporating child choice) and would benefit from further training or additional resources to implement other techniques (e.g. providing natural and contingent reinforcement). Two techniques were poorly implemented and not valued highly (i.e. incorporating turn taking and requiring a response to multiple cues). Further research was conducted to examine the necessity of these techniques and their optimal implementation.^34,35,36 This series of studies resulted in an adapted intervention referred to as Classroom Pivotal Response Teaching (CPRT) to distinguish it from the original intervention.³⁷

So far, research on CPRT has mainly focused on training teachers in CPRT.^32,38–42 Research has indicated that a combination of didactic group instruction, video or in vivo modeling, and (video-based) performance feedback during individual coaching sessions is effective in training teachers to implement most components of CPRT with an acceptable level of fidelity (i.e. at least 80% correct use). Surprisingly, very limited research to date has examined the effectiveness of CPRT on students with ASD. Robinson⁴³ found that PRT resulted in improvements in social communication skills of students with ASD in the classroom and at the playground. Stahmer et al.³⁸ reported that CPRT was moderately to highly effective for increasing active student engagement and reducing disruptive behavior. Although the effectiveness of PRT is well-established, it is yet unclear whether CPRT is equally effective, and thus additional research on the effectiveness of CPRT is warranted. Furthermore, so far, all studies on CPRT have been conducted in the United States. Whereas ABA-based interventions are widely available in North America, the use of behavioral interventions is still in its infancy in the Netherlands, particularly in educational settings.^44,45 In the Netherlands, the majority of students with ASD are attending special schools,⁴⁶ despite policies to promote inclusive education. Although most special education teachers have a master’s degree in special educational needs,⁴⁷ teachers do not receive appropriate training in EBPs. Rather, most specialized schools employ an eclectic approach.⁴⁸ Thus, it is important to provide training in an ABA-based intervention, such as CPRT, to Dutch special education teachers and to determine the effectiveness of this training and the feasibility of CPRT use in special education classrooms.

To address these needs, we conducted a pilot study to examine the effectiveness of training in CPRT for seven special education teachers in the Netherlands. Specifically, we aimed to investigate the effectiveness of training in CPRT on (a) teacher fidelity of CPRT implementation during small-group instruction, (b) communication skills and maladaptive behaviors of students with ASD, and (c) maintenance of these skills over a three-month period.

Methods

Setting and Participants

This study was conducted at one special school for primary education and two schools for special education in a large city in the eastern part of the Netherlands. Special schools for primary education are attended by children with mild to moderate learning and/or behavioral difficulties and these schools have smaller classes compared to regular schools.⁴⁹ Schools for special education are attended by children with disabilities, chronic illness, or emotional and behavioral disorders, who require more specialized support than regular or special schools for primary education can provide.

Teachers were included in the study if they met the following inclusion criteria: (a) working at a special school for primary education or school for special education, (b) at least three years of teaching experience, (c) no experience with (C)PRT prior to this study, and (d) at least one student in their class that met the inclusion criteria for students. These were the following: (a) a clinical diagnosis of ASD according to the DSM-IV-TR or DSM-5 criteria,^1,50 confirmed by a score above 60 on the Social Responsiveness Scale⁵¹ (SRS), (b) a verbal/reasoning or performance IQ above 80 on a standardized intelligence scale, (c) aged between 6 and 12 years at baseline, and (d) an individualized education plan that included at least one social-communication goal. Students were excluded if they were diagnosed with specific language impairment, received (C)PRT prior to this study, or started speech-language therapy or social skills training during data-collection.

Five schools for special (primary) education that were located in the large city in the eastern part of the Netherlands were contacted via e-mail. Three schools indicated their interest in participating in the study; two schools did not respond. The schools that indicated their interest were actively involved in the design and planning of the study. Potential participants (teachers) were identified by each school’s principal or psychologist. Informed consent was obtained from interested teachers who met the inclusion criteria. Teachers selected one student with ASD from their class based on the inclusion criteria for students. Informed consent was obtained from the parents of each student and assent was obtained from students older than 12 years. Since data collection involved videotaping small-group instructions in the classroom, teachers collected consent to be videotaped from the parents of each student in their class and students older than 12 years. If no consent was given, students were not videotaped during data collection.

Seven teachers and seven students participated in this study. Teachers (5 females) had a mean age of 44.7 years (SD = 11.2; range: 29–61). One teacher had a master’s degree in pedagogical and educational sciences; all other teachers had bachelor’s degrees in education. Two teachers worked at a special school for primary education, five teachers worked at schools for special education. On average, teachers had 20.4 years of teaching experience (SD = 8.7; range: 7–32) and 13.1 years of experience in working with students with ASD (SD = 5.1; range: 5–18). Demographic characteristics of the students are displayed in Table 1. Students (7 males) had a mean age of 9.7 years (SD = 1.2; range: 8–11). Their scores on the SRS confirmed their ASD diagnosis.

Table 1. Demographic characteristics of students at baseline

Student	Age	Diagnosis	SRS	Verbal IQ	Performance IQ	Social-communication goal
1	10;9	ASD	69	102	110	Requesting help
2	9;0	ASD	69	73	86	Requesting help
3	9;9	ASD, ADHD, tic disorder, and unspecified obsessive-compulsive disorder	84	104	98	Following instructions and gaining the teacher’s attention in an appropriate manner
4	8;0	ASD	84	90	94	Following instructions and responding to questions
5	8;11	ASD	77	74	98	Requesting help
6	10;2	ASD and Tourette’s disorder	81	90	100	Communicating annoyance in an appropriate manner
7	11;6	ASD, ADHD	80	71	104	Gaining the teacher’s attention in an appropriate manner

Note. ASD: autism spectrum disorder; ADHD: attention deficit hyperactivity disorder; SRS: social responsiveness scale. A score > 60 on the SRS is an indication for ASD.

Design

A concurrent multiple baseline design across three groups was used to investigate the effectiveness of training in CPRT.⁵² Groups consisted of two or three teachers and students from the same school. Prior to data collection, each group was randomly assigned to a baseline length using Research Randomizer.⁵³ Baseline lengths were predetermined to ensure that teachers from the same school could participate in training in CPRT together. Pre-baseline and baseline sessions started concurrently for each group and groups remained in baseline for six, nine, or 12 sessions, resulting in a sequential introduction of the intervention.

Procedures

Pre-baseline

Three pre-baseline sessions were conducted to habituate teachers and students to being videotaped and to reduce reactivity effects.⁵⁴ During pre-baseline sessions, teachers provided instruction in reading, writing, mathematics, history, geography, or science to a small group of students, including the student who participated in the study (i.e. target student). A research assistant held the camera to capture the small-group instruction. In the absence of a research assistant, teachers put the camera on a tripod and positioned it to capture interactions between teacher and students. No data on the dependent variables were collected during pre-baseline.

Baseline

Baseline consisted of six, nine, or 12 sessions. These sessions were conducted semiweekly during small-group instruction. Procedures were similar to those during pre-baseline. During baseline sessions, teachers provided instruction in reading, writing, mathematics, history, geography, or science to a group of two to five students, including the target student. They did not receive feedback on their implementation of CPRT. The first 10 to 15 minutes of each small-group instruction were videotaped by a research assistant. In case the research assistant was absent, the camera was put on a tripod. In addition, teachers were instructed to conduct planned activity checks (PLACHECKs) twice a week to measure maladaptive behavior (see Maladaptive behavior). Baseline measures were conducted to assess teachers’ fidelity of CPRT implementation prior to training in CPRT and to assess baseline level of children’s communication skills and maladaptive behavior.

Intervention

Training in CPRT was conducted by the first and second author who were both certified PRT trainers (PRT level III or IV) and had several years of experience implementing PRT and training others in PRT. Training in CPRT consisted of three two-hour group sessions followed by individual coaching sessions.^39,40 Group sessions were conducted biweekly. Specific content of group sessions is outlined in Table 2. During these group sessions, teachers were introduced to ABA, the ABC pattern of behavior, and (C)PRT. They received instruction in antecedent CPRT components (i.e. gaining the student’s attention, presenting clear and appropriate cues, interspersing maintenance and acquisition tasks, sharing control, and using multiple cues), prompting, and consequent CPRT components (i.e. providing direct and contingent reinforcement and reinforcing attempts) via a manual, didactic instruction, video-examples, worksheets, and role-plays. Teachers received a copy of ‘Classroom Pivotal Response Teaching for Children with Autism³⁷ and a summary of each CPRT component in Dutch. During the group sessions, the trainers taught them how to implement each component of CPRT with individual students and in group settings via didactic instruction and by showing video-examples of other teachers implementing CPRT. Subsequently, teachers completed worksheets and participated in role-plays to practice. Trainers provided verbal feedback on these worksheets and roleplays, but there was no mastery criterion set that teachers had to meet. Teachers were also taught to set goals and to collect data on these goals. In addition, they received a brief introduction to the assessment of fidelity of CPRT implementation.

Table 2. Overview of group sessions

Session	Session content	Homework assignment
1.	Introduction to ABA and ABC pattern of behavior Introduction to PRT and CPRT Instruction in antecedent CPRT components (gaining the student’s attention, presenting clear and appropriate cues, interspersing maintenance and acquisition tasks) Worksheet ‘student attention’ Roleplay to practice antecedent components	Reading: Chapter 2, 3 (pages 33–41), and 4 (pages 57–63) Video: Implement CPRT during small-group instruction
2.	Video feedback on implementation of CPRT Instruction in antecedent CPRT components (sharing control and using multiple cues) Instruction in prompting Roleplay to practice antecedent components	Reading: Chapter 3 (pages 41–47) and 4 (pages 63–66) Video: Implement CPRT during small-group instruction
3.	Video feedback on implementation of CPRT Instruction in setting goals and data collection Worksheet ‘student goals’ Instruction in consequent CPRT components (providing direct and contingent reinforcement and reinforcing attempts) Worksheet ‘reinforcement’ Roleplay to practice consequent components Introduction to assessment of fidelity of implementation	Reading: Chapter 3 (pages 47–54), 4 (pages 66–68) and 5 Video: Implement CPRT during small-group instruction

After each group session, teachers were instructed to practice CPRT components during instruction to a small group of students, including the target student, and to videotape their instruction. During the second and third group session, trainers provided feedback on teachers’ implementation of CPRT, using the following protocol: (a) on viewing correct use of CPRT components, the trainer provided the teacher with positive performance-based feedback and praise, (b) on viewing incorrect (or lack of) use of CRT components, the trainer provided the teacher with corrective feedback, asked the teacher open-ended questions to solicit ideas for correct use of the components, and provided suggestions for improving the use of the component(s), (c) after viewing about one minute of a videotape, the trainer made a positive statement (e.g. praise, encouragement) in case the teacher had not been provided with positive feedback on that minute of the videotape, and (d) after viewing about five minutes of the videotape, the trainer concluded with a positive statement.⁴³ From the third group session, data on the dependent variables were collected.

Individual coaching sessions started approximately two weeks after the third group session and were conducted weekly. During each coaching session, the teacher practiced implementation of CPRT components during small-group instruction, while the trainer observed and videotaped the instruction. The trainer then viewed the first 10 minutes of the videotape to assess fidelity of CPRT implementation (see Fidelity of CPRT implementation). Teachers were informed about their level of fidelity and received verbal feedback on their use of CPRT components, following the video feedback protocol. These video-feedback sessions were approximately 30 minutes in duration and were scheduled at times convenient to teachers, but always the same day the videotape was recorded. Teachers received a written summary of the video feedback and a copy of the fidelity form for their own records. Coaching sessions continued until teachers demonstrated 80% correct use of CPRT in two consecutive coaching sessions or until a teacher had participated in six coaching sessions.^42,43

During training in CPRT, teachers continued to conduct PLACHECKs twice a week to measure maladaptive behavior (see Maladaptive behavior).

Post-intervention

Post-intervention consisted of six sessions for six teachers and three sessions for one teacher due to illness. Sessions were conducted twice a week and procedures were identical to those during baseline, except that during post-intervention, teachers were instructed to implement CPRT while providing small-group instruction. Purpose of post-intervention sessions was to evaluate whether teachers and students maintained their skills immediately after training in CPRT. In addition, teachers conducted PLACHECKs twice a week to measure maladaptive behavior (see Maladaptive behavior).

Follow-up

Three or four follow-up sessions were conducted for six teachers three months after the last post-intervention session. Procedures were identical to those during baseline and post-intervention. Purpose of follow-up sessions was to assess whether teachers and students maintained their skills over time. During follow-up, teachers also conducted PLACHECKs twice a week to measure maladaptive behavior (see Maladaptive behavior).

Dependent Measures

Fidelity of CPRT Implementation

Fidelity of CPRT implementation was measured using partial interval recording.⁵⁴ The following components were recorded: (1) maximizing student motivation, (2) implementing three-term contingency learning trials, and (3) interspersing maintenance and acquisition trials.^37,38 The teacher maximized student motivation if he or she (a) incorporated choice by providing choices between activities, materials, or topics (b) followed the student’s interest or lead, or (c) incorporated turn taking by taking turns with the student or by facilitating turn taking between students.^37,38 The teacher implemented three-term contingency learning trials if he or she contrived at least two complete three-term contingencies during an interval. A three-term contingency learning trial was recorded if one of the following sequences of teacher behavior occurred: (a) gaining the student’s attention, presenting a clear and appropriate cue, student response or reasonable attempt, and reinforcing the student’s response directly and contingently, (b) gaining the student’s attention, presenting a clear and appropriate cue, prompting the student to respond, student response or reasonable attempt, and reinforcing the student’s response directly and contingently, or (c) gaining the student’s attention, presenting a clear and appropriate cue, prompting the student to respond, no student response or reasonable attempt, and withholding reinforcement.³⁰ Operational definitions of these behaviors are presented in Table 3. The teacher interspersed maintenance and acquisition trials by presenting at least three different cues or by providing at least three different prompts in two consecutive intervals.^37,38 These cues and prompts are also operationally defined in Table 3.

Table 3. Definitions of behavioral categories for three-term contingency learning trials

Behavioral category	Operational definition
Gaining the student’s attention	The teacher had the student’s attention if the student (a) was looking toward the teacher or teaching materials, (b) had oriented his or her body toward the teacher or materials, or (c) was reaching for teaching materials. If the student is engaged with other materials, he or she is not attending.
Presenting a clear and appropriate cue	The teacher presented a clear and appropriate cue if (a) the cue was uninterrupted, and (b) only one cue was presented at the same time. Six different cues (increasing in difficulty) were recorded: Instruction: giving a direct instruction telling the student exactly what to do, for example ‘Get your math book’ Multiple cues: giving a direct instruction that requires the student to attend to multiple cues, for example ‘Give me a small blue block and a big red block’ Question: asking a closed or open-ended question, for example ‘What’s your book about?’ Comment: making a statement, without giving details, for example ‘We’re going to do something fun’. Waiting: waiting while looking expectantly to the student, for example when the student needs help or when the teacher is expected to act. Environmental arrangement: setting up activities or situations to elicit communication, for example by sharing control, keeping teaching materials in sight out of reach, or keeping materials out sight out of reach.
Student response or reasonable attempt	The student responded correctly if his or her response was (a) appropriate, and (b) related to the teacher’s cue. A reasonable attempt was recorded if the student’s response was less accurate or complex than expected based on the student’s level of (social) communication skills, but appropriate and related to the teacher’s cue.
Prompting the student to respond	If the student did not initiate within 3 seconds after the teacher presented a cue, the teacher offered help by prompting. Five types of prompts were recorded: Time delay prompt: giving the student an expectant look to stimulate him or her to respond. Visual prompt: showing a picture or using gesture to stimulate the student to respond. Open-ended question prompt: asking an open-ended question to stimulate the student to respond, for example ‘What could you ask me now?’ (Verbal) model prompt: modeling the verbal or nonverbal response excepted from the student. Physical prompt: physically guiding the student to respond. The teacher continued prompting until the student responded or until three prompts were given. The order of prompts was not predetermined.
Reinforcing the student’s response directly and contingently	The teacher provided direct and contingent reinforcement if (a) he or she responded to the student within two seconds, (b) this response was the teacher’s first behavior after the student’s response, and (c) this response was directly or naturally related to the student’s response. Reinforcement was also recorded if the teacher withheld reinforcement if the student did not respond after being prompted at least once.

Videotapes of small-group instructions were coded by independent, naïve observers (i.e. research assistants). Each videotape was divided into 10 one-minute intervals. If videotapes lasted more than 10 minutes, 10 minutes from the start of the instruction were coded. Observers recorded for each one-minute interval whether or not each component (i.e. maximizing student motivation, implementing three-term contingency learning trials, and interspersing maintenance and acquisition trials) had been implemented during the interval. A plus (+)/minus (-) was recorded if the component had/had not been implemented. For each videotape, an overall percentage of CPRT fidelity was calculated by dividing the number of intervals where all three components were scored a plus by the total number of intervals, multiplied by 100.

Communication Skills

Partial interval recording was used to measure communication skills.⁵⁴ Two communication skills were distinguished: initiations and responses.^55,56 Initiations were defined as contextually appropriate attempts (verbal or nonverbal) of the student to begin or direct interactions to gain attention or a response from the teacher or a peer, for example requesting objects, protesting, requesting help, requesting (social) information, commenting to begin an interaction, providing instructions to a peer, offering help, or asking for a turn. Responses were defined as contextually appropriate behaviors (verbal or nonverbal) following the teacher’s or a peer’s cue or initiation within 5 seconds, for example following instructions, responding to questions, or commenting in response to the teacher or a peer. Only unprompted initiations and responses were recorded.⁵⁷

Ten-minute videotapes of small-group instructions were divided into 30 intervals of 20 seconds. An independent, naïve observer recorded whether or not the student initiated or responded during the interval. A plus (+)/minus (-) was recorded if the behavior occurred/did not occur during the interval. For each student, a percentage of initiations and responses was calculated by dividing the number of intervals with an initiation or response by the total number of intervals, multiplied by 100.

Maladaptive Behavior

Maladaptive behavior as experienced by teachers was measured using PLACHECKs, a variation of momentary time sampling.⁵⁴ Teachers were instructed to record twice a week from circle time till recess time at the end of each 15-minute interval whether the target student was engaged in the activity or lesson by scoring a plus (+) or displayed maladaptive behavior by scoring a minus (-). Academic engagement was defined as the student listening to teacher’s instructions, following teacher’s instructions, answering teacher’s questions and/or working on academic tasks, while looking or oriented toward the teacher or teaching materials.⁵⁸ Maladaptive behavior was defined as behavior that was disruptive or interfered with participation of the student or peers in activities or lessons, for example disruptive behavior (e.g. making noises during quiet time), noncompliance (e.g. failure to follow instructions within five seconds), verbal aggression (e.g. cursing), physical aggression (e.g. hitting), self-injurious behavior (e.g. skin picking), and self-stimulatory behavior^58,59,60 (e.g. hand flapping). A percentage of maladaptive behavior was calculated by diving the number of intervals with maladaptive behavior by the total number of intervals, multiplied by 100.

Social Validity

During post-intervention, teachers were asked to complete a questionnaire to assess the social validity of CPRT and training in CPRT. The questionnaire consisted of 23 statements (e.g. ‘Coaching sessions were useful’ and ‘I would recommend this training in CPRT to other teachers’) that were rated on a five-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). These items measured teachers’ attitude toward CPRT and whether they considered the components of CPRT training as useful and pleasant. In addition, teachers were asked to give an overall grade between 1 and 10 to rate the training in CPRT.

Interobserver Agreement

A second naïve observer independently coded 35% of the videotapes, approximately evenly distributed across teachers/students and study phases, to assess interobserver agreement. Prior to coding videotapes, observers received training in using recording systems for fidelity of CPRT implementation and communication skills. Observer training consisted of (a) discussing definitions, examples, and non-examples using written guidelines and video examples, (b) practicing coding using two-minute fragments, (c) discussing discrepancies and revising written guidelines if necessary, and (d) independent coding of 10-min training videotapes.⁶¹ Observer training continued until interobserver agreement was acceptable in two consecutive training videotapes.^62,63 Ongoing training sessions were held every two weeks to minimize observer drift.

Interobserver agreement was determined on an interval-by-interval basis by calculating prevalence-adjusted and bias-adjusted kappa⁶⁴ (PABAK). PABAK is an adjusted kappa coefficient that considers the effects of prevalence and bias. A prevalence effect exists when the proportion of agreements on occurrences (+) differs from that of non-occurrences (-) which results in a decreased value of kappa coefficient. A bias effect exists when observers disagree on the proportion of (non-)occurrences, which results in a higher value of kappa coefficient. PABAK takes into account these prevalence and bias effects. For fidelity of CPRT implementation and communication skills interobserver agreement was fair to excellent on average⁶⁵ (see Table 4).

Table 4. Interobserver agreement (PABAK) for fidelity of CPRT implementation and communication skills

		PABAK
Category		M	SD	Range
Fidelity of CPRT implementation	Total fidelity	0.63	0.26	0.20–1.00
	Maximizing student motivation	0.86	0.20	0.00–1.00
	Implementing three-term contingency learning trials	0.55	0.27	0.00–1.00
	Interspersing maintenance and acquisition trials	0.46	0.41	−0.56–1.00
Communication skills	Spontaneous initiations	0.80	0.13	0.53–1.00
	Spontaneous responses	0.66	0.16	0.33–1.00

Note. CPRT: classroom pivotal response teaching; PABAK: prevalence-adjusted bias-adjusted kappa; M: mean; SD: standard deviation.

Treatment Integrity

CPRT trainers collected data on treatment integrity after each training and coaching session using checklists. Mean percentage of treatment integrity was 91% for training sessions (SD = 7; range: 79–100) and 92% for coaching sessions (SD = 9; range: 65–100), indicating excellent treatment integrity.^65,66 For 33% of sessions an independent observer (i.e. research assistant) collected data on treatment integrity based on audiotapes of the training and coaching sessions to assess inter-observer agreement on treatment integrity using PABAK. Mean PABAK was 0.64 (SD = 0.21; range: 0.20–1.00), indicating adequate interobserver agreement on treatment integrity.^65,66

Data-analysis

Data on fidelity of CPRT implementation, communication skills, and maladaptive behavior were first examined visually to guide statistical analysis.⁶⁷ Visual analysis included systematic analysis of level, trend, and stability within and between subsequent phases for each teacher and student.⁶⁸ The split-middle method of trend estimation was used to determine baseline trend. Median percentages were compared to analyze changes in level between subsequent phases. In addition to visual analysis, statistical analysis was conducted using effect size Tau_novlap or Tau-U⁶⁹ which are single-case effect size measures that examine the proportion of nonoverlapping data between phases. In addition, Tau-U is able to control for baseline trends. Tau-U was calculated if visual analysis demonstrated a strong positive baseline trend (i.e. fidelity of CPRT implementation and communication skills) or strong negative baseline trend (i.e. maladaptive behavior). For each teacher and student effect sizes and corresponding p values were calculated for adjacent phase contrasts (i.e. baseline/intervention, intervention/post-intervention, and post-intervention/follow-up) using Single Case Research (SCR), a web-based calculator.⁷⁰ Analyses were two-tailed and p value was set at 0.05.

Results

Fidelity of CPRT Implementation

Data on fidelity of CPRT implementation are presented in Figure 1. Median fidelity percentages during baseline, intervention, post-intervention, and follow-up and values of Tau for individual teachers are provided in Table 5. Visual analysis indicated that baseline data were highly variable. Although baseline median fidelity percentages ranged from 0.00 to 30.00% across teachers, two teachers (T5 and T6) already demonstrated relatively high proficiency implementing CPRT prior to training, with fidelity percentages up to 70.00% or 80.00%. Baseline data also indicated a gradually increasing trend for these two teachers. During intervention, six teachers met the criterion of 80% correct use of CPRT in two consecutive coaching sessions according to the CPRT trainers’ ratings of fidelity within four (N = 3), five (N = 1), or six (N = 2) coaching sessions; one teacher demonstrated 80% correct use during the last coaching session. Intervention median fidelity percentages (as rated by naïve observers) ranged from 15.56% to 40.00% and increased significantly for three teachers (T3, T4, and T7), but there was a high degree of overlap between baseline and intervention fidelity percentages for the other four teachers and data were highly variable. These results indicate that the effect of CPRT training on fidelity was not consistent across teachers. During post-intervention, there was a variable decreasing trend for all teachers. Post-intervention median fidelity ranged from 15.00 to 55.00%. Follow-up data showed a variable decreasing trend for most teachers and median fidelity percentages ranging from 5.00% to 60.00%. Thus, post-intervention and follow-up data indicate that improvements in fidelity were not maintained immediately after training in CPRT or over time (i.e. three months later).

Table 5. Median percentages of fidelity of CPRT implementation and values of Tau

Teacher	Median percentage of fidelity of CPRT implementation				Tau
	Baseline	Intervention	Post-intervention	Follow-up	B-I contrast	I-PI contrast	PI-FU contrast
1	5.00%	20.00%	15.00%	5.00%	0.50	0.00	−0.13
2	5.00%	15.56%	45.00%	40.00%	0.52	0.58	−0.04
3	0.00%	30.00%	25.00%	25.00%	0.58*	−0.17	0.04
4	10.00%	20.00%	40.00%	-	0.63*	0.08	-
5	30.00%	40.00%	55.00%	60.00%	0.15	0.52	−0.17
6	20.00%	20.00%	25.00%	50.00%	−0.09	0.20	0.51
7	5.00%	25.00%	35.00%	25.00%	0.74*	0.10	−0.13

Note. CPRT: classroom pivotal response teaching; B: baseline; I: intervention; PI: post-intervention; FU: follow-up; ^a Tau-U; * significant at α = 0.05.

Figure 1. Percentage of CPRT fidelity during small-group instruction as rated by observers (squares) and trainers (crosses). Note. During intervention, data were collected from the third group session

When examining the different components of CPRT, there was variable fidelity across components, with teachers demonstrating higher proficiency maximizing student motivation and lower proficiency implementing three-term contingency learning trials and interspersing maintenance and acquisition trials prior to training CPRT (see Table 6). Although fidelity percentages for implementing three-term contingency learning trials and interspersing maintenance and acquisition trials improved during intervention, proficiency using these components remained lower.

Table 6. Median percentages of CPRT fidelity components across teachers

Category	Median percentages of fidelity of CPRT implementation
	Baseline	Intervention	Post-intervention	Follow-up
Maximizing student motivation	80.00%	90.00%	90.00%	90.00%
Implementing three-term contingency learning trials	30.00%	50.00%	60.00%	50.00%
Interspersing maintenance and acquisition trials	33.33%	55.56%	55.56%	66.67%

Note. CPRT: classroom pivotal response teaching.

Communication Skills

Data on student initiations and responses are presented in Figure 2. Median percentages of initiations and responses during all phases and values of Tau for individual students are provided in Supplemental Material.

Figure 2. Percentage of child initiations (squares) and responses (circles) during small-group instruction

Initiations

Visual analysis demonstrated a rapidly increasing baseline trend for one student (S2). The baseline median percentage of initiations ranged between 3.33% and 20.00%, indicating that students infrequently initiated interaction during small-group instruction prior to CPRT. However, baseline data were variable for all students. During intervention, there was an increasing trend for four students (S1, S3, S4, and S6), but the percentage of initiations increased significantly for only one student (S3). The intervention median percentage of initiations ranged from 3.33 to 36.67% and data remained variable for most students. During post-intervention, there was a variable increasing trend for five students (S2, S3, S4, S6, and S7), but the median percentage of initiations (range 6.67–25.00%) decreased for four of these students (S2, S3, S6, and S7). For one student (S3), this decrease in level was significant, indicating that this student did not maintain his skills. Follow-up data were variable for most students and showed an increasing trend for two students (S5 and S6) and a decreasing trend for two other students (S2 and S7), but the median percentage of initiations did not change significantly compared to post-intervention and ranged from 3.33% to 33.33%.

Responses

Visual analysis revealed a gradually increasing trend during baseline for two students (S6 and S7). The baseline median percentage of responses ranged from 16.67 to 46.67%, indicating that most students produced responses prior to CPRT. Baseline data were variable for all students. During intervention, there was an increasing trend for three students (S1, S5, and S6), a decreasing trend for three students (S2, S3, and S7) and no trend for one student (S4). The median percentage of responses ranged from 31.67% to 50.00% and did not change significantly compared to baseline. Intervention data were variable for all but one student (S2). During post-intervention, there was a decreasing trend for five students (S1, S2, S3, S6 and S7) and two of these students (S2 and S3) showed significantly less responses compared to intervention. The median percentage of responses varied between 28.33% and 55.00%. Post-intervention data were variable for most students. Follow-up data showed a decreasing trend for all students, but the percentage of responses did not change significantly compared to post-intervention. The follow-up median percentage of responses ranged from 28.33% to 53.33% and data were variable for most students.

Maladaptive Behavior

Data on student maladaptive behavior are presented in Figure 3. Median percentages of maladaptive behavior during each phase and values of Tau for individual students are provided in Supplemental Material. Visual analysis indicated a rapidly decreasing baseline trend for five students (S1, S3, S4, S5, and S6). During baseline, median percentages of maladaptive behavior varied between 0.00% and 50.00%, indicating variability in maladaptive behavior across students. During intervention, median percentages of maladaptive behavior ranged from 0.00% to 50.00% and decreased significantly for one student (S3). During post-intervention and follow-up, median percentages ranged from 0.00% to 50.00% and 5.00% to 60.00% respectively and did not change significantly.

Figure 3. Percentage of maladaptive behavior as experienced by teachers between circle and recess time

Social Validity

Six (out of seven) teachers completed the social validity questionnaire post training. Teachers rated their training in CPRT as highly informative (M = 4.32) and pleasant (M = 4.06) and would recommend the training to their colleagues (M = 4.17). With regard to the training components, video feedback was rated most informative during both group sessions (M = 5.00) and individual coaching sessions (M = 4.67). Role-plays were rated least informative (M = 2.83). Four (out of six) teachers reported that they spend a lot of time preparing CPRT lessons and recording videotapes during training. Teachers had positive attitudes toward CPRT post training (M = 4.33) and indicated that they still implemented CPRT in their classroom (M = 4.17). Two (out of six) teachers specifically reported that CPRT benefited their students. One teacher stated ‘I noticed a positive change in student engagement: students are more engaged and motivated during my lessons.’ Another teacher stated ‘It’s great to see the enormous effect of CPRT on my student.’ Overall, teachers rated the training in CPRT with an 8.33.

Discussion

In this pilot study, we examined the effectiveness of teacher training in CPRT on teacher fidelity of implementation during small-group instruction, and communication skills and maladaptive behaviors of students with ASD in schools for special (primary) education in the Netherlands. Although teachers reported a high level of satisfaction with the CPRT training, fidelity of CPRT implementation increased significantly for only three teachers, indicating that the effect of CPRT training on fidelity was not consistently replicated across teachers. In addition, improvements in fidelity were not maintained during post-intervention and follow-up. Finally, changes in students’ communication skills and maladaptive behavior were small and non-significant.

Whereas previous studies reported improvements in fidelity of CPRT implementation as a result of teacher training in CPRT,^{38,39,40,41,42} our findings were inconsistent. Therefore, it remains unclear whether a CPRT training combining group instruction and individual coaching sessions is effective in teaching Dutch special education teachers to implement CPRT with adequate fidelity in a setting with multiple students. At least two factors could explain our inconsistent findings. First, there was a high degree of variability in teachers’ individual implementation fidelity during all study phases. This lack of stability may have been the result of variability in teaching in activities and the number of students participating in a teaching activity. Teachers were asked to implement CPRT during instruction to a small group of students, including their target student, but were allowed to vary the subject taught (i.e. reading, writing, mathematics, history, geography, or science), teaching activities, and the number of students attending their instruction to ensure feasibility. This variability in teaching activities and students could have resulted in variable fidelity of CPRT implementation. Second, our baseline data indicated that teachers demonstrated relatively high proficiency implementing CPRT or components of CPRT prior to training. For example, two teachers nearly met fidelity criteria during baseline (i.e. 80% fidelity in two consecutive sessions). Also, most teachers were able to maximize student motivation during baseline, particularly by facilitating turn taking between students, suggesting that this component is naturally incorporated in teaching. This proficiency implementing CPRT prior to training has resulted in overlap between baseline and intervention fidelity percentages and may explain why several teachers did not seem to benefit from training in CPRT.

We also found variability in implementation fidelity of CPRT components during and post training in CPRT. Teachers consistently demonstrated lower proficiency implementing three-term contingency learning trials and interspersing maintenance and acquisition tasks and higher proficiency maximizing student motivation. Anecdotal evidence suggests that teachers particularly had difficulties providing direct and contingent reinforcement upon the child’s response or reasonable attempt, which is consistent with Suhrheinrich et al.^32,42 who also reported moderately lower fidelity for consequent components of CPRT. In addition, teachers found it hard to intersperse maintenance and acquisition tasks and often presented only two different cues in two consecutive intervals. They particularly used instructions and questions, which are cues commonly used in classrooms⁷¹; other cues (e.g. comments, facial expressions, or environmental arrangement) were used less often.

In addition, our results indicated that the three teachers that showed improvements in fidelity of CPRT implementation during training did not maintain their fidelity levels immediately after training or over time. Although a fidelity of implementation level at or greater than 80% is considered acceptable⁶⁶ and used in most studies on (C)PRT,^{39,42,43,72,73} it is possible that demonstration of this level of fidelity in only two consecutive sessions is insufficient to ensure that teachers maintain their fidelity levels once feedback is faded.

This lack of maintenance and the variability in fidelity of implementation across CPRT components and sessions have several implications for future research and training programs. First, training in CPRT should focus on components that teachers implement with low fidelity prior to training, for example by allocating more time to discuss, model, practice and provide feedback on the implementation of these components during group training sessions and individual coaching sessions. Thus, training in CPRT should be adapted and individualized based on teachers’ performance during baseline. Second, CPRT training should include an extensive introduction to behavioral principles as the foundation for CPRT to emphasize the rationale of each CPRT component and their necessity for student learning.⁷⁴ This is particularly important when trainees are not familiar with ABA,⁷⁵ which applies to most, if not all, special education teachers in the Netherlands. Third, self-management strategies could be added to the current CPRT training to train teachers to discriminate between correct and incorrect implementation of ‘difficult’ CPRT components and record their own behavior in order to improve their fidelity of CPRT implementation.^17,76,77 Behavioral Skills Training (BST), a training package that combines instruction, modeling, rehearsal, and feedback, may be indicated as training method, as it can be individualized to train specific components, is implemented until trainees meet predetermined criteria, and is associated with consistent improvements in implementation fidelity.^78,79 Fourth, future research should investigate in how many consecutive sessions teachers should meet the criterion of 80% fidelity to ensure that they maintain this level of fidelity post training. Finally, further research is necessary to examine how teaching activities and the number of students participating in a teaching activity affect fidelity of implementation and to investigate how teachers can be taught to implement CPRT with high fidelity in a variety of teaching activities throughout the day with various students.

Trainers’ ratings of CPRT fidelity during coaching sessions differed from observer-rated fidelity. This finding was unexpected as trainers and observers used an identical recording system to code fidelity of CPRT implementation and observers were trained in using the recording system by one of the trainers (i.e. first author) until interobserver agreement was acceptable in two consecutive videotapes. We did not collect interobserver agreement data on trainers’ ratings of fidelity and thus, these ratings may be biased. However, as observers (i.e. research assistants) were not trained in (C)PRT and had no clinical experience providing (C)PRT in contrast to both trainers, it is also possible that a difference in experience explains why trainers and observers applied the fidelity recording system differently.⁸⁰ This finding implies that the recording system may have been too complex. Complexity of the recording system for fidelity of CPRT implementation primarily concerns coding sequences of correctly implemented CPRT components. Whereas fidelity is generally determined by coding each component separately,^38,42 we coded these sequences to ensure that teachers implemented three-term contingencies, as these increase the frequency of responding and thus to enhance learning.^54,81 A recording system for fidelity of implementation should be accurate as well as feasible for use in both research and clinical practice. Future research should explore how the recording system used in the present study could be simplified.

In contrast to previous studies,^38,43 we found no improvements in students’ behavior as a result of CPRT: changes in communication skills and maladaptive behavior were small and non-significant. CPRT may have been ineffective in improving students’ behavior, but variability in fidelity of CPRT implementation may also explain this finding. Implementation fidelity is associated with student outcomes, with higher fidelity leading to better intervention outcomes.^19,82,83 Fidelity does not have to be perfect to obtain positive results,¹⁹ but it is possible that fidelity of CPRT implementation in our study was too variable or too low to produce changes in students’ behavior. Furthermore, we may not have found improvements in communication skills, because the number of students participating in small-group instruction varied across sessions. If the number of students increases, the number of teacher-created opportunities to initiate or respond per student may decrease, which could affect learning rates. Increasing the number of opportunities could result in increased student engagement, responding, and initiating that, in turn, may lead to lower levels of maladaptive behavior.^84,85 Thus, it is not possible to determine the effectiveness of CPRT on students with ASD. Further examination of the effectiveness of CPRT on students’ behavior is warranted. Specifically, it important to investigate the minimum level of CPRT fidelity and optimum number of students to produce changes in student skills during small-group instruction.

There are several limitations to this study. First, despite the fact that we used an experimental design, we did not demonstrate experimental control and thus, we cannot conclusively determine the effectiveness of teacher training in CPRT. Second, we did not set a mastery criterion for the role-plays in the group training sessions and therefore it is unclear whether teachers were able to implement components of CPRT correctly before practice in their own classroom during small-group instruction. Third, although interobserver agreement for observers’ ratings of teacher fidelity and students’ behavior was fair to excellent on average, there were several instances where interobserver agreement was below acceptable levels (i.e. PABAK < 0.60), which has impacted the accuracy of our data. Fourth, we did not collect data on student’s individual social communication goals. Finally, data on maladaptive behavior were collected by teachers and interobserver agreement data were not collected. Therefore, these data may not represent true values of maladaptive behavior but rather how teachers experienced student maladaptive behavior.

In summary, although teachers were highly satisfied with the training in CPRT, its effectiveness in teaching special education teachers in the Netherlands to implement CPRT with adequate fidelity in a group setting remains inconclusive. Consequently, the effectiveness of CPRT on students with ASD could not be determined. In future studies, training in CPRT should be adapted and individualized based on teachers’ skill level prior to training in CPRT to achieve high levels of fidelity for all components of CPRT, in a variety of teaching activities, with varying numbers of students, and throughout the day. Further research is also warranted to determine the effectiveness of CPRT on students with ASD to improve their social, behavioral, and academic outcomes.

Author Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and data analysis were performed by Rianne Verschuur and Bibi Huskens. The first draft of the manuscript was written by Rianne Verschuur and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Conflict Of Interest

The authors declare that they have no conflict of interest.

Ethics Approval

The study was registered in the Netherlands Trial Register (NL5886) and approved by the Ethics Committee of the Faculty of Social Sciences of the Radboud University in Nijmegen, the Netherlands (EC2013-1304-100a). All procedures performed were in accordance with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Acknowledgments

The authors thank the teachers, students, and schools who participated in this study. We also like to thank Margreet Weide for her assistance during data collection and for coding videos and Malon Morsinkhof, Fleur Rijnhart, Maureen van Rossum, and Anke Suijkerbuijk for coding videos.

Availability Of Data

The data that support the findings of this study are available from the corresponding author upon reasonable request.