Effects of Augmentative and Alternative Communication on Challenging Behavior: A Meta-Analysis

Abstract

The purposes of this review were to (a) use meta-analytic procedures to examine the effectiveness of single-case AAC intervention research on challenging behaviors exhibited by individuals with disabilities, (b) identify study characteristics that moderate intervention effects, and (c) evaluate the quality of research. The authors provided inferential and descriptive analyses of 54 studies representing 111 participants and estimated effect sizes using the Nonoverlap of All Pairs (NAP) method. Overall, AAC interventions were found to be equally effective across a broad spectrum of participants and interventions. AAC interventions were more effective with younger children than with adults. Interventions in which functional behavior assessments (FBA) were used had significantly larger effect sizes than those that did not use FBAs. Further, functional communication training interventions resulted in larger effect sizes than Picture Exchange Communication System interventions. Descriptive analysis revealed that (a) interventions often occurred in atypical environments (e.g., therapy room, experimental room) and with atypical interventionists (e.g., therapists, researchers), and (b) numerous studies did not exhibit quality characteristics of single-case research.

Keywords::

• Augmentative and alternative communication
• Challenging behavior
• Single-case research
• Meta-analysis

Introduction

A growing body of evidence supports the notion that unresolved challenging behavior of individuals with and without disabilities leads to negative outcomes associated with educational achievement, vocational success, and social relationships (e.g., Campbell, 1995; Dunlap et al., 2006). Furthermore, the prevalence rates of individuals with disabilities who experience challenging behavior are quite high: 10%–40% of young children (Fox & Smith, 2007) and 20%–40% of individuals in secondary settings (NLTS2, 2006). These data that reflect poor outcomes and elevated prevalence rates associated with challenging behavior have prompted research efforts addressing the development and implementation of effective practices to resolve challenging behavior.

Researchers have suggested that challenging behavior is associated with poorly developed communication (e.g., Carr & Durand, 1985; Carr et al., 1997). Children with disabilities may not develop communication competence typical of children without disabilities (Bambara & Kern, 2005; Carr et al., 1997). Fairly strong research evidence exists that suggests children who experience delays in communication skills are more likely to engage in challenging behavior (e.g., Beitchman et al., 1996; Carson, Klee, Lee, Williams, & Perry, 1998; Kaiser, Cai, Hancock, & Foster, 2002). Carr et al. (1997) propose that findings from this body of literature lend support to the communication hypothesis of problem behavior, whereby “problem behavior functions as a primitive form of communication for those individuals who do not yet possess or use more sophisticated forms of communication” (p. 22). Given the communicative function of challenging behavior, interventions that target communication skill development are likely to result in the reduction of challenging behavior.

In light of the evidence supporting the association between challenging behavior and communication, researchers have investigated communication-based interventions to address challenging behavior of individuals with disabilities; many of these investigations apply augmentative and alternative communication (AAC) strategies. Overall, these primary studies¹ demonstrate the effectiveness of AAC as an intervention to reduce challenging behavior. However, meta-analytic research is necessary to explore further the efficacy of such interventions and to examine the diverse characteristics (i.e., participant, intervention, intervention outcome, and quality-of-study) found among these primary studies. Doing so assists in the identification 
of relationships between study characteristics and the magnitude of intervention efficacy (i.e., whether study characteristics are moderators of effect size; Cooper, Hedges, & Valentine, 2009). Single-case research design often is selected as the appropriate methodology for these 
primary studies of AAC intervention research with individuals who have disabilities due to the low incidence of individuals who both have challenging behavior and use AAC and their highly individualized circumstances. Meta-analytic methods for single-case research are widely used in research involving individuals with disabilities (Gast, 2010; Scruggs, Mastropieri, & Casto, 1987); this is true despite ongoing debates about whether meta-analysis is appropriate for single-case research due to the absence of a standard effect size metric (Kazdin, 2011).

The purpose of the current review was to report the results of a meta-analysis of single-case research interventions in which AAC strategies were applied to address challenging behaviors exhibited by individuals with disabilities. Specifically, this review was designed to achieve three research aims. First, the authors sought to estimate the overall effect of different AAC interventions on challenging behavior of individuals of any age across a broad spectrum of disabilities. Several recent literature reviews have evaluated the efficacy of interventions targeting reduction of challenging behavior and AAC skills. However, these reviews are limited in several ways. Typically, they (a) assessed a broad range of behavioral interventions without a specific focus on AAC (e.g., Campbell, 2003; Didden, Korzilius, 
van Oorsouw, & Sturmey 2006; Heyvaert, Maes, & Onghena, 2010); (b) limited assessment to specific age groups (e.g., Horner, Carr, Strain, Todd, & Reed, 2002), diagnoses (e.g., Ganz, Davis, Lund, Goodwyn, & Simpson, 2012), or AAC interventions (e.g., Preston & Carter, 2009); or (c) provided only a descriptive summary of study characteristics (e.g., Snell, Chen, & Hoover, 2006). Of those reviews examining AAC interventions, a majority focused on outcome variables associated with communication and speech (e.g., Flippin, Reszka, & Watson, 2010; Millar, Light, & Schlosser, 2006). Only a few reviews have assessed challenging behavior outcomes associated with AAC interventions (e.g., Ganz et al., 2012); however, these are limited 
to studies of individuals with specific diagnoses and 
specific types of AAC intervention.

The second aim was to identify study characteristics that moderate the overall effect of AAC intervention on challenging behavior. Three categories were included in these analyses: (a) Participant characteristics (gender, age, diagnoses, pre-intervention communication characteristics, pre-intervention behavior); (b) Intervention characteristics (setting, implementation method, interventionist, intervention type, experimental design, targeted communication mode, behavior assessment); and (c) Intervention Outcome characteristics (targeted communication level, targeted behavior). Three factors contributed to the selection of these particular study characteristics. First, because a priori specification of characteristics that may be germane to a literature review is not always feasible (Cooper et al., 2009), the authors selected a broad range of study characteristics as opposed to a narrow group of characteristics. 
Second, many of these study characteristics typically have been included in existing literature reviews that address behavioral intervention research (e.g., Didden et al., 2006) and communication-based intervention research (e.g., Snell et al., 2006). Third, several selected study characteristics, such as intervention that is linked to functional behavior assessment, have been found to moderate intervention effects in other reviews (e.g., Campbell, 2003; Didden, Duker, & Korzilius, 1997).

The authors’ selection of AAC intervention categories warrants further discussion. The authors identified four categories of intervention to include in the review: (a) functional communication training (FCT; Carr & Durand, 1985), (b) choice making, (c) Picture Exchange Communication System (PECS; Bondy & Frost, 1993, 1994) and (d) milieu training (Kaiser, Hancock, & Nietfeld, 2000). FCT intervention is designed to address both communication and challenging behavior, whereas choice making, PECS, and milieu training interventions are considered to address communication exclusively. Nonetheless, the authors felt that inclusion of these three categories of communication-based intervention was appropriate, due to the availability of single-case research intervention studies and literature reviews in which challenging behavior was treated as an outcome variable (e.g., Hart & Banda, 2009; Shogren, Faggella-Luby, Bae, & Wehmeyer, 2004).

The third aim was to evaluate the quality of AAC intervention research studies that were included in the current review. Descriptive summaries of study quality allow for the evaluation of potential issues with study credibility and inadequacies in reporting. Previous literature reviews in the areas of communication and behavioral intervention have evaluated the quality of studies and report varying levels of study quality (e.g., Flippin et al., 2010; Preston & Carter, 2009; Snell et al., 2006). Five Quality of Study characteristics were evaluated in the current study, each recognized as a quality indicator 
for single-case research (e.g., Horner et al., 2005): (a) reliability (i.e., measurement of acceptable reliability or interobserver agreement associated with each dependent variable), (b) intervention fidelity (i.e., measurement of acceptable fidelity of implementation for each independent variable), (c) social validity (i.e., measurement of the value or practical nature of the intervention), (d) generalization (i.e., measurement of skill acquisition to new partners, settings, responses, etc.), and (e) maintenance (i.e., measurement of skill acquisition at least 3 months after intervention).

Method

Search Strategy

The authors identified AAC intervention studies that addressed challenging behavior through searches of the following online databases: PsychNet (PsycINFO, PsycArticles), EBSCO Host (Education Research Complete, ERIC, Psychology and Behavioral Sciences Collection, and Academic Search Complete), and 
Medline. Search terms² included combinations of 
relevant synonyms and appropriate related terms associated with AAC (e.g., “functional communication training,” “Picture Exchange Communication System”) and challenging behavior (e.g., “aberrant behavior,” “problem behavior”). The authors did not limit the literature search to a specific time period, language, or geographic region. In addition to conducting electronic searches, the authors manually searched the reference lists of articles retrieved from the online database search to locate additional studies with potential relevance. To reduce the threat of publication bias (i.e., the notion that research appearing in the published literature is systemically unrepresentative of the population of completed studies), the authors used search procedures to locate both published and unpublished studies, including dissertations and thesis papers (Cooper et al., 2009). Specifically, the authors selected search options within database search engines to explore a broad set of research literature formats (e.g., dissertations).

Inclusion and Exclusion Criteria

After eliminating 654 articles consisting of duplicates or irrelevant sources (i.e., book chapters, articles from unrelated research areas) from the literature database, the authors reviewed abstracts from the 355 remaining articles to judge studies against four inclusion criteria. Studies only were included that: (a) employed an experimental single-case research design (e.g., ABAB, multiple baseline, alternating treatment); (b) addressed expressive communication through the use of AAC intervention; (c) included one or more individuals with a disability; and (d) included one or more relevant dependent measures of challenging behavior. Furthermore, studies were excluded if (a) the study did not provide sufficient clarity of information to permit effect size calculation; (b) the complexity of the single-case research design did not permit effect size calculation [e.g., several phases introduced without a return to baseline preventing a comparison of baseline to AAC condition(s)]; (c) the initial control phase contained fewer than three data points; or (d) a description of participant characteristics was absent. Also, during the coding phase when the full article was examined, if one or more participants within any given study did not meet the inclusion criteria, those participants’ data were excluded from participant-level descriptive and inferential analyses.

To assess inter-rater agreement, both authors judged 30% (n = 105) of the 355 identified articles against the inclusion criteria. The first author, the primary coder, was a doctoral student studying special education and the second author, the secondary coder, was a special education faculty member; both had experience with AAC and behavior interventions for individuals with varying disabilities. To account for inclusion ratings due to chance, Cohen's kappa was used to quantify the degree of agreement between raters (Cooper et al., 2009; Light, 1971). When coders identify a study characteristic as occurring frequently (or infrequently), as often happens when coding data in a review such as this, the chance that the coders will agree on any given code is inflated; kappa corrects for chance agreement, thus yielding a conservative estimate of inter-rater agreement. Based on interpretive guidelines provided by Landis and Koch (1977), the mean agreement across all four inclusion criteria was interpreted as being “almost perfect” (mean kappa = 0.87; range = 0.67–0.97; p. 165). To verify agreement on the inclusion criterion that had earned the lowest agreement score during the first round of review (i.e., the study addressed expressive communication through use of AAC intervention or treatment), both authors judged an additional random sample of abstracts (30%) against this inclusion criterion. The second round of review resulted in almost perfect agreement (kappa = 0.82) on this inclusion criterion. The primary coder independently rated all remaining articles against the four inclusion criteria.

A total of 81 studies met the inclusion criteria through this review of the abstracts. An additional group of studies (n = 27) was excluded from the literature database after the authors reviewed the full article during the coding phase; a total of 54 studies³ remained in the qualifying literature database. Commonly identified reasons for excluding articles included: (a) use of non-experimental single-case research designs (i.e., AB design), (b) implementation of communication-based interventions without use of AAC (i.e., functional communication training using speech), and (c) no measurement of challenging behavior but measurement of other behaviors (i.e., targeted communicative behavior).

Coding Procedures

Coding instrument. The authors coded studies in the 
literature database using the AAC and Challenging Behavior Research Coding Form (Walker & Snell, 2012)⁴ in order to identify descriptive information relating to study characteristics; the effect sizes associated with measures of challenging behavior were calculated later. The coding instrument was based in part on the work of Snell et al. (2010). Because the focus of this review was on challenging behavior as well as AAC intervention, coding items were modified accordingly. The 
coding instrument contained 24 individual coding items organized under five categories: (a) Participant characteristics (gender, age, diagnoses, pre-intervention communication characteristics, pre-intervention behavior); (b) Intervention characteristics (setting, implementation method, interventionist, intervention type, experimental design, targeted communication mode, behavior assessment); (c) Intervention Outcome characteristics (targeted communication level, targeted behavior); (d) Quality of Study characteristics (reliability, generalization, maintenance, social validity, intervention fidelity); and (e) Effect Sizes. Although an independent quality rating scale was not used (see Flippin et al., 2010), the coding instrument included items addressing study features that aligned with several single-case research quality guidelines described by Horner et al. (2005). To refine the coding instrument, each author independently coded a small subset of randomly selected articles from the literature database (n = 5) and made necessary changes to the coding instrument after discussing problematic areas. The coding items remained the same, whereas the operational descriptions of each item were modified for clarity. A 
sample of the operational definitions for instrument coding items is presented in Table I.

Table I. A Sample of Operational Coding Definitions.

Coding item	Definition
Expressive communication levels
Multiword	Non-rote combination of two or more words/symbols – grammatical constructions
Emerging language	Single words (any mode) or rote word combination (e.g., “Thank you.”)
Prelinguistic	No real words in any mode
Challenging behaviors
Destructive	Behaviors that are harmful or threaten the safety of the individual or others. Physical aggression (e.g., hitting, kicking, pulling, biting, scratching, spitting, breaking or throwing objects, self-injury, etc.)
Disruptive	Behaviors that do not immediately endanger the individual or others, but do interfere with everyday activities and experiences. Verbal aggression (e.g., threats, teasing, etc.) Disruptive verbal behavior (e.g., yelling, screaming, loudly talking over others, interrupting, crying as part of tantrum, etc.) Disruptive physical behavior (e.g., throwing oneself on the floor as part of a tantrum, running away from classroom, poking others, etc.)
Distracting	Behaviors that deviate from what is typically expected from individuals of the same age. Nonparticipation (e.g., ignoring teacher's request for action, saying “no” to others) Stereotypical behavior (e.g., pacing, rocking, etc.)
Intervention implementation method
Massed trial	One trial follows the next trial; trials scheduled by session rather than by natural opportunities.
Distributed trial	Trials are separated by variable amounts of time and are taught as part of a routine.
Decontextualized	Removed from natural communication environment; contrived for study rather than under natural conditions; pull-out; conditions are manipulated according to time or setting; individuals present that are strikingly different than scheduled routines.
Contextualized	Conducted in a natural communication environment; generally natural but experimenter can set up preferred materials in a classroom play area in their environment.
Functional behavior assessment
Descriptive	Direct observation of challenging behavior in the natural environment to identify conditions under which challenging behavior occurs and is maintained.
Experimental	Delivery of variables hypothesized to maintain challenging behavior contingent on the occurrence of challenging behavior within brief sessions.
Generalization	Skill or behavior transfer to new stimuli (partners, settings, materials) and responses.
Maintenance	Long-term retention of a target skill or behavior measured at least 3 months after conclusion of intervention.

Coding reliability and analysis. From the 54 qualifying studies, 25 studies (46%) were randomly selected to assess inter-rater agreement. Each author independently coded the studies using the revised coding instrument. Inter-rater agreement was calculated using Cohen's kappa for categorical coding items and Pearson's product-moment correlation coefficient for continuous coding items. Agreement was interpreted as almost perfect across all categorical coding items (mean kappa = 0.91; range = 0.80–1.00) and strong across continuous coding items (mean correlation = 0.96; range = 0.90–1.00)⁵. 
All cases of disagreement were discussed until resolved and the correct code was agreed upon.

Effect size. As little consensus exists regarding the appropriate calculation of effect sizes for single-case research (Gast, 2010; Kazdin, 2011), the authors selected a nonoverlap approach to analyze intervention outcomes: Nonoverlap of All Pairs (NAP; Parker & Vannest, 2009). Parker and Vannest (2011) indicate “nonoverlap indices are more robust than indices of mean or median level shifts across phases” (p. 2). NAP has been field tested and shown to equal or outperform other overlap indices (i.e., PND, PEM, and PAND) in its accuracy and robust correlation with visual analysis (see Parker & Vannest, 2009). Further, several recent literature reviews and single-case research studies have applied NAP to estimate intervention effects (e.g., Burns, Zaslofsky, Kanive, & Parker, 2012; Ganz et al., 2012; Ramdoss et al., 2011). The NAP statistic is calculated as the percentage of all pair-wise comparisons across phases, wherein NAP equals the number of comparison pairs showing no overlap, divided by the total number of comparisons. NAP can be calculated directly as Area Under the Curve by using a Receiver Operator Characteristics (ROC) diagnostic test which is available through most statistical analysis programs (Parker & Vannest, 2009; Parker, Vannest, & Davis, 2011); the authors used this approach to calculate NAP with SPSS 12.0 for 
Windows. Interpretation of NAP scores is as follows: weak effects: 0–.65; medium effects: .66–.92; large or strong effects: .93–1.0 (Parker & Vannest, 2009, p. 364).

To calculate NAP using ROC analysis, the authors used UnGraph (2004) to identify the value of each individual data point in graphs of challenging behavior so that raw data point values could be imported into SPSS for analysis purposes. UnGraph is a digitizing software that allows for the extraction of numerical data from graphs when original raw data are not otherwise available; UnGraph has been found to have high reliability and validity (Shadish et al., 2009).

The unit of analysis was a comparison between a control phase and an intervention phase for a given participant. Due to the unique features of each single-case research design, the authors applied design-specific rules to identify appropriate phases from which to extract data point values. For graphs depicting ABAB/reversal designs or variations thereof, an effect size was calculated for each A-B and B-A pairing to measure effect of intervention implementation and withdrawal of intervention (Parker, Vannest, & Brown, 2009); resulting effect sizes were then averaged. The authors calculated separate effect sizes for each baseline-intervention comparison in multiple baseline/probe designs; resulting effect sizes were averaged. For graphs portraying alternating treatment designs or variations thereof, an effect size was calculated for each separate treatment and then averaged. If a control phase was absent and the study included an experimental functional behavior assessment (FBA) with graphed functional analysis (FA) data, FA data were used as a substitute for baseline data if the following conditions were met: (a) FA data were measured in the same units (e.g., percentage of intervals) and (b) FA conditions were identical to those in the intervention phase except that the independent variable was absent. To calculate the overall effect size associated with AAC intervention and reductions in challenging behavior, the authors averaged effect sizes (NAP values) across all participants. Effect sizes specific to individual study characteristics were calculated 
using the same approach and reported as M NAP in Tables II, III, and IV.

Table II. Findings for Participant Characteristics.

Participant characteristics (n)	M NAP	Level of effect	SD	Mean rank	χ²
Gender
Female (32)	.89	Moderate	.19	62.0	−1.27^b
Male (79)	.86	Moderate	.18	53.6
Age
Greater than 12–18 years (23)	.91	Moderate	.08	45.1	18.02***
5 years or younger (35)	.86	Moderate	.17	61.6
Greater than 5–12 years (38)	.89	Moderate	.18	67.5
Greater than 18 years or older (15)	.74	Moderate	.27	30.5
Diagnoses^a
EBD (9)	.90	Moderate	.09	81.7	1.68
IDD (83)	.87	Moderate	.19	79.0
ASD (39)	.86	Moderate	.17	73.0
GD (10)	.84	Moderate	.20	65.8
SI (10)	.73	Moderate	.37	67.7
Expressive communication levels, pre-intervention
Multiword (20)	.94	Strong	.07	45.3	6.55*
Prelinguistic (22)	.89	Moderate	.17	46.3
Emerging (38)	.85	Moderate	.17	32.9
Communication modes, pre-intervention
Non-symbolic (36)	.90	Moderate	.14	97.7	1.49
AAC – unaided (21)	.89	Moderate	.15	96.5
Speech (41)	.87	Moderate	.16	94.0
AAC – no speech output device (14)	.87	Moderate	.13	104.0
Behavior, pre-intervention
Distracting (17)	.90	Moderate	.15	84.2	2.82
Disruptive (28)	.89	Moderate	.14	73.5
Destructive (95)	.85	Moderate	.19	67.2

Note. NAP Weak effects: 0–0.65; Medium or moderate effects: 0.66–0.92; Large or strong effects: 0.93–1.00.


χ² values are derived from Kruskal-Wallis test.

^aEBD = emotional behavior disorder, IDD = intellectual or developmental disability, ASD = autism spectrum disorder, GD = genetic disorder, SI = sensory impairment, ^bZ score (Mann-Whitney U).

*p < .05, **p < .01, ***p < .001 (alpha level refers to comparison across moderator levels).

Table III. Findings for Intervention Characteristics.

Intervention characteristic (n)	M NAP	Level of effect	SD	Mean rank	χ²
Intervention setting
Community (10)	.94	Strong	.08	85.0	3.63
Pull-out (31)	.88	Moderate	.12	67.7
Classroom (54)	.87	Moderate	.19	76.4
Other (22)	.85	Moderate	.18	69.8
Home (25)	.80	Moderate	.24	61.7
Implementation method
Massed trial (65)	.86	Moderate	.17	49.0	−0.02
Distributed trial (32)	.85	Moderate	.23	48.9
Decontextualized (42)	.86	Moderate	.15	48.5	−0.74
Contextualized (59)	.86	Moderate	.21	52.8
Person(s) delivering intervention
Paraprofessional (16)	.95	Strong	.07	83.8	8.06
Teacher (34)	.94	Strong	.08	82.3
Experimenter (27)	.89	Moderate	.17	73.9
Therapist (46)	.86	Moderate	.21	66.7
Parent (20)	.83	Moderate	.09	54.7
Type of intervention
FCT (93)	.87	Moderate	.18	53.4	−2.13^a*
PECS (9)	.74	Moderate	.22	31.9
Expressive communication, targeted
Speech (16)	.89	Moderate	.09	64.8	6.71
AAC – speech output device (31)	.89	Moderate	.20	85.4
AAC – no speech output device (61)	.85	Moderate	.16	65.5
Non-symbolic (10)	.85	Moderate	.14	58.8
AAC – unaided (23)	.84	Moderate	.27	75.9
Functional assessment
Yes (98)	.88	Moderate	.17	60.3	−3.05^a**
No (13)	.72	Moderate	.21	31.3
Type of functional behavior assessment
Experimental (75)	.90	Moderate	.24	55.5	−0.52^a
Descriptive (23)	.85	Moderate	.23	52.1

Note. NAP Weak effects: 0–0.65; Medium or moderate effects: 0.66–0.92; Large or strong effects: 0.93–1.00.

χ² values are derived from Kruskal-Wallis test.

^aZ score (Mann-Whitney U).

*p < .05, **p < .01, ***p < .001 (alpha level refers to comparison across moderator levels).

Table IV. Findings for Intervention Outcome Characteristics.

Intervention outcome characteristic (n)	M NAP	Level of effect	SD	Mean rank	χ²
Expressive communication levels, targeted
Emerging (88)	.86	Moderate	.19	57.1	5.18
Prelinguistic (8)	.82	Moderate	.14	40.3
Multiword (11)	.70	Moderate	.34	38.9
Behavior targeted
Distracting (27)	.89	Moderate	.15	73.7	1.24
Destructive (84)	.88	Moderate	.15	72.4
Disruptive (34)	.85	Moderate	.17	63.9

Note. NAP Weak effects: 0–0.65; Medium or moderate effects: 0.66–0.92; Large or strong effects: 0.93–1.00. χ² values are derived from Kruskal-Wallis test. *p < .05, **p < .01, ***p < .001 (alpha level refers to comparison across moderator levels).

Moderator Analysis

To compare effect sizes across participant, intervention, and intervention outcome characteristics, the authors used SPSS 12.0 for Windows to conduct two nonparametric analyses: (a) Mann-Whitney U test (appropriate for a two independent groups design), and (b) Kruskal-Wallis one-way ANOVA (appropriate for a k independent groups design, whereby k is equal to or greater than three). Characteristics with small ns (eight or fewer) were not included in the moderator analysis.

Results

The results of this meta-analysis are summarized in two sections: (a) moderator analysis in which results of the inferential analyses are presented and (b) descriptive analysis which contains a summary of study characteristics. A total of 54 studies representing 111 participants were included in each type of analysis. It should be noted that not all studies reported information relevant to all 24 coding items. Further, for many coding items it was possible to code one or more characteristics for any given participant, thus characteristic percentages total more than 100% for certain characteristics.

Moderator Analysis

The mean effect size (NAP value) across participants was .88 (SD = .18; range = .11–1.00), which is indicative of a medium or moderate level of effect (Parker & 
Vannest, 2009). All participant, intervention, and intervention outcome characteristics were found to have either moderate or strong effects (Tables II and III); a majority of these characteristics were found to have moderate effect sizes, with the following four characteristics having strong effect sizes: AAC interventions applied (a) to participants reported to have multiword expressive communication skills prior to intervention (M = .94), (b) in community settings (M = .94), (c) by paraprofessionals (M = .95), and (d) by teachers (M = .94).

One participant characteristic and two intervention characteristics were found to moderate intervention effects. The effect of age was statistically significant (Table II), χ² (3, N = 111) = 18.02, p < .001. To evaluate pairwise differences among the four age groups, post hoc tests were conducted using the Bonferroni approach in which the alpha level is adjusted according to the number of performed comparisons to control for Type 1 error. These results revealed that effect sizes for participants 18 years and older (M = .74) were significantly weaker than those for participants 5 years or younger (M = .86) and participants ages 5 years to 12 years (M = .89), 
all ps < .05/6. Additionally, a significant difference in effect size was identified for the type of intervention (Table III), Z = −2.13, p = .03, with stronger effects for participants who received FCT intervention (M = .87) than for those who received PECS intervention (M = .74). Finally, a significant difference in effect size was found for behavior assessment (Table III), Z = −3.05, p < .002, with stronger effects for interventions that employed an FBA to determine the function of the target behavior(s) (M = .88) than for interventions that did not employ an FBA (M = .72).

Initially, a significant difference in intervention effect was found among the three pre-intervention expressive communication levels (multiword, emerging, and prelinguistic), χ² (2, N = 80) = 6.55, p = .04. However, follow-up pairwise comparisons revealed that effect sizes for participants reported to communicate at the multiword level (M = .94), emerging level (M = .85), and prelinguistic level (M = .89) prior to intervention were not statistically different (ps > .05/3) as had been shown by the initial Kruskal-Wallis one-way ANOVA. No significant differences in effect size were found for the following: (a) Participant characteristics (gender, diagnoses, pre-intervention communication modes, and pre-intervention behavior; Table II), (b) Intervention characteristics (intervention setting, implementation method, person delivering intervention, and type of functional assessment; Table III), and (c) Intervention Outcomes characteristics (expressive communication modes measured, expressive communication levels 
targeted, and behavior targeted; Table IV).

Descriptive Analysis

Frequency counts of participant, intervention, and intervention outcome characteristics reported at the participant-level and summaries of mean effect sizes and corresponding levels of effect are presented in Tables II, III, and IV, respectively. However, frequency counts are not reported across all characteristics, as those characteristics with small ns (eight or fewer) were excluded from inferential analyses. Summaries of quality of study characteristics reported at the study-level are presented in Table V.

Table V. Summary of Study Quality Characteristics, Including Number of Participants and Number who met Inclusion Criteria (# criteria), Intervention Type (IT), Functional Behavior Assessment (FBA), Social Validity (SV), and Intervention Fidelity (IF).

	# Participants	# who met criteria	IT	FBA	Reliability	Generalization	Maintenance	SV	IF
Bailey, McComas, Benavides, and Lovascz (2002)	1	1	FCT	Desc., Exp.	+		+	+	+
Bingham, Spooner, and Browder (2007)	3	3	Other		+				+
Bird, Dores, Moniz, and Robinson (1989)	1	1	FCT	Desc.	+		+
Buckley and Newchok (2005)	1	1	FCT	Exp.	+			+
Casey and Merical (2006)	1	1	FCT	Desc., Exp.	+	+	+
Charlop, Malmberg, and Berquist (2008)	3	2	PECS		+				+
Charlop-Christy et al. (2002)	3	2	PECS		+		+
Day, Horner, and O’Neill (1994)	3	3	FCT	Desc., Exp.	+
Day, Rea, Schussler, Larsen, and Johnson (1998)	3	2	FCT	Exp.	+	+			+
Derby (1995)	4	2	FCT	Desc., Exp.	+	+	+	+	+
Donovan (2003)	3	3	FCT	Desc., Exp.	+				+
Durand (1999)	5	5	FCT	Desc., Exp.	+	+			+
Durand and Kishi (1987)	5	5	FCT	Desc.	+		+
Fisher, Adelinis, Thompson, Worsdell, and Zarcone (1998)	2	2	FCT	Desc., Exp.	+
Fisher, Kuhn, and Thompson (1998)	2	1	FCT	Exp.	+
Franco et al. (2009)	1	1	FCT	Exp.	+	+	+		+
Frea, Arnold, and Vittimberga (2001)	1	1	Choice		+
Friedenthal (2009)	3	1	FCT	Desc., Exp.
Fyffe, Sung, Fittro, and Russell (2004)	1	1	FCT	Exp.	+
Ganz, Parker, and Benson (2009)	3	3	PECS		+	+	+		+
Hagopian, Kuhn, Long, and Rush (2005)	3	2	FCT	Exp.	+
Hanley, Iwata, and Thompson (2001)	3	2	FCT	Exp.	+				+
Hanley, Piazza, Fisher, and Maglieri (2005)	2	1	FCT	Exp.	+
Harding, Wacker, Berg, Winborn-Kemmerer and Lee (2009)	3	2	FCT	Exp.	+			+
Harding, Wacker, Berg, Winborn-Kemmerer, Lee, and Ibrahimovic (2009)	2	2	FCT	Exp.	+
Hetzroni and Roth (2003)	5	5	FCT	Desc.	+			+	+
Hines and Simonsen (2008)	1	1	FCT	Desc.	+
Horner et al. (1990)	1	1	FCT	Desc., Exp.	+				+
Hunt (1999)	8	8	FCT	Desc.	+			+
Hunt, Alwell, and Goetz (1988)	3	2	FCT	Desc.	+	+
Hunt, Alwell, Goetz, and Sailor (1990)	3	2	Other	Desc.	+	+			+
Kahng, Hendrickson, and Vu (2000)	1	1	FCT	Exp.	+
Kelley, Lerman, and Van Camp (2002)	3	1	FCT	Exp.	+
Kern, Gallagher, Starosta, Hickman, and George (2006)	1	1	FCT, Other	Desc.	+				+
Kettering (2009)	4	4	FCT	Exp.	+			+	+
Kuhn, Chirighin, and Zelenka (2010)	2	1	FCT	Exp.	+	+			+
Lalli, Casey, and Kates (1995)	3	2	Milieu	Exp.	+				+
Mancil, Conroy, and Haydon (2009)	3	3	FCT	Exp.	+	+		+	+
Moore, Gilles, McComas, and Symons (2010)	1	1	FCT	Desc., Exp.	+			+
Northup et al. (1994)	5	1	FCT	Desc., Exp.	+	+			+
O’Neill and Sweetland-Baker (2001)	2	2	FCT	Exp.	+	+
Schieltz et al. (2010)	10	3	FCT	Exp.	+	+
Schindler and Horner (2005)	3	2	FCT	Desc.	+	+		+	+
Sigafoos and Meikle (1996)	2	1	FCT	Exp.	+		+
Steege et al. (1990)	2	2	FCT	Exp.	+		+		+
Tang (2002)	6	1	FCT	Desc., Exp.	+
Thompson, Fisher, Piazza, and Kuhn (1998)	1	1	FCT	Exp.	+		+
Vollmer, Roane, Ringdahl, and Marcus (1999)	3	1	FCT	Exp.	+				+
Vollmer and Vorndran (1998)	1	1	FCT	Exp.	+
Wacker et al. (1990)	3	3	FCT	Exp.	+	+			+
Winborn, Wacker, Richman, Asmus, and Geirer (2002)	2	2	FCT	Exp.	+
Winborn-Kemmerer, Ringdahl, Wacker, and Kitsukawa (2009)	2	2	FCT	Exp.	+				+
Winborn-Kemmerer et al. (2010)	2	2	FCT	Exp.	+				+
Worsdell and Iwata (2000)	5	4	FCT	Exp.	+

Participant characteristics. Of the 111 participants, 71% were male and 29% were female. The age of participants varied with a majority of participants falling into two age groups: 5 years or younger (32%) or between 5 and 12 years (38%). Fewer participants were reported as being between 12 and 18 years (21%) and 18 years or older (14%). The most common diagnoses across participants were intellectual disability or developmental disability (75%) and autism spectrum disorder (35%). Fewer participants had diagnoses of sensory impairments (9%), genetic disorders (9%), and emotional behavior disorders (8%). Prior to intervention, 34% of participants were reported to communicate at an emergent level, while 18% and 20% communicated 
at a multiword and prelinguistic level, respectively. Common communication modes included speech (37%) and non-symbolic communication (e.g., gestures, vocalizations; 32%). Of the participants reported to use some mode of AAC prior to intervention, the most common were unaided AAC (e.g., sign language; 19%) and aided AAC without speech output (e.g., picture communication systems; 13%).

For comparison purposes, the authors classified pre-intervention behavior topographies by priority level (Janney & Snell, 2008): (a) destructive behavior (i.e., behavior that is harmful or threatens the safety of participant or others – physical aggression); (b) disruptive behavior (i.e., behavior that does not immediately endanger participant or others but interferes with everyday activities and experiences – verbal aggression, disorderly verbal or physical behavior); and (c) distracting behavior (i.e., behavior that deviates from what is typically expected for individuals of the same age – nonparticipation, stereotypical behavior). Prior to intervention, a majority of participants were reported to exhibit destructive behavior (86%), while significantly fewer exhibited disruptive behavior (25%) and distracting behavior (15%).

Intervention characteristics. Intervention was largely 
conducted in participant classrooms (i.e., general and special education classrooms; 49%), while interventions in pull-out settings (e.g., therapy room, experimental room; 28%), participant homes (23%), “other” settings (e.g., behavior treatment facility; 20%), and the community (10%) were less represented. The majority of participants received interventions under the following implementation conditions: individual intervention or one-to-one teaching (92%), massed trials (i.e., one trial follows the next; 59%), and contextualized surroundings (i.e., intervention conducted in natural communication environment; 53%). Implementation methods less often reported included: decontextualized surroundings (i.e., removed from natural communication environment; 38%), distributed trials (i.e., teaching as part of a routine; 29%), and group intervention (1%). Intervention was delivered by teachers (31%); experimenters (25%); parents (18%); paraprofessionals (14%); and “others”, such as therapists (41%). The primary type of intervention provided to participants was functional communication training (FCT; 84%). Less frequent types of intervention were Picture Exchange Communication System (PECS; 8%), milieu training (3%), and choice making (1%). Over half of the participants (55%) used aided AAC without speech output during intervention. Communication modes that were applied less frequently during intervention included AAC with speech output (28%), unaided AAC (21%), speech (14%), and 
non-symbolic communication (9%).

Clear descriptions of intervention dosage (i.e., frequency and duration) were infrequently reported (29%); among these studies, descriptions of daily intervention implementation (100%) and of weekly intervention implementation (59%) were more common than were descriptions of intervention duration (41%). Pre-intervention FBA was reported in 90% of interventions. Experimental FBAs were more common (68%) than descriptive FBAs (21%). Furthermore, researchers used results of the FBA to inform intervention development for 88% of participants. A range of experimental single-case research designs were used to assess intervention effect: (a) multiple baseline/probe (49%), (b) ABAB/reversal (28%), (c) alternating treatment (16%), and (d) combinations of single-case research designs (7%).

Intervention outcome characteristics. Emerging language (79%) typically was measured as an outcome of intervention; multiword (10%) and prelinguistic (7%) 
communication levels were measured less often. Topographies of destructive behavior were targeted most often for intervention (76%), while topographies of disruptive behavior (31%) and distracting behavior (24%) were targeted less often. Two categories of disruptive behavior were addressed most frequently: disruptive verbal behavior (e.g., yelling, screaming, loudly talking over others, etc.; 85%), and disruptive physical behavior (e.g., throwing oneself on the floor as part of tantrum, running away from classroom, etc.; 50%); verbal aggression was measured less frequently (e.g., threats, teasing; 6%). Interventions that focused on distracting behavior primarily measured nonparticipation (67%) and 
stereotypical behavior (22%).

Quality of study characteristics. Reliability or interobserver agreement was measured in 53 studies (98%). Generalization of targeted behavior or skills across new stimuli (i.e., partners, settings, materials) and responses was assessed with a measure of acceptable quality across 15 studies (28%). Maintenance of targeted behavior or skills was assessed across 11 studies (20%). Some measure of social validity was applied across 10 studies (19%). Finally, fidelity of intervention was measured in 25 studies (46%).

Discussion

The purposes of this meta-analysis were (a) to evaluate the effectiveness of AAC intervention on challenging behavior for individuals with disabilities, (b) to identify relationships between study characteristics and the effects of AAC intervention, and (c) to assess the quality of the studies included in the current review. The authors also provided a comprehensive summary of participant, intervention, and intervention outcome characteristics. There were several notable findings that are relevant to AAC interventions for challenging behavior of individuals with disabilities. First, this review provides evidence that AAC intervention has positive effects in decreasing challenging behavior for individuals with varying 
disabilities. That is, when the intervention for challenging behavior includes some approach for teaching individuals to communicate using AAC, the effects are likely to be positive. Although effect sizes varied across participants, the average NAP score (M = .88) is representative of the upper end of a medium or moderate effect (Parker & Vannest, 2009). Second, this review provides evidence that AAC intervention may be more beneficial when (a) applied to younger persons with disabilities; (b) an FBA is conducted prior to intervention and results of that assessment are used to inform the development of the intervention; and (c) FCT with AAC is used to address challenging behavior. Third, a majority of studies failed to measure skill generalization, maintenance of effects on challenging behavior, or the social validity of applied interventions. Further, fidelity of intervention implementation occurred in slightly less than half of the studies, thus the quality of the reviewed studies is somewhat limited. Finally, descriptive analyses revealed that almost half of participants received intervention in atypical environments (e.g., pull out room, treatment center) and over half of the interventionists were individuals who typically would not be present in participants’ natural environments (e.g., researcher, therapist). A discussion of these findings and their implications follows.

This meta-analysis contributes first to the expanding database of research that demonstrates the effective implementation of AAC interventions to resolve challenging behavior issues. Overall, AAC intervention was found to have a positive effect on challenging behavior. Further, the level of this effect (moderate or strong) was observed across all study characteristics. However, a majority of the study characteristics did not moderate intervention effects. In the case of this meta-analysis, most participant characteristics (i.e., gender, diagnoses, pre-intervention communication characteristics, pre-intervention behavior) did not contribute to different effects, nor did most intervention characteristics (i.e., setting, implementation method, interventionist, targeted communication mode, experimental design) or intervention outcome characteristics (i.e., targeted communication level, targeted behavior). These results suggest that AAC intervention is effective in resolving challenging behaviors when applied (a) across a 
broad spectrum of individuals, and (b) under a wide range of conditions (e.g., settings, interventionist, AAC mode). Although significant differences in effect size between the three pre-intervention expressive communication modes were not found following the initial Kruskal-Wallis one-way ANOVA, which may have been attributed to the conservative nature of the follow-up pairwise 
comparison procedure, the differences between participants reported as having emerging and multiword language skills prior to intervention may be noteworthy. Intervention effects were weaker for those participants with emerging language skills (M = .85) than for those with multiword language skills (M = .94), suggesting that AAC intervention potentially may be more effective for individuals with more developed language skills. However, because results of this analysis were insignificant, one must proceed with caution when drawing such conclusions; additional research evaluating the extent to which language skills affect intervention outcomes is warranted.

Results of this review also suggest that AAC intervention is more effective when applied with younger persons with disabilities. Specifically, participants 12 years or younger experienced significantly greater reductions in challenging behavior than did those participants 18 years or older. Researchers have indicated that unresolved challenging behavior during childhood leads to poor adult outcomes (McCord, 1978; Olweus, 1991). As such, it is possible that these adults have such well-established patterns of communicating through challenging behavior that they are less likely to learn alternative ways to communicate. Additional research is needed to explore this possibility.

The results of this review also lend support to the value of assessing the function of the challenging behavior prior to intervention. Interventions that were based on an FBA resulted in stronger effects than did interventions that did not take into account the function of the challenging behavior. This finding is consistent both with other reviews (e.g., Campbell, 2003; Didden, Duker, & Korzilius, 1997; Didden et al., 2006; Scotti, Evans, Meyer, & Walker, 1991) and with the communication hypothesis described by Carr et al. (1997), whereby challenging behavior serves a communicative function for those individuals who have not developed advanced communication skills. If widespread use of AAC intervention that is linked to FBA results is to become the common practice for addressing challenging behavior, it is crucial that professionals fully understand both how to conduct FBAs and how to implement the resulting intervention. Although the results of this meta-analysis suggest that interventionists implemented AAC interventions with acceptable rates of fidelity, over half of the interventionists were researchers and therapists who likely were skilled in conducting FBAs prior to intervention. Additional research is necessary to identify effective practices for teaching typical interventionists (e.g., teachers, paraprofessionals, related services personnel, and parents) to conduct FBAs (e.g., Bessette & Wills, 2007), as current research suggests that few interventionists are trained to use function-based interventions and their ability to do so varies widely (Dunlap & Fox, 2011).

Also of significance are results that suggest that the type of FBA does not differentiate the magnitude of the effect. That is, experimental FBA (i.e., analogue functional analysis) and non-experimental FBA (i.e., descriptive assessment) did not moderate behavior change. This finding suggests that both methods are equally effective, despite the large difference in skill and effort required to carry out experimental FBAs. And, while this finding is similar to that of Goh and Bambara (2010), it differs from other recent reviews (e.g., Campbell, 2003; Didden et al., 2006) that have found experimental FBAs to contribute to stronger intervention effects than non-experimental FBAs. In light of this conflicting finding, it is important to explore further the relationship between these two assessment methods and the resulting intervention effects. In particular, more work is needed first to clarify conflicting findings from meta-analyses addressing behavior interventions, and second to identify factors that contribute to such findings (e.g., conditions that are attributed to outcomes of FBA use). Furthermore, because descriptive and experimental FBAs typically differ in the type of data collected, research efforts are needed to examine the extent to which the use of each type of data to design AAC intervention affects intervention outcomes. This type of research will have strong implications for typical interventionists who are expected to conduct FBAs outside of a research context. There are many limitations to conducting experimental FBAs, including providing reinforcement for challenging behavior, obtaining results that do not apply in the natural setting, ignoring potential setting events, failing to specify predictive antecedents, and requiring sophisticated methodology (Bambara & Kern, 2005). If additional research confirms that there is little advantage in using experimental FBAs over non-experimental FBAs, those individuals who are expected to conduct behavior assessments will have the option to use non-experimental FBAs and avoid these limitations that come with experimental FBAs.

Additionally, FCT intervention, when used in conjunction with AAC approaches, was found to have significantly stronger effects than PECS intervention in reducing challenging behavior. This finding may be attributed to the function-based principles underlying FCT, which utilize FBA to inform intervention development. While FCT has been identified as an evidence-based practice to address challenging behavior (e.g., Kurtz, Boelter, Jarmolowicz, Chin, & Hagopian, 2011), the evidence of PECS as an effective behavior intervention is limited (Preston & Carter, 2009). This may be due in part to the limited number of PECS studies that also target participants with challenging behavior, and that PECS does not rely on conducting a prior FBA or on function-based principles. In light of this limitation, additional research is warranted to evaluate the efficacy of PECS in relation to challenging behavior. When interpreting these results, one must use extreme caution for two reasons. First, FCT is not an AAC intervention per se. Rather, FCT is a behavior analytic intervention that may or may not apply AAC approaches depending on whether the participant may benefit from AAC as a supplement to or a replacement for speech. Second, the authors located only three PECS intervention studies in which challenging behavior was measured as an outcome variable; this large discrepancy in number of FCT and PECS intervention studies may have skewed the results of the moderator analysis.

Finally, the descriptive analysis yielded several important findings concerning the conditions under which AAC intervention research has been conducted and the quality of such research. Intervention often occurred within atypical environments and with atypical interventionists. Although results from this review's moderator analysis indicated that decontextualized and contextualized AAC interventions lead to similar behavioral outcomes, implementation of communication-based intervention for challenging behavior under typical conditions (i.e., tasks, people, and settings that are typical of one's daily life) is thought to support skill generalization (Carr et al., 1997) and to improve the social validity of an intervention (Horner et al., 2005), and has been found to contribute to stronger intervention effects (e.g., Marquis et al., 2000). In the future, the effects of conducting AAC intervention research under typical and atypical conditions must be evaluated more precisely. Furthermore, researchers need to estimate generalization effects and determine whether AAC intervention applied under typical conditions leads to stronger generalized effects than when applied under atypical conditions. The authors of this review recommend additional research aimed at teaching typical interventionists (e.g., parents, teachers) to implement AAC interventions in typical or naturally occurring environments because these methods are thought to lead to intervention implementation that likely is less costly, requires less reliance on outside experts, and has the potential of achieving stronger intervention outcomes. Perhaps equally important, and in agreement with other reviews of behavioral intervention, are findings indicating that maintenance, generalization, social validity, and intervention fidelity were either rarely measured or measured only by half of the studies in this review. These four research tools commonly are viewed as essential characteristics for studying human behavior. It would be of value to understand the conditions under which researchers are motivated to use all four tools, as well as the conditions associated with a failure to apply these tools.

This review had several limitations that should be considered when evaluating its results. First, and foremost, the coding process was limited to broad categories of AAC intervention (i.e., FCT, PECS, choice making, milieu training). As such, the authors were unable to assess the effectiveness of specific intervention characteristics, as others have done (e.g., Campbell, 2003; Didden et al., 2006). Thus, effect size calculation was limited in this regard. For example, several studies examined the effectiveness of FCT but included comparisons among various FCT conditions (e.g., FCT alone vs. FCT plus extinction). Because the intent was to evaluate the effectiveness of AAC intervention (and not different variations thereof), complex designs were simplified by collapsing all side-by-side AAC intervention phases into one AAC intervention phase, so long as a comparison condition (i.e., no AAC intervention) was available. To address this limitation, future research efforts should assess the effect of variations of each type of AAC intervention on behavior outcomes.

Second, the authors limited effect size calculation to measures of challenging behavior and omitted effect size calculations of other measures (e.g., other behaviors, maintenance, generalization). Reviewed studies frequently measured and reported data on other behaviors beyond those reported as challenging (e.g., appropriate behavior, communicative behavior), but the only measurement requirement for being included in this review was challenging behavior. Furthermore, although studies often reported measurement of skill generalization and maintenance, the authors did not measure the effects of AAC intervention in generalization and maintenance conditions. Only additional research aimed at these limitations will determine if AAC intervention is effective longitudinally and across different conditions.

Third, the authors’ choice of NAP as an effect size metric limits the ability to compare results from this meta-analysis to those of other reviews, because a wide range of effect size methods are used in the field. This limitation is common across many single-case research literature reviews and underscores the importance of identifying a universal effect size metric to use with single-case design data or guidelines for the use of effect size metrics (Kazdin, 2011). The use of interpretative guidelines, often broadly described as “strong,” “medium,” or “weak,” offers a relatively adequate way in which to compare results between reviews. However, because NAP interpretative guidelines are “tentative” (Parker & Vannest, 2009, p. 364), the interpretation of effect sizes using the NAP statistic may be positively or negatively skewed, thus limiting the ability to compare results between reviews.

Fourth, the authors found that measuring intervention dosage presented several challenges. The coding instrument included two items to measure dosage: (a) frequency of intervention (number of intervention sessions per day, number of days per week); and (b) duration of intervention (intervention duration in weeks, months, year). However, very few studies provided a clear description of either the frequency or the duration with which the intervention was implemented, making it impossible to classify dosage into categories and include dosage in inferential analyses. Future researchers must clearly describe the extent to which AAC interventions are implemented so that connections between dosage and intervention effects can be established.

In summary, the results of this meta-analysis lend support to (a) intervening during childhood, (b) using AAC interventions to address challenging behavior across a broad spectrum of individuals with disabilities, (c) basing interventions on FBAs, and (d) implementing FCT to address challenging behavior. AAC intervention research on challenging behavior can be improved 
by studying individuals in their natural settings with typical interventionists (Pless & Granlund, 2012; 
Trottier, Kamp, & Mirenda, 2011) and by addressing the quality indicators of single-case research, including maintenance, generalization, and fidelity of implementation (Horner et al., 2005).

Author Note

Virginia L. Walker, Curriculum, Instruction, and 
Special Education, University of Virginia; Martha E. Snell, Curriculum, Instruction, and Special Education, University of Virginia.

The authors would like to thank Kimberly Vannest for sharing her knowledge of how to apply NAP to single-case research data for meta-analytic purposes.

Declaration of interest: The authors report no 
conflicts of interest. The authors alone are responsible for the content and writing of the paper.

End Notes

Notes

1. A report of original research usually published in a technical journal or appearing as a thesis or dissertation; refers to the original research reports collected for a research synthesis (Kazdin, 2011).

2. A complete list of search terms and search procedures may be obtained from the first author.

3. References for the 54 primary studies included in this review are listed in the reference list and each is denoted with an asterisk.

4. The AAC and Challenging Behavior Research Coding Form (Walker & Snell, 2012) may be obtained from the first author.

5. Inter-rater agreement scores for individual categories may be obtained from the first author.