Evidence Base Update on the Assessment of Irritability, Anger, and Aggression in Youth

ABSTRACT

Objective

Irritability, anger, and aggression have garnered significant attention from youth mental health researchers and clinicians; however, fundamental challenges of conceptualization and measurement persist. This article reviews the evidence base for assessing these transdiagnostic constructs in children and adolescents.

Method

We conducted a preregistered systematic review of the evidence behind instruments used to measure irritability, anger, aggression, and related problems in youth. Searches were conducted in PsycINFO and PubMed, identifying 4,664 unique articles. Eligibility criteria focused on self- and proxy-report measures with peer-reviewed psychometric evidence from studies in English with youths ages 3–18. Additional measures were found through ancillary search strategies (e.g. book chapters, review articles, test publishers). Measures were screened and coded by multiple raters with acceptable reliability.

Results

Overall, 68 instruments met criteria for inclusion, with scales covering irritability (n = 15), anger (n = 19), aggression (n = 45), and/or general overt externalizing problems (n = 27). Regarding overall psychometric support, 6 measures (8.8%) were classified as Excellent, 46 (67.6%) were Good, and 16 (23.5%) were Adequate. Descriptive information (e.g. informants, scales, availability, translations) and psychometric properties (e.g. reliability, validity, norms) are summarized.

Conclusions

Numerous instruments for youth irritability, anger, and aggression exist with varying degrees of empirical support for specific applications. Although some measures were especially strong, none had uniformly excellent properties across all dimensions, signaling the need for further research in particular areas. Findings promote conceptual clarity while also producing a well-characterized toolkit for researchers and clinicians addressing transdiagnostic problems affecting youth.

Consider a child who yells at their teacher, explodes in rage at their parent, and gets in fights with their peers. When such behaviors occur, it usually does not take long for the adults in the child’s life to recognize that there is a problem. These behaviors are disruptive, stressful, even scary, and, if left unaddressed, can have significant long-term consequences. Importantly, the child and others in their life are likely to experience distress, impairment, and unhappiness in relation to these behaviors, underscoring the need for clinical and research attention.

Problems like these—temper loss, aggression, irritability, arguing, anger, defiance, etc.—have been mapped empirically as overt externalizing behaviors (Frick et al., 1993). Usually beginning in childhood, overt externalizing behaviors represent some of the most common youth mental health concerns, both in the population (Ghandour et al., 2019; Polanczyk et al., 2015) and in clinical settings (Evans et al., 2022; Freeman et al., 2016; Olfson et al., 2014). These problems can predict the development of numerous poor developmental outcomes, including school suspension and expulsion, delinquency, substance use problems, depression, anxiety, severe antisocial behaviors, impaired occupational and relational functioning, and adverse physical health outcomes (Burke et al., 2005, 2014; Loth et al., 2014; Rowe et al., 2010; Stringaris & Goodman; Odgers et al., 2007, 2008). Over time, overt externalizing problems have enormous ramifications on the developmental trajectories of individual youths with large costs to families, society, health care, and justice systems (Goulter et al., 2023; Hawes et al., 2023). Thus, early identification of overt externalizing problems is critical.

Although the observable nature of overt externalizing behaviors may make it obvious that there is a problem, what is often less clear is what kind of problem it is, and what to do about it. How should professionals understand, measure, and communicate about the challenges that affect youths with externalizing problems? Considering the heterogeneity of these behaviors, even when the manifest behaviors look similar, important distinctions can be made based on factors ranging from the context in which they occur (e.g., home vs. school), to whether they are accompanied by physical aggression.

Further, it is critical to consider the affective experiences underlying overt behaviors. Disruptive behaviors are typically accompanied by emotional, social, and cognitive patterns which may represent targets for intervention and prevention. For example, irritability can precipitate aggressive behaviors, but this is not the case for most instances of feeling irritable, nor for most chronically irritable youths (Brotman et al., 2017). Nevertheless, chronic irritability is a predictor of anxiety, depression, oppositional defiant disorder (ODD), suicidality and various forms of impairment (Brotman et al., 2006; Rowe et al., 2010; Stringaris & Goodman, 2009; see; Vidal-Ribas et al., 2016 for a meta-analysis). Problems with irritability and anger are important affective dimensions of the overt externalizing spectrum. These difficulties are often impairing, under-identified, and warrant attention both within and beyond the context of behavioral dimensions that are more easily observed.

The issues outlined above are fundamental questions of conceptualization and measurement. To help address these issues, both conceptually and practically, we conducted a systematic review of instruments for assessing problems on the overt externalizing spectrum, including its affective and behavioral dimensions. It is important to note that this paper adopts a transdiagnostic framework and is therefore not about assessing for particular disorders, such as ODD or Conduct Disorder (CD).¹ In research and practice, assessment and diagnosis are closely linked. But diagnostic classification systems—including the American Psychiatric Association’s (2013; 2022) Diagnostic and Statistical Manual (DSM) and the World Health Organization’s (2022) International Classification of Diseases (ICD) have a way of driving and reifying prevailing models of psychopathology (Hyman, 2010). Many have argued that DSM and ICD fail to provide a nosological home for youth experiencing problems with irritability and related problems (e.g., Lochman, Evans, et al., 2015; Mikita & Stringaris, 2013; Stepanova et al., 2022). The DSM and ICD dedicate entire blocks for disorders of anxiety, disorders of low mood, and so on; but there is no section for disorders of irritability, anger, or aggression.² Instead, these problems are associated with many different categories across the landscape of psychopathology (e.g., for irritability, ODD, PTSD, ADHD, intermittent explosive disorder, autism spectrum disorder, anxiety and depressive disorders); at the same time, they are not necessary or sufficient for any particular diagnosis (Evans et al., 2017; Stepanova et al., 2022). This creates a significant problem in research and clinical contexts when the focus is on irritability, anger, and/or aggression, as these are not adequately captured by any one diagnosis and instead are scattered in different ways across many categories (Evans et al., 2023).

Considering the availability of research on longstanding diagnostic categories, relatively less attention has been given to measurement issues independent from existing nosologies. In light of these gaps, the present systematic review adopted a transdiagnostic conceptualization. Transdiagnostic approaches to psychopathology have gained traction in recent years, influencing both research and practice. These models provide a framework for understanding mental health, emphasizing problems, principles, and approaches that tend to cut across diagnostic categories (Dalgleish et al., 2020; Sauer-Zavala et al., 2017). By better addressing clinical heterogeneity, these approaches could lead to greater efficiency and decreasing costs in mental health care (Dalgleish et al., 2020). For youth, transdiagnostic approaches to assessment, intervention, and clinical research have been evolving to better address high rates of comorbidity, complex contextual factors, and shifting symptom profiles across development (Ehrenreich-May & Chu, 2014; Marchette & Weisz, 2017). Irritability, anger, and aggression are among the most prominent transdiagnostic features in youth mental health, as they can relate specifically to traumatic stress, mood disturbance, anxiety, sleep problems, personality, and externalizing disorders, to name a few (Evans et al., 2023). To clearly separate these transdiagnostic features from diagnostic categories, we focus on the evidence base for measures of youth emotional and behavioral problems explicitly apart from the criterial definitions given by DSM. By doing so, we aimed to contribute a less biased and more comprehensive review of instruments that were truly designed and tested for measuring these constructs while also acknowledging their interrelations.

Relations Among Irritability, Anger, and Aggression

The last two decades have seen an explosion in youth irritability research. This growth has been spurred on by converging bodies of work on the boundaries of pediatric bipolar disorder (Leibenluft, 2011), irritability in ODD (Evans et al., 2017), and closely related phenomena of tantrums (Wakschlag et al., 2012), dysregulation (Althoff et al., 2010), and “rages” or outbursts (Carlson et al., 2009). Much has been gained and learned. Youth irritability has even left its mark on the American Psychiatric Association’s (2013) DSM-5 via the addition of Disruptive Mood Dysregulation Disorder (DMDD; Roy et al., 2014) and on the World Health Organization’s (2022) ICD-11 via a new subtype for chronic irritability in ODD (Evans et al., 2017).

As with any burgeoning new area of research, questions regarding definition and measurement of irritability have been a sticking point. Given the paucity of irritability measures initially, some of the transformative early work (e.g., Brotman et al., 2006; Stringaris & Goodman, 2009) required researchers to piece together a handful of available items to measure youth irritability on a conceptual and post hoc basis. Since then, several new measures have been developed and evaluated specifically for irritability. A synthesis of the evidence is now needed.

There now seems to be reasonably strong agreement in the field coalescing around a definition of irritability as an increased proneness to anger, which can lead to aggression but often does not (Barata et al., 2016; Brotman et al., 2017; Toohey & DiGiuseppe, 2017; Vidal-Ribas et al., 2016).³ This definition requires two more definitions. First, anger has been defined as an uncomfortable, transient, negatively valanced emotional state, ranging from mild irritation to intense rage, with physiological, cognitive, phenomenological, and behavioral aspects such as arousal, hostility, and predisposition to aggression (Eckhardt et al., 2004; Sukhodolsky et al., 2016). Second, aggression is an intentional overt behavior that can result in harm to self or others, including subtypes of irritability/angry affect—i.e., reactive, impulsive, hostile, “hot-blooded” aggression—differentiated “cold-blooded” types (i.e., proactive, instrumental; Fite et al., 2018; Sukhodolsky et al., 2016). Clinically, these problems cluster together in the same multidimensional space—overt externalizing problems—that comprise parts of diagnostic criteria for ODD, CD (Frick et al., 1993), and many other disorders (Evans et al., 2017, 2023).

These definitions of irritability, anger, and aggression reveal a fundamental problem: It is difficult to characterize any one of these constructs without invoking the others. Empirically, rating scale measures of youth irritability, anger, and aggression (especially reactive aggression) correlate with each other with medium to large effect sizes, with most correlations falling between 0.3 and 0.8 when rated by the same informant (Evans et al., 2016, 2020, 2021; Ezpeleta et al., 2020; Zik et al., 2022).⁴ Such patterns suggest it is not clear whether a particular measure is really capturing the intended construct as opposed to a closely related construct or facets of a larger phenomenon (e.g., whether irritability scales actually measure irritability vs. anger or emotion dysregulation). Indeed, a close, item-level inspection often reveals similar if not overlapping item content across measures of purportedly different constructs (e.g., measures of all three constructs tend to include items related to getting angry).

The picture becomes even murkier when we consider the role of informant variation. Cross-informant correlations in this space tend to be small to medium (Evans et al., 2021; Ezpeleta et al., 2020; Zik et al., 2022), comparable to cross-informant correlations in youth psychopathology in general (De Los Reyes et al., 2015). It is likely that irritable, angry, and/or aggressive behaviors manifest in different ways in different contexts, or to different degrees (e.g., aggressive outbursts sometimes occur at home but not at school), and that each informant offers a distinct perspective that is more than just measurement error. The available evidence suggests that more variation in these variables can be accounted for by informant perspective than by differences between the constructs, but this evidence is relatively limited and new, as are the tools behind it. Clearly further work is needed to better define, measure, and model these phenomena as being related to one another, while also acknowledging their differences, within and across informants. Such work could bring researchers closer to a consistent approach, a shared understanding, language, and toolkit for meaningfully advancing research on the disruptive emotions and behaviors in youth (Althoff & Ametti, 2021). Accordingly, we hope to provide a comprehensive and practical review of instruments that researchers and professionals can use to assess problems across the irritability-anger-aggression spectrum and from different informant perspectives.

The Current Evidence Base and the Present Review

Many tools are available for assessing overt externalizing emotional and behavioral problems in youth. At a broadband level, there are comprehensive, normed, multi-informant instruments such as the Child Behavior Checklist (CBCL; Achenbach & Rescorla, 2001) and the Behavior Assessment System for Children (BASC; Reynolds & Kamphaus, 2015) that provide a picture of youths’ externalizing problems broken into subscales such as aggression, rule-breaking behaviors, and anger control (as well as internalizing, total, and other problems). Measures like the Eyberg Child Behavior Inventory (ECBI; Eyberg & Ross, 1978) have long been used to assess disruptive behaviors at baseline and throughout treatment. More recently, the field has seen more rating scales targeted for different types of aggression (e.g., Raine et al., 2006), anger (e.g., Brunner & Spielberger, 2009), and irritability (e.g., Stringaris et al., 2012), including developmentally typical and atypical problems from early childhood (e.g., Wakschlag et al., 2012) through adolescence and adulthood (e.g., Butcher et al., 1992). However, these measures draw from different literatures and disciplines (e.g., psychiatry, pediatrics, education, and clinical, counseling, developmental, social, and personality psychology). It is easy to overlook relevant work from outside one’s own discipline, justifying this comprehensive review.

Several existing literature reviews are worth noting. A few papers have recently summarized the evidence base on measures pertaining to irritability, outbursts, or emotion dysregulation (Adrian et al., 2011; Althoff & Ametti, 2021; Carlson, Singh, et al., 2022; Freitag et al., 2023; Mazefsky et al., 2021). Others have focused on more traditional conceptualizations of disruptive and externalizing problems (Becker-Haimes et al., 2020; Burke et al., 2023; Collett et al., 2003; Evans et al., 2024; McMahon & Frick, 2005; Steiner & Remsing, 2007; Walker et al., 2020). Still others have put forth methodological reviews on anger and aggression in youth (Blake & Hamrin, 2007; Feindler & Engel, 2011; Hubbard et al., 2010; Kerr & Schneider, 2008). We consulted sources like these (see Table S2 for full list) when conducting the present review, to identify leads and trends in the literature. However, existing reviews have used different or more narrow foci than the present review, identifying key limitations in the available evidence. For example, one evidence base update on measures of emotion regulation concluded that most measures had “unsatisfactory” psychometric properties (Mazefsky et al., 2021). Others have concluded that new directions in assessment are needed, including different psychological domains, social contexts, and informant perspectives on irritability (Stringaris et al., 2018). A synthesis of irritability, anger, and aggression measures and is needed to advance measurement in clinical and research settings.

Given these gaps and the assessment challenges facing researchers and clinicians, the goal of this article was to provide a comprehensive review of the evidence base behind measures for assessing overt externalizing affective and behavioral problems. Specifically, we sought to (a) identify all measures relevant to assessing irritability, anger, and/or aggression in youth; (b) summarize the characteristics of these measures; and (c) evaluate their psychometric properties according to the available evidence. To our knowledge, this is the first review of its kind in terms of its transdiagnostic conceptualization of the topic, practical orientation for assessment, and its integrative, interdisciplinary, and comprehensive scope. Our hope is that it may serve as a practical resource and toolkit for researchers and clinicians.

Methods

Procedures were developed following the guidelines put forth by De Los Reyes and Langer (2018) for evidence base updates (EBU) on assessment—in particular, for broad reviews concerning transdiagnostic constructs (p. 362). Broadly, the review involved locating measures that met our criteria and empirical articles examining those measures, then extracting descriptive information and evaluating the psychometric evidence according to established rubrics (Hunsley & Mash, 2008; Youngstrom et al., 2017). Our approach was informed by recent EBU reviews on assessing parenting stress (Holly et al., 2019), youth sleep (Van Meter & Anderson, 2020), emotion regulation (Mazefsky et al., 2021), anxiety (Etkin, Lebowitz, et al., 2021; Etkin, Shimshoni, et al., 2021), and brief, free, and accessible youth mental health measures (Becker-Haimes et al., 2020). This project was pre-registered with PROSPERO (ID: CRD42021283556). Title/abstract and full-text screenings were carried out using the Covidence online review software (https://www.covidence.org/).

Search Methods

Electronic Database Searches

Primary searches for peer-reviewed articles were conducted in PsycINFO and PubMed in October 2021. Search queries were designed to be comprehensive for the constructs of interest, with exact parameters tailored by database. Broadly, the target was peer-reviewed empirical articles that touched on at least one relevant conceptual term (e.g., irritab*, aggress*) AND a methodological term (e.g., assess*, valid*) AND an age term (e.g., child*, adolesc*). The full search strings for both databases are reported in the Supplement (Table S1). These searches yielded 5,412 results. After removing 748 duplicates that overlapped between the two databases, 4,664 unique articles remained and were subjected to screening (described below), ultimately leading to 55 measures identified via this strategy. See Figure 1 for the full PRISMA flow diagram.

Figure 1. PRISMA flow diagram for present systematic review of measures of youth irritability, anger, and aggression.

Note: Adapted from PRISMA flow diagram template (Page et al., 2021; https://prisma-statement.org/)

Ancillary Search Strategies

While the primary searches described above focused on articles about relevant measures (Table S1), we also employed a variety of strategies to identify measures directly (see Table S2). First, we examined recent review papers and chapters summarizing lists of instruments relevant for assessing at least one of the problems of interest (e.g., Althoff & Ametti, 2021; Blake & Hamrin, 2007; Kerr & Schneider, 2008; Walker et al., 2020). Second, we reviewed several online listings of assessment measures organized by professional, scientific, and health organizations (e.g., American Academy of Child and Adolescent Psychiatry [AACAP], 2022; American Psychiatry Association [APA], 2022b; Wikiversity, 2022). Third, we browsed and searched the sites of major commercial test publishers (e.g., Pearson, Psychological Assessment Resources, Multi-Health Systems). Fourth, we searched the Health and Psychosocial Instruments (HaPI) database for additional measures to evaluate for eligibility (search terms in Table S1). Finally, a few relevant measures were identified serendipitously during searching and coding. When identified through any of these ancillary strategies, new candidate measures were subjected to the same evaluation for eligibility as all other measures. Taken together, these strategies yielded an additional 13 measures that met criteria, for 68 in total (see Figure 1).

Follow-Up Searches

Once a measure was identified as potentially relevant, additional targeted literature searches were conducted specific to that measure. This practice ensured that there was sufficient evidence to carry the measure forward to the next stage for further evaluation and coding. For a measure to be included, we required that there be at least 3 peer-reviewed articles reporting original empirical results on that measure’s psychometric properties, following other recent EBU papers (Holly et al., 2019; Van Meter & Anderson, 2020). When measures fell just shy of this 3-paper threshold, additional searches were conducted to confirm no relevant papers had been missed prior to excluding. These targeted searches were conducted with an aim toward fully characterizing the evidence base for included measures. Following procedures similar to those used by Holly et al. (2019, p. 687), once measures were identified as relevant and having more than 3 articles, we then sought to locate an initial batch of ~ 10 articles (or as many as possible if 10 could not be found) that were the most appropriate sources reporting on the psychometric results⁵ for each measure. At this stage, we assigned preliminary ratings and assessed whether this initial batch of articles was sufficiently exhaustive or representative for the measure. Additional targeted searches were conducted as needed capture a more complete and representative sampling of articles in order to allow us to rate all psychometric properties.

To keep the review manageable and consistent, we imposed an upper limit such that no more than 20 unique and relevant sources (i.e., all meeting our criteria) would be reviewed to inform the ratings for any given measure. Ultimately, we reviewed a mean of 9.9 total sources (8.8 peer-reviewed) for each measure, with individual measures ranging from a minimum of 3 sources (Rage Attacks Questionnaire [RAQ]; Budman et al., 2003) to a maximum of 19 sources (Adjustment Scales for Children and Adolescents/Preschool Intervention [ASCA/ASPI]; Lutz et al., 2002; McDermott, 1993). Examples of measures that were screened out due to insufficient evidence and other reasons are provided in Table S3.

During these searches, we sometimes had to decide which articles were most relevant from among the many that reported results on the same measure. To accomplish this, we prioritized our eligibility criteria (e.g., youth samples; see below) and the following considerations (roughly in order of priority): (a) psychometric papers focusing directly on the measure of interest; (b) studies with large, diverse, and varied samples; (c) collectively selecting a variety of study types and designs to inform evaluations ranging from validity and reliability to treatment sensitivity and clinical utility; and miscellaneous other factors including (d) diversity and independence of research teams, to avoid solely relying on papers from the measure’s developer; (e) inclusion of more recent studies alongside original measure validation studies; and (f) emphasis on papers that were more highly cited or appeared in more impactful journals. Ultimately, these considerations would only affect the more widely used and studied measures, as we tried to include all available relevant papers until we reached approximately 10–20 papers for a given measure. Occasionally, test reviews and methodological reviews would be found highly relevant to the focal measures, and these were included as well. Finally, we conducted additional searches for web sites, test manuals, and review papers, and follow-up searches in Google Scholar as needed (see Table S1). Prior to finalizing ratings, we reviewed all sources for information that might have provided evidence for a higher rating in any psychometric category. Table S5 lists all sources reviewed by measure.

In sum, for each measure included in this review, we sought to find a robust and varied selection of peer-reviewed evidence that would speak to its psychometric properties to the fullest extent possible. Thus, our evaluations of each measure can be construed as the highest level of psychometric performance that the instrument appeared to reach—beyond which either we were unable to locate additional relevant articles, or including more articles would be unlikely to change any of its psychometric ratings.

Measure Eligibility

A Priori Inclusion and Exclusion Criteria

To be included, measures had to (a) assess irritability, anger, aggressive behavior, or a closely related transdiagnostic construct (e.g., hostility, rage, undifferentiated overt externalizing problems); (b) use self-report and/or proxy-report (e.g., teacher, parent, clinician) collected via rating scale, checklist, questionnaire, or interview; (c) have been used in youth samples with a mean age between 3.0 and 18.9 years, not including undergraduate samples; (d) have been written, administered, and evaluated in English; and (e) have been examined in at least three peer-reviewed empirical articles reporting psychometric information, consistent with all of the above criteria (i.e., focus of measurement, informant, methods, sample age, English language).

Measures were excluded if they did not meet all the above criteria, and specifically if they (a) were directly tied to DSM diagnostic criteria for a mental health disorder including ODD, CD, DMDD (because the focus was on transdiagnostic constructs); (b) used vignettes, hypotheticals, projective tasks, etc., or focused on prediction of risk (because the focus was on actual patterns of behaviors and emotions, not indirect inferences about them); (c) relied on peer nominations or behavioral observations at school or home (because the focus was on tools reasonably implemented at the individual level, not requiring social environment); (d) focused on other areas, such as parenting, delinquency, or mood disorders (because these were not the constructs of interest, although related); or (e) did not have enough evaluation in peer-reviewed articles (i.e., dissertations, chapters, and presentations were excluded).

Ad Hoc Eligibility Criteria

Some situations arose that required clarifications and additions to our a priori eligibility rules. First, single-item measures were excluded because they do not permit robust measurement or psychometric evaluation (e.g., no estimation of internal consistency or measurement error). Second, measures were excluded if they used items drawn from an existing instrument to measure a relevant construct, rather than being designed and developed to measure the relevant construct originally. Third, we identified some measures that appeared to fall out of common use several decades ago. To promote the review’s relevance, measures were excluded if all their articles were published >25 years ago. Fourth, many measures evolved over the years, and again, we aimed to provide a current review. We decided to prioritize the current version of each measure to determine its relevance for inclusion (however, once a measure was included, we examined its entire evidence base cumulatively). Finally, clarifications were made to differentiate constructs of interest from closely related constructs. For example, we prioritized aggression measures and excluded bullying measures by attending to terminology in the measures and evaluating them against the definition of bullying.⁶ Overall, these criteria led to excluding some measures that may be of interest. See Table S3 for all exclusionary criteria, operational definitions, and examples of measures that were excluded.

Measure Content

Decisions about eligibility were made based on measures’ titles, descriptions of their intended purpose, and direct inspection of the item content. Scales and subscales that were named and described using terms like irritability, anger, and aggression were typically included, whereas measures defined by terms that were more ambiguous or broad (e.g., externalizing, conduct, disruptive behavior) were borderline cases that required closer evaluation to decide about inclusion or exclusion. To do this, scales were coded conceptually by two authors (SCE, ARK) according to whether each item was an indicator of irritability, anger, or aggression specifically; or relevant to overt externalizing problems broadly; or tapping other constructs and therefore not relevant. Scales were included if ≥50% of their items were broadly relevant and ≥25% of their items specifically mapped to irritability, anger, or aggression. On this basis, broad externalizing scales like the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) Conduct Problems and Pediatric Symptom Checklist (PSC; Jellinek & Murphy, 1988) Externalizing Problems were included, whereas other externalizing measures (See Table S3) were excluded as they were coded as having minimal relevance to irritability, anger, and/or aggression.

Article Screening and Measure Evaluation Procedures

The project proceeded from title/abstract screening, to full-text screening, to coding (see Figure 1). These steps were carried out by the authors, all of whom have bachelor’s-level or graduate-level experience in psychology research, led by a doctoral-level clinical child psychologist (SCE). Weekly team meetings were held to allow for discussion, troubleshooting, and to promote reliability throughout screening and coding. All titles were reviewed by the lead author prior to exclusion or inclusion.

Title/Abstract Screening

The 4,664 unique article results were screened for eligibility based on broad features discernable from titles and abstracts (e.g., empirical paper, youth sample, relevant variables). During initial practice rounds, five screeners achieved perfect agreement on 10 of 20 training articles, with fair-to-moderate overall reliability between pairs of individual screeners (M pairwise agreement = 75%, range: 60–85%; M κ = 0.55, range: 0.28–0.72). Discrepancies were resolved by consensus, and procedures clarified and refined. A second training batch was then title-screened, with perfect agreement on 14/20 articles and moderate-to-substantial pairwise agreement (M = 85%, range: 80–90%; M κ = 0.62, range = 0.50–0.84). Title/abstract screening was then performed independently, leading to the exclusion of 3,852 articles, leaving 812 for full-text review (Figure 1).

Full-Text Review

All 812 articles were reviewed by two screeners, with discrepancies resolved by the first or second author. During an initial training phase, a kappa of at least 0.61 (“substantial”) with the lead author’s coding was set as the benchmark for reliability. The team approached but did not uniformly meet this target. All articles were therefore subjected to full-text review by two screeners, among whom there was moderate agreement (79%, κ = 0.56), including at least one reliable screener, among whom there was substantial agreement (82%, κ = 0.63). This process excluded 494 studies, leaving 318 studies considered relevant for inclusion (see Figure 1).

Measure Review Processes

Measure Identification

The 318 included articles were reviewed to extract the names and citations of the relevant measure(s) they focused on. This list of measures was then reviewed collectively and iteratively, including additional searches to ascertain the full evidence base for measures under consideration. From this process, 55 measures emerged as meeting criteria for inclusion. These measures were later joined by 13 additional measures, identified and from other sources (see Figure 1). Thus, a total of 68 measures were included in the review.

Psychometric Evaluation and Data Extraction

As described above, included measures were subjected to additional searches to ensure the evidence base was adequately represented. This resulted in a representative set of sources to review for each measure. Articles were reviewed following the rubrics developed for evaluating assessment instruments, as shown in Table 1 (De Los Reyes & Langer, 2018; Hunsley & Mash, 2008; Youngstrom et al., 2017). Criteria could be rated as “Excellent,” “Good,” “Adequate,” or “Insufficient Evidence to Rate.” Of the original 12 EBU criteria, 8 were included in our review and all achieved strong interrater reliability: (a) norms, κ = 0.69; (b) internal consistency, κ = 0.87; (c) test-retest reliability, κ = 0.73; (d) content validity, κ = 0.70; (e) construct validity, κ = 0.93; (f) validity generalization, κ = 0.68; (e) treatment sensitivity, κ = 0.80; (f) clinical utility, κ = 0.71. The other 4 criteria were not applicable, rarely reported, or irrelevant⁷ to our focus. All criteria were evaluated by at least two coders, with discrepancies resolved by the first or second author. Minor elaborations were made to operationalize these criteria (e.g., for clinical utility to be adequate, we required that the measure must be intended for clinician use and have at least some evidence from clinical samples; see Table 1).

Based on these ratings, we formed a few composite evaluations of each measure. First, similar to some previous EBU reviews (e.g., Becker-Haimes et al., 2020; Mazefsky et al., 2021), we sought to provide a classification of each measure as being Adequate, Good, or Excellent overall. We did this by considering each criterion rating as evidence for either external validity (norms, generalizability), reliability (internal consistency, test-retest), measurement validity (content and construct validity), and utility (treatment sensitivity, clinical utility). To be considered Excellent overall, a measure had to satisfy criteria for Excellent in at least 3 of 4 major categories and reach Good in the fourth. To be considered Good overall, a measure had to meet the criteria for Good in at least 3 of the 4 categories and reach Adequate in the fourth. To be considered Adequate overall, a measure had to reach Adequate in all 4 categories. In addition, an overall score was computed as the sum of all ratings, where Excellent = 3, Good = 2, Adequate = 1, and Insufficient Evidence = 0 (possible range = 0–24, higher scores being more favorable). Finally, similar to Etkin, Lebowitz, et al. (2021) and Etkin, Shimshoni, et al. (2021), we tallied the number of data points reaching each level (Excellent, Good, etc.) by row (measure), as well as by column (criterion) for a snapshot of the extent to which different types of psychometric evidence had accumulated. Beyond the psychometric ratings, we also extracted additional descriptive information about each measure, obtained from the measures themselves or their web sites, manuals, key papers, and the literature reviewed.

Results

Sixty-eight measures met criteria for inclusion. Table 2 lists the abbreviated and full instrument names, key citations, item counts, informant(s), age range, and focus/foci. Focus was coded in terms of whether the measure included scales targeting specific constructs of interest—irritability, anger, aggression, or general overt externalizing problems—and which of these constructs was considered the primary focus. Psychometric ratings and relevant scales are summarized by focus in Table 3 (for measures covering irritability or anger; n = 30, 44.1%) and Table 4 (for measures focusing on aggression or overt externalizing problems; n = 38, 55.9%). The supplement provides additional details for each measure including response scale, years, translations, notes, samples, norm types (e.g., t-scores, means/SDs), whether it is freely accessible, and how to access (Table S4), as well as the literature reviewed for each measure (Table S5). Most measures (n = 46, 67.6%) had been studied in 2 or more different types of samples, with school/community samples (n = 57, 83.8%) and clinical samples (n = 53, 77.9%) being the most common, followed by juvenile justice samples (n = 12, 17.6%).

Table 2. Summary of measures included in review (N = 68).

Abbr.	Full Measure Name	Key Citation(s)	Items^a	Primary Focus^b	All Foci^b				Informant^c				See Table	Ages^d
					ir	an	ag	ex	p	s	t	c
ABC	Aberrant Behavior Checklist (2nd ed.)	Aman and Singh (2017)*	58	ir	ir			ex	p		t	c	3A	5-adult
ARI	Affective Reactivity Index	Stringaris et al. (2012)^#	7	ir	ir				p	s		c	3A	3-adult
EMO-I	Emotional Outburst Inventory	Carlson, Silver, et al. (2022)^#	9	ir	ir	an	ag		p			c	3A	4-18
MAP-DB	Multidimensional Assessment of Preschool Disruptive Behavior	Wakschlag et al. (2012)^#; 2014^#	78	ir	ir		ag		p				3A	2-8
RAQ	Rage Attacks Questionnaire	Budman et al. (2003)^#	22	ir	ir	an	ag		p	s			3A	6-17
ROARS	Rage Outbursts and Anger Rating Scale	Nadeau et al. (2016)^#	3	ir	ir	an	ag					c	3A	4-18
Rothbart	Rothbart Child/Adolescent Temperament Questionnaires (CBQ, TMCQ, EATQR)	Capaldi and Rothbart (1992)^#; Rothbart et al. (2001)^#; Simonds and Rothbart (2004)^~	36–195	ir	ir		ag		p	s	t		3A	3-15
AARS	Adolescent Anger Rating Scale	Burney (2001)*	41	an		an				s			3B	11-19
AESC	Anger Expression Scale for Children	Steele et al. (2009)^#	26	an		an				s			3B	7-18
AXS	Anger Expression Scale	Spielberger (1985)^+	20	an		an				s			3B	10-adult
BYI	Beck Youth Inventories–Anger & Disruptive Behavior (2nd ed.)	Beck et al. (2005)*	20&20	an		an	ex			s			3B	7-18
CAMS	Children’s Anger Management Scales	Zeman et al (2002^#; Zeman et al., 2010^#).	11	an		an				s			3B	4-15
ChIA	Children’s Inventory of Anger	Nelson and Finch (2000)*	39	an	ir	an	ag			s			3B	6-16
MAYSI	Massachusetts Youth Screening Instrument (2nd ed.)	Grisso and Barnum (2006)*	52	an	ir	an				s			3B	12-17
MMPI	Minnesota Multiphasic Personality Inventory (Adolescent / 3rd ed.) / Cook-Medley Hostility	Ben-Porath and Tellegen (2020)*; Butcher et al. (1992)*	241-478	an	ir	an		ex		s			3B	14-adult
MSAI	Multidimensional School Anger Inventory	Smith et al. (1998)^#	12-36	an		an				s			3B	10-18
NAS-PI	Novaco Anger Scale & Provocation Inventory	Novaco (2003)*	60&25	an		an				s			3B	9-adult
PAES	Pediatric Anger Expression Scale	Jacobs et al. (1989)^#	10-15	an		an				s			3B	7-18
PROMIS	PROMIS Anger Measures	Irwin et al. (2012)^#; PROMIS (2016)	5-22	an		an			p	s			3B	5-adult
STAXI	State-Trait Anger Expression Inventory (2^nd ed., Child / Adolescent)	Brunner and Spielberger (2009)*; Spielberger (1999)*	35-57	an		an				s			3B	9-adult
AQ	Aggression Questionnaire	Buss and Warren (2000)*	34	ag		an	ag			s			3C	9-adult
AS	Aggression Scale	Orpinas and Frankowski (2001)^#	11	ag		an	ag			s			3C	8-17
CHI/HI	Children’s Hostility Inventory / Buss-Durkee	Buss and Durkee (1957)^#; Kazdin et al. (1987)^#	34-75	ag	ir	an	ag		p	s			3C	6-adult
HCSBS/SSBS	Home and Community Social Behavior Scale / School Social Behavior Scale (2nd ed.)	Merrell (2002b)*; Merrell and Caldarella (2002)*	64	ag	ir		ag		p		t		3C	5-18
PAI-A	Personality Assessment Inventory(-Adolescent)	Morey (2007)*	264-344	ag	ir		ag	ex		s			3C	12-adult
AdolBC	Adolescent Behavior Checklist	Adams et al. (1997)^#	48	ex	ir			ex		s			3D	11-17
BASC	Behavior Assessment System for Children (3rd ed.)	Kamphaus and Reynolds (2015)*; Reynolds and Kamphaus (2015)*	5-175	ex		an	ag	ex	p	s	t		3D	2-adult
BBRS	Burks’ Behavior Rating Scales (2nd ed.)	Burks and Gruber (2006)*	100	ex		an	ag	ex	p		t		3D	4-18
NCBRF	Nisonger Child Behavior Rating Form	Aman et al. (1996)^#	56-76	ex	ir			ex	p		t		3D	3-18
SCBE	Social Competence and Behavior Evaluation	LaFreniere and Dumas (1995)^#; LaFreniere & Dumas, 1996^#	30-80	ex		an	ag	ex	p		t		3D	3-11
CAS	Children’s Aggression Scale	Halperin and McKay (2008)*	23-33	ag			ag		p		t		4A	5-18
CBS	Child Behavior Scale	Ladd and Profilet (1996)^#	35-59	ag			ag				t		4A	4-13
C-PEQ	Cyber Peer Experiences Questionnaire	Landoll et al. (2015)^#	18	ag			ag			s			4A	13-19
CSBS	Children’s Social Behavior Scale	Crick (1996)^#; Crick and Grotpeter (1995)^#	6-27	ag			ag		p	s	t		4A	3-18
C-SHARP	Children’s Scale of Hostility and Aggression: Reactive/Proactive	Farmer and Aman (2009)^#	48	ag			ag		p				4A	1-21
DIAS	Direct and Indirect Aggression Scale	Björkqvist et al. (1992)^#	24-48	ag			ag		p	s	t		4A	4-18
ICS	Interpersonal Competence Scale	Cairns et al. (1995)^#	18	ag			ag		p	s	t		4A	6-18
Little	Little's Forms and Functions of Aggression	Little et al. (2003)^#	36	ag			ag		p	s	t		4A	3-16
MAVRIC	Measure of Aggression, Violence, and Rage in Children	Goodman et al. (2006)^#	19	ag			ag		p	s			4A	5-18
PBFS	Problem Behavior Frequency Scale	Farrell et al. (2016)^#	35-73	ag			ag			s	t		4A	10-18
PCS	Peer Conflict Scale	Marsee and Frick, (2007)^#; Marsee et al. (2011)^#	20-40	ag			ag		p	s	t		4A	9-19
PIPPS	Penn Interactive Peer Play Scale	Fantuzzo et al. (1995)^#	32-36	ag			ag		p		t		4A	5-8
PRA	Proactive-Reactive Aggression Scale	Dodge and Coie, 1987^#	6	ag			ag		p	s	t		4A	3-18
RBPC	Revised Behavior Problem Checklist	Quay and Peterson (1983)*	89	ag			ag		p		t		4A	5-18
R-MOAS	(Retrospective/Modified) Overt Aggression Scale	Coccaro (2020)^#; Kay et al. (1988); Sorgi et al. (1991)^#; Yudofsky et al. (1986)^#	16-20	ag			ag		p			c	4A	4-19
R-PEQ	(Revised) Peer Experiences Questionnaire	Prinstein et al. (2001)^#; Vernberg et al. (1999)	9-18	ag			ag			s			4A	11-19
RPQ	Reactive Proactive Aggression Questionnaire	Raine et al. (2006)^#	23	ag			ag		p	s	t		4A	7-16
R-TPRA/ABS	Revised Teacher Rating Scale for Proactive and Reactive Aggression/ Antisocial Behavior Scale	Brown et al. (1996)^#	16-28	ag			ag		p		t		4A	6-16
TODS	Tourette’s Disorder Scale	Shytle et al. (2003)^#	15	ag			ag		p			c	4A	6-17
WAI	Weinberger Adjustment Inventory	Weinberger (1994)*; Weinberger, (1997^#)	37-62	ag			ag			s			4A	10-20
ECBI/SESBI	Eyberg Child Behavior Inventory / Sutter-Eyberg Student Behavior Inventory	Eyberg and Pincus (1999)*	36-38	ex				ex	p		t		4B	2-16
HBRS	Human Behavior Rating Scale	Woods-Groves (2015)^#	30-94	ex				ex			t		4B	5-20
PIBS	Pediatric Inpatient Behavior Scale	Kronenberger (2022)*; Kronenberger et al. (1997)^#	25-47	ex				ex				c	4B	4-18
PSC	Pediatric Symptom Checklist	Jellinek and Murphy (1988)^#	17-35	ex				ex	p	s	t		4B	3-17
Rutter	Rutter Children’s/ Preschool Behaviour Questionnaire	Behar and Stringfield (1974)^#; Rutter (1967)^#	26-50	ex				ex	p		t		4B	3-6
SCL-90/BSI	Symptom Checklist-90, Revised / Brief Symptom Inventory (BSI)	Derogatis (1993)*; Derogatis, 1994	53-90	ex				ex		s			4B	13-adult
SDQ	Strengths and Difficulties Questionnaire	Goodman (1997)^#	35-30	ex				ex	p	s	t		4B	2-adult
SEBS	Student Externalizing Behavior Screener	Cook (2012)^#	7	ex				ex			t		4B	5-14
ASCA/ASPI	Adjustment Scales for Children and Adolescents / Preschool Intervention	Lutz et al. (2002)^#; McDermott (1993)^#	144-156	ex			ag	ex			t		4C	2-17
ASEBA	Achenbach System of Empirically Based Assessment	Achenbach and Rescorla (2001)*; Achenbach et al. (2017)*	18-116	ex			ag	ex	p	s	t		4C	1-adult
IAB	Interview for Antisocial Behavior	Kazdin and Esveldt-Dawson (1986)^#	30	ex			ag	ex	p				4C	2-15
interRAI	interRAI Child and Youth Mental Health	Stewart et al. (2015)*	5-400+	ex			ag	ex	p	s		c	4C	4-adult
Jesness	Jesness Inventory Scales Revised	Jesness (2003)*	160	ex			ag	ex		s			4C	8-adult
MHBQ	MacArthur Health and Behavior Questionnaire	Armstrong et al. (2003); Essex et al. (2002)^#	105-173	ex			ag	ex	p	s	t		4C	9-18
PKBS	Preschool and Kindergarten Behavior Scales	Merrell (2002a)*	76	ex			ag	ex	p		t		4C	3-6
SBS	Student Behavior Survey	Lachar et al. (2000)*	102	ex			ag	ex			t		4C	5-18
SSIS/SSRS	Social Skills Improvement / Rating System	Gresham and Elliott (2008)*; Gresham & Elliott, 2017*	46-58	ex			ag	ex	p	s	t		4C	3-18
TOCA	Teacher Observation of Classroom Adaptation	Koth et al. (2009)^#	21-54	ex			ag	ex	p		t		4C	5-11

Notes: ^a Item count is for the full measure, not limited to relevant scales. Where multiple versions (e.g., short and long forms, informant- or age-specific forms) are available, item count is reported as a range from shortest to longest version identified. ^b Focus represents our classification of the relevant scale(s) based on scale name, item content, and measure development. ^c Informant is coded based on the presence of any articles included in review using a given informant, even if not originally developed for that informant. ^d Ages are the prescribed age range if available; otherwise it is the age range in which the measure has been used in the reviewed articles. “Adult” = age 25 or higher. *Published assessment manual, ^#peer-reviewed journal article, ⁺chapter, ^~presentation (in order of priority for most appropriate citation). Abbreviations for focus and informant: ag = aggressive behavior, an = anger, ir = irritability, ex = general externalizing problems characterized by ag, an, and ir; p = parent-report, t = teacher, s = self-report, c = clinician. Other abbreviations not defined in table: Rothbart: CBQ = Children’s Behavior Questionnaire, TMCQ = Temperament in Middle Childhood Questionnaire, EATQR = Early Adolescent Temperament Questionnaire Revised.

Table 3. Measures covering irritability and anger (N = 30).

Abbreviated Measure Name & Relevant Scale(s)		Psychometric Ratings								Overall Rating	Rating Totals (E/G/A/I)	Rating Score (0–24)
		Norms	Internal consistency	Test-retest reliability	Content validity	Construct validity	Validity generalization	Treatment sensitivity	Clinical utility
A. Irritability-Focused Measures
ABC	Irritability Hyperactivity/ Noncompliance	Good	Excellent	Good	Good	Good	Excellent	Excellent	Adequate	Excellent	3/4/1/0	18
ARI	Irritability Sx Impairment	Good	Good	Adequate	Good	Excellent	Excellent	Excellent	Adequate	Excellent	3/3/2/0	17
EMO-I	Outburst … Severity Frequency Duration	Adequate	Adequate	Insuffic	Adequate	Excellent	Adequate	Insuffic	Adequate	Adequate	1/0/5/2	8
MAP-DB	Temper Loss Noncompliance Ag Low Concern	Adequate	Excellent	Good	Adequate	Good	Adequate	Adequate	Adequate	Adequate	1/2/5/0	12
RAQ*	Rage Attack	Good	Insuffic	Insuffic	Adequate	Adequate	Adequate	Insuffic	Adequate	Adequate	0/1/4/3	6
ROARS	Rage Frequency, Intensity, & Duration	Adequate	Good	Adequate	Adequate	Good	Adequate	Excellent	Adequate	Good	1/2/5/0	12
Rothbart*	An/Frustration Ag	Adequate	Adequate	Adequate	Adequate	Good	Adequate	Insuffic	Adequate	Adequate	0/1/6/1	8
B. Anger-Focused Measures
AARS	Total An Inst An Rea An An Control	Excellent	Good	Adequate	Excellent	Good	Good	Good	Adequate	Good	2/4/2/0	16
AESC	Trait An An Expression An In An Control	Good	Good	Adequate	Adequate	Good	Excellent	Good	Adequate	Good	1/4/3/0	14
AXS	An In An Out An Control An Expression	Good	Adequate	Good	Adequate	Adequate	Good	Adequate	Adequate	Adequate	0/3/5/0	11
BYI	An Disruptive Bx	Excellent	Good	Adequate	Adequate	Good	Good	Good	Adequate	Good	1/4/3/0	14
CAMS	An Coping An Dysreg An Inhibition	Good	Adequate	Adequate	Good	Good	Excellent	Adequate	Adequate	Adequate	1/3/4/0	13
ChIA*	Frustration Phys Ag Peer Relations Authority Relations	Excellent	Good	Good	Good	Good	Good	Excellent	Adequate	Good	2/5/1/0	17
MAYSI	An-Irritable	Excellent	Good	Adequate	Good	Good	Good	Insuffic	Adequate	Good	1/4/2/1	13
MMPI*	An Explosive Bx Irritability Ag Conduct Px Antisocial Bx	Good	Adequate	Good	Good	Excellent	Good	Insuffic	Adequate	Good	1/4/2/1	13
MSAI	An Experience Hostility Destructive Expression Positive Coping	Good	Adequate	Adequate	Adequate	Adequate	Good	Insuffic	Adequate	Adequate	0/2/5/1	9
NAS-PI	An Total Cognitive Arousal Bx An Regulation Provocation Total	Excellent	Good	Adequate	Good	Good	Good	Good	Adequate	Good	1/5/2/0	15
PAES	An Out An Control An Reflection An Suppression	Good	Adequate	Adequate	Adequate	Adequate	Excellent	Adequate	Adequate	Adequate	1/1/6/0	11
PROMIS	An	Good	Good	Adequate	Good	Good	Excellent	Excellent	Adequate	Good	2/5/1/0	16
STAXI*	State An Trait An An Express-Out An Express-In An Control	Excellent	Good	Adequate	Good	Good	Good	Excellent	Adequate	Good	2/4/2/0	16
C. Aggression Measures Covering Irritability and/or Anger
AQ	Phys Ag Verbal Ag An Hostility Indirect Ag	Excellent	Adequate	Good	Good	Good	Excellent	Good	Adequate	Good	2/4/2/0	16
AS	Phys/Verbal Ag An Total Ag	Adequate	Good	Good	Good	Adequate	Adequate	Good	Adequate	Good	0/4/4/0	12
CHI/HI*	Ag Hostility Assaultiveness Indirect Hostility Verbal Hostility Resentment Irritability	Good	Good	Insuffic	Adequate	Good	Good	Adequate	Adequate	Good	0/4/3/1	11
HCSBS/ SSBS*	Antisocial Bx Hostile/Irritable Antisocial/Ag Def/Disruptive	Excellent	Excellent	Adequate	Good	Good	Good	Good	Adequate	Good	2/4/2/0	16
PAI-A*	Irritability Antisocial Bx Ag Attitude Verbal Ag Phys Ag Affective Instability	Excellent	Good	Adequate	Excellent	Excellent	Good	Adequate	Adequate	Good	3/2/3/0	16
D. General Overt Externalizing Measures Covering Irritability and/or Anger
AdolBC	Conduct Px Emotional Lability	Good	Good	Adequate	Adequate	Good	Good	Insuffic	Adequate	Good	0/4/3/1	11
BASC*	Ag An Control Externalizing Px Bullying	Excellent	Excellent	Good	Good	Good	Excellent	Excellent	Adequate	Excellent	4/3/1/0	19
BBRS	Disruptive Bx Poor An Control Ag Tendencies Rebelliousness	Excellent	Good	Adequate	Good	Good	Adequate	Insuffic	Adequate	Good	1/3/3/1	12
NCBRF*	Disruptive Bx Opp Conduct Px Irritable	Good	Good	Insuffic	Good	Excellent	Good	Excellent	Adequate	Good	2/4/1/1	15
SCBE	Angry Ag Opp Externalizing	Adequate	Good	Adequate	Adequate	Good	Adequate	Adequate	Adequate	Adequate	0/2/6/0	10
Totals	(E/G/A/I) Mdn Rating	11/13/6/0 Good	4/17/8/0 Good	0/8/18/4 Adequate	2/15/13/0 Good	5/20/5/0 Good	8/14/8/0 Good	8/7/7/8 Adequate	0/0/30/0 Adequate	3/18/9/0 Good	M (SD)	13.23 (3.21)

Note: Ag—aggression or aggressive, An = anger, Bx = behavior(s), Insuffic = insufficient evidence to rate, Inst = instrumental, Phys = physical, Pro = proactive, Px = problems, Rea = reactive, Rel = relational, Sx = symptoms. Opp = oppositional(ity). *Additional relevant subscales not listed here or varies across versions.

Table 4. Measures focused on aggression and general overt externalizing problems (N = 38).

Abbreviated Measure Name & Relevant Scale(s)		Psychometric Ratings								Overall Rating	Rating Totals (E/G/A/I)	Rating Score (0–24)
		Norms	Internal consistency	Testretest reliability	Content validity	Construct validity	Validity generalization	Treatment sensitivity	Clinical utility
A. Aggression-Focused Measures
CAS*	Verbal Ag Ag Against Objects/Animals Provoked Phys Ag Initiated Phys Ag Use of Weapons Total Ag Index	Good	Good	Adequate	Good	Good	Excellent	Excellent	Adequate	Good	2/4/2/0	16
CBS	Ag with Peers	Good	Good	Adequate	Adequate	Good	Adequate	Good	Adequate	Good	0/4/4/0	12
C-PEQ	Cyber Peer Ag	Good	Adequate	Insuffic	Good	Good	Good	Adequate	Adequate	Good	0/4/3/1	11
CSBS	Overt Ag Rel Ag	Good	Good	Good	Good	Good	Good	Excellent	Adequate	Good	1/6/1/0	16
C-SHARP	Verbal Ag Phys Ag Covert Ag Hostility Bullying	Good	Good	Insuffic	Adequate	Good	Adequate	Insuffic	Adequate	Good	0/3/3/2	9
DIAS	Phys Ag Direct Verbal Ag Indirect Ag	Good	Good	Insuffic	Adequate	Adequate	Adequate	Insuffic	Adequate	Adequate	0/2/4/2	8
ICS	Ag	Adequate	Adequate	Adequate	Adequate	Adequate	Adequate	Insuffic	Insuffic	Adequate	0/0/6/2	6
Little	Pure Overt Ag Rea Overt Ag Inst Overt Ag Pure Rel Ag Rea Rel Ag Inst Rel Ag	Good	Good	Insuffic	Adequate	Adequate	Good	Insuffic	Adequate	Adequate	0/3/3/2	9
MAVRIC	Total Ag	Adequate	Good	Insuffic	Adequate	Good	Excellent	Adequate	Adequate	Good	1/2/4/1	11
PBFS	Phys Ag Verbal Ag Rel Ag	Good	Good	Insuffic	Adequate	Good	Good	Good	Adequate	Good	0/5/2/1	12
PCS	Phys-Pro Ag Phys-Rea Ag Rel-Pro Ag Rel-Rea Ag	Good	Good	Insuffic	Good	Good	Excellent	Adequate	Adequate	Good	1/4/2/1	13
PIPPS	Play Disruption	Adequate	Good	Adequate	Good	Good	Good	Insuffic	Adequate	Good	0/4/3/1	11
PRA	Pro Ag Rea Ag	Good	Good	Good	Good	Good	Excellent	Insuffic	Adequate	Good	1/5/1/1	14
RBPC	Conduct Disorder Socialized Ag	Good	Excellent	Adequate	Good	Good	Excellent	Excellent	Adequate	Excellent	3/3/2/0	17
R-MOAS	Verbal Ag Property Ag Self Ag Phys Ag Total Score	Adequate	Adequate	Insuffic	Good	Adequate	Adequate	Good	Adequate	Adequate	0/2/5/1	9
R-PEQ	Overt Ag Rel Ag Reputational Ag	Adequate	Good	Insuffic	Good	Good	Good	Adequate	Adequate	Good	0/4/3/1	11
RPQ	Pro Ag Rea Ag	Adequate	Good	Adequate	Adequate	Good	Good	Adequate	Adequate	Good	0/3/5/0	11
R-TPRA/ ABS	Rea Ag Pro Ag Covert Antisocial	Good	Good	Good	Good	Good	Good	Adequate	Adequate	Good	0/6/2/0	14
TODS	Ag	Adequate	Excellent	Insuffic	Adequate	Good	Adequate	Excellent	Adequate	Good	2/1/4/1	12
WAI	Suppression of Ag	Adequate	Good	Good	Adequate	Good	Excellent	Excellent	Adequate	Good	2/3/3/0	15
B. Measures of General Overt Externalizing Problems
ECBI/ SESBI	Intensity Problem	Excellent	Excellent	Good	Adequate	Excellent	Excellent	Good	Adequate	Excellent	4/2/2/0	18
HBRS	Externalizing Affect	Adequate	Excellent	Insuffic	Adequate	Good	Good	Insuffic	Insuffic	Good	1/2/2/3	9
PIBS	Opp- Noncompliant Conduct Px	Good	Adequate	Insuffic	Good	Good	Adequate	Insuffic	Adequate	Adequate	0/3/3/2	9
PSC	Conduct	Excellent	Adequate	Adequate	Good	Good	Excellent	Adequate	Adequate	Adequate	2/2/4/0	14
Rutter*	Conduct Px/Opp Ag Antisocial	Good	Good	Good	Good	Good	Good	Adequate	Adequate	Good	0/7/1/0	15
SCL-90/ BSI	Hostility	Good	Good	Insuffic	Adequate	Good	Good	Good	Adequate	Good	0/5/2/1	12
SDQ	Conduct Px	Excellent	Good	Adequate	Adequate	Good	Good	Adequate	Adequate	Good	1/3/4/0	13
SEBS	Externalizing	Adequate	Good	Adequate	Adequate	Good	Adequate	Good	Insuffic	Good	0/3/4/1	10
C. Measures of Overt Externalizing Problems that Cover Aggression
ASCA/ ASPI*	Ag Scales (vary) Opp	Excellent	Excellent	Adequate	Good	Good	Excellent	Adequate	Insuffic	Good	3/2/2/1	15
ASEBA	Ag Bx Externalizing Px Opp Def Px	Excellent	Good	Good	Good	Excellent	Excellent	Excellent	Adequate	Excellent	4/3/1/0	19
IAB	Arguing/Fighting Covert Antisocial	Adequate	Adequate	Insuffic	Adequate	Good	Adequate	Good	Adequate	Adequate	0/2/5/1	9
interRAI	Disruptive/Ag Bx Externalizing Pro Ag/ Externalizing Rea Ag/ Externalizing	Good	Good	Insuffic	Excellent	Good	Adequate	Adequate	Adequate	Good	1/3/3/1	12
Jesness	Manifest Ag Conduct Disorder Opp Def Disorder	Excellent	Good	Adequate	Good	Good	Good	Adequate	Adequate	Good	1/4/3/0	14
MHBQ	Externalizing Sx Opp Conduct Px Overt Hostility Rel Ag	Good	Good	Good	Good	Good	Excellent	Insuffic	Insuffic	Good	1/5/0/2	13
PKBS	Externalizing Px Self-Centered/ Explosive Antisocial/Ag	Excellent	Excellent	Good	Good	Good	Good	Good	Adequate	Good	2/5/1/0	17
SBS	Verbal Ag Phys Ag Bx Px Opp Def Px Conduct Px	Excellent	Good	Good	Good	Good	Excellent	Good	Adequate	Good	2/5/1/0	17
SSIS/ SSRS	Bullying Externalizing	Excellent	Good	Good	Good	Good	Good	Excellent	Adequate	Good	2/5/1/0	17
TOCA	Disruptive/Ag Bx	Adequate	Good	Good	Adequate	Good	Good	Good	Insuffic	Good	0/5/2/1	12
Totals	(E/G/A/I) Mdn Rating	9/17/12/0 Good	6/26/6/0 Good	0/12/11/15 Adequate	1/20/17/0 Good	2/32/4/0 Good	12/16/10/0 Good	7/10/12/9 Adequate	0/0/32/6 Adequate	3/28/7/0 Good	M (SD)	12.66 (3.10)

Note: Ag—aggression or aggressive, An = anger, Bx = behavior(s), Insuffic = insufficient evidence to rate, Inst = instrumental, Phys = physical, Pro = proactive, Px = problems, Rea = reactive, Rel = relational, Sx = symptoms. Opp = oppositional(ity). *Additional relevant subscales not listed here or varies across versions.

In terms of overall psychometric evaluations, only 6 measures (8.8%) were classified as Excellent, while 46 (67.6%) were considered Good, and the remaining 16 (23.5%) were Adequate. The numeric scores (i.e., the sum of psychometric ratings; possible range = 0–24) were roughly normally distributed, with a median of 13 (M = 12.91, SD = 3.14; observed range = 6–19). Thus, measures with scores ≥ 16 (75^th percentile) in Tables 3–4 can be viewed as having relatively strong psychometric evidence, whereas those with scores ≤ 11 (25^th percentile) might benefit from further empirical examination. Ratings did not differ between anger/irritability vs. aggression/externalizing measures, t(66) = 0.747, p = .46. However, measures that were accompanied by manuals (n = 28), which typically had to be purchased from the publisher, had better psychometric properties (M = 15.07, SD = 2.65) than instruments without manuals (M = 11.43, SD = 2.56), t(66) = 5.695, p < .001, d = 1.40.

We also looked at the overall state of the evidence on each criterion, pooling across all measures (see bottom of Tables 3–4). The evidence reached a median rating of Good for norms, internal consistency, content validity, construct validity, and validity generalization. However, the evidence was lower (i.e., Adequate) for test-retest reliability, treatment sensitivity, and clinical utility. Notably, no measures reached Good or Excellent on clinical utility. Despite such differences in the distributions of ratings, all eight criteria seemed to provide useful information about individual measures.

As shown in Table 2, only 7 instruments (10.3%) were classified as primarily for irritability, 13 (19.1%) for anger, 25 (36.8%) for aggression, and 23 (33.8%) for general overt externalizing problems. However, 30 (44.1%) measures contained multiple scales that collectively covered 2 or more of these constructs, with 10 (14.7%) covering 3 or more. This seemed to support our broad search and coding strategies, as the results yielded relevant and overlapping scales that might otherwise be overlooked.

Measures Focusing on Irritability

Overall, 15 instruments were identified that measured irritability, including 7 that we classified as having irritability as their primary focus. These measures are listed in Table 2, with psychometric ratings listed in see Table 3 (Sections A, C, and D). Of these measures, only 2—the ARI and the Aberrant Behavior Checklist (ABC)—emerged as having Excellent overall psychometric properties (9 were Good, 4 were Adequate). Both the ARI and the ABC have been widely used with at least good evidence for internal consistency, norms, content and construct validity, generalization, and treatment sensitivity. The ABC was developed for use with children, adolescents, and adults with developmental delays, in various settings and proxy raters (Aman & Singh, 2017). It is commercially available and a relatively comprehensive measure (58 items) of different behavior problems common in youth with autism spectrum disorder (e.g., irritability, hyperactivity/noncompliance), although there is evidence specifically for its use in addressing irritability/aggression (Frazier et al., 2010; Stoddard et al., 2020). For a free, brief, and multi-informant measure of irritability, the ARI has emerged as a leading standard over the last decade, with psychometric evidence from diverse clinical and community samples (Dougherty et al., 2021; Evans et al., 2021; Mulraney et al., 2014; Stringaris et al., 2012; Tseng et al., 2017; Wilson et al., 2022). Recently, a clinician version, the CL-ARI, has been developed (Haller et al., 2020); it has also been adapted into a short-term version, translated into 15+ languages, with initial psychometric evidence for teacher-report in Spanish (Ezpeleta et al., 2020).

The other 13 measures covering irritability and related constructs were rated as Good or Adequate. These include instruments rated by single or multiple informants including parent, teacher, self, and clinician. Notably, these measures cover some specific aspects of irritability, including emotional or aggressive outbursts (ROARS, RAQ, EMO-I), general externalizing problems (NCBRF), personality or temperament (MMPI, PAI-A, Rothbart measures), as well as broader measures for specific age ranges and purposes from early childhood (MAP-DB) to childhood and adolescence (CHIA, AdolBC, MAYSI, CHI/HI, HCSBS/SSBS). More information about the characteristics, strengths, weaknesses, and evidence base for these measures can be found in Tables 2, 3, S4, and S5.

Measures Focusing on Anger

We identified 19 measures covering anger (13 as a primary focus). Of these, only one measure—the BASC—reached criteria for Excellent and, with a score of 19, was one of the two highest-rated instruments overall (see Table 3, Sections B, C, and D). The BASC is a comprehensive multi-informant system for measuring emotional and behavioral functioning in youth (Reynolds & Kamphaus, 2015). As a system, the BASC also includes tools for screening and progress-monitoring (Kamphaus & Reynolds, 2015). The BASC is included here due to its Anger Control scale, which is new to the third edition, although it also includes Aggression, Conduct Problems, Externalizing Problems, and other broadband and narrowband scales for emotional and adaptive functioning (Reynolds & Kamphaus, 2015). The BASC is commonly used in clinical as well as school settings, with nationally representative norms and evidence accruing over 3 editions and more than 3 decades (e.g., Canivez et al., 2021; Flanagan et al., 1996; Lochman, Dishion, et al., 2015).

Of the remaining measures covering anger, 13 were classified as Good and 5 as Adequate. Of note, many of these instruments offer multiple dimensions of anger-related constructs, such as anger expression in and out, anger control, coping, dysregulation, inhibition, hostility, cognitive, arousal, behavioral, and provocations (e.g., AARS, AESC, AXS, CAMS, ChIA, NAS-PI, PAES, STAXI; see Table 3). Other measures offer only a single or total scale related to anger (e.g., BYI, PROMIS), although these can offer other benefits (e.g., a more comprehensive measurement suite, parent- and youth-report progress-monitoring). It is apparent from these anger measures that—unlike irritability and aggression—anger has historically been measured primarily by self-report, with all having a youth self-report version and only two (PROMIS, CAMS) having a corresponding parent-report version. We found no teacher-report measures of anger. It may be that views of “anger” have focused more on one’s subjective experiences of that emotion, whereas irritability and aggression measures are more likely to draw upon ratings from others (e.g., parents, teachers). We also observed a pattern that was generally unique to anger measures, where these instruments were often developed initially with a focus on adults (e.g., personality, hostility), with several measures being downward extensions (AQ, AXS, MMPI, CHI/HI, NAS-PI, PAI). Nonetheless, the evidence reviewed all came from youth studies.⁸

Measures Focusing on Aggression

Overall, 45 measures (66.2%) were identified for aggression (25 as the primary focus)—by far the largest focus of measurement included in this review. Ratings are listed in Table 4 (Sections A and C) and Table 3 (Sections C and D). Of these, only a few were classified as having Excellent psychometrics overall: the BASC-3, the Achenbach System of Empirically Based Assessment (ASEBA; Achenbach & Rescorla, 2001) and the Revised Behavior Problem Checklist (RBPC; Quay & Peterson, 1983). Comparable to the BASC (already discussed above), ASEBA is a suite of multi-informant measures of emotional and behavioral functioning that received the other highest psychometric rating in the sample. Comprehensive (CBCL, YSR, TRF; Achenbach & Rescorla, 2001) and progress-monitoring (BPM; Achenbach et al., 2017) instruments are available. The ASEBA goes back several decades, a variety of translations and norms are available, and the evidence base for validity, reliability, and treatment sensitivity is extensive (Ebesutani et al., 2011; Ivanova et al., 2019; Weisz et al., 2020; Youngstrom et al., 2003). Of note, Aggressive Behavior is one of the original empirically derived syndrome scales on the CBCL/YSR/TRF instruments, with relations to the externalizing composite, DSM Oppositional Problems, and various other scales. The RBPC, with non-diagnostic scales for Conduct Disorder and Socialized Aggression, has a similarly rich history going back several decades (Hinshaw et al., 1987; Lahey & Piacentini, 1985; Quay, 1983; Santisteban et al., 2017); however, this measure has been used infrequently in recent years, and our search suggests it may have gone out of print.

Among the remaining aggression measures classified as Good (n = 33) or Adequate (n = 8), a significant variety of subtypes and specialized foci can be seen. For example, some measures focus on cyber-aggression (CPEQ; Landoll et al., 2015) or aggression during play (PIPPS; Fantuzzo et al., 1995). In particular, several attend to the proactive vs. reactive functions of aggression, including the PRA (Dodge & Coie, 1987), the revised teacher PRA (Brown et al., 1996), and the RPQ (Raine et al., 2006), or different forms (e.g., direct, indirect; overt, relational) of aggression including RPEQ (Prinstein et al., 2001), CSBS (Crick, 1996), DIAS (Björkqvist et al., 1992), and the PBFS (Farrell et al., 2016). Notably, a few measures focus on both the forms and the functions of aggression, including the PCS (Marsee & Frick, 2007) and Little et al. (2003). A handful of measures approach aggression as a single monothetic scale or conceptualization (e.g., ASCA/ASPI, IAB, interRAI, Jesness, MHBQ, PKBS, SBS, SSIS/SSRS, TOCA, TODS, WAI, MAVRIC, ICS, CBS), while others offer many different types of aggression (e.g., AQ, AS, CAS, C-SHARP, R-MOAS).

It is important to point out that many of the measures discussed above are relevant to aggression, even if not classified as such. For example, the ChIA, CHI/HI, EMO-I, RAQ, ROARS, and MAP-DB cover outbursts and tantrums—forms of aggressive behavior. Similarly, the, MMPIA, HCSBS/SSBS, and PAI-A all cover aggression in the context of personality or general behavior. Among this tier, rather than some measures being clearly better than others, what is most evident is varying foci as described above, with different strengths and weaknesses for different purposes (e.g., free vs. commercial; lengthy vs. brief; school vs. clinical). Tables 4 and S4 offer useful insights for deciding among measures in this way.

Measures of General Overt Externalizing Problems

Finally, 27 measures (39.7%) included scales covering general overt externalizing problems, 23 (33.8%) with this as the primary focus. Ratings of these measures are listed in Table 3 (Section D) and Table 4 (Sections B and C). In this category, 3 of the 4 measures (ABC, ASEBA, BASC) rated as Excellent have already been discussed above. The other was the ECBI/SESBI (Eyberg & Pincus, 1999). The ECBI is a widely used for measuring disruptive behavior problems in children, for screening and treatment monitoring and outcomes evaluation in a variety of populations (Boggs et al., 1990; Eyberg & Ross, 1978; Jeter et al., 2017; Nixon et al., 2004). The SESBI adapts the ECBI to school populations, with evidence from preschool to high school age (Bagner et al., 2010; Floyd et al., 2004; Querido & Eyberg, 2003). Several other measures in this category (e.g., ASCA/ASPI, PSC, Ruttter, SCL-90/BSI, SDQ, SBS, SEBS, SSIS/SSRS) are widely used brief or broad measures that cover overt externalizing problems in some form. Most measures in this category (n = 19 of 27) were rated as Good, while 4 were rated as Adequate. For brevity we will not summarize these ratings here, but the findings can be reviewed in Tables (primarily in Tables 3d and 4(b, c)). Of note, these overt externalizing measures may not specifically assess for irritability, anger, or aggression. However, our search and evaluation methods led us to classify them as having scales or items that were mostly related to these constructs, broadly construed.

Discussion

We sought to provide a comprehensive review of the evidence base for assessing irritability, anger, aggression, and related problems in children and adolescents. Sixty-eight measures were identified and summarized in terms of their psychometric properties and descriptive characteristics, intended to provide a resource that may be useful to researchers and clinicians. Although these results are rich with detailed findings, a fine-grained methodological analysis of individual measures is beyond our scope (interested readers are referred to the tables and the supplement). Instead, results paint a general but reliable picture of the state of the evidence base for measuring these constructs, which we have conceptualized as a heterogeneous spectrum of overt externalizing affective and behavioral problems.

Evidence Base Update and Evaluation of Psychometric Properties

Several methodological themes emerged from the measures and evidence reviewed. Broadly, findings reflect favorably on the availability of research-supported tools for assessing irritability, anger, and aggression in youth. In aggregate, the evidence reached at least “Good” for most of the measures (52 of 68; 76.5%) and most of the psychometric dimensions (5 of 8; 62.5%) used to evaluate the measures: norms, internal consistency, content validity, construct validity, and validity generalization. These conclusions should be qualified by constraints that limited what measures were included, what evidence was reviewed, and other factors affecting psychometric ratings (see Limitations). Regarding specific psychometric dimensions, findings underscore the need for more targeted research on clinical utility, treatment sensitivity, and test-retest reliability—i.e., the three domains with the lowest overall ratings (“Adequate”). In addition, more work is needed on inter-rater reliability, repeatability, discriminative validity, and prescriptive validity—the original EBU dimensions (Youngstrom et al., 2017) that could not be included in this review. A few of these criteria warrant specific comment.

First, test-retest reliability and sensitivity to change both require two or more measurement occasions. In this regard, it is not surprising that these dimensions might lag behind others given that cross-sectional designs are so common. We recommend that measure developers and psychometric researchers strive to collect longitudinal data when possible, as these data are needed to evaluate test-retest reliability, sensitivity to change, and other key questions such as longitudinal invariance (Little, 2013). However, important tensions can arise between these two assumptions: that measures should be both stable over time (as evidence for test-retest reliability) and show evidence of change over time (as evidence for change sensitivity). Different measures may value these goals differently. For example, some measures of personality (e.g., MMPI-A) and temperament (Rothbart measures) provide “Adequate” or better long-term stability and test-retest reliability, but they are not necessarily sensitive to change (both rated as Insufficient Evidence for treatment sensitivity). Conversely, measures like the NCBRF and the CAS may function as “Excellent” measures for treatment sensitivity precisely because sensitivity to change is prioritized over longitudinal stability (rated as Insufficient Evidence and Adequate, respectively, for test-retest reliability). The standard for “Adequate” test-retest reliability (i.e., r = 0.7 over a week) may be sufficient, but stretching this period longer, over months and years, begins to reflect trait-like longitudinal stability of the construct, not just the reliability of the instrument. As researchers increasingly study within-person changes and variability on shorter time scales (e.g., over days, hours, and minutes), alternative indicators of reliability will be needed. Such measures were excluded from this review, largely due to the use of single-items and lack of proper names and standardized formats, but they represent an important future direction for research in this area (Evans et al., 2023).

Clinical utility is gaining momentum in many areas of assessment (De Los Reyes & Langer, 2018; Youngstrom et al., 2017), from diagnosis and classification (Keeley et al., 2016) to measurement-based care (McLeod et al., 2022). Whether something is clinically useful is an empirical question, and usually there is vanishingly little evidence to answer it. Of note, most measures are likely to offer some degree of clinically useful information—hence, 62 of 68 (91.2%) measures reviewed here were classified as Adequate for clinical utility, while those that fell short of this threshold were not intended for clinical use. But none reached “Good” or higher, as this requires “published evidence that using the assessment data confers clinical benefit (e.g., better outcome, lower attrition, greater satisfaction), in areas important to stakeholders.” Such evidence might require, for example, randomized trials showing better outcomes were achieved when clinicians used a certain measurement tool or method, relative to a control group that did not. Generating this evidence will take time and possibly a paradigm shift toward implementation science and measurement-based care, especially testing empirical questions about whether and how different assessment practices could actually affect clinical outcomes (McLeod et al., 2022).

Other psychometric categories do not require much elaboration (norms, internal consistency, content validity, construct validity, and validity generalization); however, it is worth noting what was not captured. For instance, norms focus on the representativeness of the samples in which they have been used, but do not capture whether or not a measure comes with t-scores or an empirically derived cutoff for clinical significance—features which would greatly enhance the utility of normative data for clinicians and researchers (see Table S4 for this information). Additional measurement characteristics may also be relevant to a measure’s utility, such as cost, length, and available translations, which were also not captured in existing categories rated, but are provided as additional information in the current review. Research on repeatability and prescriptive validity is so new that it was not useful to include in the review, and inter-rater reliability rarely applies beyond correlations between different informants (e.g., teacher with parent) which arguably should be considered as informants that are expected to have low correlations, around r = 0.28 (De Los Reyes et al., 2015). Lastly, discriminative validity, as defined by Youngstrom et al. (2017), is primarily applicable to discriminating a particular diagnosis from non-diagnosis within a given population—not applicable to transdiagnostic approaches that are intended to cut across, rather than identify, specific diagnostic categories.

Although not an EBU dimension itself, the concepts of incremental validity and incremental utility are particularly important here. If irritability, anger, and aggression are indeed related to functioning, perhaps distinctly or in combination with one another, it follows that some future research is needed to focus on the validation and “value-added” from measuring each of these domains. For example, research is needed to examine whether irritability, anger, and aggression distinctly and incrementally predict clinically relevant validity criteria such as psychopathology, treatment response, performance on domain-relevant tasks, and associated features of the domains themselves (e.g., parenting, peer relations, social skills, sleep functioning). To test whether a measure confers some benefit “above and beyond” measurement of other variables is an important approach but rarely used in the articles we reviewed. Such work calls for multiple regression models, machine learning methods, and longitudinal designs to help tease apart what measures are the most robust, incremental predictors of relevant outcomes

We also call for more research on cross-informant associations—not necessarily as evidence for “inter-rater reliability,” but because findings regarding cross-informant patterns are informative in and of themselves. In particular, there is a need for research to quantify the level of agreement on assessing each of these constructs (irritability, anger, and aggression) according to each of several informants (parent, youth, teacher), within- vs. cross-informant, and same- vs. different-construct. Populating this correlation matrix would help elucidate the distinctions among these constructs and the role informants play, and could provide useful in understanding how context impacts displays of the behaviors used to estimate their presence. The present review found that researchers have relied almost exclusively on youth self-report for measuring anger, finding few studies/measures using parent-report and none that used teacher-report. This scenario is subject to mono-informant bias, making it difficult for researchers to understand different perspectives as well as different constructs. Informant discrepancies speak to issues surrounding measurement validity best understood from a multi-informant perspective (De Los Reyes et al., 2023). We call specifically for research on the role of informant perspectives and context in affecting ratings from specific informants and discrepancies among them. Multi-informant measures, particularly with parallel item content, should be prioritized, as they permit more robust tests of these questions.

Implications for Research on Youth Irritability, Anger, and Aggression

Several interesting findings were revealed by virtue of the types and properties of measures that were included. First, the overlap among the scales seems to support the view that our targeted constructs can reasonably represent a coherent spectrum of overt externalizing problems, which include affective and behavioral manifestations. Many, though not all, measures construed as measuring “externalizing behavior” also include many items, if not entire scales, that capture the prevailing emotions (irritability, anger, frustration) associated with externalizing problems. Although some measures and scales reflect the empirical associations between these problems, they also reveal that they are empirically distinct (e.g., MAP-DB: Temper Loss and Aggression scales).

Second, our searches, screening, and coding revealed that the measures of interest here cut across a number of contexts, including outpatient, inpatient, school-based, preschool, research, developmental, intervention, experimental, social, personality, and other types of work. Because these boundaries have led to some confusion and lack of clarity in the literature, we sought to be as inclusive as possible in order to align measures from different contexts side by side in the same synthesis. Terminological differences emerged from this process, warranting careful consideration. For example, aggression and bullying measures can capture the same behaviors, but contemporary definitions of bullying have focused these instruments in ways that go beyond aggression and should be considered separately. At the same time, aggression itself has several different subtypes, which are well-captured in this review (e.g., proactive vs. reactive, e.g., Dodge & Coie, 1987; overt vs. relational, e.g., Crick, 1996), and researchers and clinicians interested in better understanding aggressive behavior are advised to consider whether they are best served by a unidimensional instrument or by more focused subtypes of aggression.

Third, readers may note that specific measures appear to have been left out, notably including measures relevant to diagnostic assessment such as the DBDRS or the KSADS. As noted in the introduction, our transdiagnostic approach precluded us from including measures that were explicitly diagnostic. Yet at the same time, these excluded measures are often relevant to include in a comprehensive assessment of externalizing problems. For more comprehensive coverage of this topic, we refer readers to relevant chapters/books (e.g., Burke et al., 2023; Evans et al., 2024; Lochman & Matthys, 2018; Matthys & Lochman, 2017; Walker et al., 2020) and articles (McKinney & Morse, 2012; McMahon & Frick, 2005; Steiner & Remsing, 2007; Stringaris et al., 2018). However, we would also emphasize the importance of a transdiagnostic perspective, as these problems not only cut across a wide range of diagnostic categories, but they are also not well captured by any single category (e.g., Stepanova et al., 2022). These challenges have led us as a field to “reverse-engineer” provisional solutions to our diagnostic systems, such as a specifier for outbursts in DSM-5-TR (Carlson, Singh, et al., 2022), a subtype of ODD in DSM-5 (Evans et al., 2017), and syndromic targets for research programs (Leibenluft, 2011; Stepanova et al., 2022). Clearly this is both an active area of research and an area in which much work has yet to be done. The present review underscores the promise of a transdiagnostic and transdisciplinary approach in pursuing research on irritability, anger, and aggression.

Strengths, Limitations, and Future Directions

The present review has several strengths offering particular value for those interested in understanding or measuring irritability, anger, or aggression in youth. First, this is one of the most comprehensive reviews available, covering 4,664 unique articles during screening, and considering hundreds more sources through measure database queries, ancillary strategies, and follow-up searches (See Figure 1) and Supplement. This ultimately led us to identify an average of 9.9 relevant sources (8.8 peer-reviewed) per measure upon which to base our psychometric ratings. The stringent inclusion and exclusion criteria allowed us to reliably evaluate each measure and accurately judge whether or not it should be included for review. One of these criteria was that a measure must have at least three published, peer-reviewed articles on its use; dissertations and theses that had not been peer-reviewed were also excluded. The downside of this approach is that it omits newer measures and measures that have not been used as often in research contexts. Further, our screening workflow was carried out in such a way that we are not able to provide a comprehensive list of all the measures that were possibly relevant but were excluded for this or that specific reason (but see Table S3 for exemplar excluded measures and reasons why). However, the strength of this approach is that it allowed us to focus on only measures that had been implemented and studied in research, with sufficient peer-reviewed evidence for evaluation.

An important exclusion of any measure that was directly tied to any DSM diagnostic criteria sets, such (e.g., ODD, CD, or DMDD) as this review aimed to investigate transdiagnostic measures relating to irritability, anger, and aggression. This transdiagnostic approach allows these measures to be applicable across diagnoses because it is not for the purpose of giving a diagnosis. Further, this approach is cross-disciplinary, relevant to psychology, psychiatry, education, and other fields wanting to find an applicable measure for youth in their care. Our systematic review also includes detailed information about the 68 measures included, such as the number of items, age range, constructs of interest, and translations available. This information allows the review to also function as a tool to compare measures and determine which would be most applicable to the setting in question. Thus, this review provides a comprehensive look into the field of actionable measures of irritability, anger, and aggression in youth, where a current gap is apparent. We encourage researchers specializing in different substantive areas (e.g., trauma, ADHD, disruptive behavior problems, mood and anxiety disorders) to approach these measurement questions in a manner that is most appropriate for their population and problem of interest. At the same time, maintaining a dialogue with irritability research and researchers from different areas will likely help advance assessment practices more effectively for the most people.

Some further limitations also should be noted. First, although necessary for a transdiagnostic approach, our exclusion of measures tied to DSM led to the exclusion of measures that might otherwise be relevant clinically, while other exclusion criteria omitted newer measures and measures that have not been widely used in research (see Table S3 for details and examples). Second, the focus on English-speaking populations led to the omission of psychometric research and instruments from other languages and countries, which is important for future research. It would be useful for future work to examine the cross-cultural applicability between English and non-English versions of the measures evaluated in this study. Similarly, this study is not representative of measurement in adults, infants, or toddlers; measures developed for these age groups may be omitted here or have stronger evidence for their properties in those ages than reviewed here. Third, the psychometric evaluation process used in this study (Hunsley & Mash, 2008; Youngstrom et al., 2017) was vital, offering a way to tier the extent of evidence supporting each measure and provided a rubric from which to evaluate existing evidence. However, there are few psychometric studies with a specific focus in categories such as treatment sensitivity and clinical utility (Holly et al., 2019). Therefore, certain measures have lower psychometric strength overall despite having high ratings for properties like test-retest reliability and content validity. Further, the rubric has limited room for nuance in evaluating how good a particular piece of evidence is (e.g., large, randomized trials vs. small correlational studies), instead taking any evidence as sufficient to pass a given threshold. Future EBU projects should revisit how psychometric criteria are defined and operationalized, to promote greater reliability and validity in these ratings within and across review papers.

Interesting differences emerged between free and commercially available measures, such that the latter had better properties on average. This pattern is understandable given that companies have large, funded efforts behind them, with resources to draw from large, nationally representative samples for norms and data. Free measures, on the other hand, are commendable for their accessibility, and in some cases may be more extensively used in research providing a more complete and transparent picture of their psychometric properties. Thus, commercial and noncommercial test developers can learn from one another how to improve the tools they produce.

We have attempted to be as comprehensive as possible, and to define clear criteria to explain what gets omitted and why. Nonetheless, irritability, anger, and aggression measurement is an active and robust area of research, and it is possible that relevant measures were omitted. In particular, we carried out our searches and coding in 2021–2023. More research has been done during this time and will continue to be done in the years ahead. Thus, this EBU review can be viewed as a snapshot of measurement methods and evidence at a given time, underscoring the need for continued methodological development and research synthesis moving forward.

Summary and Conclusions

Despite its limitations, the present paper offers a useful review and toolkit for youth mental health assessment. Research on overt externalizing problems has come a long way over the past few decades, converging on a set of measures with empirical support for assessing irritability, anger, and/or aggressive behavior as transdiagnostic constructs. We identified, reviewed, and evaluated 68 of these measures. The availability of measures with Good to Excellent psychometric ratings is encouraging. At the same time, areas identified as Adequate or Insufficient Evidence highlight potential directions for future research, particularly clinical utility, incremental validity, and longitudinal stability vs. change sensitivity. Other priority areas for future research include advancing multi-informant approaches and understanding; disentangling overlapping constructs for a clearer theoretical conceptualization; clarifying associations with ecologically or clinically meaningful variables; and understanding the cross-cultural variability and applicability of measures. Although there is a place for developing new measures, what is more pressing now is continued evaluation of existing measures, and careful review, consolidation, and elaboration of these measures, including harmonizing and bridging to other informants and populations, as a foundation for future research in youth irritability, anger, and aggression.

Findings from this review are consistent with the impression that progress in this area may have been limited by diagnostic, disciplinary, methodological, and terminological boundaries. Moving forward, researchers are encouraged to be aware of and actively work to dismantle these boundaries, as this could help strengthen assessment methods and the empirical evidence base. When selecting measures, researchers and clinicians are advised to review multiple measures and consider what the evidence says about their strengths and weaknesses, for what purposes, and in what populations and settings. The present paper was written to help inform such decisions. In the tables and supplement, we have summarized key characteristics of these instruments, including number of items, age ranges, sample types, norms, instructions to access the measures, and whether alternative translations are available. There is no shortage of tools for assessing irritability, anger, or aggression in youth. Future research can continue to grow this toolkit—and evaluate the tools within it—to better meet the needs of irritable, angry, and aggressive youths in clinical and research settings.

Editor’s Note

This article is part of a special issue, “The Affective Side of Disruptive Behavior: Toward Better Understanding, Assessment, and Treatment,” published in the Journal of Clinical Child and Adolescent Psychology in 2024. Spencer C. Evans and Jeffrey D. Burke served as editors of the special issue; Andres De Los Reyes served as conflict-of-interest editor.

Preregistration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021283556

Acknowledgments

The authors thank Nour Abduljawad, Patricia Gao, Sophia Ross, and Kate Simmons for their assistance in carrying out this project.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

This is a review paper that has no data associated with it. The supplement provides detailed information (references, links) on all the sources consulted to inform this review.

Supplementary Data

Supplemental material for this article can be accessed online at https://doi.org/10.1080/15374416.2023.2292041.

Additional information

Funding

This project was not specifically associated with any source of funding. SCE received partial support from the AIM Clinical Science Fellowship, NIH Loan Repayment Program (L30MH120708), and University of Miami faculty start-up funds during the preparation of this work.

Notes

¹ For relevant diagnostic guidance on assessment, readers are referred to the following sources (Burke et al., 2023; Evans et al., 2024; Lochman & Matthys, 2018; Matthys & Lochman, 2017; McKinney & Morse, 2012; McMahon & Frick, 2005; Steiner & Remsing, 2007; Stringaris et al., 2018; Walker et al., 2020).

² For the sake of illustration, nearly all specific problems related to anxiety (e.g., separation anxiety disorder, specific phobia, social anxiety disorder, panic disorder, agoraphobia, generalized anxiety disorder) are classified as Anxiety Disorders in DSM-5; and most specific problems related to low mood (e.g., major depressive disorder, major depressive episodes, persistent depressive disorder) are classified as Depressive Disorders in DSM-5. There is no such section for disorders of irritability, anger, or aggression. We lump together “irritability, anger, and aggression” here for reasons outlined in the next section.

³ Some iterations of this definition specify that irritability is an increased proneness to anger compared to peers, a between-person conceptualization. More recently, there has been increasing recognition that irritability can also be viewed as an increased proneness to anger compared to one’s oneself or one’s own baseline (i.e., a within-person, contextual, or longitudinal conceptualization). We adopt a definition that accommodates both views.

⁴ Multi-informant data are limited, and this is not a comprehensive review, but these studies suggest that within-informant correlations vary somewhat across informants (roughly 0.4–0.8 for parent-report, 0.6–0.9 for teacher-report, and 0.2–0.7 for youth-report) and drop considerably when the same construct is rated by different informants (most cross-informant correlations around 0.1–0.5), with no clear construct-specific patterns.

⁵ We define “psychometric results” broadly based on the rubric’s domains (see Table 1). Importantly, this definition did not limit our focus exclusively to assessment papers. For example, any relevant empirical study reporting Cronbach’s alpha could be considered for rating internal consistency, or means/SDs for rating norms. At the same time, however, we did not simply review all empirical studies that used a measure and reported statistics from it. In practice, this approach meant that we leaned toward including all relevant empirical papers for measures that were newer or less frequently used. In contrast, for widely used measures with dozens or hundreds of studies, we were more selective to identify a robust corpus of relevant evidence to inform our evaluations. More on this below.

Table 1. Rubric for evaluating norms, validity, and utility (Hunsley & Mash, 2008; extended by Youngstrom et al., 2017).

Criterion	Adequate	Good	Excellent
Norms	M and SD for total score (and subscores if relevant) from a large, relevant clinical sample	M and SD for total score (and subscores if relevant) from multiple large, relevant samples, at least one clinical and one nonclinical	Same as “good,” but must be from representative sample (i.e., random sampling, or matching to census data)
Internal consistency (Cronbach’s alpha, split half, etc.)	Most evidence shows alpha values of 0.70–0.79, based on median Cronbach’s alpha across relevant scales and samples	Most reported alphas 0.80–0.89, based on median Cronbach’s alpha across relevant scales and samples	Most reported alphas ≥ 0.90, based on median Cronbach’s alpha across relevant scales and samples
Inter-rater reliability^a	Most evidence shows kappas of 0.60–0.74, or intraclass correlations of 0.70–0.79	Most reported kappas of 0.75–0.84, ICCs of 0.80–0.89	Most kappas ≥ 0.85, or ICCs ≥ 0.90
Test – retest reliability (stability)	Most evidence shows test–retest correlations ≥ 0.70 over period of several days or weeks,  ≥ 0.50 over several months, or ≥ 0.30 over a year or longer	Most evidence shows test—retest correlations ≥ 0.70 over period of several months, or ≥ 0.40 over a year or longer	Most evidence shows test–retest correlations ≥ 0.70 over a year or longer
Repeatability^a	Bland–Altman (Bland & Altman, 1986) plots show small bias, and/or weak trends; coefficient of repeatability is tolerable compared to clinical benchmarks (Vaz et al., 2013)	Bland–Altman plots and corresponding regressions show no significant bias, and no significant trends; coefficient of repeatability is tolerable	Bland–Altman plots and corresponding regressions show no significant bias, and no significant trends; established for multiple studies; coefficient of repeatability is small enough that it is not clinically concerning
Content validity	Test developers clearly defined domain and ensured representation of entire set of facets	Same as “adequate,” plus all elements (items, instructions) evaluated by judges (experts or pilot participants)	Same as “good,” plus multiple groups of judges and quantitative ratings
Construct validity (e.g., predictive, concurrent, convergent, and discriminant validity)	Some independently replicated evidence of construct validity	Bulk of independently replicated evidence shows multiple aspects of construct validity	Same as “good,” plus evidence of incremental validity with respect to other clinical data
Discriminative validity^a	Statistically significant discrimination in multiple samples; AUCs < 0.6 under clinically realistic conditions (i.e., not comparing treatment seeking and healthy youth)	AUCs of 0.60 to < 0.75 under clinically realistic conditions	AUCs of 0.75 to 0.90 under clinically realistic conditions
Prescriptive validity^a	Statistically significant accuracy at identifying a diagnosis with a well-specified matching intervention, or statistically significant moderator of treatment	Same as “adequate,” with good kappa for diagnosis, or significant treatment moderation in more than one sample	Same as “good,” with good kappa for diagnosis in more than one sample, or moderate effect size for treatment moderation
Validity generalization	Some evidence supports use with either more than one specific demographic group or in more than one setting	Bulk of evidence supports use with either more than one specific demographic group or in multiple settings	Bulk of evidence supports use with either more than one specific demographic group AND in multiple settings
Treatment sensitivity	Some evidence of sensitivity to change over course of treatment, statistically significant improvement for treated group	Independent replications show evidence of sensitivity to change over course of treatment, requiring at least 2 studies	Same as “good,” plus sensitive to change across different types of treatments, requiring at least 3 studies
Clinical utility	After practical considerations (e.g., costs, respondent burden, ease of administration and scoring, availability of relevant benchmark scores, patient acceptability), assessment data are likely to be clinically actionable, must be intended for clinician use and have evidence from clinical samples	Same as “adequate,” plus published evidence that using the assessment data confers clinical benefit (e.g., better outcome, lower attrition, greater satisfaction), in areas important to stakeholders	Same as “good,“plus independent replication

Note: Table from De Los Reyes and Langer (2018); reprinted with permission. ^a Inter-rater reliability, repeatability, discriminative, and prescriptive validity were not evaluated in this manuscript for reasons noted in text. ICC = intraclass correlation coefficient; AUC = area under the curve. Italics = our elaborations made to existing criteria.

⁶ Aggression refers to any behavior enacted with an intent to harm another person who does not want to be harmed (Baron & Richardson, 1994). Bullying is defined by three criteria: (a) the intentionality criterion from aggression, plus (b) repetitiveness of the aggressive behaviors over time, and (c) a power imbalance favoring the perpetrator over victim (Olweus, 2013). Given the conceptual overlap, we included “bullying” in our search strategies—to capture tools that measured youth aggression broadly—but we explicitly excluded scales that were defined by the full definition of bullying, as that was beyond the focus of the review.

⁷ The following criteria were omitted: (a) interrater reliability, because only multi-informant data could be construed as such; this was rarely reported, and we view different informants as different perspectives, not meant to show high levels of agreement (De Los Reyes et al., 2015, 2023); (b) repeatability, because evaluation of this essentially required Bland-Altman plots and/or a coefficient of repeatability, which was virtually never reported; (c) discriminative validity, because this usually focused on discriminating among those with vs. without a diagnosis, but our transdiagnostic approach excludes diagnoses, leaving little basis for discriminative validity; and similarly (d) prescriptive validity, involves identifying a diagnosis with a well-specified matching intervention or a significant treatment moderator, and this seemed rare and not an appropriate fit for our transdiagnostic approach.

⁸ This was the case for all measures and studies included in this review. Thus, any “adult/youth measures” were evaluated solely for their performance among children and adolescents, defined as non-undergraduate samples with a mean age between 3.0 years and 18.9 years. Evidence from adult samples was not considered.