Introduction to the Special Issue. A Dozen Years of Demonstrating That Informant Discrepancies are More Than Measurement Error: Toward Guidelines for Integrating Data from Multi-Informant Assessments of Youth Mental Health

ABSTRACT

Validly characterizing youth mental health phenomena requires evidence-based approaches to assessment. An evidence-based assessment cannot rely on a “gold standard” instrument but rather, batteries of instruments. These batteries include multiple modalities of instrumentation (e.g., surveys, interviews, performance-based tasks, physiological readings, structured clinical observations). Among these instruments are those that require soliciting reports from multiple informants: People who provide psychometrically sound data about youth mental health (e.g., parents, teachers, youth themselves). The January 2011 issue of the Journal of Clinical Child and Adolescent Psychology (JCCAP) included a Special Section devoted to the most common outcome of multi-informant assessments of youth mental health, namely discrepancies across informants’ reports (i.e., informant discrepancies). The 2011 JCCAP Special Section revolved around a critical question: Might informant discrepancies contain data relevant to understanding youth mental health (i.e., domain-relevant information)? This Special Issue is a “sequel” to the 2011 Special Section. Since 2011, an accumulating body of work indicates that informant discrepancies often contain domain-relevant information. Ultimately, we designed this Special Issue to lay the conceptual, methodological, and empirical foundations of guidelines for integrating multi-informant data when informant discrepancies contain domain-relevant information. In this introduction to the Special Issue, we briefly review the last 12 years of research and theory on informant discrepancies. This review highlights limitations inherent to the most commonly used strategies for integrating multi-informant data in youth mental health. We also describe contributions to the Special Issue, including articles about informant discrepancies that traverse multiple content areas (e.g., autism, implementation science, measurement validation, suicide).

1985: One of the Guest Editors of this Special Issue (CCE) was a graduate student trainee and met with her very first client. Like any good graduate student, even back then, she obtained parent-report, teacher-report, child self-report, and did behavioral observations in the “playroom.” She triple-checked her hand-scored informants’ measures (as no computer programs for scoring were available then), plotted their scores on the respective informant’s profile sheet, listened to her cassette audiotape of her “behavioral observations” (as no video or digital recording was available then), and then attended her first supervision meeting. Her supervisor asked, “So what do we have”? She basically said, “I have no clue, the mother says the kid is very depressed, and nothing else; the teacher says the kid has significant aggressive behavior, and nothing else; the kid does not report a single symptom of anything; and in playing games with him it seemed to me that he was incredibly anxious about everything.” She remembers a lot about that first client. She remembers having to hospitalize the mother for high suicide risk the next time she met with the mother and boy. She remembers that the assessment “data” inspired her master’s thesis paper, master’s thesis research, and subsequent dissertation (and that the 1987 meta-analysis, Achenbach et al., 1987, was published just 2 years after her first client). She does not recall whether or not that discrepant assessment information was used in the treatment plan for that client (likely it was not). She does know that in her 29 years of supervising doctoral students in clinical practicum: 1) she always spent the first class covering suicide risk and management (“as clients are like a box of chocolates, you never know what you’ll get”); and 2) not once were informant discrepancies per se involved in treatment planning or case conceptualization (but they were almost always there!).

2006–2007: Roughly 20 years after the first story, one of the Guest Editors of this Special Issue (ADLR) was a graduate student trainee. He was making his way through interviews for placement in a psychology internship; a key requirement of doctoral training in clinical psychology. At an internship site that will remain nameless, during an interview with a faculty member who will remain nameless, the Guest Editor was asked to describe his research interests. His research interests were then as they are now: to understand why informants like those described in the first story often make discrepant reports when asked to rate the mental health of a child or adolescent (i.e., collectively referred to as youth unless otherwise specified). The faculty member’s response: “That is an area of research I am actively ignoring.” Incidentally, the faculty member’s expertise was in assessing and treating youth at risk for suicide.

Validly characterizing youth mental health phenomena requires evidence-based approaches to assessment (Hunsley & Mash, 2018). An evidence-based assessment cannot rely on scores taken from a “gold standard” instrument. Rather, evidence-based assessments produce multiple scores; the byproducts of administering batteries of instruments. These batteries include multiple modalities of instrumentation (e.g., surveys, interviews, performance-based tasks, physiological readings, structured clinical observations). Among these instruments are those that require soliciting reports from multiple informants: Untrained raters who provide psychometrically sound data about youth mental health (e.g., parents, teachers, youth themselves).

Each of the stories at the opening of this article highlight a ubiquitous observation when taking a multi-informant approach to assessing youth mental health: the presence of discrepant estimates or measurement outcomes regarding a given youth’s mental health, depending on the informants who provide reports about that youth and their psychological functioning (i.e., informant discrepancies). In fact, these stories are part of a long history of research and theory on informant discrepancies in this area of work. The earliest empirical observations of informant discrepancies trace back decades before the “1985 story” (e.g., Lapouse & Monk, 1958). Soon after this story, Achenbach et al. (1987) published the seminal meta-analysis that first documented the robust nature of these discrepancies, based on the findings of over 100 studies. As part of that meta-analysis, Achenbach and colleagues advanced a notion they referred to as situational specificity: The informant discrepancies they observed in their meta-analytic findings reflected the idea that (a) informants like parents and teachers vary in terms of where they observe youth (i.e., home vs. school) and (b) youth vary in terms of where they display behaviors indicative of the mental health domains about which informants make reports. Stated another way, informant discrepancies reflect “real stuff”―data germane to understanding the very youth mental health domains about which informants provide reports (i.e., domain-relevant information). Yet, much of the history between the two stories (i.e., work published between 1985 and 2007) largely consisted of research and theory focused not on situational specificity, but rather on explaining away the informant discrepancies observed in youth mental health assessments purely as a function of measurement confounds, such as random error or rater biases (e.g., De Los Reyes, 2011; Richters, 1992). Perhaps because of this history surrounding measurement confounds, many scholars―including the faculty member in the “2006–2007 story”―did not feel a need to probe whether these informant discrepancies ought to be understood as something more than what McGuire (1969) referred to as research artifacts. By “research artifacts,” we mean errors or flaws inherent to the process of estimating the mental health needs of youth clients.

This article serves as the introduction to a Special Issue on informant discrepancies in youth mental health assessments. In fact, this Special Issue serves as a “sequel” to a Special Section on this same topic, published in Issue 1 of the 2011 volume of this very journal―the Journal of Clinical Child and Adolescent Psychology (JCCAP; De Los Reyes, 2011). In the 12 years since publication of this Special Section in JCCAP, we have seen a flourishing of research and theory on informant discrepancies as they manifest in youth mental health assessments, and as we explain below, this work indicates that informant discrepancies often contain domain-relevant information (for reviews, see De Los Reyes et al., 2013, 2015, 2020, 2022).

However, we must be cognizant of this key reality of youth mental health: In this area of work, the ubiquitous presence of informant discrepancies traverses research and service delivery settings, mental health domains, the instruments used to assess mental health domains, the developmental periods of youth undergoing evaluation, and the types of clinical decisions informed by the outcomes of mental health assessments such as diagnosis, treatment planning, and treatment response (e.g., Al Ghriwati et al., 2018; Augenstein et al., 2022; Becker-Haimes et al., 2018; Bonadio et al., 2022; De Los Reyes, Cook, et al., 2019; De Los Reyes, Lerner, et al., 2019; De Los Reyes, Ohannessian et al., 2019; Humphreys et al., 2017; Lerner et al., 2017; Makol et al., 2020; Nichols & Tanner-Smith, 2022; Trang & Yates, 2020; Watts et al., 2022; Zilcha-Mano et al., 2021). If these informant discrepancies affect all the work that the community of youth mental health scholars produces, then addressing the issues these discrepancies raise cannot be self-contained into a set of research interests. We cannot expect the task of addressing these issues to fall on a few scholars who declare informant discrepancies as their primary area of expertise. The task of advancing knowledge about informant discrepancies falls on all scholars in this area. Consequently, we see a need for increased engagement among scholars in youth mental health who invariably encounter informant discrepancies in their work.

The central thesis of the current Special Issue is this: The informant discrepancies observed in youth mental health assessments cannot be ignored―they must be engaged. As we describe in this article, the consequences of failing to engage these informant discrepancies and understand their inner workings extend well beyond the findings of empirical studies. They affect the lives of the youth clients we serve, and across all the settings where we use mental health assessments to make high-stakes decisions about the services these clients receive.

We see a great deal of remaining work on informant discrepancies in youth mental health assessments. In particular, we must address a key barrier to research on assessment and in tandem, evidence-based service delivery. The “1985 story” illustrates this barrier, namely the lack of guidelines for conducting multi-informant assessments, interpreting their outcomes, and integrating data derived from these assessments (see also Beidas et al., 2015; De Los Reyes, Tyrell, et al., 2022). Along these lines, in this introductory article we address a series of aims. First, we briefly describe research documenting the presence of informant discrepancies in assessments of all known mental health domains. Second, we summarize research on the degree to which informant discrepancies contain domain-relevant information, focusing on work produced in the intervening years between the 2011 JCCAP Special Section and this current Special Issue. Third, we discuss a key driver of this current Special Issue: The most widely used approaches to integrating multi-informant data assume that informant discrepancies cannot contain domain-relevant information. Fourth, we highlight the benefits of developing and testing approaches to integrating multi-informant data that capitalize on the domain-relevant information in informant discrepancies, by describing conceptual models designed to guide work on these very benefits. Fifth, we briefly describe the contributions to this Special Issue, and we close with a renewed call to action to probe informant discrepancies in youth mental health.

Informant Discrepancies Define the Outcomes of Youth Mental Health Assessments

We previously noted that the body of work on informant discrepancies in youth mental health assessments numbers in the hundreds of studies. As in other areas that have accumulated large bodies of evidence (e.g., controlled trials of intervention efficacy; see Smith & Glass, 1977; Weisz et al., 2017), several meta-analyses have sought to characterize the nature and extent of the informant discrepancies that occur when assessing youth mental health. We previously cited the first of these meta-analyses (Achenbach et al., 1987), which examined 119 studies published between 1960 and 1986, and estimated that, on average, informants’ reports of youth mental health domains (i.e., internalizing and externalizing concerns) correspond at low-to-moderate magnitudes (i.e., mean r = .28; Cohen, 1988). Remarkably, in the years that followed this meta-analysis, very little changed with regard to the magnitudes of these effects. In fact, a meta-analysis conducted 30 years later―and based on examinations of 341 studies published after 1987 (i.e., between 1989 and 2014)―detected the exact same mean correspondence estimate observed by Achenbach and colleagues (i.e., r = .28; De Los Reyes et al., 2015).

The stability of the effects reported by both Achenbach et al. (1987) and De Los Reyes et al. (2015) is further corroborated by several other elements of the meta-analytic literature on informant discrepancies. First, this stability is not simply a function of all of these studies being conducted on homogenous samples (i.e., WEIRD effects; see Henrich et al., 2010). Indeed, recent meta-analytic work finds remarkably stable levels of correspondence (i.e., mean r = .28, with a 95% confidence interval of r = .24–.31) across hundreds of studies conducted in over 30 countries and spread across all inhabited continents (De Los Reyes, Lerner, et al., 2019). Second, this stability is not simply a function of all of these studies focused on the same youth mental health domains. Achenbach and colleagues and De Los Reyes and colleagues meta-analyzed cross-informant correspondence data related to multi-informant assessments of concerns on the internalizing and externalizing spectra. Beyond these two quantitative reviews, meta-analyses have revealed low-to-moderate magnitudes of correspondence when assessing (a) the mental health of youth with histories of maltreatment (Romano et al., 2018), (b) concerns on the autism spectrum (Stratis & Lecavalier, 2015), (c) associated features of mental health concerns (e.g., parenting; Hou et al., 2020; Korelitz & Garber, 2016), and (d) assessments of psychosocial strengths (e.g., social competence; Renk & Phares, 2004). Third, these informant discrepancies are not exclusive to understanding them as they manifest in a particular content area of youth mental health. In fact, meta-analyses find that these informant discrepancies occur when interpreting meta-analytic estimates of the effects from controlled trials (Casey & Berman, 1985; De Los Reyes & Kazdin, 2005, 2006, 2008; Weisz et al., 1987, 1995, 2006, 2017), and when estimating links between youth mental health concerns and domains thought to contribute to the development and maintenance of these concerns (e.g., parenting, peer victimization, parental depression; Hawker & Boulton, 2000; Hou et al., 2020; Ivanova et al., 2022). Across all of these meta-analyses, we find that informant discrepancies occur regardless of the study design in which assessments are administered: from highly controlled laboratory settings to uncontrolled field settings (e.g., community samples, school-based services, community mental health clinics). Taken together, decades of research make clear that informant discrepancies define assessments of all known youth mental health domains.

When Informant Discrepancies Reflect Domain-Relevant Information

One of the key contemporary issues in mental health research, as well as in psychological research generally, has to do with the replicability and reproducibility of research findings (see Open Science Collaboration, 2015; Tackett et al., 2017). The logic goes: How might we arrive at conclusions about or draw principles from research findings if (a) independent research teams investigating the same phenomenon using the same procedures do not arrive at similar conclusions (i.e., replication; Schmidt, 2009) and/or (b) similar conclusions do not manifest when testing a hypothesis―within the same study or dataset―using varied procedures for hypothesis testing such as different analytic procedures, outcome measures, or measurement sources (i.e., reproduction; Schekman, 2016)? In this respect, informant discrepancies are an outlier in the broader literature on replication and reproduction. Perhaps ironically, the most replicable finding in youth mental health is that multiple informants’ reports of the same youth fail to reproduce the same estimates of that youth’s mental health.

Beyond the replicability of the informant discrepancies phenomenon, we see an interesting parallel between how researchers have interpreted informant discrepancies and what they reflect, and how researchers have interpreted low levels of replicability and reproducibility in psychological research findings. That is, in both cases interpretations have largely focused on research artifacts (i.e., McGuire, 1969). For example, researchers have attributed the grand majority of low rates of replication and/or reproduction in psychological research to the culture of research practices in Psychology, namely (a) a focus on the novelty of research findings, (b) an emphasis on reporting significant research findings, and thus (c) a reinforcement of research practices that increase the likelihood of observing “a” and “b” (e.g., p-hacking, selective reporting of findings, selective use of analytic procedures; see Boutron et al., 2010; Open Science Collaboration, 2015; Silberzahn & Uhlmann, 2015). In essence, these interpretations frame the lack of replication and/or reproducibility as factors that are irrelevant to understanding psychological phenomena. As such, this interpretation has had a large influence on recommendations to address replication and reproduction in Psychology, namely in changing the culture of research practices and decreasing researcher behaviors geared toward novelty and emphasizing reports of significant findings (for a review, see Nosek et al., 2012).

In ways that are analogous to research on replication and reproduction in Psychology, we previously mentioned that, for much of the history of informant discrepancies research in youth mental health, these discrepancies have been attributed to measurement confounds such as random error and/or rater biases (De Los Reyes, 2011; Richters, 1992). Crucially, this attribution rests on the weakest of empirical evidence (for a recent review, see De Los Reyes & Makol, 2022). Nevertheless, large swaths of researchers who have sought to understand and interpret informant discrepancies in youth mental health assessments have grounded their work on this very idea―that informant discrepancies reflect measurement confounds and nothing more (cf. the faculty member in the “2006-2007 story,” and for a recent review, see De Los Reyes, Tyrell, et al., 2022).

Earlier in this paper we also highlighted Achenbach et al. (1987), namely their notion of situational specificity. What we have yet to mention is that, unlike the “measurement confounds interpretation” of informant discrepancies, a considerable body of evidence has rapidly accumulated over the last 15 years, supporting the idea that informant discrepancies often reflect the kinds of domain-relevant information indicated by situational specificity. We graphically depict this evidence in Figure 1. The studies cited in Figure 1 have been reviewed extensively in recent work (e.g., De Los Reyes, Lerner, et al., 2019; De Los Reyes, Ohannessian et al., 2019; De Los Reyes, Talbott, et al., 2022; De Los Reyes, Tyrell, et al. 2022).

Figure 1. Graphical depiction of studies supporting links between informant discrepancies and domain-relevant criterion variables.

Note: Citations supporting links described in this figure: Al Ghriwati et al. (2018), Augenstein et al. (2022), Becker-Haimes et al. (2018), Bonadio et al. (2022), Borelli et al. (2016), Charamut et al. (2022), De Los Reyes et al. (2009), De Los Reyes, Lerner, et al., 2013, De Los Reyes, Salas, et al., 2013, De Los Reyes, Alfano, et al., 2016, De Los Reyes, Ohannessian et al., 2016, De Los Reyes, Cook, et al., 2022, Follet et al. (2022), Human et al. (2016), Humphreys et al. (2017), Laird and De Los Reyes (2013), Lerner et al. (2017), Leung et al. (2016), Luo et al. (2020), Makol and Polo (2018), Makol et al. (2019), Makol et al. (2020), Makol et al. (2021), Nelemans et al. (2016), Nichols and Tanner-Smith (2022), Ohannessian and De Los Reyes (2014), Ohannessian et al. (2016), Okuno et al. (2022), Trang and Yates (2020), Van Heel et al. (2019), Van Petegem et al. (2020), and Zilcha-Mano et al. (2021).

As we depict in Figure 1, several characteristics of this evidence enhance the interpretability of the findings. In particular, studies have demonstrated links between informant discrepancies and criterion variables that traverse multiple modalities, ruling out the possibility that shared method variance explains the findings (see Garb, 2003). Further, these links exist for informant discrepancies observed with assessments across multiple domains relevant to understanding youth mental health, including symptoms (e.g., internalizing, externalizing, autism spectrum), areas of impairment or life interference, parenting, and family relationships.

Here, we highlight studies published in the years following the 2011 JCCAP Special Section. For instance, Lerner et al. (2017) studied an intake clinical assessment of 283 diagnosed youth receiving outpatient services for autism spectrum concerns. In this study, the authors leveraged person-centered models of data analysis (see Bartholomew et al., 2002) to characterize patterns of parent and teacher reports of youth autism spectrum concerns and identified parent-teacher dyads who agreed in their reports of such concerns as well as disagreed (i.e., parent > teacher; teacher > parent). These patterns of parent-teacher discrepancies demonstrated links to independent assessments of clinical severity as indexed by diagnostic observation tasks (i.e., the Autism Diagnostic Observation Schedule; Lord et al., 2012), such that greater agreement between parent and teacher reports predicted greater levels of clinical severity (as indexed by receiving medication, special education, and an ASD diagnosis). Thus, this study highlighted the ability of variations in informant discrepancies to reflect domain-relevant variations in clinical phenomena (i.e., clinical severity) that inform our understanding of the current functioning of youth undergoing evaluation. Similar findings for various combinations of informants (e.g., parent-teacher, parent-youth, mother-father) have been made for informant discrepancies in assessments of such domains as depression (Makol & Polo, 2018), family relationships (Borelli et al., 2016), psychosocial impairments (De Los Reyes, Cook, et al., 2022), social anxiety (Deros et al., 2018), and broadband internalizing/externalizing concerns (De Los Reyes, Alfano, et al., 2016).

In the years following the 2011 JCCAP Special Section, we also learned about instances in which informant discrepancies can be used as tools for not only characterizing functioning concurrently as was demonstrated by Lerner et al. (2017), but also for predicting youth outcomes longitudinally. Similar to Lerner and colleagues, Makol et al. (2019) leveraged person-centered models of data analysis to characterize parent and youth reports of youth internalizing symptoms assessed at the intake assessments of 765 youth receiving psychiatric inpatient care. In this study, not only did the authors observe patterns of agreement and discrepancies between parent and youth reports, but also these patterns predicted clinically relevant characteristics of the care delivered to these youth. For instance, when parents reported greater internalizing concerns than youth self-reported, youth were particularly likely to exhibit suicidality when admitted, and to be administered intensive treatment services during their inpatient stay (e.g., locked-door seclusion, standing antipsychotics). Patterns of agreement were also predictive, such that instances in which parents and youth agreed on high levels of internalizing concerns was a marker of a relatively longer hospital stay. The predictive utility of discrepancies between parent and youth reports has been demonstrated with a range of predictors and outcomes, including (a) intake assessments of youth anxiety symptoms predicting treatment response (Becker-Haimes et al., 2018), (b) baseline assessments of parenting practices predicting internalizing and externalizing symptoms at 12-month follow-up among youth at risk for substance use relapse (Nichols & Tanner-Smith, 2022), and (c) baseline assessments of depressive symptoms predicting youth suicidal thoughts at three-month follow-up (Augenstein et al., 2022). In sum, since the 2011 JCCAP Special Section on informant discrepancies in youth mental health assessments, researchers have identified a variety of instances in which informant discrepancies reflect domain-relevant information.

When Procedures for Integrating Multi-Informant Data Make Measurements Less Valid

In considering the findings depicted in Figure 1, we must make one thing clear: Informant discrepancies reflect unique variance. Specifically, among a set of informants’ reports about a youth’s mental health, this is unshared data or data that are particular to each informant’s report. Informant discrepancies appear to reflect the grand majority of variance in youth mental health assessments (see also De Los Reyes et al., 2015). As such, learning that at least a portion of these discrepancies contains domain-relevant information should be welcome news for scholars in youth mental health. Yet, we must make another thing clear: Our infrastructure for understanding, scoring, and modeling multi-informant data was not built to capitalize on domain-relevant unique variance. In fact, the dominant procedures for integrating, modeling, or otherwise interpreting multi-informant data and the informant discrepancies they often contain are simply ill-equipped to handle these discrepancies in any substantive sense. In Table 1, we list these procedures, and highlight the issues raised with using them, particularly when informant discrepancies contain domain-relevant information.

Table 1. Limitations of commonly implemented analytic procedures for integrating or modeling multi-informant data in youth mental health.

Procedure	Description	Key Usage Assumption	Limitation	Representative Citations Supporting Limitation
Difference scores	Measured (or latent) index of the “space” between two informant’s reports of the same rated domain (e.g., anxiety, parenting)	The “space” between reports represents incrementally valuable information (e.g., index of informant discrepancies), over-and-above the component scores used to create it (i.e., the two informants’ reports)	Procedure does not provide users with the opportunity to test its key usage assumption; difference scores may not provide any incrementally valuable data, over-and-above the component scores used to create them	Edwards (1994), and Laird (2020)
Combinational algorithms	Used to integrate discrete data, based on whether two or more informants endorse presence of domain (AND rule) or at least one endorses its presence (OR rule)	It is unnecessary to consider the “source” of endorsement or the specific informants who made the endorsement(s)	Combined scores cannot detect context-specific associated features (e.g., parenting; school grades) that individual informant’s reports can detect; depresses measurement validity	Offord et al. (1996) and Rubio-Stipec et al. (2003)
Composite scores	Measure of central tendency (e.g., average of standardized scores) among multiple informants’ reports, typically on parallel instruments	When integrating multi-informant data, common variance contains the only domain-relevant variance	If applied to data conditions that contain domain-relevant informant discrepancies, composite score will be unable to incorporate them; depresses measurement validity	Makol et al. (2020)
Multi-rater structural models	Latent modeling procedures that estimate sources of common and unique variance among multiple informants’ reports	When modeling multi-informant data, common variance contains the only domain-relevant variance	Cannot distinguish domain-relevant informant discrepancies from those that reflect measurement confounds	Watts et al. (2022)
Measurement invariance	Means for refining item content of multi-informant instruments based, in part, on the outcomes of tests focused on detecting items for which informants make variant responses (i.e., tests of differential item functioning)	When constructing instruments for collecting multi-informant data, the only items worth retaining are those that function similarly across informants’ item responses	Does not provide you with the opportunity to test its key usage assumption; differential item functioning may reflect domain-relevant information; depresses measurement validity	De Los Reyes, Tyrell, et al. (2022)

To start, it is important to keep in mind that what all of the procedures listed in Table 1 have in common is they were developed during an era of conceptual and methodological paradigms that emphasized common variance: Among a set of informants’ reports about a youth’s mental health, this is shared data across informants’ reports. This is true of Converging Operations (Garner et al., 1956)―the conceptual paradigm that has dominated thinking for the last half-century on how to interpret patterns of research findings. Converging Operations holds that the degree to which multiple findings derived from tests of the same hypothesis point to the same conclusion should be seen as synonymous with the degree of empirical support for that hypothesis. For instance, much of the evidence supporting the efficacy of interventions tested within randomized controlled trials consists of outcome measures completed by multiple informants (see Weisz et al., 2005). Researchers use these outcome measures to gauge relative differences between the intervention examined in a trial and comparison conditions (e.g., wait-list control, alternative treatment), usually under an “umbrella hypothesis” that the intervention significantly improves functioning (e.g., reductions in symptoms of a specific mental health condition), relative to the improvements observed for the comparison condition (see De Los Reyes & Kazdin, 2006). Converging Operations holds that one should gauge an intervention’s efficacy based on the degree to which the multiple informants’ reports used as outcome measures in the study point to the same conclusion (e.g., intervention outperforms control condition). Assumptions underlying use of Converging Operations also hold true for the similarly dominant measurement validation paradigm for the last half-century―the Multi-Trait Multi-Method Matrix (MTMM; Campbell & Fiske, 1959)―which drew inspiration from Converging Operations (see Grace, 2001; Highhouse, 2009). With the MTMM, users assume that one should gauge measurement validity based on the degree to which a measure assessing a given domain (e.g., self-report of anxiety) yields estimates that converge with other measures of “fairly similar concepts and plausibly independent methods” (Fiske & Campbell, 1992, p. 393).

Together, both the Converging Operations and MTMM paradigms inspired users of the analytic procedures listed in Table 1 to (a) prioritize common variance and treat unique variance components like informant discrepancies as measurement confounds and/or (b) make assumptions about informant discrepancies that cannot be empirically verified within the procedures themselves. De Los Reyes, Wang et al. (2023) review these procedures extensively, discuss their limitations with regard to domain-relevant informant discrepancies, and cite many examples of how they are currently implemented in youth mental health research.

In highlighting issues raised with applying the analytic procedures listed in Table 1, we must make yet another thing clear: Highlighting the issues we raise is not synonymous with labeling an analytic procedure as “bad” or a procedure that researchers should avoid using. Indeed, no one analytic procedure is inherently “right” or “wrong” (see also Levy, 1969). Rather, a user might be “wrong” in applying the procedure to data conditions that do not “fit” the assumptions underlying its use. In the case of the procedures listed in Table 1, their use within data conditions that contain domain-relevant informant discrepancies has the logical consequence of depressing measurement validity. Indeed, when a user leverages a procedure like composite scoring or structural models focused on estimating common variance, they must also assume that common variance is the only source of domain-relevant variance. If the data conditions indicate otherwise―that domain-relevant informant discrepancies exist―these procedures leave all that domain-relevant variance on the “cutting room floor.” If a great deal of variance is left out of the equation, then what other conclusion can one come to other than that these procedures―when applied to data conditions that contain domain-relevant informant discrepancies―make measurements less valid? In essence, two sources of valid variance (i.e., both common variance and domain-relevant informant discrepancies) are better than one (e.g., only common variance).

When Informant Discrepancies Boost Measurement Validity

Taken together, scholars in youth mental health currently lack an infrastructure for understanding, scoring, and integrating multi-informant data when those data contain domain-relevant informant discrepancies. Yet, we see hope on the horizon for building this infrastructure. In fact, another development since the 2011 JCCAP Special Section is that we now have a conceptual foundation for such an infrastructure, in the form of a framework―the Operations Triad Model (De Los Reyes, Thomas, et al., 2013)―for conceptualizing the patterns of data observed within multi-informant assessments. We graphically depict the Operations Triad Model in Figure 2. A key feature of the Operations Triad Model, unlike Converging Operations and the analytic procedures listed in Table 1, is that it acknowledges that domain-relevant variance can come from instances in which reports from multiple informants point to the same conclusion (i.e., Converging Operations; Figure 2a), but also from instances in which informants’ reports point to different conclusions. However, within the Operations Triad Model not all of these informant discrepancies are created equal. That is, the Operations Triad Model distinguishes between the informant discrepancies that reflect domain-relevant information (i.e., Diverging Operations; Figure 2b) from those discrepancies that reflect measurement confounds (i.e., Compensating Operations; Figure 2c).

Figure 2. The Operations Triad Model (Reproduced from De Los Reyes, Thomas, et al., 2013).

Note: The top half (A) represents Converging Operations: a set of measurement conditions for interpreting patterns of findings based on the consistency within which findings yield similar conclusions. The bottom half denotes two circumstances within which researchers identify discrepancies across empirical findings derived from multiple informants’ reports and thus discrepancies in the research conclusions drawn from these reports. On the left (B) is a graphical representation of Diverging Operations: a set of measurement conditions for interpreting patterns of inconsistent findings based on hypotheses about variations in the behavior(s) assessed. The solid lines linking informants’ reports, empirical findings derived from these reports, and conclusions based on empirical findings denote the systematic relations among these three study components. Further, the presence of dual arrowheads in the figure representing Diverging Operations conveys the idea that one ties meaning to the discrepancies among empirical findings and research conclusions and thus how one interprets informants’ reports to vary as a function of variation in the behaviors being assessed. Lastly, on the right (C) is a graphical representation of Compensating Operations: a set of measurement conditions for interpreting patterns of inconsistent findings based on methodological features of the study’s measures or informants. The dashed lines denote the lack of systematic relations among informants’ reports, empirical findings, and research conclusions. Originally published in De Los Reyes, Thomas et al. (2013). © Annual Review of Clinical Psychology. Copyright 2012 Annual Reviews. All rights reserved. The Annual Reviews logo, and other Annual Reviews products referenced herein are either registered trademarks or trademarks of Annual Reviews. All other marks are the property of their respective owner and/or licensor.

The last decade of work demonstrating that informant discrepancies often contain domain-relevant information has largely been informed by the Operations Triad Model (see also De Los Reyes & Ohannessian, 2016). Behind the scenes of the findings depicted in Figure 1 are several innovations in measurement, study design, and analytic procedures; each of which highlight the potential for translating research about informant discrepancies into tangible improvements to measurement validity. We highlight one line of work here to illustrate these improvements.

The Operations Triad Model in Action: Assessments of Adolescent Social Anxiety

Mental health services for adolescent social anxiety are often informed by reports collected from adolescent clients as well as the stakeholder who often initiates services on their behalf, typically a parent or caregiver (De Los Reyes & Makol, 2019; Hunsley & Lee, 2014). When relying on just these two informants, not only do discrepancies commonly manifest between reports, but also these discrepancies have historically been interpreted as measurement confounds. Specifically, as adolescents who experience social anxiety often also avoid situations where they may be perceived in a negative light, the idea was that these discrepancies could be explained by adolescents downplaying their concerns when completing self-reports (e.g., social desirability bias; see De Los Reyes et al., 2012, 2015). Yet, here too situational specificity may play a key role in explaining informant discrepancies, particularly when one considers two key elements of adolescents and their social anxiety concerns. First, adolescents, relative to younger children, often spend considerably greater time outside of the home context (Smetana, 2008); places where they are quite likely to encounter the key kinds of social interactions germane to social anxiety (i.e., interactions with unfamiliar people; see Alfano & Beidel, 2011). Second, one crucial set of social interactions germane to adolescents receiving care involves interacting with same-age unfamiliar peers (see Cannon et al., 2020; Hofmann et al., 1999). Taken together, these issues beg the question: Might discrepancies between parent and adolescent reports of adolescent social anxiety occur, in part, because adolescents often display social anxiety concerns within clinically relevant contexts outside of the home?

Addressing this question required a few innovations in measurement and study design. To start, researchers required an additional informant who observes adolescents within interactions with unfamiliar peers, in an effort to compare and contrast adolescents’ and parents’ reports with those of an observer outside of the home. Researchers also required a context to gather such reports that could facilitate acquiring standardized data about these interactions, as well as data from additional, trained observers to reduce the impact of shared method variance on the interpretability of study findings. These considerations resulted in the development of the Unfamiliar Peer Paradigm (Cannon et al., 2020), a set of simulated social interactions in which adolescents interact with youthful-looking research personnel trained to “stand in” as same-age unfamiliar peers (i.e., peer confederates). Following these interactions, peer confederates complete parallel versions of the same sets of social anxiety surveys completed by parents and adolescents. Because peer confederates make these reports naive to assessment training, they essentially serve as a “proxy informant” used to estimate how same-age unfamiliar peers might perceive adolescents’ social anxiety.

The innovations in measurement and study design built into the Unfamiliar Peer Paradigm were not only informed by the Operations Triad Model, but they also resulted in several advancements in our understanding about how to improve measurement validity in multi-informant assessments of adolescent social anxiety. Consistent with situational specificity―and by extension Diverging Operations (Figure 2b)―adolescents’ self-reports displayed distinct patterns of correlations with parents’ and peer confederates’ reports (Makol et al., 2020). Further, these three informants’ reports varied in their relations to trained independent observers’ assessments of adolescents’ reactions to the tasks within the Unfamiliar Peer Paradigm (Glenn et al., 2019). Moreover, all three of these informants’ reports demonstrated psychometric soundness as evidenced by their links to (a) established survey measures of adolescent social anxiety and (b) their ability to distinguish adolescents on a clinically relevant index, namely referral status (Deros et al., 2018).

All of these findings led to a discovery: Might these reports, collectively, contain not only domain-relevant common variance but also domain-relevant unique variance as reflected in the discrepancies among the reports? If so, might improvements to measurement validity come not only from using all three of these informants’ reports to assess adolescent social anxiety, but also from identifying ways to integrate these multi-informant data with methods that retain both common variance and domain-relevant unique variance? Using just such a method developed by Kraemer et al. (2003), Makol et al. (2020) successfully integrated adolescent, parent, and peer confederate reports. Further, in this study scores taken from the Kraemer et al. (2003) method outperformed the composite scoring procedures described in Table 1 when predicting trained independent observers’ ratings of adolescent social anxiety. This last finding has since informed additional innovations in measurement, namely in tests of the psychometric soundness of reports taken from unfamiliar untrained observers―informants who based their reports on videos of adolescents’ interactions within the Unfamiliar Peer Paradigm (see Charamut et al., 2022; Follet et al., 2022; Okuno et al., 2022; Rezeppa et al., 2021). In sum, this line of work illustrates the potential for informant discrepancies research informed by the Operations Triad Model to result in improvements to measurement validity.

When Informant Discrepancies Optimize Clinical Decision-Making

Throughout this introductory article to the Special Issue, we have focused our attention on informant discrepancies as they manifest in research settings. However, we must return to a key reality of youth mental health assessments: These informant discrepancies are everywhere. As indicated by the “1985 story” that opened this introductory article, informant discrepancies often manifest when mental health professionals leverage multi-informant assessments to make clinical decisions in their work with individual clients (e.g., Fisher et al., 2017; Hawley & Weisz, 2003; Hoffman & Chu, 2015; Yeh & Weisz, 2001). Within these settings, what are the implications of informant discrepancies if here too they contain domain-relevant information?

Only recently have researchers laid the conceptual foundations for addressing questions surrounding the role of informant discrepancies in service delivery, and we graphically depict these foundations in Figures 3 and 4. In essence, these conceptual foundations serve as an extension of the Operations Triad Model into research and theory in implementation science. Specifically, researchers designed the Needs-to-Goals Gap framework to characterize how informant discrepancies impact the outcomes of youth mental health service delivery (De Los Reyes, Talbott, et al., 2022). To understand this impact, it is important to note key themes linking informant discrepancies as they manifest in research settings to those discrepancies seen in service settings. We previously mentioned that informant discrepancies manifest across mental health domains studied in research (e.g., symptoms, impairments, associated features) as well as across topics of study (e.g., treatment outcomes, risk factors, basic laboratory research). Similarly, informant discrepancies occur across stages of care in service delivery. In this respect, these discrepancies factor into clinical decisions about not only for which general domains a youth client might require services (i.e., needs assessments), but also the specific aspects of these domains to target for intervention and the contexts in which these treatment targets manifest (i.e., goal-setting).

Figure 3. The Needs-to-Goals Gap framework (Reproduced from De Los Reyes, Talbott, et al., 2022).

Note: When professionals base their decisions regarding service goals on a single informant (e.g., client’s parent), such a decision would be evidence-based insofar as the client’s needs and the goals of services to address those needs manifest identically across domain-relevant contexts, such as home and school (A). However, when the evidence indicates that the client’s needs and/or goals of services manifest differently across contexts (B), or the evidence is inconclusive as to whether the client’s needs and/or goals of services manifest differently across contexts (C), professionals should base their decisions regarding service goals on the views of multiple informants (e.g., client’s parent, client’s teacher, client). A mismatch between assessment approaches (i.e., single vs. multiple informants) and the context(s) in which the client’s needs manifest is thought to increase the risk of creating a Needs-to-Goals Gap in service delivery, whereby the goals set for service delivery fail to meet the client’s needs.

Figure 4. Graphical depiction of how needs assessment and goal setting processes conducted as part of service delivery might appear, if conducted consistent with the science of multi-informant assessment and the Needs-to-Goals Gap framework (Reproduced from De Los Reyes, Talbott, et al., 2022).

Note: In this depiction, we assume that caregivers initiate referral on behalf of the youth client.

Understanding how informant discrepancies impact goal-setting is of particular relevance to treatment plans within the frameworks of evidence-based interventions for youth mental health. Generally, a core feature of these interventions involves tailoring therapeutic activities to the unique needs of the clients receiving them (see Alfano & Beidel, 2011; Beck, 1993; Kazdin, 2013; Skinner, 1953). This necessitates understanding both the specific goals of services (e.g., improve social skills, decrease oppositional behavior), and the specific maintaining factors (i.e., contingencies) present in clinically relevant contexts that, if properly addressed and/or manipulated, may facilitate meeting clients’ needs.

If informant discrepancies contain domain-relevant data of the kind indicated by situational specificity, then their utility in service delivery is dictated by revealing to service providers the specific contexts in which needs assessments and goal setting processes ought to be focused. As such, the Needs-to-Goals Gap framework highlights the value of informant discrepancies when delivering services. By extension, they also highlight the consequences of ignoring them or constructing assessments that prevent one from detecting domain-relevant informant discrepancies. For example, it is common lore in service delivery that “optimal informants” exist for assessing particular mental health conditions, such as teachers for attention-deficit/hyperactivity disorder (ADHD; see Loeber et al., 1990). What if an intake assessment relies only on the teacher to assess ADHD-related needs? Within the Needs-to-Goals Gap framework, relying only on a single informant could only be justified if the client truly displayed symptoms and impairments consistently across contexts (Figure 3a), such that one informant could serve as an effective proxy for how the client behaves across relevant contexts (e.g., home and school). However, as we discussed previously these Converging Operations scenarios rarely occur, particularly within assessments conducted in service delivery settings. In fact, prior work indicates that informants such as parents and teachers cannot be relied upon to assess ADHD symptoms that manifest outside of their own context of observation (e.g., teacher for home symptoms, parent for school symptoms; see de Nijs et al., 2004; Talbott et al., 2021).

Under those circumstances that reflect Diverging Operations scenarios (Figure 3b), relying on a single informant would not only result in an inaccurate needs assessment, but would also have profound downstream effects on all aspects of services going forward. In particular, such an assessment would not have accurate context-sensitive data available for goal-setting, let alone for accurately estimating treatment responses that, presumably, might manifest in context-sensitive ways (e.g., symptom reductions observed mainly at home). Consequently, the Needs-to-Goals Gap framework highlights the importance of conducting needs assessments and engaging in goal-setting processes with an eye toward understanding whether patterns of multi-informant assessments reflect Converging Operations, Diverging Operations, or Compensating Operations scenarios (see Figure 4). Understanding which of these scenarios best fits the data obtained from informants facilitates accurate decision-making not only about crucial aspects of service delivery (e.g., goal-setting, monitoring intervention response), but also which informants are best positioned to guide decisions about contexts and contingencies germane to the client’s needs and goals for the services they receive. Taken together, we expect the Needs-to-Goals Gap framework to play an important role in future work seeking to understand the role of informant discrepancies and the implementation of multi-informant approaches to assessment within the delivery of youth mental health services.

Overview of Contributions to the Special Issue

This Special Issue includes articles that traverse several content areas. The Special Issue begins with an article that reconceptualizes measurement validation in youth mental health, by highlighting problems with applying the MTMM (Campbell & Fiske, 1959) to understand and interpret the outcomes of youth mental health assessments, and advancing a validation paradigm designed to address these problems (De Los Reyes, Wang et al., 2023). Three contributions focus on informant discrepancies observed within implementation science and the study of therapeutic processes, namely the therapeutic alliance (Roest et al., 2023), treatment fidelity (McLeod et al., 2023), and shared decision-making (Fitzpatrick et al., 2023). Three contributions focus on informant discrepancies observed when assessing specific mental health domains, including autism spectrum (Kang et al., 2023), the limited prosocial emotions specifier linked to assessments of conduct disorder (Castagna & Waschbusch, 2023), and suicide risk (Spears et al., 2023). We close the Special Issue with an editorial statement that highlights key findings, recommendations, and future directions for the study of informant discrepancies in youth mental health assessments (De Los Reyes, Epkins et al., 2023). The 70 coauthors on the statement traverse disciplines, career stages, and areas of expertise, and include journal editors and presidents of scholarly societies.

Concluding Comments

My hope is that this work paves the way toward a strong consensus among clinical researchers and practitioners that our field would greatly benefit from carefully conducted studies of the basic science behind informant discrepancies. Ideally, these studies should seek to actively distinguish the circumstances in which informant discrepancies reflect real constructs that are capable of informing clinical research and practice from the circumstances in which they should, in fact, be treated as methodological artifacts. Our field can then use the findings from this basic research to develop methods for increasing the interpretative power of informant discrepancies within clinical assessments of children. I am very curious about the work that lies ahead. After reading these articles, I hope you are curious as well. (De Los Reyes, 2011, p. 8)

We opened our concluding comments to this introductory article with the concluding paragraph of the introductory article from the 2011 JCCAP Special Section. A dozen years later, we see an accumulating body of research that clearly articulates instances in which informant discrepancies reflect domain-relevant information―data pertinent to the very domains about which informants provide reports (Figure 1). In these last dozen years, we saw the advent of analytic models conducive to testing questions about informant discrepancies, as well as innovations in measurement and study design. A dozen years later, we now have conceptual models that help us (a) distinguish informant discrepancies that reflect domain-relevant information from discrepancies that reflect measurement confounds (Operations Triad Model; Figure 2), as well as (b) understand the implications of informant discrepancies when delivering mental health services to individual youth clients (Needs-to-Goals Gap; Figures 3 and 4). A key goal in 2011 was to question the prevailing approach to informant discrepancies. Are these discrepancies solely the byproducts of research artifacts as described by McGuire (1969)? What if we revised our approach to informant discrepancies, and framed them as topics of curiosity in their own right? If given appropriate empirical and theoretical attention, might informant discrepancies even improve our understanding of youth mental health? To the degree that those of us who study informant discrepancies took this revised approach and made others more curious about what these discrepancies might reflect, then it appears we met the moment, so to speak.

Now, we all have a new moment to meet. Although we in youth mental health often encounter Diverging Operations scenarios when administering multi-informant assessments and interpreting their outcomes, we have all been carrying out this work in a “Converging Operations world.” In this world, our most commonly implemented analytic procedures for integrating multi-informant reports assume that Diverging Operations scenarios do not exist (Table 1). This reality of our research arguably has had downstream effects on settings beyond the laboratory; settings in which, much like the laboratory, we take multi-informant approaches to assessment. Unfortunately, we have yet to learn how best to integrate data from multi-informant assessments when making clinical decisions. In this moment, this is the conflict we face: We frequently encounter data conditions that are incompatible with the infrastructure we have in place for understanding, scoring, and integrating multi-informant data of all kinds—from the data collected in large-sample studies to the data resulting from clinical assessments administered to individual clients.

At the same time, we see a path to resolving this conflict. We see emerging work that sparks our curiosity and motivates us to do more. This motivation helps us build capacity to ask new questions and confront head-on the conflict we face. When informant discrepancies reflect domain-relevant information, what analytic procedures facilitate retaining this information? Do we need to develop entirely new analytic procedures, in addition to refining existing procedures? Might modified versions of procedures that were originally developed to integrate multi-informant data at the sample level facilitate such integration at the level of the individual client? To fully optimize these integrative procedures, should we also consider modifying the item content of our multi-informant instruments, or perhaps develop new instruments?

Are you curious about the answers to these questions? Do you think you have the motivation and will to address them, and perhaps extend the work of scholars who contributed to this Special Issue? Do you have new questions of your own that warrant addressing? If so, then we welcome your work on these questions. There is more than enough space for you to contribute. If informant discrepancies define the outcomes of youth mental health assessments, then we need every one of you who reads the articles in this Special Issue to put thought into questions surrounding these discrepancies. What more must we learn? What ideas must we explore? What tools must we develop? The days of actively ignoring these informant discrepancies are over. For the sake of youth mental health, we need your active engagement from here on out.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

Efforts by the first author were supported by grants from the Fulbright U.S. Scholars Program (Fulbright Canada Research Chair in Mental Health), and the Institute of Education Sciences [R324A180032].