The Operations Triad Model and Youth Mental Health Assessments: Catalyzing a Paradigm Shift in Measurement Validation

Figures & data

Table 1. Differences between CONTEXT and the MTMM paradigm in guidance on theory surrounding informant discrepancies.

Validation Paradigm	Core Assumption Regarding Informant Discrepancies	How Assumption Guides Conceptual Foundations of Paradigm-Informed Research	Examples of Theories Applied in Prior Research	Citations of Prior Work That Applied Paradigm-Informed Theories
MTMM	All informant discrepancies reflect measurement confounds	Guides users to apply theories about informant discrepancies that posit factors that produce rater biases	Depression-Distortion Hypothesis (Richters, 1992)	Bauer et al. (2013); Clark et al. (2016); Jungersen and Lonigan (2021); T.M. Olino et al. (2018, 2021)
CONTEXT	At least some informant discrepancies contain domain-relevant information	Guides users to apply theories about informant discrepancies that posit factors that produce domain-relevant information	Situational Specificity (Achenbach et al., 1987)	De Los Reyes et al. (2022a, 2022b, 2022c); Deros et al. (2018); Lerner et al. (2017); Makol et al. (2020); Watts et al. (2022)

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 2. Differences between CONTEXT and the MTMM paradigm in epistemology surrounding informant discrepancies.

Validation Paradigm	Test Implication	Logical Extension of Test Implication	Research Practices That Logically Extend from Test Implication	Relevant Citations
MTMM	Converging findings are the only data to contain domain-relevant information	Anything less than converging findings ought to be interpreted as inconclusive evidence	primary outcome measure methods; tests of measurement invariance; use of structural models that prioritize common variance; eschew estimating common or unique variance altogether and just examine informants’ reports individually	Bauer et al. (2013); Boutron et al. (2010); De Los Reyes et al. (2011); Howe et al. (2019); T.M. Olino et al. (2018)
CONTEXT	At least some informant discrepancies reflect domain-relevant information	Informant discrepancies may optimize rather than hinder the interpretability of findings	strategic selection of informants that produce context-sensitive information; use of analytic procedures that facilitate identifying informant discrepancies and testing whether they contain domain-relevant information	Deros et al. (2018); Kraemer et al. (2003); Laird (2020); Lakind et al. (2022); Makol et al. (2020)

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 3. Differences between CONTEXT and the MTMM paradigm in guidance on research practices surrounding selection of informants.

Validation Paradigm	Central Focus of Paradigm	Treatment of Individual Informants’ Reports that Logically Results from Focus	Research Practice That Logically Extends from Focus	Relevant Citations
MTMM	Attain precise estimates of common variance	If the means for selecting informants is to achieve the end of estimating common variance, then no one informant’s report has value unique to itself; each report simply reflects “error” around a measure of central tendency (i.e., common variance)	Select multiple informants for the sole purpose of maximizing precision in estimating common variance; the actual informants do not matter (e.g., could be structurally different, random raters, parents, teachers, youth, peers), so long as one attains precise estimates of common variance	Bauer et al. (2013); Edgeworth (1888); Fiske and Campbell (1992); Roberts and Caspi (2001)
CONTEXT	Strike a balance between attaining precise estimates of both common variance and domain-relevant unique variance (i.e., informant discrepancies)	If the means of the multi-informant approach involve collecting reports from structurally different informants, and the informant discrepancies resulting from this approach ought to contain domain-relevant information, then determining which specific informants will provide reports is as an important decision as determining the content of the measures the informants will complete	Strategically select informants who will produce both common variance and informant discrepancies that contain domain-relevant information; the actual informants are instrumental to optimizing measurement validity	De Los Reyes et al. (2009); Kraemer et al. (2003); Makol et al. (2020)

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 4. Differences between CONTEXT and the MTMM paradigm in guidance on research practices surrounding selection of analytic procedures.

Validation Paradigm	Central Focus of Paradigm	Logical Result of Focus	Research Practices That Logically Extend from Focus	Relevant Citations
MTMM	Attain precise estimates of common variance, because informant discrepancies reflect measurement confounds	If informant discrepancies reflect measurement confounds, then there is no legitimate reason for modeling them, other than creating estimates of error for modeling purposes	Use of analytic procedures that cannot estimate the incremental value of informant discrepancies	Alvarez et al. (2021); Murray et al. (2021); T.M. Olino et al. (2018); Russell et al. (2016); van der Ende et al. (2020)
CONTEXT	Strike a balance between attaining precise estimates of both common variance and unique variance, because informant discrepancies may reflect domain-relevant information	If informant discrepancies may reflect domain-relevant information, and common variance estimates are likely domain-relevant as well, then one must leverage analytic procedures that allow one to determine the relative value of both common and unique variance for the purpose of scholarly inquiry (e.g., characterizing behavior, predicting outcomes, theory testing)	Use of analytic procedures that facilitate testing for the incremental value of informant discrepancies	Becker-Haimes et al. (2018); De Los Reyes et al. (2009); Laird and De Los Reyes (2013); Makol et al. (2019, 2020)

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 5. Differences between CONTEXT and the MTMM paradigm in guidance on research practices surrounding study design.

Validation Paradigm	Core Assumption Regarding Informant Discrepancies	Logical Result of Focus	Research Practices That Logically Extend from Focus	Relevant Citations
MTMM	All informant discrepancies reflect measurement confounds	If informant discrepancies reflect measurement confounds, then it is logical to assume that discrepancies between informants’ reports and other non-informant modalities (e.g., observed behavior) also reflect measurement confounds	Use of study designs that cannot distinguish informant discrepancies that reflect measurement confounds from those that reflect domain-relevant information	Bauer et al. (2013); De Los Reyes et al. (2019a); Dunning et al. (2004); van der Ende et al. (2020); Watts et al. (2022)
CONTEXT	At least some informant discrepancies contain unique domain-relevant information	If the means of the multi-informant approach involve collecting reports from structurally different informants, and the informant discrepancies resulting from this approach ought to contain domain-relevant information, then detecting this information requires careful attention to selecting non-informant modalities to serve as domain-relevant validity indicators; selecting validity indicators is as an important decision as selecting the informants	Use of study designs that facilitate distinguishing informant discrepancies that reflect measurement confounds from those that reflect domain-relevant information; these designs involve “matching” the structural characteristics of the informants providing reports with the structural characteristics of the non-informant modalities used to construct validity indicators	Al Ghriwati et al. (2018); De Los Reyes et al. (2009, 2013c, 2022c); Makol et al. (2019, 2020)

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 6. Examples of how features of CONTEXT facilitate using informant discrepancies to optimize characterizations of youth mental health concerns.

Authors	Informants	Domains Informants Reported	Analytic Procedure Used to Characterize Informant Discrepancies	Criterion Variables
De Los Reyes et al. (2009)	Parent, Teacher	Disruptive behavior	Latent class analysis	Behavior rated by independent observers, based on interactions between the child and adult authority figures on a controlled laboratory task
De Los Reyes et al. (2022c)	Parent, Teacher, Child	Psychosocial impairments	Kraemer et al. (2003) Satellite Model	Behavior rated by independent observers, based on a combination of school observations and parent-child interaction tasks
Lerner et al. (2017)	Parent, Teacher	Autism spectrum	Latent class analysis	Trained raters’ evaluations of clinical severity, based on observations of client during a standardized diagnostic observation task
Makol et al. (2020)	Adolescent, Parent, Peer confederate	Social anxiety	Kraemer et al. (2003) Satellite Model	Behavior rated by independent observers, based on interactions between the adolescent and personnel trained to simulate a same-age, unfamiliar peer

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 7. Examples of how features of CONTEXT facilitate using informant discrepancies to optimize predictions of youth mental health outcomes.

Authors	Informants	Domains Informants Reported	Analytic Procedure Used to Characterize Informant Discrepancies	Criterion Variables
Becker-Haimes et al. (2018)	Parent, Child/Adolescent	Anxiety	Polynomial regression	Diagnostic remission
Makol et al. (2019)	Parent, Adolescent	Internalizing	Latent class analysis	Characteristics of care during adolescent’s stay on inpatient unit

CONTEXT = Classifying Observations Necessitates Theory, Epistemology, and Testing. MTMM = Multi-Trait Multi-Method Matrix.

Table 8. Between-group differences in identification probabilities of videotaped social skills behavior, as a function of variations in situational specificity.

Behaviors Observed by Experimental Groups	Group N	Mean Probabilities of Video Identification	Standard Deviations of Video Identification	Effect Sizes of Mean Differences Reflecting Situational Specificity
Videotaped behaviors observed by Group 1 and not Group 2 (n = 10 items)	Group 1 N: 87	Group 1 Mean: 74.71	Group 1 SD: 20.79	Cohen’s d: +2.50
	Group 2 N: 95	Group 2 Mean: 22.00	Group 2 SD: 21.42
Videotaped behaviors observed by Group 2 and not Group 1 (n = 10 items)	Group 1 N: 87	Group 1 Mean: 29.08	Group 1 SD: 18.40	Cohen’s d: -2.82
	Group 2 N: 95	Group 2 Mean: 79.16	Group 2 SD: 17.11
Videotaped behaviors observed by both groups (n = 15 items)	Group 1 N: 87	Group 1 Mean: 89.43	Group 1 SD: 13.08	Cohen’s d: -0.05
	Group 2 N: 95	Group 2 Mean: 90.11	Group 2 SD: 13.11
Behaviors observed by neither group (n = 85 items)	Group 1 N: 87	Group 1 Mean: 20.07	Group 1 SD: 12.85	Cohen’s d: +0.13
	Group 2 N: 95	Group 2 Mean: 18.25	Group 2 SD: 14.24

Richters, J. E. (1992). Depressed mothers as informants about their children: A critical review of the evidence for distortion. Psychological Bulletin, 112(3), 485–499. https://doi.org/10.1037/0033-2909.112.3.485

 PubMed Web of Science ®Google Scholar

Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. https://doi.org/10.1037/a0032475

 PubMed Web of Science ®Google Scholar

Clark, D. A., Listro, C. J., Lo, S. L., Durbin, C. E., Donnellan, M. B., & Neppl, T. K. (2016). Measurement invariance and child temperament: An evaluation of sex and informant differences on the child behavior questionnaire. Psychological Assessment, 28(12), 1646–1662. https://doi.org/10.1037/pas0000299

 PubMed Web of Science ®Google Scholar

Jungersen, C. M., & Lonigan, C. J. (2021). Do parent and teacher ratings of ADHD reflect the same constructs? A measurement invariance analysis. Journal of Psychopathology and Behavioral Assessment, 43(4), 778–792. https://doi.org/10.1007/s10862-021-09874-3

 PubMed Web of Science ®Google Scholar

Olino, T. M., Finsaas, M., Dougherty, L. R., & Klein, D. N. (2018). Is parent–child disagreement on child anxiety explained by differences in measurement properties? An examination of measurement invariance across informants and time. Frontiers in Psychology, 9, 1295. https://doi.org/10.3389/fpsyg.2018.01295

 PubMed Web of Science ®Google Scholar

Olino, T. M., Michelini, G., Mennies, R. J., Kotov, R., & Klein, D. N. (2021). Does maternal psychopathology bias reports of offspring symptoms? A study using moderated non‐ linear factor analysis. Journal of Child Psychology and Psychiatry, 62(10), 1195–1201. https://doi.org/10.1111/jcpp.13394

 PubMed Web of Science ®Google Scholar

Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101(2), 213–232. https://doi.org/10.1037/0033-2909.101.2.213

 PubMed Web of Science ®Google Scholar

De Los Reyes, A., Talbott, E., Power, T., Michel, J., Cook, C. R., Racz, S. J., & Fitzpatrick, O. (2022a). The needs-to-goals gap: How informant discrepancies in youth mental health assessments impact service delivery. Clinical Psychology Review, 92, 102114. https://doi.org/10.1016/j.cpr.2021.102114

 PubMed Web of Science ®Google Scholar

De Los Reyes, A., Tyrell, F. A., Watts, A. L., & Asmundson, G. J. G. (2022b). Conceptual, methodological, and measurement factors that disqualify use of measurement invariance techniques to detect informant discrepancies in youth mental health assessments. Frontiers in Psychology, 13, 931296. https://doi.org/10.3389/fpsyg.2022.931296

 Google Scholar

De Los Reyes, A., Cook, C. R., Sullivan, M., Morrell, N., Gresham, F. M., Wang, M., Gresham, F. M., Makol, B. M., Keeley, L. M., & Qasmieh, N. (2022c). The Work and Social Adjustment Scale for Youth: Psychometric properties of the teacher version and evidence of contextual variability in psychosocial impairments. Psychological Assessment, 34(8), 777–790. https://doi.org/10.1037/pas0001139

 Google Scholar

Deros, D. E., Racz, S. J., Lipton, M. F., Augenstein, T. M., Karp, J. N., Keeley, L. M., Qasmieh, N., Grewe, B. I., Aldao, A., & De Los Reyes, A. (2018). Multi-informant assessments of adolescent social anxiety: Adding clarity by leveraging reports from unfamiliar peer confederates. Behavior Therapy, 49(1), 84–98. https://doi.org/10.1016/j.beth.2017.05.001

 PubMed Web of Science ®Google Scholar

Lerner, M. D., De Los Reyes, A., Drabick, D. G., Gerber, A. H., & Gadow, K. D. (2017). Informant discrepancy defines discrete, clinically useful autism spectrum disorder subgroups. Journal of Child Psychology and Psychiatry, 58(7), 829–839. https://doi.org/10.1111/jcpp.12730

 PubMed Web of Science ®Google Scholar

Makol, B. A., Youngstrom, E. A., Racz, S. J., Qasmieh, N., Glenn, L. E., & De Los Reyes, A. (2020). Integrating multiple informants’ reports: How conceptual and measurement models may address long-standing problems in clinical decision- making. Clinical Psychological Science, 8(6), 953–970. https://doi.org/10.1177/2167702620924439

 Web of Science ®Google Scholar

Watts, A. L., Makol, B. A., Palumbo, I. M., De Los Reyes, A., Olino, T. M., Latzman, R. D., DeYoung, C. G., Wood, P. K., & Sher, K. J. (2022). How robust is the p-factor? Using multitrait-multimethod modeling to inform the meaning of general factors of psychopathology in youth. Clinical Psychological Science, 10(4), 640–661. https://doi.org/10.1177/21677026211055170

 PubMed Web of Science ®Google Scholar

Boutron, I., Dutton, S., Ravaud, P., & Altman, D. G. (2010). Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA, 303(20), 2058–2064. https://doi.org/10.1001/jama.2010.651

 PubMed Web of Science ®Google Scholar

De Los Reyes, A., Kundey, S. M. A., & Wang, M. (2011). The end of the primary outcome measure: A research agenda for constructing its replacement. Clinical Psychology Review, 31(5), 829–838. https://doi.org/10.1016/j.cpr.2011.03.011

 PubMed Web of Science ®Google Scholar

Howe, G. W., Dagne, G. A., Brown, C. H., Brincks, A. M., Beardslee, W., Perrino, T., & Pantin, H. (2019). Evaluating construct equivalence of youth depression measures across multiple measures and multiple studies. Psychological Assessment, 31(9), 1154–1167. https://doi.org/10.1037/pas0000737

 PubMed Web of Science ®Google Scholar

Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(9), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566

 PubMed Web of Science ®Google Scholar

Laird, R. D. (2020). Analytical challenges of testing hypotheses of agreement and discrepancy: Comment on Campione-Barr, Lindell, and Giron (2020). Developmental Psychology, 56(5), 970–977. https://doi.org/10.1037/dev0000763

 PubMed Web of Science ®Google Scholar

Lakind, D., Bradley, W. J., Patel, A., Chorpita, B. F., & Becker, K. D. (2022). A multidimensional examination of the measurement of treatment engagement: Implications for children’s mental health services and research. Journal of Clinical Child and Adolescent Psychology, 51(4), 453–468. https://doi.org/10.1080/15374416.2021.1941057

 PubMed Web of Science ®Google Scholar

Edgeworth, F. Y. (1888). The statistics of examinations. Journal of the Royal Statistical Society, 51(3), 598–635. https://www.jstor.org/stable/2339898

 Google Scholar

Fiske, D. W., & Campbell, D. T. (1992). Citations do not solve problems. Psychological Bulletin, 112(3), 393–395. https://doi.org/10.1037/0033-2909.112.3.393

 Web of Science ®Google Scholar

Roberts, B. W., & Caspi, A. (2001). Personality development and the person-situation debate: It’s déjà vu all over again. Psychological Inquiry, 12(2), 104–109. https://doi.org/10.1207/S15327965PLI1202_04

 Web of Science ®Google Scholar

De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37(5), 637–652. https://doi.org/10.1007/s10802-009-9307-3

 PubMed Web of Science ®Google Scholar

Alvarez, I., Herrero, M., Martínez‐Pampliega, A., & Escudero, V. (2021). Measuring perceptions of the therapeutic alliance in individual, family, and group therapy from a systemic perspective: Structural validity of the SOFTA‐s. Family Process, 60(2), 302–315. https://doi.org/10.1111/famp.12565

 PubMed Web of Science ®Google Scholar

Murray, A. L., Speyer, L. G., Hall, H. A., Valdebenito, S., & Hughes, C. (2021). Teacher versus parent informant measurement invariance of the strengths and difficulties questionnaire. Journal of Pediatric Psychology, 46(10), 1249–1257. https://doi.org/10.1093/jpepsy/jsab062

 PubMed Web of Science ®Google Scholar

Russell, J. D., Graham, R. A., Neill, E. L., & Weems, C. F. (2016). Agreement in youth-parent perceptions of parenting behaviors: A case for testing measurement invariance in reporter discrepancy research. Journal of Youth and Adolescence, 45(10), 2094–2107. https://doi.org/10.1007/s10964-016-0495-1

 PubMed Web of Science ®Google Scholar

van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2020). Multitrait-multimethod analyses of change of internalizing and externalizing problems in adolescence: Predicting internalizing and externalizing DSM disorders in adulthood. Journal of Abnormal Psychology, 129(4), 343–354. https://doi.org/10.1037/abn0000510

 PubMed Web of Science ®Google Scholar

Becker-Haimes, E. M., Jensen-Doss, A., Birmaher, B., Kendall, P. C., & Ginsburg, G. S. (2018). Parent–youth informant disagreement: Implications for youth anxiety treatment. Clinical Child Psychology and Psychiatry, 23(1), 42–56. https://doi.org/10.1177/1359104516689586

 PubMed Web of Science ®Google Scholar

Laird, R. D., & De Los Reyes, A. (2013). Testing informant discrepancies as predictors of early adolescent psychopathology: Why difference scores cannot tell you what you want to know and how polynomial regression may. Journal of Abnormal Child Psychology, 41(1), 1–14. https://doi.org/10.1007/s10802-012-9659-y

 PubMed Web of Science ®Google Scholar

Makol, B. A., De Los Reyes, A., Ostrander, R., & Reynolds, E. K. (2019). Parent-youth divergence (and convergence) in reports of youth internalizing problems in psychiatric inpatient care. Journal of Abnormal Child Psychology, 47(10), 1677–1689. https://doi.org/10.1007/s10802-019-00540-7

 PubMed Web of Science ®Google Scholar

De Los Reyes, A., Lerner, M. D., Keeley, L. M., Weber, R., Drabick, D. A. G., Rabinowitz, J., & Goodman, K. L. (2019a). Improving interpretability of subjective assessments about psychological phenomena: A review and cross-cultural meta-analysis. Review of General Psychology, 23(3), 293–319. https://doi.org/10.1177/108926801983764

 Web of Science ®Google Scholar

Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5(3), 69–106. https://doi.org/10.1111/j.1529-1006.2004.00018.x

 PubMedGoogle Scholar

Al Ghriwati, N., Winter, M. A., Greenlee, J. L., & Thompson, E. L. (2018). Discrepancies between parent and self-reports of adolescent psychosocial symptoms: Associations with family conflict and asthma outcomes. Journal of Family Psychology, 32(7), 992–997. https://doi.org/10.1037/fam0000459

 PubMed Web of Science ®Google Scholar