Clinical Significance in Counseling Outcome Research and Program Evaluation

This is a pivotal time for counseling outcome researchers and evaluation scientists. The efficacy of interventions and programs intended to promote well-being are increasingly challenged through precipitous changes in the complexity of our socio-ecological context. In response, our colleagues within the clinical and program development arms of the counseling enterprise are working ardently to innovate culturally-responsive and developmentally-informed approaches. Their activities have provided roadmaps to embodying the protective factors and resiliency characteristics that have been demonstrated as generative across the lifespan. I see this emerging body of work and am grateful for the intention, effort, and output. Our colleagues’ activities have not only inspired hope, but also a sense of promise that our profession will continue to meet individuals, families, schools, neighborhoods, and communities where they are and support their journey forward.

It is on the bridge between intervention and meaningful representation of results that we are called. As counseling outcome researchers and evaluation scientists, we are charged with prudent, even-handed, and contextually-referenced representation of findings. Statistical significance? No problem. Effect sizes? Check! Clinical significance? Now that is one area many of us could use some support. Although the Publication Manual of the American Psychological Association (American Psychological Association, 2020) and several best practice commentaries encourage reporting of clinical significance (LaGreca, 2005; Thompson, 2002), the presence of related outputs is largely absent within the professional counseling literature. While the publication values of Counseling Outcome Research and Evaluation (CORE) have embraced a position methodological diversity and transparent reporting practices (Lenz, 2018, 2020a), our pages are no exception to the trend (Cade et al., 2018; Johnson et al., in press). I believe that CORE can make a difference in how practitioners, researchers, and policy makers use the results of our articles. We are able to support our authors in the statistical reporting of clinical significance for relevant measures and are committed to related activities moving forward.

Conceptualizing Clinical Significance

Plainly stated, clinical significance represents the real-world meaningfulness that an intervention or program has on quality of life for clients or those with whom they interact. While statistical significance and effect size warrant embrace in their usefulness to convey the presence and magnitude of counseling intervention effects (Watson et al., 2016), clinical significance can support inferences about the convincingness of results (Lenz, 2020b). How do we infer that degree of convincingness? Kendall (1999) suggested that clinical significance can be inferred by answering either of 2 questions: (a) Has a personally meaningful amount of change occurred for individuals or groups; or (b) Are the those who received an intervention indistinguishable in meaningful ways from individuals or groups who did not require it in the first place?

Calculating Clinical Significance

We live in an unprecedented time of access to resources that support computation of clinical significance using indices such as Percent Improvement (PI; Blanchard & Schwarz, 1988), the Reliable Change Index (RCI; Jacobson & Truax, 1991), and normative group comparisons (Kendall et al., 1999). The decision of which strategy to use will be based on the characteristics of your design and data. For example, authors in this issue of CORE (Balkin & Russo, 2021; Ikonomopoulos et al., 2021) reported PI based on outcomes that were represented by the frequency or severity of symptoms measured using interval or ratio-level scales. The PI index is easily calculated as a proportion representing intervention gains with respect to pre-intervention severity/functioning: $$\text{Percent Improvement}=\mathrm{ }100\text{ * }\frac{\left[\mathit{Preintervention}\mathrm{ }\mathit{Scores}-\mathrm{ }\mathit{Postintervention}\mathrm{ }\mathit{Scores}\right]}{\mathit{Preintervention}\mathrm{ }\mathit{Scores}}$$

By establishing the metric as a function of pre-intervention severity/functioning, estimations of clinical significance based on PI are not subject to the biases of arbitrary cutoffs. As a result, sample-specific improvement can be represented across individual, single-group, and between-groups data contrasts.

Interpreting and Reporting Clinical Significance

Each index of clinical significance is associated with unique interpretative conventions. For example, when RCI values are greater than 1.96, the observed difference is regarded as representing change that exceeds the influence of a measure’s unreliability. While useful, this may produce a difficult dichotomy—do we categorically reject an RCI of 1.95 as unacceptable? In instances when an observed value is near the categorical fulcrum, some authors have recommended supplementing with another measure such as normative group comparisons (Ferguson et al., 2002). Alternatively, measures such as the PI offer non-dichotomous representations such as those provided by Devineni and Blanchard (2005) wherein values greater than 50% are regarded as improved and clinically significant, those between 25 and 49% represent slight improvement that is not clinically significant, and values less than 25% suggest unimproved conditions that lack clinical significance.

Reporting Clinical Significance

Reporting practices for clinical significance are possible at the individual and group levels (see Lenz, 2020b). Whether indicating the percentage of individuals that experienced clinically significant change or basing estimates on group means, effort should be made to situate the findings back into context from which they are drawn. That is, our reporting clinical significance estimates is not context-free and perhaps at its best when the degree of convincingness is readily transparent. Because there are not readily available indicators of context in any of the clinical significance formulae, it is up to authors to paint the picture for readers with an even and objective hand. Furthermore, while narrative accounts and tables are the traditional mechanisms for reporting results, visual representations that integrate findings meaningfully are a great support to non-technical readers or other stakeholders.

The Journey Forward

The CORE editorial team recognizes that a valuing of clinical significance in reporting practices has to go beyond good intentions. Therefore, we have viewed our most proximal opportunities for influence as: (a) acknowledging that clinical significance may not apply to every submission, (b) modifying our reviewer forms to include space for feedback related to the clinical significance of findings as appropriate, and (c) providing feedback through the editorial process to support our authors' reporting practices. The CORE editorial team is excited about this initiative and welcomes this opportunity to support the integrity of scholarship within the counseling profession and advance our field through the process.