Enhancing the effect of psychotherapy through systematic client feedback in outpatient mental healthcare: A cluster randomized trial.

ABSTRACT

Objective: Systematic client feedback (SCF), the regular monitoring and informing of patients’ progress during therapy to patient and therapist, has been found to have effects on treatment outcomes varying from very positive to slightly negative. Several prior studies have been biased by researcher allegiance or lack of an independent outcome measure. The current study has taken this into account and aims to clarify the effects of SCF in outpatient psychological treatment.

Method: Outpatients (n = 1733) of four centers offering brief psychological treatments were cluster randomized to either treatment as usual (TAU) or TAU with SCF based on the Partners for Change Outcome Management System (PCOMS). Primary outcome measure was the Outcome Questionnaire (OQ-45). Effects of the two treatment conditions on treatment outcome, patient satisfaction, dropout rate, costs, and treatment duration were assessed using a three-level multilevel analysis. DSM-classification, sex, and age of each patient were included as covariates.

Results: In both analyses, SCF significantly improved treatment outcome, particularly in the first three months. No significant effects were found on the other outcome variables.

Conclusions: Addition of systematic client feedback to treatment as usual, is likely to have a beneficial impact in outpatient psychological treatment. Implementation requires a careful plan of action.

Clinical or methodological significance of this article: This study, with large sample size and several independent outcome measures, provides strong evidence that addition of systematic client feedback to outpatient psychological treatment can have a beneficial effect on treatment outcome (symptoms and wellbeing), particularly in the first three months. However, implementation requires a careful plan of action.

Keywords:

• systematic client feedback
• PCOMS
• outpatient psychological treatment
• implementation strategy

Introduction

Throughout the world, in different mental health care settings, ranging from counseling centers to inpatient care, there is a growing interest in systematic client feedback which refers to regularly monitoring patients’ progress in therapy and informing the therapist and/or patient of this progress. Meta-analyses demonstrated that such feedback significantly improves outcomes on symptom reduction or lower drop-out rates (De Jong et al., 2021; Lambert et al., 2018; Østergård et al., 2020). In addition, some studies have reported a reduction of treatment duration (e.g., Janse et al., 2016) or a reduction of costs (e.g., Delgadillo et al., 2017, 2021).

Several systematic client feedback systems are currently in use, each administering different aspects of treatment outcome (Kendrick et al., 2016). Most systems address a broad domain of outcomes, such as quality of life in the Medical Outcomes Study Short Form SF-36 (Wells et al., 1989) or symptoms and functioning as measured by the Outcomes Questionnaire System (OQ-45; Lambert et al., 2004), the Partners for Change Outcome Management System (PCOMS; Duncan et al., 2003; Duncan & Sparks, 2002; Miller et al., 2003) or the Clinical Outcomes in Routine Evaluation Outcome Measure (CORE-OM; Barkham et al., 2006)

PCOMS is a widely used feedback system that systematically administers the Outcome Rating Scale (ORS; Miller et al., 2003), assessing wellbeing in daily life on a interpersonal and intrapersonal level as well as on social roles, and the Session Rating Scale (SRS; Duncan et al., 2003), assessing the therapeutic alliance aspects as defined by Bordin (1979): the affective bond, topics and goals, and approach or method. The ORS is administered at the beginning of each therapy session, the SRS at the end of each session.

Østergård et al. (2020) explored the effects of PCOMS in their systematic review and meta-analysis of 18 studies, finding a small overall positive effect of using feedback, but with substantial heterogeneity between the different studies. The effect was moderated by the setting; the effect of PCOMS on outcomes was only found in counseling settings, not in psychiatric settings. Similarly, Davidson et al. (2015) concluded that the beneficial effect of feedback diminishes with more severe psychiatric populations. Some studies that were conducted in severe settings have even found small negative effects of feedback, such as Oenen et al. (2016) in emergency psychiatry.

However, most of the studies Østergård et al. examined in their meta-analysis, used PCOMS not only as a feedback instrument but also as an instrument for outcome. This absence of an independent outcome measure is likely to have biased the results (also see Lambert et al., 2018). In addition, the majority of studies were conducted by researchers affiliated to PCOMS, and researcher allegiance was associated with a positive effect (Østergård et al., 2020). Finally, implementing PCOMS requires proper training and Bovendeerd et al. (2019) found that the amount of education in the use of feedback has varied substantially across studies, ranging from 0 to 17 h of training.

Thus, while adding systematic client feedback to therapy can be beneficial, due to methodological weaknesses in earlier research, it is still unclear to what extent a specific patient population will benefit or not. Using feedback when it has no added value is inefficient, and could negatively influence therapy outcome or discourage therapists (Wolpert, 2014).

In summary, research on systematic client feedback must take several issues into account. First, there is a possible researcher allegiance bias. If feedback only works when the researcher beliefs it will work, it is questionable if it should be implemented widely. Secondly, using a feedback instrument simultaneously as monitoring instrument and as outcome measurement for research, should be avoided. Finally, to ensure adherence and to prevent that any lack of added value of feedback could be attributed to poor implementation, training in the use of the feedback instrument should be thorough.

The current study aimed to further evaluate the beneficial effect of systematic client feedback in outpatient psychological treatment by adding PCOMS to treatment as usual (TAU), as well as identify potential moderators of feedback. To optimize research conditions, therapists were trained in the use of the feedback instrument in a one and a half day training program, and independent outcome measures were used. In addition, an independent research group conducted the study. Our primary hypothesis was that systematic client feedback would enhance treatment outcome, measured in psychiatric symptoms and wellbeing, and our secondary hypothesis was that systematic client feedback would improve patient satisfaction, and would decrease drop-out rates, treatment duration, and costs.

Method

Design

This study is a four-center cluster randomized trial in outpatient psychological treatment. The four participating centers were randomized into two conditions by an independent researcher using the http://www.randomization.com website. All participating patients received brief therapy (TAU), and in two centers PCOMS was added (TAU-PCOMS). Data were collected from January 1, 2016 till December 31, 2017. The DSM IV-TR classification (APA, 2000), sex, and age of each patient were included as covariates. The design and methods have been comprehensively described in the published study protocol (Bovendeerd et al., 2019).

Setting

The data were acquired at four centers of Mindfit, a medium sized outpatient mental health organization in the Netherlands. These centers are treating patients with mild to moderate psychological disorders with a DSM classification (APA, 2000). Time-limited brief therapy of on average 7 face to face sessions is offered, with a maximum of 12 sessions over a period of 12 months supplemented with telehealth interventions and online exercises. Their therapeutic interventions are based on (positive) cognitive behavioral therapy (Seligman, 2011) and solution focused therapy (Shazer et al., 2007).

In the year prior to the study, the four participating centers showed no significant differences in therapy outcome. In addition, no significant differences in outcomes were found when these centers were compared to the overall Mindfit organization (Bovendeerd et al., 2019).

Participants

All included patients had mild to moderate psychological disorders with a DSM classification, were 18 years or older, mastered Dutch as their first or second language, and did not object against using their anonymized Routine Outcome Monitoring (ROM) data for scientific research. During the inclusion period, 2552 patients were referred to psychological treatment by their general practitioner. Patients refraining from treatment in advance (n = 98), not completing questionnaires at intake (n = 507), or younger than 18 (n = 14) were excluded. Patients were not excluded based on a specific diagnosis. In total, data of 1933 patients were included; 936 in the centers assigned to TAU, and 997 in the centers assigned to TAU-PCOMS.

After intake, 200 patients (10.3%) did not receive treatment, either because their disorder was too severe or too mild to receive treatment in outpatient psychological treatment, or because they only received psychological assessment. The remaining 1733 patients were analyzed using the intention to treat analysis (ITT-analysis).

In the per protocol analysis (PP), patients receiving less than three full sessions of therapy (n = 95), not completing therapy (dropout, n = 150), or not completing PCOMS for at least three sessions during treatment in the TAU-PCOMS condition (n = 586), were excluded. The remaining 902 patients were included in the PP analysis. Patient flow through the study is presented in Figure 1.

Figure 1. Enrollment.

This trial was registered on September 30, 2015 in the Dutch Trial Register NTR5466. The Medical Ethics Committee of the University of Twente (Enschede) approved this study (registration number: K15-11, METC Twente).

Measurements

The Dutch version of the OQ-45 (De Jong et al., 2009; Lambert et al., 2004) measuring psychiatric symptoms and wellbeing was used as the primary outcome measure. It contains 45 items, using a five-point scale ranging from 0 (never) to 4 (almost always) on three subscales: Symptom Distress, Interpersonal Relationships, and Social Role. In this study, we only used the total score (ranging from 0 to 180), which is calculated adding all the item-scores together. An example of an item of the OQ-45 is “I have thoughts of ending my life.” The higher the score, the more problems the patient reports. The reliable change index is 14 points, and its clinical cut-off score is 63. De Jong et al. examined the reliability and validity of the Dutch translation (2007) finding psychometric properties similar to the original OQ-45 and a cut-off score for clinical dysfunctioning eight points below the American cut-off point (55 instead of 63).

The Mental Health Continuum Short Form, Dutch version (MHC-SF; Keyes, 2002; Lamers et al., 2011), and the Consumer Quality Index short version for mental healthcare (CQI; Wijngaarden et al., 2008) were used as secondary outcome measures. The MHC-SF contains 14 items on emotional, psychological and social wellbeing, ranging from 0 to 5. The total sum score is the overall experienced positive mental health. An item of the MHC-SF is “How often did you feel that you liked most parts of your personality?”, and its psychometric properties were empirically tested by Lamers et al. (2011) finding high internal reliability (Cronbach’s α = 0.89 for the total MHC-SF) and moderate test-retest reliability. The reliable change (RC) for the MHC-SF is 0.88 (initial SD = 0.96, Cronbach’s α = 0.89).

The CQI, administered at the end of the therapy, measures customer experiences in mental healthcare on interpersonal conduct, accessibility to the therapist, received information and therapy options. One item questions the overall rating: “What mark do you give to the treatment? A Score from 0 to 10, where 0 means ‘very bad’ and 10 means ‘excellent’.” Wijngaarden et al. (2008) studied the psychometric properties of the CQI and concluded that the CQI is applicable for patients receiving brief (psycho)therapy.

The therapy dropout percentage, costs, and therapy duration were also assessed, using data from the electronic health record. Several patient variables (sex, age, DSM classification, treatment duration, costs, and reason for therapy ending) were gathered automatically as part of treatment.

Sample Size Calculation

To achieve 80% power in a test with a two-sided alternative hypothesis, type I error rate α = 0.05 and a small effect size (d = 0.2), we needed 26 patients per therapist, or a total of 208 patients per center. The power analysis is described in more detail in the study protocol (Bovendeerd et al., 2019).

Intervention

PCOMS is a systematic client feedback system using the ORS and SRS, two brief (four items), 10 cm long, visual analogue scales. In this study they were both administered digitally. The ORS, administered at the beginning of each session, assesses the domains of intrapersonal, interpersonal, social and overall wellbeing. The improvement or deterioration of the results are discussed immediately, and the topics of the forthcoming therapy session are specified, based on that conversation. Its psychometric properties were tested by Miller et al. (2003), finding a high internal consistency, test-retest reliability, and sensitivity to change. The SRS, completed at the end of each therapy session, administers the affective bond, topics and goals, and approach or method of the working alliance (Bordin, 1979). The improvement or deterioration of the working alliance, and any suggestions for improving upcoming sessions are instantly reviewed. Duncan et al. (2003) examined its psychometric properties and found that the SRS has similar reliability and validity as longer alliance measures.

To ensure therapists paid attention to stagnating or deteriorating patients, the ORS and SRS graphs were accompanied with green upward pointing arrows to signal progress compared to the ORS score of the previous session, or red downward pointing arrows in case of stagnation or deterioration. The intervention is described in more detail in the study protocol (Bovendeerd et al., 2019).

Therapists in the feedback condition were given a full day training in the use of PCOMS, then practiced for a month in its use, after which they received a 4-hour follow-up training. Training was given by an experienced and certified trainer of PCOMS, following the Dutch PCOMS manual translated by Crouzen (2010). After the follow-up training, therapists first practiced for two more months in adding PCOMS to TAU before the first patient was included in this study. During the inclusion period, every four to six weeks, therapists of all four centers were given supervision by the lead researcher, a PCOMS senior therapist. In the PCOMS condition, therapists were encouraged to discuss the ORS and SRS patterns in this supervision. None of the therapists, nor the lead researcher had any competing interest in the PCOMS method.

Procedure

In all four centers, therapy was offered by registered psychologists and psychiatric nurses. In the TAU-PCOMS condition, PCOMS was added to therapy. Administering the outcome measures is part of ROM and was already daily practice prior to the study in the participating centers. The intention was to assess patients four times: at the beginning, after 5 weeks, after 13 weeks, and at the end of therapy. The Outcome Questionnaire (OQ-45) and Mental Health Continuum Short Form (MHC-SF) were administered in all assessments, the Consumer Quality Index (CQI) was added in the final measurement.

Statistical Analysis

Data were analyzed using IBM SPSS Statistics version 22, and MLWin version 3.00. Pre-treatment differences between the two conditions in sex and diagnosis were tested with chi squared tests. T-tests were used to examine differences in age and in initial score on the OQ-45 and MHC-SF.

In the primary analysis, the outcomes of the two treatment conditions on treatment outcome (OQ-45 and MHC-SF), patient satisfaction (CQI), costs and duration were compared using a multilevel model (Hox et al., 2018). The DSM classification, sex, and age of each patient were included as covariates. Two analyses were carried out; the Intention To Treat analysis (ITT) to examine the general effect and the Per Protocol-analysis (PP) as a sensitivity analysis. Although planned in our study-protocol (Bovendeerd et al., 2019), education level was not included as a covariate in our model, due to too many missing variables (24.2% in the ITT and 23.7% in the PP). On account of the limited number of participating centers, we used fixed effects regression to correct for the clustering on center-level (Moerbeek et al., 2003). Missing data were not imputed due to the large dataset with considerable power. All available data were taken into account for calculation of the model.

The model comparing the TAU and the TAU-PCOMS group over time, was built with a three-level multilevel analysis due to the nested structure. To decide whether to use a two- or three-level model, we first built three three-level models. In the first model with no dummies for center, we found very little variance at the therapist level as compared to the level of patient and the level of repeated measures (Intra Class Correlation = 0.0008). In the second three-level model, with dummies for center, we found no variance on therapist level either. In the third model, however, with dummies for center, we added a time variable and the effect of time was allowed to vary across therapists and patients. In this third model the variance of the effect of time at therapist level was small but significant. Such significant variance of the effect of time at therapist level was found in four other analysis (ITT costs, ITT duration, PPA costs & PPA duration). In the five remaining analyses (PPA-OQ, ITT-MSF, PPA-MSF, ITT-CQi & PPA-CQi) the variance of the effect of time at therapist level was small and not significant.

In addition, we tested the effect of removing the therapist level from all analyses. This marginally affected the results. Because significant variance of the effect of time was found at therapist level in half of the analyses, including the main analysis, we decided to use a three-level model in all analyses to ensure uniformity.

The model was built with repeated measures on the OQ total score of each patient on the first level, patients on the second level, and therapists on the third level. Time was measured in days and baseline was coded with the value 0. Treatment was coded 0 for TAU and +1 for TAU-PCOMS and was used as predictor of both the intercept and time effect. The rate of change was allowed to vary across the patients and therapists. Dummy variables were used to represent the centers. In addition, we checked the assumptions underlying the multilevel model, in particularly homoscedasticity, normality of residuals and linearity between continuous predictor variables and the outcome variable. The deviance test was used to compare nested models to each other, where the deviance is calculated as -2*loglikelihood. We fitted both a loglinear and a linear-quadratic model and found that the linear-quadratic model had a slightly better fit (Akaike Information Criterion 31031.4 versus 31056.3, a difference of 0.12%). A possible explanation for this finding is the fact that the results include treatments of different duration (De Jong et al., 2007; Knaup et al., 2009). However, loglinear growth cures are more in line with the phase model of psychotherapy outcome (e.g., Sembill et al., 2019) and the difference between the two models is very small. We, therefore, decided to use the loglinear model.

The drop-out rate of the two treatment conditions was compared with a chi-square test.

Results

Preliminary Analysis

Patients. Patients included in this study had mild to moderate psychological disorders with a DSM classification, mainly with an anxiety or a depressive disorder. Pre-treatment tests between the TAU and TAU-PCOMS group showed no significant differences in number of males and females, diagnosis, age or initial score on the MHC-SF. A small but significant difference was found on the initial score in the OQ-45 (t(1733) = 2.217, p = 0.03, 95% CI [0.26, 4.20]; d = 0.11).

Characteristics of patients including the distribution of patients diagnoses, can be found in Table I.

Table I. Characteristics of patients in both treatment conditions.

	Condition
	TAU			TAU-PCOMS
	N	%		N	%
Sex
Male	286	35.0		351	38.3
Female	531	65.0		565	61.7
Diagnosis
Anxiety disorder	341	41.7		422	46.1
Depressive disorder	240	29.4		260	28.4
Psychosomatic disorder	140	17.1		139	15.2
Developmental disorder	41	5.0		54	5.9
Otherwise	55	6.7		41	4.5
Age	817	M = 37.04	SD = 13.8	916	M = 37.91	SD = 13.0
OQ-45 total at intake	817	M = 73.89*	SD = 21.18	916	M = 76.12	SD = 20.57
MHC-SF total at intake	776	M = 2.43	SD = 0.96	695	M = 2.38	SD = 0.93

Note: OQ-45 = Outcome Questionnaire, MHC-SF = Mental Health Continuum-Short Form. TAU = Treatment as Usual, PCOMS = Partners for Change Outcome Management System. *p ≤ 0.05.

Primary Outcome

Effect on treatment outcome on the OQ-45. The model comparing the TAU and the TAU-PCOMS group over time, was built with a three-level multilevel analysis using dummy variables to represent the centers (see Table II).

Table II. Three level model and dummy variables of the effect of systematic client feedback, ITT-analysis.

	Measure
	OQ-45		MHC-SF
	Estimate	SE	Estimate	SE
Fixed effects
Intercept	53.345***	4.555	2.959***	0.207
Dummy TAU-PCOMS center	0.917	0.665	−0.099**	0.034
Dummy TAU center	−1.574**	0.563	0.043	0.032
Female	2.739**	1.001	−0.003	0.046
Anxiety disorder	6.049**	2.170	0.033	0.101
Depressive disorder	14.793***	1.995	−0.480***	0.104
Psychosomatic disorder	7.234**	2.106	−0.056	0.110
Developmental disorder	5.598*	2.453	−0.090	0.137
Age	0.626**	0.199	−0.022*	0.009
Age^2	−0.008***	0.002	0.000*	0.000
Log10Time	−7.656***	0.492	0.256***	0.021
Condition	1.939*	0.887	−0.060	0.048
Condition*Log10Time	−2.051**	0.651	0.064*	0.030
Random effects
Level 1 (measurement)
Residual	172.191	10.924	0.292	0.020
Level 2 (patient)
Variance intercept	237.114	17.583	0.547	0.037
Covariance Log10Time/intercept	18.282	7.23	−0.025	0.015
Variance Log10Time	13.234	5.297	0.041	0.010
Level 3 (therapist)
Variance intercept	0.000	0.000	0.000	0.000
Covariance time/intercept	0.000	0.000	0.000	0.000
Variance Log10Time	1.369	0.81	0.002	0.001

Note. OQ-45 = Outcome Questionnaire, MHC-SF = Mental Health Continuum-Short Form, ITT = Intention to Treat, TAU = Treatment as Usual, PCOMS = Partners for Change Outcome Management System, Time is in days. *p < 0.05; **p < 0.01; ***p ≤ 0.001.

At intake, females showed higher OQ-scores than males and significant differences between the diagnoses were found. We also found a linear and negative quadratic effect of age. Furthermore, a significant log10 linear effect of time was found. The main effect of condition was significant, meaning the OQ-scores did differ across the two conditions at baseline. Inspection of QQ-plots of residuals and scatterplots of predicted OQ-score versus residual showed the assumptions of normality and homoscedasticity were not violated.

In the ITT-analysis, the model revealed a different change of the OQ over time across treatments in favor of the TAU-PCOMS condition, showing a significant interaction between condition and the log10 linear time trend. The improvement on the OQ-45 during treatment in the two conditions is shown in Figure 2.

Figure 2. Growth curves of the OQ-45.ITT.

On average, therapy lasted 179 days (M = 178.88, SD = 74.86). The OQ-score decreased in both conditions, but at a higher rate in the TAU-PCOMS condition, particularly in the first three months. After 1 year, patients in the TAU condition displayed a 19.63 point improvement on the OQ-45, whereas patients in the TAU-PCOMS condition displayed an improvement of 24.88 points, a difference of 5.25 points or 26.8%. The majority (76.4%) of this additional improvement (4.02 out of 5.25 points) was achieved in the first three months. The interaction between log10 linear time and condition is significant (B = −2.051, SE = 0.651, p = 0.002, 95% CI [−3.327, −0.774]).

Secondary Outcomes

Effect on wellbeing on the MHC-SF. In the ITT-analysis, a significant interaction effect between log10linear time and condition, as was also found with the OQ-45, was found using the MHC-SF as the dependent variable, showing more improvement in the TAU-PCOMS condition, primarily in the first three months. The three-level model can be found in Table II and the improvement on the MHC-SF during treatment in the two conditions can be found in Figure 3.

Figure 3. Growth curves of the MHC-SF.ITT.

After 1 year, in the ITT-analysis, patients in the TAU condition show a 0.66 point improvement on the MHC-SF and patients in the TAU-PCOMS condition show an improvement of 0.82 points, a difference of 0.16 points or 25.0%. 76.2% of this additional improvement (0.13 out of 0.16 points) was achieved in the first three months. The interaction between log10 linear time and condition is significant (B = 0.064, SE = 0.030, p = 0.032, 95% CI [0.006, 0.123]).

Effect on patient satisfaction, costs and duration. No effect of adding PCOMS to TAU was found on patient satisfaction as measured by the CQI (B = −0.276, SE = 0.204, p = 0.177, 95% CI [−0.676, 0.124]). No effects of adding PCOMS were found between the two conditions on costs (B = 18.847, SE = 38.176, p = 0.622, 95% CI [−55.976, 93.670]) and duration (B = 24.846, SE = 16.161, p = 0.124, 95% CI [−6.829, 56.521]).

Effect on drop-out rate. In the TAU condition, 81 patients terminated their therapy premature (9.9%); in the TAU-PCOMS 69 patients ((7.5%), also see Figure 1). The difference between these groups is not significant, χ² (1, N = 1733) = 3.1, p = .078.

Sensitivity Analysis

The adherence of adding PCOMS to TAU is relatively low (see Figure 1). Only 206 of the 792 patients (26%) who could have received PCOMS in addition to TAU actually received PCOMS. Low therapists’ treatment adherence is a common problem in psychotherapy, Tschuschke et al. (2015) found rates of adherence of specific interventions in eight different treatment approaches ranging from 4.2% to 27.8%. Brattland et al. (2018) emphasized the importance of adherence specifically in PCOMS and also found that therapists’ adherence to PCOMS increases over time.

Low adherence rates may dilute the measured effect of adding PCOMS in the ITT analysis. On the other hand, the ITT sample is relatively large, which actually increases the likelihood of finding significant effects. As advised by Thabane et al. (2013), a PP analysis was performed to assess the robustness of the results with respect to protocol deviations.

Effect on treatment outcome on the OQ-45. In the PP-analysis, the same log10 linear model comparing the TAU and the TAU-PCOMS group over time, was built with a three-level multilevel analysis using dummy variables to represent the centers (see Table III).

Table III. Three level model and dummy variables of the effect of systematic client feedback, PP-analysis.

	Measure
	OQ-45		MHC-SF
	Estimate	SE	Estimate	SE
Fixed effects
Intercept	57.688***	5.792	2.783***	0.263
Dummy PCOMS center	0.891	2.130	−0.066	0.106
Dummy TAU center	−1.912**	0.735	0.058	0.035
Female	2.011	1.348	−0.017	0.061
Anxiety disorder	4.269	2.870	0.053	0.129
Depressive disorder	14.411***	2.946	−0.480***	0.132
Psychosomatic disorder	6.344*	3.114	−0.063	0.140
Developmental disorder	5.832	4.394	−0.126	0.199
Age	0.470	0.267	−0.011	0.012
Age^2	−0.006	0.003	0.000	0.000
Log10Time	−7.866***	0.441	0.259***	0.021
Condition	3.177	2.307	−0.105	0.114
Condition*Log10Time	−2.160**	0.823	0.104**	0.008
Random effects
Level 1 (measurement)
Residual	167.288	11.826	0.298	0.022
Level 2 (patient)
Variance intercept	227.087	21.884	0.547	0.046
Covariance Log10Time/intercept	15.099	8.701	−0.034	0.018
Variance Log10Time	16.709	6.070	0.040	0.012
Level 3 (therapist)
Variance intercept	0.000	0.000	0.000	0.006
Covariance time/intercept	0.000	0.000	0.000	0.002
Variance Log10Time	0.544	0.747	0.002	0.002

Note. OQ-45 = Outcome Questionnaire, MHC-SF = Mental Health Continuum-Short Form, PP = Per Protocol, Time is in days. TAU = Treatment as Usual, PCOMS = Partners for Change Outcome Management System. *p < 0.05;**p < 0.01;***p ≤ 0.001.

At intake, significant differences between the diagnoses were found. We also found a linear and quadratic effect of age. Furthermore, a significant log10 linear effect of time was found. The main effect of condition was not significant and inspection of QQ-plots of residuals and scatterplots of predicted OQ-score versus residual showed the assumptions of normality and homoscedasticity were not violated. The improvement on the OQ-45 during treatment in the two conditions is shown in Figure 4.

Figure 4. Growth curves of the OQ-45.PP.

In the PP-analysis, the model revealed a change of the OQ over time across treatments in favor of the TAU-PCOMS condition, similar to the ITT analysis, showing a significant interaction between condition and the log10 linear time trend. The OQ-score decreased in both conditions, but at a higher rate in the TAU-PCOMS condition, mainly in the first three months. After 1 year, patients in the TAU condition showed a 20.16 point improvement on the OQ-45, whereas patients in the TAU-PCOMS condition showed an improvement of 25.70 points, a difference of 5.54 points or 27.5%. The majority of this additional improvement (76.4%; 4.23 out of 5.54 points) was achieved in the first three months. The interaction between log10 linear time and condition is significant (B = −2.160, SE = 0.823, p = 0.009, 95% CI [−3.773, −0.547]).

Effect on wellbeing on the MHC-SF. In the PP-analysis, a similar significant interaction effect between log10linear time and condition was found using the MHC-SF as the dependent variable, showing more improvement in the TAU-PCOMS condition. The results of the three-level model can be found in Table III.

After 1 year, in the PP-analysis, patients in the TAU condition show a 0.66 point improvement on the MHC-SF and patients in the TAU-PCOMS condition show an improvement of 0.93 points, a difference of 0.27 points or 40.2%. The majority of this additional improvement (76.4%; 0.20 out of 0.27 points) was achieved in the first three months. The effect is significant (B = 0.104, SE = 0.039, p = 0.008, 95% CI [0.028, 0.180]).

Effect on patient satisfaction, costs and duration. No effect of adding PCOMS to TAU was found on patient satisfaction as measured by the CQI (B = −0.282, SE = 0.303, p = 0.351, 95% CI [−0.876, 0.311]). No effects of adding PCOMS were found between the two conditions on costs (B = 48.359, SE = 55.485, p = 0.383, 95% CI [−60.389, 157.106]) and duration (B = 40.341, SE = 24.808, p = 0.104, 95% CI [−8.283, 88.965]).

In summary, the results found in the PP analysis are similar to those of the ITT analysis.

Discussion

In this study, we examined the effect of systematic client feedback in outpatient psychological treatment. Adding feedback to TAU was effective and led to steeper growth curves in treatment outcome on psychiatric symptoms and wellbeing. This highly significant effect was found in both the ITT-analysis and the PP-analysis, despite the fact that the sample size of the PP-analysis is about half the size of the ITT analysis. The effects on drop-out, costs, and duration were not significant.

Recent meta-analyses have suggested that the positive effect of feedback may be attributable to researcher allegiance or lack of independent outcome measures and can only be found in counseling settings (Lambert et al., 2018; Østergård et al., 2020). The current study has taken these methodological limitations of previous studies into account, and still found a positive effect of systematic feedback. Moreover, it was not conducted in a counseling setting but in a psychiatric setting treating patients with mild to moderate psychological disorders. Adding systematic client feedback to TAU led to a significant decrease of psychiatric symptoms and a significant increase in wellbeing for both independent outcome measures. This effect was found in the ITT as well as in the PP-analysis.

A significant difference was found on both outcome measures. The mean decrease after one year on the OQ-45 is well above the reliable change (RC) of 14 points in both conditions, with an additional decrease of 5.25 points in the TAU-PCOMS condition. The mean decrease on the MHC-SF in the ITT analysis was below the RC in both conditions (TAU = 0.66; TAU-PCOMS = 0.82). In the PP analysis the decrease in the TAU condition was below RC (0.66), but in the TAU-PCOMS condition a decrease was found above the reliable change (0.93).

The added value of PCOMS is found mainly at the beginning of therapy, 76.4% (OQ-45) and 76.2% (MHC-SF) of the improvement was achieved in the first three months. This is consistent with the research of Horvath and Symonds (1991) who found that a good working relationship at the earliest sessions was the best predictor of outcome. It is possible that PCOMS contributes to this objective, as suggested by Brattland et al. (2019).

The adherence rate (26%) of adding PCOMS to TAU was low, with a large difference between the two centers participating in this condition. One center added PCOMS to TAU in 184 of the 349 patients included, an adherence rate of 52.7%. The adherence rate in the other center was only 5.0%, with only 22 of 443 included patients received TAU with PCOMS added. This difference is highly significant, χ² (1, N = 792) = 231.34, p < .001, V = 0.54.

The large difference between the centers in adherence rate may be explained using the distinction made by Schneider et al. (2013) between organizational culture (basic values) and organizational climate (daily experiences). The same culture prevailed in both centers at the beginning of the study; therapists had included the use of PCOMS in their basic values and using PCOMS was part of daily practice. At the center with the low adherence, however, the organizational climate changed significantly after the first months of the study.

In this center, three out of eight therapists (37.5%) dropped out due to long-term absenteeism, while in the other center no therapists were on long term sick leave. At the same time, there was a large increase in referrals. The workload thus increased and the number of available practitioners vastly decreased. In the monthly supervision meetings the therapists stated that they tried to meet the minimum quality standards of treatment, but often failed to add PCOMS to TAU. By the time the long term absenteeism had diminished, the inclusion period of this study was nearing completion. Differences between therapists in the use of PCOMS will be explored in more detail in a separate article, as indicated in the study protocol (Bovendeerd et al., 2019). The large difference at the center level is reflected in the three-level multilevel analysis in which most of the variance is found on the center level and the therapist effect is small.

No effect was found on duration and costs in this outpatient setting. In this setting, the maximum reimbursement, as well as the length for a therapy episode is fixed. All therapy sessions that exceed the maximum reimbursement or overpass the time-limited period of one year, are not financially compensated. At the start of treatment, three treatment duration variants can be chosen; short, medium, or long, and the chosen variant is generally completed. This may have led to less variation in the number of sessions, as well as less variation in costs. Two similar studies have been conducted in the Netherlands by Janse et al. in a treatment setting where the number of sessions is less fixed (Janse et al., 2016, 2020). They examined the effect of adding PCOMS to cognitive behavioral therapy and found an effect on treatment duration, but, apart from some subgroups of patients, not on treatment outcome. It can be hypothesized that adding PCOMS either leads to improved outcome or reduced treatment duration, as it is unlikely that adding PCOMS would lead to both simultaneously.

No effect was found on client satisfaction. This may be due to a ceiling effect, client satisfaction was already very high in the TAU group (M = 8.57 on a range of 0–10).

To what extent can these results be applied to psychotherapy in general? As stated by Brattland et al. (2018) and also reflected in our results, implementation of systematic feedback requires careful planning to create a helpful culture and climate. An added effect is not always found and the effects found vary considerably, varying from shortening of treatment duration (Janse et al., 2016, 2020), or reduction of drop-out (Janssen et al., 2021).

This study found an improvement in treatment outcomes, especially in the first three months. It would be interesting to examine in further research whether significant differences would still be found if systematic feedback were reduced or discontinued after this initial phase. How can we use SCF as efficiently as possible and keep the burden on the therapists as low as possible at the same time? It may be sufficient to get systematic feedback in this first phase only and to continue at a lower frequency thereafter. If it turns out that the working relationship is good, it might even be stopped permanently. Perhaps the more relevant research question is when we should use feedback, rather than if.

Strengths, limitations and future directions. The naturalistic design in this study provides insight into how adding systematic feedback to TAU can provide in daily practice. However, the findings also have their limitations. The relatively small number of participating centers leads to limited generalizability, and the cluster randomization of a small number of centers may result in systematic differences between the centers. Furthermore, although no significant differences were found between the participating centers, or compared to the rest of the organization, it is possible that differences exist in unmeasured variables, that may post threats to the internal validity.

While the inclusion within these centers was high – 69% of the eligible patients were analyzed – it is possible that there is a selection bias, with certain patients not entering the study, which may limit the generalizability of our results. In particular in the PP-analysis, a relatively large number of patients were excluded, partly due to the strict inclusion criteria and to the changed organizational climate in one of the centers. However, the majority of those excluded patients only used PCOMS once or twice, and including them would have potentially diluted the effect of the feedback.

Strengths of the study are the lack of researcher allegiance, its naturalistic setting, the thorough training of the full staff, and the use of independent outcome measures.

In future research, a cost–benefit analysis of adding systematic feedback would be relevant. Adding systematic feedback to therapy can have an enhancing effect but not on all patients or subgroups, and implementation can be challenging. It will, therefore, be helpful to know when to add systematic feedback, to optimize the balance between costs and benefits. The effect of therapist factors and patients perspective on feedback could also be taken into account.

Conclusion

This study provides evidence that with the relatively small effort of adding systematic feedback, the outcome of therapy can be substantially improved. Adding systematic client feedback to treatment as usual probably has a beneficial impact in outpatient psychological treatment. Further research should focus on implementation and profiling those patients who will most likely benefit from systematic feedback.

Availability of data and materials

The data that support the findings of this study are available from the corresponding author, BB, upon reasonable request.

Acknowledgements

The authors wish to thank the therapists from the participating centers of Mindfit for their participation in the study, the patient advisory board of the Dimence Group for their advice, Alfred Kaal and Esther Scherjon for their data-file support, Sjoerd Colijn and Anton Hafkenscheid for their substantial contribution to the development of this study and numerous colleagues of the Dimence Group for their additional help. BB wrote the study proposal, the manuscript and led the research project. BB, EdG and JdK developed the study design and coordinated the data acquisition. JdK and KdJ supervised the research project. MM, BB and KdJ developed the data-analysis. All authors provided comments on manuscript drafts and approved the final manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Research-foundation of Postmaster Psychologie Opleidingen (PPO) the Netherlands [grant number PPO-RF-18CM]; The Dimence Groep, the Netherlands [grant number CWOBB112014]; The University of Groningen.