The effects of motivation feedback in patients with severe mental illness: a cluster randomized controlled trial

Abstract

Objective

To evaluate the effectiveness of providing clinicians with regular feedback on the patient’s motivation for treatment in increasing treatment engagement in patients with severe mental illness.

Methods

Design: cluster randomized controlled trial (Dutch Trials Registry NTR2968). Participants: adult outpatients with a primary diagnosis of a psychotic disorder or a personality disorder and their clinicians, treated in 12 community mental health teams (the clusters) of two mental health institutions in the Netherlands. Interventions: monthly motivation feedback (MF) generated by clinicians additional to treatment as usual (TAU) and TAU by the community mental health teams. Primary outcome: treatment engagement at patient level, assessed at 12 months by clinicians. Randomization: teams were allocated to MF or TAU by a computerized randomization program that randomized each team to a single treatment by blocks of varying size. All participants within these teams received similar treatment. Clinicians and patients were not blind to treatment allocation at the 12-month assessment.

Results

The 294 randomized patients (148 MF, 146 TAU) and 57 clinicians (29 MF, 28 TAU) of 12 teams (6 MF, 6 TAU) were analyzed according to the intention-to-treat principle. No statistically significant differences between treatment groups on treatment engagement were found (adjusted mean difference =0.1, 95% confidence interval =−2.2 to 2.3, P=0.96, d=0). Preplanned ancillary analyses showed statistically significant interaction effects between treatment group and primary diagnosis on treatment motivation and quality of life (secondary outcomes), which were beneficial for patients with a primary diagnosis of a personality disorder but not for those with a psychotic disorder. There were no reports of adverse events.

Conclusion

The current findings imply that monitoring and discussing the patient’s motivation is insufficient to improve motivation and treatment engagement, and suggests that more elaborate interventions for severe mental illness patients are needed.

Keywords:

• randomized controlled trial
• feedback
• motivation
• adherence
• psychotic disorders
• personality disorders

Introduction

Background

A common consideration in clinical practice is that evaluation of the patient’s motivation may help understand how a patient may best be engaged in treatment.^1–5 Patients with severe mental illness (SMI), such as those with psychotic disorders or severe personality disorders, are often considered not motivated to seek treatment⁶ or fail to adhere to treatment programs.^7,8 Regular assessment of motivation for engaging in treatment and providing this as feedback to the clinician might be a promising approach to both monitor the patient’s motivation and provide a useful structure in the communication about it. Such communication may help to improve motivation for treatment and treatment engagement.⁹ Meta-analyses have shown beneficial effects of employing feedback to clinicians on their patients’ mental health outcomes.^10–12 However, most clinician feedback research has focused primarily upon treatment outcomes^13–16 and was unable to determine which specific elements from the feedback provided the mechanism(s) of action. Motivation for treatment has been used as part of such feedback systems, yet to our knowledge and based on an extensive review,¹⁰ no previous study has investigated the effects of providing feedback that is exclusively based on the SMI patient’s motivation for treatment. This warrants the current investigation.

The theoretical basis of the motivation feedback (MF) intervention was founded on Self-determination Theory (SDT); a theory of motivation that defines several types of motivation that fall along a continuum of self-determination and describes how social and cultural factors can facilitate or undermine people’s sense of self-determination.^17,18 SDT states that the most externally determined form of treatment motivation is when a patient remains in treatment because he feels pressured to do so.¹⁸ This external motivation could, for example, be present in a patient who is court-ordered into treatment.¹⁹ Also, relatively external yet somewhat more autonomous is introjected motivation, where a patient is driven by feelings of guilt or shame. A patient with introjected motivation might act to avoid disapproval or guilt or receive approval or praise (eg, from the mental health worker or important others). More autonomous motivation is present in a patient with identified motivation, who recognizes and accepts that treatment is useful for achieving personally relevant goals.¹⁸ An example is a patient who finds it important to take medications as a way of preventing relapse. According to SDT, engaging in treatment for a long time requires that patients internalize treatment values since behaviors that are more autonomous (ie, more self-determined) are more likely to be performed again, whereas behaviors that are primarily driven by external motives will only be performed in the presence of such perceived external pressures.^18,20

Objectives and hypothesis

The current study aimed at evaluating the effectiveness of MF intervention compared to treatment as usual (TAU) in outpatients with SMI treated by community mental health teams. It was hypothesized that SDT-based MF would lead to increased treatment engagement (primary outcome), and to a beneficial shift in the SDT motivation continuum toward more autonomous motivation and improved psychosocial functioning and quality of life (secondary outcomes) in outpatients with SMI.

Methods

Trial design and ethics statement

This study was a two-center cluster randomized trial comparing MF and TAU. Cluster randomization was chosen to avoid contamination bias.²¹ The full trial protocol is available elsewhere.⁹ Briefly, the cluster randomized controlled trial was designed with the primary objective of determining the effectiveness of MF intervention on treatment engagement (primary outcome) of outpatients with psychotic disorders and personality disorders, compared to TAU in community mental health care teams (the clusters). Secondary outcomes include treatment motivation, psychosocial functioning, and quality of life. The specific aspects of the broader trial design, including details of settings, interventions, randomization, and blinding, are also addressed. Discrepancies between the original protocol and the current report are described under “Methods” and “Results” sections; these included the handling of skewed outcomes in the statistical analyses and the smaller sample size due to lower than expected recruitment rate.

This study was approved by the Medical Ethical Committee for Mental Health Care Institutions (MotivaTe-IT; trial number NTR2968, Netherlands Trial Register, http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=2968) as well as by the scientific committees of the Western North Brabant Mental Health Center and Breburg Mental Health Center, the specialty mental health institutions where the data were collected. The authors confirm that all ongoing and related trials for this intervention are registered. All procedures were conducted according to the principles expressed in the Declaration of Helsinki, including the obtainment of written informed consent by the participants. Results are presented in accordance with the CONSORT statement for cluster randomized controlled trials.²²

Setting and participants

This study was initiated by GGZ Westelijk Noord Brabant, a specialty mental health center located in a semiurban area in the south-west of the Netherlands. GGz Breburg, another specialty mental health center in the same semiurban area, was also approached and agreed to join the study.

Within these two centers, patients were eligible for participation if they had a primary diagnosis of a psychotic disorder or a personality disorder (as diagnosed by the psychiatrist of the team using the Diagnostic and Statistical Manual of Mental Disorders-Text Revision [fourth edition] criteria and obtained from the medical record), were aged between 18 and 65 years, and received individual outpatient treatment for their mental disorder. Exclusion criteria were an insufficient command of the Dutch language (which was estimated by the clinician who was most frequently involved with the patient), and a documented diagnosis of dementia or chronic toxic encephalopathy.

Treatment teams were eligible for participation if they provided outpatient assertive community mental health care to eligible patients. At the start of the study, 12 teams of the two mental health centers fulfilled this criterion and were approached for participation. Specifically, these teams included a forensic psychiatric outpatient clinic, three specialized psychotic outpatient treatment programs and eight function-assertive community treatment teams (FACT-teams²³). FACT-teams provide assertive, outreaching, community-based, and supportive psychiatric services to individuals with SMI.²³ Clinicians within the approached teams were eligible for participation if they were the primary health-care practitioner involved with the patient, meaning that this clinician had the most frequent contact with the patient. All 12 teams agreed to participate in the study and as such, the trial was conducted within these teams between May 2011 and October 2013.

Interventions: TAU and MF

TAU

In the TAU group (consisting of six teams or clusters), treatment was provided by multidisciplinary assertive outreaching community mental health teams. TAU was guided by the patient’s individual symptoms and needs for care and could include assertive outreach, medication, social and financial management, job counseling, crisis interventions, cognitive (behavioral) therapy, the strengths-based approach, and/or supportive structured therapy.²³ Individual case management was offered to patients who were more stable and needed long-term care, but intensive assertive outreach was offered to patients at risk of relapse or neglect, often by several clinicians working with a shared caseload.²³ We did not seek for uniformity in TAU as such diversity reflects reality. Clinicians in the control group continued TAU during the course of the study.

MF

MF was provided in addition to TAU in patients randomized to the MF group (consisting of six teams or clusters). Patients and clinicians in the MF intervention group were asked to fill in a Short Motivation Feedback List (SMFL) every month up to 12 months after baseline assessment. The SMFL consists of eight statements that relate to the level and type of the patient’s treatment motivation, based on three types of motivation postulated by SDT:^18,20 external, introjected, and identified motivation. The SMFL was shown to be reliable for these three types of motivation; congeneric estimates of reliability ranged from 0.81 to 0.93.^24,25

Before commencing the study, clinicians were trained by the principal investigator (PI) in the principles of SDT and the use of MF. The training consisted of a presentation about the principles and concepts of SDT, exercises to learn how to distinguish the needs for autonomy, competence and relatedness, and practicing MF assessments with other clinicians (not yet with patients) during this training, to familiarize themselves with the feedback and how to communicate it. Clinicians received three booster sessions over the course of the study to evaluate and discuss their progress and experiences together with other colleagues who also participated in MF. During the course of the study, the PI received filled-out SMFLs from the clinicians and subsequently provided the clinician with MF graphs via email. An example of such a graph can be found in our published research protocol.⁹ The evaluation of the SMFL and the graph could serve as a starting point for conversations between the patient and the clinician regarding the motivation of the patient. Clinicians were instructed to stimulate internalization of motivation by supporting the patient’s basic psychological needs of autonomy, competence, and relatedness, in line with SDT.²⁶ The intention was that the conversation would revolve around sources of motivation behind treatment goals. Clinicians were free to decide for themselves how they would structure this conversation with the patient, such as discussing only one item or several, or discuss differences between patient’s and clinician’s vision, and they were free to decide how long this would take. The duration and frequency of SMFL assessments were monitored by the research team. Both the number of face-to-face contacts between patient and clinician and the number of performed SMFL assessments were counted to evaluate how many of the possible SMFL assessments were actually performed.

During the course of the study, clinicians were regularly contacted by the PI to monitor the MF intervention and to discuss progress and experiences together with other colleagues who also participated in the MF intervention. These evaluation sessions took place four times over the course of the study. To aid clinicians in remembering to perform SMFL assessments, they were given bookmarks to use in their paper planners, posters of the study were hung up in the team offices, electronic reminders were regularly placed in the electronic planners, the PI was regularly present in the team office to check up on progress, and emails were sent to remind the clinicians of using MF.

Outcomes

The outcomes of interest were treatment engagement (primary outcome) and treatment motivation, psychosocial functioning, and quality of life (secondary outcomes).⁹ We also administered a number of other comprehensive instruments, including measures for baseline characteristics used in the current study, for which we refer to our research protocol.⁹ All instruments were administered in Dutch language.

Primary outcomes

Treatment engagement was measured with the Service Engagement Scale (SES) that was filled out by clinicians. The SES was developed to measure engagement with community mental health services.²⁷ It comprises 14 items that assess availability, collaboration, help seeking, and treatment engagement behaviors (including medication adherence). The items are rated on a 4-point scale ranging from 0 (not at all) to 3 (most of the time). The SES has shown good psychometric properties and has previously been used in studies with patients with psychotic disorders.^27–29 The SES total scale score was used as the outcome measure in this study, where higher scores denote higher treatment engagement. Reliability of the total scale score in the current sample was considered good, as evaluated by a congeneric estimate of reliability of 0.91. Additionally, we included the number of missed appointments (no-shows), as a more objective measure of treatment engagement.⁹ These were obtained from the medical records.

For patients with a primary diagnosis of a psychotic disorder, the Morisky Medication Adherence Scale³⁰ was used to assess patient self-reported antipsychotic medication adherence. The Morisky Medication Adherence Scale is a self-report scale that consists of eight items asking about a specific medication-taking behavior, such as “When you feel that your symptoms are under control, do you sometimes stop taking your medicine?”. The items can be scored “yes” or “no” and the total scale score theoretically ranges from 0 to 8, with higher scores indicating better medication adherence. The congeneric estimate of reliability was 0.82 in the current sample.

Secondary outcomes

Motivation for engaging in treatment as postulated by SDT was measured with the Treatment Entry Questionnaire (TEQ)^24,31 that was administered to both patients and clinicians. It contains three subscales (external, introjected, and identified motivation), each with six items rated on a scale from 1 (strongly disagree) to 7 (strongly agree), and subscale scores are computed by averaging the item scores and multiplying this by the number of items. The congeneric estimates of reliability for TEQ subscales were acceptable in the current study sample as evaluated by congeneric estimates of reliability; 0.78 for identified motivation, 0.72 for introjected motivation, and 0.75 for external motivation.²⁴ Construct validity for the TEQ was supported by significant associations with therapist-rated service engagement (correlations between −0.15 and 0.58 (P<0.01), depending on the subscale), patient- and clinician-rated therapeutic alliance (eg, r=0.47 and r=0.25, P<0.01, respectively with identified motivation) and legally mandated treatment.²⁴ Higher scale scores denote higher levels of that type of motivation.

The patient’s psychosocial functioning was measured with the Dutch version of the Health of the Nations Outcome Scales (HoNOS).³² The HoNOS was administered as a semistructured interview with the patient, performed by independent research assistants (mostly graduate students in psychology and medicine). The researchers had no involvement in the patient’s treatment. Patients were interviewed at the team office or at home, depending on their preference. The HoNOS quantifies health and social problems of the previous 2 weeks and contains 12 items that refer to behavioral problems, cognitive and physical impairments, symptoms, and social (dis) functioning. HoNOS items are scored on a scale from 0 (no problem) to 4 (severe problem). The total scale score is computed by adding the 12 items. A higher total score on the HoNOS denotes more severely impaired psychosocial functioning. The psychometric properties of the total scale score were shown to be acceptable and sensitive to change.³² Reliability of the total scale score was adequate in the current sample, as reflected by a congeneric estimate of reliability of 0.77.

The patient’s quality of life was assessed with the Manchester Short Assessment of Quality of Life (MANSA).^33,34 The MANSA is a self-report questionnaire that asks the patient how satisfied he/she is in the following life domains: living situation, social relationships, physical health, mental health, safety, financial situation, work situation, and life as a whole. The mean score on the 12 MANSA items was used as the outcome measure, of which the psychometric properties are considered satisfactory.³⁴ The congeneric estimate of reliability was 0.91 for the MANSA total score in the current sample. Higher scores denote a higher perceived quality of life.

Sample size

The sample size was calculated on the basis of our primary hypothesis that MF would be more effective than TAU in enhancing treatment engagement, as measured with the SES at 12 months after baseline assessment. The difference between the MF group and control group for the primary outcome was based on a power of 0.80, an alpha of 0.05 (two-tailed), and an effect size (standardized mean difference) of 0.40.⁹ The clustering of patients within clinicians was accounted for using the variance inflation factor formula f =1+ (m−1) ρ, with an estimated cluster size (m) of six patients per clinician and the within-cluster correlation (ρ) was estimated from a previous study to be around 0.07.³⁵ Using these parameters and including an additional correction for expected loss-to-follow-up, it was estimated that the required total sample size should be 350 patients.⁹

Randomization

Sequence generation

A computer-generated list of random numbers was used to randomly assign each team to a treatment condition, such that all clinicians and patients in the same team were randomized to a similar treatment.

Type

The randomization sequence was created using software from www.randomization.org with a 1:1 allocation ratio using random block sizes of 1, 2, and 3.

Implementation

The random allocation sequence was performed by authors ECJ and HJD prior to approaching treatment teams, such that treatment teams and their members were still unknown and were numbered blindly before entering team numbers into the computer program.

Blinding

At baseline, patients were unaware (blind) as to which treatment condition they had been randomized to. Clinicians had to be made aware of treatment condition as those randomized to MF needed to receive the necessary training prior to baseline assessments such that MF could start immediately thereafter. This blinding procedure is common in psychiatric intervention research.²¹ At the 12-month assessment, clinicians and patients were not blind to treatment condition while filling in questionnaires, whereas independent research assistants who looked up information from the medical record and performed interviews with patients were blind to treatment allocation.

Procedures

Treatment teams were approached by the PI and clinicians working in these teams received oral and written information about the study and were asked for informed consent. Subsequently, clinicians were asked to provide their caseload to the PI, who randomly selected ten eligible patients for participation (or if fewer than ten eligible patients were available, all the eligible patients were selected). Clinicians explained to the selected patients the contents and procedure of the study and asked for participation. To enhance the likelihood of participation, patients were given an incentive of €15 for participating. If a patient consented to participate, an appointment was made with the PI, sometimes accompanied by the clinician for the patient’s comfort and/or the investigator’s safety. The patient received oral and written information about the contents and procedures of the study once more before signing informed consent. Subsequently, patients and clinicians completed the baseline assessments. Independent research assistants accompanied patients during the assessment, such that they could help if necessary. This could, for example, include reading items aloud to accommodate patients with concentration problems and/or explaining items that were not readily understood. This procedure took about 2 hours for most patients and about 20 minutes for clinicians.

After 12 months, patients and clinicians were contacted for the follow-up assessment. Patients who had ended treatment or dropped out from treatment were nonetheless contacted for a follow-up assessment. Clinicians were asked to complete their follow-up assessment for all patients who were enrolled at start of the study. Furthermore, it was assessed to what extent the MF intervention was performed by the clinicians.

Statistical methods

Several outcomes, including the primary outcome and the motivation questionnaires, were not normally distributed at follow-up assessment and transformations were not successful. To deal with this, difference scores were calculated as follow-up assessment minus baseline assessment. The difference scores showed normal distributions for all outcomes, as evaluated by histograms and normal probability plots. Subsequently, they were used as outcomes in this study.

Differences in demographic and clinical variables at baseline between the intervention group and control group, and between participants and nonparticipants, were evaluated with independent samples t-tests and chi-square tests. All analyses were conducted using a significance level of P<0.05 (two-sided) and unstandardized estimates of regression coefficients (β), 95% confidence intervals (95% CI), interquartile ranges, intraclass correlation coefficients, and standardized mean differences are reported where appropriate. Statistical tests were performed using IBM SPSS Statistics 21 (IBM Corporation, Armonk, NY, USA) and SAS version 9.3 (SAS Institute Inc., Cary, NC, USA).

Intention-to-treat analyses

All outcomes were analyzed with multilevel linear regression models. As stated in the study protocol,⁹ it was explored which and how many levels would be appropriate for inclusion in the multilevel analyses. Not all available levels (ie, mental health institutions, teams, clinicians, patients, and measurements) could simultaneously be included as random effects due to singularity problems. Considering the variances explained at each level (on average: 81% at patient-level, 8% at clinician-level, 1% at team-level, and 0% at mental health institution) and considering that the dispersion of patients over teams was larger than the dispersion over clinicians (12–38 patients per team versus 1–10 patients per clinician), it was decided to include “team” as the second level in the analyses.

Furthermore, all analyses were performed both unadjusted and adjusted for baseline imbalances between treatment groups. In unadjusted analyses, models included treatment as fixed effect and clustering at team-level as a random effect. In adjusted analyses, models included treatment and a multivariate confounder score as fixed effect, and clustering at team-level as random effect. The multivariate confounders score was calculated using a set of observed potential confounders to control for the observed differences in the distribution of baseline variables between treatment groups.³⁶ The multivariate confounder score included ethnicity, sex, educational level, primary diagnosis, addiction problems, the clinician’s years of clinical working experience, and the baseline value of the respective outcome (eg, the baseline score on the SES was added to the confounder score in the analyses for SES at follow-up). Further, for all models, missing data on baseline variables were not imputed; only all observed data were used. Missing data on outcomes were considered missing at random. Restricted maximum likelihood was used as the estimation method.

Additional analyses

Per protocol analyses

As stated in the study protocol,⁹ we wanted to investigate the effect of actual exposure to the intervention on outcomes. To this end, a per protocol analyses was performed in which a median split was performed on the number of SMFL assessments, such that patients who performed MF less than four times were removed from analyses despite their randomization to MF. The modeling approach of these per protocol analyses was similar to the intention-to-treat analyses.

Test of interaction effects: the role of primary diagnosis and age

As stated in the study protocol,⁹ we were also interested in determining whether treatment effects were dependent on baseline characteristics of the sample. To limit the number of tests (and accompanying problems of multiple testing), it was decided to test for differences between patients with psychotic disorders and patients with personality disorders and to test for potential differential effects of age. The two diagnostic groups constitute the great majority of patients treated in assertive outreach teams in the Netherlands;²³ but previous studies have largely focused on patients with psychotic disorders and ignored the experiences of service users with personality disorders in motivational interventions and/or outcome feedback systems.^15,37,38 Therefore, exploratory analyses were performed to detect whether the effects of treatment on all outcomes were modified by the primary diagnosis. Additionally, as previous studies in community mental health care for patients with SMI in the Netherlands have shown that treatment outcomes such as psychosocial functioning were dependent on patient age,³⁹ it was decided to explore whether the effects of treatment on all outcomes were modified by age. We tested these possible interactions (treatment group by primary diagnosis and treatment group by age) for significance on all outcomes.

Results

Participant flow, recruitment, and numbers analyzed

The numbers of participants who were randomly assigned, received intended treatment, were lost to follow-up, and the total numbers analyzed are shown in the flowchart (Figure 1). Ultimately, a total of 57 clinicians and 294 eligible patients signed the written informed consent and completed baseline assessments between May 2011 and September 2012. The recruitment process was slower and more difficult than expected, and despite extending the inclusion period by 4 months, the inclusion of patients did not reach the estimated necessary 350 patients.

Figure 1 Flow diagram of MotivaTe-IT.

Abbreviations: IQR, interquartile range; min, observed minimum; max, observed maximum.

In total, 58% of the eligible patients (294 out of 507) and 80% of clinicians (57 out of 71) actually agreed to participate. Of the 155 patients who declined participation in the trial, 53 patients (34%) did not feel capable of filling in the questionnaires. For example, they found it too long or too much (n=18) or they felt they were too ill or incapable at the moment (n=12). Another 51 patients (33%) were not interested or did not feel the need to participate, and 16 patients (10%) said they did not want to have anything to do with mental health affiliations. Another 18 patients (12%) did not see the use of scientific research in general, nine patients (6%) started out with the baseline assessment but quit before completing, and eight patients (5%) did not give a reason for declining participation. Additionally, 58 patients could not be contacted despite several attempts. Patients who declined participation were significantly more often those with a primary diagnosis of a psychotic disorder and less often those with a personality disorder (χ² [1, N=470] =8.70, P<0.01).

At 12 months, 253 patients (86%) were reassessed. Numbers lost to follow-up were not significantly different between intervention groups. The group that was lost to follow-up was significantly more often of non-Dutch ethnicity (48% versus 26%, P<0.01) and more often had a legal mandate for treatment (18% versus 7%, P=0.03) compared to completers. Clinicians completed their follow-up assessments for 278 patients (95%).

Baseline characteristics

The baseline sociodemographic and clinical characteristics of participating patients are shown in Table 1. At baseline, several patient characteristics were unequally distributed over the two treatment groups (Table 1). Clinicians in the MF group had an average of 5 more years of working experience (20 versus 15 years, P<0.01).

Table 1 Baseline characteristics of participating patients

	Motivation feedback (N=148)	Treatment as usual (N=146)
Age, years, mean (SD)	45.47 (10.4)*	42.50 (10.0)*
Male sex, n (%)	98 (66.2)	81 (55.5)
Dutch ethnicitya, n (%)	116 (78.4)**	92 (63.0)**
Education level, n (%)
No education/elementary	57 (38.5)	51 (34.9)
Secondary school	57 (38.5)	67 (45.9)
≥Upper high school	32 (21.6)	27 (18.5)
Living situation, n (%)
Alone	88 (59.5)**	59 (40.4)**
With partner and/or children	49 (33.1)*	70 (47.9)*
Mental health center facility	10 (6.8)	16 (11.0)
Homeless	1 (0.7)	1 (0.7)
Primary diagnosis, n (%)
Psychotic disorder	104 (70.2)	95 (65.1)
Personality disorder	44 (29.7)	51 (34.9)
Comorbid substance use problemsb, n (% yes)	42 (28.4)	32 (21.9)
Prescribed medication, n (%)
Classical antipsychotics	37 (25.0)	26 (17.8)
Atypical antipsychotics	63 (42.6)	67 (45.9)
Combination of typical and atypical	12 (8.1)	15 (10.3)
Benzodiazepines	42 (28.4)	39 (26.7)
Antidepressants	40 (27.0)	53 (36.3)
Age of first contact with mental health, mean (SD)	27.16 (10.34)	24.95 (10.24)
One or more previous hospitalizations, n (% yes)	113 (76.4)	114 (78.1)
Legal mandate, n (% yes)	11 (7.4)	13 (8.9)

Notes:

a The definition of Dutch ethnicity was based on the definition by the Dutch Bureau of Statistics.⁵⁷

b Substance abuse problem was defined as having a DSM-IV diagnosis of substance abuse and/or dependence in the medical record.

* P<0.05,

** P<0.01.

Abbreviations: SD, standard deviation; DSM-IV, Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition.

Adherence to MF intervention

On average, four assessments with the SMFL were done per patient (standard deviation =3, observed range =0–11), representing 45% of the possible SMFL assessments that could have been performed considering the frequency of contacts with patients. Clinicians reported that the median time of discussing the SMFL with the patient was 10 minutes (interquartile ranges =5–15 minutes). Eighteen out of 148 patients (12%) never completed any SMFL assessments (reasons are shown in Figure 1). All clinicians in MF had at least one patient who was actively involved in MF, so there was no clinician who never performed SMFL assessments.

Intention-to-treat analyses of outcomes

Table 2 shows pre- and postintervention medians and the results of intention-to-treat analyses for all outcome measures, both unadjusted and adjusted for the multivariate confounder score. In the following, we will describe the results of the adjusted analyses, which are similar to the results of unadjusted analyses in terms of interpretation.

Table 2 Effects of motivation feedback on outcomes in the total patient sample (intention-to-treat analyses)

Outcomea	Treatment group	T0 (baseline) Median (IQR)	T12 (posttest) Median (IQR)	Mean difference between treatment groupsb (95% CI)	P-valueb	Effect size	Adjusted mean difference between treatment groupsc (95% CI)	P-valuec	Effect size	ICC
Treatment engagement
Clinician-reported engagement (SES; min =0, max =42)	MF	31 (25–37)	34 (30–37)	−0.0 (−2.2 to 2.2)	0.99	0	0.1 (−2.2 to 2.3)	0.96	0	0.05
	TAU	31 (24–36)	36 (30–38)
Missed appointments (number of no-shows)	MF	0 (0–2)	0 (0–1)	−0.1 (−0.8 to 0.6)	0.83	0.01	0.1 (−0.7 to 0.8)	0.89	0	0.05
	TAU	0 (0–0)	0 (0–0)
Patient-reported motivation
Identified motivation (TEQ; min =6, max =42)	MF	36 (29–40)	33 (30–37)	−1.4 (−3.1 to 0.3)	0.10	−0.10	−0.8 (−2.7 to 1.0)	0.37	−0.03	0
	TAU	36 (30–39)	36 (29–39)
Introjected motivation (TEQ; min =6, max =42)	MF	21 (14–28)	20 (14–26)	0.1 (−1.8 to 2.1)	0.90	0.01	−0.8 (−2.9 to 1.4)	0.49	−0.03	0
	TAU	24 (18–31)	22 (17–28)
External motivation (TEQ; min =6, max =42)	MF	15 (11–21)	12 (8–18)	1.7 (−0.2 to 3.6)	0.07	−0.11	1.0 (−1.1 to 3.1)	0.34	0.04	0
	TAU	18 (12–24)	12 (9–20)
Clinician-reported motivation
Identified motivation (TEQ; min =6, max =42)	MF	30 (26–33)	29 (25–34)	−0.4 (−2.1 to 1.3)	0.66	−0.03	−0.6 (−2.6 to 1.3)	0.53	−0.03	0.01
	TAU	32 (27–35)	32 (25–35)
Introjected motivation (TEQ; min =6, max =42)	MF	20 (15–25)	17 (13–21)	−3.0 (−4.7 to −1.3)	<0.01	−0.20	−4.5 (−6.4 to −2.6)	<0.001	−0.18	0
	TAU	22 (16–28)	21 (17–27)
External motivation (TEQ; min =6, max =42)	MF	20 (14–26)	17 (12–24)	-2.3 (−5.0 to 0.4)	0.09	−0.11	−2.0 (−4.9 to 0.9)	0.17	−0.07	0.09
	TAU	21 (14–27)	21 (15–27)
Psychosocial functioning	MF	8 (6–13)	10 (6–14)	1.1 (−0.9 to 3.1)	0.27	0.07	0.6 (−1.5 to 2.6)	0.60	0.02	0.04
(HoNOS; min =0, max =48)	TAU	9 (6–14)	10 (7–15)
Quality of life	MF	5 (4–5)	5 (4–5)	0.0 (−0.3 to 0.3)	0.91	0	0.0 (−0.3 to 0.3)	0.85	0	0.04
(MANSA; min =1, max =7)	TAU	5 (4–5)	5 (4–5)

Notes: Effect size was calculated as the standardized mean difference between intervention groups.

a The theoretically possible range of scores is reported next to the abbreviations of scale names, indicated by min and max.

b Results represent the effects of treatment allocation (MF relative to TAU), adjusted for clustering at team-level.

c Results represent the effects of treatment allocation (MF relative to TAU), adjusted for clustering at team-level and a multivariate confounder score (which included patient sex, ethnicity, educational level, comorbid addiction problems, years of working experience of the clinician, and the baseline value of the outcome).

Abbreviations: MF, motivation feedback; TAU, treatment as usual; SES, Service Engagement Scale; TEQ, Treatment Entry Questionnaire; HoNOS, Health of the Nation Outcome Scales; MANSA, Manchester Short Assessment for quality of life; IQR, interquartile range; ICC, intracluster correlation coefficient at the level of treatment programs (teams); CI, confidence interval; min, minimum score; max, maximum score.

It can be seen from Table 2 that we found no statistically significant differences between the MF and TAU groups in terms of treatment engagement, neither as measured with the SES nor as measured by the number of no-shows. Neither did we find significant differences between treatment groups in patients with primarily psychotic disorders, regarding their self-reported medication adherence. Regarding motivation for treatment, no statistically significant treatment effects were found for patient-reported motivation, but clinicians reported that MF reduced patients’ introjected motives for engaging in treatment more than TAU (adjusted mean difference [AMD] =−4.5, 95% CI =−6.4 to −2.6, P<0.001). Neither any of the other motivation scales nor the patient’s psychosocial functioning and quality of life were significantly differently affected between the two treatment groups.

Ancillary analyses

Per protocol analyses

The results of the per protocol analyses were comparable to the findings of the intention-to-treat analyses. That is, no statistically significant differences between the MF and TAU groups were found in terms of treatment engagement (as measured with the SES and number of no-shows) and patient-reported motivation. The findings on clinician-reported introjected motivation were confirmed such that clinicians reported a significantly higher reduction of introjected motivation in MF than in TAU (AMD =−4.9, 95% CI =−7.4 to −2.4, P<0.001). Additionally, we found an effect on clinician-reported external motivation such that clinicians reported less external motivation in MF compared to TAU (AMD =−3.2, 95% CI=−6.2 to −0.3, P<0.03). No statistically significant differences between the MF and TAU groups were found for psychosocial functioning and quality of life.

Test of interaction effects: the role of primary diagnosis and age

Results of the interaction analyses showed no statistically significant differential treatment effect of the primary diagnosis on changes in treatment engagement as assessed by the SES (P=0.50) and the number of no-shows (P=0.09). No differential effects on patient-reported treatment motivation were found, but the interaction effect between treatment group and primary diagnosis was significant for clinician-reported identified motivation (β=−3.77, 95% CI =−7.12 to −0.42, P=0.03) and clinician-reported introjected motivation (β=−5.07, 95% CI =−8.67 to −1.59, P<0.01) (Figure 2 for estimates of accompanying main effects). As depicted in Figure 2, clinicians reported opposing treatment effects for identified motivation in the two diagnostic groups, such that they reported increased identified motivation in patients with a primary diagnosis of a personality disorder and decreased identified motivation in patients with a primary diagnosis of a psychotic disorder. For introjected motivation, it was found that clinicians reported a higher increase in introjected motivation in TAU than in MF for both diagnostic groups, but it was more pronounced for patients with a psychotic disorder.

Figure 2 Statistically significant moderation effects of primary diagnosis on treatment effects (intention-to-treat analyses).

Notes: Results predict change values based on a two-level multiple regression model that included treatment, primary diagnosis, treatment by primary diagnosis (interaction effect) and a multivariate confounder score which included patient sex, ethnicity, educational level, comorbid addiction problems, years of working experience of the clinician, and the baseline value of the outcome. Treatment was coded as “0” (treatment as usual) and “1” (motivation feedback); primary diagnosis was coded as “0” (personality disorder) and “1” (psychotic disorder). ^aModel: intercept (β=−2.72, df =11, P=0.07), treatment (β=1.90, df =246, P=0.19), primary diagnosis (β=2.03, df =246, P=0.08), treatment by primary diagnosis (β=−3.77, df =246, P=0.03), adjusted for the multivariate confounder score (β=2.00, df =246, P=0.41). ^bModel: intercept (β=−5.86, df =11, P<0.01), treatment (β=−1.0, df =246, P=0.52), primary diagnosis (β=3.11, df =246, P=0.01), treatment by primary diagnosis (β=−5.07, df =246, P<0.01), adjusted for the multivariate confounder score (β=7.86, df =246, P<0.01). ^cModel: intercept (β=0.12, df =11, P=0.56), treatment (β=0.45, df =221, P=0.04), primary diagnosis (β=0.17, df =221, P=0.32), treatment by primary diagnosis (β=−0.62, df =221, P<0.01), adjusted for the multivariate confounder score (β=−0.27, df =221, P=0.42).
Abbreviations: TEQ, treatment entry questionnaire; MANSA, Manchester Short Assessment for quality of life.

Further, the interaction effect between treatment group and primary diagnosis was significant for patient-reported quality of life (β=−0.62, 95% CI =−1.08 to −0.15, P=0.01), such that patients reported opposing treatment effects depending on their diagnostic group; those with a primary diagnosis of a personality disorder reported a significantly higher quality of life in MF whereas patients with a primary diagnosis of a psychotic disorder reported lower quality of life in MF (Figure 2). No significant interaction effects between treatment group and primary diagnosis were found on the patient’s psychosocial functioning. Finally, none of the interaction effects between treatment group and age reached statistical significance, suggesting that the effects of treatment were not dependent on patient age.

Harms

No adverse or unintended effects of the MF intervention or of TAU were reported.

Discussion

Main findings

There were no significant differences between MF and TAU regarding clinician-rated treatment engagement and the number of no-shows (primary outcome), and there were no differences regarding the patient’s psychosocial functioning and quality of life (secondary outcomes). Regarding the secondary outcome motivation for engaging in treatment, we found no statistically significant differences between the MF and TAU groups on patient-reported motivation. Clinicians, however, did report that MF reduced introjected motives for engaging in treatment more than TAU, albeit that the effect size was small.

Interpretation and possible mechanisms

Apparently, SMI patients felt that talking about their motivation with their clinician did not change their motivation nor treatment engagement, and clinicians felt that MF did not improve their patient’s treatment engagement. These findings thus question whether monitoring and discussing the current motivational profile of the patient is necessary and sufficient to cause beneficial change in treatment motivation and behaviors such as treatment engagement.

Interestingly, the results show that clinicians did feel that MF changed their patient’s motivation as they reported lower introjected motivation and, when four or more MF sessions were performed, also lower external motivation. This suggests that clinicians noticed a reduction in relatively external motivation for engaging in treatment in their patients in response to MF, signifying that their perception of the patient’s motivation has changed in response to the intervention. We have previously found that introjected motivation for treatment was especially difficult for clinicians to estimate in the current patient sample,⁴⁰ and it is therefore reasonable to assume that the repeated conversations with patients regarding their motives has led to a change in the clinician’s perception of the motivation such that it became more closely aligned with the patient’s perspective. As such, the intervention may have enhanced the clinician’s ability to estimate the patient’s perspective on motivation.

The finding regarding the change in clinician’s ratings of the patient’s motivation raises the question of whether this is a beneficial outcome. If one takes a process-oriented perspective,⁴¹ improving the ability of clinicians to estimate their patient’s motivation or – if one were to assume that clinicians actually perceived a reduction in external motives – improving the overall quality of motivation for engaging in treatment can be considered beneficial in itself. On the other hand, if one takes an outcome-oriented perspective and observes that changes in motivation do not result in beneficial changes in treatment engagement or functional outcomes, the clinical relevance is ambiguous.

Furthermore, the interaction analyses of treatment with primary diagnosis suggest that clinicians felt the MF intervention had opposing effects depending on the primary diagnosis. It should be noted that, considering the theoretically possible changes on the TEQ scales (−36 to 36), the observed changes are small and the clinical relevance and implications are not straightforward. Nevertheless, impaired cognitive functioning in patients with psychotic disorders,^42,43 including problems with synthetic metacognition^44,45 (which involves integrating and bringing together several perceptions into complex ideas about the self and others⁴⁵), may explain why the interaction analyses showed that MF was less effective for patients with psychotic disorders. Offering an intervention that requires patients to repeatedly reflect on internal motivational states while these patients may suffer such (meta) cognitive impairments may have been experienced as overdemanding or even frustrating, even more so for patients who had a relatively high level of motivation at the start of the study. Alternatively, the “additional” personal attention that clinicians in MF give to their patients, as reflected by an explicit interest in the nature of their motivation, may be experienced as positive for those with primarily a personality disorder, whereas this may be experienced more neutral or even negative by patients with primarily a psychotic disorder.

Comparison to other studies

Although the rationale of our study was plausible as demonstrated in several studies,^10,37,46 this was the first study to test a feedback intervention that was explicitly based on motivation for engaging in treatment and the first study to test it in a real-life heterogeneous SMI patient population. Our results are not consistent with results from other feedback studies.^37,46,47 For example, a study by Raes et al⁴⁶ found that providing feedback to substance abuse patients about their personal resources and readiness to change (using a motivational questionnaire based on the Transtheoretical Model^48,49) resulted in more patients attending eight sessions of treatment compared to a control group who did not receive such feedback. Other feedback studies, performed in non-SMI patient populations, have shown that the use of clinical support tools based on therapeutic alliance, social network support, and motivation alongside outcome feedback systems are more effective at improving treatment retention and outcomes than outcome feedback alone.^47,50 It appears that successful outcome monitoring systems include additional support and services that are necessary alongside MF to cause changes in clinical outcomes and that solely monitoring and discussing motivation is an insufficient cause for such changes.

Generalizability

The estimated sample size was not reached which may have compromised the current results, although we feel that it is reasonable to assume that the statistical inferences would not have been different from a somewhat larger sample size. The current patient sample already showed relatively high levels of identified motivation, treatment engagement, and psychosocial functioning and low levels of no-shows to begin with (considering the range of scores). Further, the reasons that nonparticipants gave for declining participation in the trial, including feeling too ill or incapable, and the finding that nonparticipants were more likely to be patients with a psychotic disorder, suggest that the most severely ill patients did not participate. This may reflect that the current study was not successful at recruiting SMI patients with substantial problems in their motivation for engaging in treatment, treatment engagement, and psychosocial functioning. Such ceiling effects and selection bias may explain why MF was not able to improve outcomes, and this suggests that the findings of the current study may not be generalizable to the general SMI outpatient population but are limited to patients who are already relatively well engaged in treatment and function at a relatively high level.

Strengths and limitations of the study

This study had several strengths including the implementation of this study in everyday practice of the community mental health teams, representative TAU, independent raters, multiple methods for assessing motivation, intention-to-treat analysis, and the feasibility and low costs of the intervention. Finally, the number of participating patients (N=294) and the follow-up rate for patients (86%) was high considering the patient population. Despite its strengths, this study may be viewed as a “negative trial”, and common causes for negative trials include failures of concept, design/methodology, and/or logistics. We will address each of these issues in the following sections.

Concept and rationale

Numerous studies suggest that evaluation of the patient’s motivation may help understand how a patient may best be engaged in treatment,^1–5 while other studies have found that clinicians have difficulties in estimating their patient’s motivation for treatment,⁴⁰ suggesting that the plausibility of the rationale for the current study was high. However, it should be noted that in the Netherlands, the accessibility and quality of mental health care for patients with SMI are currently at a relatively high level.⁵¹ TAU was provided by multidisciplinary treatment teams that provided tailored care guided by the patient’s individual symptoms and needs for care and could include assertive outreach, medication, social and financial management, job counseling, crisis interventions, and psychotherapy. Such care may have been sufficiently effective in engaging patients with SMI, especially highly motivated patients who were more likely to participate in this study, such that MF did not prove to be superior to TAU because the contrast between TAU and MF was (too) small.

Design and methodology

Although this current study was well designed,⁹ in hindsight we may conclude that the expected effect size was too high and that the timing of our outcome evaluation might have been suboptimal. A meta-analysis on the effects of feedback in mental health care showed that outcome feedback had beneficial effects if outcomes were measured within 9 weeks after initial assessment (d=0.10, 95% CI =0.01–0.19), but these effects did not persist after 3 months.¹⁰ Another meta-analyses on continuous feedback in outpatient psychotherapy found similar results.⁵² As our study measured outcomes after 12 months, potential short-term beneficial changes of the MF intervention will have gone unnoticed and may have worn off by the end of follow-up. An additional assessment moment within the first 3 months of our study could have been informative in this respect, but due to practical and financial limitations this was not feasible.

Another methodological issue is that clinicians and patients were not blinded for treatment allocation, and this may have influenced the information that they gave on the outcome questionnaires (ie, information bias). Although this is a common design in mental health research²¹ and because blinding was not feasible, this might have biased the results toward no differences between the MF and TAU treatment groups or toward counterproductive effects of the MF intervention if clinicians generally did not expect the intervention to work or felt that the MF intervention was less/not appropriate for patients with psychoses.

Further, although we performed evaluation sessions with clinicians about MF alongside the trial, we have limited insight into what happened during MF sessions as these were neither recorded nor supervised. The exact communication processes within the sessions and whether or not they were autonomy supportive remain unclear, but such processes might explain why the MF intervention was not successful. Despite the training and evaluation sessions for clinicians in MF, we may have failed in providing the professionals with the necessary competencies and tools to be able to address different types of motivation for engaging in treatment, and how to provide support for the needs of autonomy, competence, and relatedness in patients with SMI. More attention for the implementation process, including the influence of contextual factors as well as a minimum intensity of the feedback intervention, may be needed to reach favorable effects. Encouragement of both clinicians and patients to actively involve with MF is already difficult when facing patients with highly prevalent cognitive impairments, communication difficulties, and comorbidities, let alone in a health-care context faced with reorganizations, and as such this requires a unique set of competencies from both researchers and clinicians to ensure sufficient implementation.

The heterogeneity of this study sample is considered both a strength and a limitation. Our sample largely represents a broad population of outpatients with diagnoses of psychotic and personality disorders with a variety of comorbid psychiatric disorders, which strengthens the generalizability of the study and enhances the probability of adoption in clinical practice. However, MF may have different effects in different subgroups of patients, which could only be addressed in an exploratory manner in the current study.

Measuring treatment engagement and motivation for engaging in treatment is complex, and gold standards are lacking.^3,53 A strength of the current study is that we had both patient and clinician reports of motivation and two methods to assess treatment engagement. Other objective measures for treatment engagement and medication compliance, such as pill counts, electronic methods, prescription monitoring, or urine assay tests, were not available. Future studies may use such objective measures, although all have their own strengths and limitations.⁵³ To our knowledge, this was the first study to use the TEQ in a population of patients with SMI. The reliability and validity of the TEQ in the current sample were shown to be acceptable,²⁴ but should be improved upon and should be investigated more extensively to further determine the construct validity and sensitivity to change.

Logistics

Logistic issues that have likely negatively impacted this study include the difficulties in recruiting the intended number of patients – especially the recruitment of patients with low levels of motivation and low levels of treatment engagement – and organizational changes in mental health care during the course of the study, including changes in the no-show policy and costs of mental health care for SMI patients. Further, there were large variations between teams and clinicians in the number and duration of SMFL assessments, reflecting the pragmatic nature of the trial. Our findings may reflect that too few MF sessions were actually utilized (ie, 45% on average) or that the way MF was used in the sessions was not able to beneficially affect motivation and treatment engagement. Frequently, clinicians admitted that they regularly forgot to do SMFL assessments despite efforts from the research team to help them remember, and some reported that they were burdening the patient with “yet another list to fill out”. Such comments seem reflective of a controlling health-care context, where external demands and contingencies pressure people to behave in particular ways.⁵⁴ If this was the case, this is likely to have been a counterproductive mechanism in the MF intervention.²⁶

Implications for theory and practice

Theoretically, MF was expected to lead to a higher level of autonomous motivation which would in turn lead to a higher level of treatment engagement. The question remains if this hypothesis can be retained, but the negative results should not be taken as evidence against SDT, as the MF intervention may not have been able to successfully affect SDT constructs such as patient autonomy. The motivational constructs may still be able to predict treatment engagement in both conditions, and this should be addressed in subsequent investigations. Future studies should address which contextual factors influence the implementation and interpretation of (motivation) feedback interventions, as these contextual factors can impact the motivational constructs that the intervention is trying to affect.^55,56

Regarding implications for clinical practice, our study provided no evidence for the effectiveness of MF in outpatients with SMI, and this discourages the implementation of the SDT-based MF intervention into community mental health care for such patients. Nevertheless, although this study did not show beneficial effects of MF in SMI outpatients, it contributes to the evidence base for optimal clinical decision-making and is relevant to prevent an overestimation of the benefits of feedback interventions. The findings imply that monitoring and discussing the patient’s motivation is insufficient to improve motivation and treatment engagement in outpatients with SMI. It appears that successful outcome monitoring systems include additional support and services alongside MF, and this allows for beneficial changes in clinical outcomes. In the future, there may be a place for SDT-based MF as a communication tool for the clinician to explore the patient’s perspective, after which other tailored interventions and services may be applied to improve patient motivation, treatment engagement, and, most importantly, symptomatic and functional outcomes.

Acknowledgments

This study was funded by GGZ Westelijk Noord Brabant and approved by the Medical Ethical Committee for Mental Health Care Institutions (Dutch Trials Registry NTR2968). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data collection was performed in GGZ Westelijk Noord Brabant and in GGz Breburg by the first author and several collaborating students and clinicians. Analyses were performed by the first author at the Erasmus University Medical Center. GGz Breburg contributed time, energy, and resources to the study for the inclusion of patients, but was otherwise not a funder of the study. We are grateful to all patients, students, and health care professionals who made this study possible.

Disclosure

ECJ and HJD report no competing interests. AD reports that he has paid employment by the funder. The funder had no commercial interest in the results of the current study. CMFC reported grants from Eli Lilly, NIHR, EU FP7 program, ZonMw, and Achmea outside the submitted work. CLM reported grants from Johnson & Johnson, Janssen Cilagg, NWO, and ZonMw outside the submitted work. The authors report no other conflicts of interest in this work.