Routine Outcome Monitoring (ROM) and Feedback: Research Review and Recommendations

Abstract

Objective

To provide a research review of the components and outcomes of routine outcome monitoring (ROM) and recommendations for research and therapeutic practice.

Method

A narrative review of the three phases of ROM – data collection, feeding back data, and adapting therapy – and an overview of patient outcomes from 11 meta-analytic studies.

Results

Patients support ROM when its purpose is clear and integrated within therapy. Greater frequency of data collection is more important for shorter-term therapies, and use of graphs, greater specificity of feedback, and alerts are helpful. Overall effects on patient outcomes are statistically significant (g ≈ 0.15) and increase when clinical support tools (CSTs) are used for not-on-track cases (g ≈ 0.36–0.53). Effects are additive to standard effects of psychological therapies. Organizational, personnel, and resource issues remain the greatest obstacles to the successful adoption of ROM.

Conclusion

ROM offers a low-cost method for enhancing patient outcomes, on average resulting in an ≈ 8% advantage (success rate difference; SRD) over standard care. CSTs are particularly effective for not-on-track patients (SRD between ≈ 20% and 29%), but ROM does not work for all patients and successful implementation is a major challenge, along with securing appropriate cultural adaptations.

Keywords:

• routine outcome monitoring
• ROM
• outcome measures
• feedback
• clinical support tools
• deterioration
• psychotherapy outcome
• measurement-based care

Clinical or methodological significance of this article: ROM is a method that integrates data into the process of therapy and enables adjustments when patients are not on track. While the effects can, traditionally, be labeled as small, they are additive to standard outcome effects, thus enhancing the overall effectiveness of psychotherapy.

Introduction

Since the mid-1900s, statistical (i.e., actuarial) models have been claimed (Meehl, 1954) and evidenced (Ægisdóttir et al., 2006) to be more accurate than clinical judgements, since psychological therapists are known to be overconfident in evaluating their own effectiveness (Walfish et al., 2012). This issue is especially critical when it potentially blinds a therapist to a patient’s evolving worsening outcomes. Given this context, enhancing the validity of in-session therapist behaviors and patient outcomes requires a measurement-based approach to psychological therapies informed by data, with feedback as a central therapeutic process (Lutz, Schwartz et al., 2022). This has particular relevance for patients at risk of deterioration, with Hannan et al.’s (2005) often-cited study supporting the superiority of actuarial methods over clinical judgement in correctly identifying patients who deteriorate.

Routine outcome monitoring (ROM), including feedback, have their roots in the paradigm of practice-based evidence, specifically patient-focused research (Howard et al., 1996). Such an approach encompasses research attesting to the fact that not all patients improve and a proportion of them deteriorate (e.g., Lambert, 2010). In such cases, administering outcome measures, either continuously or at regular intervals as a means of checking progress and informing treatment decision-making processes, can yield an additive effect over and above the existing effects of the standard delivery of psychological therapies (De Jong et al., 2021; Lutz, De Jong et al., 2021).

ROM is a cost-effective method that has the potential to enhance existing therapy outcomes for some patients at a relatively small additional cost (Delgadillo et al., 2021). Developments in information technology and software packages have likely made the adoption of repeated session-by-session outcome measures less onerous for individual practitioners and have contributed to standardization of the production of progress graphs (Ogles et al., 2022). The role of software packages is important, with ROM systems using increasingly advanced statistical procedures that identify whether a patient’s progress is not on track (NOT) compared with historical data from patients presenting with similar clinical profiles (e.g., nearest neighbors; Lutz et al., 2005). Notwithstanding these developments, however, the single area of major concern and focus for action relates to implementation (Bovendeerd et al., 2022; Lutz, De Jong et al., 2021). The present article provides an overview of the research evidence relating to the components (processes) and outcomes of routine outcome monitoring and feedback.

Definitions and Policy

Definitions

Routine outcome monitoring (ROM) has been referred to by various terms (e.g., progress monitoring, measurement-based care, feedback-informed treatment), but all contain features that can be grouped into three sequential phases: (1) collecting patient data on a regular basis; (2) feeding back data to the therapist and, on many occasions, also to the patient; and (3) when appropriate, adapting the process or focus of therapy in light of the feedback. These three phases have been presented as a transtheoretical model of measurement-based care – Collect, Share, Act (Barber & Resnick, 2022) – while a definition of ROM capturing these three phases describes it as “the implementation of standardized measures, usually on a session-to-session basis, to guide clinical decision-making, monitor treatment progress, and indicate when treatment adjustment is needed” (Pinner & Kivlighan, 2018, p. 248). ROM has been presented as a “relatively straightforward evidence-based practice … that the clinician can add to any type of psychotherapy … without requiring changes in that psychotherapy” (Persons et al., 2016, p. 25).

Policy

The American Psychological Association (APA) has long recommended the use of ROM and feedback methodology in routine care (APA, 2006; Wright et al., 2020). A recent APA governance-appointed Advisory Committee for Measurement-Based Care and the Mental and Behavioral Health Registry has argued for a professional practice guideline focusing on measurement-based care comprising a draft statement: “Psychologists aim to routinely assess treatment process and outcomes and integrate that information in ongoing collaboration with their patients” (Boswell et al., 2022, p. 9). The Joint Commission (2018) requires organizations “to accomplish this [assessment of outcomes] through the use of a standardized tool or instrument. Feedback derived through these standardized instruments may be used to inform goals and objectives, monitor individual progress, and inform decisions related to individual plans for care, treatment, or services” (p.1). The Roadmap for Mental Health Research in Europe has also supported ROM (Emmelkamp et al., 2014). Additionally, national policy makers and regulatory bodies in some countries have made measuring treatment outcomes a requirement, such as in Australia (Burgess et al., 2015), Canada (Tasca et al., 2019), England (Clark, 2018), and Norway (Knapstad et al., 2018).

ROM has been widely espoused in multiple adult therapeutic modalities; for example, psychodynamic therapy (Winkeljohn Black et al., 2017), couple therapy (Anker et al., 2009), and group therapy (Slone et al., 2015); as well as with specific presenting problems, such as substance abuse (Crits-Christoph et al., 2012); and in youth settings (Bickman, 2008). It is pan-theoretical and, as a therapeutic method, combines elements of supervision, continuous assessment, and overall quality assurance. However, in the absence of policy or clinical directives to implement ROM, the willingness to adopt and use this method is likely to be a function of the openness of individual therapists (e.g., Rye et al., 2019).

Research Review of ROM and Feedback Components

Defining Hallmarks

The extent to which all ROM phases are implemented varies, but the most complete format comprises outcome measures being reported back to the therapist, who in turn feeds back to the patient in a way that is responsive to the patient’s current state and within a discussion that can inform subsequent treatment (for different levels of use of outcome measures, see Krägeloh et al., 2015). An outcome measure or system provides a signal or alert for the therapist when a patient’s treatment response is not on track (NOT) as determined by the expected treatment response. Some adaptation to the treatment plan is therefore indicated and achieved via a therapeutic method termed clinical troubleshooting in which a therapist identifies obstacles that are interfering with treatment progress and devises a plan to systematically overcome these obstacles following a hypothesis-testing approach (De Jong et al., 2023). Various clinical support tools (CSTs) are available to help identify and address problems that might be interfering with treatment progress (e.g., Lambert, 2010). See Supplemental Materials for an account of a clinical case across the three ROM phases.

Phase 1: Collecting Outcome Data

The two most frequently researched feedback systems are the Outcome Questionnaire-45 (OQ-45) System comprising the OQ-45 measure and Analyst software (Lambert et al., 2013) and the Partners for Change Outcome Management System (PCOMS; Duncan & Reese, 2015), the latter often referred to by its two separate components – the Outcome Rating Scale (ORS; Miller et al., 2003) and the Session Rating Scale (SRS; Duncan et al., 2003). Other outcome measures have been used for routine outcome monitoring (see Drapeau, 2012; also, a special issue in Psychotherapy, 52(4), 2015). Recently developed ROM systems incorporate sophisticated prediction and decision-making tools (e.g., Demir et al., 2022; Lutz et al., 2019; McAleavey et al., 2021; Schiepek et al., 2016). Multiple factors need to be considered in selecting ROM measures or systems: suitability with the patient population, clinical utility, psychometric properties, financial viability, and of minimal burden to patients (De Jong et al., 2023).

Patients generally support monitoring outcomes during therapy (e.g., Lutz et al. [2011] as reported in Castonguay et al. [2013]), and prefer using a short measure to monitor treatment progress (Thew et al., 2015). Yet, qualitative data also shows patients’ doubts about completing measures when their providers do not utilize the information (Talib et al., 2018). Four meta-themes of patients identified by Solstad et al. (2019) are: (1) concern about motives for adopting ROM; (2) dominance of symptom focus; (3) the need to provide a rationale, engaging with patients, and explaining how the data will be used (see Börjesson & Boström, 2020); and (4) developing a collaborative practice in which ROM becomes a clinical process tool to direct and deepen the therapeutic dialogue (see Faija et al., 2022). Furthermore, Solstad et al. (2021) found that while ROM can enhance patients’ awareness of emotions and experiences, they can also be uncertain about the veracity of a rating scale compared with their felt experience.

Regarding frequency of data completion, no significant difference has been reported between continuous feedback (i.e., every session) and less frequent feedback (De Jong et al., 2021) and although Janse et al. (2020) reported a similar result, continuous feedback resulted in fewer therapy sessions and a lower patient dropout rate. Prediction modeling has shown the use of more sophisticated analyses is not dependent on session-by-session data (Mütze et al., 2022). Hence, a more basic data collection regime is viable for prediction purposes but the key is ensuring that there are sufficient data points, meaning that continuous data collection may be more critical in shorter duration treatments. Continuous data collection also facilitates clinical decision making by increasing the odds of identifying early on that a patient is not progressing well.

Challenges to practice relate to broadening out from symptom-only measures to consider, for example, quality of life and relationship measures (Barkham, 2021) as well as idiographic measures (see Sales et al., 2022). In addition, consideration needs to be given to cultural adaptations that would increase the accessibility of ROM for patients from culturally diverse and ethnic minority populations (e.g., Koslofsky & Rodíguez, 2017).

Phase 2: Feeding Back Data

While completion of self-report measures can result in greater self-learning by some patients, the impact on outcomes and treatment likely depends on how the information is subsequently shared and used in therapy (Greenhalgh et al., 2018). Data need to be reviewed collaboratively with patients (Hepner et al., 2019), preferably in the form of a conversation (Faija et al., 2022), and viewed within the context of shared decision-making rather than their simply being told the scores. But a randomized controlled trial showed no evidence that ROM enhanced shared decision-making overall across a range of patient presenting problems (Metz et al., 2019). However, it was associated with better outcomes for patients experiencing mood disorders with the suggestion that the feedback was not sufficiently targeted to the other presenting conditions. There also needs to be an appropriate balance between personalizing and over-protocolizing the procedures of feedback (Drew et al., 2021). Marked change in individual items, either showing improvement or deterioration, can provide a clinical focus and thereby act as a clinical process tool for therapeutic discussions (Faija et al., 2022). Such practice integrates the outcome measure and its individual items into the therapy session and conversation.

Research suggests both patients and therapists have similar preferences for how feedback is presented, preferring greater specificity in relation to the nature of predictors as well as of advice. Both have preferences for feedback to be presented as either a continuous outcome or an outcome that is expressed in terms of a probability, with the feedback representation comprising both text and images (Hilhorst et al., 2022).

Clinical support tools (CSTs) are a major component in maximizing the effectiveness of ROM and robust evidence supports their use to augment feedback. Studies focusing on NOT patients in which a feedback-only arm was compared with one accessing feedback + CSTs have consistently favored CSTs (d = 0.36) compared with expected treatment response (d = 0.12) or use of raw scores (d = 0.04; De Jong et al., 2021). Results from other meta-analyses have yielded slightly larger effects for CSTs (e.g., d = 0.49; Lambert et al., 2018).

CSTs assist in identifying key obstacles to treatment progress; for example, therapeutic alliance, motivation, social support, and issues coping with problematic life events (e.g., The Assessment for Signal Clients [ASC]; Lambert et al., 2015). Evidence from various studies have suggested that lack of support (White et al., 2015) and suicidality, motivation, and life events (Schilling et al., 2021) are more associated with deterioration than alliance. These findings relating to the role of life events and the lack of social support in the lives of patients are a salutary reminder of the importance of the social and interpersonal world of patients outside of therapy and contrasts with the considerable attention paid to in-session concepts.

As an example of the application of clinical support tools, the ASC has been used as a central component in the Trier Treatment Navigator (Lutz et al., 2019), supplemented with the Affective Style Questionnaire (Hofmann & Kashdan, 2010) with both measures administered at every fifth session. Figure 1 displays the output from the Trier Treatment Navigator showing routinely collected data [A] as against the expected treatment response curve [C] based on a nearest neighbor approach – that is, based on a defined number of patients from a data archive who most closely resemble the specific patient (Lutz et al., 2005). The crucial flag occurs at session 9 when the outcome measure exceeds the dynamic failure boundary [B], producing a warning flag [D]. Completion of the CSTs [E] shows that the patient is experiencing difficulties regarding motivation/therapy goals and the therapeutic alliance, which then provide a refocusing of therapy to address these domains in the third phase of ROM.

Figure 1: Example of feedback graph with clinical support tools. Note: The screenshot shows the symptomatic progress of a patient measured with the Hopkins-Symptom-Checklist-11 (HSCL-11) within the clinical navigation system by Lutz et al. (2019); A: Individual measurement points for the patient measured at the beginning of each session; B: Dynamic failure boundary; C: Expected treatment response curve; D: As soon as the patient’s HSCL-11 score exceeds the failure boundary (marked in the graph with an arrow), the therapist receives a warning signal (top right corner); E: CSTs are divided into five domains. The exclamation mark indicates the domains in which the patient has specific problems. The therapist is able to click on these icons to gain access to the activated tools. The check mark signals that the patient has few or no problems in the respective domain.

Phase 3: Adapting Therapy in Light of Feedback

Evidence of adapting the focus or direction of therapy has been shown from analyses of patient-therapist dyads (Brooks Holliday et al., 2021). Within session adjustments include: Setting and monitoring treatment goals; determining the most appropriate therapeutic approach for a patient; adjusting the pace of therapy; focusing the nature of the discussion in a session and/or assigning treatment “homework”; or adjusting therapeutic modalities. Overall, a set of best practices for discussing feedback proposed providing a strong rationale for ROM, discussing ROM every time measures are administered; actively engaging patients in the discussion of ROM; and using a graph to show progress to patients.

When CSTs are not available, the principle is the same but relies on the therapist implementing clinical troubleshooting, which is informed by their knowledge of the clinical and research evidence relating to a patient’s response to treatment (see De Jong et al., 2023). This might include contextual (e.g., lack of social support in the patient’s life), process (e.g., a difficult therapeutic relationship), and patient factors (e.g., the presence of comorbid conditions). A therapeutic plan is developed, implemented, and evaluated through subsequent ROM data and reviewed in the context of the overall treatment plan.

Summary of Meta-analytic Reviews

At least 50 controlled, quasi-experimental, observational, and implementation studies have investigated the effectiveness of ROM feedback systems applied in several countries over the last 20 years. These studies have been synthesized across numerous systematic reviews (e.g., Carlier et al., 2012; Davidson et al., 2014; Gondek et al., 2016; Krägeloh et al., 2015; Mackrill & Sørensen, 2020) and at least 11 meta-analyses (Bergman et al., 2018; De Jong et al., 2021; Kendrick et al., 2016; Knaup et al., 2009; Lambert et al., 2003, 2018; Østergård et al., 2020; Pejtersen et al., 2020; Rognstad et al., 2022; Shimokawa et al., 2010; Tam & Ronan, 2017).

Early meta-analyses suggested that ROM feedback improves distal treatment outcomes relative to usual psychological care. However, trials published at that time (around a dozen) were predominantly from the US and located in university counseling settings (Knaup et al., 2009; Lambert et al., 2003; Shimokawa et al., 2010). A seminal meta-analysis reported a pooled (intention-to-treat; ITT) effect size (ES) of g = 0.28 (p = .003) favoring feedback-informed treatment relative to usual care in cases classed as NOT using the OQ-45 measure (Shimokawa et al., 2010). The pooled (ITT) ES was larger when examining data from studies that supplemented feedback with CSTs, g = 0.44. These promising results led to feedback studies in other countries, treatment settings, age groups, and outcome measures.

Recent meta-analyses have examined the effects of feedback with specific measures, principally the OQ-45 and PCOMS (Lambert et al., 2018; Østergård et al., 2020; Pejtersen et al., 2020), and on specific populations (e.g., children and young people; Bergman et al., 2018; Tam & Ronan, 2017) and presenting conditions (e.g., common mental health disorders; Rognstad et al., 2022). As the literature from trials accumulated, meta-analyses of ROM with heterogeneous populations and outcome measures emerged reporting discrepant findings. For example, one review reported a pooled ES of g = 0.10 favoring feedback in short-term therapies but no significant effect in long-term therapies (Knaup et al., 2009), while another review reported no significant overall effect of feedback, but reported a significant ES of g = 0.22 favoring feedback in NOT cases (Kendrick et al., 2016). The most comprehensive meta-analysis comprising 58 studies (49 RCTs and 9 cohort studies) reported ESs (g) ranging from 0.15 and 0.17 for the total sample and NOT cases, respectively, to 0.36 for NOT cases with the addition of CSTs (De Jong et al., 2021). A further recent meta-analysis comprising 31 RCTs reported a virtually identical ES of 0.14 for a total sample and a slightly larger ES of 0.29 for NOT cases (Rognstad et al., 2022).

Overall, the additive ESs from these meta-analyses of psychological therapy with feedback compared to psychological therapy without feedback can be summarized as ranging from small (g = 0.14; Lambert et al., 2018; Rognstad et al., 2022) when based on all cases, to medium (g = 0.33; e.g., Lambert et al., 2018) when based on NOT cases, and increasing to 0.49 when CST feedback was used (Lambert et al., 2018). These mixed findings seem to be a function of the analysed samples and their heterogeneity, which requires a comprehensive examination of potential moderators of effects sizes across studies.

Cumulative and Causal Evidence

Considering the broader evidence base provides some clarity on the impact of feedback. The ESs for psychological therapy with versus without client feedback confirm the lower estimate (g = 0.15; De Jong et al., 2021) as previously reported (g = 0.14; Lambert et al., 2018). The largest effect is obtained in the NOT sample for feedback enhanced by CSTs (g = 0.36; De Jong et al., 2021), an effect slightly lower than previously reported for NOT with CSTs (g = 0.49; Lambert et al., 2018), but similar to both the effect of PCOMS for all samples (g = .40) and NOT samples using the OQ-45 (g = .33; Lambert et al., 2018).

ESs regarding improvement for NOT patients with feedback vs. treatment as usual (TAU) have been reported in seven meta-analyses (De Jong et al., 2021; Kendrick et al., 2016; Lambert et al., 2003, 2018; Østergård et al., 2020; Rognstad et al., 2022; Shimokawa et al., 2010). Effects range between g = .17 (De Jong et al., 2021) and g = .53 (Shimokawa et al., 2010). Three meta-analyses assessed the effects of feedback with CSTs compared to no-feedback controls in NOT cases and found ESs ranging from g = 0.36–0.53 (De Jong et al., 2021; Lambert et al., 2018; Shimokawa et al., 2010).

Additional findings have shown feedback to increase the rate of improvement in NOT cases when CSTs were included (OR = 2.40 vs. OR = 1.89 without CSTs; Lambert et al., 2018). In terms of the effect of feedback on treatment duration, De Jong et al. (2021) found no evidence of feedback impacting on treatment duration. In contrast, other studies have reported differential effects between OT vs. NOT cases with OT cases receiving on average 0.69 fewer sessions when feedback was provided, while NOT cases received 0.73 more sessions when feedback was provided (Kendrick et al., 2016). While these results concur with earlier findings (e.g., Lambert et al., 2003), results need further investigation. And regarding dropout, one salient finding from the meta-analyses of De Jong et al. (2021) is the effect of feedback on reducing dropout (i.e., by 20%).

Recall that these effects are additive to the effects of standard treatment with no feedback. The average ES of feedback (g = 0.15) corresponds to a success rate difference (SRD; Furukawa & Leucht, 2011)¹ of 8.45%. This ES increases when clinical support tools for NOT cases are used to g ≈ 0.36–0.53, which corresponds to an SRD between 20.09% and 29.22%. In this context, interpretation of ESs is crucial, particularly when between-group ESs are compared with those from correlational studies (Kraft, 2020) as the former reflect causal effects whereas the latter represent only descriptive relations between two variables. Correlational designs and the resulting ESs are, on average, substantially larger than those derived from between group designs (Kraft, 2020). Accordingly, ROM feedback is a relatively simple method provided in addition to psychological treatment, where the effects compared to a range of control conditions are moderate to large (e.g., Barkham & Lambert, 2021).

Moderators

Feedback studies have yielded a range of moderators that impact on patient outcomes. De Jong et al.’s (2021) comprehensive meta-analysis reported moderating effects for measures and systems, finding larger effects for the ORS compared with the OQ-45 (and other outcome instruments), while the PCOMS feedback system returned larger effects compared with the OQ System (or other feedback systems). However, the differential effect was more nuanced in that the PCOMS had a larger effect when all patients were sampled but not for the NOT patients, while the OQ System appeared particularly effective with this latter group, especially in conjunction with CSTs. Such findings appear to be consistent with the differing rationales for these two systems and have also been reported by Rognstad et al. (2022).

Both De Jong et al. (2021) and Rognstad et al. (2022) found the use of an independent outcome instrument rather than the feedback instrument to yield smaller effects, while the former study also reported smaller effects in more recently conducted studies, those conducted outside the US, and those without the developer of the feedback system as a co-author. An implication of these latter findings is that as the research evidence for feedback extends and becomes less centralized, the resulting effect sizes will provide increasingly precise and reliable estimates of its potential in the field.

Therapist effects have been shown to moderate feedback effects (Bovendeerd et al., 2022; Janse et al., 2020). A reanalysis of six earlier ROM studies using the OQ-45 found the provision of feedback reduced the size of the therapist effect, thereby leveling the variability between more and less effective therapists (Delgadillo et al., 2022). Specific effects have been reported for female therapists while those with a higher commitment to feedback show a higher probability of using the information provided by a feedback system, and those therapists who use a feedback system are also more effective with NOT patients (De Jong et al., 2012). Therapists with a low internal feedback propensity (i.e., less likely to trust their own opinions rather than feedback from an external source) who were more committed to using the feedback at the beginning of the study saw patients who improved more quickly. By contrast, therapists with a high internal feedback propensity (i.e., trusted their own opinions over those from an external source) saw patients with a slower rate of change (De Jong et al., 2012). In addition, therapist positive attitude to feedback and using specific modifications in light of feedback have been found to be associated with enhanced effects (Lutz et al., 2015), while therapist-rated usefulness of feedback has been reported to be a significant moderator of feedback outcome associations (Lutz, Deisenhofer et al., 2022).

Improving patient outcomes as a function of feedback has been enhanced by combining traditional ROM feedback and CSTs with data-informed decision and support tools at the start of therapy. In such a comprehensive feedback system, an algorithm is used to generate feedback on the optimal treatment strategy and dropout risk at the beginning of treatment, while during the course of treatment feedback to therapists is given on patients’ progress, including CSTs for patients at risk for treatment failure. In a study comprising 538 patients that evaluated both components of such a comprehensive feedback system, results showed patients who received the prospectively predicted optimal treatment strategy displayed greater early improvements (Lutz, Deisenhofer et al., 2022).

However, feedback arising from ROM is not necessarily a panacea for enhancing treatment outcomes for all patients. Indeed, evidence suggests two groups of patients might experience negative effects of ROM: patients with severe psychopathology, and those experiencing Cluster B personality disorders. Regarding the former, patients experiencing greater severity have been reported to experience an aversive effect when in receipt of negative feedback in which CSTs were not available (Errázuriz & Zilcha-Mano, 2018; van Oenen et al., 2016). Hence, CSTs may be all the more important in the context of severe psychopathology. And for the latter, in a sample drawn from inpatient and day-patient settings, patients presenting with cluster B personality disorders or personality disorders not otherwise specified (PD-NOS), showed higher levels of symptom distress over the first 6-months of treatment in response to patient and therapist feedback as compared with patients presenting with cluster C personality disorder, although these differences subsequently dissipated by 9 months (De Jong et al., 2018). Overall, these findings suggest that the combination of discouraging feedback and greater severity or vulnerability may make negative feedback an unwelcome component, particularly in situations where there are not the resources to support delivery of CSTs or where the flexibility of the therapist may be limited (e.g., in inpatient settings).

Implementation has been shown to impact ROM effects (Bovendeerd et al., 2022; Simon et al., 2013; van Sonsbeek et al., 2021). Multi-center studies found differential effects of feedback within trials, with some locations showing medium effects and other locations showing no effect at all (e.g., Bovendeerd et al., 2022). The effects of ROM have been found to increase over time, with later recruited patients yielding larger gains than those recruited at the commencement of the trial (Brattland et al., 2018).

Training Implications

Training in ROM has been examined as a potential moderator but has not been found to significantly impact feedback effects on symptom reduction or dropout, although it has resulted in decreasing the percentage of deteriorated cases (De Jong et al., 2021). Other studies have shown that positive attitudes towards ROM (Edbrooke-Childs et al., 2016) and rates of ROM adoption (Persons et al., 2016) can be enhanced following well-designed training. The top two ranked reasons for using ROM from a US survey were to help trainees determine when there was a need to adapt treatment and to help them make better treatment decisions (Peterson & Fagan, 2017).

Strategies that include behavioral rehearsal and modeling of practical actions are likely to enhance the delivery of ROM (Beidas et al., 2014). These include components that are primarily interactions with patients (e.g., explaining the rationale, responding to patients’ reactions to data, and addressing adaptations to treatment). The adoption of deliberate practice is likely to support a more active and method-based approach to learning key components of ROM (see Rousmaniere, 2017). Actions to improve the adoption of ROM include training the trainers as well as trainees. ROM has been espoused as a tool in supervision, emphasizing the view that supervisors and the supervision process is integral to securing the adoption of ROM by trainees (Swift et al., 2015). Accounts of differing training programs premised on ROM have been reported in the US (e.g., Cooper et al., 2021) and Germany (e.g., Lutz et al., 2023). While some ROM activities can be achieved in months, others may take longer (e.g., Cooper et al., 2021).

Implementation Science

While the literature on the benefits of ROM have yielded both positive and null findings, the literature is almost unanimous in identifying implementation as the main barrier for successful ROM with obstacles grouped into three main categories: (1) the people, (2) organizational aspects, and (3) systems (Lewis et al., 2019; Van Wert et al., 2021). Unlike most therapeutic methods, ROM is dependent on a climate of organizational support: the existence of a “culture for feedback” (Bertolino & Miller, 2012). Cultural and philosophical issues are potentially greater obstacles to implementation (Boswell et al., 2015; Boyce et al., 2014).

Commonly reported barriers include ease of accessing and using ROM systems, guidance on selecting outcome measures, and organizational accountability (Van Wert et al., 2021). Mackrill and Sørensen (2020) identified a wide range of factors comprising leadership, inter-organizational factors, feedback culture, implementation team, coordinators and champions, supervision, training, measures, and generating a language for ROM use in clinical practice. Rye et al. (2019) reported that holding more positive attitudes regarding the adoption of ROM predicted greater use of standardized instruments. Limitations centered on ROM being seen as too narrowly focused, not suitable for patients presenting with multiple problems, and hindering the relationship between patient and therapist. Such concerns predicted poorer uptake of standardized measures. Therapists with a higher commitment to client feedback also had a higher probability of using feedback and those therapists were more effective with NOT patients (De Jong et al., 2021). Similarly, therapists’ satisfaction with ROM systems and use of feedback information tend to predict the magnitude of feedback effects (Lutz et al., 2015). Many authors have described strategies for improving implementation (e.g., Bear et al., 2022; Mellor-Clark et al., 2016).

Limitations of the Research

Besides the predominantly Western samples, the major limitations concern poor implementation and lack of statistical power. Null findings highlight research dilemmas in these two specific areas (e.g., van Sonsbeek et al., 2021), along with more complex designs (e.g., Errázuriz & Zilcha-Mano, 2018), mismatch between feedback system and the patient population (e.g., van Oenen et al., 2016), and infrequent application of feedback (e.g., Schöttke et al., 2019). A significant portion of the literature has been characterized by biases arising from researcher allegiance and from a lack of independent outcome measures (i.e., independent of the feedback measure), both of which were addressed by Bovendeerd et al. (2022). Designs need to accommodate both patient and therapist and utilize multilevel modeling in the analysis of the data to reflect the nesting effect. Adequate power is also essential to advance understanding of potential moderators, which, along with a greater focus of mechanisms and theoretical models (e.g., Sapyta et al., 2005) is urgently needed.

Reliance on highly selected samples of committed practitioners, sometimes with leading international experts as advisors, lessens the generalizability of such studies. Independence of authorship and ROM systems as well as a variety of clinical populations are required together with follow-up data. The utility of using idiographic outcome measures should be considered (Jensen-Doss et al., 2018) as evidence suggests no differences in preferences between idiographic and nomothetic approaches (Bugatti & Boswell, 2022). A balance between standardization and personalization in ROM may yield better rates of adoption by practitioners (Bjaastad et al., 2019) and the limitations of relying on fixed-predictions based solely on an initial assessment are being addressed by moves towards more dynamic modeling (e.g., Bone et al., 2021).

Conclusion: Research and Therapeutic Practices

This article has drawn together the current research evidence relating to the components and impact of routine outcome monitoring with feedback and framed as three sequential phases that combine measurement-based methods and clinical skills. The overall impact amounts to an average 8% advantage and higher when used with clinical support tools. From the reported evidence, we conclude with key actionable strategies and practices for research and therapeutic practice.

Research

Independence and generalizability

1. The single and simplest action to enhance the validity of feedback evaluation would be to adopt an independent outcome measure distinct from that used in implementing the feedback process. In addition, independent evaluations of feedback systems conducted by people other than the developers of such measures/systems will enhance the quality of evidence regarding impact and potential allegiance bias.

Size and sampling

2. As the impact of feedback, when effective, lies in the region of smaller effects, studies need to be powered on this assumption. This power is also necessary in order to better determine key moderators. In addition, a broader sampling strategy needs to be adopted across countries, cultures, appropriate settings, and clinical populations, with longer-term follow-up data collected.

Clinical support tools and decision making

3. To further enhance the effect of feedback, invest in conducting studies using clinical support tools (CSTs) during treatment as this approach appears particularly effective, but also extend it to combine with data-informed decision and support tools at the start of therapy. Conducting prospective studies combining data-informed decision and support tools at the start of therapy with traditional ROM feedback during therapy will enable testing methods for tailoring treatments to specific patient groups as well as testing treatment adaptations during therapy.

Dosage

4. Whatever research design is employed, it should ensure there is a clear contrast in that participants receive a sufficient dose of feedback, in whatever form it takes, for it to be a fair test of the feedback method against any control or comparator condition.

Therapeutic Practices

Adoption and implementation

1. Given the additive effect of ROM, particularly for patients who are not progressing as expected, there are potential gains in adopting ROM regardless of theoretical orientation. ROM provides transparency regarding patient outcomes, is viewed favorably by patients, and acts as a complement to clinician judgements of patient outcomes.

2. It is crucial to identify and address obstacles to successful implementation of ROM before initiating it and provide sufficient time and resources to support its adoption. The time taken to implement ROM will depend on the specific aims of the project, the resources available, and the compliance of all stakeholders involved. However, attitudinal aspects (e.g., viewing ROM positively) are key factors for practitioners and a significant factor in the success of ROM, supported by a community of ROM champions.

Context and setting

3. ROM is not a panacea and the clinical population and setting needs to be considered. Patients presenting with high severity levels or cluster B personality disorders may not benefit due to repeated negative feedback, while clinical settings where there is insufficient flexibility for the therapist to adapt therapy may not benefit from ROM.

4. Regardless of setting, frame ROM in-session activity in terms of three actionable phases: administering measures, feeding back results, and adapting therapy where indicated.

Collecting data

5. Adopting an outcome measure that is both psychometrically sound and clinically practical is crucial along with therapists being familiar with the measure, its scoring, interpretation, meaning, and how such information can be used to help patients progress.

6. It is important to consider culturally appropriate adaptations to the content or processes of ROM to ensure that it does not disadvantage culturally diverse populations.

Integrating ROM into therapy

7. Introduce ROM together with a clear rationale for its use to each patient early in the course of therapy, preferably in the first session so that the expectations are set out clearly and ROM is presented as an integral part of therapy. And administer outcome measures frequently. Sufficient data density is required to recognize patterns of deterioration at the earliest opportunity to enact therapy adaptations. Shorter treatment durations require more frequent ROM administration.

8. Supplement ROM with clinical support tools, particularly for not-on-track patients, as these have consistently yielded the largest effects in research on ROM. These provide therapists with a clinical focus to address potential reasons for a lack of progress.

Feeding back

9. Engage patients in discussions about ROM with a particular emphasis on the data using teach-back methods to facilitate dialogue about the match or mis-matches between the data and their psychological health.

10. Use a graph to show progress to patients as visualization enables a better understanding of the overall concept of ROM as well as connecting data with possible treatment options. Computer programs will generate expected treatment curves if available; if not, using graphs generated by Excel can be informative. The use of expected treatment curves also results in lower deterioration rates.

Adapting therapy

11. The function of clinical support tools can be emulated using clinical troubleshooting by the therapist in which their knowledge of the clinical and research evidence identifies contextual, processes, and patient factors that may be obstacles to improvement and adjusts the course of therapy accordingly.

Infrastructure

12. Implement training in ROM for therapists as this enhances reduction in patient deterioration rates. Active models of training involving role plays and deliberate practice are likely to enhance the interactive components of ROM above and beyond passive knowledge acquisition. Also, provide ROM training for the trainers.

In conclusion, the implementation of these research and practice activities will consolidate and advance both the robustness of research and the therapeutic impact of ROM and feedback. But these practices are not panaceas, although together with other methods yielding discrete but smaller effects (e.g., treatment matching), the cumulative impact can lead to noticeable improvements in the effectiveness of psychological therapies (see Barkham, 2022; Barkham & Lambert, 2021).

Declaration

This article is adapted, by special permission of Oxford University Press, by the same authors in C. E. Hill & J. C. Norcross (Eds.) (2023), Psychotherapy skills and methods that work. New York: Oxford University Press. The interorganizational Task Force on Psychotherapy Methods and Skills was cosponsored by the APA Division of Psychotherapy/Society for the Advancement of Psychotherapy.

Disclosure Statement

WL developed a ROM and feedback system called Trier Treatment Navigator, but receives no financial gain from it. MB is a co-developer of the Clinical Outcomes in Routine Evaluation-Outcome Measure (CORE-OM) and CORE-10, both of which have been used to support ROM activities but receives no financial gain from these measures. JD co-developed feedback technologies with industry partners, but receives no financial benefit from this work. KdJ, JD, and MB have published a book on ROM and feedback for which KdJ and JD receive royalties. All co-authors are academics in the field and teach about ROM and feedback in academic courses, scientific conferences, and professional training courses. KdJ, JD, and WL acknowledge receipt of occasional honoraria for external workshops or talks on ROM.

Notes

1 The success rate difference (SRD) is the difference between the probability that a patient in the treatment group has an outcome preferable to one in the control group and the probability that a patient in the control group has an outcome preferable to one in the treatment group. These probabilities are expressed as percentages throughout this paper (see Furukawa & Leucht, 2011).