Routine Outcome Monitoring (ROM) and Feedback: Research Review and Recommendations

References

• Ægisdóttir, S., White, M. J., Spengler, P. M., Maugherman, A. S., Anderson, L. A., Cook, R. S., Nichols, C. N., Lampropoulos, G. K., Walker, B. S., Cohen, G., & Rush, J. D. (2006). The meta-analysis of clinical judgment project: Fifty-six years of accumulated research on clinical versus statistical prediction. The Counseling Psychologist, 34(3), 341–382. https://doi.org/10.1177/F0011000005285875
   Web of Science ®Google Scholar
• American Psychological Association, Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. https://doi.org/10.1037/0003-066X.61.4.271
   PubMed Web of Science ®Google Scholar
• Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback to improve couple therapy outcomes: A randomized clinical trial in a naturalistic setting. Journal of Consulting and Clinical Psychology, 77(4), 693–704. https://doi.org/10.1037/a0016062
   PubMed Web of Science ®Google Scholar
• Barber, J., & Resnick, S. G. (2022). Collect, share, act: A transtheoretical clinical model for doing measurement-based care in mental health treatment. Psychological Services, Advance online publication. doi: 10.1037/ser0000629
   PubMed Web of Science ®Google Scholar
• Barkham, M. (2021). Towards greater bandwidth for standardised outcome measures. The Lancet Psychiatry, 8(1), 17. https://doi.org/10.1016/S2215-0366(20)30488-0
   PubMed Web of Science ®Google Scholar
• Barkham, M. (2022). Smaller effects matter in the psychological therapies: 25 years on from Wampold et al. (1997). Psychotherapy Research, 1–3. Advance online publication. https://doi.org/10.1080/10503307.2022.2141589
   Google Scholar
• Barkham, M., & Lambert, M. J. (2021). The efficacy and effectiveness of psychological therapies. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change (7th ed., pp. 135–189). Wiley.
   Google Scholar
• Bear, H. A., Dalzell, K., Edbrooke-Childs, J., & Wolpert, M. (2022). Applying behaviour change theory to understand the barriers to implementing routine outcome monitoring. The British Journal of Clinical Psychology, 61(3), 557–578. https://doi.org/10.1111/bjc.12322
   PubMed Web of Science ®Google Scholar
• Beidas, R. S., Cross, W., & Dorsey, S. (2014). Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice, 21(1), 1–11. https://doi.org/10.1016/j.cbpra.2013.04.002
   PubMed Web of Science ®Google Scholar
• Bergman, H., Kornør, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefsen, T. K., & Bjørndal, A. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews, 8, CD011729. https://doi.org/10.1002/14651858.CD011729.pub2
   PubMedGoogle Scholar
• Bertolino, B., & Miller, S. D. (2012). ICCE manuals on feedback-informed treatment (Vol. 1–6). ICCE Press.
   Google Scholar
• Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114–1119. https://doi.org/10.1097/CHI.0b013e3181825af8
   PubMed Web of Science ®Google Scholar
• Bjaastad, J. F., Jensen-Doss, A., Moltu, C., Jakobsen, P., Hagenberg, H., & Joa, I. (2019). Attitudes toward standardized assessment tools and their use among clinicians in a public mental health service. Nordic Journal of Psychiatry, 73(7), 387–396. https://doi.org/10.1080/08039488.2019.1642383
   PubMed Web of Science ®Google Scholar
• Bone, C., Simmonds-Buckley, M., Thwaites, R., Sandford, D., Merzhvynska, M., Rubel, J., Deisenhofer, A. K., Lutz, W., & Delgadillo, J. (2021). Dynamic prediction of psychological treatment outcomes: Development and validation of a prediction model using routinely collected symptom data. The Lancet Digital Health, 3(4), e231–e240. https://doi.org/10.1016/S2589-7500(21)00018-2
   PubMedGoogle Scholar
• Börjesson, S., & Boström, P. K. (2020). "I want to know what it is used for": Clients' perspectives on completing a routine outcome measure (ROM) while undergoing psychotherapy. Psychotherapy Research, 30(3), 337–347. https://doi.org/10.1080/10503307.2019.1630780
   PubMed Web of Science ®Google Scholar
• Boswell, J. F., Hepner, K. A., Lysell, K., Rothrock, N. E., Bott, N., Childs, A. W., Douglas, S., Owings-Fonner, N., Wright, C. V., Stephens, K. A., Bard, D. E., Aajmain, S., & Bobbitt, B. L. (2022). The need for a measurement-based care professional practice guideline. Psychotherapy, Advance online publication. https://doi.org/10.1037/pst0000439
   Google Scholar
• Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25(1), 6–19. https://doi.org/10.1080/10503307.2013.817696
   PubMed Web of Science ®Google Scholar
• Bovendeerd, B., De Jong, K., De Groot, E., Moerbeek, M., & De Keijser, J. (2022). Enhancing the effect of psychotherapy through systematic client feedback in outpatient mental healthcare: A cluster randomized trial. Psychotherapy Research, 32(6), 710–722. https://doi.org/10.1080/10503307.2021.2015637
   PubMed Web of Science ®Google Scholar
• Boyce, M. B., Browne, J. P., & Greenhalgh, J. (2014). The experiences of professionals with using information from patient-reported out- come measures to improve the quality of healthcare: A systematic review of qualitative research. BMJ Quality and Safety, 23(6), 508–518. https://doi.org/10.1136/bmjqs-2013-002524
   PubMed Web of Science ®Google Scholar
• Brattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C., Linaker, O. M., Ryum, T., Wampold, B., Lara-Cabrera, M. L., & Iversen, V. C. (2018). The effects of routine outcome monitoring (ROM) on therapy outcomes in the course of an implementation process: A randomized clinical trial. Journal of Counseling Psychology, 65(5), 641–652. https://doi.org/10.1037/cou0000286
   PubMed Web of Science ®Google Scholar
• Brooks Holliday, S., Hepner, K. A., Farmer, C. M., Mahmud, A., Kimerling, R., Smith, B. N., & Rosen, C. (2021). Discussing measurement-based care with patients: An analysis of clinician-patient dyads. Psychotherapy Research, 31(2), 211–223. https://doi.org/10.1080/10503307.2020.1776413
   PubMed Web of Science ®Google Scholar
• Bugatti, M., & Boswell, J. F. (2022). Clinician perceptions of nomothetic and individualized patient-reported outcome measures in measurement-based care. Psychotherapy Research, 1–12. Advance online publication. https://doi.org/10.1080/10503307.2022.2030497
   Google Scholar
• Burgess, P., Pirkis, J., & Coombs, T. (2015). Routine outcome measurement in Australia. International Review of Psychiatry, 27(4), 264–275. https://doi.org/10.3109/09540261.2014.977234
   PubMed Web of Science ®Google Scholar
• Carlier, I. V., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J., & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104–110. https://doi.org/10.1111/j.1365-2753.2010.01543.x
   PubMed Web of Science ®Google Scholar
• Castonguay, L. G., Barkham, M., Lutz, W., & McAleavy, A. (2013). Practice oriented research: Approaches and applications. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed, pp. 85–133). Wiley.
   Google Scholar
• Clark, D. M. (2018). Realizing the mass public benefit of evidence-based psychological therapies: The IAPT program. Annual Review of Clinical Psychology, 14(1), 159–183. https://doi.org/10.1146/annurev-clinpsy-050817-084833
   PubMed Web of Science ®Google Scholar
• Cooper, L. D., Murphy, H. G., Delk, L. A., Fraire, M. G., Van Kirk, N., Sullivan, C. P., Waldron, J. C., Halliburton, A. E., Schiefelbein, F., & Gatto, A. (2021). Implementing routine outcome monitoring in a psychology training clinic: A case study of a process model. Training and Education in Professional Psychology, 15(2), 87–96. https://doi.org/10.1037/tep0000298
   Web of Science ®Google Scholar
• Crits-Christoph, P., Ring-Kurtz, S., Hamilton, J. L., Lambert, M. J., Gallop, R., McClure, B., Kulaga, A., & Rotrosen, J. (2012). A preliminary study of the effects of individual patient-level feedback in outpatient substance abuse treatment programs. Journal of Substance Abuse Treatment, 42(3), 301–309. https://doi.org/10.1016/j.jsat.2011.09.003
   PubMed Web of Science ®Google Scholar
• Davidson, K., Perry, A., & Bell, L. (2014). Would continuous feedback of patient's clinical outcomes to practitioners improve NHS psychological therapy services? Critical analysis and assessment of quality of existing studies. Psychology and Psychotherapy: Theory, Research and Practice, 88(1), 21–37. https://doi.org/10.1111/papt.12032
   PubMed Web of Science ®Google Scholar
• De Jong, K., Conijn, J. M., Gallagher, R., Reshetnikova, A. S., Heij, M., & Lutz, M. C. (2021). Using progress feedback to improve outcomes and reduce drop-out, treatment duration, and deterioration: A multilevel meta-analysis. Clinical Psychology Review, 85, 102002. https://doi.org/10.1016/j.cpr.2021.102002
   PubMed Web of Science ®Google Scholar
• De Jong, K., Delgadillo, J., & Barkham, M. (2023). Routine outcome monitoring and feedback in psychological therapies. Open University Press.
   Google Scholar
• De Jong, K., Segaar, J., Ingenhoven, T., van Busschbach, J., & Timman, R. (2018). Adverse effects of outcome monitoring feedback in patients with personality disorders: A randomized controlled trial in day treatment and inpatient settings. Journal of Personality Disorders, 32(3), 393–413. https://doi.org/10.1521/pedi_2017_31_297
   PubMed Web of Science ®Google Scholar
• De Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22(4), 464–474. https://doi.org/10.1080/10503307.2012.673023
   PubMed Web of Science ®Google Scholar
• Delgadillo, J., Deisenhofer, A. K., Probst, T., Shimokawa, K., Lambert, M. J., & Kleinstäuber, M. (2022). Progress feedback narrows the gap between more and less effective therapists: A therapist effects meta-analysis of clinical trials. Journal of Consulting and Clinical Psychology, 90(7), 559–567. https://doi.org/10.1037/ccp0000747
   PubMed Web of Science ®Google Scholar
• Delgadillo, J., McMillan, D., Gilbody, S., De Jong, K., Lucock, M., Lutz, W., Rubel, J., Aguirre, E., & Ali, S. (2021). Cost-effectiveness of feedback-informed psychological treatment: Evidence from the IAPT-FIT trial. Behaviour Research and Therapy, 142, 103873. https://doi.org/10.1016/j.brat.2021.103873
   PubMed Web of Science ®Google Scholar
• Demir, S., Schwarz, F., & Kaiser, T. (2022). Therapy from my point of view: A case illustration of routine outcome monitoring and feedback in psychotherapeutic interventions. Journal of Clinical Psychology, 78(10), 2029–2040. https://doi.org/10.1002/jclp.23408
   PubMed Web of Science ®Google Scholar
• Drapeau, M. (2012). Ten tools for progress monitoring in psychotherapy. Integrating Science and Practice, 2(2), 1–45. https://www.mcgill.ca/psy/files/psy/10_tools_pdf_final.pdf
   Google Scholar
• Drew, P., Irvine, A., Barkham, M., Faija, C., Gellatly, J., Ardern, K., Armitage, J. C., Brooks, H., Rushton, K., Welsh, C., Bower, P., & Bee, P. (2021). Telephone delivery of psychological interventions: Balancing protocol with patient-centred care. Social Science & Medicine, 277, 113818. https://doi.org/10.1016/j.socscimed.2021.113818
   PubMed Web of Science ®Google Scholar
• Duncan, B. L., Miller, S. D., Sparks, J., Claud, D., Reynolds, L., Brown, J., & Johnson, L. (2003). The session rating scale: Preliminary psychometric properties of a “working” alliance measure. Journal of Brief Therapy, 3(1), 3–12. https://www.scottdmiller.com/wp-content/uploads/documents/SessionRatingScale-JBTv3n1.pdf
   Google Scholar
• Duncan, B. L., & Reese, R. J. (2015). The partners for change outcome management system (PCOMS): revisiting the client’s frame of reference. Psychotherapy, 52(4), 391–401. https://doi.org/10.1037/pst0000026
   PubMed Web of Science ®Google Scholar
• Edbrooke-Childs, J., Wolpert, M., & Deighton, J. (2016). Using patient reported outcome measures to improve service effectiveness (UPROMISE): training clinicians to use outcome measures in child mental health. Administration and Policy in Mental Health, 43(3), 302–308. https://doi.org/10.1007/s10488-014-0600-2
   PubMed Web of Science ®Google Scholar
• Emmelkamp, P. M., David, D., Beckers, T., Muris, P., Cuijpers, P., Lutz, W., Andersson, G., Araya, R., Banos Rivera, R. M., Barkham, M., Berking, M., Berger, T., Botella, C., Carlbring, P., Colom, F., Essau, C., Hermans, D., Hofmann, S. G., Knappe, S., … Vervliet, B. (2014). Advancing psychotherapy and evidence-based psychological interventions. International Journal of Methods in Psychiatric Research, 23(Suppl 1), 58–91. https://doi.org/10.1002/mpr.1411
   PubMedGoogle Scholar
• Errázuriz, P., & Zilcha-Mano, S. (2018). In psychotherapy with severe patients discouraging news may be worse than no news: The impact of providing feedback to therapists on psychotherapy outcome, session attendance, and the alliance. Journal of Consulting and Clinical Psychology, 86(2), 125–139. https://doi.org/10.1037/ccp0000277
   PubMed Web of Science ®Google Scholar
• Faija, C. L., Bee, P., Lovell, K., Lidbetter, N., Gellatly, J., Ardern, K., Rushton, K., Brooks, H., McMillan, D., Armitage, C. J., Woodhouse, R., & Barkham, M. (2022). Using routine outcome measures as clinical process tools: Maximising the therapeutic yield in the IAPT programme when working remotely. Psychology and Psychotherapy, 95(3), 820–837. https://doi.org/10.1111/papt.12400
   PubMed Web of Science ®Google Scholar
• Furukawa, T. A., & Leucht, S. (2011). How to obtain NNT from Cohen's d: Comparison of two methods. PLoS ONE, 6(4), e19070. https://doi.org/10.1371/journal.pone.0019070
   PubMed Web of Science ®Google Scholar
• Gondek, D., Edbrooke-Childs, J., Fink, E., Deighton, J., & Wolpert, M. (2016). Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 325–343. https://doi.org/10.1007/s10488-015-0710-5
   PubMed Web of Science ®Google Scholar
• Greenhalgh, J., Gooding, K., Gibbons, E., Dalkin, S., Wright, J., Valderas, J., & Black, N. (2018). How do patient reported outcome measures (PROMs) support clinician-patient communication and patient care? A realist synthesis. Journal of Patient-Reported Outcomes, 2(1), 42. https://doi.org/10.1186/s41687-018-0061-6
   PubMedGoogle Scholar
• Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology: In Session, 61(2), 155–163. https://doi.org/10.1002/jclp.20108
   PubMed Web of Science ®Google Scholar
• Hepner, K. A., Farmer, C. M., Brooks Holliday, S., Bharil, S., Kimerling, R. E., McGee-Vincent, P., McCaslin, S. E., & Rosen, C. (2019). Displaying behavioral health measurement-based care data: Identifying key features from clinician and patient perspectives. RAND Corporation. https://www.rand.org/pubs/research_reports/RR3078.html
   Google Scholar
• Hilhorst, L., Stappen, J. V., Lokkerbol, J., Hiligsmann, M., Risseeuw, A. H., & Tiemens, B. G. (2022). Patients' and psychologists' preferences for feedback reports on expected mental health treatment outcomes: A discrete-choice experiment. Administration and Policy in Mental Health, 49(5), 707–721. https://doi.org/10.1007/s10488-022-01194-2
   PubMed Web of Science ®Google Scholar
• Hofmann, S. G., & Kashdan, T. B. (2010). The affective style questionnaire: Development and psychometric properties. Journal of Psychopathology and Behavioral Assessment, 32(2), 255–263. https://doi.org/10.1007/s10862-009-9142-4
   PubMed Web of Science ®Google Scholar
• Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy. Efficacy, effectiveness, and patient progress. American Psychologist, 51(10), 1059–1064. https://doi.org/10.1037//0003-066x.51.10.1059
   PubMed Web of Science ®Google Scholar
• Janse, P. D., De Jong, K., Veerkamp, C., van Dijk, M. K., Hutschemaekers, G., & Verbraak, M. (2020). The effect of feedback-informed cognitive behavioral therapy on treatment outcome: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 88(9), 818–828. https://doi.org/10.1037/ccp0000549
   PubMed Web of Science ®Google Scholar
• Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., & Hawley, K. M. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0
   PubMed Web of Science ®Google Scholar
• Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., Bhattacharya, A., Gimson, A., Brütt, A. L., De Jong, K., & Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. The Cochrane Database of Systematic Reviews, 7(7), CD011119. https://doi.org/10.1002/14651858.CD011119.pub2
   PubMed Web of Science ®Google Scholar
• Knapstad, M., Nordgreen, T., & Smith, O. R. (2018). Prompt mental health care, the Norwegian version of IAPT: Clinical outcomes and predictors of change in a multicenter cohort study. BMC Psychiatry, 18(1), 1–16. https://doi.org/10.1186/s12888-018-1838-0
   PubMedGoogle Scholar
• Knaup, C., Koesters, M., Schoefer, D., Becker, T., & Puschner, B. (2009). Effect of feedback of treatment outcome in specialist mental healthcare: Meta-analysis. The British Journal of Psychiatry, 195(1), 15–22. https://doi.org/10.1192/bjp.bp.108.053967
   PubMed Web of Science ®Google Scholar
• Koslofsky, S., & Rodíguez, M. M. D. (2017). Cultural adaptations to psychotherapy: Real-world applications. Clinical Case Studies, 16(1), 3–8. https://doi.org/10.1177/1534650116668273
   Web of Science ®Google Scholar
• Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
   Web of Science ®Google Scholar
• Krägeloh, C. U., Czuba, K. J., Billington, D. R., Kersten, P., & Siegert, R. J. (2015). Using feedback from patient-reported outcome measures in mental health services: A scoping study and typology. Psychiatric Services, 66(3), 224–241. https://doi.org/10.1176/appi.ps.201400141
   PubMed Web of Science ®Google Scholar
• Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, & feedback in clinical practice. American Psychological Association Press.
   Google Scholar
• Lambert, M. J., Bailey, R. J., White, M., Tingey, K. M., & Stevens, E. (2015). Clinical support tool manual (brief version-40). OQMeasures.
   Google Scholar
• Lambert, M. J., Kahler, M., Harmon, C., Burlingame, G. M., Shimokawa, K., & White, M. M. (2013). Administration and scoring manual: Outcome questionnaire OQ®-45.2. OQMeasures.
   Google Scholar
• Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301. https://doi.org/10.1093/clipsy.bpg025
   Web of Science ®Google Scholar
• Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55(4), 520–537. https://doi.org/10.1037/pst0000167
   PubMed Web of Science ®Google Scholar
• Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., Hoffman, M., Scott, K., Lyon, A., Douglas, S., Simon, G., & Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324–335. https://doi.org/10.1001/jamapsychiatry.2018.3329
   PubMed Web of Science ®Google Scholar
• Lutz, W., Böhnke, J. R., & Köck, K. (2011). Lending an ear to feedback systems: Evaluation of recovery and non-response in psychotherapy in a German outpatient setting. Community Mental Health Journal, 47(3), 311–317. https://doi.org/10.1007/s10597-010-9307-3
   PubMed Web of Science ®Google Scholar
• Lutz, W., Deisenhofer, A.-K., Rubel, J., Bennemann, B., Giesemann, J., Poster, K., & Schwartz, B. (2022). Prospective evaluation of a clinical decision support system in psychological therapy. Journal of Consulting and Clinical Psychology, 90(1), 90–106. https://doi.org/10.1037/ccp0000642
   PubMed Web of Science ®Google Scholar
• Lutz, W., Deisenhofer, A.-K., Weinmann-Lutz, B., & Barkham, M. (2023). Data and feedback-informed clinical training and practice. In L. G. Castonguay & C. E. Hill (Eds.), Becoming better therapists: Advancing training and supervision in psychotherapy. APA.
   Google Scholar
• Lutz, W., De Jong, K., Rubel, J. A., & Delgadillo, J. (2021). Measuring, predicting, and tracking change in psychotherapy. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change (7th ed., pp. 89–133). Wiley.
   Google Scholar
• Lutz, W., Leach, C., Barkham, M., Lucock, M., Stiles, W. B., Evans, C., Noble, R., & Iveson, S. (2005). Predicting rate and shape of change for individual clients receiving psychological therapy: Using growth curve modeling and nearest neighbor technologies. Journal of Consulting and Clinical Psychology, 73(5), 904–913. https://doi.org/10.1037/0022-006X.73.5.904
   PubMed Web of Science ®Google Scholar
• Lutz, W., Rubel, J., Schiefele, A. K., Zimmermann, D., Böhnke, J. R., & Wittmann, W. W. (2015). Feedback and therapist effects in the context of treatment outcome and treatment length. Psychotherapy Research, 25(6), 647–660. https://doi.org/10.1080/10503307.2015.1053553
   PubMed Web of Science ®Google Scholar
• Lutz, W., Rubel, J. A., Schwartz, B., Schilling, V., & Deisenhofer, A.-K. (2019). Towards integrating personalized feedback research into clinical practice: Development of the Trier Treatment Navigator (TTN). Behaviour Research and Therapy, 120, 103438. https://doi.org/10.1016/j.brat.2019.103438
   PubMed Web of Science ®Google Scholar
• Lutz, W., Schwartz, B., & Delgadillo, J. (2022). Measurement-based and data-informed psychological therapy. Annual Review of Clinical Psychology, 18(1), 71–98. https://doi.org/10.1146/annurev-clinpsy-071720-014821
   PubMedGoogle Scholar
• Mackrill, T., & Sørensen, K. M. (2020). Implementing routine outcome measurement in psychosocial interventions – a systematic review. European Journal of Social Work, 23(5), 790–808. https://doi.org/10.1080/13691457.2019.1602029
   Web of Science ®Google Scholar
• McAleavey, A. A., Nordberg, S. S., & Moltu, C. (2021). Initial quantitative development of the Norse feedback system: A novel clinical feedback system for routine mental healthcare. Quality of Life Research, 30(11), 3097–3115. https://doi.org/10.1007/s11136-021-02825-1
   PubMed Web of Science ®Google Scholar
• Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press.
   Google Scholar
• Mellor-Clark, J., Cross, S., Macdonald, J., & Skjulsvik, T. (2016). Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Administration and Policy in Mental Health, 43(3), 279–285. https://doi.org/10.1007/s10488-014-0587-8
   PubMed Web of Science ®Google Scholar
• Metz, M. J., Veerbeek, M. A., Twisk, J. W. R., van der Feltz-Cornelis, C. M., de Beurs, E., & Beekman, A. T. F. (2019). Shared decision-making in mental health care using routine outcome monitoring: Results of a cluster randomised-controlled trial. Social Psychiatry and Psychiatric Epidemiology, 54(2), 209–219. https://doi.org/10.1007/s00127-018-1589-8
   PubMed Web of Science ®Google Scholar
• Miller, S. D., Duncan, B. L., Brown, J., Sparks, J., & Claud, D. (2003). The outcome rating scale: A preliminary study of the reliability, validity, and feasibility of a brief visual analog measure. Journal of Brief Therapy, 2(2), 91–100. https://scottdmiller.com/wp-content/uploads/documents/OutcomeRatingScale-JBTv2n2.pdf
   Google Scholar
• Mütze, K., Witthöft, M., Lutz, W., & Bräscher, A. K. (2022). Matching research and practice: Prediction of individual patient progress and dropout risk for basic routine outcome monitoring. Psychotherapy Research, 32(3), 358–371. https://doi.org/10.1080/10503307.2021.1930244
   PubMed Web of Science ®Google Scholar
• Ogles, B. M., Goates-Jones, M. K., & Erekson, D. M. (2022). Treatment success or failure? Using a narrative interview to supplement ROM. Journal of Clinical Psychology, 78(10), 1986–2001. https://doi.org/10.1002/jclp.23345
   PubMed Web of Science ®Google Scholar
• Østergård, O. K., Randa, H., & Hougaard, E. (2020). The effect of using the partners for change outcome management system as feedback tool in psychotherapy-A systematic review and meta-analysis. Psychotherapy Research, 30(2), 195–212. https://doi.org/10.1080/10503307.2018.1517949
   PubMed Web of Science ®Google Scholar
• Pejtersen, J. H., Viinholt, B. C. A., & Hansen, H. (2020). Feedback-informed treatment: A systematic review and meta-analysis of the partners for change outcome management system. Journal of Counseling Psychology, 67(6), 723–735. https://doi.org/10.1037/cou0000420
   PubMed Web of Science ®Google Scholar
• Persons, J. B., Koerner, K., Eidelman, P., Thomas, C., & Liu, H. (2016). Increasing psychotherapists' adoption and implementation of the evidence-based practice of progress monitoring. Behaviour Research and Therapy, 76, 24–31. https://doi.org/10.1016/j.brat.2015.11.004
   PubMed Web of Science ®Google Scholar
• Peterson, A. P., & Fagan, C. (2017). Training the next generation in routine outcome monitoring: Current practices in psychology training clinics. Training and Education in Professional Psychology, 11(3), 182–189. https://doi.org/10.1037/tep0000148
   Web of Science ®Google Scholar
• Pinner, D. H., & Kivlighan, D. M. I. I. I. (2018). The ethical implications and utility of routine outcome monitoring in determining boundaries of competence in practice. Professional Psychology: Research and Practice, 49(4), 247–254. https://doi.org/10.1037/pro0000203
   Web of Science ®Google Scholar
• Rognstad, K., Wentzel-Larsen, T., Neumer, P.-S., & Kjøbli, J. (2022). A systematic review and meta-analysis of measurement feedback systems in treatment for common mental health disorders. Administration and Policy in Mental Health and Mental Health Services Research, 50(2), 269–282. https://doi.org/10.1007/s10488-022-01236-9
   PubMed Web of Science ®Google Scholar
• Rousmaniere, T. (2017). Deliberate practice for psychotherapists: A guide to improving clinical effectiveness. Routledge/Taylor & Francis Group.
   Google Scholar
• Rye, M., Rognmo, K., Aarons, G. A., & Skre, I. (2019). Attitudes towards the use of routine outcome monitoring of psychological therapies among mental health providers: The EBPAS-ROM. Administration and Policy in Mental Health, 46(6), 833–846. https://doi.org/10.1007/s10488-019-00968-5
   PubMed Web of Science ®Google Scholar
• Sales, C., Ashworth, M., Ayis, S., Barkham, M., Edbrooke-Childs, J., Faísca, L., Jacob, J., Xu, D., & Cooper, M. (2022). Idiographic patient reported outcome measures (I-PROMs) for routine outcome monitoring in psychological therapies: Position paper. Journal of Clinical Psychology, 79(3), 596–621. https://doi.org/10.1002/jclp.23319
   PubMed Web of Science ®Google Scholar
• Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153. https://doi.org/10.1002/jclp.20107
   PubMed Web of Science ®Google Scholar
• Schiepek, G., Aichhorn, W., Gruber, M., Strunk, G., Bachler, E., & Aas, B. (2016). Real-time monitoring of psychotherapeutic processes: Concept and compliance. Frontiers in Psychology, 7, 604. https://doi.org/10.3389/fpsyg.2016.00604
   PubMed Web of Science ®Google Scholar
• Schilling, V. N. L. S., Zimmermann, D., Rubel, J. A., Boyle, K. S., & Lutz, W. (2021). Why do patients go off track? Examining potential influencing factors for being at risk of psychotherapy treatment failure. Quality of Life Research, 30(11), 3287–3298. https://doi.org/10.1007/s11136-020-02664-6
   PubMed Web of Science ®Google Scholar
• Schöttke, H., Unrath, M., & Uhlmann, C. (2019). The effect of patient progress feedback on psychotherapy outcome. Verhaltenstherapie, 1–11. http://doi.org/10.1159/000503765
   Google Scholar
• Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311. https://doi.org/10.1037/a0019247
   PubMed Web of Science ®Google Scholar
• Simon, W., Lambert, M. J., Busath, G., Vazquez, A., Berkeljon, A., Hyer, K., Granley, M., & Berrett, M. (2013). Effects of providing patient progress feedback and clinical support tools to psychotherapists in an inpatient eating disorders treatment program: A randomized controlled study. Psychotherapy Research, 23(3), 287–300. https://doi.org/10.1080/10503307.2013.787497
   PubMed Web of Science ®Google Scholar
• Slone, N. C., Reese, R. J., Mathews-Duvall, S., & Kodet, J. (2015). Evaluating the efficacy of client feedback in group psychotherapy. Group Dynamics: Theory, Research, and Practice, 19(2), 122–136. https://doi.org/10.1037/gdn0000026
   Web of Science ®Google Scholar
• Solstad, S. M., Castonguay, L. G., & Moltu, C. (2019). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research, 29(2), 157–170. https://doi.org/10.1080/10503307.2017.1326645
   PubMed Web of Science ®Google Scholar
• Solstad, S. M., Kleiven, G. S., & Moltu, C. (2021). Complexity and potentials of clinical feedback in mental health: An in-depth study of patient processes. Quality of Life Research, 30(11), 3117–3125. https://doi.org/10.1007/s11136-020-02550-1
   PubMed Web of Science ®Google Scholar
• Swift, J. K., Callahan, J. L., Rousmaniere, T. G., Whipple, J. L., Dexter, K., & Wrape, E. R. (2015). Using client outcome monitoring as a tool for supervision. Psychotherapy, 52(2), 180–184. https://doi.org/10.1037/a0037659
   PubMed Web of Science ®Google Scholar
• Talib, T. L., DeChant, P., Kean, J., Monahan, P. O., Haggstrom, D. A., Stout, M. E., & Kroenke, K. (2018). A qualitative study of patients' perceptions of the utility of patient-reported outcome measures of symptoms in primary care clinics. Quality of Life Research, 27(12), 3157–3166. https://doi.org/10.1007/s11136-018-1968-3
   PubMed Web of Science ®Google Scholar
• Tam, H. E., & Ronan, K. (2017). The application of a feedback-informed approach in psychological service with youth: Systematic review and meta-analysis. Clinical Psychology Review, 55, 41–55. https://doi.org/10.1016/j.cpr.2017.04.005
   PubMed Web of Science ®Google Scholar
• Tasca, G. A., Angus, L., Bonli, R., Drapeau, M., Fitzpatrick, M., Hunsley, J., & Knoll, M. (2019). Outcome and progress monitoring in psychotherapy: Report of a Canadian psychological association task force. Canadian Psychology, 60(3), 165–177. https://doi.org/10.1037/cap0000181
   Web of Science ®Google Scholar
• The Joint Commission. (2018). R3 Report: Requirement, rationale, reference. A complimentary publication of The Joint Commission Issue 13, February 6. https://www.jointcommission.org/-/media/tjc/documents/standards/r3-reports/r3_13_outcome_measures_1_30_18_final.pdf
   Google Scholar
• Thew, G., Fountain, L., & Salkovskis, P. (2015). Service user and clinician perspectives on the use of outcome measures in psychological therapy. The Cognitive Behaviour Therapist, 8, E23. https://doi.org/10.1017/S1754470X15000598
   Google Scholar
• van Oenen, F. J., Schipper, S., Van, R., Schoevers, R., Visch, I., Peen, J., & Dekker, J. (2016). Feedback-informed treatment in emergency psychiatry; a randomised controlled trial. BMC Psychiatry, 16(1), 110. https://doi.org/10.1186/s12888-016-0811-z
   PubMedGoogle Scholar
• van Sonsbeek, M. A. M. S., Hutschemaekers, G. J. M., Veerman, J. W., Vermulst, A., Kleinjan, M., & Tiemens, B. G. (2021). Challenges in investigating the effective components of feedback from routine outcome monitoring (ROM) in youth mental health care. Child & Youth Care Forum, 50(2), 307–332. https://doi.org/10.1007/s10566-020-09574-1
   Web of Science ®Google Scholar
• Van Wert, M. J., Malik, M., Memel, B., Moore, R., Buccino, D., Hackerman, F., Kumari, S., Everett, A., & Narrow, W. (2021). Provider perceived barriers and facilitators to integrating routine outcome monitoring into practice in an urban community psychiatry clinic: A mixed-methods quality improvement project. Journal of Evaluation and Clinical Practice, 27(4), 767–775. https://doi.org/10.1111/jep.13457
   PubMed Web of Science ®Google Scholar
• Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110(2), 639–644. https://doi.org/10.2466/02.07.17
   PubMed Web of Science ®Google Scholar
• White, M. M., Lambert, M. J., Ogles, B. M., Mclaughlin, S. B., Bailey, R. J., & Tingey, K. M. (2015). Using the assessment for signal clients as a feedback tool for reducing treatment failure. Psychotherapy Research, 25(6), 724–734. https://doi.org/10.1080/10503307.2015.1009862
   PubMed Web of Science ®Google Scholar
• Winkeljohn Black, S., Owen, J., Chapman, N., Lavin, K., Drinane, J. M., & Kuo, P. (2017). Feedback informed treatment: An empirically supported case study of psychodynamic treatment. Journal of Clinical Psychology, 73(11), 1499–1509. https://doi.org/10.1002/jclp.22529
   PubMed Web of Science ®Google Scholar
• Wright, C. V., Goodheart, C., Bard, D., Bobbitt, B. L., Butt, Z., Lysell, K., McKay, D., & Stephens, K. (2020). Promoting measurement-based care and quality measure development: The APA mental and behavioral health registry initiative. Psychological Services, 17(3), 262–270. https://doi.org/10.1037/ser0000347
   PubMed Web of Science ®Google Scholar