Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions

References

• Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Consideration of organizational context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14, 255–271. http://dx.doi.org/10.1016/j.chc.2004.04.008
   PubMed Web of Science ®Google Scholar
• Adelman, R. (2005). Reducing adolescent clients' anger in a residential substance abuse treatment facility. Journal on Quality and Patient Safety, 31, 325–327.
   PubMedGoogle Scholar
• American Psychological Association. (2009). Criteria for the Evaluation of Quality Improvement Programs and the Use of Quality Improvement Data. American Psychologist, 64, 551–557. doi:10.1037/a0016744
   PubMed Web of Science ®Google Scholar
• Anker, M. G., Duncan, B. L., & Sparks, J.A. (2009). Using client feedback to improve couple therapy outcomes: a randomized clinical trial in a naturalistic setting. Journal of Consulting & Clinical Psychology, 77, 693–704. doi:10.1037/a0016062
   PubMed Web of Science ®Google Scholar
• Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C., … McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69, 184–196. doi:10.1037/0022-006X.69.2.184
   PubMed Web of Science ®Google Scholar
• Barkham, M., Mellor-Clark, J., Connell, J., & Cahill, J. (2006). A core approach to practice-based evidence: A brief history of the origins and applications of the CORE-OM and CORE System. Counselling & Psychotherapy Research, 6, 3–15. doi:10.1080/14733140600581218
   Google Scholar
• Bickman, L. (2000). Summing up program theory. New Directions for Evaluation, 103–112. doi:10.1002/ev.1186
   Google Scholar
• Bohanske, R. T., & Franczak, M. (2010). Transforming public behavioral health care: A case example of consumer-directed services, recovery, and the common factors. In B.L. Duncan, S.D. Miller, B.E. Wampold, & M.A. Hubble (Eds.), The heart and soul of change: Doing what works in therapy (2nd ed., pp. 299–322). Washington DC: American Psychological Association.
   Google Scholar
• Boswell, J. F., & Castonguay, L. G. (2007). Psychotherapy training: Suggestions for core ingredients and future research. Psychotherapy: Theory, Research, Practice, Training, 44, 378–383. doi:10.1037/0033-3204.44.4.378
   PubMed Web of Science ®Google Scholar
• Brownson, R. C., Kreuter, M. W., Arrington, B. A., & True, W. R. (2006). Translating scientific discoveries into public health action: How can schools of public health move us forward? Public Health Reports, 121, 97–103.
   PubMed Web of Science ®Google Scholar
• Castonguay, L. G., Barkham, M., Lutz, W., & McAleavey, A. A. (2013). Practice-oriented research: Approaches and applications. In M.J. Lambert (Ed.), Bergin and Garfield's handbook of therapeutic change (6th ed.). New York: Wiley & Sons.
   Google Scholar
• Castonguay, L. G., Boswell, J. F., Constantino, M. J., Goldfried, M.R., & Hill, C.E. (2010). Training implications of harmful effects of psychological treatments. American Psychologist, 65, 34–49. doi:10.1037/a0017330
   PubMed Web of Science ®Google Scholar
• Castonguay, L. G., Boswell, J. F., Zack, S. E., Baker, S., Boutselis, M. A., Chiswick, N. R. … Grosse Holtforth, M. (2010). Helpful and hindering events in psychotherapy: A practice research network study. Psychotherapy: Theory, Research, Practice, Training, 47, 327–344. doi:10.1037/a0021164
   PubMed Web of Science ®Google Scholar
• Castonguay, L. G., Pincus, A. L., & McAleavey, A. A. (in press). Practice-research networks in training clinics. Psychotherapy Research.
   Google Scholar
• Chin, R., & Benne, K. D. (1969). General strategies for effecting changes in human systems. In W.G. Bennis, K.D. Benne, & R. Chin (Eds.), The planning of change (pp. 32–59). New York: Holt, Rinehart & Winston.
   Google Scholar
• Cook, I. A., Balasubramani, G. K., Eng, H., Friedman, E., Young, E. A., Martin, J., … Wisniewski, S. R. (2007). Electronic source materials in clinical research: Acceptability and validity of symptom self-rating in major depressive disorder. Journal of Psychiatry Research, 41, 737–743. doi:10.1016/j.jpsychires.2006.07.015
   PubMed Web of Science ®Google Scholar
• De Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22, 464–474. doi:10.1080/10503307.2012.673023
   PubMed Web of Science ®Google Scholar
• Diamond, G. A., & Kaul, S. (2009). Evidence-based financial incentives for healthcare reform: Putting it together. Circulation: Cardiovascular quality and outcomes, 2, 134–140. doi:10.1161/CIRCOUTCOMES.108.825695
   PubMed Web of Science ®Google Scholar
• Drapeau, M. (2012). The value of tracking in psychotherapy. Integrating Science & Practice, 2, 2–6.
   Google Scholar
• Eby, L. T., Burk, H., & Maher, C. P. (2010). How serious of a problem is staff turnover in substance abuse treatment? A longitudinal study of actual turnover. Journal of Substance Abuse Treatment, 39, 264–271. doi:10.1016/j.jsat.2010.06.009
   PubMed Web of Science ®Google Scholar
• Ellsworth, J. R., Lambert, M. J., & Johnson, J. (2006). A comparison of the Outcome Questionnaire-45 and Outcome Questionnaire-30 in classification and prediction of treatment outcome. Clinical Psychology & Psychotherapy, 13, 380–391. doi:10.1002/cpp.503
   Web of Science ®Google Scholar
• Fernández-Álvarez, H., Gómez, B., & García, F. (in press). Bridging the gap between research and practice in a clinical and training network. Psychotherapy Research
   Google Scholar
• Finch, R. A., & Phillips, K. (2005). Center for Prevention and Health Services. An employer's guide to behavioral health services: A roadmap and recommendations for evaluating, designing, and implementing behavioral health services. Washington DC: National Business Group on Health.
   Google Scholar
• Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).
   Google Scholar
• Garb, H. N. (2005). Clinical judgment and decision making. Annual Review of Clinical Psychology, 1, 67–89. doi:10.1146/annurev.clinpsy.1.102803.143810
   PubMed Web of Science ®Google Scholar
• Gilbody, S., House, A., & Sheldon, T. (2002). Psychiatrists in the United Kingdom do not use outcomes measures. British Journal of Psychiatry, 180, 101–103. doi:10.1192/bjp.180.2.101
   PubMed Web of Science ®Google Scholar
• Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581–629. doi:10.1111/j.0887-378X.2004.00325.x
   PubMed Web of Science ®Google Scholar
• Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., & Sutton, S. W. (2005). A lab test and algorithms for identifying clients at risk for treatment failure. Journal of Clinical Psychology: In Session, 61, 155–163. doi:10.1002/jclp.20108
   PubMed Web of Science ®Google Scholar
• Hansen, N. B., Lambert, M. J., & Forman, E. V. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329–343. doi:10.1093/clipsy.9.3.329
   Web of Science ®Google Scholar
• Harford, T. (2011). Adapt: Why success always starts with failure. London: Macmillan.
   Google Scholar
• Harmon, S. C., Lambert, M. J., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Slade, K., & Lutz, W. (2007). Enhancing outcome for potential treatment failures: Therapist/client feedback and clinical support tools. Psychotherapy Research, 17, 379–392. doi:10.1080/10503300600702331
   Web of Science ®Google Scholar
• Hatfield, D., McCullough, L., Plucinski, A., & Krieger, K. (2010). Do we know when our clients get worse? An investigation of therapists' ability to detect negative client change. Clinical Psychology & Psychotherapy, 17, 25–32.
   PubMed Web of Science ®Google Scholar
• Hatfield, D. R., & Ogles, B. M. (2004). The current climate of outcome measures use in clinical practice. Professional Psychology: Research and Practice, 35, 485–491. doi:10.1037/0735-7028.35.5.485
   Web of Science ®Google Scholar
• Hatfield, D. R., & Ogles, B. M. (2007). Why some clinicians use outcome measures and others do not. Administrative Policy Mental Health and Mental Health Services Research, 34, 283–291. doi:10.1007/s10488-006-0110-y
   PubMed Web of Science ®Google Scholar
• Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14, 308–327. doi:10.1093/ptr/kph027
   Web of Science ®Google Scholar
• Holmqvist, R., Philips, B., & Barkham, M. (in press). Developing and delivering practice-based evidence: Observations, tensions, and challenges. Psychotherapy Research.
   Google Scholar
• Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy. American Psychologist, 51, 1059–1064. doi:10.1037/0003-066X.51.10.1059
   PubMed Web of Science ®Google Scholar
• Kazdin, A. E. (1996). Dropping out of child therapy. Issues for research and implications for practice. Clinical Child Psychology and Psychiatry, 1, 133–156. doi:10.1177/1359104596011012
   Google Scholar
• Kazdin, A. E. (2005). Evidence-based assessment for children and adolescents: Issues in measurement development and clinical application. Journal of Clinical Child & Adolescent Psychology, 34, 548–558. doi:10.1207/s15374424jccp3403_10
   PubMed Web of Science ®Google Scholar
• Kiefe, C. I., Allison, J. J., Williams, O. D., Person, S. D., Weaver, M. T., & Weissman, N. W. (2001). Improving quality improvement using achievable benchmarks for physician feedback: a randomized controlled trial. JAMA, 285, 2871–2879. doi:10.1001/jama.285.22.2871
   PubMed Web of Science ®Google Scholar
• Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254–284. doi:10.1037/0033-2909.119.2.254
   Web of Science ®Google Scholar
• Koestner, R. (2008). Reaching one's personal goals: A motivational perspective focused on autonomy. Canadian Psychology, 49, 60–67. doi:10.1037/0708-5591.49.1.60
   Web of Science ®Google Scholar
• Koocher, G. P., & Keith-Spiegel, P. (2008). Ethics in psychology and the mental health professions: Standards and cases. New York: Oxford University Press.
   Google Scholar
• Kraus, D. R., Boswell, J. F., Wright, A. G. C., Castonguay, L. G., & Pincus, A. L. (2010). Factor structure of the treatment outcome package for children. Journal of Clinical Psychology, 66, 627–640.
   PubMed Web of Science ®Google Scholar
• Kraus, D. R., Castonguay, L. G., Boswell, J. F., Nordberg, S. S., & Hayes, J. A. (2011). Therapist effectiveness: Implications for accountability and patient care. Psychotherapy Research, 21, 267–276. doi:10.1080/10503307.2011.563249
   PubMed Web of Science ®Google Scholar
• Kraus, D. R., Seligman, D., & Jordan, J. R. (2005). Validation of a behavioral health treatment outcome and assessment tool designed for naturalistic settings: The Treatment Outcome Package. Journal of Clinical Psychology, 61, 285–314. doi:10.1002/jclp.20084
   PubMed Web of Science ®Google Scholar
• Kraus, D. R., Wolf, A., & Castonguay, L. (2006). A kinder philosophy to the management of outcomes. Psychotherapy Bulletin, 41, 23–31.
   Google Scholar
• Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, and feedback in clinical practice. Washington DC: American Psychological Association Press.
   Google Scholar
• Lambert, M. J. (2012). Helping clinicians to use and learn from research-based systems: The OQ-Analyst. Psychotherapy, 49, 109–114. doi:10.1037/a0027110
   PubMed Web of Science ®Google Scholar
• Lambert, M. J. (2013). The efficacy and effectiveness of psychotherapy. In M. J. Lambert (Ed.) Bergin & Garfield's Handbook of psychotherapy & behavior change (6th ed.). New York: Wiley.
   Google Scholar
• Lambert, M. J., Burlingame, G. M., Umphress, V., Hansen, N. B., Vermeersch, D. A., Clouse, G. C., & Yanchar, S. C. (1996). The reliability and validity of the outcome questionnaire. Clinical Psychology & Psychotherapy, 3, 249–258. doi:10.1002/(SICI)1099-0879(199612)3:4<249::AID-CPP106>3.0.CO;2-S
   Web of Science ®Google Scholar
• Lambert, M. J. & Shimokawa, K. (2011). Collecting client feedback. In J.C. Norcross (Ed.), Psychotherapy relationships that work (2nd ed., pp. 203–223). New York: Oxford University Press.
   Google Scholar
• Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, G. V., & Finch, A. E. (2002). Comparison of empirically derived and rationally derived methods for identifying clients at risk for treatment failure. Clinical Psychology and Psychotherapy, 9, 149–164. doi:10.1002/cpp.333
   Web of Science ®Google Scholar
• Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S.L., & Hawkins, E.J. (2001). The effects of providing therapists with feedback on client progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49–68. doi:10.1080/713663852
   PubMed Web of Science ®Google Scholar
• Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., & Goates, M. K. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology and Psychotherapy, 9, 91–103. doi:10.1002/cpp.324
   Web of Science ®Google Scholar
• Latham, G. P., & Locke, E. A. (2006). Enhancing the benefits and overcoming the pitfalls of goal setting. Organizational Dynamics, 35, 332–340. doi:10.1016/j.orgdyn.2006.08.008
   Web of Science ®Google Scholar
• Locke, E. A., & Latham, G. P. (2006). New directions in goal-setting theory. Current Directions in Psychological Science, 15, 265–268. doi:10.1111/j.1467-8721.2006.00449.x
   Web of Science ®Google Scholar
• Marty, D., Rapp, C. A., McHugo, G. J., & Whitley, R. (2007). Factors influencing consumer outcome monitoring in implementation of evidence-based practices: Results from the National EBP Implementation Project. Administration and Policy in Mental Health, 35, 204–211. doi:10.1007/s10488-007-0157-4
   PubMed Web of Science ®Google Scholar
• McAleavey, A. A., Lockard, A. J., Castonguay, L. G., Hayes, J. A., & Locke, B.D. (in press). Research by, with, and for clinicians: Lessons from a large-scale PRN for University and College Counseling Centers. Psychotherapy Research.
   Google Scholar
• McAleavey, A. A., Nordberg, S. S., Kraus, D. R., & Castonguay, L.G. (2012). Errors in treatment outcome monitoring: Implications of multidimensional and general measurements for real-world psychotherapy. Canadian Psychology, 53, 105–114. doi:10.1037/a0027833
   Web of Science ®Google Scholar
• McHugh, R. K., & Barlow, D. H. (2012). Dissemination and implementation of evidence-based psychological interventions. New York: Oxford University Press.
   Google Scholar
• Merten, T., & Ruch, W. (1996). A comparison of computerized and conventional administration of the German versions of the Eysenck Personality Questionnaire and the Carroll Rating Scale for Depression. Personality and Individual Differences, 20, 281–291. http://dx.doi.org/10.1016/0191-8869(95)00185-9
   Web of Science ®Google Scholar
• Miller, S. D., Duncan, B. L., & Hubble, M. A. (2004). Beyond integration: The triumph of outcome over process in clinical practice. Psychotherapy in Australia, 10, 32–43.
   Google Scholar
• Miller, S. D., Duncan, B. L., Sorrell, R., & Brown, G. S. (2005). The Partners for Change Outcome Management System. Journal of Clinical Psychology: In-Session, 61, 199–208. doi:10.1002/jclp.20111
   PubMed Web of Science ®Google Scholar
• Miller, S. D., Hubble, M. A., Chow, D. L., & Seidel, J. A. (2013). The outcome of psychotherapy: Yesterday, today, and tomorrow. Psychotherapy, 50, 88–97. doi:10.1037/a0031097
   PubMed Web of Science ®Google Scholar
• Nordberg, S. S., Boswell, J. F., Kraus, D. R., Castonguay, L. G., Hayes, J., & Wampold, B. (2010, June). Therapist effectiveness treating depression with and without co-morbid substance abuse. Paper presented at the Society for Psychotherapy Research Conference, Asilomar, CA.
   Google Scholar
• Ogles, B. M., France, C. R., Lunnen, K. M., Bell, M. T., & Goldfarb, M. (1998). Computerized depression screening and awareness. Community Mental Health Journal, 34, 27–38. doi:10.1023/A:1018760128239
   PubMed Web of Science ®Google Scholar
• Okiishi, J., Lambert, M. J., Eggett, D., Nielsen, S. L., Dayton, D.D., & Vermeersch, D.A. (2006). An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients’ psychotherapy outcome; Journal of Clinical Psychology, 62, 1157–1172. doi:10.1002/jclp.20272
   PubMed Web of Science ®Google Scholar
• Peterson, L., Johannsson, V., & Carlsson, S. (1996). Computerized testing in a hospital setting: Psychometric and psychological effects. Computers in Human Behavior, 12, 339–350. http://dx.doi.org/10.1016/0747-5632(96)00012-X
   Web of Science ®Google Scholar
• Pope, K. S., Tabachnick, B. G., & Keith-Spiegel, P. (1987). Ethics of practice: The beliefs and behaviors of psychologists as therapists. American Psychologist, 42, 993–1006. doi:10.1037/0003-066X.42.11.993
   PubMed Web of Science ®Google Scholar
• Riemer, M., Rosof-Williams, J., & Bickman, L. (2005). Theories related to changing clinician practice. Child and Adolescent Psychiatric Clinics of North America, 14, 241–254. doi:10.1016/j.chc.2004.05.002
   PubMed Web of Science ®Google Scholar
• Rogers, E. (2003). The diffusion of innovation (5th ed.). New York: The Free Press.
   Google Scholar
• Rosenthal, M. B. (2008). Beyond pay for performance—emerging models of provider-payment reform. New England Journal of Medicine, 359, 1197–1200. doi:10.1056/NEJMp0804658
   PubMed Web of Science ®Google Scholar
• Shimokawa, K., Lambert, M. J., & Smart, D. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting & Clinical Psychology, 78, 298–311. doi:10.1037/a0019247
   PubMed Web of Science ®Google Scholar
• Slade, K., Lambert, M. J., Harmon, S. C., Smart, D. W., & Bailey, R. (2008). Improving psychotherapy outcome: The use of immediate electronic feedback and revised clinical support tools. Clinical Psychology & Psychotherapy, 15, 287–303. doi:10.1002/cpp.594
   PubMed Web of Science ®Google Scholar
• Slade, M., Thornicroft, G., & Glover, G.S.O. (1999). The feasibility of routine outcome measures in mental health. Social Psychiatry and Epidemiology, 34, 243–249. doi:10.1007/s001270050139
   PubMed Web of Science ®Google Scholar
• Strauss, B., Lutz, W., Steffanowski, A., Wittmann, W., Böhnke, J.R., Rubel, J., … Kirchmann, H. (in press). Benefits and challenges in practice-oriented psychotherapy research in Germany: The TK and the QS-PSY-BAY projects of quality assurance in outpatient psychotherapy. Psychotherapy Research.
   Google Scholar
• Stricker, G., & Trierweiler, S. J. (1995). The local clinical scientist: A bridge between science and practice. American Psychologist, 50, 995–1002. doi:10.1037/0003-066X.50.12.995
   PubMed Web of Science ®Google Scholar
• Strupp, H. H., Horowitz, L. M., & Lambert, M. J. (1997). Measuring patient changes in mood, anxiety, and personality disorders: Toward a core battery. Washington DC: American Psychological Association Press.
   Google Scholar
• Trauer, T., Gill, L., Pedwell, G., & Slattery, P. (2006). Routine outcome measurement in public mental health—what do clinicians think? Australian Health Review, 30, 144–147. doi:10.1071/AH060144
   PubMedGoogle Scholar
• Tremblay, M. C., Hevner, A. R., & Berndt, D. J. (2012). Design of an information volatility measure for health care decision making. Decision Support Systems, 52, 331–341. doi:10.1016/j.dss.2011.08.009
   Web of Science ®Google Scholar
• Walfish, S., McAlister, B., O'Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110, 639–644. doi:10.2466/02.07.17
   PubMed Web of Science ®Google Scholar
• Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., & Burlingame, G.M. (2010). Youth psychotherapy change trajectories & outcome in usual care: Community mental health versus managed care. Journal of Clinical & Consulting Psychology, 78, 144–155. doi:10.1037/a0018544
   PubMed Web of Science ®Google Scholar
• Weissman, N. W., Allison, J. J., Kiefe, C. I., Farmer, R., Weaver, M.T., Williams, O.D., … Baker, S. (1999). Achievable benchmarks of care: The ABCs of benchmarking. Journal of Evaluation in Clinical Practice, 5, 269–281. doi:10.1046/j.1365-2753.1999.00203.x
   PubMed Web of Science ®Google Scholar
• West, J. C., Mościcki, E. K., Duffy, F. F., Wilk, J.E., Countis, L., Narrow, W.E., & Regier, D.A. (in press). APIRE Practice Research Network: Accomplishments, challenges, and lessons learned. Psychotherapy Research.
   Google Scholar
• Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S.L., & Hawkins, E.J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment failure and problem solving strategies in routine practice. Journal of Counseling Psychology, 58, 59–68. doi:10.1037/0022-0167.50.1.59
   Web of Science ®Google Scholar
• Wierzbicki, M. & Pekarik, G. (1993). A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice, 24, 190–195. doi:10.1037/0735-7028.24.2.190
   Web of Science ®Google Scholar
• Youn, S.J., Kraus, D. R., & Castonguay, L.G. (2012). The Treatment Outcome Package: Facilitating practice and clinically relevant research. Psychotherapy, 49, 115–122. doi:10.1037/a0027932
   PubMed Web of Science ®Google Scholar
• Zimmerman, M., & McGlinchey, J. B. (2008). Why don't psychiatrists use scales to measure outcome when treating depressed patients? Journal of Clinical Psychiatry, 69, 1916–1919.
   PubMed Web of Science ®Google Scholar