The Psychometric Properties of the Outcome Rating Scale Used in Practice: A Narrative Review

References

• Abell, N., Springer, D. W., & Kamata, A. (2009). Developing and validating rapid assessment instruments. New York, NY: Oxford University Press, Inc.
   Google Scholar
• American Medical Directors Association. (2000). White paper on hospice in long term care. Retrieved from https://paltc.org/sites/default/files/hospiceinltc.pdf.
   Google Scholar
• Anker, M. G., Duncan, B. L., & Sparks, J. A. (2009). Using client feedback to improve couple therapy outcomes: A randomized clinical trial in a naturalistic setting. Journal of Counseling and Clinical Psychology, 77, 693–704. doi:10.1037/a0016062
   PubMed Web of Science ®Google Scholar
• Anker, M. G., Owen, J., Duncan, B. L., & Sparks, J. A. (2010). The alliance in couple therapy: Partner influence, early change, and alliance patterns in a naturalistic sample. Journal of Counseling and Clinical Psychology, 78, 635–645. doi:10.1037/a0020051
   PubMed Web of Science ®Google Scholar
• Atzil-Slonim, D., Bar-Kalifa, E., Fisher, H., Peri, T., Lutz, W., Rubel, J., & Rafaeli, E. (2018). Emotional congruence between clients and therapists and its effect on treatment outcome. Journal of Counseling Psychology, 65, 51–64. doi:10.1037/cou0000250
   PubMed Web of Science ®Google Scholar
• Bardhoshi, G., & Erford, B. T. (2017). Processes and procedures for estimating score reliability and precision. Measurement and Evaluation in Counseling and Development, 50, 256–263. doi:10.1080/07481756.2017.1388680
   Web of Science ®Google Scholar
• Baumann, E. C., & Hill, C. E. (2016). Client concealment and disclosure of secrets in outpatient psychotherapy. Counseling Psychology Quarterly, 29, 53–75. doi:10.1080/09515070.2015.1023698
   Web of Science ®Google Scholar
• Bijur, P. E., Silver, W., & Gallagher, E. J. (2001). Reliability of the visual analog scale for measurement of acute pain. Academy of Emergency Medicine, 8, 1153–1157. doi:10.1111/j.1553-2712.2001.tb01132.x
   PubMed Web of Science ®Google Scholar
• Bringhurst, D. L., Watson, C. W., Miller, S. D., & Duncan, B. L. (2006). The reliability and validity of the outcome rating scale: A replication study of a brief clinical measure. Journal of Brief Therapy, 5, 23–30.
   Google Scholar
• Burckhardt, C. S., & Anderson, K. L. (2003). The quality of life scale (QOLS): Reliability, validity and utilization. Health and Quality of Life Outcomes, 60, 1–7. doi:10.1186/1477-7525-1-60
   Google Scholar
• Campbell, A., & Hemsley, S. (2009). Outcome rating scale and session rating scale in psychological practice: Clinical utility of ultra-brief measures. Clinical Psychologist, 13, 1–9. doi:10.1080/13284200802676391
   Web of Science ®Google Scholar
• Charter, R. A. (2003). A breakdown of reliability coefficients by test type and reliability method, and the clinical implications of low reliability. The Journal of General Psychology, 130, 290–304. doi:10.1080/00221300309601160
   PubMed Web of Science ®Google Scholar
• Cook, D. A. (2015). Much ado about differences: Why expert-novice comparisons add little to the validity argument. Advances in Health Science Education, 20, 829–834. doi:10.1007/s10459-014-9551-3
   PubMed Web of Science ®Google Scholar
• Cranford, J. A., Shrout, P. E., Iida, M., Rafaeli, E., Yip, T., & Bolger, N. (2006). A procedure for evaluating sensitivity to within person change: Can mood measures in diary studies detect change reliably? Personality Social Psychology Bulletin, 32, 917–929. doi:10.1007/s10459-014-9551-3
   PubMed Web of Science ®Google Scholar
• DeSantis, B., Jackson, M. J., Duncan, B. L., & Reese, R. J. (2017). Casting a wider net in behavioral health screening in primary care: A preliminary study of the outcome rating scale. Primary Health Care Research & Development, 18, 188–193. doi:10.1017/S1463423616000311
   PubMed Web of Science ®Google Scholar
• Duncan, B. (2012). The Partners for Change Outcome Management System (PCOMS): The heart and soul of change project. Canadian Psychology, 53, 93–104. doi:10.1037/a0027762
   Web of Science ®Google Scholar
• Duncan, B. L., Miller, S. D., Sparks, J. A., Claud, D. A., Reynolds, L. R., Brown, J., & Johnson, L. D. (2003). The session rating scale: Preliminary psychometric properties of a “working” alliance measure. Journal of Brief Therapy, 3, 3–12.
   Google Scholar
• Duncan, B. L., Sparks, J. A., Miller, S. D., Bohanske, R. T., & Claud, D. A. (2006). Giving youth a voice: A preliminary study of the reliability and validity of a brief outcome measure for children, adolescents, and caretakers. Journal of Brief Therapy, 5, 71–88.
   Google Scholar
• Fisher, H., Slonim, D., Bar-Kalifa, E., & Peri, T. (2016). Emotional experience and alliance contribute to therapeutic change in psychodynamic therapy. Psychotherapy, 53, 105–116. doi:10.1037/pst0000041
   PubMed Web of Science ®Google Scholar
• Frost, M. H., Reeve, B. B., Liepa, A. M., Stauffer, J. W., & Hays, R. D.; the Mayo/FDA PROCMG. (2007). What is sufficient evidence for the reliability and validity of patient-reported outcome measures? Value in Health, 10, S94–S105. doi:10.1111/j.1524-4733.2007.00272.x
   PubMed Web of Science ®Google Scholar
• Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice, 44, 231–246. doi:10.1037/a0032605
   Web of Science ®Google Scholar
• Gould, D., Kelly, D., Goldstone, L., & Gammon, J. (2001). Examining the validity of pressure ulcer risk assessment scales: Developing and illustrated patient simulations to collect the data. Journal of Clinical Nursing, 10, 697–706.
   PubMed Web of Science ®Google Scholar
• Grossl, A. B., Reese, R. J., Norsworthy, L. A., & Hopkins, N. B. (2014). Client feedback data in supervision: Effects on supervision and outcome. Training and Education in Professional Psychology, 8, 182–188. doi:10.1037/tep0000025
   Web of Science ®Google Scholar
• Hacohen, N., Atzil-Slonim, D., Tuval-Mashiach, R., Bar-Kalifa, E., & Fisher, H. (2017). Multiplicity and mutuality in the transition of patient and therapist’s self-states: Comparison of good vs. poor outcome groups. Psychotherapy Research, 29, 1–14. doi:10.1080/10503307.2017.141162
   Web of Science ®Google Scholar
• Hafkenscheid, A., Duncan, B. L., & Miller, S. D. (2010). The outcome and session rating scales: A cross-cultural examination of the psychometric properties of the Dutch translation. Journal of Brief Therapy, 7, 1–12.
   Google Scholar
• Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35, 485–491. doi:10.1037/0735-7028.35.5.485
   Web of Science ®Google Scholar
• Hawker, G. A., Mian, S., Kendzerska, T., & French, M. (2011). Measures of adult pain: Visual Analog Scale for Pain (VAS Pain), Numeric Rating Scale for pain (NRS pain), McGill Pain Questionnaire (MPQ), Short-Form McGill Pain Questionnaire (SF-MPQ), Chronic Pain Grade Scale (CPGS), Short Form-36 Bodily Pain Scale (SF-36 BPS), and Measure of Intermittent and Constant Osteoarthritis Pain (ICOAP). Measures of Pathology and Symptoms, 63, S240–S252. doi:10.1002/acr.20543
   Google Scholar
• Higginson, I. J., & Carr, A. J. (2001). Measuring quality of life: Using quality of life measures in the clinical setting. The British Medical Journal, 322, 1297–1300. doi:10.1136/bmj.322.7297.1297
   PubMed Web of Science ®Google Scholar
• Klimek, L., Bergmann, K., Biedermann, T., Bousquet, J., Hellings, P., Jung, K., … Pfaar, O. (2017). Visual analogue scales (VAS): Measuring instruments for the documentation of symptoms and therapy monitoring in cases of allergic rhinitis in everyday healthcare. Allegro Journal International, 26, 16–24. doi:10.1007/s40629-016-0006-7
   PubMedGoogle Scholar
• Kuhlman, I., Tolvanen, A., & Seikkula, J. (2013). Couple therapy for depression within a Naturalistic setting in Finland: Factors related to change of the patient and the spouse. Contemporary Family Therapy, 35, 656–672. doi:10.1007/s10591-013-9246-6
   Google Scholar
• Lalande, L., King, R., Bambling, M., & Schweitzer, R. D. (2017). An uncontrolled clinical trial of guided respiration mindfulness therapy (GRMT) in the treatment of depression and anxiety. Journal of Contemporary Psychotherapy, 47, 251–258. doi:10.1007/s10879-017-9360-0
   Web of Science ®Google Scholar
• Lambert, M. J., Burlingame, G. M., Umphress, V., Hansen, N. B., Vermeersch, D. A., Clouse, G. C., & Yanchar, S. C. (1996). The reliability and validity of the outcome questionnaire. Clinical Psychology and Psychotherapy, 3, 249–258. doi:10.1002/(SICI)1099-0879(199612)3:4<249::AID-CPP106>3.0.CO;2-S
   Web of Science ®Google Scholar
• Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., … Kroenek, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76, 324–335. doi:10.1001/jamapsychiatry.2018.3329
   PubMed Web of Science ®Google Scholar
• Lovibond, P. F., & Lovibond, S. H. (1995). The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the beck depression and anxiety inventories. Behavior Research and Therapy, 33, 335–343. doi:10.1016/0005-7967(94)00075-U
   PubMed Web of Science ®Google Scholar
• Mikeal, C. W., Gillaspy, J. A., Jr., Scoles, M. T., & Murphy, J. J. (2016). A dismantling study of the partners for change outcome management system. Journal of Counseling Psychology, 63, 704–709. doi:10.1037/cou0000168
   Web of Science ®Google Scholar
• Miller, S. D., Duncan, B. L., Brown, J., Sparks, J. A., & Claud, D. A. (2003). The outcome rating scale: A preliminary study on the reliability, validity and feasibility of a brief visual analog measure. Journal of Brief Therapy, 2, 91–100.
   Google Scholar
• Miller, S. D., Hubble, M. A., Chow, D. L., & Seidel, J. A. (2013). The outcome of psychotherapy: Yesterday, today and tomorrow. Psychotherapy, 50, 88–97. doi:10.1037/a0031097
   PubMed Web of Science ®Google Scholar
• Mueller, R. M., Lambert, M. J., & Burlingame, G. M. (1998). Construct validity of the outcome questionnaire: A confirmatory factor analysis. Journal of Personality Assessment, 70, 248–262. doi:10.1207/s15327752jpa7002_5
   Web of Science ®Google Scholar
• Nezlek, J. B. (2017). A practical guide to understanding reliability in studies of within-person variability. Journal of Research in Personality, 69, 149–155. doi:10.1016/j.jrp.2016.06.020
   Web of Science ®Google Scholar
• Overington, L., & Ionita, G. (2012). Progress monitoring measures: A brief guide. Canadian Psychology, 53, 82–92. doi:10.1037/a0028017
   Web of Science ®Google Scholar
• Owen, J., Duncan, B., Anker, M., & Sparks, J. (2012). Initial relationship goal and couple therapy outcomes at post and six month follow up. Journal of Family Psychology, 26, 179–186. doi:10.1037/a0026998
   Web of Science ®Google Scholar
• Owen, J., Duncan, B., Reese, R. J., Anker, M., & Sparks, J. (2014). Accounting for variability in couple therapy outcomes: What really matters? Journal of Sex and Marital Therapy, 40, 488–502. doi:10.1080/0092623X.2013.772552
   PubMed Web of Science ®Google Scholar
• Owen, J., Miller, S. D., Seidel, J., & Chow, D. (2016). The working alliance in treatment of military adolescents. Journal of Consulting and Clinical Psychology, 84, 1–12. doi:10.1037/ccp0000035
   Web of Science ®Google Scholar
• Reese, R. J., Gillaspy, J. A., Owen, J. J., Flora, K. L., Cunningham, D. A., & Marsden, T. (2013). The influence of demand characteristics and social desirability on clients’ ratings of the therapeutic alliance. Journal of Clinical Psychology, 69, 696–709. doi:10.1002/jclp.21946
   PubMed Web of Science ®Google Scholar
• Reese, R. J., Norsworthy, L. A., & Rowlands, S. R. (2009b). Does a continuous feedback system improve psychotherapy outcome? Psychotherapy, 46, 418–431. doi:10.1037/a0017901
   PubMed Web of Science ®Google Scholar
• Reese, R. J., Toland, M., Sloane, N. C., & Norsworthy, L. A. (2010). Effect of client feedback on couple psychotherapy outcomes. Psychotherapy, 47, 616–630. doi:10.1037/a0021182
   PubMed Web of Science ®Google Scholar
• Reese, R. J., Usher, E. L., Bowman, D. C., Norsworthy, L. A., Halstead, J. L., Rowlands, S. R., & Chisholm, R. R. (2009a). Using client feedback in psychotherapy training: An analysis of its influence on supervision and counselor self-efficacy. Training and Education in Professional Psychology, 3, 157–168. doi:10.1037/a0015673
   Web of Science ®Google Scholar
• Seidel, J., Andrews, W. P., Owen, J., Miller, S. D., & Buccino, D. L. (2017). Preliminary validation of the rating of outcome scale and equivalence of ultra-brief measures of well-being. Psychological Assessment, 29, 65–75. doi:10.1037/pas0000311
   PubMed Web of Science ®Google Scholar
• Shaw, S., & Murray, K. (2014). Monitoring alliance and outcome with client feedback measures. Journal of Mental Health Counseling, 36, 43–57. doi:10.17744/mehc.36.1.n5g64t3014231862
   Google Scholar
• Sloane, N., Matthews-Duvall, S., Reese, R., & Kodet, J. (2015). Evaluating the efficacy of client feedback in group psychotherapy. Group Dynamics, Theory, Research and Practice, 19, 122–136. doi:10.17744/mehc.36.1.n5g64t3014231862
   Web of Science ®Google Scholar
• Souza, A. C., Alexandre, N. M., & Guirardello, E. (2017). Psychometric properties in instruments evaluation of reliability and validity. Applications of Epidemiology, 26, 649–659. doi:10.5123/S1679-49742017000300022
   Google Scholar
• Sullivan, G. M. (2011). A primer on the validity of assessment instruments. Journal of Graduate Medical Education, 3, 119–120. doi:10.4300/JGME-D-11-00075.1
   PubMedGoogle Scholar
• Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. doi:10.5116/ijme.4dfb.8dfd
   PubMedGoogle Scholar
• Terwee, C. B., Bot, S. D., de Boer, M. R., Windt, D. A., Knol, D. L., Bouter, L. M., & de Vet, H. C. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60, 34–42. doi:10.1016/j.jclinepi.2006.03.012
   PubMed Web of Science ®Google Scholar
• Torrey, W. C., Mueser, K. T., Meltugo, G. H., & Drake, R. E. (2000). Self-esteem as an outcome measure in studies of vocational rehabilitation for adults with severe mental illness. Psychiatric Services, 51, 229–233. doi:10.1176/appi.ps.51.2.229
   PubMed Web of Science ®Google Scholar
• Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). Reliability coefficients and generalizability theory. Handbook of Statistics, 26, 81–124.
   Google Scholar