Visualizing Agreement: Bland–Altman Plots as a Supplement to Inter-Rater Reliability Indices

References

• Altman, D. G., & Bland, J. M. (1983). Measurement in medicine: The analysis of method comparison studies. Journal of the Royal Statistical Society, 32(3), 307–317. https://doi.org/10.2307/2987937
   Google Scholar
• Ardito, R. B., & Rabellino, D. (2011). Therapeutic alliance and outcome of psychotherapy: Historical excursus, measurements, and prospects for research. Frontiers in Psychology, 2, 1–11. https://doi.org/10.3389/fpsyg.2011.00270
   PubMed Web of Science ®Google Scholar
• Beck, J. S. (2011). Cognitive behavior therapy: Basics and beyond (2nd ed.). Guilford Press.
   Google Scholar
• Bland, J. M., & Altman, D. G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571–582. https://doi.org/10.1080/10543400701329422
   PubMed Web of Science ®Google Scholar
• Bland, M. (2015). How can i decide the sample size for a study of agreement between two methods of measurement? https://www-users.york.ac.uk/~mb55/meas/sizemeth.htm
   Google Scholar
• Bobak, C. A., Barr, P. J., & O’Malley, A. J. (2018). Estimation of an inter-rater intra-class correlation coefficient that overcomes common assumption violations in the assessment of health measurement scales. BMC Medical Research Methodology, 18(1), 93–93. https://doi.org/10.1186/s12874-018-0550-6
   PubMedGoogle Scholar
• Carkeet, A. (2015). Exact parametric confidence intervals for Bland-Altman limits of agreement. Optometry and Vision Science, 92(3), 71–80. https://doi.org/10.1097/OPX.0000000000000513
   Web of Science ®Google Scholar
• Carter, J. D., McIntosh, V. V. W., Jordan, J., Porter, R. J., Douglas, K., Frampton, C. M., & Joyce, P. R. (2018). Patient predictors of response to cognitive behaviour therapy and schema therapy for depression. Australian & New Zealand Journal of Psychiatry, 52(9), 887–897. https://doi.org/10.1177/0004867417750756
   PubMed Web of Science ®Google Scholar
• Carter, J. D., McIntosh, V. V. W., Jordan, J., Porter, R. J., Frampton, C. M., & Joyce, P. R. (2013). Psychotherapy for depression: A randomized clinical trial comparing schema therapy and cognitive behavior therapy. Journal of Affective Disorders, 151(2), 500–505. https://doi.org/10.1016/j.jad.2013.06.034
   PubMed Web of Science ®Google Scholar
• Costa-Santos, C., Bernardes, J., Ayres de Campos, D., Costa, A., & Costa, C. (2011). The limits of agreement and the intraclass correlation coefficient may be inconsistent in the interpretation of agreement. Journal of Clinical Epidemiology, 64(3), 264–269. https://doi.org/10.1016/j.jclinepi.2009.11.010
   PubMed Web of Science ®Google Scholar
• Datta, D. (2017). blandr: A Bland-Altman method comparison package for R (Version 0.5.3). https://github.com/deepankardatta/blandr
   Google Scholar
• Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24(4), 295–300. https://doi.org/10.1111/j.2044-8260.1985.tb00662.x
   PubMedGoogle Scholar
• Doğan, N. Ö. (2018). Bland-Altman analysis: A paradigm to understand correlation and agreement. Turkish Journal of Emergency Medicine, 18(4), 139–141. https://doi.org/10.1016/j.tjem.2018.09.001
   PubMed Web of Science ®Google Scholar
• Fairburn, C. G., Marcus, M. D., & Wilson, G. T. (1993). Cognitive-behavioural therapy for binge eating and bulimia nervosa: A comprehensive treatment manual. In C. G. Fairburn & G. T. Wilson (Eds.), Binge eating: Nature, assessment and treatment (pp. 361–406). Guilford Press.
   Google Scholar
• Giavarina, D. (2015). Understanding Bland Altman analysis. Biochemia Medica, 25(2), 141–151. https://doi.org/10.11613/BM.2015.015
   PubMed Web of Science ®Google Scholar
• Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023
   PubMedGoogle Scholar
• Hartley, D. E., & Strupp, H. H. (1983). The therapeutic alliance: Its relationship to outcome in brief psychotherapy. In J. Masling (Ed.), Empirical studies of psychoanalytical theories (Vol. 1, pp. 1–38). Analytical Press.
   Google Scholar
• Hopkins, W. G. (2000). Measures of reliability in sports medicine and science. Sports Medicine, 30(1), 1–15. https://doi.org/10.2165/00007256-200030010-00001
   PubMed Web of Science ®Google Scholar
• Jordan, J., McIntosh, V. V. W., Carter, J. D., Rowe, S., Taylor, K., Frampton, C. M. A., McKenzie, J. M., Latner, J., & Joyce, P. R. (2014). Bulimia nervosa‐nonpurging subtype: Closer to the bulimia nervosa‐purging subtype or to binge eating disorder? International Journal of Eating Disorders, 47(3), 231–238. https://doi.org/10.1002/eat.22218
   PubMed Web of Science ®Google Scholar
• Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). Dyadic data analysis. Guilford Press.
   Google Scholar
• Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
   PubMed Web of Science ®Google Scholar
• Koyama, S., Tanabe, S., Itoh, N., Saitoh, E., Takeda, K., Hirano, S., Ohtsuka, K., Mukaino, M., Yanohara, R., Sakurai, H., & Kanada, Y. (2018). Intra- and inter-rater reliability and validity of the tandem gait test for the assessment of dynamic gait balance. European Journal of Physiotherapy, 20(3), 135–140. https://doi.org/10.1080/21679169.2017.1414304
   Web of Science ®Google Scholar
• Krupnick, J. L., Sotsky, S. M., Simmens, S., Moyer, J., Elkin, I., Watkins, J., & Pilkonis, P. A. (1996). The role of the therapeutic alliance in psychotherapy pharmacotherapy outcome: Findings in the national institute of mental health treatment of depression collaborative research program. Journal of Consulting and Clinical Psychology, 64(3), 532–539. https://doi.org/10.1037/0022-006X.64.3.532
   PubMed Web of Science ®Google Scholar
• LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852. https://doi.org/10.1177/1094428106296642
   Web of Science ®Google Scholar
• Lu, M.-J., Zhong, W.-H., Liu, Y.-X., Miao, H.-Z., Li, Y.-C., & Ji, M.-H. (2016). Sample size for assessing agreement between two methods of measurement by Bland−Altman method. The International Journal of Biostatistics, 12(2), 20150039. https://doi.org/10.1515/ijb-2015-0039
   Google Scholar
• Ludbrook, J. (2010). Confidence in Altman–Bland plots: A critical review of the method of differences. Clinical and Experimental Pharmacology and Physiology, 37(2), 143–149. https://doi.org/10.1111/j.1440-1681.2009.05288.x
   PubMed Web of Science ®Google Scholar
• McIntosh, V. V. W., Jordan, J., Carter, J. D., Frampton, C. M. A., McKenzie, J. M., Latner, J. D., & Joyce, P. R. (2016). Psychotherapy for transdiagnostic binge eating: A randomized controlled trial of cognitive-behavioural therapy, appetite-focused cognitive-behavioural therapy, and schema therapy. Psychiatry Research, 240, 412–420. https://doi.org/10.1016/j.psychres.2016.04.080
   PubMed Web of Science ®Google Scholar
• McIntosh, V. V. W., Jordan, J., Carter, J. D., Latner, J. D., & Wallace, A. (2007). Appetite focused CBT for binge eating. In G. T. Wilson & J. D. Latner (Ed.), Self-help for obesity and eating disorders (pp. 325–346). Guilford Press.
   Google Scholar
• Myles, P. S., & Cui, J. (2007). Using the Bland-Altman method to measure agreement with repeated measures. British Journal of Anaesthesia, 99(3), 309–311. https://doi.org/10.1093/bja/aem214
   PubMed Web of Science ®Google Scholar
• O’Malley, S. S., Suh, C. S., & Strupp, H. H. (1983). The Vanderbilt psychotherapy process scale: A report on the scale development and a process-outcome study. Journal of Consulting and Clinical Psychology, 51(4), 581–586. https://doi.org/10.1037/0022-006X.51.4.581
   PubMed Web of Science ®Google Scholar
• Quarfoot, D., & Levine, R. A. (2016). How robust are multirater interrater reliability indices to changes in frequency distribution? The American Statistician, 70(4), 373–384. https://doi.org/10.1080/00031305.2016.1141708
   Web of Science ®Google Scholar
• Ryan, T. P., & Woodall, W. H. (2005). The most-cited statistical papers. Journal of Applied Statistics, 32(5), 461–474. https://doi.org/10.1080/02664760500079373
   Web of Science ®Google Scholar
• Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
   PubMed Web of Science ®Google Scholar
• R Studio Team. (2020). R studio: Integrated development for R. R Studio, Inc. http://www.rstudio.com
   Google Scholar
• Suh, C. S., O’Malley, S. S., Strupp, H. H., & Johnson, M. E. (1989). The Vanderbilt psychotherapy process scale (VPPS). Journal of Cognitive Psychotherapy: An International Quarterly, 3(2), 123–154. https://doi.org/10.1891/0889-8391.3.2.123
   Google Scholar
• Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60(2), 174–195. https://doi.org/10.1177/0013164400602002
   Web of Science ®Google Scholar
• Trevethan, R. (2017). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology, 17(2), 127–143. https://doi.org/10.1007/s10742-016-0156-6
   Web of Science ®Google Scholar
• Tryon, G. S., Blackwell, S. C., & Hammel, E. F. (2007). A meta-analytic examination of client–therapist perspectives of the working alliance. Psychotherapy Research, 17(6), 629–642. https://doi.org/10.1080/10503300701320611
   Web of Science ®Google Scholar
• Wickham, H. (2016). Ggplot2: Elegant graphics for data analysis (Version 3.3.2). Springer-Verlag. https://doi.org/10.1007/978-3-319-24277-4
   Google Scholar
• Windholz, M. J., & Silberschatz, G. (1988). Vanderbilt psychotherapy process scale: A replication with adult outpatients. Journal of Consulting and Clinical Psychology, 56(1), 56–60. https://doi.org/10.1037/0022-006X.56.1.56
   PubMed Web of Science ®Google Scholar
• Young, J. E., Klosko, J. S., & Weishaar, M. E. (2003). Schema therapy: A practitioner’s guide. Guilford Press.
   Google Scholar