Properties of the Achievement of Therapeutic Objectives Scale (ATOS): A Generalizability Theory study

References

• Bakali , J.V. , Wilberg , T. , Hagtvet , K. , & Lorentzen , S. 2010 . Sources accounting for alliance and cohesion at three stages in group psychotherapy: Variance ms . Group Dynamics: Theory, Research, and Practice , 14 , 368 – 383 . doi: 10.1037/a0019170 .
   Web of Science ®Google Scholar
• Bozarth , J.D. , & Krauft , C.J. 1972 . Accurate empathy ratings: Some methodological considerations . Journal of Clinical Psychology , 23 , 408 – 411 . doi: 10.1002/1097-4679(197207) 28:3+< 408 .
   Web of Science ®Google Scholar
• Brennan , R.L. 1983 . Elements of Generalizability Theory . Iowa City . American College Testing Program .
   Google Scholar
• Brennan , R.L. 2003 . Coefficients and indices in Generalizability Theory . Center for Advanced Studies in Measurement and Assessment Report 1 .
   Google Scholar
• Cicchetti , D.V. 1994 . Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology . Psychological Assessment , 6 , 284 – 290 . doi: 10.1037/1040-3590.6.4.284 .
   Google Scholar
• Cone , J.D. 1977 . The relevance of reliability and validity for behavioral assessment . Behavior Therapy , 8 , 411 – 426 . doi: 10.1016/S0005–7894(77)80077–4 .
   Web of Science ®Google Scholar
• Crick , J.E. , & Brennan , R.L. 1983 . Manual for GENOVA. A generalized analysis of variance system . ACT Technical Bulletin. No, 43 . Iowa City : American College Testing .
   Google Scholar
• Crits-Christoph , P. , Johnson , J. , Gallop , R. , Gibbons , M.B.C. , Ring-Kurtz , S. , Hamilton , J.L. , & Tu , X. 2011 A generalizability theory analysis of group process ratings in the treatment of cocaine dependence . Psychotherapy Research , 22 , 252 – 266 . doi: 10.1080/10503307.2010.551429 .
   Web of Science ®Google Scholar
• Cronbach , L.J. and Meehl , P.E. 1955 . Construct validity in psychological tests . Psychological Bulletin , 52 : 281 – 302 .
   PubMed Web of Science ®Google Scholar
• Cronbach , L.J. , Gleser , G.C. , Nanda , H. and Rajaratnam , N. 1972 . The dependability of behavioral measurements: Theory of generalizability for scores and profiles , New York : Wiley .
   Google Scholar
• Davanloo , H. 1980 . Short-term dynamic therapy , New York : Jason Aronson .
   Google Scholar
• Erb , H.P. , Kruglanski , A.W. , Chun , W.Y. , Pierro , A. , Mannetti , L. and Spiegel , S. 2003 . Searching for commonalities in human judgment: The parametric unimodel and its dual mode alternatives . European Review of Social Psychology , 14 : 1 – 49 .
   Google Scholar
• Fosha , D. 2000 . The transforming power of affect: A model of accelerated change , New York : Basic Books .
   Google Scholar
• Gilbert , P. 2005 . Compassion-focused therapy: An introduction to theory, research and practice , London : Routledge .
   Google Scholar
• Greenberg , L. 2001 . Emotion-focused therapy: Coaching clients to work through their feelings , Washington , DC : American Psychological Association .
   Google Scholar
• Hagtvet , K.A. 1997 . The function of indicators and errors in construct measures: An application of generalizability theory . Journal of Vocational Education Research , 22 : 247 – 266 .
   Google Scholar
• Hill , C.E. and Lambert , M.J. 2004 . “ Methodological issues in studying psychotherapy processes and outcome ” . In Bergin & Garfield's handbook of psychotherapy and behavior change , Edited by: Lambert , M.J. 84 – 135 . New York : Wiley & Sons .
   Google Scholar
• Hintze , J.M. and Matthews , W.J. 2004 . The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation . School Psychology Review , 33 : 258 – 270 .
   Web of Science ®Google Scholar
• Høglend , P. , Bøgwald , K.P. , Amlo , S. , Heyerdahl , O. , Sørbye , Ø. , Marble , A. , Sjaastad , M.C. and Bentsen , H. 2000 . Assessment of change in dynamic psychotherapy . The Journal of Psychotherapy Practice and Research , 9 : 190 – 199 .
   PubMedGoogle Scholar
• Hoyt , W.T. 2000 . Rater bias in psychological research: When is it a problem and what can we do about it? Psychological Methods , 5 , 64 – 86 . doi: 10.1037/1082–989X.5.1.64 .
   PubMed Web of Science ®Google Scholar
• Hoyt , W.T. 2002 . Bias in participant ratings of psychotherapy process: An initial generalizability study . Journal of Counselling Psychology , 49 , 35 – 46 . doi: 10.1037/0022–0167.49.1.35 .
   Web of Science ®Google Scholar
• Hoyt , W.T. , & Kerns , M.D. 1999 . Magnitude and moderators of bias in observer ratings: A meta-analysis . Psychological Methods , 4 , 403 – 424 . doi: 10.1037/1082-989X.4.4.403 .
   Web of Science ®Google Scholar
• Kane , T.M. 1982 . A sampling model for validity . Applied Psychological Measurement , 6 : 125 – 160 .
   Web of Science ®Google Scholar
• Kallestad , H. , Valen , J. , McCullough , L. , Svartberg , M. , Høglend , P. , & Stiles , T.C. 2010 . The relationship between insight gained during therapy and long-term outcome in short-term dynamic psychotherapy and cognitive therapy for Cluster C personality disorders . Psychotherapy Research , 20 , 526 – 534 . doi: 10.1080/10503307.2010.492807 .
   PubMed Web of Science ®Google Scholar
• Kazdin , A. 2007 . Mediators and mechanisms of change in psychotherapy research . Annual Review of Clinical Psychology , 2 , 1 – 27 . doi: 10.1146/annurev.clinpsy.3.022806.091432 .
   Web of Science ®Google Scholar
• Kazdin , A. 2008 . Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care . American Psychologist , 63 , 146 – 159 . doi: 10.1037/0003–066X.63.3.146 .
   PubMed Web of Science ®Google Scholar
• Kazdin , A.E. 2009 . Understanding how and why psychotherapy leads to change . Psychotherapy Research , 19 , 418 – 428 . doi: 10.1080/10503300802448899 .
   PubMed Web of Science ®Google Scholar
• Kenny , D.A. 1991 . A general model of consensus and accuracy in interpersonal perception . Psychological Review , 98 , 155 – 163 . doi: 10.1207/s15327957pspr0803_3 .
   PubMed Web of Science ®Google Scholar
• Klein , D.F. , & Cleary , T.A. 1967 . Platonic true scores and error in psychiatric rating scales . Psychological Bulletin , 68 , 77 – 80 . doi: 10.1037/h0024761 .
   PubMed Web of Science ®Google Scholar
• Lakes , K.D. , & Hoyt , W.T. 2008 . What sources contribute to variance in observer ratings? Using generalizability theory to assess construct validity of psychological measures . Infant and Child Development , 17 , 269 – 284 . doi: 10.1002/icd.551 .
   PubMed Web of Science ®Google Scholar
• Lambert , M.J. 2004 . Bergin and Garfield's Handbook of psychotherapy and behavior change , New York : Wiley and Sons .
   Google Scholar
• Lambert , M.J. and Ogles , B.M. 2004 . “ The efficacy and effectiveness of psychotherapy ” . In Bergin and Garfield‘s Handbook of psychotherapy and behavior change , Edited by: Lambert , M.J. 139 – 193 . New York : Wiley and Sons .
   Google Scholar
• Lipsey , M.W. , & Wilson , D.B. 1993 . The efficacy of psychological, educational, and behavioral treatment . American Psychologist , 48 , 1181 – 1201 . doi: 10.1037/0003–066X.48.12.1181 .
   PubMed Web of Science ®Google Scholar
• Luborsky , L. , Rosenthal , R. , Diguer , L. , Andrusyna , T.P. , Berman , J.S. , Levitt , J.T. , Seligman , D.A. , & Krause , E.D. 2002 . The Dodo bird verdict is alive and well-mostly . Clinical Psychology: Science and Practice , 9 , 2 – 12 . doi: 10.1093/clipsy.9.1.2 .
   Web of Science ®Google Scholar
• Luborsky , L. , Singer , B. and Luborsky , L. 1975 . Comparative studies of psychotherapies: Is it true that “Everyone has won and all must have prizes”? . Archives of General Psychiatry , 32 : 995 – 1008 .
   PubMed Web of Science ®Google Scholar
• Lutz , W. , & Hill , C.E. 2009 . Quantitative and qualitative methods for psychotherapy research: Introduction to special section . Psychotherapy Research , 19 , 369 – 373 . doi: 10.1080/10503300902948053 .
   PubMed Web of Science ®Google Scholar
• Marcoulides , G.A. 1989 . The application of generalizability analysis to observational studies . Quality & Quantity , 23 , 115 – 127 . doi: 10.1007/BF00151898 .
   Web of Science ®Google Scholar
• McCullough , L. , Bhatia , M. , Ulvenes , P. , Berggraf , L. , & Osborn , K. 2011 . Learning how to rate video-recorded therapy sessions: A Practical Guide for Trainees and Advanced Clinicians . Psychotherapy , 48 , 127 – 137 . doi: 10.1037/a0023131 .
   PubMed Web of Science ®Google Scholar
• McCullough , L. , Kuhn , N. , Andrews , S. , Valen , J. , Hatch , D. and Osimo , F. 2004 . The reliability of the Achievement of Therapeutic Objectives Scale (ATOS): a research and teaching tool for psychotherapy . Journal of Brief Therapy , 2 : 75 – 90 .
   Google Scholar
• McCullough , L. , Larsen , A.E. , Schanche , E. , Andrews , S. , & Kuhn , N. (with Hurley , C.L. , Carley , M. , & Francis-Ranier , E. ). 2003 . Achievement of Therapeutic Objectives Scale: ATOS scale . Retrieved from http://www.affectphobia.org/videoatos.html .
   Google Scholar
• Mercer , R.C. , & Loesch , L.C. 1979 . Audio tape ratings: Comments and guidelines . Psychotherapy: theory, research and practice , 16 , 79 – 85 . doi: 10.1037/h0085879 .
   Web of Science ®Google Scholar
• Messer , S.B. , Thishby , O. , & Spillman , A. 1992 . Taking context seriously in psychotherapy research: Relating therapist interventions to patient progress in brief psychodynamic therapy . Journal of Consulting and Clinical Psychology , 60 , 678 – 688 . doi: 10.1037/0022–006X.60.5.678 .
   PubMed Web of Science ®Google Scholar
• Messick , S. 1995 . Validity of psychological assessment. Validation of inferences from persons' responses and performances as scientific inquiry into score meaning . American Psychologist , 50 , 741 – 749 . doi: 10.1037/0003–066X.50.9.741 .
   Web of Science ®Google Scholar
• Saal , F.E. , Downey , R.G. , & Lahey , M.A. 1980 . Rating the ratings: Assessing the psychometric quality of rating data . Psychological Bulletin , 88 , 413 – 428 . doi: 10.1037/0033–2909.88.2.413 .
   Web of Science ®Google Scholar
• Schanche , E. , Nielsen , G.H. , McCullough , L. , Valen , J. , & Mykletun , A. 2010 . Training graduate students as raters in psychotherapy research. Reliability of ratings with the Achievement of Therapeutic Objectives Scale (ATOS) . Nordic Psychology , 62 , 4 – 20 . doi: 10.1027/1901–2276/a000013 .
   Web of Science ®Google Scholar
• Shavelson , R.J. and Webb , N.M. 1981 . Generalizability theory: 1973–1980 . British Journal of Mathematical and Statistical Psychology , 34 : 133 – 166 .
   Web of Science ®Google Scholar
• Shavelson , R.J. and Webb , N.M. 1991 . Generalizability Theory: A primer , Newbury Park , CA : Sage .
   Google Scholar
• Shavelson , R.J. , Webb , N.M. , & Rowley , G.L. 1989 . Generalizability theory . American Psychologist , 44 , 922 – 932 . doi: 10.1037/0003–066X.44.6.922 .
   Web of Science ®Google Scholar
• Shedler , J. 2010 . The efficacy of psychodynamic psychotherapy . American Psychologist , 65 , 98 – 109 . doi: 10.1037/a0018378 .
   PubMed Web of Science ®Google Scholar
• Siefert , C.J. , Defife , J.A. and Baity , M.R. 2009 . “ Process measures for psychodynamic therapy ” . In Handbook of evidence based therapy: Bridging the gap between science and practise , Edited by: Levy , R.A. and Ablon , J.S. Totowa , NJ : Humana Press .
   Google Scholar
• Smith , G.T. 2005 . On construct validity: Issues of method and measurement . Psychological Assessment , 17 , 396 – 408 . doi: 10.1037/1040–3590.17.4.396 .
   PubMed Web of Science ®Google Scholar
• Strossen , R.J. , Coates , T.J. , & Thoresen , C.E. 1979 . Extending generalizability theory to single-subject designs . Behavior Therapy , 10 , 606 – 614 . doi: 10.1016/S0005–7894(79)80064–7 .
   Web of Science ®Google Scholar
• Svartberg , M. , Stiles , T.C. and Seltzer , M. 2004 . Randomized controlled trial of the effectiveness of short-term dynamic psychotherapy and cognitive therapy for cluster C personality disorders . American Journal of Psychiatry , 161 : 810 – 817 .
   PubMed Web of Science ®Google Scholar
• Thorndike , E.L. 1920 . A constant error in psychological ratings . Journal of Applied Psychology , 4 : 25 – 29 .
   Google Scholar
• Ulvenes , P.G. , Berggraf , L. , Wampold , B.E. , Hoffart , A. , & McCullough , L. 2011 . Observer ratings of psychotherapy process: facets contributing variability to scores . Submitted for publication .
   Google Scholar
• Valen , J. , Ryum T. , Svartberg , M. , Stiles , T.C. , & McCullough , L. 2011 . The Achievement of Therapeutic Objective Scale: Interrater reliability and sensitivity to change in short term dynamic therapy and cognitive therapy . Psychological Assessment. Advance online publication . doi: 10.1037/a0023649 .
   Google Scholar
• Wampold , B.E. 2001 . The great psychotherapy debate: Models, methods, and findings , Mahwah , NJ : Erlbaum .
   Google Scholar
• Wampold , B.E. 2007 . Psychotherapy: The humanistic (and effective) treatment . American Psychologist , 62 , 857 – 873 . doi: 10.1037/0003–066X.62.8.857 .
   Web of Science ®Google Scholar
• Wampold , B.E. , Mondin , G.W. , Moody , M. , Stich , F. , Benson , K. , & Ahn , H. 1997 . A meta-analysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes” . Psychological Bulletin , 122 , 203 – 215 . doi: 10.1037/0033–2909.122.3.203 .
   Web of Science ®Google Scholar
• Wasserman , R.H. , Levy , K.N. , & Loken , K.N. 2009 . Generalizability theory in psychotherapy research: The impact of multiple sources of variance on the dependability of psychotherapy process ratings . Psychotherapy Research , 19 , 397 – 408 . doi: 10.1080/10503300802579156 .
   PubMed Web of Science ®Google Scholar