Assessing the reliability of clinical scales when the data have both nominal and ordinal features: Proposed guidelines for neuropsychological assessments

References

• American Psychiatric Association . 1987 . Diagnostic and statistical manual of mental disorders , 3rd ed. Washington, DC
   Google Scholar
• Apgar , V. A. 1953 . A proposal for a new method of evaluation of the newborn infant . Anesthesia and Analgesia , 32 : 260 – 267 .
   Google Scholar
• Cicchetti , D. V. 1976 . Assessing inter‐rater reliability for rating scales: Resolving some basic issues . British Journal of Psychiatry , 129 : 452 – 456 .
   PubMed Web of Science ®Google Scholar
• Cicchetti , D. V. 1988 . When diagnostic agreement is high, but reliability is low: Some paradoxes occurring in independent neuropsychological assessments . Journal of Clinical and Experimental Neuropsychology , 10 : 605 – 622 .
   PubMed Web of Science ®Google Scholar
• Cicchetti , D. V. , Aivano , S. L. and Vitale , J. 1977 . A computer program for assessing rater agreement and rater bias for qualitative data . Educational and Psychological Measurement , 37 : 195 – 201 .
   Google Scholar
• Cicchetti , D. V. and Feinstein , A. R. 1990 . High agreement but low kappa: II. Resolving the paradoxes . Journal of Clinical Epidemiology , 43 : 551 – 568 .
   PubMed Web of Science ®Google Scholar
• Cicchetti , D. V. and Fleiss , J. L. 1977 . Comparison of the null distributions of weighted kappa and the C ordinal statistic . Applied Psychological Measurement , 1 : 195 – 201 .
   Google Scholar
• Cicchetti , D. V. and Heavens , R. 1979 . RATCAT (Rater Agreement/Categorical Data) . The American Statistician , 33 : 91
   Web of Science ®Google Scholar
• Cicchetti , D. V. , Lee , C. , Fontana , A. F. and Dowds , B. N. 1978 . A computer program for assessing specific category rater agreement for qualitative data . Educational and Psychological Measurement , 38 : 805 – 813 .
   Web of Science ®Google Scholar
• Cicchetti , D. V. and Prusoff , B. A. 1983 . Reliability of depression and associated clinical symptoms . Archives of General Psychiatry , 40 : 987 – 990 .
   PubMedGoogle Scholar
• Cicchetti , D. V. and Showalter , D. 1988 . A computer program for determining the reliability of dimensionally scaled data when the numbers and specific sets of examiners may vary at each assessment . Educational and Psychological Measurement , 48 : 717 – 720 .
   Web of Science ®Google Scholar
• Cicchetti , D. V. , Showalter , D. and Tyrer , P. 1985 . The effect of number of rating scale categories upon levels of interrater reliability: A Monte Carlo investigation . Applied Psychological Measurement , 9 : 31 – 36 .
   Web of Science ®Google Scholar
• Cicchetti , D. V. , Sparrow , S. S. , Volkmar , F. , Cohen , D. and Rourke , B. P. 1991 . Establishing the reliability and validity of neuropsychological disorders with low base rates: Some recommended guidelines . Journal of Clinical and Experimental Neuropsychology , 13 : 328 – 338 .
   PubMed Web of Science ®Google Scholar
• Cicchetti , D. V. and Sparrow , S. S. 1981 . Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior . American Journal of Mental Deficiency , 86 : 127 – 137 .
   PubMedGoogle Scholar
• Cicchetti , D. V. and Tyrer , P. 1988 . “ Reliability and validity of personality assessment ” . In Personality disorders: Diagnosis, management and course , Edited by: Tyrer , P. J. 63 – 73 . London : Butterworth Scientific Ltd .
   Google Scholar
• Cohen , J. 1960 . A coefficient of agreement for nominal scales . Educational and Psychological Measurement , 20 : 37 – 46 .
   Web of Science ®Google Scholar
• Feinstein , A. R. 1987 . Clinimetrics , New Haven, CT : Yale University Press .
   Google Scholar
• Feinstein , A. R. and Cicchetti , D. V. 1990 . High agreement but low kappa: I. The problems of two paradoxes . Journal of Clinical Epidemiology , 43 : 543 – 549 .
   PubMed Web of Science ®Google Scholar
• Fleiss , J. L. 1981 . Statistical methods for rates and proportions , 2nd ed. , New York : Wiley .
   Google Scholar
• Fleiss , J. L. , Cohen , J. and Everitt , B. S. 1969 . Large sample standard errors of kappa and weighted kappa . Psychological Bulletin , 72 : 323 – 327 .
   Web of Science ®Google Scholar
• Hamilton , M. 1960 . A rating scale for depression . Journal of Neurological and Neurosurgical Psychology , 23 : 56 – 62 .
   PubMed Web of Science ®Google Scholar
• Heavens , R. H. Jr. and Cicchetti , D. V. 1978 . A computer program for calculating rater agreement and bias statistics using contingency table input . Proceedings of the American Statistical Association (Statistical Computing Section) , 21 : 366 – 370 .
   Google Scholar
• Horwitz , R. I. , Cicchetti , D. V. and Horwitz , S. M. 1984 . A comparison of the Norris and Killip Coronary Prognostic Indices . Journal of Chronic Diseases , 37 : 369 – 375 .
   PubMedGoogle Scholar
• Killip , T. III and Kimball , J. T. 1967 . Treatment of myocardial infarction in a coronary care unit . American Journal of Cardiology , 20 : 457 – 464 .
   PubMed Web of Science ®Google Scholar
• Landis , J. R. and Koch , G. G. 1977 . The measurement of observer agreement for categorical data . Biometrics , 33 : 159 – 174 .
   PubMed Web of Science ®Google Scholar
• Norris , R. M. , Brandt , P. W. T. , Caughey , D. E. , Lee , A. J. and Scott , P. J. 1969 . A new coronary prognostic index . Lancet , 1 : 274 – 278 .
   PubMed Web of Science ®Google Scholar
• Overall , J. E. and Gorham , D. R. 1962 . The Brief Psychiatric Rating Scale . Psychological Reports , 10 : 799 – 812 .
   Web of Science ®Google Scholar
• Overall , J. E. and Klett , C. J. 1972 . Applied multivariate analysis , New York : McGraw‐Hill .
   Google Scholar
• Schopler , E. , Reichler , R. J. and Renner , B. R. 1986 . The Childhood Autism Rating Scales (CARS) for diagnostic screening and classification of Autism , New York : Irvington .
   Google Scholar
• Shrout , P. E. , Spitzer , R. L. and Fleiss , J. L. 1987 . Quantification of agreement in psychiatric diagnosis revisited . Archives of General Psychiatry , 44 : 172 – 177 .
   PubMed Web of Science ®Google Scholar
• Sparrow , S. S. , Balla , D. A. and Cicchetti , D. V. 1984a . The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by Edgar A. Doll. I. Survey Form , Circle Pines, MN : American Guidance Service .
   Google Scholar
• Sparrow , S. S. , Balla , D. A. and Cicchetti , D. V. 1984b . The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by Edgar A. Doll. II. Expanded Form , Circle Pines, MN : American Guidance Service .
   Google Scholar
• Sparrow , S. S. , Balla , D. A. and Cicchetti , D. V. 1985 . The Vineland Adaptive Behavior Scales: A revision of the Vineland Social Maturity Scale by Edgar A. Doll. III. Classroom Edition , Circle Pines, MN : American Guidance Service .
   Google Scholar
• Sparrow , S. S. and Cicchetti , D. V. 1984 . The Behavior Inventory for Rating Development (BIRD): Assessments of reliability and factorial validity . Applied Research in Mental Retardation , 5 : 219 – 231 .
   PubMedGoogle Scholar
• Steinberg , M. , Rounsaville , B. and Cicchetti , D. V. 1990 . The Structured Clinical Interview for DSM‐III‐R Dissociative Disorders: Preliminary report on a new diagnostic instrument . American Journal of Psychiatry , 147 : 76 – 82 .
   PubMed Web of Science ®Google Scholar
• Tyrer , P. , Alexander , M. S. , Cicchetti , D. , Cohen , M. S. and Remington , M. 1979 . Reliability of a schedule for rating personality disorders . British Journal of Psychiatry , 135 : 168 – 174 .
   PubMed Web of Science ®Google Scholar
• Tyrer , P. , Owen , R. and Cicchetti , D. V. 1984 . The Brief Scale for Anxiety: A subdivision of the Comprehensive Psychopathological Rating Scale . Journal of Neurology, Neurosurgery and Psychiatry , 47 : 970 – 975 .
   PubMed Web of Science ®Google Scholar
• Williams , J. B. W. 1988 . A structured interview guide for the Hamilton Depression Rating Scale . Archives of General Psychiatry , 45 : 742 – 747 .
   PubMed Web of Science ®Google Scholar
• Wing , J. K. , Cooper , J. E. and Sartorius , N. 1974 . The measurement and classification of psychiatric symptoms , Cambridge : Cambridge University Press .
   Google Scholar