Advanced search
Publication Cover
Journal of the American Statistical Association Volume 80, 1985 - Issue 389
Submit an article Journal homepage
71
Views
99
CrossRef citations to date
0
Altmetric
Theory and Method

Modeling Agreement among Raters

Martin A. Tanner Departments of Statistics and Human Oncology, University of Wisconsin, Madison, WI, 53706, USA
&
Michael A. Young Departments of Psychology and Psychiatry, Rush-Presbyterian-St. Luke's Medical Center, Chicago, IL, 60612, USA
Pages 175-180 | Received 01 Apr 1983, Published online: 12 Mar 2012

Citations (99)

Keep up to date with the latest research on this topic with citation updates for this article.

Subscribe to citation updates

Read on this site (5)

Elaine Conway. (2019) To Agree or Disagree? An Analysis of CSR Ratings Firms. Social and Environmental Accountability Journal 39:3, pages 152-177.
Read now
Alireza Akbarzadeh Bagheban & Farid Zayeri. (2010) A generalization of the uniform association model for assessing rater agreement in ordinal scales. Journal of Applied Statistics 37:8, pages 1265-1273.
Read now
Wessel N. van Wieringen & Edwin R. van den Heuvel. (2005) A Comparison of Methods for the Evaluation of Binary Measurement Systems. Quality Engineering 17:4, pages 495-507.
Read now
A. JOLLY, D. GUYON & J. RIOM. (1996) Utilisation des donnees du moyen infrarouge de Landsat Thematic Mapper pour la mise en evidence des coupes rases sur le Massif Forestier Landais. International Journal of Remote Sensing 17:18, pages 3615-3645.
Read now
ValenE. Johnson. (1996) On Bayesian Analysis of Multirater Ordinal Data: An Application to Automated Essay Grading. Journal of the American Statistical Association 91:433, pages 42-51.
Read now

Articles from other publishers (94)

Ömer Emre Can ALAGÖZ, Yılmaz Orhun GÜRLÜK, Mediha KORMAZ & Gizem CÖMERT. (2023) An Illustration of a Latent Class Analysis for Interrater Agreement: Identifying Subpopulations with Different Agreement Levels. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 14:4, pages 492-507.
Crossref
Michael E. Sobel, Gregory J. Wawro & Sean Farhang. (2023) Association and causation: Attributes and effects of judges in equal employment opportunity commission litigation outcomes. The Annals of Applied Statistics 17:4.
Crossref
Katelyn A. McKenzie & Jonathan D. Mahnken. (2023) Simulating and estimating agreement in the presence of multiple raters and covariates. Statistics in Medicine 42:11, pages 1687-1698.
Crossref
Adebola Rafiu BAKAREAdebola Rafiu BAKARE. 2023. National Assembly and Legislative Effectiveness in Nigeria’s Fourth Republic. National Assembly and Legislative Effectiveness in Nigeria’s Fourth Republic 77 86 .
Gokcen ALTUN. (2021) Analysis of the Multiraters Agreement with Log-Linear ModelsLog-lineer Modeller ile Çoklu Değerlendiriciler Uyum Analizi. Bilge International Journal of Science and Technology Research 5:2, pages 107-110.
Crossref
Mauro Giammarino, Silvana Mattiello, Monica Battini, Piero Quatto, Luca Maria Battaglini, Ana C. L. Vieira, George Stilwell & Manuela Renna. (2021) Evaluation of Inter-Observer Reliability of Animal Welfare Indicators: Which Is the Best Index to Use?. Animals 11:5, pages 1445.
Crossref
Adebola Rafiu Bakare. 2021. Two Decades of Legislative Politics and Governance in Nigeria’s National Assembly. Two Decades of Legislative Politics and Governance in Nigeria’s National Assembly 121 159 .
Andrés Bravo-Oviedo, Maurizio Marchi, Davide Travaglini, Francesco Pelleri, Maria Chiara Manetti, Piermaria Corona, Fátima Cruz, Felipe Bravo & Susanna Nocentini. (2020) Adoption of new silvicultural methods in Mediterranean forests: the influence of educational background and sociodemographic factors on marker decisions. Annals of Forest Science 77:2.
Crossref
David Bock, Eva Angenete, Anders Bjartell, Jonas Hugosson, Gunnar Steineck, Sofie Walming, Peter Wiklund & Eva Haglind. (2019) Agreement between patient reported outcomes and clinical reports after radical prostatectomy - a prospective longitudinal study. BMC Urology 19:1.
Crossref
Anna K. Porter, Fang Wen, Amy H. Herring, Daniel A. Rodríguez, Lynne C. Messer, Barbara A. Laraia & Kelly R. Evenson. (2018) Reliability and One-Year Stability of the PIN3 Neighborhood Environmental Audit in Urban and Rural Neighborhoods. Journal of Urban Health 95:3, pages 431-439.
Crossref
Kerrie P. Nelson & William Barlow. 2014. Wiley StatsRef: Statistics Reference Online. Wiley StatsRef: Statistics Reference Online 1 7 .
Kerrie P Nelson & Don Edwards. (2016) A measure of association for ordered categorical data in population-based studies. Statistical Methods in Medical Research 27:3, pages 812-831.
Crossref
Kerrie P. Nelson, Aya A. MitaniDon Edwards. (2017) Assessing the influence of rater and subject characteristics on measures of agreement for ordinal ratings. Statistics in Medicine 36:20, pages 3181-3199.
Crossref
Pankaj K Choudhary & Haikady N Nagaraja. 2017. Measuring Agreement. Measuring Agreement 319 330 .
Rashid Almehrizi. (2016) Normalization of mean squared differences to measure agreement for continuous data. Statistical Methods in Medical Research 25:5, pages 1955-1974.
Crossref
T. S. Akkerhuis & J. de Mast. (2016) Quantifying the Random Component of Measurement Error of Nominal Measurements Without a Gold Standard. Quality and Reliability Engineering International 32:6, pages 1993-2003.
Crossref
TP ErdmannJ De MastMJ Warrens. (2012) Some common errors of experimental design, interpretation and inference in agreement studies. Statistical Methods in Medical Research 24:6, pages 920-935.
Crossref
Kerrie P. NelsonDon Edwards. (2015) Measures of agreement between many raters for ordinal classifications. Statistics in Medicine 34:23, pages 3116-3132.
Crossref
Jason J. Z. Liao & Robert C. Capen. 2014. Wiley StatsRef: Statistics Reference Online. Wiley StatsRef: Statistics Reference Online.
William Barlow. 2014. Wiley StatsRef: Statistics Reference Online. Wiley StatsRef: Statistics Reference Online.
Mark P. Becker. 2014. Wiley StatsRef: Statistics Reference Online. Wiley StatsRef: Statistics Reference Online.
Guangchao Charles Feng. (2014) Estimating intercoder reliability: a structural equation modeling approach. Quality & Quantity 48:4, pages 2355-2369.
Crossref
Jason J. Z. Liao & Robert C. Capen. 2014. Methods and Applications of Statistics in Clinical Trials. Methods and Applications of Statistics in Clinical Trials 446 456 .
Guangchao Charles Feng. (2012) Factors affecting intercoder reliability: a Monte Carlo experiment. Quality & Quantity 47:5, pages 2959-2982.
Crossref
Tenko Raykov, Dimiter M. Dimitrov, Alexander von Eye & George A. Marcoulides. (2012) Interrater Agreement Evaluation. Educational and Psychological Measurement 73:3, pages 512-531.
Crossref
Jennifer K. Straughen, Cleopatra H. Caldwell, Theresa L. Osypuk, Laura Helmkamp & Dawn P. Misra. (2013) Direct and Proxy Recall of Childhood Socio‐Economic Position and Health. Paediatric and Perinatal Epidemiology 27:3, pages 294-302.
Crossref
Shahram Khoshbin, Amy Herring, Gregory L. Holmes, Donald Schomer, Daniel Hoch, Elizabeth C. Dooling, Eileen P.G. Vining & Lewis B. Holmes. (2013) Inter-rater agreement for diagnoses of epilepsy in pregnant women. Epilepsy & Behavior 27:1, pages 148-153.
Crossref
Yuan Horng Lin. (2013) Fuzzy Kappa Coefficient with Simulated Comparisons. Applied Mechanics and Materials 303-306, pages 372-375.
Crossref
Alexander von Eye & Eun‐Young Mun. 2012. Log‐Linear Modeling. Log‐Linear Modeling 425 440 .
Alessandro Foddai, Laura E Green, Sam A Mason & Jasmeet Kaler. (2012) Evaluating observer agreement of scoring systems for foot integrity and footrot lesions in sheep. BMC Veterinary Research 8:1.
Crossref
Jørgen Holm Petersen, Klaus Larsen & Svend Kreiner. (2010) Assessing and quantifying inter-rater variation for dichotomous ratings using a Rasch model. Statistical Methods in Medical Research 21:6, pages 635-652.
Crossref
Yu‐Kang Tu & Mark S. Gilthorpe. (2012) Key statistical and analytical issues for evaluating treatment effects in periodontal research. Periodontology 2000 59:1, pages 75-88.
Crossref
Anna Klimova, Tamás Rudas & Adrian Dobra. (2012) Relational models for contingency tables. Journal of Multivariate Analysis 104:1, pages 159-173.
Crossref
Hisayuki Hara, Tomonari Sei & Akimichi Takemura. (2012) Hierarchical subspace models for contingency tables. Journal of Multivariate Analysis 103:1, pages 19-34.
Crossref
Amita K Manatunga, José Nilo G Binongo & Andrew T Taylor. (2011) Computer-aided diagnosis of renal obstruction: utility of log-linear modeling versus standard ROC and kappa analysis. EJNMMI Research 1:1.
Crossref
Levent Dumenci. (2010) The Psychometric Latent Agreement Model (PLAM) for Discrete Latent Variables Measured by Multiple Items. Organizational Research Methods 14:1, pages 91-115.
Crossref
Kerrie P. Nelson & Don Edwards. (2010) Improving the reliability of diagnostic tests in population‐based agreement studies. Statistics in Medicine 29:6, pages 617-626.
Crossref
Reem Hasan, Michele L. Jonsson Funk, Amy H. Herring, Andrew F. Olshan, Katherine E. Hartmann & Donna D. Baird. (2009) Accuracy of reporting bleeding during pregnancy. Paediatric and Perinatal Epidemiology 24:1, pages 31-34.
Crossref
Serpil Aktaş & Tülay Saraçbaşı. (2008) Estimation of symmetric disagreement using a uniform association model for ordinal agreement data. AStA Advances in Statistical Analysis 93:3, pages 335-343.
Crossref
Kerrie P. Nelson & Don Edwards. (2010) On population‐based measures of agreement for binary classifications. Canadian Journal of Statistics 36:3, pages 411-426.
Crossref
Alireza Akbarzadeh Bagheban, Mahtab Nouri & Mohammadreza Safavi. (2008) Assessment of agreement in measuring orthodontic treatment need with the modified DHC. Australasian Orthodontic Journal 24:1, pages 10-14.
Crossref
Jason J. Z. Liao & Robert C. Capen. 2007. Wiley Encyclopedia of Clinical Trials. Wiley Encyclopedia of Clinical Trials 1 9 .
Peter H. Van NessVirginia R. Towle & Manisha Juthani-Mehta. (2007) Testing Measurement Reliability in Older Populations. Journal of Aging and Health 20:2, pages 183-197.
Crossref
Fabien Valet, Christiane Guinot & Jean Yves Mary. (2006) Log‐linear non‐uniform association models for agreement between two ratings on an ordinal scale. Statistics in Medicine 26:3, pages 647-662.
Crossref
Alexander von Eye. (2006) An Alternative to Cohen's κ. European Psychologist 11:1, pages 12-24.
Crossref
Mousumi Banerjee. 2004. Encyclopedia of Statistical Sciences. Encyclopedia of Statistical Sciences.
Ying Guo & Amita K. Manatunga. (2005) Modeling the Agreement of Discrete Bivariate Survival Times using Kappa Coefficient. Lifetime Data Analysis 11:3, pages 309-332.
Crossref
Mark P. Becker. 2005. Encyclopedia of Biostatistics. Encyclopedia of Biostatistics.
William Barlow. 2005. Encyclopedia of Biostatistics. Encyclopedia of Biostatistics.
Fabio Rapallo. (2005) Algebraic exact inference for rater agreement models. Statistical Methods & Applications 14:1, pages 45-66.
Crossref
Alexander von Eye & Maxine von Eye. (2005) Can One Use Cohen’s Kappa to Examine Disagreement?. Methodology 1:4, pages 129-142.
Crossref
Mousumi Banerjee. 2004. Encyclopedia of Statistical Sciences. Encyclopedia of Statistical Sciences.
Mônica Rodrigues Campos, Maria do Carmo Leal, Paulo Roberto de Souza Jr. & Cynthia Braga da Cunha. (2004) Consistência entre fontes de dados e confiabilidade interobservador do Estudo da Morbi-mortalidade e Atenção Peri e Neonatal no Município do Rio de Janeiro. Cadernos de Saúde Pública 20:suppl 1, pages S34-S43.
Crossref
Jason J. Z. Liao. (2003) An improved concordance correlation coefficient. Pharmaceutical Statistics 2:4, pages 253-261.
Crossref
Helena Chmura Kraemer, Vyjeyanthi S. Periyakoil & Art Noda. (2002) Kappa coefficients in medical research. Statistics in Medicine 21:14, pages 2109-2129.
Crossref
Susan M. Perkins & Mark P. Becker. (2002) Assessing rater agreement using marginal association models. Statistics in Medicine 21:12, pages 1743-1760.
Crossref
H. Lester Kirchner & Jon H. Lemke. (2002) Simultaneous estimation of intrarater and interrater agreement for multiple raters under order restrictions for a binary trait. Statistics in Medicine 21:12, pages 1761-1772.
Crossref
John Schafer, Raul Caetano & Catherine L. Clark. (2016) Agreement About Violence in U.S. Couples. Journal of Interpersonal Violence 17:4, pages 457-470.
Crossref
Christof Schuster & Alexander von Eye. (2001) Models for Ordinal Agreement Data. Biometrical Journal 43:7, pages 795-808.
Crossref
Mekibib Altaye, Allan Dormer & Neil Klar. (2004) Inference Procedures for Assessing Interobserver Agreement among Multiple Raters. Biometrics 57:2, pages 584-588.
Crossref
Kent Grayson & Roland Rust. (2001) Interrater Reliability. Journal of Consumer Psychology 10:1-2, pages 71-73.
Crossref
F. Neijenhuis, H.W. Barkema, H. Hogeveen & J.P.T.M. Noordhuizen. (2000) Classification and Longitudinal Examination of Callused Teat Ends in Dairy Cows. Journal of Dairy Science 83:12, pages 2795-2804.
Crossref
Jennifer C NelsonMargaret S Pepe. (2016) Statistical description of interrater variability in ordinal ratings. Statistical Methods in Medical Research 9:5, pages 475-496.
Crossref
Mousumi Banerjee, Michelle Capozzoli, Laura McSweeney & Debajyoti Sinha. (2008) Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics 27:1, pages 3-23.
Crossref
John R Bergan, Richard D SchwarzLinda A Reddy. (2016) Latent Structure Analysis of Classification Errors in Screening and Clinical Diagnosis: An Alternative to Classification Analysis. Applied Psychological Measurement 23:1, pages 69-86.
Crossref
Eduardo Freitas da Silva & Maurício Gomes Pereira. (1998) Avaliação das estruturas de concordância e discordância nos estudos de confiabilidade. Revista de Saúde Pública 32:4, pages 383-393.
Crossref
Patrick E Shrout. (2016) Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research 7:3, pages 301-317.
Crossref
Jouni Kuha & Chris Skinner. 1997. Survey Measurement and Process Quality. Survey Measurement and Process Quality 633 670 .
Bjørn O. Eriksen, Sven M. Almdahl, Anne Hensrud, Steinar Jæger, Ivar S. Kristiansen, Fred A. Mürer, Erik Nord, Jan Fr. Pape, Reidar Robertsen & Glen Thorsen. (2009) Assessing Health Benefit from Hospitalization: Agreement Between Expert Panels. International Journal of Technology Assessment in Health Care 12:1, pages 126-135.
Crossref
Irene Guggenmoss‐Holzmann. (2007) Modelling covariate effects in observer agreement studies: The case of nominal scale agreement. Statistics in Medicine 14:20, pages 2285-2288.
Crossref
Patrick Graham. (2007) Modelling covariate effects in observer agreement studies: The case of nominal scale agreement. Statistics in Medicine 14:3, pages 299-310.
Crossref
Clifford C. Clogg. 1995. Handbook of Statistical Modeling for the Social and Behavioral Sciences. Handbook of Statistical Modeling for the Social and Behavioral Sciences 311 359 .
N. T. Longford. (2016) Reliability of Essay Rating and Score Adjustment. Journal of Educational Statistics 19:3, pages 171-200.
Crossref
Irene Guggenmoos‐Holzmann. (2006) HOW reliable are change‐corrected measures of agreement?. Statistics in Medicine 12:23, pages 2191-2205.
Crossref
N. T. Longford. (2014) RELIABILITY OF ESSAY RATING AND SCORE ADJUSTMENT. ETS Research Report Series 1993:2.
Crossref
Ambrogio S. Fassina, Maria C. Montesco, Vito Ninfo, Paolo Denti & Guido Masarotto. (2018) Histological Evaluation of Thyroid Carcinomas: Reproducibility of the «Who» Classification. Tumori Journal 79:5, pages 314-320.
Crossref
P. Faglioni & C. Botti. (1993) How to Differentiate Retrieval from Storage Deficit: A Stochastic Approach to Semantic Memory Modeling. Cortex 29:3, pages 501-518.
Crossref
Patrick Graham & Rodney Jackson. (1993) The analysis of ordinal agreement data: beyond weighted kappa. Journal of Clinical Epidemiology 46:9, pages 1055-1062.
Crossref
Steven S. Coughlin, Linda W. Pickle, Marc T. Goodman & Lynne R. Wilkens. (1992) The logistic modeling of interobserver agreement. Journal of Clinical Epidemiology 45:11, pages 1237-1241.
Crossref
Alan Agresti. (2016) Modelling patterns of agreement and disagreement. Statistical Methods in Medical Research 1:2, pages 201-218.
Crossref
Mark P. Becker & Alan Agresti. (2013) Log‐linear modelling of pairwise interobserver agreement on a categorical scale. Statistics in Medicine 11:1, pages 101-114.
Crossref
Mitchell H. Gail. (2006) A bibliography and comments on the use of statistical models in epidemiology in the 1980s. Statistics in Medicine 10:12, pages 1819-1885.
Crossref
J. Barry Garner. (2006) The standard error of Cohen's Kappa. Statistics in Medicine 10:5, pages 767-775.
Crossref
Alexander von Eye & Silvia Sörensen. (2007) Models of Chance when Measuring Interrater Agreement with Kappa. Biometrical Journal 33:7, pages 781-787.
Crossref
E. Teju Jolayemi. (2007) A Multiraters Agreement Index for Ordinal Classification. Biometrical Journal 33:4, pages 485-492.
Crossref
John S. Uebersax & William M. Grove. (2006) Latent class analysis of diagnostic agreement. Statistics in Medicine 9:5, pages 559-572.
Crossref
E. T. Jolayemi. (2007) Relative Frequency Estimation in Multiple Outcome Measurement with Misclassifications. Biometrical Journal 32:6, pages 707-711.
Crossref
Mark P. Becker. (2006) Using association models to analyse agreement data: Two examples. Statistics in Medicine 8:10, pages 1199-1207.
Crossref
JOSEPH S. VERDUCCI, MICHAEL E. MACK & MORRIS H. DEGROOT. 1989. Multivariate Statistics and Probability. Multivariate Statistics and Probability 539 562 .
Joseph S Verducci, Michael E Mack & Morris H DeGroot. (1988) Estimating multiple rater agreement for a rare diagnosis. Journal of Multivariate Analysis 27:2, pages 512-535.
Crossref
J. N. Darroch & P. I. McCloud. (2008) Category Distinguishability and Observer Agreement. Australian Journal of Statistics 28:3, pages 371-388.
Crossref
Andre O. Varma. (2005) Discussion: A procedure for evaluating the reliability of a gingival index, and, A comparison of 3 clinical indices for measuring gingivitis. Journal of Clinical Periodontology 13:5, pages 396-397.
Crossref
Albert Kingman. (2005) A procedure for evaluating the reliability of a gingivitis index. Journal of Clinical Periodontology 13:5, pages 385-391.
Crossref
R.W. Valachovic, C.W. Douglass, C.S. Berkey, B.J. McNeil & H.H. Chauncey. (2016) Examiner Reliability in Dental Radiography. Journal of Dental Research 65:3, pages 432-436.
Crossref

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Order Reprints Request Corporate Permissions

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

Request Academic Permissions

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.