2,530
Views
231
CrossRef citations to date
0
Altmetric
Original Articles

An Overview on Assessing Agreement with Continuous Measurements

, &
Pages 529-569 | Received 08 Feb 2007, Accepted 08 Mar 2007, Published online: 05 Jul 2007

REFERENCES

  • American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME) . ( 1999 ). The Standards for Educational and Psychological Testing . Washington , D.C. : American Psychological Association .
  • Anderson , S. , Hauck , W. W. ( 1990 ). Consideration of individual bioequivalence . Journal of Pharmacokinetics and Biopharmaceutics 18 : 259 – 273 .
  • Atkinson , G. , Nevill , A. ( 1997 ). Comment on the use of concordance correlation to assess the agreement between two variables . Biometrics 53 : 775 – 777 .
  • Barnhart , H. X. , Williamson , J. M. ( 2001 ). Modeling concordance correlation via GEE to evaluate reproducibility . Biometrics 57 : 931 – 940 .
  • Barnhart , H. X. , Haber , M. , Song , J. ( 2002 ). Overall concordance correlation coefficient for evaluating agreement among multiple observers . Biometrics 58 : 1020 – 1027 .
  • Barnhart , H. X. , Song , J. , Haber , M. (2005a). Assessing intra, inter, and total agreement with replicated measurements. Statistics in Medicine 24:1371–1384.
  • Barnhart , H. X. , Song , J. , Lyles , R. ( 2005b ). Assay validation for left censored data . Statistics in Medicine 24 : 3347 – 3360 .
  • Barnhart , H. X. , Haber , M. , Kosinski , A. S. ( 2007a ). Assessing individual agreement . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Barnhart , H. X. , Haber , M. , Lokhnygina , Y. , Kosinski , A. S. ( 2007b ). Comparison of concordance correlation coefficient and coefficient of individual agreement in assessing agreement . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Bartko , J. J. ( 1966 ). The intraclass correlation coefficient as a measure of reliability . Psychological Reports 19 : 3 – 11 .
  • Bartko , J. J. ( 1974 ). Corrective note to the intraclass correlation coefficient as a measure of reliability . Psychological Reports 34 : 418 .
  • Bland , J. M. , Altman , D. G. ( 1986 ). Statistical methods for assessing agreement between two methods of clinical measurement . Lancet i : 307 – 310 .
  • Bland , J. M. , Altman , D. G. ( 1990 ). A note on the use of the intraclass correlation coefficient in the evaluation of agreement between two methods of measurement . Computers in Biology and Medicine 20 : 337 – 340 .
  • Bland , J. M. , Altman , D. G. ( 1992 ). This week's citation classic: comparing methods of clinical measurement . Current Contents CM 20(40) Oct 5:8 .
  • Bland , J. M. , Altman , D. G. ( 1995 ). Comparing methods of measurement: why plotting difference against standard method is misleading . Lancet 346 : 1085 – 1087 .
  • Bland , J. M. , Altman , D. G. ( 1999 ). Measuring agreement in method comparison studies . Statistical Methods in Medical Research 8 : 135 – 160 .
  • Bland , J. M. , Altman , D. G. ( 2007 ). Agreement between methods of measurement with multiple observations per individual . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Brennan , R. L. ( 2001 ). Generalizability Theory . New York : Springer .
  • Carrasco , J. L. , Jover , L. ( 2003a ). Estimating the generalized concordance correlation coefficient through variance components . Biometrics 59 : 849 – 858 .
  • Carrasco , J. L. , Jover , L. ( 2003b ). Assessing individual bioequivalence using the structural equation model . Statistics in Medicine 22 : 901 – 912 .
  • Carrasco , J. L. , Jover , L. ( 2005a ). The structural error-in-equation model to evaluate individual bioequivalence . Biometrical Journal 47 : 623 – 634 .
  • Carrasco , J. L. , Jover , L. ( 2005b ). Concordance correlation coefficient applied to discrete data . Statistics in Medicine 24 : 4021 – 4034 .
  • Carrasco , J. L. , King , T. , Chinchilli , V. , Jover , L. ( 2007 ). Comparison of concordance correlation coefficient estimating approaches with skewed data . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Chen , M. L. ( 1997 ). Individual bioequivalence—A regulatory update . Journal of Biopharmaceutical Statistics 7 : 5 – 11 .
  • Chen , C. , Barnhart , H. X. ( 2007 ). Comparison of ICC and CCC for assessing agreement for data without and with replications . ENAR 2007 conference presentation in Atlanta, GA .
  • Chinchilli , V. M. , Martel , J. K. , Kumanyika , S. , Lloyd , T. ( 1996 ). A weighted concordance correlation coefficient for repeated measures designs . Biometrics 52 : 341 – 353 .
  • Choudhary , P. K. ( 2007a ). A tolerance interval approach for assessment of agreement in method comparison studies with repeated measurements . Journal of Statistical Planning and Inference DOI: 10.1016/j.jspi.2007.03.056
  • Choudhary , P. K. ( 2007b ). A tolerance interval approach for assessment of agreement with left censored data . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Choudhary , P. K. (2007c). Semiparametric regression for assessing agreement using tolerance bands. Computational Statistics and Data Analysis DOI: 10.1016/j.csda.2007.01.006
  • Choudhary , P. K. , Nagaraja , H. N. ( 2004 ). Measuring agreement in method comparison studies – a review . In: Balakrishnan , N. , Kannan , N. , Nagaraja , H. N. , eds. Advances in Ranking and Selection, Multiple Comparisons, and Reliability . Boston : Birkhauser , pp. 215 – 244 .
  • Choudhary , P. K. , Nagaraja , H. N. ( 2005a ). Assessment of agreement using intersection-union principle . Biometrical Journal 47 : 674 – 681 .
  • Choudhary , P. K. , Nagaraja , H. N. ( 2005b ). Selecting the instrument closest to a gold standard . Journal of Statistical Planning and Inference 129 : 229 – 237 .
  • Choudhary , P. K. , Nagaraja , H. N. ( 2005c ). A two-stage procedure for selection and assessment of agreement of the best instrument with a gold-standard . Sequential Analysis 24 : 237 – 257 .
  • Choudhary , P. K. , Ng , H. K. T. ( 2006 ). Assessment of agreement under non-standard conditions using regression models for mean and variance . Biometrics 62 : 288 – 296 .
  • Choudhary , P. K. , Nagaraja , H. N. ( 2007 ). Tests for assessment of agreement using probability criteria . Journal of Statistical Planning and Inference 137 : 279 – 290 .
  • Cohen , J. ( 1960 ). A coefficient of agreement for nominal scales . Educational and Psychological Measurement 20 : 37 – 46 .
  • Cohen , J. ( 1968 ). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit . Psychological Bulletin 70 : 213 – 220 .
  • Cronbach , L. J. , Gleser , G. C. , Nanda , H. , Rajaratnam , N. ( 1972 ). The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles . New York : Wiley .
  • Donner , A. ( 1998 ). Sample size requirements for the comparison of two or more coefficients of inter-observer agreement . Statistics in Medicine 17 : 1157 – 1168 .
  • Dunn , D. ( 1989 ). Statistical Evaluation of Measurement Errors: Design and Analysis of Reliability Studies . New York : Oxford University Press Inc .
  • Dunn , D. ( 1992 ). Design and analysis of reliability studies . Statistical Methods in Medical Research 1 : 123 – 157 .
  • Dunn , D. ( 2004 ). Statistical Evaluation of Measurement Errors: Design and Analysis of Reliability Studies . New York : Oxford University Press Inc .
  • Dunn , D. ( 2007 ). Regression models for method comparison data . Journal of Biopharma-ceutical Statistics 17 ( 4 ): xx – xx .
  • Dunn , G. , Roberts , C. ( 1999 ). Modelling method comparison data . Statistical Methods in Medical Research 8 : 161 – 179 .
  • Eliasziw , M. , Young , S. L. , Woodbury , M. G. , Fryday-Field , K. ( 1994 ). Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example . Physical Therapy 74 : 777 – 788 .
  • Fagot , R. F. ( 1993 ). A generalized family of coefficients of relational agreement for numerical scales . Psychometrika 58 : 357 – 370 .
  • Fay , M. P. ( 2005 ). Random marginal agreement coefficients: rethinking the adjustment for chance when measuring agreement . Biostatistics 6 : 171 – 180 .
  • Fisher , R. A. ( 1925 ). Statistical Methods for Research Workers . Edinburgh : Oliver and Boyd .
  • Fleiss , J. L. ( 1986 ). The Design and Analysis of Clinical Experiments . New York : Wiley .
  • Food and Drug Administration (FDA) ( 2001 ). Guidance for Industry: Bioanalytical Method Validation . http://ww.fda.gov/cder/guidance/index.htm
  • Galton , F. ( 1886 ). Family likeness in stature . Proceedings of the Royal Society 40 : 42 – 73 .
  • Goodwin , L. D. ( 1997 ). Changing concepts of measurement validity . Journal of Nursing Education 36 : 102 – 107 .
  • Guo , Y. , Manatunga , A. K. ( 2007 ). Nonparametric estimation of the concordance correlation coefficient under univariate censoring . Biometrics 83 : 164 – 172 .
  • Guttman , I. (1988). Statistical tolerance regions. In: Kotz , S. , Johnson , N. L. , eds. Encyclopedia of Statistical Sciences . Vol. 9 . New York : Wiley, pp. 272–287.
  • Haber , M. , Barnhart , H. X. ( 2006 ). Coefficients of agreement for fixed observers . Statistical Methods in Medical Research 15 : 1 – 17 .
  • Haber , M. , Barnhart , H. X. ( 2007 ). A general approach to evaluating agreement between two observers or methods of measurement . Journal of Data Science , in press .
  • Haber , M. , Barnhart , H. X. , Song , J. , Gruden , J. ( 2005 ). Observer variability: a new approach in evaluating interobserver agreement . Journal of Data Science 3 : 69 – 83 .
  • Haber , M. , Gao , J. , Barnhart , H. X. ( 2007 ). Assessing observer agreement in studies involving replicated binary observations . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Hand , D. J. ( 2004 ). Measurement Theory and Practice . New York : Oxford University Press Inc .
  • Harris , I. R. , Burch , B. D. , St. Laurent , R. T. ( 2001 ). A blended estimator for measure of agreement with a gold standard . Journal of Agricultural, Biological, and Environmental Statistics 6 : 326 – 339 .
  • Hawkins , D. M. ( 2002 ). Diagnostics for conformity of paired quantitative measurements . Statistics in Medicine 21 : 1913 – 1935 .
  • Hutson , A. D. , Wilson , D. C. , Geiser , E. A. ( 1998 ). Measuring relative agreement: echocar-diographer versus computer . Journal of Agricultural, Biological and Environmental Statistics 3 : 163 – 174 .
  • International Organization for Standardization (ISO) ( 1994 ). Accuracy (Trueness and Precision) of Measurement Methods and Results—Part 1: General Principles and Definitions (5725-1) . Geneva, Switzerland : ISO .
  • Janson , H. , Olsson U. ( 2001 ). A measure of agreement for interval or nominal multivariate observations . Educational and Psychological Measurement 61 : 277 – 289 .
  • Janson , H. , Olsson U. ( 2004 ). A measure of agreement for interval or nominal multivariate observations by different sets of judges . Educational and Psychological Measurement 64 : 62 – 70 .
  • Kelly , G. E. ( 1985 ). Use of the structural equations model in assessing the reliability of a new measurement technique . Applied Statistics 34 : 258 – 263 .
  • King , T. S. , Chinchilli , V. M. ( 2001a ). A generalized concordance correlation coefficient for continuous and categorical data . Statistics in Medicine 20 : 2131 – 2147 .
  • King , T. S. , Chinchilli , V. M. ( 2001b ). Robust estimators of the concordance correlation coefficient . Journal of Biopharmaceutical Statistics 11 : 83 – 105 .
  • King , T. S. , Chinchilli , V. M. , Carrasco , J. ( 2007a ). A repeated measures concordance correlation coefficient . Statistics in Medicine DOI: 10.1002/sim.2778
  • King , T. S. , Chinchilli , V. M. , Carrasco , J. L. , Wang , K. ( 2007b ). A class of repeated measures concordance correlation coefficients . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Krippendorff , K. ( 1970 ). Bivariate agreement coefficients for reliability of data . In: Borgatta , B. F. , Bohrnstedt , G. E. , eds. Social Methodology . San Francisco : Jessey-Bass , pp. 139 – 150 .
  • Konishi , S. Khatri , C. G. , Rao , C. R. ( 1991 ). Inferences on multivariate measures of interclass and intraclass correlations in familial data . Journal of the Royal Statistical Society Series B 53 : 649 – 659 .
  • Kraemer , H. C. , Periyakoil V. S. , Noda , A. ( 2002 ). Kappa coefficients in medical research . Statistics in Medicine 21 : 2109 – 2129 .
  • Lee , J. , Koh , D. , Ong , C. N. ( 1989 ). Statistical evaluation of agreement between two methods for measuring a quantitative variable . Computers in Biology and Medicine 19 : 61 – 70 .
  • Liao , J. J. Z. (2003). An improved concordance correlation coefficient. Pharmaceutical Statistics 2:253–261.
  • Liao , J. J. Z. ( 2005 ). Agreement for curved data . Journal of Biopharmaceutical Statistics 15 : 195 – 203 .
  • Lin , L. I. ( 1989 ). A concordance correlation coefficient to evaluate reproducibility . Biometrics 45 : 225 – 268 .
  • Lin , L. I. ( 1992 ). Assay validation using the concordance correlation coefficient . Biometrics 48 : 599 – 604 .
  • Lin , L. I. ( 2000a ). A note on the concordance correlation coefficient . Biometrics , 56 : 324 – 325 .
  • Lin , L. I. ( 2000b ). Total deviation index for measuring individual agreement with applications in laboratory performance and bioequivalence . Statistics in Medicine 19 : 255 – 270 .
  • Lin , L. I. ( 2003 ). Measuring Agreement . In: Chow , S. , ed. Encyclopedia of Biopharmaceutical Statistics . Philadelphia : Informa Healthcare , pp. 561 – 567 .
  • Lin , L. I. , Hedayat , A. S. , Sinha , B. , Yang , M. ( 2002 ). Statistical methods in assessing agreement: models, issues and tools . Journal of American Statistical Association 97 : 257 – 270 .
  • Lin , L. I. , Hedayat , A. S. , Wenting , W. ( 2007 ). A unfied approach for assessing agreement for continuous and categorical data . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Lin , S. C. , Whipple , D. M. , Ho , C. S. ( 1998 ). Evaluation of statistical equivalence using limits of agreement and associated sample size calculation . Communications in Statistics—Theory and Methods 27 : 1419 – 1432 .
  • Liu X. , Du Y. , Teresi J. , Hasin , D. S. ( 2005 ). Concordance correlation in the measurements of time to event . Statistics in Medicine 24 : 1409 – 1420 .
  • Lord , F. M. , Novick , M. R. ( 1968 ). Statistical Theories of Mental Test Scores . Reading , MA : Addison-Wesley .
  • McGraw , K. O. , Wong , S. P. ( 1996 ). Forming inferences about some intraclass correlation coefficients . Psychological Methods 1 : 30 – 46 .
  • Molenberghs , G. , Vangeneugden , T. , Laenen A. ( 2007 ). Estimating reliability and general-izability from hierarchical biomedical data . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Müller , R. , Büttner , P. ( 1994 ). A critical discussion of intraclass correlation coefficients . Statistics in Medicine 13 : 2465 – 2476 .
  • Nickerson , C. A. ( 1997 ) Comment on a concordance correlation to evaluate reproducibility . Biometrics 53 : 1503 – 1507 .
  • Pearson , K. ( 1896 ). VII. Mathematical contributions to the theory of evolution—III. Regression, heredity and panmixia . Philosophical Transactions of the Royal Society, Series A 187 : 253 – 318 .
  • Pearson , K. ( 1901 ). VIII. Mathematical contributions to the theory of evolution—IX. On the principle of homotyposis and its relation to heredity, to the variability of the individual, and to that of the race. Part I. Homotyposis in the vegetable kingdom . Philosophical Transactions of the Royal Society, Series A 197 : 285 – 379 .
  • Pearson , K. , Lee , A. , Bramley-Moore , L. ( 1899 ). VI. Mathematical contributions to the theory of evolution—VI. Genetic (reproductive) selection: Inheritance of fertility in man, and of fecundity in thoroughbred racehorses . Philosophical Transactions of the Royal Society, Series A 192 : 257 – 330 .
  • Raghavachari , M. ( 2004 ). Measures of concordance for assessing agreement in ratings and rank order data . In: Balakrishnan , N. , Kannan , N. , Nagaraja , H. N. , eds. Advances in Ranking and Selection, Multiple Comparisons and Reliability . Boston : Birkhauser , pp. 245 – 263 .
  • Robieson , W. (1999). On Weighted Kappa and Concordance Correlation Coefficient. Ph.D. thesis, University of Illinois in Chicago.
  • Quan , H. , Shih , W. J. ( 1996 ). Assessing reproducibility by the within-subject coefficient of variation with random effects models . Biometrics 52 : 1195 – 1203 .
  • Quiroz , J. ( 2005 ). Assessment of equivalence using a concordance correlation coefficient in a repeated measurement design . Journal of Biopharmaceutical Statistics 15 : 913 – 928 .
  • Ryan , T.P. , Woodall , W.H. ( 2005 ). The most-cited statistical papers . Journal of Applied Statistics 32 : 461 – 474 .
  • Shao , J. , Zhong , B. ( 2004 ). Assessing the agreement between two quantitative assays with repeated measurements . Journal of Biopharmaceutical Statistics 14 : 201 – 212 .
  • Shavelson , R. J. , Webb , N. M. ( 1981 ). Generalizability: 1973–1980 . British Journal of Mathematical and Statistical Psychology 34 : 133 – 166 .
  • Shavelson , R. J. , Webb , N. M. ( 1991 ). Generalizability Theory: A Primer . Newbury Park , CA : SAGE Publication .
  • Shavelson , R. J. , Webb , N. M. ( 1992 ). Generalizability theory . In: Alkin , M. C. , ed. Encyclopedia of Education Research . Vol. 2 . New York : Macmillan , pp. 538 – 543 .
  • Shavelson , R. J. , Webb , N. M. , Rowley , G. I. ( 1989 ). Generalizability theory . American Psychologist 44 : 922 – 932 .
  • Shoukri , M. M. ( 1998 ). Measurement of agreement . In: Armitage , P. , Colton , T. , eds. The Encyclopedia of Biostatistics . New York : Wiley , pp. 102 – 116 .
  • Shoukri , M. M. , Asyali , M. H. , Donner , A. ( 2004 ). Sample size requirements for the design of reliability study: review and new results . Statistical Methods in Medical Research 13 : 251 – 271 .
  • Shrout , P. E. ( 1998 ). Measurement reliability and agreement in psychiatry . Statistical Methods in Medical Research 7 : 301 – 317 .
  • Shrout , P. E. , Fleiss , J. L. ( 1979 ). Intraclass correlations: uses in assessing rater reliability . Psychological Bulletin 86 : 420 – 428 .
  • St. Laurent , R. T. ( 1998 ). Evaluating agreement with a gold standard in method comparison studies . Biometrics 54 : 537 – 545 .
  • Stine , W. W. ( 1989 ). Interobserver relational agreement . Psychological Bulletin 106 : 341 – 347 .
  • Vangeneugden , T. , Laenen , A. , Geys , H. , Renard , D. , Molenberghs , G. ( 2004 ). Applying linear mixed models to estimate reliability in clinical trials with repeated measurements . Controlled Clinical Trials 25 : 13 – 30 .
  • Vangeneugden , T. , Laenen , A. , Geys , H. , Renard , D. , Molenberghs , G. ( 2005 ). Applying concepts of generalizability theory on clinical trial data to investigate sources of variation and their impact on reliability . Biometrics 61 : 295 – 304 .
  • Wang , W. , Hwang , J. R. G. ( 2001 ). A nearly unbiased test for individual bioequivalence problems using probability criteria . Journal of Statistical Planning and Inference 99 : 41 – 58 .
  • Williamson , J. M. , Crawford , S. B. , Lin , H. ( 2007 ). Permutation testing for comparing dependent concordance correlation coefficients . Journal of Biopharmaceutical Statistics 17 ( 4 ): xx – xx .
  • Zhong , B. , Shao , J. ( 2003 ). Evaluating the agreement of two quantitative assays with repeated measurements . Journal of Biopharmaceutical Statistics 13 : 75 – 86 .
  • Zegers , F. E. ( 1986 ). A family of chance-corrected association coefficients for metric scales . Psychometrika 51 : 559 – 562 .

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.