Inter-rater Agreement Indices for Multiple Informant Methodology

References

• Anderson, J. C. (1985). A measurement model to assess measure-specific factors in multiple-informant research. Journal of Marketing Research, 22 (1), 86–92. doi:10.2307/3151554
   Web of Science ®Google Scholar
• Babbie, E. (2001). The practice of social research (9th ed.). Belmont, CA: Thomson Wadsworth.
   Google Scholar
• Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83 (5), 762–765. doi:10.1037/0033-2909.83.5.762
   Web of Science ®Google Scholar
• Biemann, T., Ellwart, T., & Rack, O. (2014). Quantifying similarity of team mental models: An introduction of the rRG index. Group Processes & Intergroup Relations, 17 (1), 125–140. doi:10.1177/1368430213485993
   Google Scholar
• Bradley-Geist, J. C., & Landis, R. S. (2012). Homogeneity of personality in occupations and organizations: A comparison of alternative statistical tests. Journal of Business and Psychology, 27 (2), 149–159. doi:10.1007/s10869-011-9233-6
   Web of Science ®Google Scholar
• Brown, D. R., & Hauenstein, N. M. A. (2005). Interrater agreement reconsidered: An alternative to the rWG indices. Organizational Research Methods, 8 (2), 165–184. doi:10.1177/1094428105275376
   Web of Science ®Google Scholar
• Burke, M. J., & Dunlap, W. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5 (2), 159–172. doi:10.1177/1094428102005002002
   Web of Science ®Google Scholar
• Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2 (1), 49–68. doi:10.1177/109442819921004
   Web of Science ®Google Scholar
• Cai, Y., Jia, L., You, S., Zhang, Y., & Chen, Y. (2013). The influence of differentiated transformational leadership on knowledge sharing and team creativity: A social network explanation. Acta Psychologica Sinica, 45 (5), 585–598. doi:10.3724/SP.J.1041.2013.00585
   Google Scholar
• Christenfeld, N. J., & Hill, E. A. (1995). Whose baby are you? Nature, 378 (6558), 669. doi:10.1038/378669a0
   PubMed Web of Science ®Google Scholar
• Cicchetti, D. V., & Sparrow, S. A. (1981). Developing criteria for establishing interrater reliability of specific items: Applications to assessment of adaptive behavior. American Journal of Mental Deficiency, 86 (2), 127–137.
   PubMedGoogle Scholar
• Cohen, J. (1960). Kappa: Coefficient of concordance. Educational and Psychological Measurement, 20 (1), 37–46.
   Web of Science ®Google Scholar
• Cronbach, L. J., Ikeda, H., & Avner, R. A. (1964). Intraclass correlation as an approximation to the coefficient of generalizability. Psychological Reports, 15 (3), 727–736. doi:10.2466/pr0.1964.15.3.727
   Web of Science ®Google Scholar
• De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. A. (2013). Principles underlying the use of multiple informants’ reports. Annual Review of Clinical Psychology, 9, 123–149. doi:10.1146/annurev-clinpsy-050212-185617
   PubMed Web of Science ®Google Scholar
• Dickman, B. M. (2014). Conceptions of creativity in elementary school mathematical problem posing (Doctoral dissertation). Columbia University. Retrieved from https://academiccommons.columbia.edu/catalog/ac:176065
   Google Scholar
• Ellwart, T., Konradt, U., & Rack, O. (2014). Team mental models of expertise location: Validation of a field survey measure. Small Group Research, 45 (2), 119–153. doi:10.1177/1046496414521303
   Web of Science ®Google Scholar
• Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley & Sons.
   Google Scholar
• Giraudeau, B. (1996). Negative values of the intraclass correlation coefficient are not theoretically possible. Journal of Clinical Epidemiology, 49 (10), 1205. doi:10.1016/0895-4356(96)00053-4
   PubMed Web of Science ®Google Scholar
• Gisev, N., Bell, J. S., & Chen, T. F. (2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9 (3), 330–338. doi:10.1016/j.sapharm.2012.04.004
   PubMed Web of Science ®Google Scholar
• González-Romá, V., & Hernández, A. (2014). Climate uniformity: Its influence on team communication quality, task conflict, and team performance. Journal of Applied Psychology, 99 (6), 1042–1058. doi:10.1037/a0037868
   PubMed Web of Science ®Google Scholar
• Huang, C. E., Cassels, S. L., & Winer, R. L. (2015). Self-reported sex partner dates for use in measuring concurrent sexual partnerships: Correspondence between two assessment methods. Archives of Sexual Behavior, 44 (4), 873–883. doi:10.1007/s10508-014-0414-z
   PubMed Web of Science ®Google Scholar
• James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67 (2), 219–229. doi:10.1037/0021-9010.67.2.219
   Web of Science ®Google Scholar
• James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69 (1), 85–98. doi:10.1037/0021-9010.69.1.85
   Web of Science ®Google Scholar
• Jigjidsuren, D. (2014). School readiness: Does it matter if parents and caregivers think alike?. (Doctoral dissertation). The University of North Carolina at Chapel Hill. Retrieved from https://cdr.lib.unc.edu/indexablecontent/uuid:a986b052-9f6f-4414-b8dd-70c20ed2414a
   Google Scholar
• Kenny, D. A., & Acitelli, L. K. (1994). Measuring similarity in couples. Journal of Family Psychology, 8 (4), 417–431. doi:10.1037/0893-3200.8.4.417
   Google Scholar
• Klein, K. J., & Kozlowski, S. W. (2000). Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions. San Francisco, CA: Jossey-Bass.
   Google Scholar
• Kline, T. J. B., & Hambley, L. A. (2007). Four multi-item inter-rater agreement options: Comparisons and outcomes. Psychological Reports, 101 (3), 1001–1010. doi:10.2466/pr0.101.3.1001-1010
   PubMed Web of Science ®Google Scholar
• Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160 (9), 1566–1577. doi:10.1176/appi.ajp.160.9.1566
   PubMed Web of Science ®Google Scholar
• Lanz, M., Scabini, E., Tagliabue, S., & Morgano, A. (2015). How should family interdependence be studied? The methodological issues of non-independence. TPM-Testing, Psychometrics, Methodology in Applied Psychology, 22 (2), 1–12.
   Web of Science ®Google Scholar
• LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11 (4), 815–852. doi:10.1177/1094428106296642
   Web of Science ®Google Scholar
• Levecque, K., Roose, H., Vanroelen, C., & Van Rossem, R. (2014). Affective team climate: A multi-level analysis of psychosocial working conditions and psychological distress in team workers. Acta Sociologica, 57 (2), 153–166. doi:10.1177/0001699313498262
   Web of Science ®Google Scholar
• Lindell, M. K., & Brandt, C. J. (1997). Measuring interrater agreement for ratings of a single target. Applied Psychological Measurement, 21 (3), 271–278. doi:10.1177/01466216970213006
   Web of Science ®Google Scholar
• Lindell, M. K., Brandt, C. J., & Whitney, D. J. (1999). A revised index of interrater agreement for multi-item ratings of a single target. Applied Psychological Measurement, 23 (2), 127–135. doi:10.1177/01466219922031257
   Web of Science ®Google Scholar
• Liu, S., & Li, Y. (2014). A longitudinal study on the impact mechanism of employees’ boundary spanning behavior: Roles of centrality and collectivism. Acta Psychologica Sinica, 46 (6), 852–863. doi:10.3724/SP.J.1041.2014.00852
   Google Scholar
• Lohse-Bossenz, H., Kunina-Habenicht, O., & Kunter, M. (2014). Estimating within-group agreement in small groups: A proposed adjustment for the average deviation index. European Journal of Work & Organizational Psychology, 23 (3), 456–468. doi:10.1080/1359432X.2012.748189
   Web of Science ®Google Scholar
• McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1 (1), 30–46. doi:10.1037/1082-989X.1.1.30
   Web of Science ®Google Scholar
• Moreno, J., Silverman, W. K., Saavedra, L. M., & Phares, V. (2008). Fathers’ ratings in the assessment of their child’s anxiety symptoms: A comparison to mothers’ ratings and their associations with paternal symptomatology. Journal of Family Psychology, 22 (6), 915–919. doi:10.1037/a0014097
   PubMed Web of Science ®Google Scholar
• Parade, S. H., Supple, A. J., & Helms, H. M. (2012). Parenting during childhood predicts relationship satisfaction in young adulthood: A prospective longitudinal perspective. Marriage & Family Review, 48 (2), 150–169. doi:10.1080/01494929.2011.629078
   Google Scholar
• Pearsall, M. J., & Venkataramani, V. (2015). Overcoming asymmetric goals in teams: The interactive roles of team learning orientation and team identification. Journal of Applied Psychology, 100 (3), 735–748. doi:10.1037/a0038315
   PubMed Web of Science ®Google Scholar
• Pedon, A. (2009). Dizionario di statistica e metodologia per le scienze del comportamento. Roma, IT: Alpes Italia.
   Google Scholar
• Peñarroja, V., Orengo, V., Zornoza, A., & Hernández, A. (2013). The effects of virtuality level on task-related collaborative behaviors: The mediating role of team trust. Computers in Human Behavior, 29 (3), 967–974. doi:10.1016/j.chb.2012.12.020
   Web of Science ®Google Scholar
• Ratelle, J. T., Kelm, D. J., Halvorsen, A. J., West, C. P., & Oxentenko, A. S. (2015). Predicting and communicating risk of clinical deterioration: An observational cohort study of internal medicine residents. Journal of General Internal Medicine, 30 (4), 448–453. doi:10.1007/s11606-014-3114-4
   PubMed Web of Science ®Google Scholar
• Roberson, Q. M., Sturman, M. C., & Simons, T. L. (2007). Does the measure of dispersion matter in multilevel research?. A comparison of the relative performance of dispersion indexes. Organizational Research Methods, 10 (4), 564–588. doi:10.1177/1094428106294746
   Web of Science ®Google Scholar
• Semrau, M., Burns, A., Djukic-Dejanovic, S., Eraslan, D., Han, C., Lecic-Tosevski, D., … Sartoriusp, N. (2015). Development of an international schedule for the assessment and staging of care for dementia. Journal of Alzheimer’s Disease, 44 (1), 139–151, doi:10.3233/JAD-141599
   PubMed Web of Science ®Google Scholar
• Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86 (2), 420–428. doi:10.1037/0033-2909.86.2.420
   PubMed Web of Science ®Google Scholar
• Silva, N., Crespo, C., Carona, C., Bullinger, M., & Canavarro, M. C. (2015). Why the (dis)agreement?. Family context and child–parent perspectives on health‐related quality of life and psychological problems in paediatric asthma. Child: Care, Health and Development, 41 (1), 112–121. doi:10.1111/cch.12147
   PubMed Web of Science ®Google Scholar
• Smith-Crowe, K., Burke, M. J., Cohen, A., & Doveh, E. (2014). Statistical significance criteria for the rWG and average deviation interrater agreement indices. Journal of Applied Psychology, 99 (2), 239–261. doi:10.1037/a0034556
   Google Scholar
• Smith-Crowe, K., Burke, M. J., Kouchaki, M., & Signal, S. M. (2012). Assessing inter-rater agreement via the average deviation index given a variety of theoretical and methodological problems. Organizational Research Methods, 16 (1), 127–151. doi:10.1177/1094428112465898
   Web of Science ®Google Scholar
• Tang, C. Y., Curran, M., & Arroyo, A. (2014). Cohabitors’ reasons for living together, satisfaction with sacrifices, and relationship quality. Marriage & Family Review, 50 (7), 598–620. doi:10.1080/01494929.2014.938289
   Google Scholar
• Tijdens, K. G., de Ruijter, E., & de Ruijter, J. (2014). Comparing tasks of 160 occupations across eight European countries. Employee Relations, 36 (2), 110–127. doi:10.1108/ER-05-2013-0046
   Web of Science ®Google Scholar
• Tinsley, H. E. A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22 (4), 358–376. doi:10.1037/h0076640
   Web of Science ®Google Scholar
• Unsworth, C., Harries, P., & Davies, M. (2015). Using social judgment theory method to examine how experienced occupational therapy driver assessors use information to make fitness-to-drive recommendations. The British Journal of Occupational Therapy, 78 (2), 109–120. doi:10.1177/0308022614562396
   PubMed Web of Science ®Google Scholar
• van Bruggen, G. H., Lilien, G. L., & Kacker, M. (2002). Informants in organizational marketing research: Why use multiple informants and how to aggregate responses. Journal of Marketing Research, 39 (4), 469–478. doi:10.1509/jmkr.39.4.469.19117
   Web of Science ®Google Scholar
• van Vianen, A. E. M., De Pater, I. E., Bechtoldt, M. N., & Evers, A. (2011). The strength and quality of climate perceptions. Journal of Managerial Psychology, 26 (1), 77–92. doi:10.1108/02683941111099637
   Web of Science ®Google Scholar
• Vera, M., Martínez, I. M., Lorente, L., & Chambel, M. J. (2015). The role of co-worker and supervisor support in the relationship between job autonomy and work engagement among Portuguese nurses: A multilevel study. Social Indicators Research, 126 (3), 1143–1156. doi:10.1007/s11205-015-0931-8
   Web of Science ®Google Scholar
• Wagner, S. M., Rau, C., & Lindemann, E. (2010). Multiple informant methodology: A critical review and recommendations. Sociological Methods & Research, 38 (4), 582–618. doi:10.1177/0049124110366231
   Web of Science ®Google Scholar
• Wholey, D. R., Zhu, X., Knoke, D., Shah, P., Zellmer-Bruhn, M., & Witheridge, T. F. (2012). The teamwork in assertive community treatment (TACT) scale: Development and validation. Psychiatric Services, 63 (11), 1108–1117. doi:10.1176/appi.ps.201100338
   PubMed Web of Science ®Google Scholar
• Wittenborn, A. K., Dolbin-MacNab, M. L., & Keiley, M. K. (2013). Dyadic research in marriage and family therapy: Methodological considerations. Journal of Marital and Family Therapy, 39 (1), 5–16. doi:10.1111/j.1752-0606.2012.00306.x
   PubMed Web of Science ®Google Scholar
• Yurdusev, A. N. (1993). ‘Level of analysis’ and ‘unit of analysis’: A case for distinction. Millennium: Journal of International Studies, 22 (1), 77–88. doi:10.1177/03058298930220010601
   Web of Science ®Google Scholar
• Zohar, D., & Luria, G. (2010). Group leaders as gatekeepers: Testing safety climate variations across levels of analysis. Applied Psychology: An International Review, 59 (4), 647–673. doi:10.1111/j.1464-0597.2010.00421.x
   Web of Science ®Google Scholar