Dear Sir
We want to take this opportunity to respond to the concerns raised about the reliability analysis conducted in the study. Dr Sabour has pointed out the appropriate use of intraclass correlation coefficient (ICC) as a preferred analysis to assess reliability in quantitative variables and has criticized our use of Pearson correlation coefficient.
As Dr Sabour is likely aware, the G-coefficient in generalizability analysis and ICC are both based in classical test theory and are closely related. While ICC analysis examines a single facet, generalizability analysis provides the opportunity to look at multiple facets of measurement error in a single design (Shrout & Fleiss 1979; Barch & Mathalon Citation2011). In our analysis, while the correlation coefficients were used to establish the relationship between peer and faculty ratings, the generalizability analysis provided the reliability measure.
We appreciate his interest in our research and the opportunity to clarify the analysis conducted.
Declaration of interest: The authors report no conflicts of interest.
References
- Barch DM, Mathalon DH. 2011. Using brain imaging measures in studies of precognitive pharmacological agents in schizophrenia: Psychometric and quality assurance considerations (Supplement). Biol Psychiatry 70(1):13–18
- Basehore PM, Pomerantz SC, Gentile M. 2014. Reliability and benefits of medical student peers in rating complex clinical skills. Med Teach 36(5):409–414
- Shrout PE, Fleiss JL. 1979. Intraclass correlations: Uses in assessing rater reliability. Psych Bull 86(2):420–428