Reliability and benefits of medical student peers in rating complex clinical skills; Common mistake

Dear Sir

I was interested to read the paper by Basehore PM and colleagues published in the March 2014 issue of Medical Teacher, where the authors investigated the reliability of student peers of the same level of training in rating complex clinical skills in a geriatric medicine based objective structured clinical exam (OSCE). They reported that the reliability of the OSCE was moderately strong (G-coefficient = 0.70) with strong correlations between peer and faculty ratings for the overall OSCE (r = 0.78, p = 0.001) and for each case (r = 0.70–0.85, p = 0.001; Basehore et al. 2014). This result has nothing to do with reliability and actually is one of the common mistakes in reliability analysis (Rothman et al. 2008). Reliability (repeatability or reproducibility) is often assessed by different statistical tests such as Pearson r, least square and paired t. ‘Mistakes in reliability analysis are common’ (Lawrence & Kuei 1989; Rothman et al. 2008).

For quantitative variables the Intra Class Correlation Coefficient (ICC) should be used. For qualitative variables the weighted kappa, which should be used with caution because kappa has its own limitation too (Lawrence & Kuei 1989; Rothman et al. 2008). It is crucial to know that there is no value of kappa that can be regarded universally as an indication good agreement. An important weakness of k value to assess agreement of a qualitative variable is that it depends upon the prevalence in each category. This means that it is be possible to have a different kappa value based on the same percentage of both concordant and discordant cells.

The authors point out in their conclusion, “peer raters” of the same level of training can provide accurate ratings of complex clinical tasks and can serve as an important resource in assessing student performance in an OSCE, but have not investigated the concordance of the pass/fail decisions with respect to individual candidates

Reliability (precision) and validity (accuracy) are two completely different and important methodological issues in all fields of researches. To assess the accuracy (validity) the following tests are used:-

• sensitivity (the percentage with the disease who test positive, True Positives / (True Positives + False Negative)),

• specificity (the percentage of healthy who test negative, True Negatives / (True Negatives + False Positive))

• positive predictive value (PPV), (percentage of positive tests who actually are diseased, True Positives / (True Positives + False Positive)),

• negative predictive value (NPV) (the percentage of negative tests who are healthy, True Negatives / (True Negatives + False Negative)),

• likelihood ratio positive and likelihood ratio negative as well as diagnostic accuracy [(both true positive and true negative results / total)× 100]

• odds ratio (true results / false results) preferably more than 50.

These are the tests to evaluate the validity (accuracy) of a test compared to a gold standard (Rothman et al. 2008).

Therefore, the authors’ conclusion is due to the confusion of reliability (precision) with validity (accuracy) and is, therefore, misleading.

Declaration of interest: The author reports no conflicts of interest.