How Good Is Good? Students and Assessors’ Perceptions of Qualitative Markers of Performance

Abstract

Background: Qualitative markers of performance are routinely used for medical student assessment, though the extent to which such markers can be readily translated to actionable pieces of information remains uncertain. Purpose: To explore (a) the perceived value to be indicated by descriptor phrases commonly used for describing student performance, (b) the perceived weight of the different performance domains (e.g. communication skills, work ethic, knowledge base, etc), and (c) whether or not the perceived value of the descriptors changes as a function of the performance domains. Methods: Five domains of performance were identified from the thematic coding of past medical student transcripts (N = 156). From the transcripts, 91 distinct descriptors indicating the language commonly used by assessors were also identified. From the list of 91 descriptors, Thurstone's method of equal-appearing intervals was used to extract 10 descriptors that were representative of the continuum of student performance. A modified paired comparisons method was then used to enable the relative ranking of each of 10 descriptors combined with each of 5 different domains of performance. A web-based survey was used to collect responses from participants (N = 209), which consisted of medical students and faculty members who were previously involved in student assessment. Results: Results demonstrated that respondents did not simply sum positive and negative descriptors in a uniform manner. Rather, comments on some domains (e.g., “ability to apply patient centred medicine”) were seen as particularly positive when associated with positive descriptors but not particularly negative when associated with negative descriptors. For others (e.g., “receptivity and responsiveness to feedback”) the reverse was true. Comments on “knowledge-base” elicited a relatively muted perception at both ends of the scale. Finally, the results also revealed moderate misalignment in the perceptions of assessors and students. Conclusions: The findings from this study suggest that the use of any given descriptor conveys slightly different meaning dependent on the context in which it is used. This helps to address some key issues surrounding the application of qualitative markers to performance assessment in medical education.