Evaluating a workshop designed to help medical students improve their skills in communicating about cancer: Methodological considerations—and worries

Abstract

Increasingly, communication skills training is being introduced into medical schools. Evaluations of post-training assessments document its success and add momentum to this movement. In addition, evaluations of student feedback shape course content and teaching methods. Two types of self-selection might bias evaluations and hence, lead to erroneous decisions: (1) students who attend a course may differ from nonattenders; and (2) students who give feedback may differ from those who do not. These two types of bias were examined in the context of a workshop designed to help medical students talk to patients about cancer. Overall, the need for communication skills training was endorsed. However, sex and experiential differences between attenders and non-attenders were identified and if replicated, could confound post-training evaluations. Similarly, differences between students who were more and less willing to provide feedback suggested that decisions based on student evaluations might systematically shape courses to the needs of unrepresentative student groups. Furthermore, qualitative data suggested that students who are least likely to give feedback, are those most likely to have critical or negative comments. Such a bias could mislead evaluators to underestimate the need for change. Rather than cancel each other out, biases associated with attendance and motivation to provide feedback combined. This is serious, because in practice, evaluators are likely to receive data only from those who both attend a given course and provide feedback. The danger is that evaluators may base decisions on information from a doubly self-selected minority of students.