A predictive validity study of medical judgment vignettes to assess students’ noncognitive attributes: A 3-year prospective longitudinal study

Abstract

Background: The admissions interview still remains the most common approach used to describe candidates’ noncognitive attributes for medical school.

Aim: In this prospective study, we have investigated the predictive validity of a semi-structured interview for admissions to medical school based on medical judgment vignettes: (1) ethical decision-making (moral), (2) relationships with patients and their families (altruistic), and (3) roles and responsibilities in professional relationships (dutiful).

Method: A group of 26 medical students from the Class of 2007 participated in the interview process and provided their subsequent performance results from clerkship 3 years later.

Results: Inter-rater reliability of the scored interviews was high (κ = 0.96). Our results provided evidence for both convergent and divergent predictive validity. Medical judgment vignettes scores correlated significantly with seven mandatory clerkship rotation in-training evaluation reports (r = 0.39, p < 0.05; to r = 0.55, p < 0.01).

Conclusion: This semi-structured interview based on clearly defined and scored medical judgment vignettes that focus on the assessment of medical students’ noncognitive attributes is promising for student's selection into medical school. The high reliability and evidence of predictive validity of clinical performance over a 3-year period suggests a workable approach to the assessment of ‘compelling personal characteristics’ beyond merely cognitive variables.

Introduction

In the process of making decisions about the qualifications of students admitted to medical school, the primary criteria has been weighted largely towards measures of academic achievement and cognitive measures. While these measurements are thought to be relatively good predictors of students’ academic ability, on average they have been found to account for about 23 and 6% of the variance in overall medical school and post-graduate performance,respectively (Ferguson et al. 2002), and has little, if any, relationship to the personal characteristics attributed to being a ‘good’ doctor. In a recent meta-analysis of 10 studies that examined the predictive power of interviews on clinical performance, an overall correlation coefficient of 0.17 (0.11–0.22, 95% confidence intervals) indicates a small, but positive effect related to existing interview processes (Goho & Blackman 2006). The modest predictive validity of cognitive variables (Donnon et al. 2007) and the poor predictive validity of noncognitive variables (e.g. interviews, letters of reference, etc) for medical school performance and beyond have resulted in concern about the admissions process to medical school.

With an increased demand for accountability in the decision about whom to admit to medical school and the continuing growth in the number of qualified applicants, the president of the Association of American Medical Colleges (AAMC) expressed serious concern that the truly ‘compelling’ personal characteristics of individual applicants are rejected for one or two isolated blemishes in the academic record (Cohen 2001). There has been a fundamental shift in the emphasis medical schools are now placing on the integration of both curricular strategies and cultural shifts that promote an environment conducive to professionalism development for both students and faculty (Whitcomb 2007). In criticism of the predominance of the research on measures of previous academic achievement, however, Ferguson et al. (2002) emphasizes the need for more studies that focus on identifying admission criteria to medical school that predicts clinical performance. The challenge at most medical schools is that there is no agreement on what noncognitive characteristics are most important, how they are to be measured appropriately, and an objective protocol for this information to be used in a consistent manner for student selection (Albanese et al. 2003).

Table

Practice points

In this prospective study, we investigate the predictive validity of a semi-structured interview for admissions to medical school based on medical judgment vignettes: (1) ethical decision-making (moral), (2) relationships with patients and their families (altruistic), and (3) roles and responsibilities in professional relationships (dutiful). Our semi-structured interview based on clearly defined and scored medical judgment vignettes that focus on the assessment of medical students’ noncognitive attributes establishes a promising protocol with high reliability and evidence of predictive validity of clinical performance over a 3-year period.

Although conducting interviews are costly and issues about their predictive validity exist, advocates of the process claim that there is no other reasonable way to select the most appropriate candidates from such a highly qualified pool of applicants. Traditionally, the interview process has been the primary method of compiling non-academic information about the applicant, such as personal qualities, that are difficult or impossible to obtain in any other way. The primary goal is to identify candidates with desirable characteristics appropriate for the practice of medicine as opposed to admitting students without these particular attributes and consequently left with the task of trying to instill them. Although an admission interview protocol is nearly a universal approach used at most medical schools to measure candidates’ humanistic or noncognitive attributes, the interview process is criticized for being poorly structured, typically biased, and having low reliability and validity (Puryear & Lewis 1981; Johnson & Edwards 1991; Ferguson et al. 2002; Goho & Blackman 2006).

Although the reliability of the interview has been improved through the use of structured approaches (Puryear & Lewis 1981; Powis et al. 1988; Edwards et al. 1990; Patrick et al. 2001; Eva et al. 2005), there still remains the issue of measuring relevant noncognitive domains of interest that extend beyond the self-confidence, motivation, and compatibility concerns related to the particular medical school. In identifying personality characteristics to measure, Taylor (1990) reported that 87 positive qualities of successful physicians are assessed in self-reported evaluation forms given to their applicants. Meridith et al. (1982) provides some of the best evidence in support of the semi-structured interview process. They found that applicant ratings on maturity, personal achievement/ability, motivation to practice medicine, and interpersonal skills correlated with the clinical assessment–accounting for twice the variance (10.4%) as the next highest predictor variable (MCAT science = 5.0%). Collins et al. (1995) focused on measuring communication, maturity, caring qualities/kindliness, and knowledge of community, political, social, and medical issues. Although these findings support the use of objective-structured interviews, further research is needed on identifying which compelling personal characteristics are to be assessed in an interview and how they can be measured.

In a move by the AAMC to establish consistent medical school objectives that meet society's expectations of physicians, a consensus was reached among leaders of 14 countries regarding the attributes that new doctors need to practice medicine (Collins et al. 1995; The Medical School Objectives Writing Group 1999). Of the four principal attributes identified, the first (physicians must be altruistic) is related entirely to the promotion of specific altruistic, noncognitive characteristics and the fourth (physicians must be dutiful) emphasizes the importance of enhancing the ability to work collaboratively with other healthcare professionals and develop strong interpersonal skills. With the current ratio of qualified applicants to positions available increasing, there is the concern that by establishing a strictly cognitive performance threshold cut-off (e.g. previous grades, standardized test scores, etc.) efforts to identify students with the personal characteristics, more conducive to medical practice may be impeded.

Accordingly, the main purpose of the present study was to investigate the reliability and validity of a semi-structured interview approach for assessing personal characteristics of good physicians. Specifically, we wished to conduct a 3-year prospective longitudinal study of the predictive validity of a semi-structured interview with students from the beginning of medical school through to their performance in clerkship. The major purpose of this present study was to evaluate the predictive validity of an alternative, semi-structured interview approach to measuring the compelling personal characteristics attributed to ‘good’ physicians (i.e. moral, altruistic, and dutiful).

Method

Medical judgment vignettes and scoring keys

The semi-structured interview was designed to measure students’ open-ended responses to medical scenarios that were classified into three broad categories: (1) major ethical dilemmas in medicine (end-of-life–euthanasia), (2) relationships with patients and their families (altruistic commitment and compassionate treatment), and (3) collaboration and clarification with staff and colleagues (dutifulness and understanding of medical relationships). The purpose of the semi-structured interview format used in this study was to develop a standard protocol that presented the medical dilemmas with a set of pre-determined or probing questions (Appendix) that were asked consistently to all participants while allowing for an open-ended response format (Powis et al. 1988; Edwards et al. 1990; Nowacek et al. 1996). Open-ended questions provide an opportunity for the respondent to introduce relevant information, personal ideas, and conceptual understandings that are potentially unique to the individual and may not appear on the established scoring rubric. Essential to the semi-structured interviewing process, however, an objective approach was maintained by the interviewer with each of the respondents. In particular, the same approach was used in the presentation of the vignettes, the asking of the probing questions, and the clarification and completion of responses obtained from the interviewees. For example, a simple ‘yes’ or ‘no’ response or short answer to a probing question would trigger the interviewer to encourage the respondent to elaborate further or use paraphrasing to clarify understanding.

In adhering to Kohlberg's stage theory of cognitive and moral development (Kohlberg 1981), we assessed participants’ stages of moral, altruistic, dutiful development using pre- to post-conventional levels (Table 1). As such, it is important to understand that there is no ‘right or wrong’ answer to the dilemmas and the stage we assigned to a particular response is based on their reasoning and rationale within a specific domain (i.e. moral, altruistic, dutifulness). Research conducted by Kohlberg and his associates has shown that while individuals may fluctuate above and below one particular stage, the majority of adults in Western societies typically reason at one predominant stage. Stage 6, the Universal Ethical Principle Orientation, is removed as a general measure of moral development as this stage reflects decisions of conscience-based self-chosen ethical principles appealing to comprehensiveness and universality associated with leaders such as Martin Luther King, Gandhi, and the Dalai Lama.

Table 1.  Overview of Kohlberg's stages of moral development

Pre-conventional level
Stage 1: Obedience and punishment orientation	Focus on avoidance of punishment by not breaking the rules.
Stage 2: Individualism and exchange orientation	Acceptance of alternative views as right and wrong is determined by what satisfies the individual's particular needs.
Conventional level
Stage 3: Good interpersonal relationships orientation	Meet the expectations of what is right because people expect it as part of mutual interpersonal relationships.
Stage 4: Maintaining social order orientation	Emphasis is on obeying social order, respecting the dignity of all while conforming to the laws of the group or institution.
Post-conventional level
Stage 5: Balance of social contract and individual rights orientation	Conceptualize society in a theoretical manner, stepping back from existing society and considering the relativity of group and individual values with respect to what society ought to uphold.

We did, however, slightly modify the standard guidelines defined by Colby and Kohlberg (1987) to expand across the other domains measured in the medical judgment vignettes. For example, at Stage 1 of the ‘altruistic’ or ‘dutiful’ domains the respondents do not identify themselves as members of a society and see the decision process regarding interpersonal or collegial relationships as something external to themselves. These respondents justify their reasoning from an authoritarian perspective about how the physician should interact with patients, staff, and colleagues based on a fixed set of rules established by the medical, health, and governing authorities.

Participants

All 100 newly admitted students to the medical school at the University of Calgary from the Class of 2007 were invited to participate voluntarily in the completion of a 20–30 min, semi-structured interview and provide subsequent clinical performance data from their in-training evaluation reports (ITERs) completed at the end of their clerkship final year. The 26 medical students that participated consisted of 16 females (61%) and 10 males (39%) with a mean age of 26.4 years (SD = 3.9; range 19–36). These participants were found to be representative of the Class of 2007 by both sex (60% females and 40% males) and age (M = 25.6 years, SD = 4.3); p > 0.05. This study was approved by the Conjoint Health Research Ethics Board of the University of Calgary and signed consent was provided by all participants.

Procedures and data collection

The semi-structured interview was administered by an experienced counselling psychologist. Each of the medical judgment vignettes were read aloud while the medical student followed along on a printed copy and, in turn, was asked to respond to a set of 3–4 open-ended probing questions developed specifically for each of the vignettes (Appendix). Each of the students’ responses were tape-recorded and their transcribed responses to each vignette was coded independently by two of the authors (TD and EOP) on the five-stage moral, altruistic, and dutiful response key. Although the respondents were told that the interview would take about 20–30 min, the time varied from as short as 12.5 min to as long as 23.2 min (M = 15.6 min, SD = 2.3). In addition, students’ scores from a six judge panel interview that were routinely employed at the medical school were collected. In this interview, typically five physicians and at least one representative of the community (e.g. a former patient) are given a general framework from which to ask candidates questions about why they are interested in being a doctor, their general academic performance, how they would deal with ambiguity, and their future career aspirations (e.g. interest in general practice or specialty). Each member of the panel independently rated the candidate on a global scale from 1 to 5 and an average score was tabulated from the summed total of the six judges’ interview scores.

The undergraduate medical education program at this medical school is 3 years. Students complete seven mandatory clerkship rotations (internal medicine, paediatrics, surgery, obstetrics/gynaecology, family medicine, psychiatry, and anaesthesia) in their third and final year. As part of the evaluation of the clerks’ performance in each of these rotations, an eight-item ITER is completed by the attending physician. The items are scored on a five-point scale from ‘Unsatisfactory’ to ‘Outstanding’ and are a reflection of the physicians’ perceptions of clerks’ knowledge of basic and clinical disease mechanisms (cognitive reasoning ability) to more practical skills related to history taking and physical examinations, communication skills with patients and families, and a sense of professionalism and responsibility. An average performance score on each item of the ITER was derived for each of the 26 medical students based on their scores across all seven of the clerkship rotations.

Results

Responses to medical judgment vignettes

An overall inter-rater reliability coefficient of κ = 0.96 was achieved between the two raters on the three vignettes for all 26 transcribed interviews. As shown in Table 2, students’ scores range across all five stages in their responses to the open-ended questions posed for each of the medical judgment vignettes (Appendix). In the first vignette, an emphasis is placed on the respondents’ moral reasoning capacity regarding a doctor's decision to support a terminal ALS (Amyotrophic Lateral Sclerosis) patient's request to die at home. Acting as her medical doctor of 15 years, he agrees to set up a self-administered IV system that will increase the rate of her morphine injections over a set period of time, until a lethal injection will be released to end her life. In this dilemma, the respondents recorded the highest mean score of all the vignettes at stage 2.65 (SD = 1.41).

Table 2.  Means (SDs) and raw scores of medical students (n = 26) on the moral, altruistic, and dutifulness medical judgment vignettes

Medical judgment vignette	Stage 1	Stage 2	Stage 3	Stage 4	Stage 5	Mean (SD)
Moral: Ethical decision-making in medicine	7	6	6	3	4	2.65 (1.41)
Altruistic: Relationships with patients and their families	4	10	8	2	2	2.54 (1.10)
Dutiful: Roles and responsibilities in professional relationships	7	10	2	6	1	2.38 (1.24)
Medical judgment vignettes total score (Min. 3–Max. 15)						7.58 (2.87)

The second vignette focuses on the altruism of the respondents as the interpersonal relationships with the patient and family are emphasized in this refusal of treatment scenario. The family physician in this dilemma is faced with a daughter and mother who have decided to end chemotherapy treatment, which is making the 15-year-old girl feel constantly sick and disoriented to pursue alternative therapies. The participants are asked to respond to questions about what the doctor should say to the mother and daughter about the medical system's ability to provide care, their decision to pursue alternative therapies, and the father's request for assistance in pursuing legal support for sole custody of the daughter to continue the chemotherapy treatment (M = 2.54, SD = 1.10). As illustrated in Table 3, excerpts of various students’ responses to and subsequent scores on the ‘altruistic’ Medical Judgment Vignette varied across all five stages. At Stage 1 of the altruistic scoring key, relationships with patients and families are based on authoritative positions. In this example, the student begins and ends their response without moving beyond the ‘power of the medical system’ and adhering to a ‘doctor knows best’ attitude removed from the development of an interpersonal relationship with the patient and her family. At Stage 2, the interactions with the patient and family members reflect acceptance of alternative approaches, but the response continues to show an indifference to the establishment of clearly defined interpersonal relationships with the doctor. By Stage 3, we see that the focus of the response involves the establishment of a ‘good’ doctor relationship with the patient and family that focuses on good motives and interpersonal feelings through a moderating role as ‘this girl is getting sicker and sicker and you cannot have a conflict between mom and dad at this time.’ At Stage 4, the relationship with the patient and family becomes more complex and involves elements of the previous stages such as acceptance of alternative views and the influential role the doctor can play through both professional and interpersonal support. While more broadly concerned with the interpersonal relationships with patients and their family, emphasis is also placed on using ‘evidence-based medicine,’ the seeking of legal advice, and the obligation of keeping the governing body informed to ensure that the social order is maintained as part of the process. Finally, at Stage 5 we see that the respondent has explored the next level where a balance between the expectations of a good and just society (‘… understanding the frameworks around Western medicine and alternative medicine …’) and the need of individual values of respect and dignity (‘helping the father maybe to understand the motivations behind the mother and daughter … but also helping the father and mother realize that the social benefits to their daughter of good family relations are going to do way more for their daughter's health than either of the two treatment modalities’) must be upheld in any society.

Table 3.  Excerpts of various responses to the ‘altruistic’ medical judgment vignette

Pre-conventional level
Stage 1: Obedience and punishment orientation	‘… I don’t think it's the doctor's role to suggest to people that they go elsewhere. He/she should stick to what they know best, mainly medicine, and that includes describing the limitations of medicine.’
Stage 2: Individualism and exchange orientation	‘… I think that as far as I know, it's always really up to the patient to determine what course they want to go on. If they want to pursue an alternative treatment, then I think that if the doctor has explained everything that he can to them …’
Conventional level
Stage 3: Good interpersonal relationships orientation	‘I think you’re always kind of the moderator on these. Especially in the role of family physician, I think you have to pursue both sides of this … that it's a horrible decision to have to make. Especially for a 15-year-old girl … I think the family doctor could play a role in somehow moderating this …’
Stage 4: Maintaining social order orientation	‘… I guess the family doctor would have to say that if they are confident that the 15-year-old girl and her mother have made sound decisions in their practices, and he/she would have to defend them in their decisions. And that he would probably need to outline the conflict to this governing body because that seems to be of pretty big importance to this case.’
Post-conventional level
Stage 5: Balance of social contract and individual rights orientation	‘The family doctor can offer the treatments that are available through the Western system which in this case seems to be ongoing chemotherapy. So, the family doctor's obligations are to outline the options available, which are to be on chemotherapy or not be on chemotherapy … As a family doctor, I would try to work with the mother and daughter to try and blend the two approaches and find a middle ground they might be comfortable with …, but also helping the father and mother realize that the social benefits to their daughter of good family relations are going to do way more for their daughter's health than either of the two treatment modalities.’

The third vignette focuses on an emergency room patient encounter where the attending physician, is presented with an aboriginal youth who has been stabbed in the arm and brought in with his parents by two police officers. The scenario focuses on explaining to a new Canadian doctor the complexities related to addressing the needs of persons’ of aboriginal descent and providing personal perspectives on how to reach out to aboriginal communities to potentially deal with health-related concerns unique to this culture (M = 2.38, SD = 1.24).

Validity of medical judgment vignettes interview

The correlation of the composite medical judgment vignettes interview with that of the mean scores from the six panel judges’ interview was zero (r = 0.03, p > 0.05). The correlation coefficients between ITER scores with each of the three vignettes, a medical judgment vignettes total score and scores from the six panel judges’ interview are presented in Table 4. Internal consistency reliability for all seven ITERs were high (Cronbach's α = 0.85–0.93).

Table 4.  Correlations between ITER items in clerkship and scores from the traditional six judge panel and medical judgment vignettes (i.e. moral/altruistic/dutiful) interview formats

		Medical judgment vignettes
ITER Items (for all seven mandatory clerkship rotations)	Six panel judges total	Moral	Altruistic	Dutiful	Total
1. Fund of knowledge (basic and clinical) and understanding of disease mechanisms	0.55**	0.21	0.35	0.03	0.25
2. Clinical skills: History taking and physical examination	0.20	0.34	0.49*	0.24	0.46*
3. Communication skills: Patients interviews and oral case presentations	0.21	0.39*	0.22	0.23	0.37
4. Record keeping: conscientious case write-ups and progress patient management notes	0.09	0.39*	0.16	0.22	0.35
5. Problem solving consistency and strong patient management skills	0.21	0.47*	0.45*	0.50**	0.62**
6. Professionalism and responsibility: Assumes patient care responsibilities and is conscientious	0.05	0.51**	0.26	0.46*	0.55**
7. Motivation: Self improvement and educational initiative	0.23	0.50**	0.20	0.30	0.45*
8. Relationships: Sensitive to patients and fulfils responsibilities as a team member	0.18	0.42*	0.21	0.19	0.37

Note: *p < 0.05, **p < 0.01.

The global assessments of the six panel judges show a moderate correlation with the medical students’ performance on the first item related to demonstrated knowledge and understanding of disease mechanisms (r = 0.55, p < 0.01), but fails to correlate significantly with any of the other seven ITER items. The correlations between the ITER items and the three vignettes (i.e. moral, altruistic, dutiful) and a total composite score representing the sum total of students’ vignette scores (i.e. range of scores from a low of 3 to a high of 15) are shown separately in Table 4. The significant correlations between students’ total medical judgment vignettes score with most ITER items were significant and ranged from 0.39 (p < 0.05) to 0.51 (p < 0.01).

Discussion

The main findings of the present study are (1) high inter-rater reliability was achieved for the medical judgment vignettes, (2) there was only one significant correlation between the six judge global ratings and ITER scores 3 years later, (3) there were no significant correlations between the six judge global ratings and the medical judgment vignettes scores, and (4) there were a number of significant correlations between the medical judgment vignettes and ITER scores attained 3 years later.

The significant correlations between the medical judgment vignettes and the ITER items provide evidence of predictive validity. Specifically, correlations between total vignette score and their ratings across the seven mandatory clerkship rotation ITERs on ‘Professionalism and Responsibility: assumes patient care responsibilities and is conscientious’ (r = 0.55, p < 0.01) and ‘Problem-solving consistency and Strong patient management skills’ (r = 0.62, p < 0.01). For the remaining noncognitive ITER items, small correlation coefficients (r = 0.35, p = 0.08, to r = 0.46, p < 0.05) were found for measures such as ‘Communication Skills’, ‘Relationships’, ‘Motivation’, and ‘Clinical Skills: History Taking and Physical Examination.’ These correlations provide convergent validity evidence for predictive validity (performance in similar domains–noncognitive) while the zero correlation with ITER item 1 (knowledge and cognitive variables) provides divergent validity evidence for predictive validity.

These results are particularly notable as the data have been collected 3 years apart. The relevancy of these findings can, in part, be compared to the findings of a recent exploratory meta-analysis of the effectiveness of academic admission interviews to health-related professions (Goho & Blackman 2006). In this study, a mean effect size of 0.17 (0.11–0.22, 95% confidence intervals) was found for predicting clinical success from performance on an admission interview. In comparison, the predictive performance on the medical judgment vignette total score with noncognitive measures of students’ clinical performance across the seven mandatory clerkship rotations ranged from an effect size of 0.35 to 0.62 (accounting for 12–38% of the variance). Separately, the moral vignette appeared to be the best predictor of clinical performance related to motivation (r = 0.50, p < 0.01) and relationships (r = 0.42, p < 0.05) while the altruistic vignette predicted students’ clinical skills related to history taking and physical examination (r = 0.49, p < 0.05). All three medical judgment vignettes showed a mean effect size of r = 0.48, p < 0.05, in predicting performance on ‘problem-solving consistency and strong patient management skills.’

The conventional subjective approach to interviewing candidates for medical school (the six judge panel of global ratings) did not correlate with noncognitive components of the ITER forms in clerkship. Interestingly, there was significant correlation with knowledge (basic science and clinical) of disease mechanisms. It is probable that the judges are focusing on candidates’ cognitive characteristics (e.g. MCAT scores and undergraduate achievement in the students’ files). Conversely, the medical judgment vignettes scores were not correlated with the knowledge item of the ITERs yet were correlated with ITER items of clinical performance. Similarly, the medical judgment vignettes were not correlated with the six judge global ratings that are correlated with knowledge. This pattern of correlations provides both convergent (significant correlations with clinical performance) and divergent (no correlations with knowledge measures) evidence for the predictive validity of medical judgment vignettes scores for clinical performance 3 years later.

Although maturity (age) has been shown to play an important role in Kohlberg's (1981) theory, nearly half of the present participants (n = 14; 48.3%) were classified as pre-conventional (Stages 1 and 2) moral reasoning and only four (13.8%) were in the post-conventional phase. Similar results were obtained for altruism and dutifulness. It is evident from these results that many in the present sample are functioning at a low level of moral reasoning, altruism, and dutifulness. Few (around 14%) have achieved higher levels of these characteristics (post-conventional) as might be expected of students at this level of educational maturity (medical students). About half of our sample is functioning at the lowest level that is authoritarian and rule-bound reasoning (Stage 1: Obedience and Punishment; Stage 2: Individualism and Exchange). Importantly, the stages of development are correlated with clerkship ITER scores 3 years later as the higher level students outperform their less well-developed colleagues on noncognitive measures of clinical performance.

The results indicate that our semi-structured, medical judgment vignettes interview has promise as a screening device for admission to medical school. Nonetheless, there are limitations to the present study. First, the size of the sample was relatively small. Second, the process we used in the selection of the participants was voluntarily (although they were representative of their class on age and sex composition). Third, the limited range of domains explored through the vignettes (i.e. moral, altruistic, and dutiful) could have been expanded to look at a broader range of characteristics as the time to complete the interviews was found to take just over 15 min on average.

Currently, research is focusing on the reliability and validity of the medical judgment vignettes interview employing larger samples of applicants. Studies that also look at the temporal changes (e.g. educational, maturational) of moral, altruistic, and dutiful development are also being considered. Although the current findings support a consistency in how the students present on noncognitive measures from the initial interview and 3 years later in clerkship, it is important to investigate whether there is a corresponding increase in the development of these attributes over the course of their program that may have implications for the promotion and teaching of medical professionalism. Longitudinal research, however, does allow for further investigation into the predictive validity of the vignettes with subsequent performance on clerkship, residency, and beyond into independent clinical practice.

In conclusion, the medical profession has placed a great emphasis on identifying the personal characteristics or noncognitive attributes in the selection of medical students. Current approaches involving loosely structured individual or panel interviews (frequently with poorly trained interviewers) are not working well. Our semi-structured interview based on clearly defined and scored medical judgment vignettes that focus on the assessment of medical students’ noncognitive attributes establishes a promising protocol with high reliability and evidence of predictive validity of clinical performance over a 3-year period. Nevertheless, the promotion of these noncognitive attributes also needs to be explored as there is an increasing expectation that the development of such personal characteristics as altruism and dutifulness will increase over the course of medical school and beyond. Meanwhile, we have demonstrated a workable approach to the assessment of ‘compelling personal characteristics’ beyond merely cognitive variables.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Tyrone Donnon

TYRONE DONNON is an Assistant Professor in the Department of Community Health Sciences and Member in the Medical Education and Research Unit, Faculty of Medicine, University of Calgary, Canada.

Elizabeth Oddone-Paolucci

ELIZABETH ODDONE-PAOLUCCI is an Assistant Professor in the Departments of Community Health Sciences and Surgery, Faculty of Medicine, University of Calgary, Canada.

Claudio Violato

CLAUDIO VIOLATO is a Professor in the Department of Community Health Sciences and Director in the Medical Education and Research Unit, Faculty of Medicine, University of Calgary, Canada.

Appendix: Medical judgment vignettes

Principles of ethical decision-making (Moral)

Table

For the past 5 years, a 45-year-old woman has been suffering from muscle degenerative effects of terminal Amyotrophic Lateral Sclerosis (ALS). Since being diagnosed with ALS, her condition has steadily deteriorated but most rapidly in the last year to the point that she is confined to a motorized wheel chair and is now completely dependent on 24-h nursing care for most aspects of her daily life. Although there is no cure for her condition, it is expected that she will live for at least another 2–3 years, but in that time her health will continue to deteriorate to the point that she will completely lose her ability to speak and will eventually require the use of a life support system to stay alive (i.e. ALS is a disease that will ultimately disable all body movement, and yet a person with ALS can continue to be kept alive artificially). In support of her request to die at home, her medical doctor of 15 years has agreed to set up a self-administered IV system that will increase the rate of her morphine injections over a set period of time. At a certain point, the system will release what will be a lethal injection of morphine to end her life.

Probing Questions:

• Do you think the doctor made the right decision in helping her to commit euthanasia?

• Should the doctor lose his license and, hence, ability to practice medicine?

• Should the doctor be arrested and serve time for his actions?

Relationships with patients and their families (altruistic)

Table

A 15-year-old teenage girl is suffering from swollen glands and complains of being lethargic. After a number of tests the family doctor and specialists confirm the diagnosis of astrocytoma, grade 4, brain cancer (i.e. advance stage of development). The family is informed that there is no known cure for this type of cancer, but the doctors’ suggest that they can slow the process if they begin to address the disease immediately through ongoing chemotherapy. After a few days, the mother and daughter decide to end the chemotherapy treatments, which they say are leaving the girl feeling constantly sick and disorientated. Instead, the mother and daughter decide to pursue a variety of alternative non-toxic therapies outside of the recognized medical system of practice (e.g. herbology, nutritional modification, vitamin therapy). The girl's father, however, is in direct conflict with his wife and daughter and wants them to return to the original chemotherapy treatment plan. After a frustrating week of family discussions, the father has decided to take legal action against his wife for sole custody of his daughter in support of his decision to get her back into chemotherapy.

Probing Questions:

• What should the family doctor say to the mother and daughter about the medical system's ability to provide care at the cancer treatment centre?

• What should the family doctor say to the mother and daughter regarding their decision to pursue alternative therapies?

• What should the family doctor say to the father when he comes to ask for assistance in pursuing support from the doctor in convincing his wife and daughter to continue chemotherapy treatment?

• The specialists and health authority administration are requesting further information on the patient's care, how should the doctor respond?

Roles and responsibilities in professional relationships (dutiful)

Table

A doctor with little experience working in Canada has just asked you for assistance with a patient, and his family, that was checked into emergency for a knife wound about two hours ago. The knife wound is not serious, but the patient and family are being difficult in cooperating with the acting doctor, nursing staff, and the police officers in the treatment of the injury. The 17-year-old male patient is of aboriginal descent and has consumed a considerable amount of alcohol. While at a party on the Reservation, he got into a fight with another young aboriginal male his age and was stabbed twice in the arm before the fight was broken up. The patients’ mother and father also live on this high-needs Reservation and simply want the doctor to patch up and release their son so that they can take him home. They are refusing to lay charges against the other young man and ignore the frustrated police officers’ attempts to answer any further questions. After the patient has been bandaged and treated, the family leaves quickly.

Probing Questions:

• The new Canadian doctor asks you to explain why the aboriginal parents were so reluctant to assist the doctor and police in helping their son out. What do you say?

• The new Canadian doctor asks you to explain what some of the issues are facing the health of aboriginal peoples in Canada. What is your response?

• In a follow-up visit with the patient and his mother and father what should the doctor suggest the family do next?

• Upon further investigation, you find that there is a history of aboriginal patients that come through the emergency room, get various inflictions ‘patched up’, and leave as quickly as possible. What can be done to improve this situation?