Letters to the editor

Progress testing

Dear Sir

We congratulate Dr Van der Vleuten and his colleagues (Van der Vleuten et al., 2004) for implementing and sharing progress testing material within a School of Medicine and between Schools. They present progress testing as a way of measuring objectively the changes in knowledge of students. However, the suggested instrument needs further development before it can be used as one of the tools for such an assessment and we suggest a new dimension for consideration.

For the sample question described, and for which data are presented in the article, the authors discuss the differing patterns of increasing correct knowledge between schools in reference to their different curricula. The number of correct responses certainly does increase but so does the number of incorrect responses. Obviously the number of don’t know responses must decrease. In all three medical schools there were many students who gave an incorrect response in the later years. In one School in the final year, 38% gave the correct answer, 36% did not know and 26% gave the wrong answer.

The number of both correct and incorrect responses increased as the students progressed through the course and was highest in their final year. There are two possible reasons for these wrong responses, the students may have been guessing or they may have genuinely believed that their responses were correct, or there may have been a mixture of the two causes. Negative marking for wrong responses should minimize guessing.

If we assume that all incorrect responses were due to guessing based on ignorance, then if 26% guessed wrong, logic demands that 26% guessed correctly which means that of the 38% who gave the correct answer only 12% were based on knowledge. If we exclude guessing, then if a correct response indicates true knowledge, we must also accept that a wrong response shows false knowledge (some prefer to call this false belief). The students started with admitted ignorance. There is cause for concern if almost as many students gain false knowledge (belief) as true knowledge.

We have found the same trend in students at the School of Medicine of the University of Queensland. In a study one of us (Alexander, 1998) did pre- and post-testing of a group of final year medical students at the same institution during a Paediatric term. The number of correct responses rose in proportion to the decrease in ‘don't know’ responses, but the number of wrong responses remained steady.

In another study, we gave the same multiple true/false/don't know questions to different groups of students during their orientation on entry to the post-graduate medical course, and also at the beginning of Year 3 and near the end of Year 4. There were 92 questions covering the anatomy, physiology, pathology and management of two clinical cases. We also found the percentage of correct responses increased, and the percentage of ‘don't know' responses decreased from Year 1 to Year 4. Like Dr Van Der Vleuten et al., we found many incorrect responses that did not always decline with increased training.

Concerns about false knowledge are not new. The Greek philosophers discussed the subject but the best aphorism is by Colton (1780–1832) who wrote ‘knowledge is two-fold and consists not only in affirmation of what is true, but in the negation of what is false’. We know from studies addressing safety and quality issues that a significant number of errors in medical practice are knowledge-based mistakes (Wilson et al., 1995). Van der Vleuten and colleagues have shown that we may be doing reasonably well increasing true knowledge; we hope that their new development will give equal consideration to reducing false knowledge and/or guessing.

Heather Alexander

School of Medicine,

Griffith University

Queensland, Australia

Email: [email protected]

Alan Dugdale

Department of Paediatrics and Child Health,

School of Medicine

University of Queensland

Queensland, Australia

Email: [email protected]

Progress Testing: Response from C.P.M. van der Vleuten et al.

We thank our colleagues Alexander and Dugdale for their letter concerning our paper on cross-institutional collaboration. Of course it is interesting to see what could be inferred theoretically from response patterns. But we think one should be careful with over-inferring from one single item and with using estimates as true values. Please let us explain why.

The sample item in the paper was merely intended to demonstrate how item analyses could be used to compare curricula. Of course, inferences on the bases of only one item are very weak, as is the case with any n = 1 sample. Our intent was therefore to discuss how the whole 250-item test or the scores on four such tests per year could be used to assist the schools in their quality control. In fact, the focus of the paper was on cross-institutional collaboration. The particular test item was merely an example. That is why we would like to annotate Alexander and Dugdale's letter with some further nuances.

The authors seem to imply that the reasons why and how students respond to a certain question are a mere ‘know’ versus ‘don’t know’ dichotomy. We assume this because only if this were true would it be logical to assume that with many students and a large number of items the correctly and incorrectly guessed items would be symmetrically divided. But one could never infer this based on a single item. In fact, at this level it is even more logical that there is no symmetry.

Moreover, students are known to use various strategies to improve their chance of answering the question correctly. This implies that in such tests the majority of the items would be scored correctly more often than incorrectly. This is precisely the reason why items with a higher p-false value than p-correct value should be revisited for wording and content. We therefore disagree with our colleagues that one could simply subtract the percentage incorrect for the percentage correct and take the result as true knowledge on a single item.

We do not want to imply that our progress test is perfect. As reported in our paper we are constantly evaluating and improving it. The key message was to report on a successful collaboration, highlighting not only success factors but also problem factors and possible benefits. We hope the sample item used will not draw readers’ attention away from this.

C.P.M. van der Vleuten

L.W.T. Schuwirth

A.M.M. Muijtjens

A.J.N.M. Thoben

J. Cohen-Schotanus

C.P.A. van Boven

University of Maastricht, Netherlands

Email: [email protected]

Teaching chronic illness care

Dear Sir

Chronic illness care has become an essential issue of the German healthcare system. Medical students should be prepared for this. We have tried to develop attitude, skills and knowledge for chronic care management for undergraduate students.

We presented Wagner's ‘Chronic Care Model’ (Wagner et al., 1996) as a trans-sectoral approach to promoting continuous care: case management is an effective element including: (1) identification, (2) assessment, (3) planning, (4) coordination, (5) monitoring. Students were encouraged to understand that optimal chronic care is achieved when a ‘proactive practice team’ interacts with an ‘activated patient’. Students planned depression management for outpatients in small-group discussions (diagnostic, therapeutic and supportive procedures). We did a before-and-after evaluation (Regina, 1991) with 31 students (f = 14, m = 17) in the final year of their undergraduate training. They joined an obligatory 30-hour general practice course including this two-hour session (two groups: 14, 17). To evaluate students’ effort we presented two similar paper-cases on depression to be solved as an essay. We defined evaluation criteria from the Chronic Care Model to be used for a text analysis: right diagnosis, good clinical reason for diagnosis, comprehensive assessment of patient needs, using a patient register, active involvement of the patient, clear therapy targets, multidimensional aspects of treatment (social, psychological, medical), interdisciplinary therapy strategy, proactive case management, patient education, activating the patient for self-management. The maximum score was 11 points. The cut-off point for effort was defined at 70% (8 points). We analysed the essays from n = 25 (six students did not attend T1). At baseline (T0) students showed a mean score of 3.7 (SD 2.05); 8% achieved the cut-off point of 8. Two weeks after intervention (T1) students presented a mean score of 7.4 (SD 1.7) and 60% were above the cut-off point.

We found some limitations in our evaluation: (1) we used an instrument that has not been validated; (2) these were paper cases and not real student practice; (3) there was no randomized controlled design.

Medical students in undergraduate general practice courses do understand optimal chronic care. Medical students should be taught using the Chronic Care Model as a new paradigm in caring for chronic illness during their studies (Holman, 2004).

Jochen Gensichen Institute for General Practice, Johann Wolfgang Goethe-University, Frankfurt a.M., Germany Email: [email protected] Website: http://www.allgemeinmedizin.uni-frankfurt.de