Effectiveness of a programme design for the development of competence in solving clinical problems

Abstract

Background: To apply what has been learned theoretically in a clinical context is for many students a major challenge. In order to ease their transition into practice, a training programme was developed, focusing on learning to solve clinical problems.

Aims: The programme is designed for veterinary medicine students in the preclinical phase with already a sound theoretical base in biomedical and clinical sciences. The design is based on the engagement in learning and work processes derived from clinical practice and exposure to a large variety of real and paper-based cases. This article addresses the effectiveness of this programme design.

Method: Programme effectiveness was defined in terms of the progress made by the students. This progress was established using methodological triangulation of the results from student questionnaires, performance observations and (pre and post) assessment.

Results: On all dimensions of effectiveness, the results consistently showed that the programme was perceived as effective and led to improved performances in solving clinical problems and better solutions in the assessment. The students’ progress was substantial.

Conclusions: At preclinical level, a course design based on the work processes in clinical practice and a mixture of real and paper-based patients is effective in enhancing problem-solving competence.

Introduction

The ability to solve clinical problems and make the required decisions is nowadays recognised by many as a key competence in clinical practice (e.g. Higgs et al. 2008). For a long time, clinical problem-solving had been regarded as a rational process of merely applying logic to medicine (Phillips 1995; Rao 2007). Due to a growing understanding of the complexity of problem-solving processes and the many sources and opportunities for making erroneous decisions, training programmes and assessments explicitly containing aspects of clinical reasoning and decision making have become part of medical curricula.

In the preclinical phase, case-based learning and problem-based learning (PBL) are commonly used as instructional designs to support the development of clinical problem-solving competence. Whereas the strengths of these learning approaches have been recognised (Walker & Leary 2009) as well as debated (Kirschner et al. 2006), alternative designs to optimise learning have been explored (Moust et al. 2005; Srinivasan et al. 2007). This study concerns such an alternative design that combines a case-based instructional format with opportunities to practise and experience real-life clinical situations. It builds on theories about authentic learning (Anderson et al. 1996; Seel 2001) and instructional design (Paas et al. 2003; Jonassen 2004), employing a mixture of paper-based cases (tutorials) with real patients (clinical practicals) to engage students with a sound knowledge of basic and clinical sciences in solving high-fidelity clinical problems. The programme is intended to reinforce links between theory and practice and to ease the transfer from learning preclinical subjects to their application during clerkships.

Table

Practice points

Instructional formats built on authentic work processes offer valuable opportunities to practice at preclinical level with clinical problem-solving. Just-in-time provision of case information can be used to raise case fidelity, while maintaining a level of complexity students can handle. Providing students with just-in-time case information as well as scaffolding the process requires teachers to manage roles and activities concurrently. Evident performance improvements concern the students’ fluency and speed of work, ability to handle processes concurrently, their scope of relevant aspects and the appropriateness of solutions.

A preceding proof-of-concept study established that the instructional design, on which the programme is based, is valid with regard to facilitating the processes and learning functions that are essential for development of competence in clinical problem-solving (Ramaekers et al. 2011a). Table 1 presents the key features of the instructional design. In this study, the question is whether the programme proves to be as effective in practice as might be expected from the theoretical arguments on which the design was based. Discrepancies between what was actually attained from the training programme and its original design may result from conceptual shortcomings underlying that design (Jonassen 2004; Norman et al. 2005), as well as from practical considerations (Van den Akker 2003; McKenny et al. 2006).

Table 1  Key features of the clinical lessons’ course design (Ramaekers et al. 2011a)

Key features at the level of ‘separate meetings within the course’

1. (Authentic) cases are the unit of work

2. Activities and sequence: the problem-solving process during the meetings is based on the procedures underlying patient assessment and preventive screening

3. The required case-specific information is provided just in time

4. Role of students: the instructional formats require/build on a high level of self-directed learning and critical co-operation between peers

5. Role of teachers: teachers combine the provision of information with scaffolding students in their problem-solving process and assessing their performance

6. Regulation: between phases in the process and at the end of a case discussion, time is allocated specifically to reflection and feedback

Key features at the level of the ‘course as a whole’

7. The course covers a large variety of animals and conditions, representing the situations and problems that veterinarians are confronted with in their practice

8. As the course progresses, cases are placed in a sequence of progressive complexity and scaffolding support is gradually reduced

9. The different work formats have complementary strengths with regard to achieving the course objectives

10. Longitudinally repeated assessments are used to monitor progress and establish the achieved level of competence in clinical problem-solving

The purpose of this study is to establish the extent to which the design is effective in enhancing the development of competence in solving clinical problems. To achieve this, the issue of effectiveness is approached in three complementary ways:

1. Did the students perceive the programme as effective with regard to raising their level of competence in solving clinical problems?

2. Did the programme lead to changes in the students’ approach to clinical problems that are consistent with progress in the development of problem-solving competence?

3. Did the programme lead to an improved performance in terms of an improved quality of solutions to clinical problems?

Educational context and design

In 2004, the curriculum components for training clinical problem-solving in veterinary medicine (Utrecht University) were re-designed as part of a large-scale curriculum revision. The new programme, the so-called ‘clinical lessons’, was designed to suit current views about the development of competence in solving clinical problems and effective instructional designs. These views express: (a) the importance of learning around realistic examples, (b) which accurately represent the range of ways in which conditions clinically present, (c) in sufficient numbers and starting early in the curriculum, (d) providing practice in both analytic (deduction) and non-analytic (recognition) problem-solving processes, and (e) supporting the linking of specific cases and contexts with general theories, comparing across cases and practice with error-checking strategies. (e.g. Jonassen 2004; Eva 2005; Ramaekers et al. 2011a)

The clinical lessons extend through to the last (fourth) year before clinical clerkship and take up a substantial part of the weekly coursework. The core of the clinical lessons consists of three complementary educational formats: clinical practicals, demonstrations and tutorials. Whereas the first two involve real clinical patients, the latter build on paper-based cases. In the clinical demonstrations, students demonstrate their patient assessment and clinical problem-solving process to their peers. In all formats, the students direct the exploration of clinical problems and discussions to establish optimal ‘solutions’. The teachers’ primary roles are to support and guide students in the process, provide them with additional patient information at their request and assess performances and progress.

The instructional design of the clinical lessons shares many characteristics with traditional PBL. Few differences between these instructional formats may affect the dynamics of sessions:

• The clinical lessons are made up of paper-based cases mixed with real patients.

• In the clinical lessons, case information is provided just in time. Beforehand, students only receive limited case information; in the process, they receive additional information at their request.

• Even though cases are intended as examples and primarily provide an opportunity to practise, students focus on solving the case rather than identifying the knowledge they still lack and need to acquire.

• As a consequence of the instructional format with just-in-time provision of additional case information, the role of the teacher in the tutorials of the clinical lessons is more prominent than in PBL.

• Whereas in PBL, student collaboration is considered essential, the simulated patient assessments that are part of the tutorials of the clinical lessons emphasise an approach with only one or two peers. Active involvement of other peers is confined to the discussions in between phases of the simulated patient assessment.

As regards the development of competence in clinical problem-solving, empirical research and theories have shown some changes in the approach of cases that indicate progress:

1. An increased speed and fluency when concurrently taking patient history or executing tests, processing case information, reasoning and making decisions, elaborating on findings or justifying choices (Custers et al. 1996; Van de Wiel et al. 1999).

2. A transitory increase in explicit application of biomedical theories and integrating knowledge from different domains when analysing and explaining the specific case problems (Boshuizen & Schmidt 1992; Groothuis et al. 1998; Peterson 1999; Van de Wiel et al. 2000).

3. A widening scope and awareness of relevant contextual features, including underlying functional and structural components, enabling conditions and case dynamics. This awareness supports an increased accuracy of problem analysis and decisions (Randel et al. 1996; Hmelo-Silver & Pfeffer 2004).

4. Improvements with regard to early diagnostic hypotheses, the use of obtained information to guide the assessment of these hypotheses and meta-cognitive monitoring of the progress made (Elstein et al. 1978).

Together, these changes result in (5) an ability to handle more complex cases successfully, (6) reduced dependence on teacher guidance and (7) mastering a variety of problem-solving approaches, including non-analytic processing (McLaughlin et al. 2008).

Methods

In line with the methodology of design-based research, a specific component of the programme has been adjusted every year and the effects of this adjustment have been studied. In 2005–2006, adjustments were geared towards optimisation of teaching, particularly regarding student guidance (Ramaekers et al. 2011b). In 2006–2007, the cases for tutorials and progress in case complexity were reviewed to achieve optimisation. This study focuses on the academic year 2007–2008, the first year after both student guidance and cases had been optimised.

This study drew on a mixture of qualitative and quantitative methods and used ‘methodological triangulation’ complementary (Johnson & Onwuegbuzie 2004) to address the multidimensional character of educational outcomes and effectiveness. Each of the used methods had its own perspective on dimensions of effectiveness with regard to competence in clinical problem-solving. Table 2 presents the conceptual relationship between the data collected and the methods used (Duffy 1987).

Table 2  Data structure in triangulation

Level	Processes	Instrument/method	Categories of variables	Outcomes
Case level	• Problem-solving process	Observations (B) throughout the course	Components of clinical problem-solving	Changes in approach to clinical problems
	• Learning process	Questionnaire (A2) at the end of case discussion	Qualities of cases, teacher performance and work format	Perceived effectiveness of single case discussion related to learning
Course level	• Development process	Script Concordance Test (SCT) (C)	Different problems, levels of complexity and different types of judgements and decisions	Improved accuracy of solutions to clinical problems
		Questionnaire (A1) at the end of the course	Opportunities to practise and support learning and development processes	Perceived effectiveness of course as a whole related to competence development

Note: The additions A1, A2, B and C refer to the three different methods of analysis.

Table 3  The coding scheme (main categories) for observations of the clinical lessons

Problem-solving phases	Supportive-learning phases
Initial case information (C-INFO) Checking vital functions (C-VITA) Anamnesis (C-ANAM) Initial problem description (C-PROB) General patient assessment (C-GENA) Initial diagnostic hypothesis (C-INIT) Specific patient assessment (C-SPEA) Differential diagnosis (C-DDX) Choice of treatment modalities (C-RX) Execution of treatment (C-EXEC) Review of effectiveness (C-EFF)	Instruction beforehand (E-INFO) Time-out (E-TO) Evaluation (E-EVAL) Teacher-guided discussion (E-COLL)
Teacher behaviours	Students’ reasoning (behaviours)
Providing answers (T-ANSW) Asking questions (T-QUES) Additional statements (T-ADDS) Scaffolding the process (T-PROC) Stimulating group interactions (T-GROU) Guiding reflection and feedback (T-EVAL)	Choice of strategy (R-STRAT) Gathering information (R-GATH) Organising information (R-ORG) Interpreting information (R-INTP) Making judgements (R-JUDG) Making decisions (R-DECI) Justifying judgements and decisions (R-JUST) Other (R-OTHR)

Table 4  Results questionnaire at the course level

N = 129	M	SD
Opportunities (α = 67)
The clinical lessons offered a valuable opportunity to:
• practise with solving clinical problems	4.39	0.64
• learn from instructive cases and patients	4.25	0.61
• extend and deepen my knowledge base	4.30	0.63
• practise with performing relevant clinical skills	3.83	0.85
Effects (α = 0.82)
The clinical lessons have increased my:
• ability to deal effectively with clinical problems	4.13	0.68
• ability to justify my clinical decisions	4.07	0.62
• awareness of my own competence level with regard to professional conduct	3.39	0.96
• motivation for this degree study	4.29	0.71
• understanding of how various subjects within this degree study relate to each other	4.04	0.70
• readiness for clinical clerkship	4.09	0.67

Data collection

Questionnaires: Perceived effectiveness of the programme

The perceived effectiveness of the programme with regard to clinical problem-solving competence was established in two ways:

• (A1) Course level: The students’ appreciation of the course as a whole, its design and effectiveness were evaluated at the end of year. This questionnaire consisted of 26 items structured around the instructional design, opportunities to practise and perceived effectiveness of the course; 129 questionnaires were returned, covering 75.8% of all participating students.

• (A2) Case level: A questionnaire with 15 items about the ‘case attributes’, ‘teacher performance’ and ‘general qualities of the work format’ (including its educational effectiveness) was completed immediately after each clinical lesson by four students, two leading the case exploration and two observing. A total of 1814 completed questionnaires were returned, covering 627 (94.4%) of the sessions that took place.

The items in the first questionnaire were derived from the literature about case-based learning formats and instructional design, while the second questionnaire was based on the results of evaluations and observations during the 2 years prior to this study. In both questionnaires, students were asked to indicate their degree of agreement on a five-point Likert scale (1 = completely disagree, 5 = completely agree) for each item. The full questionnaires are available from the first author.

(B) Observations: Problem-solving process

A total of 14 clinical lessons at the beginning and end of the course were observed and recorded on audio tape to allow for an in-depth qualitative analysis. These observations related to 14 different cases, six student groups and six teachers. Each student group and teacher was observed at least twice. They gave informed consent to be recorded on audio tape. The observing researcher SR did not actively participate in any case discussion.

Recordings of the clinical lessons were analysed, with a single case discussion as the unit of analysis. The coding scheme was developed and refined on the basis of the conceptual framework underlying this instructional design and the behaviours actually observed (Ramaekers et al. 2011b). Four main code categories were distinguished (Table 3): problem-solving phases, supportive-learning phases, student reasoning behaviours and teacher behaviours. The ‘behaviour’ categories were nested within the ‘phases’. Behaviours concern single utterances while phases cover larger segments of a case discussion. All case discussions were coded by the first author (SR). Inter-rater agreement was used to establish the consistency of the coding scale. The inter-rater agreements for the ‘problem-solving’ and ‘supportive-learning’ phases were high (K = 0.92) while those for ‘teacher behaviours’ and ‘students’ reasoning behaviours’ were substantial (K = 0.75).

Student utterances were coded interpretatively (Miles & Huberman 1994), linking their behaviour to the (cognitive) activities that make up ‘clinical problem-solving’: the gathering, interpreting and organising of information, establishing and testing hypothesis, drawing conclusions, making and justifying decisions and choices. Next, qualitative analysis of the specific content of the discussion and cross-case comparisons were used to shed light on changes in the students’ reasoning during the year. These changes in reasoning were checked against the afore-mentioned indicators of progress in problem-solving competence.

(C) Assessment: Quality of problem solutions

To establish, independent of teacher guidance and interventions, to what extent the students at the end of the course had improved in solving clinical cases they were confronted with, an SCT was developed, which specifically focused on clinical reasoning and decision making in realistic cases in veterinary medicine (SCT-VM). This test was administered twice, near the beginning and at the end of the course, to reveal the progress made during this course. Students participated voluntarily; test results were neither part of the course assessment programme nor revealed to the teaching staff. The students received individual feedback about their own scores and guidance in the interpretation of results.

The SCT-VM consisted of 30 cases and 120 items representing a variety of conditions that are common in primary veterinary care and different types of clinical judgements and decisions. Items in an SCT contain a hypothesis or a proposal for action as well as additional case information (Figure 1). The participants were asked to consider whether the suggested hypothesis or action was supported by all available case information or not. As the cases were intended to represent authentic problems, they included the kind of uncertainties that are common in clinical practice. The answer key of an SCT was, therefore, established by a reference panel of experts. The degree to which answers from participants concurred with those of the experts determined the score of the participants and indicated their level of competence.

Figure 1. Example case in the SCT-VM.

The answer key was based on the responses of 27 experts. Of all students on the course, 168 (97.7%) participated in one of the test admissions and 148 (86.0%) in both. Details about the methodological qualities of this test have been reported earlier (Ramaekers et al. 2010).

Methodological triangulation

Triangulation was geared towards combining complementary data to achieve a comprehensive view on the effectiveness of the course design. All data involved the cohort of students participating in the clinical lessons for 1 year, with the same cases (tutorials and demonstrations) and teachers executing the programme. Although various data were available at the level of individual students (e.g. scores on the SCT-VM), findings from the observations and some results from the questionnaires were probably influenced by group dynamics. Therefore, the student cohort was taken as the unit of analysis for triangulation.

As for the triangulation procedure, the data from each method were first analysed separately from the other methods (Black 1994). Then, findings were combined at the conceptual level to elaborate on separate results and establish the extent to which results converged or diverged.

Results

Perceived effectiveness of the programme

Course level

At the course level, student appreciation ranged from 4.39 (valuable opportunity to practise with solving clinical problems) to 2.94 (clarity about the expected depth of preparation), with an average of 3.75. Ten items from this questionnaire concerned the students’ appreciation of the learning opportunities and the perceived effects of the programme. Eight of them scored above 4.0 (Table 4).

About the conditions possibly affecting their learning opportunities (not included in Table 4), the students were least satisfied with the time available for preparation (3.02 ± 0.94), clarity about the expected depth of case preparation (2.94 ± 0.99), the aptness of their prior knowledge (3.27 ± 0.84) and the transparency of assessment criteria (3.26 ± 1.05). In addition to this, a number of students expressed that the differences between teachers sometimes created uncertainty that had a negative effect on case discussions and learning.

Case level

At the level of separate cases, student appreciation ranged from 4.45 (authentic clinical problems and circumstances) to 3.11 (frequent teacher interventions), with an overall mean of 3.87. During the year, these scores did not change significantly, except the one for ‘clarity about teacher expectations’. As the students progressed, this clarity increased significantly from 3.00 to 3.89 (Spearman's r_s = 0.106, p < 001).

The programme features that were most valued by the students were: authenticity of the case (4.45 ± 0.65), practice with clinical problem-solving (4.25 ± 0.69), teacher guidance (4.16 ± 0.71), attention given to evaluation/reflection (4.28 ± 0.83) and transparency of changes in teacher role (4.12 ± 0.88). Overall, case discussions were considered highly instructive (4.29 ± 0.67) and inspiring for their study (4.25 ± 0.72).

Changes in the students’ approach to clinical problems

Although student performances at the beginning and end of the course had some common characteristics, various differences were identified that exemplified the progress in solving clinical problems at this stage of student development.

Speed of work and fluency

It took the students about 4–6 weeks to master applying the procedures of patient assessment and farm health management screening at a level that supported their problem-solving process. Parallel to the development of elementary routines in performing recurrent parts of the assessment (anamnesis, observation, basic examination of organ systems and vital functions), their speed and fluency of work increased. Questions and considerations were more to the point and students became less hesitant about the choices to be made. Furthermore, emphasis gradually changed from taking history (relevant questions) and communication (clarity and building rapport) to interpreting and relating findings, drawing conclusions and deciding how to proceed.

Explicit use of theory

Throughout the course, the students reproduced vast amounts of factual knowledge relevant to the cases. During the first few months, however, mechanisms underlying case problems and findings were only discussed in depth when probed by the teachers. Rather than ‘building’ a comprehensive case-specific model relating problems and findings to theories and hypotheses, the students persisted in a stepwise process of elimination, centred around one diagnostic hypothesis at a time. It was near the end of the course that different lines of thoughts were handled simultaneously and new information was weighted against several hypotheses.

Scope

The range of potential relevant aspects that the students included in their case analyses increased during the course. In the first series of cases, student attention was typically focused on issues related to the pathophysiological process. Gradually, aspects such as enabling conditions, prevention and animal health management, costs of additional tests, owner expectations, public health and ethical issues were included as well. Similarly, a gradual change in focus took place from identification and concentration on one organ system to awareness of potential effects on other organ systems and systemic diseases.

Early hypotheses, meta-cognitive monitoring and procedural adjustments

The students’ first hypotheses were often mere lists of differential diagnostic possibilities linked to particular symptoms and not suppositions based on case-specific combinations of signs and symptoms. During the year, this did not change very much. Likewise, restricting the gathering of information to what was necessary remained an issue of discussion throughout the year. The possibility of taking ‘time-out’ for reflection on the progress made and how to proceed was highly appreciated and frequently used by the students.

Quality of solutions

To what extent the quality of the solutions improved owing to student progress could not be observed independent from other influences. The interactions with peers and support and guidance by the teacher commonly influenced problem solutions. Yet, during the last weeks of the course, the students managed successfully to practise with selected, complex cases covering many different aspects simultaneously.

Dependence on guidance and support

When considering teacher support only quantitatively, hardly any reduction was observed during the year. Nevertheless, the nature and focus of this guidance changed in due time from pointed questioning about theoretical background and strategy towards facilitating in-depth discussion about relevant issues and elaboration on clinical practices.

Variety in problem approaches

Despite the observed changes in scope and fluency in handling basic examination procedures, a clear development towards more variety in approaches to problems was not observed. Their strategic choices were actually often influenced by their teachers’ interventions. A majority of the teachers stressed a systematic–analytic, almost algorithmic, approach. Others reinforced a more heuristic approach and the need to selectively gather information that is required to achieve the level of certainty considered necessary.

Improved performance in solutions to clinical problems

Progress with regard to the quality of solutions to clinical problems, established independently from the discourse of case discussions and guidance from teachers or peers, was made clear with the SCT-VM. The students’ overall scores on the SCT-VM improved from the pre-course test (M = 74.9, SD = 5.5) to the post-course test (M = 79.6; SD = 4.9). This improvement was significant (t = 12.753, df = 147, p < 00025) and individual student scores in the pre- and post-tests correlated positively (r = 0.653, N = 148, p < 001). The effect size was large (Cohen's d = 89). Figure 2 shows a scatter plot based on both scores, indicating the student's progress. As represented by the regression line, the students with the lowest scores on the pre-test made the largest relative improvement.

Figure 2. Scatter plot showing student progress in the SCT.

Discussion and conclusion

On all three dimensions of effectiveness, the results were consistent and indicated that the programme led to the projected growth of student competence in solving clinical problems.

Perceived effectiveness with regard to competence in solving clinical problems

The students perceived the clinical lessons as effective for their ability to solve clinical problems and their readiness for clerkships, as well as for their study motivation and understanding relationships between subjects. The opportunities in this course to practise with clinical problem-solving and practical skills on instructive, real and paper-based clinical cases were highly appreciated. Further analysis of these results (Ramaekers et al. 2011b) indicated that the students attributed the effectiveness of case discussions largely to the quality of teacher guidance, the cases themselves, reflection at the end of each case and the feedback they received.

Changes in the students’ approach to clinical problems

Previous studies have highlighted potential discrepancies between student self-efficacy (or confidence) and performance in real-practice situations (e.g. Tousignant & DesMarchais 2002). In this study, the progress perceived by students in handling problem-solving processes was confirmed by the observed changes during the year. Most evident were the increased fluency and speed of work, advances in the ability to handle processes and activities systematically and a widening scope of relevant case aspects. The observed changes with regard to applying knowledge at a conceptual level and to procedural adjustments based on the interpretation of findings remained limited. This might be related to limitations on what can be achieved within the time frame of this course and in this phase of student development.

In this learning process, cases were solved by students interacting with their peers and a teacher. Hence, the quality of solutions cannot be taken as an independent measure of student learning outcomes. Nevertheless, by the end of the course, students were able to solve more complicated cases within the same amount of time and with a similar amount of interaction, and this indicates progress.

Improved quality of solutions to clinical problems

Whereas the findings from the observations mostly reflect student progress in handling the problem-solving process, the results from the SCT-VM express their competence development in terms of improved professional judgements and decisions. The strength of these results indicates that the progress the students made in this course was substantial. In his synthesis of over 800 meta-analyses relating to educational achievement, Hattie (2009) referred to an effect size of d = 0.4 as the so-called hinge point. He considered this as a minimum; desired effects are those above this point that are attributable to specific interventions or methods. The performance improvements (d = 0.89) that were established with SCT-VM were clearly above this standard.

In conclusion, the programme has been shown to be effective in the enhancement of student competence in solving clinical problems and has received much appreciation. None of the findings in this study revealed specific limitations with regard to the use of these design features and underlying principles in comparable educational settings. Even though the specific content is different for different health professions, the instructional format is general enough to cover a variety of clinical problem-solving tasks. Still, the effectiveness of the instructional format centred on work processes in practice and just-in-time provision of the required information may be dominated by the extent of the students’ prior knowledge, particularly when such a format is applied in knowledge-intensive domains. A second premise concerns the teacher's ability to handle these instructional formats effectively. This requires managing the following different roles and activities almost concurrently: providing information, scaffolding the process, monitoring performance and stimulating group interaction if necessary, taking care of patient well-being and safety issues. Such a combination of roles and activities is difficult to manage, and the teachers in this course were sometimes not successful in achieving it. Teacher training and support of the implementation process can remedy this.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.