Educating capable doctors—A portfolio approach. Linking learning and assessment

Abstract

Background: Teachers want students to focus on their learning to become capable doctors; yet, students primarily want to focus on passing their exams. How much of this paradox is explained by learning and assessment being seen as two different entities rather than as the continuum of one and the same process? How may the two areas be more closely and effectively linked?

Aim: This article describes and illustrates a conceptual framework for an approach termed capability-based portfolio assessment.

Results and conclusions: Thinking about capability, i.e. the ability to perform in the real world, is needed for a contemporary curriculum and assessment design. A capability-focus will help students to integrate the foundations of medical practice with learning how to become a capable, reflective and life-long learner.

A well-structured capability portfolio, regularly presented and reviewed, will be a useful tool to guide the journey, and should have the potential to help drive deep learning and allow the assessment of capabilities that are hard to assess using conventional approaches.

Assessment based on portfolio approaches should not equate to increasing the overall assessment burden as it will reduce the need for more traditional assessment methods.

The ultimate aim of medical education is to produce a graduate who is a capable performer in clinical practice. Capability, i.e. the ability to act in the uncertainty environment of real practice, is the highest goal one can aim at. To teachers it means one truly has transferred the complex knowledge and skills required for professional performance, for learners it means the confirmation of mastery over and above having learnt basic competencies, i.e. the abilities to perform specific tasks against a given standard.

Since knowledge and content are constantly changing (Sturmberg & Martin 2008), teaching will never cover every last entity so gaps will be inevitable at any point in time. Hence, a capable graduate besides being a skilled clinician will also be a learner who is able to identify and remedy any clinically relevant gaps and health service deficiencies throughout his/her life.

However, students often show little enthusiasm for these issues. Their agenda invariably focusses more on passing exams rather than immersing themselves in the learning process—a dichotomy that almost all institutions have yet to overcome.

Table

Practice points
The curriculum should define the expected capabilities of the program Capability should be the basis for the definition of assessment tasks that allow learners to demonstrate their professional achievements Professional achievements reflect the complex interplay between knowledge, skill and attitudes, and are better conveyed through portfolios

How much of this—longstanding—paradox is explained by learning and assessment being seen as two different entities rather than as the continuum of one and the same process? In this article, we offer a conceptualisation of a capability-focussed learning and assessment framework we hope will create a more stimulating and relevant learning environment.

The basis of teaching and assessment

Professional education is a continuous process from undergraduate through postgraduate to continuing professional development. This process involves multidimensional activities and processes and is broader than merely the accumulation of facts and rules. Learning equally is a social process that involves amongst others the recognition of patterns, seeing and experiencing situations as prototypical and observed outcomes as stimuli to reflect, developing situational awareness and sensing the limitations of assumed mental frameworks and reworking them (Weick 2002).

Learning therefore is as much perceptual as it is computational and thinking better means seeing better, not calculating better. This concept is known as perceptual learning and enables graduates to develop expertise rather than merely to accumulate facts and rules (Weick 2002). Our learning environments and activities must link these two dimensions of learning.

These ideas complement the modern educational emphasis on the importance of self-directed learning (Knowles 1990). We contend that such a learning environment should embrace ‘self-directed assessment’ as part of the learning process, a process that needs to be valued and supported by the teaching facility as an integral part of the learning process.

Assessment in an environment that values and encourages self-directedness and reflective/perceptive learning should not be punitive (the way most students perceive traditional summative assessment) since it will hinder learning, negates self-directedness and assumes that all learners progress at the same pace (Schön 1983).

Rather, assessment should be seen as a reflective process (formative assessment) aimed to reassure teachers and learners alike about achieving learning goals, regardless of its status in relation to pass/fail decisions. Then feedback drives learning rather than being prepared to ‘perform’ a predefined assessment task.

Is summative assessment more rigorous than formative?

Traditionally, statutory and legal requirements have enforced summative assessment, opposing the educational view that formative assessment, if well done, should be sufficient. However, given that valid assessment is always based on well-defined objectives, either approach can be rigorous (Driessen et al. 2007; Overeem et al. 2007) and contribute to the pass/fail decision (Driessen et al. 2007).

More importantly though, assessment should aim to test or closely approximate the learner's capabilities, i.e. the attributes that the learner should demonstrate in actual clinical practice. This type of assessment corresponds to tasks at the top of Miller's triangle (Miller 1990) whereas the foundation and the middle of the triangle deal with the different aspects of the learner's knowledge base, i.e. ‘basic knowledge’ is required for ‘knowing how’ to approach a task, ‘knowing how’ is required to ‘demonstrate’ the performance of that task, and ‘demonstrating’ the ability to perform that task is required to ‘perform’ it in the appropriate clinical situation at the bedside or in the consulting room. Thus, constructing higher-order assessment tasks will—by implication—also always include lower-order knowledge and basic skills (Wass et al. 2001).

The value of assessing lower-order knowledge and basic skills in themselves lies in the ability to ‘deconstruct’ them for the purpose of feedback towards progressive mastery of the complex tasks of clinical practice. However, most modern summative assessments focus unduly on lower-order assessment tasks. While these have been shown to predict higher-order ability (Norcini et al. 1984; Ram et al. 1999; Stern et al. 2003; Tamblyn et al. 2002), for example, between performance in MCQ tests and clinical performance, such assessments still fall short of confirming directly the clinical capability of the learner.

Towards the assessment of capabilities—the capability portfolio

A different focus is required to address the nexus between learning and assessment. The ideas reported here are based on a workshop for medical faculty conducted by the author (J.P. Sturmberg) involving 35 staff of the American University—Beirut, and have subsequently been further developed by the authors.

Changing assessment requires a return to the curriculum as it is here where the mindset of a course is articulated. We suggest that any curriculum development should start with a clear definition of the expected capabilities (like conceptualised in the outcomes-based curriculum approach (Bowden & Marton 1998)) at the end of a programme, rather than describing numerous singular simple discrete competencies. The distinction is important as capability refers to the ability to perform in the real world and requires expertise and capacity and as such is broader than the building blocks entailed in standardised competencies required for the performance of specific tasks.

Hence, capabilities should be the basis for the definition of assessment tasks that allow learners to demonstrate their professional achievements at the bedside or in the consulting room. These could then be collected in a capability portfolio for future reference and presentation at a final review.

The differences in conceptualisation between the competency-based (typically summative involving the demonstration of specific discrete tasks) and capability-based (emphasis on portfolio construction and review demonstrating actual integrated performance and reflection on real patient experiences) assessment approaches are summarised in Figure 1.

Figure 1. Capability vs. competency approaches to learning and assessment (adapted from Semantech (Semantech 2007)).

The results of mind-mapping exercises based on the above framework exemplify the relationship between expected student capabilities and related potential portfolio components. Figures 2 and 3 illustrate how capabilities and portfolio components can be placed in a pyramid based on Miller's triangle where knowledge is on the base and performance is at the apex.

Figure 2. Capability components.

Figure 2 demonstrates that most of the commonly used assessment strategies—e.g. MCQs, case studies and even OSCEs—rank below the capability level in the pyramid.

Figure 3. Portfolio components.

We have argued that the prevailing competency-based approach to assessment of graduates currently falls short of assessing the full range of desired capabilities of the graduate. Figure 3 shows a number of interesting assessment approaches that could fill this deficiency, but also highlights that many of the identified capabilities are currently not tested at all.

Since capabilities reflect complex interactions between knowledge, skills and attitudes, the assessment of capabilities will require approaches that reflect these complexities. Complex issues cannot be reduced to simple and linear descriptions (Cilliers 1998; Sturmberg 2007), and typically we use metaphors to understand, describe and communicate these (Lakoff & Johnsen 2003). Portfolios offer the broadest possible way of conveying the achievement of one's complex capabilities in descriptive ways. Since constructing and presenting a relevant portfolio is a reflective process, it will foster further learning based on clinical experiences (Challis 1999; Pitts 1999).

Portfolios may also capture the complexities of learners’ contextual knowledge, practice-based experiences and learning style preferences (Challis 1999; Pitts 1999). Having constructed a personal portfolio allows a graduate to demonstrate the achievement of the predefined graduate capabilities while the process also contributes to the meta-cognitive long-term assimilation of information, knowledge and wisdom (Figure 1) (Parboosingh 1996; Snadden & Thomas 1998; Pitts 1999).

A portfolio that captures performance in practice will therefore need to contain evidence of professional and ethical behaviours, critical thinking and dealings with uncertainty, recognition of limitations and leadership and team work skills (Figure 2). These items require a reflective evaluation of personal performance complemented by feedback from a designated trained mentor (Driessen et al. 2007), patients, supervisors and other members of the team/staff.

The value of the portfolio as a learning tool also lies in the demonstration of good as well as poor/er examples of performance, and in particular, critical incidents (Flanagan 1954; Mathers et al. 1999). Examples identifying deficiencies may contribute to future learning plans and initiate remedial actions, on the one hand strengthening the link between professional learning and demonstrating performance improvement, on the other supporting its validity as an assessment tool.

For a portfolio approach to be effective as a learning and assessment tool, an institution must develop a structured operational framework defining its use (Challis 1999; Wilkinson et al. 2002; Driessen et al. 2007). If based on defined capabilities and combined with suggested ways of demonstrating these, capability-based portfolio assessment could truly drive deep learning and foster the achievement of authentic professional development (Wilkinson et al. 2002).

The capability-portfolio approach requires a supportive learning environment and mentors who are able to work with the learner to discuss and draw out meaning from portfolio examples. Helping the learner to adopt a critical reflective approach should further develop their future learning (Challis 1999; Wilkinson et al. 2002; Gordon 2003). This could be done similar to Balint groups (Balint 1957) where learners share and reflect on their experiences. Such a group approach offers the advantage of moderating the often much more critical self-assessment of one's performance compared to external examiners (Schön 1983; Gordon 2003).

Seen on a continuum, the concept of capability-based portfolio assessment could become an important tool for life-long reflective learning and self-directed quality assurance. The value of the approach lies in its flexibility as well as its ability to tap into some of the hard to assess humanistic, team based and organisational domains of clinical practice (Farmer et al. 2002; Hays et al. 2002; Wilkinson et al. 2002; van Roessel & Shafer 2006).

Conclusions

Thinking about capability, i.e. the ability to perform in the real world, is needed for a contemporary curriculum and assessment design. A capability-focus will help students to integrate the foundations of medical practice with learning how to become a capable, reflective and life-long learner, requirements that patients expect from their doctors. A well-structured capability portfolio, regularly presented and reviewed, will be a useful tool to guide the journey to mastery.

However, changing assessment approaches should not equate to increasing the overall assessment burden. When introducing portfolio-based approaches we should also reduce the time and emphasis on more traditional assessment methods, to improve the reach and ability of our assessment methods to capture what matters in daily clinical practice without burdening the recipient. Concrete examples of portfolios used as part of an overall assessment programme and a rigorous evaluation of their qualities would form the next step to enhancing the capacity of this approach.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

J.P. Sturmberg

Both authors are involved in medical education at their universites, and had been National Segment Coordinators on the Board of Assessment of the RACGP for 14 years.

L Farmer

Both authors are involved in medical education at their universites, and had been National Segment Coordinators on the Board of Assessment of the RACGP for 14 years.