Examiner perceptions of a portfolio assessment process

Abstract

Background: The portfolio assessment process is important for assessing learner achievement.

Aims: To study examiner perceptions of Dundee Medical School's portfolio assessment process, in years 4 and 5 of the 5-year curriculum, in relation to: outcomes as a framework for the portfolio assessment process; portfolio content; portfolio assessment process; end points of the portfolio assessment process; appropriateness of the two part final exam format and examiner training.

Methods: A questionnaire containing statements and open questions was used to obtain examiner feedback. Responses to each statement were compared over 3 years: 1999, 2000 and 2003.

Results: Response rates were 100%, 88% and 61% in 1999, 2002 and 2003, respectively. Examiners were positive about the ability of institutionally set learning outcomes (Dundee 12 exit learning outcomes) to provide a framework for the portfolio assessment process. They found difficulties, however, with the volume of portfolio content and the time allocated to assess it. Agreeing a grade for each learning outcome for the candidate with their co-examiner did not present difficulties. The comprehensive, holistic picture of the candidate provided by the portfolio assessment process was perceived to be one of its strengths. Examiners were supportive of the final examination format, and were satisfied with their briefing about the process.

Conclusions: The 12 exit learning outcomes of Dundee curriculum provide an appropriate framework for the portfolio assessment process, but the content of the portfolio requires fine-tuning particularly with regard to quantity. Time allocated to examiners for the portfolio assessment process needs to be balanced against practicability. The holistic picture of the candidate provided by the process was one of its strengths.

Introduction

Portfolio assessment for medical students’ final examination was introduced at Dundee Medical School in 1997 to tackle the issues of assessing student attitudes, professionalism and performance in work-place settings. This article describes the examiner perceptions of the portfolio assessment process and will be of interest to health professions’ teachers involved in portfolio assessment and those considering the introduction of a portfolio assessment process. The student perceptions of the portfolio assessment process have been described elsewhere (Davis et al. 2009).

Table

Practice points

Examiners are strongly supportive of the portfolio assessment process. The volume of expected portfolio material from the students needs to be balanced against the workload of the examiners and students. Studies are needed to identify how to minimise the portfolio content, while retaining the holistic picture of the candidate that the portfolio assessment process can provide.

Background

The curriculum at Dundee Medical School is a 5-year, integrated, systems-based course (Harden et al. 1997). During this study (1999–2003), it had three phases: phase 1 dealing with normal structure, function and behaviour; phase 2 with abnormal structure, function and behaviour; and phase 3 with clinical learning and a task-based approach (Harden et al. 2000). The examiner perceptions of the phase 3 portfolio assessment process are the subject of this study.

The portfolio assessment process has five steps: (1) collection of evidence of achievement of learning outcomes by the student; (2) student reflection on the evidence; (3) examiner evaluation of the evidence; (4) defence of the portfolio by the student with two examiners and (5) an assessment decision taken by groups of examiners (Friedman Ben David et al. 2001). The 12 exit learning outcomes of the Dundee Medical School (Harden et al. 1999) provide the framework for the portfolio in years 4 and 5 (i.e. phase 3). Material is included in the portfolio at regular intervals throughout the 2 years. The material is graded by teaching staff on an A–G scale, required by the University of Dundee, and feedback on the material is returned to the students for formative purposes. The feedback follows each portfolio entry that the student submits; i.e. feedback is provided throughout the 2-year process. Students sit part 1 of their finals at the end of year 4, comprising an objective structured clinical examination (OSCE), extended matching item multiple choice exam, and a constructed response paper. Part 2 of the final exam is the portfolio assessment process, which takes place towards the end of year 5. Once a student submits the completed portfolio, two examiners have approximately 1 week to read it to arrive at an initial judgement of the student's strengths and weaknesses in terms of the 12 learning outcomes, based on the gradings awarded to the student over the 2–year period and their own reading of the portfolio material. The examiners meet to discuss how they will probe the individual student's strengths and weaknesses. This initial judgement is later confirmed or refuted at an interview with the student lasting 40 minutes. A grade is awarded for each outcome and included in the portfolio assessment summary sheet that includes all the grades awarded over the 2-year period. Through a consensus approach, the examiners arrive at the final summative decision as to whether the student has passed, failed or further evidence is required to reach a decision. At the time of this study, students with two or more ‘D’ final outcome grades or any single grade below ‘D’ were either requested to submit further evidence of achievement of the outcomes where they were weak or failed the exam.

The internal examiners were staff who taught at some point in the 5-year curriculum, but not necessarily in years 4 or 5. Basic scientists, laboratory-based staff and clinicians accepted invitations to examine the students’ portfolios. All examiners participated in a 2-hour examiner training programme comprising an initial briefing regarding the rationale of the portfolio assessment process, portfolio content material that provided information about each of the 12 curriculum exit outcomes and instructions about the rating scale used in the portfolio assessment process. Ten external examiners from outwith Dundee Medical School were invited to join the examiner group, in keeping with University of Dundee regulations requiring the contribution of external examiners. The external examiners either attended the internal examiner briefing session or were given an individual briefing. They took an active part as examiners in the portfolio assessment process and also observed internal examiners during discussion of portfolios with individual students.

The present study is based on an examiner evaluation questionnaire, completed by both external and internal examiners, designed to study examiners’ perceptions of the outcome framework of the portfolio process; the portfolio content; the portfolio assessment process; the end points of the portfolio assessment process; the appropriateness of the two part final exam format and the examiner training.

Methods

The examiner questionnaire was developed following discussions with the staff members who designed the portfolio assessment process, i.e. the phase 3 assessment working group, and with the internal and external examiners. The questionnaire contained both items to be rated by the examiners and open questions. A five-point Likert rating scale was used, where strongly agree = 5 and strongly disagree = 1, except for *items in Table 3, where a 1–3 scale denoting ‘too little’, ‘about right’, and ‘too much’, respectively, was used. The questionnaire underwent some change each year to reflect the changes made to the portfolio assessment process, based on the student/staff feedback of the previous years (Davis et al. 2009). All the items are provided in the tables with the year in which they were used. The questionnaire was distributed immediately after the examiners’ meeting, held at the end of the portfolio assessment process, in years 1999, 2002 and 2003.

The yearly median ratings for each statement were compared using the Kruskal Wallis test, at a statistical significance of p ≤ 0.05. To estimate how large the change was from year to year (i.e. to estimate the contribution by the ‘year’ to the overall variability of average scores for a given questionnaire item) effect sizes were calculated using partial η². This effect size test was used in the absence of a suitable effect size measure for non-parametric comparisons involving more than two variables. All statistical analyses were carried out using an SPSS version 15.0.

The number of subjects precluded factor analysis of the examiner perceptions. The examiner responses to the open questions of each year were grouped into categories by the first author and then compared between the years.

Results

The questionnaire response rates were 100% (33) in 1999, 88% in 2002 (58 out of 66) and 61% (40 out of 66) in 2003.

All the questionnaire ratings are shown in Tables 1–4. Table 1 shows the ratings for statements about the outcome framework for the portfolio assessment process. Table 2 shows the ratings for statements about the portfolio content. Though the examiner opinion significantly changed from that of the first year, they were still unsure as to the manageability of the volume of information (with a relatively substantial effect size). Table 3 shows the ratings for statements about the portfolio assessment process.

Table 1.  The comparison of the median (mean ± SD) ratings for the questionnaire statements on the outcome framework of the portfolio process, over the 3 years

Statements: outcome framework for the portfolio process	1999	2000	2003	p-value
The portfolio identifies students’ strengths and weaknesses	4 (3.9 ± 0.9)	4 (3.9 ± 0.7)	4 (4 ± 0.7)	0.84	0.00
The 12 outcomes provide a useful framework for reviewing the students’ strengths and weaknesses	4 (4.2 ± 0.6)	4 (4.1 ± 0.7)	4 (4 ± 0.7)	0.37	0.02
12 outcome sheets provided me with helpful information	4 (4.3 ± 0.7)	4 (4.1 ± 1)	NA	0.67	0.01

, partial η²; NA, not available.

Table 2.  The comparison of the median (mean ± SD) ratings for the questionnaire statements on portfolio content, over the 3 years

Statements: portfolio content	1999	2000	2003	p-value
PRHO learning plan assessment form provided me with helpful information	4 (3.9 ± 0.7)	4 (3.7 ± 0.9)	4 (3.5 ± 1.1)	0.44	0.02
Clinical SSC assessment form provided me with helpful information	4 (4.0 ± 0.7)	4 (3.9 ± 0.7)	4 (3.9 ± 0.7)	0.70	0.00
Record of achievement provided me with helpful information	4 (3.8 ± 0.8)	4 (4 ± 0.8)	4 (3.8 ± 0.9)	0.28	0.02
Case discussions and grades provided me with helpful information	4 (4.2 ± 0.5)	4 (4.2 ± 0.6)	4 (4.3 ± 0.6)	0.67	0.01
Fourth year assignment and report provided me with helpful information	4 (4.0 ± 0.5)	4 (4.1 ± 0.5)	4 (4.1 ± 0.6)	0.83	0.00
Theme-based SSC assessment form provided me with helpful information	4 (4.2 ± 0.5)	4 (4.1 ± 0.6)	4 (4.2 ± 0.6)	0.80	0.00
General practice SSC report provided me with helpful information	4 (4.3 ± 0.7)	4 (4.4 ± 0.7)	4 (4.2 ± 0.6)	0.23	0.01
Elective report provided me with helpful information	4 (3.5 ± 0.9)	4 (4.1 ± 0.8)	4 (3.8 ± 1)	0.01	0.07
Patient presentations provided me with helpful information	4 (4.0 ± 0.5)	4 (4.0 ± 0.6)	NA	0.65	0.00
Curriculum vitae provided me with helpful information	4 (3.5 ± 0.9)	NA	NA
Additional material submitted provided me with helpful information	3 (3.3 ± 1.1)	3.5 (3.5 ± 0.6)	NA	0.69	0.01
Results of year 5 progress test provided me with helpful information	4 (3.9 ± 1.0)	NA	NA
The volume of information was manageable	2 (2.3 ± 0.5)	3 (2.8 ± 0.4)	3 (2.8 ± 0.4)	0.00	0.19

p ≤ 0.05; , partial η²; NA, not available; PRHO, Pre-registration House Officer; SSC, student selected components.

Table 3.  The comparison of the median (mean ± SD) ratings for the questionnaire statements on portfolio assessment process, over the 3 years

Statements: portfolio assessment process	1999	2000	2003	p-value
Reviewing the portfolio in detail before the oral was an important part of the process	5 (4.7 ± 0.5)	5 (4.7 ± 0.6)	5 (4.4 ± 1.3)	0.95	0.02
The discussion with the other examiner before the oral was an important part of the process	5 (4.6 ± 0.6)	5 (4.4 ± 0.7)	5 (4.6 ± 0.9)	0.19	0.01
The discussion with the other examiner after the oral at the examiners’ meeting was an important part of the process	5 (4.5 ± 0.7)	4 (4.3 ± 0.7)	4 (4.3 ± 1)	0.33	0.01
Two examiners rather than one examiner reviewing the portfolio before the examination was unnecessary	1 (1.3 ± 1.3)	1 (1.6 ± 1.1)	2 (1.9 ± 1.2)	0.01	0.04
The briefing meeting for examiners was helpful in clarifying my role	4 (3.9 ± 1.2)	4 (4.1 ± 0.6)	NA	0.53	0.03
Adequate time was allocated for review of portfolio	2 (1.9 ± 0.4)	3 (2.7 ± 0.7)	3 (2.3 ± 0.9)	0.00	0.21
Adequate time was allocated for discussion with other examiner	2 (2.2 ± 0.4)	3 (2.7 ± 0.5)	3 (2.4 ± 1)	0.00	0.12
Adequate time was allocated for oral examination	2 (1.9 ± 0.4)	3 (2.9 ± 0.4)	3 (2.7 ± 0.5)	0.00	0.54
The portfolio assessment summary sheet was user friendly	4 (3.8 ± 0.5)	4 (3.8 ± 0.8)	4 (3.8 ± 0.8)	0.97	0.00
The oral examination was unnecessary as part of the portfolio assessment	1 (1.1 ± 0.9)	1 (1.4 ± 0.9)	2 (1.9 ± 1.1)	0.00	0.11
The pre oral discussion between the two examiners was valuable in identifying areas to be explored during the oral	5 (4.4 ± 0.7)	4 (4.3 ± 0.6)	5 (4.4 ± 0.9)	0.26	0.01
I would have found it useful to have received information about the course before the examination.	2 (2.3 ± 1.3)	3 (2.8 ± 1.2)	3 (2.8 ± 1.5)	0.21	0.03

Statements were rated on a four-point rating scale from 0 to 3.

p ≤ 0.05;, partial η²; NA, not available.

Table 4.  The comparison of the median (mean ± SD) ratings for the questionnaire statements on outcomes of the portfolio assessment process, over the 3 years

Statements: end points of the portfolio assessment process	1999	2000	2003	p-value
The combination of an OSCE and written exams in 4th year and a portfolio exam in 5th year is an effective approach to the assessment of student competence to graduate	4 (3.9 ± 1.0)	4 (4 ± 0.9)	4 (4.2 ± 0.8)	0.32	0.02
The portfolio assessment process is valuable in assessing the individual student's fitness to practice	4 (4.0 ± 0.8)	4 (3.8 ± 0.9)	NA	0.24	0.02
I was comfortable with the final allocation of students into the categories exempt from final assessment, possible distinction, potential exemption, not exempt	4 (4.1 ± 0.6)	4 (4 ± 1)	4 (3.9 ± 0.7)	0.33	0.02
I was pleased by the standard achieved by the students in this examination	4 (4.0 ± 0.5)	4 (4.1 ± 0.7)	4 (4.3 ± 0.6)	0.19	0.02
The standard achieved by these students is higher than that achieved by comparable students elsewhere	3 (2.9 ± 1.0)	3 (3.1 ± 0.7)	3 (2.1 ± 1.5)	0.01	0.13

p ≤ 0.05; , partial η²; NA, not available.

The examiners feel rushed in their contribution to the portfolio assessment process. Although there was a significant change in this opinion with substantial effect sizes in the last 2 years, as opposed to the first year, the examiners were still uncertain as to the adequacy of the time allocated to them for their part of the portfolio assessment process. Similarly, though they were generally satisfied with the training that they received, the examiners were unsure whether it would have been useful if more information about the course was given to them in advance. Table 4 shows the ratings for statements about the end points of the portfolio assessment process.

The examiners consistently agreed with the statement ‘the combination of an OSCE and written examination in the fourth year and a portfolio exam in the fifth year is an effective approach to the assessment of the competence to graduate’.

There was not much variability in assessors’ perceptions (i.e. median questionnaire item scores) over the 3 years. Overall, the examiners have favourable opinions about portfolio assessment, and that opinion has not changed over the years.

The open-ended response categories that had more than 10 responses in at least 1 year are shown in Table 5.

Table 5.  Comparison of open-ended responses

	Number of responses (%)
Comment category	1999	2002	2003
(1) What material might be omitted from the portfolio for next year's examination?
None	5 (15.2)	12 (20.7)	3 (7.5)
(2) Do you think building the portfolio serves as a learning experience for students?
Yes	27 (81.8)	53 (91.4)	33 (82.5)
(3) Examiners were asked to allocate a grade for each outcome after the oral examination. What, if any, difficulties did you have allocating such a grade?
None/reasonable agreement	6 (18.2)	26 (44.8)	19 (47.5)
(4) Please comment on strengths of the portfolio assessment process
Comprehensive coverage of student performance	7 (21.2)	22 (37.9)	13 (32.5)
(5) Please comment on weaknesses of the portfolio assessment process
Inadequate assessment of core clinical skills	7 (21.2)	20 (34.5)	7 (17.5)

Discussion

The outcome framework

The examiners are confident in the ability of the institutionally set exit learning outcomes to provide a useful framework for the portfolio assessment process. This provides reassurance about the adoption of such an outcome framework for the portfolio assessment process. The students also thought that the outcomes provided a framework for portfolio building and assessment (Davis et al. 2009).

Portfolio content

The volume of information in the portfolio was a problem for examiners increasingly over the years. Thus, both the examiners and students (Davis et al. 2009) thought that there was too much material in the portfolio, despite a reduction in portfolio content over the years of this study (Davis et al. 2009). Too much portfolio material is a well-documented problem that seems hard to overcome (Athenases 1997). To address this issue, material that provides no new information about the student and insufficiently reliable material should be omitted from the portfolio.

Portfolio assessment process

The examiners were satisfied with most aspects of the portfolio assessment process, reflecting their confidence in the five-step process (Davis et al. 2001; Friedman Ben David et al. 2001). There were uncertainties over the amount of time allocated for their part of the portfolio assessment process. Portfolio assessment is well recognised to be a time consuming undertaking for both examiners and students (Race 1996). These time demands should be balanced against the benefits of a portfolio assessment process. There was a realisation within the medical school that only the best and worst students needed to discuss their portfolios with the examiners reducing both the amount of staff time expended on the process and student anxiety (Davis et al. 2009). Attempts to streamline the portfolio assessment process, however, had to be abandoned in the light of examiner and student opinion. Having expended large amounts of time in compiling their portfolios the students and examiners thought that all the students’ portfolios should be read and discussed.

The majority of the examiners found little or no difficulty in agreeing upon the student grade with the co-examiner (Table 5, item 3). This finding contrasts with the evidence on inter-rater reliability by Pitts et al. (1999, 2001, 2002) and Roberts et al. (2002), but concurs with the more recent inter-rater reliability evidence related to rating scales (Olson et al. 2003).

The end points of portfolio assessment process

The strength of the portfolio assessment as identified by the examiners revolves around its ability to assess candidates holistically with comprehensive coverage of performance (Table 5, item 4). It is recognised that portfolios facilitate the assessment of integrated and complex abilities, taking into account the level and context of learning (Friedman Ben David et al. 2001), combining information from both subjective and objective assessment procedures (Schuwirth et al. 2002), ‘to see the whole picture’ (Schuwirth 2002).

A few examiners were concerned by the inability of the portfolio to assess clinical skills (Table 5, item 5). This is not a deficit of the portfolio assessment process, but of the portfolio content. The inclusion in the portfolio of a workplace-based assessment such as the mini-clinical evaluation exercise (mini-CEX) in appropriate numbers should be able to provide reliable evidence of the student performance of clinical skills in the workplace. Evidence of student competence in clinical skills can be provided by the inclusion of the year 4 OSCE results in the portfolio.

Examination format

The examiner support for the combination of the OSCE and the written examination in the 4th year, followed by the portfolio assessment in the 5th year confirms the appropriateness of the present examination format to assess all four levels of the Miller's pyramid (Miller 1990). The written examinations assess at the ‘knows’ and ‘knows how’ levels and the OSCE assesses at the ‘shows how’ level in the 4th year. The portfolio contains evidence that is assessed at ‘does’ level in the 5th year. If the 4th year written exams and OSCE results are included in the portfolio, the portfolio assessment process itself is capable of assessing the students at all four levels of Miller's pyramid.

Examiner training

Though the examiners were satisfied with the briefing that they received, they would have preferred more information about the course in advance. This may further add to the logistics and the cost of the examination. The cost of assessment, in terms of both expenditure and examiner time (Snadden et al. 1996; Roberts et al. 2002), is an accepted disadvantage of portfolio assessment. This, however, may well be the price that one has to pay for an authentic assessment of performance.

The evaluation process

Examiner participation in the evaluation was high in the first 2 years although in 2003 participation was lower. Examiners made use of the open questions in the evaluation and have freely expressed their opinion, both for and against the portfolio assessment process. The data in this article present a clear picture of examiner opinion. The results of the above study, however, need to be considered with other extant evaluation evidence, such as student perceptions, external examiner reports and student portfolio assessment results to give a full picture of the evaluation of the portfolio assessment process.

Conclusions

Despite the drawbacks of increased examiner and student workload, the examiners acknowledge the suitability of portfolio assessment (together with an OSCE to assess the student competence) for final year medical undergraduate assessment. Future studies need to be directed towards finding the right balance of portfolio content that will provide a manageable workload for the students and examiners, without compromising the comprehensive assessment of student performance that the portfolio assessment process offers.

Declaration of interest: The authors report no conflict of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Margery H. Davis

MARGERY DAVIS is a professor in Centre for Medical Education, University of Dundee, UK.

Gominda G. Ponnamperuma

GOMINDA PONNAMPERUMA is in Medical Education Development and Research Centre (MEDARC), Faculty of Medicine, University of Colombo, Sri Lanka.