Does a summative portfolio foster the development of capabilities such as reflective practice and understanding ethics? An evaluation from two medical schools

Abstract

Portfolios need to be evaluated to determine whether they encourage students to develop in capabilities such as reflective practice and ethical judgment. The aims of this study were (i) to determine whether preparing a portfolio helps promote students’ development in a range of capabilities including understanding ethical and legal principles, reflective practice and effective communication, and (ii) to determine to what extent the format of the portfolio affected the outcome by comparing the experiences of students at two different medical schools.

A questionnaire was designed to evaluate undergraduate medical students’ experiences of completing a portfolio at two medical schools.

A total of 526 (45% response rate) students answered the on-line questionnaire. Students from both medical schools gave the highest ranking for the portfolio as a trigger for reflective practice. 63% of students agreed their portfolio helped them develop reflective practice skills (p < 0.001), whereas only 22% disagreed. 48% of students agreed portfolios helped them understand ethical and legal principles whereas 29% disagreed (p < 0.001). In contrast, only 34% of students thought the portfolio helped them to develop effective communication.

Students perceive portfolio preparation as an effective learning tool for the development of capabilities such as understanding ethical and legal principles and reflective practice, whereas other capabilities such as effective communication require complementary techniques and other modes of assessment.

Introduction

The complexity and demands of medical practice continue to evolve and medical schools need to ensure educational objectives match the medical needs of society (McCurdy et al. 1997; Carraccio et al. 2002; Leinster 2003). While the emphasis differs, the list of graduate capabilities articulated by medical schools remains remarkably similar throughout the literature (Whitcomb 2002; General Medical Council 2003; ACGME 2007). Regulatory bodies are stipulating outcome and competency based assessments of personal and professional capabilities, as well as the more traditional knowledge and application of basic and medical science.

As developing professionals, medical graduates are required to understand the psychosocial and cultural aspects of health, communicate effectively both orally and in writing, and understand the ethical and legal principles which underpin modern medicine (Howe 2002; Epstein 2007). Doctors also must develop the capacity to recognize their own limits through reflection, and to sustain lifelong learning in order to support professional practice (Mathers et al. 1999; Driessen et al. 2008).

Portfolio assessment has been adopted by a number of medical schools as a method that can ensure educational goals and competencies are met in areas which are not covered by traditional medical assessments (Davis et al. 2001; Driessen et al. 2007; Buckley et al. 2009). Portfolios should require the student to reflect on their achievements or the evidence presented, their progress, and on plans for the future (Wilkinson et al. 2002; Rees et al. 2005). Student evaluations of portfolios have generally been confined to their ability to demonstrate capabilities such as reflection or communication skills (Rees & Sheard 2004b; Davis et al. 2009), but little is known about whether portfolio assessments help students to think or perform in more narrowly defined areas of competency such as recognizing one's own strengths and limitations, setting learning goals and plans, respecting patient rights and the integration of psychosocial and cultural aspects into medical practice (Roberts et al. 2002; Johnston 2004).

Therefore, the primary aim of this study was to determine whether the process of preparing a portfolio report or essay helped students appreciate and develop in a range of capabilities, such as understanding ethical and legal principles and reflective practice; capabilities which are not the focus of traditional assessments such as clinical or written examinations. The format of the portfolio assessment was different in each of the two medical schools in the study, and so a secondary aim was to determine to what extent the format of the assessment affected the outcome by comparing the student experiences with their portfolio assessment in the two schools.

Table

Practice points

Portfolios should be linked to graduate capabilities such as reflective practice and understanding ethics to help drive learning towards specified outcomes. Students should be required to build their portfolio prospectively thus promoting regular review of progress and encouraging self-direction and reflection. The explicit request to produce a reflective document is associated particularly with the development of skills in reflective practice and self-direction. Students perceive portfolio preparation as an effective learning tool for the development of capabilities such as reflective practice and ethical judgment, whereas other capabilities such as effective communication require complementary techniques and other modes of assessment.

Methods

A questionnaire was developed after reviewing the literature and previous portfolio evaluations (Appendix 1). The questionnaire was designed to evaluate medical students’ experiences of completing a portfolio report or essay, specifically related to six key capabilities developed by recognized regulatory organizations (ACGME 2007; General Medical Council 2003; Frank 2005). The questions were also linked to the philosophy, capabilities and learning outcomes expected at graduation from the two medical schools involved in this research. Both schools used written material for their portfolios but these were framed differently as a report or essay. Eighteen questions were developed, three linked to each of six key capabilities (Table 1). The questions focused on whether the portfolio helped the students understand or develop in specific areas, rather than seeking information on their personal attributes. The questionnaire was cross-sectional, and completed by students from two medical schools who had completed portfolio assessments during their program, thus relying on student self-reporting and perceptions. The questionnaire was delivered online and all students in the medical programs at both universities were invited to complete it via email which was sent out approximately 2 months after the completion of their portfolio. Research Ethics Committee approval was granted at both medical schools and all subjects gave informed consent (University 1: 2009-7-42, University 2: 2008035).

Table 1.  Capability/competency framework for portfolio examination.

University 1 Graduate capabilities	University 2 Philosophy and learning outcomes	ACGME 2007 Competencies	CanMEDS 2005 Competency framework	GMC – Tomorrow's doctors 2003	Numbers refer to the questionnaire
Social and cultural aspects of health & disease	Respect different cultures, recognises social and economic forces in health	Knowledge of social-behavioral sciences	Identify & respond to the determinants of health in the community	Integrate the behavioral and social sciences into medicine	20, 21 & 22
Effective communication	Managing and presenting written information	Comprehensive written communication	Accurately convey information	Communicate effectively	5, 6 & 7
Teamwork	Work effectively as a member of a team	Evaluate feedback Work effectively in a team	Participate effectively in an inter-professional team	Respond constructively to performance review Demonstrate effective team-working	17, 18 & 19
Self-directed learning & critical evaluation	Skills to continue learning Critical appraisal	Set learning & improvement goals	Maintain professional activities Critically evaluate information	Integrate new knowledge Plan professional development	11, 12 & 13
Ethics and legal responsibilities	Respect patient autonomy	Compassion & integrity Accountability Patient respect	Develop ethical relationships with patients	Duty to protect patients Understand the rights of patients	8, 9 & 10
Reflective practice	Reflective Practice Recognises problems	Identify strengths & limitations	Recognize the limits of their expertise	Recognize personal & professional limits	14, 15 & 16

The numbers in the final column refer to the question numbers in the questionnaire in Appendix 1.

One school's medical program is a 6-year undergraduate program (University 1). At the end of years 2, 4 and 6 students present a portfolio essay in which they discuss their progress in each of eight graduate capabilities based on the evidence of achievement indicated by assessments in the program (McNeil et al. 2006). This portfolio is integrated into the curriculum and it is described in more detail in the manuscript by O'sullivan et al. (2011). Most assessments in the program focus explicitly on a number of capabilities, and students reflect on their grades and feedback, and discuss actions taken in response, or plans for improvement. The reflective essay varies from 1500 to 3500 words depending on the year. The assessments that form the evidentiary base of the portfolio comprise individual essay assignments and group project reports. Informal “evidence of achievement” such as a research activity or a letter of commendation can also be submitted. Students have access to mentoring from portfolio advisors who are senior academic staff. The portfolio essay is graded overall and per capability on a 4 point scale (F: fail, P-: borderline, P: pass, P+: distinction).

The other medical school's program is a 5-year undergraduate course (University 2) with clinical experience fully integrated into all five years of the course. Students prepare a 2,000 word portfolio report at the end of each of their five academic years. For more details on the format and assessment criteria, readers are referred to the manuscript by Howe et al. (2009). Formal teaching on professionalism is integrated into the curriculum through a variety of teaching activities and the portfolio draws these together as the assessment. The portfolio report assessment is identical in each year group, with the exception that the reports are framed around a different specific theme each year, with a focus on aspects of professional development: learning from experience (Year 1), working in groups (Year 2), the doctor-patient relationship (Year 3), sex, race and power (Year 4), becoming a doctor (Year 5). Students are asked to reflect on experiences from clinical settings, their campus-based teaching and any appropriate personal events from their reflective diaries that they are asked to keep throughout the five years of the undergraduate medical course to aid when writing the report. The examiner does not see these diaries. Students provide relevant examples of experiences that relate to the theme of that year's report, reflect on these experiences, consider what they have learned from these experiences. They need to show how their learning of appropriate attitudes and values, related to GMC guidance, relates to what is expected of doctors, and show that they are setting appropriate goals to develop their professional practice. Students have access to mentoring from academic and teaching staff. The yearly portfolio reports are graded overall on a three-point scale (fail, pass, distinction).

The portfolios at both medical schools are summative barrier assessments; that is, students can fail to progress if they do not pass these components. Although, the two portfolios are structurally different, both portfolios have the same educational goal of requiring students to reflect on their progress, and completion of the portfolios require a degree of self-directed action. The questionnaire was designed to determine if the students perceived the portfolio as an appropriate means of helping develop reflective practice, understanding ethics and self-directed learning, rather than interrogating the specific structure of the portfolio. The responses to the questionnaire from students at both medical schools were combined and compared to examine the development of relevant professional capabilities.

Data analysis

The data were analyzed using PASW Statistics 18.0 (SPSS 2010, Chicago, Illinois). Comparison of combined university responses was done using Chi-square test. Comparisons of students’ perceptions of their experiences with the portfolio at the two schools (University 1 versus University 2) and gender were performed using a Kruskal-Wallis Test. Spearman's correlation coefficients between variables were also calculated. For data presentation strongly disagree and disagree were combined as were agree and strongly agree. To maintain anonymity of the medical schools, they will be referred to as University 1 and University 2.

Results

Respondents

A total of 526 out of an eligible 1158 students (45% response rate) answered the on-line questionnaire, 411 (49%) from University 1 and 115 (36%) from University 2 (Years 1 to 4). A similar number of students from each year responded. Of the respondents 60% were female and 40% male compared with the eligible cohort of 56% female and 44% male.

Ratings of capabilities

When the responses from the two universities were combined, 63% of students agreed their portfolio helped them develop in reflective practice (p < 0.001) whereas only 22% disagreed (Table 2). 48% of students agreed portfolios helped them understand ethical and legal principles (p < 0.001) whereas 29% disagreed (Table 2). Aiding self-directed learning was rated third highest with 48% agreeing and 33% disagreeing. In contrast, 34% of students thought the portfolio helped them to develop effective communication whereas 42% disagreed. 37% of students thought the portfolio helped them to appreciate teamwork whereas 38% disagreed. All responses to the capabilities positively correlated with each other (p < 0.001, data not shown) indicating that students generally rated most capabilities consistently positive or negative.

Table 2. University 1 and University 2 combined questionnaire responses (n = 526) to the three questions linked to each capability (1578 responses) expressed as percentage from both medical schools.

Capability	Disagree	Neutral	Agree	Chi-square (df = 6)
Reflective practice	22%	16%	63%	622.0*
Ethical and legal principles	29%	23%	48%	162.7*
Self-directed learning	33%	19%	48%	197.9*
Social and cultural aspects	33%	20%	47%	167.3*
Teamwork	38%	25%	37%	50.8*
Effective communication	42%	24%	34%	73.3*

*Significant at the p < 0.001 level.

Differences between groups

When the responses from the students from each university were looked at separately, students at both universities reported that the portfolio process helped them understand or develop in capabilities such as reflective practice, understanding ethical and legal principles, self-directed learning and social and cultural aspects as shown in Table 3. In contrast, students at University 2 reported that preparing a portfolio helped them also in the other two aspects, effective communication and teamwork, whereas students at University 1 did not. Generally, students at University 2 rated preparing a portfolio higher than the students at University 1 across all the capabilities. When data were analysed comparing responses from different years in the medical programs, no significant differences were found (data not shown). As shown in Table 4, a greater percentage of female students rated preparing a portfolio higher than the male students across all the capabilities, with reflective practice and teamwork showing the greatest difference.

Table 3. Questionnaire responses expressed as percentage comparing University 1 (n = 411) to University 2 (n = 115).

Capability	University	Disagree	Neutral	Agree	P value*
Reflective practice	Uni 1	24%	16%	60%	<0.001
	Uni 2	14%	14%	72%
Ethical and legal principles	Uni 1	33%	22%	45%	<0.001
	Uni 2	16%	26%	58%
Self-directed learning	Uni 1	37%	19%	45%	<0.001
	Uni 2	28%	20%	52%
Social and cultural aspects	Uni 1	34%	19%	48%	0.77
	Uni 2	30%	26%	43%
Teamwork	Uni 1	44%	24%	32%	<0.001
	Uni 2	18%	27%	56%
Effective communication	Uni 1	47%	25%	29%	<0.001
	Uni 2	23%	23%	54%

*Significant difference University 1 versus University 2, Kruskal-Wallis test.

Table 4. University 1 and University 2 combined questionnaire responses expressed as percentage of each gender comparing male (n = 211) to females (n = 315).

Capability	Gender	Disagree	Neutral	Agree	P value*
Reflective practice	male	29%	17%	54%	<0.001
	female	17%	15%	68%
Ethical and legal principles	male	34%	22%	44%	<0.001
	female	26%	23%	51%
Self-directed learning	male	41%	16%	43%	<0.001
	female	28%	21%	51%
Social and cultural aspects	male	40%	18%	42%	<0.001
	female	28%	22%	50%
Teamwork	male	44%	27%	29%	<0.001
	female	34%	23%	43%
Effective communication	male	49%	20%	31%	<0.001
	female	37%	27%	36%

*Significant difference male versus female, Kruskal-Wallis test.

Discussion

Reflective practice, understanding ethics and self-directed learning

This research suggests that the introduction of a summative reflective portfolio into the assessment program of a medical school can impact positively on learning in capabilities not examined by traditional medical assessments. Although the structure of the portfolios were different, their educational aims were similar, and students in both contexts gave high ratings to the impact of this educational approach on their development of reflective practice, understanding ethical and legal principles, and self-directed learning. This suggests that the tasks of collecting evidence and completing a reflective activity can drive learning in these areas, regardless of the structure of the portfolio that the reflective task is embedded in (Cole 2005; Driessen et al. 2008). However, more information on the quality of reflection and what aspects of the portfolio drive on students to learn in these capabilities is required (Zeichner & Wray 2001).

Communication

In contrast, students at both universities rated the impact of the portfolio assessment on the development of communication skills lower. When comparing the responses of the students from the two medical schools, it needs to be considered that both schools had different curricula and therefore, their responses may reflect the effect of their respective curriculum on learning as well as differences in the portfolio format. However, neither portfolio included an oral component and so this result suggests students regard, correctly, that communication has important components in addition to writing, such as oral communication and non-verbal visual skills (Rees & Sheard 2004b). Students from University 1 gave lower ratings to the portfolio for helping with communication skills than University 2, but this may relate to other differences in the curricula and assessment programs. University 1 requires students to complete many written assignments in the first two years in addition to the portfolio, whereas University 2 does not have this degree of emphasis on written assignments. Thus when asked specifically in two questions, if preparing the portfolio report has helped with their written communication skills and appreciation of the value of such skills, a higher rating for University 2 is perhaps not surprising given that the portfolio report is one of the main forms of written assessment at University 2. The amount of written work required for a portfolio does influence its acceptability by students (Rees et al. 2005; Davis et al. 2009;), and the ‘tie-in’ between portfolio and communication may be stronger in some medical programs than others.

Teamwork

The influence of completing a portfolio on the development of teamwork skill was rated lower by students at both universities. This finding is possibly explained by the fact that producing and writing portfolio submissions is an individual activity, even though some of the evidence in the portfolio that students are writing about was produced in groups or teams. However, students at University 2 rated the impact of the portfolio on their development in the capability of teamwork relatively higher than students at University 1 possibly related to differences in teaching structure. The process of developing the evidence for the portfolio at University 1 requires students to do a considerable amount of group work, and the results of this group work along with evidence of the students’ contributions to the group process, have a significant influence on the grading of this capability (Hughes et al. 2008). At University 2, students prepare their portfolios by responding to a specific focused question, which may highlight teamwork as an issue. Furthermore, many of the learning experiences they will be reflecting on will have occurred in a group situation due to the fact that the majority of both university-based and clinical learning occurs in small groups. The results above, for the capabilities of teamwork and communication, may suggest that students at University 1 distinguished between the processes of generating evidence of development on the one hand, from the preparation of the reflective essay itself on the other, when they responded to the survey.

Portfolio acceptance

Although the portfolios evaluated in this study were generally positively rated by students, findings in other studies have differed (Rees & Sheard 2004b; Davis et al. 2009; Howe et al. 2009). Studies using portfolios to assess written reflections have shown that students demonstrate a high level of professional insight and are able to link experiences to their development as a professional (Howe et al. 2009). Other investigators have found support for portfolio validity, although the students had quite polarized views on the process of reflection (Rees & Sheard 2004a). Some students seem to appreciate the need to improve their reflective skills (Rees & Sheard 2004a), and others have reported positive attitudes towards reflective learning (Davis et al. 2009), whereas negative attitudes may arise where trainees lack understanding of the method (Pee et al. 2000), or feel reluctant to participate in this style of learning activity (Grant et al. 2006). Therefore, there appears to be a greater need to facilitate learners to accept reflective learning (Davis et al. 2009). Interestingly in the present study, female students at both universities rated preparing a portfolio more highly than male students and further qualitative research in this area may be helpful.

Portfolio structure

It is clear that portfolios require other features to help improve their acceptance and effectiveness. Evidence suggests significant supervisor and mentor input is needed and mentor training is required (Driessen et al. 2007; Kalet et al. 2007; O'Sullivan et al. 2011). The amount of work required for the portfolio must also be manageable for the student (Schuwirth & Cantillon 2005), and some investigators have recommended keeping minimal paperwork (Rees et al. 2005; Davis et al. 2009). In fact, due to resource issues, portfolios have been regarded as to resource intensive for developing countries (Burch & Seggie 2008). These demands have led to the development of e-portfolios which may be effective (O'Sullivan et al. 2011), and they are reviewed positively by supervisors (Bashook et al. 2008), however electronic delivery is not always favoured (Dornan et al. 2002). Therefore, while further development of portfolios need to take these factors into account, the essential reflective component must be preserved.

A number of limitations of this research need to be raised. While the response rate for the survey was acceptable, the non-responding group may have included a large number of students who did not find the process of preparing a portfolio reflective essay beneficial for learning in the focus competencies. Secondly, the survey data identifies some positive student perceptions, but does not allow us to draw conclusions as to the actual impact of the portfolio on the use of the capabilities and skills in practical settings. However, evidence suggests students’ perceptions concerning the effectiveness of their education and teaching do reflect educational improvements and learning outcomes (March 1987; Zonia & Stommel 2000; Lizzio et al. 2002). However, it will be important to further evaluate portfolio assessments with outcomes especially if they are to be used as summative and high-stakes exams (Roberts et al. 2002; Driessen 2009).

Conclusion

Portfolio examination can successfully link assessment to the development of capabilities such as reflective practice, understanding ethical and legal aspects, and self-directed learning. Other capabilities such as effective communication and teamwork may require complementary examinations such as oral clinical examinations and vivas. This study demonstrates that despite structural differences in a summative portfolio, the explicit request to produce a reflective document is associated particularly with the development of skills in reflective practice and self-direction as well as an enhanced understanding of ethical and legal principles. Further qualitative work may help explore the mechanisms behind these expressions of understanding, the quality of reflection and the role of the portfolio in driving learning compared with being an assessment tool (Zeichner & Wray 2001). More research is required to determine whether fostering of these capabilities by the portfolio actually translates into improved performance in these capabilities. Further research is required into why females respond more positively than males to portfolio assessment as well as methods to improve student acceptance.

Acknowledgments

The authors wish to acknowledge the contributions of Professors Patrick McNeil and Rakesh Kumar, Associate Professor Gary Velan and Drs Chinthaka Balasooriya and Susan Toohey towards the development of the UNSW Medical Program assessments which enabled the development of the UNSW portfolio.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Previous presentations

Data from this manuscript was presented in abstract form at the following meeting: Summative portfolio assessment: A method to assess reflective practice in undergraduate medical students. Annual Conference for the Association for Medical Education in Europe, Glasgow, Scotland 2010, Oral No. 2H2.

Appendix 1. Portfolio evaluation questionnaire

Instructions

• This questionnaire has been designed to evaluate your experiences with the portfolio essay you completed during the last academic year.

• For each question please select the response that best describes your opinion. There are spaces for comments at the end of the questionnaire.

• This questionnaire is for anonymous completion, so please do not provide your name or student ID number.

• Please note, the term “evidence” in the questionnaire describes your assignments and projects and the encounters experienced in clinical attachments or on campus, notes from discussions and groups, or other key experiences.

• You will be asked to grade your answers over a five point scale

1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree

1) Gender: __________

2) Age: _____________

3) Year or Phase? (i.e. your Year or Phase when you last submitted your portfolio, not your current Year or Phase): _____________

4) Theme of portfolio _________ [University 2 only, drop-down list of themes for each year group provided]

Communication

5) Writing my portfolio essay helped me develop my written communication skills:

6) Writing my portfolio essay made me appreciate that developing written communication skills are important for a doctor:

7) The evidence I used in my portfolio essay helped me think about the importance of good oral communication between doctors and patients:

Doctor and the Patient

8) The evidence I thought about for my portfolio essay helped me understand that patients have rights that need to be respected:

9) The evidence I used in my portfolio essay helped me think about ethical situations with patients that I may not have thought of:

10) The evidence I considered for my portfolio essay helped me understand the patients’ role in the management of their condition:

Professionalism

11) Collecting evidence for my portfolio and writing the essay required me to think critically about the information I would include:

12) Collecting evidence for my portfolio and writing the essay helped me realise that setting learning goals is an important skill for a doctor to develop:

13) Collecting evidence for my portfolio and writing the essay helped me understand the need to have a plan to keep up-to-date because medicine is an ever developing field of knowledge:

14) Writing my portfolio essay helped me realise that whilst every clinician has strengths and weaknesses, it is important that they recognise their own limitations:

15) Writing my portfolio essay made me aware that I have weaknesses in some areas which I need to deal with in future:

16) Writing my portfolio essay made me think about methods I might use to improve my performance, such as seeking assistance from peers and senior colleagues:

Teamwork

17) Collecting evidence for my portfolio and writing the essay helped me to understand the importance of working in an interprofessional team:

18) Collecting evidence for my portfolio and writing the essay helped me appreciate how important effective peer teamwork is for the functioning of a peer working/learning group:

19) Collecting evidence for my portfolio and writing the essay helped me to recognise the importance of senior and junior members of the team working together effectively:

Diversity

20) Collecting evidence for my portfolio and writing the essay helped me appreciate how different cultures can have different attitudes towards health and disease:

21) Deciding which evidence I would use for my portfolio essay helped me realise that a person's socio-economic status can influence the illnesses they develop:

22) Collecting evidence for my portfolio essay has helped me understand the importance of the patient's social context in the management of their illness:

Comments

23) Any comments about your experiences in collecting evidence for your portfolio, including any suggestion for how the MB/BS program could help to you to collect appropriate evidence:

24) Any comments about how to make the portfolio essay assessment more useful for your development as a doctor: