Combined formative and summative professional behaviour assessment approach in the bachelor phase of medical school: A Dutch perspective

Abstract

Background: Teaching and assessment of professional behaviour (PB) has been receiving increasing attention in the educational literature and educational practice. Although the focus tends to be summative aspects, it seems perfectly feasible to combine formative and summative approaches in one procedural approach.

Aims and method: Although, many examples of frameworks of professionalism and PB can be found in the literature, most originate from North America, and only few are designed in other continents. This article presents the framework for PB that is used at Maastricht medical school, the Netherlands.

Results: The approach to PB used in the Dutch medical schools is described with special attention to 4 years (2005–2009) of experience with PB education in the first 3 years of the 6-year undergraduate curriculum of Maastricht medical school. Future challenges are identified.

Conclusions: The adages ‘Assessment drives learning’ and ‘They do not respect what you do not inspect’ [Cohen JJ. 2006. Professionalism in medical education, an American perspective: From evidence to accountability. Med Educ 40, 607–617] suggest that formative and summative aspects of PB assessment can be combined within an assessment framework. Formative and summative assessments do not represent contrasting but rather complementary approaches. The Maastricht medical school framework combines the two approaches, as two sides of the same coin.

Introduction

Medical schools today are increasingly recognising the importance of professionalism and professional behaviour (PB; Stephenson et al. 2006; van Mook et al. 2009a, e). Concurrently, postgraduate training programmes are undergoing a developmental shift towards competency-based programmes, most of which include professionalism (Accredition Council for Graduate Medical Education 1999; Rowley et al. 2000; Edelstein et al. 2005; Lee et al. 2007; Stuurgroep Modernisering Opleiding en Beroepsuitoefening in de Gezondheidszorg 2007; Royal College of Physicians and Surgeons in Canada 2008). After completion of postgraduate training, clinicians are expected to attend continuous medical education (CME) programmes to continue to strive to be ‘better professionals’ (Larkin 1999; Gruen et al. 2003; Stern & Papadakis 2006). Teaching around professionalism involves ‘setting expectations’, ‘providing experiences in the curriculum’ and ‘evaluating outcomes by assessment’ (Stern & Papadakis 2006; van Mook et al. 2009d). Assessment of PB serves a formative (‘assessment (derived feedback) drives learning’) as well as a summative purpose (‘no learning, no pass’ or ‘go/no-go decisions’). The formative aspect is obviously essential and should benefit students and doctors alike (Aultman 2006; Stern 2006; Thistlethwaite & Spencer 2008b; Wear & Cruess et al. 2009). The summative aspects, identifying the few ‘bad apples’, however, often receive more emphasis (O’Neill 2000; Esmail 2005; Postma et al. 2006; Rynja et al. 2006). Unprofessional behaviour should indeed be taken seriously since it has been shown to be associated with later poor adherence to guidelines, impediments to communication, collaboration, information transfer and workplace relationships, low staff morale and high staff turnover, medical errors, adverse outcomes and malpractice suits, while it can seriously undermine public trust in the profession (Rosenstein et al. 2002; Rosenstein & O’Daniel 2005a, b, 2006, 2008a, b; Felps et al. 2006; Leape & Fromson 2006; McLemore 2006; Hickson et al. 2007). Identifying, measuring, addressing and assessing unprofessional behaviour may thus be a complementary approach alongside teaching and promoting professionalism (Papadakis et al. 1999; van Luijk et al. 2000; Hickson et al. 2007). Although, formative and summative aspects of PB may appear to be incompatible and mutually exclusive at first glance, it has been proven possible to join both aspects in a structured framework of PB. However, definitions of and approaches to concepts of professionalism and PB as well as legal possibilities for dismissal of dysfunctioning students differ not only between countries but also between continents. Whereas many North American examples of such frameworks can be found in the literature, published examples from other continents are scarce (Schonrock-Adema et al. 2007). In this article, we present the prevailing educational approach to PB in the Netherlands with special attention to the experiences with assessment of PB in the bachelor programme of Maastricht medical school. We also describe some future challenges.

Table

Practice points

One should realise that approaches to the concepts of professionalism and PB as well as legal possibilities for dismissal of dysfunctioning students differ not only between countries, but also within continents. Likewise, the approach to assessment may differ. Although the focus tends to be on the summative aspects of PB assessment, it seems perfectly feasible to combine teaching, as well as formative and summative assessment in the same procedural approach. PB assessment may be performed combining judgements during regular educational activities with critical incidents regarding unprofessional behaviour. Although many students occasionally receive a negative judgement during regular educational activities, the number of persistently dysfunctioning students after counselling and guidance is limited. So far, dismissal of persistently dysfunctioning students is not possible within Dutch legal regulations.

Legal boundaries and current guidelines regarding assessment of PB in the Netherlands

The current and future Higher Education and Research Act

The current Higher Education and Research Act (WHW) does not include provisions regarding expulsion from medical school on the grounds of unprofessional behaviour. The only legal provision for mandatory cessation of studies within the act is the possibility for the Board of Directors of a university to issue a ‘negative, obligatory study advice’ in the first year of medical school. Thereafter, mandatory cessation of studies is no longer an option. But the new Higher Education and Research Act (WHOO), which is currently in preparation, does contain a provision for mandatory cessation of medical studies and permanent dismissal in the cases of persisting and non-remediable unprofessional behaviour. The specification of cases where dismissal is warranted prevents dismissal on the basis of relatively minor, isolated events, such as an incidental breach of the peace.

National guidelines for PB

There are striking international differences between approaches to professionalism (van Mook et al. 2009e). In the Netherlands, the so-called Blueprint for undergraduate medical education (which has been revised recently) (Blueprint 2001) formulates final competency levels for graduating doctors regarding knowledge, skills and PB (Metz et al. 2001). The working-group ‘Consilium Abeundi’ of the Association of Universities in the Netherlands, comprising members from the Dutch faculties of medicine, dentistry, and veterinary medicine, added requirements concerning PB. ‘Consilium Abeundi’ refers to non-binding advice given to a student to leave the study programme he or she is attending. The working group proposed a practical definition of professional behaviour (Project Team Consilium Abeundi van Luijk 2005): observable behaviours that reflect the norms and values of the medical profession. The group also formulated nationwide guidelines with regard to the learning, teaching, and assessment of PB and recommendations for guidance of students showing unprofessional behaviour (Project Team Consilium Abeundi 2002). Three PB related categories are distinguished: ‘Dealing with daily work and tasks’, ‘Dealing with others’ and Dealing with self-functioning’ (van Luijk et al. 2000; Project Team Consilium Abeundi 2002). An overview of the different categories and attributes is provided in Table 1. The use of observable behaviours as the basis for assessment and guidance was intended to facilitate the practical implementation of education related to aspects of PB. Maastricht medical school has implemented a teaching and assessment framework that is in line with the recommendations of the Consilium Abeundi working group and complies with the current legal requirements.

Table 1.  Dimensions of PB as defined by the Project Team Consilium Abeundi of the Association of Universities in the Netherlands (Project Team Consilium Abeundi van Luijk 2005) (reproduced with permission from van Mook et al. 2009e)

Dimension	Aspects	Subcategory	Details
Dealing with tasks/work	Time management Information management Ability to work independently Requesting supervision when necessary Preparation and presentation of patient demonstration Punctuality Honouring commitments Observable dedication, commitment and sense of responsibility Coping with stress Leadership	Monitoring level of competence	Keeping up to date in the professional field, loyally to the profession, responsibility and independence
		Monitoring limitations of competence	Indicating where competence ends, abstaining from behaviour/statements that are not within one's competence, even when colleagues put pressure on one to exceed these limitations
		Participation in team competence	Forms of teamwork to achieve team competence, delegating and division of tasks, chairing meetings, substituting for others, honouring commitments, loyalty to decisions made
Dealing with others	Communicating in understandable language Adequate command of Dutch language Adequate non-verbal behaviour Adequate behaviour in gathering and giving information Giving a presentation/reporting Ability to construct a structured consultation Honouring commitments Collaborating Dealing with privacy and doctor patient confidentiality Empathising with patient's expectations Clearly explaining one's role as clerk and its possibilities and limitations Handling conflict Handling emotions of others Adequate interaction with colleagues and nurses Being courteous and respectful Negotiating skills	Gathering information from another person's perspective	Includes empathy, being unprejudiced, finding contextual information, active awareness of the emotional implications of information
		Providing information from another person's perspective	All behaviours directed at information and advice tailored to the recipient: e.g. adapting information to the other person's emotions and capabilities for understanding, presenting concrete information, checking whether the information has actually been received, inviting response
		Aligning other person's perspective and one's own competence in decision making	All behaviours directed at facilitating decision making, taking into account both the other person's perspective and the doctor's competence. E.g. encouraging someone to respond, meta-communication, implementation of informed consent procedures
Dealing with oneself	Ability to self-reflect Dealing with feedback and criticism Good personal appearance Self-management/setting boundaries Dealing with uncertainty Handling own emotions	Reflection on own behaviour and the underlying dynamics	Self-observation, gaining insight into and dealing with one's emotions, motivation, cognitions including values, standards, prejudices, as well as how these have evolved, and their effect on one's own behaviour
		Testing one's reflections against the opinions of others	Being receptive to and stimulating feedback on the and criticism on one's own behaviour
		Implementation of the results of reflections against the opinions of others	Developing and experimenting with behavioural alternatives on the basis of self-reflection and feedback

The Maastricht framework

Brief introduction to the Maastricht curriculum

In order to understand how PB is evaluated and assessed in the Maastricht curriculum, some knowledge of this curriculum is essential. The school offers a 3-year bachelor followed by a 3-year master curriculum in medicine and uses a predominantly problem-based learning (PBL) approach. Annually, 340–370 students are admitted to the first bachelor year. The first 2 years emphasise knowledge and skills acquisition and simulated patients are used as ‘problems’, serving as the starting points for learning by problem solving. The curriculum is structured in 6–10 week thematic blocks during which the main educational format is small groups of around ten students facilitated by a tutor, which meet twice weekly. The first session of the week is devoted to understanding and brainstorming around a (patient) problem, culminating in concrete learning goals. After the session students collect information regarding these issues, which they report and discuss in the second session of the week. In year 3, paper patient problems are replaced by real patient encounters in the student outpatient clinic in the academic hospital, and the master phase is dominated by clinical clerkships (Diemers et al. 2008).

Organisation of education in PB

The Examination Committee (EC) of Maastricht medical school (the regulatory body responsible for all examinations) underscored the importance of students’ development with regard to PB and instituted a Committee on Professional Behaviour (CPB), which is formally responsible for teaching and assessment of PB in the bachelor programme. In brief, the CPB longitudinally discusses all judgements of all students’ PB by different assessors during the educational sessions in one academic year (see the section on Assessment below), determines the end-of-year judgements, and intensively communicates with the EC. The EC also meets regularly with student advisors to discuss all students who have received negative judgements (‘fail’) or missed too many sessions to allow reliable judgement of PB (‘no judgement possible’). The CPB consists of five members, in varying combinations of doctors, educationalists and students, and is supported by a secretary appointed by the medical school. Student members are not present when personal issues regarding their peers are discussed. The following sections are concerned with the current framework for teaching and assessing PB during the bachelor programme.

Creating awareness of and staff development regarding PB

Failure to detect and respond to even minor lapses in PB is frequently reported in both undergraduate and postgraduate training (Project Team Consilium Abeundi 2002; Thistlethwaite & Spencer 2008a). In the past, staff were often uninformed or indulgent with regard to students’ non-PB. Possible reasons were fear of unpleasant or time consuming encounters with students or staff or fear of lawsuits (Pellegrino 2002). To overcome these problems, new and experienced Maastricht tutors are provided with practical guidelines for assessing PB. At least two important messages are conveyed. First, in cases of doubt (‘pass’ or ‘fail’) staff are explicitly advised to opt for a ‘fail’ instead of a ‘borderline-pass’. This policy is supported by evidence that equivocal feedback on assessment forms for PB correlates negatively with quantitative judgements (marks) in the same assessment (Frohna et al. 2005). Second, it is the tutor who determines the usefulness of the assessment, e.g. by writing extensive feedback on the form. Without formal, written feedback the assessment is far less useful, if not useless (van Mook et al. 2007).

(Mandatory) tutor instruction (teach-the-teacher sessions) takes place in 6-week cycles at the beginning of each new block and is complemented by an annual meeting in which teachers share their experiences and discuss questions and problems regarding evaluation and assessment of PB. Web-based information about the background and practice in relation to PB is available to students and staff. Course coordinators are requested to schedule time for assessment of PB, in recognition of the status of PB as an integral and important part of the curriculum. This policy has reduced the practice of superficial and brief evaluations of PB.

Teaching and learning PB during the Consultation and Reflection Programme

In addition to teaching (awareness) and assessment of PB in all curricular blocks with changing groups of students and tutors, PB has a central role in a longitudinal programme in each bachelor year, the Consultation and Reflection (CORE) programme. This programme uses problem based and experimental teaching methods, such as individual video-recorded consultations with simulated patients, to help students acquire theoretical knowledge and skills in communicating with patients and reflecting on their future role as a doctor. Students remain in the same group with the same teacher for the duration of the academic year. Ten days after taking part in individual consultations with simulated patients, the CORE group meets to discuss these contacts based on students’ individual requests for feedback, which result in individual strength–weakness analyses. During discussions moderated by experienced, trained teachers, students’ awareness and mindfulness of PB are stimulated. Due to its content and longitudinal nature, the CORE programme is eminently suitable for learning and assessment of PB. Over 20 additional optional modules, for example dealing with non-verbal and intercultural communication and dealing with emotions and commercialisation in medicine, are offered to facilitate further development of PB.

Other ways of teaching and learning around PB

All first year students have to create a portfolio aimed at promoting self-reflection skills and containing reflections on educational activities, including the CORE programme. As part of the portfolio process, mentors guide individual students during their longitudinal development of strengths and weaknesses in communication and reflection, and help students to set intermediate learning goals (formative assessment). Formal, summative portfolio assessment, however, is performed by an assessor with no involvement in the student's guidance. Furthermore, background information about PB is provided during a plenary lecture early in the first year, and aspects of medical history, ethics, sociology, humanism, law and other areas involving professionalism issues are dispersed over the curriculum. Notwithstanding the strong importance of the above-mentioned educational formats, (behavioural) skills relevant for adequate functioning in a group are primarily learned during tutorials, with feedback on students’ functioning being provided by the tutors.

Assessment of PB

We hope that the preceding has shown that in the first 2 years, education (and consequently assessment) in PB is mainly focused on the student–teacher and student–peers relationship. In later years, the emphasis shifts to the student–patient and student–physician relationships, and assessment comprises two different approaches (van Mook et al. 2009c), which we will discuss in the following sections: (1) reflection of staff members on students’ professional performance over a longer period of close observations and guidance and (2) reports of ‘critical incidents’.

Assessment by staff during scheduled educational activities

For assessment of students’ performance during formal educational activities, a PB assessment form is used (van Luijk et al. 2000; Project Team Consilium Abeundi 2002) (Figure 1). The form provides ample space for narrative feedback, which staff are strongly encouraged to provide. During each block PB is discussed at the start, halfway through and at the end of the block. In the first group, session assessment forms are distributed, the background of evaluation and assessment of PB is discussed briefly and the timing of evaluations is clarified. The session halfway the course is the most time consuming of the three evaluations, lasting an hour on average (for 9–11 students in one tutorial group). Careful home preparation and reflection on their behaviour during group sessions precede individual students’ plenary reflections as to which aspects of their PB need work and which are already acceptable or even excellent. Peers and tutor give their views, with feedback and suggestions for improvement. Based on these discussions, the tutor formulates an intermediate formative judgement of each student. Negative judgements are to be followed by confrontation, feedback and proposals for remediation from the tutor (van Luijk et al. 2000; Project Team Consilium Abeundi 2002). The mid-course judgement combined with new qualitative comments derived from the second part of the course are the basis for the final summative judgement in the form of a ‘pass’, ‘fail’ or ‘no judgement possible’ when a student has missed too many (or all) sessions to enable reliable and valid judgement. The end-of-course assessment is substantiated verbally and in writing, and the student is requested to sign the assessment form, although this does not imply agreement with the judgement. In addition to these regular assessments of PB, the CPB has instituted the possibility of filing reports on critical incidents concerning PB.

Figure 1. The PB assessment form currently in use in Faculty of Health, Medicine and Life Sciences, Maastricht.

Negative critical incident reports

Reports of ‘critical incidents’ by staff document in detail a specific event in which a student showed negative PB. These reports are directed in particular to ‘the extreme outliers of behaviours, students on whom staff spend an inordinate amount of time’ (Papadakis & Loeser 2006). The nature of these so-called incident reports is often more serious than negative judgements during formal educational activities and may result in a student being denied access to an educational activity. Examples are reports of sexual harassment or intimidation, verbal aggression or threatening behaviour. Dealing with these reports is often urgent, which makes this task an educational emergency. Starting in the 2005–2006 academic year the procedure for dealing with cases of ‘fail’, ‘no judgement possible’ and critical incidents has been refined. The current procedure and the associated time frame will be discussed in detail.

Structure for dealing with regular fail judgements and critical incident reports

Early identification of lapses in behaviour is critical to achieve remediation before behaviour has become refractory to change (Sullivan & Arnold 2009). The threshold for reporting unprofessional behaviour should therefore be kept low. The CPB gathers and files all judgements of PB and requests all students to elaborate on the reasons for negative judgements by e-mail, even if only one single lapse is concerned. In accordance with the adversarial principle, all students who receive a negative judgement are also invited to clarify their behaviour in a meeting with the CPB. If necessary, the tutor is asked to explain his/her judgement as documented in the PB form. Persistent failure to respond to these requests is considered unprofessional behaviour and results in a negative end-of-year judgement. Prior to the CPB meeting all judgements are discussed with the student advisors in order to optimise advice, guidance and possible subsequent counselling. During the CPB meeting the background and principles of assessment of PB are explained to the student in question, and the reasons for the negative judgement are discussed extensively. Since every professional has a range of behaviour, incidental occurrences of inappropriate behaviour (so-called lapses) are not necessarily considered synonymous with unprofessional behaviour. In decisions whether a certain behaviour is acceptable or not, the behaviour should be considered within its appropriate context, including the environment in which it occurred as well as the student's phase of training (Fochtmann 2006; Sullivan & Arnold 2009). This may shape both the behaviour and influence assessment quality (Sullivan & Arnold 2009).

It should be stressed that a negative judgement of PB can never be compensated for by favourable judgements of other aspects of clinical competence (Project Team Consilium Abeundi 2002). The CPB decides whether a judgement provides sufficient grounds for an immediate negative end-of-year assessment and/or an immediate leave of absence. This decision is sent to the EC, the student, the tutor and, if appropriate, the mentor, since students are obliged to incorporate all judgements of PB in their portfolios and a ‘fail’ must always be discussed with the mentor. Whenever required, students are advised of remediation strategies. If the CPB advice alone is considered insufficient, the student is referred to a student advisor, student psychologist or mentor. In extreme cases the student's GP, spiritual advisor or psychiatrist can be consulted, with the student's consent. A clear remediation plan should be tailored to the student's needs, and may involve meetings with, for example, the student advisors, the CPB and a psychologist. Ideally, there should be a regular exchange of information between all parties involved, provided the student consents to this. Practical requirements for productive meetings with the student include setting clear objectives before the scheduled meeting, the presence of witnesses, note taking and setting transparent goals and expectations. A timeframe for the process and adequate guidance with frequent feedback sessions are also of paramount importance (Sullivan & Arnold 2009). Any progress and agreements that are reported should be tested against the original plan of action (Project Team Consilium Abeundi 2002). Candid documentation of every incident, the remediation strategy and the student's response is required. All reports should be handled confidentially and complainants should receive follow-up of review and resulting actions (Sullivan & Arnold 2009). Normally, the extent to which behaviour can be remedied, the willingness and ability of the person involved to change his or her unprofessional behaviour as well as possible persistence of the behaviour over time all determine whether the behaviour will ultimately be categorised as unprofessional or not. Insight into and awareness of the problem, willingness and ability to change, consistency in adhering to the remediation plan can contribute to success. Likewise, denial and a weak follow-up plan are contributors to remediation failure (Sullivan & Arnold 2009). Longitudinal tracking of negative judgements of PB and reports of critical incidents can thus be used to monitor the effectiveness of remediation strategies (Papadakis & Loeser 2006).

A similar procedure to that for critical incidents is used in the cases of repeated instances of ‘no judgement possible’, generally due to frequent non-attendance. This can be an early clue and warning sign of structural problems, especially if they occur in succession. In the authors’ experience repeated absence is indeed predictive of other signs of unprofessional behaviour, and is frequently also associated with unsatisfactory results on skills and knowledge tests. When deemed necessary by the CPB and/or student advisors, a student is asked to provide written clarification and/or invited to meet with the CPB.

The algorithm in Figure 2 provides an overview of the assessment procedure for PB during regular courses and critical incident reports in the first 2 year. A comparable framework is available for the end-of-year assessment procedure.

Figure 2. Algorithm displaying the procedure for dealing with the judgements of students’ PB after all scheduled educational activities, as well as critical incident reports.

The end-of-year assessment procedure

For the end-of-year judgement, all available information (regular judgements of PB and critical incident reports) is weighed qualitatively with account being taken of the number, nature and order of receipt of negative judgements over the year as well as the educational activities involved. An algorithmic advice (summation and subtraction of the numbers of negative and positive judgements) has been abandoned for several reasons. First of all, the sequence of judgements is meaningful. A series of negative judgements followed by several ‘passes’ has other implications than a series of negative judgements after a successful start. After all, PB develops longitudinally. Second, the contexts of judgements may differ and the weight attributed to a judgement will vary accordingly. The CPB gives more weight to judgements resulting from educational activities with longitudinal follow-up, such as the CORE programme. Third, algorithmic advice ignores the qualitative aspects of each negative judgement. For example, a negative judgement of a student who structurally shows up late for tutorials but whose participation and communication are deemed good carries a different weight and requires a different approach regarding advice and guidance than combined negative judgements concerning timeliness, knowledge, participation and communication. Finally, incidental reports of critical incidents are not considered in an algorithmic advice. The diagram in Figure 3 provides an overview of the end-of-year procedure in the first 2 years.

Figure 3. Algorithm displaying the final (end-of-year) judgement procedure regarding.

No judgement due to non-attendance or a negative end-of-year judgement means that the student is not awarded the associated educational credit points, which may preclude admission to the next year of the medical school programme. When a pass on PB would result in the required credits, a conditional pass can be granted and the student is allowed to enter the next year. In the following years, the students must gain ‘passes’ on all assessments of PB to compensate for the earlier ‘fail’. Repeated negative judgements, despite adequate remediation efforts, are followed by a more formal response and ultimately lead to penalties (Papadakis & Loeser 2006). Thus, depending on the nature and extent of their unsatisfactory PB, students may either not progress to the next year or provisionally pass and move on to the next year, in both cases with mandatory, individualised remediation programmes. Practical tips based on the preceding discussion of the implementation of assessment of PB are summarised in Table 2.

Table 2.  Lessons learned regarding teaching and assessment of PB

No.	Tip
1	Create awareness regarding the importance of PB in students early in the curriculum, for example by giving a lecture or implementing a white coat ceremony.
2	Aspects of medical history, ethics, sociology, humanism, law and other areas where professionalism issues are represented, should be dispersed throughout the curriculum.
3	The formative aspect of PB assessment is obviously essential, and should compliment summative PB assessment.
4	Use PB forms which provide ample space for narrative evaluative feedback, which faculty are highly encouraged to complete.
5	Create awareness regarding the importance of PB assessment by instituting mandatory staff development programs, and making it a mandatory and scheduled part of each course.
6	In case of doubt between ‘pass’ or ‘fail’ the committee urges staff to preferably give the judgement ‘fail’ over ‘pass-by-mercy’.
7	The tutor fulfils a pivotal role regarding PB assessment, and largely determines the quality of PB assessment. The instrument itself (the form) itself has limited intrinsic value.
8	Development of additional training programmes (such as the described CORE program), underscoring the importance of aspects of professionalism and emphasis on PB development can compliment professional development during regular educational activities.
9	Since every professional has a range of behaviours, incidental occurrences of inappropriate behaviour (so-called lapses) are not per se synonymous with unprofessional behaviour.
10	Early lapse identification is critical to facilitate remediation attempts before behaviour becomes refractory to change
11	Furthermore, a negative judgement regarding PB should never be compensated for by other aspects of clinical competence.
12	A clear remediation plan should be individually tailored to the student's needs, and may involve alternating meetings with for example the student advisors, the CPB and a psychologist, with regular information exchange between all those involved.
13	Practical aspects are having clear objectives before scheduled meetings, having witnesses present, scribing, and setting transparent goals and expectations.
14	Any progress reported and agreements made, should be tested against the original plan of action.
15	A timeframe in which these should be fulfilled, and assuring adequate guidance and frequent performance feedback sessions are also of paramount importance.
16	Nevertheless, all reports should be confidentially handled, and complainants should receive follow-up of review and resulting actions.
17	An algorithmic advice (summation and subtraction of number of negative and positive judgements) should be abandoned for several reasons.
18	Changes will certainly not occur in the absence of strong institutional leadership!

Voluntary ending of studies versus dismissal

Although, most students improve their PB after feedback, some continue to show unprofessional behaviour. As was also reported by Papadakis et al., we think that multiple critical incident reports are a sign of significant shortcomings in a student's professional development. These students often refuse to accept responsibility for their problems and accuse their evaluators of unfairness and inaccuracy (Papadakis & Loeser 2006). However, we should point out that these students are rarely encountered in our institution. In the same group, knowledge and skills test results as well as clinical performance are also frequently below average (Rhoton 1994; Project Team Consilium Abeundi 2002; Haurani et al. 2007). This combination seems indicative of a structural problem often caused by a personality disorder and/or persisting conditions in a student's social environment (Project Team Consilium Abeundi 2002). The EC determines the consequences for the student's study progress and decides on appropriate penalties after consultation with the CPB. Students whose behaviour fails to improve can ultimately be advised to end their studies. When a student complies with this advice, the medical school is under an obligation to help him or her find an alternative career, albeit that this obligation is not required by law and of limited duration. Possible solutions include pursuing an alternative degree in medicine, which does not involve contact with patients, or enrolment in a related master's degree programme, such as medical biology) (Project Team Consilium Abeundi 2002). For such measures to be feasible, it is of paramount importance that assessment of PB is firmly embedded in the medical school's examination regulations. In the Netherlands, however, changes in the law are also required to enable medical schools to deny persistently dysfunctioning students access to education if they refuse to accept one of the above-mentioned alternatives. As matters stand today, the law only recognises a negative study advice issued in the first year by the EC (Project Team Consilium Abeundi 2002) as a valid reason to expel a student from medical school. Thereafter, definitive dismissal is only possible after a legal verdict. Such problems do not arise in the USA, for example, where persistent unprofessional behaviour is considered a violation of the contract signed on admission to medical school, which automatically results in dismissal.

From theory to practice: The results of the Maastricht framework 2005–2010

Quantitative and qualitative data were gathered prospectively. The most apparent findings will be presented briefly.

Numerical facts regarding regular educational activities and critical incident reports

Table 3 summarises quantitative data obtained in Maastricht between 2005 and 2010. Since year 3 was only recently added to the CPB's responsibilities, limited data for that year is available. Several phenomena can be noticed. First, the number of negative assessments of PB (‘fails’) during regular educational courses has remained stable over the years in the first and second years. In contrast, there is a recent increase in the number of ‘no judgement possible’ in first year, whereas this number has remained stable in second year. The increase in the number of meetings of students with the CPB is a reflection of the repetitive nature of these judgements, students’ (unsatisfactory) explanatory responses and/or the perceived necessity of a meeting with the CPB suggested by the student advisors. The number of critical incident reports also shows a recent increase among first-year students. Third, with the exception of one year, male students received more negative judgements and ‘no judgement possible’ compared to their female counterparts, a finding for which we could find no clear explanation. Whether gender differences, for example in interpersonal communication skills, make male students more prone to unprofessional behaviour is not clear. In the study by Papadakis et al. (2005) gender was not statistically associated with disciplinary action by medical boards (odds ratio 2.24, p = 0.09), but the same authors reported a greater proportion of male students displaying problematic behaviour in, for example, obstetrics–gynaecology (46 vs. 29 male vs. female students) (Papadakis & Loeser 2006).

Table 3.  Overview of numerical data on negative PB judgements during regular educational activities and critical incident reports in the period 2005–2009 for the first 2 years of Faculty of Health, Medicine and Life Sciences, Maastricht, The Netherlands

Year 1
	Number of students in year 1				Number of judgements				Number of students involved				Gender of students in year 1 Male/Female
Judgements during scheduled educational activities	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009
Fail					10	23	23	30	10	19	20	26
No judgement possible					78	73	82	147	54	56	58	88
(Sub)total					88	96	105	177	64	75	78	114
Incidental reports regarding critical incidents					3	6	6	21	2	6	6	18
Total	359	355	341	375	91	102	111	198	66	81	84	132	131/228	115/240	119/222	147/228
	Gender of students involved
	Male				Female				Meeting with Committee on PB				Referral student counsellors
Judgements during scheduled educational activities	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009
Fail	8	7	11	21	2	12	9	5	9	16	12	19	3	5	4	9
No judgement possible	28	23	3	49	26	33	1	39	5	1	1	7	1	0	0	0
(Sub)total	36	30	14	70	28	45	10	44	14	17	13	26	4	5	4	9
Incidental reports regarding critical incidents	2	4	4	14	0	2	2	4	1	3	1	3	1	3	0	2
Total	38	34	18	84	28	47	12	48	15	20	14	29	5	8	4	11
Year 2
	Number of students in year 2				Number of judgements				Number of students involved				Gender of students in year 2 (Male/Female)
Judgements during scheduled educational activities	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009
Fail					28	21	25	21	24	21	24	18
No judgement possible					75	104	92	97	56	72	71	77
(Sub)total					103	125	117	118	80	93	95	95
Incidental reports regarding critical incidents					3	5	2	12	3	5	2	12
Total	367	359	344	326	106	130	119	130	83	98	97	107	149/218	141/218	121/223	117/209
	Gender of students involved
	Male				Female				Meeting with Committee on PB				Referral student counsellors
Judgements during scheduled educational activities	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009	2005–2006	2006–2007	2007–2008	2008–2009
Fail	18	10	16	14	6	2	8	4	13	6	13	11	3	2	3	3
No judgement possible	30	41	40	42	26	31	31	35	15	6	9	4	2	3	0	2
(Sub)total	48	51	56	56	32	33	39	39	28	12	22	15	5	5	3	5
Incidental reports regarding critical incidents	2	1	1	10	1	4	0	2	2	1	0	030	2	2	0	1
Total	50	52	57	66	33	37	39	41	30	13	22	15	7	7	3	6

Reasons for negative judgements of PB

The reasons for negative judgements of PB during regular courses in the first two years are comparable, comprising frequent non-attendance and unsatisfactory performance in preparing for, reporting back and participation in tutorials and, to a lesser extent, insufficient cooperation with peers, lack of improvement of participation and poor planning and timekeeping. Combinations of the above occur frequently. It is noticeable that students repeatedly fail to respond to tutors’ communication requests. Poor self-reflection is reported rarely. Thus, negative judgements almost always relate to the categories ‘Dealing with work’ and ‘Dealing with others’. The relative absence of issues relating to ‘Dealing with self-functioning’ may be attributable to first year tutorials being less suitable for judgements of students’ self-reflection and responses to feedback. Nevertheless, many negative judgements and reports of critical incidents occur after repeated feedback has failed to result in behaviour change. This suggests that students lack self-reflective abilities (and/or abilities to change). The past few years have shown a significant decrease in unauthorised absence as a reason for negative judgements. Although this may have contributed to the increase in ‘no judgement possible’, the magnitude of the increase in ‘no judgement’ far exceeds the decrease in negative judgements. An example of a negative judgement of PB during regular courses is displayed in Table 4, box 1.

Table 4.  Illustrative examples of negative PB judgement during a regular educational course, an isolated critical incident report, as well as multiple and persisting negative PB judgements despite remediation attempts

Box	Example
1	Isolated negative PB judgement during a regular educational course
	Student A was given a negative PB judgement by the tutor during a 6-week course in the second year for reasons of unexplained absence during a tutorial group meeting. The student was previously known with neither PB, nor knowledge and skills deficits. During the CPB meeting the student explained that absence during 2 tutorial group meetings was unavoidable due to participation in the management of an international student association. This absence had been cleared with the tutor prior to the meeting. Returning from the meeting student was confronted with the severe illness of a relative, which he subsequently visited. Student failed to inform the tutor of this absence, considering it merely a private matter. Soon thereafter the relative passed away. The funeral was not attended since it would have resulted in additional absence. In retrospect the student regrets non-informing the tutor, designates it as unprofessional and not acceptable. Honesty would have been more appropriate. The CPB empathises with the reasons for the absence, but prior explaining would have contributed to the tutor's understanding. The advice was given to communicate more clearly with staff regarding such matters. No further negative PB judgements have been received thereafter.
2	Isolated critical incident report
	The CPB received a critical incident report on student B written by the mentor and CORE-coach. For reasons of absence during simulated patient contacts and their subsequent discussion during the CORE meetings, and related insufficient communication with the coach and peers. After several remediation attempts the student was refused further participation in the CORE programme. The student responded in writing, and later before the CPB, that the student struggled with chronic health problems. Complaints, medical testing, treatment and doctors’ visits limited the ability to participate in the training programmes. The student was not aware of special arrangements for students with a chronic illness. Referral to the student advisors and dean for student affairs resulted in less strict attendance requirements, pre-registration possibilities for skills training and tutorial groups, and individual examination possibilities of extended duration. The CPB advised to re-discuss the persisting health problems with the mentor and CORE-coordinator, and be frank to peers regarding possible future limitations to peers. Ultimately re-entering of the CORE programme was not feasible, and the student repeated the programme a year later.
3	Multiple persisting negative PB judgements despite remediation attempts
	Student C was repeatedly given negative PB judgements as well as received multiple critical incident reports during the first, second and third years. This pattern was paralleled by frequent and repeated failure on knowledge and skills testing. Passing the first two year of medical school took more than a lustrum. Major problems identified were communications problems and not being able to adequately cooperate regarding assignments with peers, lack of self-reflection, repeatedly ignoring feedback, perception of intentional humiliation and insulting by tutors, aggressive writing style when responding to questions by tutors and peers, criticising tutors and coaches when provided with constructive feedback, not showing up for resits and too frequent unauthorised absence. Despite intensive guidance by the student advisor, no improvement was observed. A comparable pattern was observed during correspondence and meetings with the EC and CPB. Reponses were primary, with an unpleasant, pejorative undertone which seemed to confirm instead of refuting the assessors’ views. Although communication during meetings was less hostile, e-mails sent to the EC and CPB, often without delay, accused them of e.g. ‘wilful bullying, intimidation, abuse of power and rights, and willingly obstructing study progress’. Repeatedly, an advice to end the study was not appreciated. Since mandatory cessation of study is so far impossible in the Netherlands, expiring study results ultimately formed the basis for preventing graduation from medical school.

Reasons for critical incident reports

Critical incident reports in the first year are frequently related to absence without notice from simulated patient encounters or resits, not meeting obligations related to the portfolio, or repeatedly failing to respond to hepatitis B immunisation appeals. The recent increase in critical incident reports from CORE programme coaches as well as unauthorised absence from resits may be due to increased awareness of the possibility of and procedure for filing critical incident reports. The immunisation non-compliance reports are especially interesting in view of recent reports by Stern et al. that only immunisation non-compliance and failure to complete required evaluation forms in the first two years of medical school were predictive of subsequent discussion in the academic review board in third year of medical school. These two predictors accounted for almost 14% of the variance in academic review board appearances. Immunisation non-compliance predicted low overall internal medicine clerkship professional evaluation scores accounting for just over 10% of the variance of this outcome (Stern et al. 2005). Absence without notice and non-compliance with remediation assignments during electives, absence without notice during resits, and absence and failure to meet other obligations related to simulated patient contacts (e.g. providing peer feedback) were reasons for critical incident reports in second year. The literature is non-informative regarding no-shows at standardised patient encounters. However, students who underestimated their actual performance on a standardised patient exercise early in medical school received higher (more positive) ratings and students who overestimated their performance received lower ratings on the same exercise (Stern et al. 2005). An example of a critical incident report is displayed in Table 4, box 2.

Results of end-of-year assessment

An overview is provided in Table 5. Although the number of student advisor-referrals has remained relatively stable for first year medical students, an increase can be observed in the final curriculum year, probably due to the increased number of both negative judgements and ‘no judgement possible’. The number of negative end-of-year judgements in the first year shows a parallel steep increase, even after correction for changes in the total number of judgements. By contrast, the emerging pattern for the second-year is stable again. The overall relatively stable numbers of fail judgements and critical incident reports may be suggestive of failure to change behaviour during the educational experience. It should be noted, however, that overall numbers of students are compared between years, and individualised data are not presented. Although the overall number of negative judgements is relatively stable, these judgements are infrequently generated by the same students during the (first three) years. One-time professional lapses are far more common than persisting unprofessional behaviour. In other words, a small minority of students (one to three per cohort) display severe and persisting dysfunctional behaviour despite intensive long-term counselling and guidance (and are ultimately advised to end their studies). Since this selected group has been proven to be lacking in reflective abilities, it is hardly surprising that this advice is hardly ever heeded. Cessation of studies can then only result from expiring examination results, lack of funds or a legal process. An example of a student with persisting unprofessional behaviour is presented in Table 4, Box 3.

Table 5.  Overview of end-of-year judgements in the period 2005–2009 for the first two years of Faculty of Health, Medicine and Life Sciences, Maastricht, The Netherlands

Year 1
	Number of judgement (%)
Final end-of-year judgement	2005–2006	2006–2007	2007–2008	2008–2009
Pass	326 (96)	319 (93)	302 (92)	297 (90)
No judgement possible	9 (3)	20 (6)	21 (6)	16 (5)
Fail	3 (1)	3 (1)	5 (2)	17 (5)
Total	338 (100)	342 (100)	328 (100)	330 (100)
Year 2
Pass	331 (94)	314 (93)	310 (93)	278 (94)
No judgement possible	14 (4)	19 (6)	19 (6)	16 (5)
Fail	6 (2)	5 (1)	2 (1)	4 (1)
Total	351 (100)	338 (100)	321 (100)	298 (100)

Future challenges

Several aspects of necessary innovation and change should be mentioned here. First, the current framework is labour intensive and thus relatively costly. A planned reduction of the number of courses incorporating assessment of PB could reduce costs without compromising reliability and validity, provided the option of critical incident reports is maintained. This management decision, however, opposes the face validity argument of integration of PB in the entire curriculum. Furthermore, the longitudinal aspect of assessment of PB currently receives insufficient attention. During several courses, teachers should be able to follow up on end-of-course judgements and learning goals from previous courses. Transfer of results and goals between activities, phases (bachelor-master, undergraduate-graduate) and teaching facilities (e.g. teaching hospitals offering clerkship rotations) is currently being explored. Other methods to strengthen longitudinal development which are being considered are: non-block and non-course related sessions in which PB is evaluated and assessed (e.g. sessions in which small groups of students reflect on paper cases, video vignettes or cases experienced in practice) (Alexander et al. 2004; Boenink 2006; van Mook et al. 2009b). Regular CPB meetings to optimise formal communication with the EC facilitate longitudinal tracking of student performance. Expansion of the CPB's responsibilities to include all six years of medical school is considered pivotal. The contexts in which PB is assessed in the first three years are limited to tutorials, but extend to the broader context of hospital wards in the clinical years. Continued assessment of PB during the clinical years coordinated by the same CPB would be essential to investigate to which extent behaviour in the first three years predicts later behaviour (in medical school). But this expansion is hampered by obvious financial constraints, and its implementation will therefore, most likely, be gradual. Third, emphasis is placed on teach-the-teacher sessions to create awareness of PB, teach the necessary feedback, communication and assessment skills, stimulate provision of in-depth qualitative feedback and improve the educational climate in all curriculum years. Attention to role modelling during clinical training is of particular significance if we are to make our practice congruent with our rhetoric.

Conclusions

The adages ‘Assessment drives learning’ and ‘They do not respect what you do not inspect’ (Cohen 2006) suggest that formative and summative aspects of assessment of PB can be combined within an assessment framework. Formative and summative assessments are not so much contrasting as complementary approaches. The Maastricht medical school framework combines these approaches: they are two sides of the same coin.

Acknowledgements

The authors thank Ms. Mereke Gorsira, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands, for critically reviewing the manuscript regarding use of the English language.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.