Regulatory end-point assessment of the consultation competence of family practice trainees in Kuwait

Abstract

Background: No single approach to the regulatory assessment of global consultation competence has been shown to possess the required levels of validity, reliability and feasibility. Objective: To evaluate the approach adopted in Kuwait to the regulatory end-point assessment of the global consultation competence of family practice trainees with particular reference to validity, reliability and feasibility. Methods: Family practice trainees in Kuwait were individually and directly observed for 3 hours in consultation with a minimum of 10 patients by a pair of examiners. Performance was judged against the explicit criteria of consultation competence as contained in the Leicester Assessment Package (LAP). Results: The marks independently allocated by the pairs of examiners to 126 trainees between 1994 and 2001 were within five percentage points on 91% of occasions. A reliability coefficient of 0.82 was achieved when two examiners independently marked candidates consulting with 10 real patients; this rose to 0.95 at the critical 50% pass–fail margin. The main sources of variance contributing to the reliability of marks allocated were candidate performance (42%) and the interaction of candidate performance across cases, i.e., case specificity (30%). The clinical challenges presented by the patients were judged by both examiners to be sufficient to enable performance to be assessed across the seven LAP consultation categories as follows: behaviour and relationship with patients (100% of consultations), interviewing/history taking (100%), record keeping (99%), patient management (99%), problem solving (98%), physical examination (95%), and anticipatory care (86%). Each assessment involved a pair of examiners and lasted approximately 3.5 hours.

Conclusion: The Kuwait clinical examination achieves high content validity and authenticity as it uses direct observation of performance, validated and explicit criteria against which performance is judged, and real patient challenges. It can discriminate between different levels of consultation performance and satisfies the recognized reliability threshold for regulatory examinations (0.82 vs 0.80). Accordingly, we recommend the use of such an approach in the regulatory end-point assessment of the global consultation competence of trainees in family practice. Such an approach is more valid, and is likely to be more feasible, than simulated surgeries or the short-case OSCE format.

• Consultation competence
• family practitioners
• feasibility
• regulatory assessment
• reliability
• trainees
• validity

Introduction

Since the core activity in clinical practice is the consultation between doctor and patient [1], it follows that the focal point of the assessment of clinical competence must be the systematic observation and analysis of the performance of a clinician in the consultation [2]. To date, no single procedure for the regulatory assessment of global consultation performance has been shown to possess the required levels of validity, reliability and feasibility.

We now report a study whose aims were to evaluate the regulatory end-point assessment of the global consultation competence of trainees in the Kuwait Family Practice Training Programme with particular reference to content validity and reliability (contributions to variance and generalizability).

Background

The Kuwait Family Practice Training Programme [3], which began in 1984, is a collaborative venture between the Royal College of General Practitioners (RCGP) and the Kuwait Ministry of Health. Following a selection process, all trainees undergo a full-time speciality training programme lasting 3 years.

Trainees initially spend 3 months in a family practice training centre, followed by 24 months in a rotation of hospital posts, during which they attend a weekly day-release programme supervised by family practice trainers. The final 9 months are spent in a family practice training centre.

Before being eligible to undertake independent clinical practice, trainees have to pass an end-point regulatory assessment which consists of three written papers (45% of available marks) and clinical (30%) and oral (25%) examinations. The overall pass mark is 60%, but candidates must also achieve a minimum mark of 50% in each of the three components. Internal and external examiners (the latter nominated by the RCGP and approved by the Kuwait Examination Board) share examining duties. Successful candidates are awarded the Diploma in Family Practice (RCGP/Kuwait).

The assessment tool used was the Leicester Assessment Package (LAP) [4], [5], which was developed to facilitate educational and regulatory assessment and enhancement of the global consultation competence of a doctor in the context of general practice. The LAP is an integrated assessment tool which contains seven prioritized categories of consultation competence and 39 component competences (see Box 1). The LAP criteria, against which consultation performance is judged, have been demonstrated to be valid for general practice in the United Kingdom [6] and Hong Kong [7]. The LAP also contains descriptive criteria to assist assessors in the allocation of marks (see Box 2). Use of the LAP requires consulting doctors at various stages in the consultation to answer certain questions to enable a teacher or examiner to become more aware of the reasoning which underpins the doctor's actions (see Methods).

Box 1. Diploma in Family Practice (RCGP/Kuwait): consultation competences assessed in the clinical examination.

1.	Interviewing/history taking (relative weighting: 20%)
	Introduces self to patients; puts patients at ease; allows patients to elaborate presenting problem fully; listens attentively; seeks clarification of words used by patients as appropriate; phrases questions simply and clearly; uses silence appropriately; recognizes patients' verbal and non-verbal cues; identifies patients' reasons for consultation; elicits relevant and specific information from patients and/or their records to help distinguish between working diagnoses; considers physical, social and psychological factors as appropriate; exhibits well-organized approach to information gathering.
2.	Physical examination (relative weighting: 10%)
	Performs examination and elicits physical signs correctly and sensitively; uses the instruments commonly used in family practice in a competent and sensitive manner.
3.	Patient management (relative weighting: 20%)
	Formulates management plans appropriate to findings and circumstances in collaboration with patients; makes discriminating use of investigations, referral and drug therapy; is prepared to use time appropriately; demonstrates understanding of the importance of reassurance and explanation, and uses clear and understandable language; checks patients' level of understanding; arranges appropriate follow-up; attempts to modify help-seeking behaviour of patients as appropriate.
4.	Problem solving (relative weighting: 20%)
	Generates appropriate working diagnoses or identifies problem(s) depending on circumstances; seeks relevant and discriminating physical signs to help confirm or refute working diagnoses; correctly interprets and applies information obtained from patient records, history, physical examination and investigations; is capable of applying knowledge of basic, behavioural and clinical sciences to the identification, management and solution of patients' problems; is capable of recognizing limits of personal competence and acting accordingly.
5.	Behaviour/relationship with patients (relative weighting: 10%)
	Maintains friendly but professional relationship with patients with due regard to the ethics of medical practice; conveys sensitivity to the needs of patients; demonstrates an awareness that the patient's attitude to the doctor (and vice versa) affects management and achievement of levels of cooperation and compliance.
6.	Anticipatory care (relative weighting: 10%)
	Acts on appropriate opportunities for health promotion and disease prevention; provides sufficient explanation to patients for preventive initiatives taken; sensitively attempts to enlist the cooperation of patients to promote change to healthier lifestyles.
7.	Record keeping (relative weighting: 10%)
	Makes accurate, legible and appropriate record of every doctor–patient contact and referral. The minimum information recorded should include date of consultation, relevant history and examination findings, any measurement carried out (e.g., BP, peak flow, weight, etc.), the diagnosis/problem (preferably “boxed”), outline of management plan, investigations ordered and follow-up arrangements. If a prescription is issued, the name(s) of drug(s), dose, quantity provided and special precautions intimated to the patient should be recorded.

Box 2. Criteria for the allocation of marks.

The following descriptors of performance are to be used as yardsticks of levels of competence.
Marks
80% or above	Consistently demonstrates mastery of all components: the criterion performance.
70–79%	Consistently demonstrates mastery of most components and capability in all.
60–69%	Consistently demonstrates capability in almost all components to a high standard and a satisfactory standard in all.
50–59%	Demonstrates capability in most components to a satisfactory standard: demonstrates minor omissions and/or defects in some components. Duration of most consultations appropriate.
40–49%	Demonstrates inadequacies in several components. No major omissions or defects but not currently ready for independent practice.
39% or below	Demonstrates several major omissions and/or serious defects; clearly unacceptable standard overall. Not safe to practise independently.

The LAP can be used with “real” or simulated patients and with “live” or videotaped consultations. It can also facilitate the achievement of reliable assessments of doctors consulting with simulated patients in an experimental situation [8] and in identifying medical students with unsatisfactory levels of consultation performance [9].

Methods

The procedure for the Kuwait clinical examinations is laid down in an Examiners’ Code of Practice by the Kuwait Family Practice Examination Board, and all examiners are required to follow it.

The clinical component consists of 3 hours of direct and independent observation by two examiners of candidates in consultation with a minimum of 10 consenting but unselected patients in a Family Practice Training Centre. There is one internal and one external examiner. The criteria against which clinical performance is assessed are specified in Box 1, and marks are allocated according to the criteria specified in Box 2. Whenever possible—and with the additional permission of the patient—examiners directly observe candidates conducting physical examinations and/or using instruments, and personally verify positive physical findings.

In each consultation, the examiner is required to seek clarification from the candidate by asking (in English) the following questions:

1. At the completion of (initial) history taking (the candidate to inform the examiner when this has occurred):

   • What are your diagnostic hypotheses at this stage?

   • Why have you erected these hypotheses?

   • What physical examination do you intend to carry out, and why?

2. After any physical examination:

   • What did you find on examination of the patient?

   • How have these findings affected your thoughts?

3. After the patient has left:

   • Why did you choose your management plan?No comments are allowed, and examiners must not enter into discussion with candidates.

Since the consultations are conducted mainly in Arabic, candidates are required to articulate a brief summary in English of the salient points of the history prior to the asking of the first LAP questions. Furthermore, an external examiner has the discretion to consult his internal examiner colleague to seek clarification on matters of fact relating to candidate/patient interactions. In order to maintain the independence of judgement of external examiners, internal examiners are not permitted to provide opinions.

According to the procedures described above, the examiners independently allocated marks to reflect the candidates’ performance in every LAP consultation category challenged in individual consultations. The sum of the consultation category marks represented the global performance in individual consultations. If a consultation category was not challenged in a particular consultation, the denominator was altered appropriately. For example, if anticipatory care had not been challenged, the denominator would be 90 instead of 100 in that consultation. All marks for individual consultations had to be allocated before the next consultation began.

Once the required number of consultations had been observed, the allocated marks were transferred to a master mark sheet. Examiners then awarded final marks to reflect overall performance in all seven categories of competence in turn. Since examiners were required to take account of the nature and difficulty of the clinical challenges presented, the overall mark did not automatically represent the average of marks awarded for individual consultations. The final mark was the sum of the marks allocated by each examiner divided by two.

To estimate reliability, a generalizability analysis [10], [11] was carried out with candidates (P), consultations (C) and examiners (A) as factors and sources of variance. Since real (i.e., unstandardized) patients were used, different candidates were assessed dealing with differing sorts of patient challenges. Thus, case variance was nested within candidate variance (C:P). Furthermore, the same examiners were used across all cases for an individual candidate, but different examiners were allocated to different candidates (A:P). Variance components were estimated using this design and subsequently generalizability coefficients were computed for a number of different samples of assessors and cases. (The complexity of clinical assessment procedures creates problems in both testing and determining their reliability. Although further explanation is provided in the Discussion, readers interested in more details of the educational and statistical principles underpinning the complex methodology involved can consult Fraser et al. [8].)

Content validity was investigated by determining the extent to which the LAP categories of consultation competence were sufficiently challenged by the real patients encountered to enable the assessors to arrive at judgments of actual clinical performance of candidates.

The local examiners were all senior trainers and experienced clinicians who had received formal training in the use of the LAP for regulatory purposes, occupying a minimum of 7 days. The external examiners were all experienced examiners and GPs in the UK. All had received a minimum of 2 days’ specific training, and all had undergone a programme to familiarize them with the organization and delivery of training and clinical care within Kuwait and with local culture and tradition. Several of the examiners had made multiple visits to Kuwait. All examiners also attended a formal briefing session immediately prior to every examination diet.

Results

Between 1994 and 2001, 126 clinical assessments were conducted, involving a total of 1328 patient consultations. There was a range of 6–21 candidates per diet. The assessments were performed by 11 internal and nine RCGP external examiners who examined a range of 1–28 candidates and 3–29 candidates, respectively. The mean global consultation score was 57% (range 45–72%, SD 6.8). The standard deviations for the scores allocated by internal and external examiners were 6.4 and 7.1, respectively. Nineteen candidates (15%) were allocated scores below 50%, and three candidates (2.4%) achieved scores above 70%. The scores independently and individually allocated by paired examiners were identical in five assessments (4%), within two percentage points of each other in 57% of assessments and within five percentage points in 91%. The maximum difference in scores allocated to an individual candidate was 9.4 percentage points.

Table I sets out the contribution of each source of variance to the reliability of the marks allocated. The variance calculations are based on the case-specific data which were collected from 1997 onwards involving 73 candidates. More than two-fifths (42%) of all varience were attributed to the variance between candidates, while less than one-third was associated with the cases (30%). Only 10% were the confounded effect of the case and the examiners.

Table I.  Contribution of each source of variance to the reliability of the marks allocated.

Source of variance	Estimated variance component	Standard error	Contribution (%) to variance
P	19.15	3.82	42%
C:P	13.43	0.93	30%
A:P	4.68	0.88	10%
CA:P	7.94	0.42	18%

P—candidates; C:P—consultations nested within candidates; A:P—examiners nested within candidates; CA:P—consultations×examiners nested within candidates.

Table II shows reliability (generalizability) coefficients as a function of the number of consultations and examiners. The actual numbers used were 10 consultations with two examiners, leading to a reliability of 0.82. The same reliability can be achieved by reducing the number of consultations to six if the number of examiners is increased to three. The reliability coefficient increased to 0.95 at the critical pass–fail level of 50%.

Table II.  Generalizability coefficients as a function of the number of consultations (cases) and examiners, using the same examiners rating across all consultations for a single candidate.

	Number of examiners
Number of consultations	1	2	3
2	0.55	0.63	0.67
4	0.66	0.74	0.77
6	0.70	0.79	0.82
8	0.72	0.81	0.84
10	0.74	0.82	0.86
12	0.75	0.83	0.87
14	0.76	0.84	0.88

The clinical challenges presented by the patients were judged by both assessors to be sufficient to enable performance to be assessed across the seven consultation categories as follows: behaviour and relationship with patients (100% of consultations), interviewing/history taking (100%), record keeping (99.2%), patient management (99.2%), problem solving (97.8%), physical examination (94.8%), and anticipatory care (85.9%).

Each candidate consumed approximately 7 hours of assessor time: 3 hours per examiner of direct observation plus up to half-an-hour for collating component marks and arriving at final mark allocations. To this has to be added travelling time to the candidate's health centre and examiner training and briefing (see Methods).

Discussion

“In high stakes (regulatory) assessments, credibility of the method is of major importance” [12]. Primarily, this means that the assessment must be both valid and reliable [13], although compromises usually have to be made in the “real world” on the grounds of feasibility [13]. Nevertheless, “validity is the sine qua non of assessment” [14], and “all assessments in medical education require evidence of validity” [14].

For an assessment to be valid, it must measure what it is supposed to measure. In the context of the Kuwait assessments, this means that the criteria against which consultation competence is judged should be professionally important and relevant, and the nature of the clinical challenges encountered should be suitable, reflecting day-to-day practice as closely as possible [13]. Furthermore, the assessment process must allow any examiner to directly observe the consultation performance of the candidate at all times [15]. The Kuwait examination satisfies all these validity criteria.

Although the criteria of consultation competence against which performance in the Kuwait clinical examinations was judged have been formally validated in the UK [6] and Hong Kong [7], this has not been specifically done in Kuwait. Nevertheless, despite the differences between the healthcare systems, “the close similarity in the acceptance of the LAP criteria in Hong Kong and the UK support the conceptualisation of the LAP as a generic assessment tool that can be applied to consultations in widely differing settings” [7]. Furthermore, the LAP has proved both acceptable and useful since the early days of the Kuwait Family Practice Training Programme for both formative and regulatory purposes [3]. Trainees, trainers and local examiners have made repeated use of the LAP and are very familiar with, and have made no changes to, the 39 LAP criteria of consultation competence. Furthermore, because the LAP criteria are generic, i.e., not case specific, this makes it very unlikely that any important aspect of the consultation would be overlooked. Consequently, it is reasonable to claim that the criteria of consultation competence used in the Kuwait clinical examinations are also valid in that context, since they command such consensual support from local experts. After all, “validity is judgemental … to hunt for validity in statistical procedures is to pursue a will o’ the wisp” [16].

Direct observation of actual performance in daily practice with real patients is the most authentic and valid approach to the assessment of consultation competence as it most closely represents real clinical practice [12], [17], [18]. Indeed, a systematic review [15] of published articles (1966–2001) on the validity and reliability of measures of clinical competence of physicians, medical students and residents identified the direct observation of trainees “in real-life situations” as the most valid form of assessment. Nevertheless, unlike Kuwait, “few assessments observe trainees in real-life situations” [15].

Although using real patients is highly authentic, it has the potential to compromise the content validity of the clinical challenge [19]. Perhaps the optimum method for overcoming this potential problem is the use of a blueprint with specific criteria for selecting a representative sample of patient challenges [12]. Our approach was to record the proportion of consultations in which each of the seven LAP categories of consultation competence was deemed by the assessors to have been sufficiently challenged for them to make a judgement of the consulting doctor's performance. This condition was satisfied in over 97.8% of consultations for five of the seven consultation categories. In respect of the other two categories, it is well recognized and accepted that a physical examination does not need to be conducted in every consultation, and that anticipatory care opportunities do not exist in all consultations. Accordingly, it is reasonable to conclude that 10 patients can provide a valid set of assessable challenges, although a minimum of eight patients produces reliable results. Nevertheless, it cannot be guaranteed that every candidate in the Kuwait examination encountered a totally representative sample of clinical challenges or identical degrees of difficulty. However, examiners were required to take account of the nature and difficulty of the clinical challenges in making their judgements (see Methods). Although the extent to which this occurred was not quantified, the difficulty of the cases has been factored into the generalizability analysis (see below).

“Reliability is defined as the extent to which a result reflects all possible measurements of the same construct” [10]. In the Kuwait examination, the “construct of interest” [10] is the candidate's consultation competence. The analysis of variance components facilitate measurement of all the possible sources of error in the assessment process (see Table I). With a reliable assessment instrument, differences in scores should reflect true differences between candidates (P). The other sources of variance in any assessment of consultation performance are the influence of the examiners (A) and the nature and difficulty of the clinical challenges (C), i.e., case specificity. Combinations of any of these sources of variance can cause distortion of true scores.

Determining true variations in candidate performance is the principal function of a regulatory assessment in order to accurately identify candidates who deserve to pass or fail. In Kuwait, over two-fifths (42%) of all the variance in the reliability of marks allocated can be attributed to the differences in performance between candidates. Accordingly, the Kuwait examination succeeded satisfactorily in discriminating between candidate performance. The influence of the nature and difficulty of the patient challenge (C) and the effects of the interaction between candidates across cases (C:P) was less than one-third (30%). Although both effects cannot be disentangled because of the nesting of cases within candidates, the reported influence of case specificity on performance is frequently much larger [13]. Indeed, it is accepted wisdom that “professional behaviour is highly dependent upon the nature and details of the problem being faced” [10]. On the other hand, the examiner contribution to variance is minimal, both nested within candidates (A:P = 10%) and in the confounded influence of the case by examiners within candidates (plus all other non-explained residual variance) (CA:P = 18%). Overall, the relative contribution of these variance components indicates that the Kuwait examination is able to discriminate very well in the assessment of consultation competence of candidates. Unfortunately, comparing variances across studies with different designs is extremely difficult, if not impossible. Nevertheless, the small differences in marks independently allocated to individual candidates by the examiners were virtually identical to those achieved in assessments of clinical competence using the Objective Structured Long Examination Record (OSLER) [20], [21] and when using the LAP in a regulatory assessment in Hong Kong [22].

Although “the relative size of the separate sources of variation (variance) provide rich information in their own right, they can also be combined using equations … to express the extent to which the result reflects all possible measurements of the construct of interest. The result is a fraction between zero and one called the generalisability coefficient (G). It integrates the discriminating ability of the test and the reproducibility of the result. Essentially, it provides a measure of how confident you can be that any differences detected between assesses are real differences … simply because it takes account of all possible sources of error at the same time” [10]. By mathematically modelling G in different hypothetical scenarios, it is possible to estimate G with different numbers of observers and cases (analogous to the power calculation in an intervention trial) [10].

By convention, a regulatory assessment should achieve a reliability coefficient (G) of 0.8. The Kuwait examination is able to achieve this with a minimum of eight consultations and two examiners. It is likely that the following factors contributed to the achievement of this level of reliability:

• As a consequence of their systematic programme of preparation, all examiners were familiar with the content and application of the assessment package and the local context.

• The LAP contains explicit descriptors of performance linked to mark allocations to aid calibration in the allocation of marks by examiners (see Box 2).

These two measures have been demonstrated to reduce subjectivity and the potential for bias in examiners [23].

• The responses to the LAP questions provide examiners with major insights into the thinking processes underpinning the actions of the consulting doctors. This advantage is unique to the LAP.

• The inclusion of 10 different patient challenges increases both the reliability and validity of the assessments. Increasing the number of cases makes it more likely that examiners will be exposed to the true performance of the consulting doctor, and minimizes the impact of the well-recognized variations in physician performance which occur from case to case [24].

• The LAP contains seven categories of consultation competence and 39 component competences (see Box 1); the inclusion of so many explicit and different aspects of consultation competence also adds to increased validity and reliability [25].

Concerns about the reliability of assessments involving unstandardized “long cases” have focussed on three main influencing factors: “case specificity, examiner stringency and the aspects of a competence evaluated” [26]. Although these concerns have principally been addressed to long cases in a hospital setting, they equally apply in the setting of family practice. If all these areas are addressed, as they have been in the Kuwait clinical examinations, problems of reliability can be successfully overcome without resorting to less valid methods of assessing consultation competence, such as simulated surgeries [27], [28] and objective structured clinical examinations (OSCEs) [29]. Our study shows that reliable inferences can be achieved in assessments involving real (i.e., unstandardized) patients, providing adequate sampling is undertaken. This confirms previous research in a hospital setting [30] and provides further evidence that concerns over the reliability of long-case assessments can be overcome. Indeed, in terms of reliability, a recent review of the relevant evidence has concluded that “… for equal testing time, the long case turns out a bit better than the OSCE” [31].

There are also some well-documented concerns about the use of simulated patients and OSCE stations in assessments which render them less valid than direct observation of the global consultation competence of physicians in daily practice with real patients [15], [32]. Although both techniques can be reliable [15], OSCE stations usually test fragmented components of consultation competence [29], and short OSCE stations, in particular, risk “trivialisation of isolated elements of what should be a coherent whole” [15]. It has also been demonstrated that “simulated patients are limited in the number and complexity of the medical problems they can portray” [18]. Furthermore, “standardised patient assessments … under-emphasise important domains of professional competence”, such as “content of care” and “patient–physician relationships” [15]. There are also potential problems of feasibility in the use of simulated patients. For example, “[a]lthough few cases are needed to assess straightforward skills, up to 27 cases may be necessary to assess inter-personal skills reliably in high stakes examinations” [15]. This can be achieved with as few as eight patient encounters using the approach adopted in Kuwait. Accordingly, we would argue in favour of using the most authentic form of assessment, i.e., real-life consultations being directly observed and holistically judged against a sufficient number of relevant criteria.

The approach to consultation assessment in Kuwait is also feasible to conduct as each assessment requires approximately 3.5 hours×2 of examiner time per candidate, excluding travelling time. As the assessments are based on largely unselected real patients, time required for their development and maintenance is minimal compared with OSCE stations or simulated surgeries. Examiner training and briefing would be required for all three approaches.

Conclusion

The direct observation and assessment of a trainee family practitioner in consultation with 10 real patients by two independent examiners using the LAP has content validity, produces reliable results and is feasible to conduct. Accordingly, we would recommend the use of such an approach in the regulatory end-point assessment of the global consultation competence of trainees in family practice. This approach is more valid, and is likely to be more feasible, than simulated surgeries or short-case OSCE stations, although all three approaches can achieve satisfactory levels of reliability.

We gratefully thank all external and internal examiners for their cooperation in collecting the additional data required for this study, and Ron Hoogenboom for his contribution to the data processing.