Perceptions and experiences of trainers and trainees of UK workplace-based assessment for general practice licensing: a mixed methods survey

ABSTRACT

Background

Workplace-Based Assessment (WPBA) forms a major component of the UK General Practitioner (GP) licensing, together with knowledge and clinical skills examination. WPBA includes Case-based Discussion, Consultation Observation Tool, Mini-Consultation Exercise, Multisource Feedback, Patient Satisfaction Questionnaire, Clinical Examination and Procedural Skills, Clinical Supervisor’s Report, and Educational Supervisor Review. We aimed to investigate GP trainees’ and trainers’ perceptions and experiences of WPBA regarding validity and fairness.

Methods

We used a national online survey, with Likert-scaled and free-text responses, to a convenience sample of GP trainees and trainers, on perceptions and experiences of WPBA. Analysis included descriptive statistics, scale development, and regression models to investigate factors associated with attitudes towards WPBA, with thematic analysis of free text responses supported by NVivo 12.

Results

There were 2,088 responses from 1,176 trainees and 912 trainers. Both groups were generally positive towards WPBA, with trainers more positive or similar to trainees towards individual assessments. In a multivariable regression model, accounting for sex, ethnicity and country of primary medical qualification, trainees were significantly less positive (p < 0.001) while international medical graduates (IMGs) trained outside the European Economic Area (EEA) were significantly more (p < 0.001) positive towards WPBA. Qualitative analysis revealed varying concerns about validity and relevance, assessment burden, potential for bias, fairness to protected characteristics groups, gaps in assessment, and perceptions of individual assessments.

Discussion

Trainers’ greater positivity towards elements of WPBA accords with their role as assessors. Despite concerns about bias, IMGs from outside the EEA were significantly more positive towards WPBA.

KEYWORDS:

• Workplace-Based Assessment
• assessment
• licensing
• general practice
• cross-sectional study
• perceptions

Introduction

Workplace-Based Assessment (WPBA) forms a major component, together with the Applied Knowledge Test (AKT) and Recorded Consultation Assessment (RCA), of the Membership of the Royal College of General Practitioners (MRCGP) licensing assessment for general practitioners (GPs) in the United Kingdom (UK) [1].

WPBA uses a range of defined and individually validated assessment tools together with professional and triangulated judgements to evaluate trainees’ progress during the training programme [2]. WPBA includes formative elements including Case-Based Discussion (CBD), Consultation Observation Tool (COT), Mini-Consultation Exercise (miniCEX), Multisource Feedback (MSF), Patient Satisfaction Questionnaire (PSQ), Clinical Examination and Procedural Skills (CEPS), Clinical Supervisors Report (CSR), and Educational Supervisor Report (ESR) (see Box 1), which together with other requirements combine to form a summative Annual Review of Competence Progression (ARCP) during each year of GP training [2]. While the WPBA is long-established, little is known about its validity [3,4] and fairness in GP speciality training [5].

An assessment is valid when it measures what it purports to measure, in this case the competence of a GP trainee to practise effectively and safely. Successful achievement of WPBA contributes to a licence to work independently in general practice, so validity is of importance to trainees undertaking assessments, educators delivering them, and patients and their families, colleagues, employers and those affected directly or indirectly by a doctor’s clinical practice.

There are different types of test validity, summarised in the concept of construct validity which, according to Downing, is informed by multiple sources of evidence including the content, response process, internal structure, relationship to other variables and consequences of the assessment [6]. Various methods have been used to investigate the validity of different workplace-based assessments, including expert assessment of portfolios [7], factor analysis and expert consensus [8], randomised comparisons of different rating tools [9] and validity mapping using multiple sources of evidence [10].

Despite research investigating fairness of the MRCGP [5,11], perceptions of a lack of fairness of WPBA for different groups of trainees, particularly those in protected characteristic groups persist. The Equality Act 2010 [12] prohibits discrimination because of a protected characteristic such as age; disability; gender reassignment; marriage and civil partnership; pregnancy and maternity; race; religion and belief; sex; and sexual orientation.

Box 1. Formative components of WPBA.

WPBA	Definition and purpose
Audio Consultation Observation Tool (Audio-COT)	The Audio-COT uses the same methodology and process of completing the assessment as the COT (see below) but is used in a different setting. It provides an additional tool to assess trainees’ telephone consultation skills.
Case-based Discussion (CBD)	In CBD, trainees are expected to select a case for discussion and demonstrate how they demonstrated their capabilities in that case. Trainers are expected to review the case, establish what the trainee did in the case, and whether the case was of sufficient complexity.
Consultation Observation Tool (COT)	Trainees are expected to review the consultations in various settings (GP surgery, home visits, etc.). with their trainer, who grades their performance and provides them with feedback. It is an in-person assessment, in contrast to the Audio-COT (see above). This can be via video or directly observed.
Mini-Consultation Exercise (miniCEX)	A real-life observation of a patient-doctor consultation. The assessment critiques clinical skills, attitudes and behaviours. Assessments should be spread out evenly in the duration of the post and last no longer than 15 minutes.
Multisource Feedback (MSF)	MSF collects feedback from colleagues about trainees’ clinical performance and professional behaviour. The Educational Supervisor (ES) has access to the feedback once MSF closes.
Patient Satisfaction Questionnaire (PSQ)	Gives trainees feedback from patients on the extent to which they have shown empathy and sought to build a relationship with patients.
Clinical Examination and Procedural Skills (CEPS)	Trainers must be satisfied through observed evidence or documented evidence from others that trainees are generally competent. Trainees are assessed on five intimate examinations and a range of non-intimate examinations.
Clinical Supervisors Report (CSR)	This is a report that is completed by the trainee’s clinical supervisor in each post. Trainees receive feedback about their performance.
Educational Supervisor Report (ESR)	These reports provide feedback based on evidence within the portfolio on performance within the 13 capabilities and identifies areas for further improvement.

One way of assessing content validity of WPBA and fairness for licensure is to survey those undertaking and administering the assessment to determine stakeholder acceptability [13]. Our study aimed to investigate GP trainees’ and trainers’ perceptions and experiences of the validity and fairness of individual assessments and WPBA more generally.

Methods

Design

We used a cross-sectional mixed methods survey design, employing self-administered online questionnaires using the JISC (https://beta.jisc.ac.uk/online-surveys) platform. A survey was used to be an efficient method of data collection but included quantitative and qualitative elements to allow triangulation through integration.

Study population

We surveyed GP trainees and trainers.

Data collection

The GP trainee and trainer questionnaires included: demographic data (gender, ethnicity, place of primary medical qualification (UK, European Economic Area [EEA], Rest of the World), disability, UK nation (England, Northern Ireland, Scotland, Wales); rurality; and full-time versus less than full-time [LTFT] training). We distinguished International Medical Graduates from the EEA and Rest of the World because a proportion of UK nationals train in the EEA.

The questionnaire was designed by active GP trainers based on their experience of trainer and trainee engagement with the WPBA to provide feedback on the new elements of the WPBA and its workings. It included information on engagement with the portfolio, such as time spent each week on portfolio, and time spent on the Educational Supervisor Report (ESR), and which Care Assessment Tools (CATs) had been used. Potential participants were contacted through UK GP training schemes. The questionnaire was piloted on an initial 100 respondents and was revised slightly following this initial pilot. An email was circulated with a link to the survey around the Committee of General Practice Education Directors (COGPED), with a request to circulate it to their trainee and trainer networks. A reminder email was sent. Links were added to the trainee and trainer surveys as a banner at the top of the trainee and Educational Supervisor (ES) portfolio approximately one month after the survey was launched.

There were free-text questions specifically on whether a participant thought that a trainee could be disadvantaged in any area of WPBA because of their protected characteristics, and whether there were any areas of general practice that the current package of assessments did not adequately address.

Finally, participants were asked to indicate the extent to which they agreed or disagreed, on a 5-point Likert scale from strongly disagree ( =1) to strongly agree ( =5) with neither agree or disagree ( =3), that each assessment was valuable in developing a trainee’s learning and contributing to a global judgement on a trainee’s performance, and any other (free-text) comments were invited.

Data analysis

Quantitative analysis, using SPSS 28 and Stata 15, included descriptive statistics, scale development measuring attitudes towards WPBA and multivariable linear regression. This analyses the relationship between at least two independent variables (demographic characteristics) and the dependent variable, positive attitudes to WPBA, such that the association between each take into account other independent variables. We undertook qualitative analysis of free-text responses, supported by NVivo 12, with codes identified and decided by two researchers (VP, NS) to reduce risk of bias, and subsequently agreed and confirmed by all the authors [14].

Results

As of May 2022, there were 15,383 RCGP Associate in Training (AIT) RCGP members, the RCGP identifier for GP trainees. In 2020, there were approximately 8,500 registered GP trainers.

Respondent characteristics

There were 2,088 responses from 1,176 trainees (approximately 7.6% based on the 2022 figures above) and 912 trainers (approximately 10.7% based on the 2020 figures above). The two groups were similar in sex, but trainers were more likely to be White British and Irish, and less likely to be Asian and Black, compared with GP trainees. They were more likely to be UK trained, work in an urban practice, from a UK nation (England, Northern Ireland, Scotland, Wales), and work less than full time (Table 1).

Table 1. Respondent characteristics.

		GP trainees		GP trainers
		N	%*	N	%	Chi square
Gender	Female	701	(60.0)	539	(59.6)	0.45
	Male	435	(37.2)	343	(37.7)
	Other	0	(0.0)	2	(0.2)
	Prefer not to say	33	(2.8)	26	(2.9)
Ethnicity
	White British or Irish	333	(34.6)	629	(65.4)
	White Other	55	(4.7)	39	(4.3)
	Asian	406	(34.5)	178	(19.5)	<0.001
	Black African Caribbean	232	(19.7)	8	(0.9)
	Mixed/Other	90	(7.7)	21	(2.3)
	Prefer not to say	60	(5.1)	36	(4.0)
Country of Primary Medical Qualification
	UK	466	(39.6)	771	(62.3)
	EEA	65	(5.5)	25	(2.8)	<0.001
	Rest of the world	618	(52.6)	709	(34.1)
	Prefer not to say	27	(2.3)	14	(1.6)
Practice type
	Urban	523	(44.5)	506	(55.5)
	Rural	154	(13.1)	109	(12.0)	<0.001
	Urban/Rural mixed	459	(39.0)	285	(31.3)
	Prefer not to say	40	(3.4)	12	(1.3)
UK nation of training
	England	973	(82.9)	683	(75.6)
	Scotland	113	(9.6)	130	(14.4)	0.001
	Wales	48	(4.1)	51	(5.6)
	Northern Ireland	30	(2.6)	33	(3.7)
	Prefer not to say
Full time or less than full time
	Full time	865	(73.7)	384	(42.4)
	Less than full time	297	(25.3)	510	(56.4)	<0.001
	Prefer not to say	12	(1.0)	11	(1.2)

*Percentage of those responding.

Time spent on educational portfolio or completing the educational supervisor review (ESR) report

Overall, GP trainees spent significantly more time engaging with the portfolio than trainers but less time to complete an ESR report (Table 2).

Table 2. Time spent engaging with the educational portfolio or completing the educational supervisor review report.

		GP trainees		GP trainers
		Number	Percent	Number	Percent	Chi square
Time engaging with portfolio (hours per week)	0–1	199	(16.9)	273	(29.9)
	1–2	450	(38.3)	473	(51.9)
	2–3	299	(25.4)	124	(13.6)	<0.001
	3–4	131	(6.3)	31	(1.5)
	>4	97	(8.2)	11	(1.2)
Time to complete an ESR report (hours)
	0–1	155	(13.2)	8	(0.9)
	1–2	351	(29.8)	184	(20.2)
	2–3	268	(22.8)	334	(36.6)	<0.001
	3–4	181	(15.4)	247	(27.1)
	>4	221	(18.8)	139	(15.2)

How trainees valued the individual assessments

Trainers and trainees generally agreed that the WPBA developed trainees’ learning and contributed to a global judgement on performance, with all mean ratings exceeding the midpoint of 3 on a range of 1 to 5 (Table 3). The Consultation Observation Tool (COT)/Audio Consultation Observation Tool (Audio-COT) was rated highest (mean 4.21) and Leadership Multi-Source Feedback (Leadership MSF) lowest (mean 2.78).

Table 3. Item statistics for scaled responses.

	Mean	Std. Deviation
The Care Assessment Tool (CAT) would be an appropriate assessment of ability for ST1/2 in General Practice	3.47	1.39
Please indicate the extent to which you agree or disagree that the Patient Satisfaction Questionnaire (PSQ) contributes to a global judgement on your performance	3.41	1.60
Please indicate the extent to which you agree or disagree that the Multi-Source Feedback (MSF) contributes to a global judgement on your performance	4.17	0.95
Please indicate the extent to which you agree or disagree that the Leadership Multi-Source Feedback (Leadership MSF) contributes to a global judgement on your performance	2.78	1.63
Please indicate the extent to which you agree or disagree that the Quality Improvement Project (QIP) contributes to a global judgement on your performance	3.52	1.13
Please indicate the extent to which you agree or disagree that the Care Assessment Tool (CAT) contributes to a global judgement on your performance	3.69	1.43
Please indicate the extent to which you agree or disagree that the Consultation Observation Tool (COT)/Audio Consultation Observation Tool (Audio-COT) contributes to a global judgement on your performance	4.21	1.18
Please indicate the extent to which you agree or disagree that the Learning Event Analysis (LEA) contributes to a global judgement on your performance	3.88	1.02
Please indicate the extent to which you agree or disagree that the Placement planning meeting contributes to a global judgement on your performance	3.64	1.01
Please indicate the extent to which you agree or disagree that the Leadership Activity contributes to a global judgement on your performance	3.26	1.36
Please indicate the extent to which you agree or disagree that the Quality Improvement Activity (QIA) contributes to a global judgement on your performance	3.62	1.08
Please indicate the extent to which you agree or disagree that the Supporting Documentation contributes to a global judgement on your performance	3.72	0.96
The assessment of performance statements are helpful to determine progress and development needs	3.81	0.87
I find a global grade judgement on individual assessments helpful	3.82	0.89
I find it useful in assessing trainee progression to have a judgement of level of required supervision within the Clinical Supervisor Report (CSR)	3.92	0.94
The different learning log options provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement within the Educational Supervisor Report (ESR)	3.78	0.88
The whole package of assessment tools provides sufficient information to allow an Educational Supervisor (ES) to develop an overall judgement within the Educational Supervisor Report (ESR)	3.84	0.87
The format of the clinical case reviews supports effective reflection on clinical encounters using the ‘what will I maintain, improve and stop’ format	3.58	1.02
A broad range of Care Assessment Tools (CATs) gives a more accurate assessment of ability than using Case-based Discussions (CbDs) alone	3.58	1.28
The leadership activity, Leadership Multi-source Feedback (Leadership MSF) and the leadership, management and professionalism learning log allows a trainee to demonstrate leadership skills	3.02	1.49
The Quality Improvement Project (QIP)/Quality Improvement Activity (QIA) helps understand the methods behind quality improvement theory and adds value to development	3.56	1.05
The overall range of WPBA facilitates early identification of developmental needs	3.58	.97
The nine Clinical Experience Groups are a true representation of the spectrum and range of patients in general practice	3.57	.97
The current range of WPBA provides sufficient opportunity to detail performance against the capabilities	3.75	.87
The Trainee Portfolio is user friendly	3.59	1.09
There is sufficient information about the requirements for WPBA on the RCGP website (including the WPBA handbook)	3.74	.99

Both trainers and trainees were positive towards all of the individual assessments, with the former being more so than the latter (Table 4). We constructed a scale, combining Likert scaled responses to all the individual assessments and this showed good reliability (Cronbach’s alpha 0.91), with alpha remaining above 0.9 when any item was removed. The assessments represented the independent variable in a multivariable regression model accounting for sex, ethnicity and country of primary medical qualification. Trainees were significantly less positive than trainers (p < 0.001) while IMGs trained outside the EEA were significantly more positive than UK graduates (p < 0.001) towards WPBA (Table 5).

Table 4. Comparing trainers’ and trainees’ views of assessment tools.

	Trainers			Trainees
	Number agreeing*	% Agreeing	Total	Number agreeing	% Agreeing	Total
The Care Assessment Tool (CAT) would be an appropriate assessment of ability for ST1/2 in General Practice	683	74.9%	912	742	63.3%	1,173
Please indicate the extent to which you agree or disagree that the Patient Satisfaction Questionnaire (PSQ) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	743	81.5%	912	653	55.6%	1,174
Please indicate the extent to which you agree or disagree that the Multi-Source Feedback (MSF) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	853	93.5%	912	938	79.9%	1,174
Please indicate the extent to which you agree or disagree that the Leadership Multi-Source Feedback (Leadership MSF) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	497	54.5%	912	443	37.7%	1,174
Please indicate the extent to which you agree or disagree that the Quality Improvement Project (QIP) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	670	73.5%	912	661	56.3%	1,174
Please indicate the extent to which you agree or disagree that the Care Assessment Tool (CAT) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	777	85.2%	912	824	70.2%	1,174
Please indicate the extent to which you agree or disagree that the Consultation Observation Tool (COT)/Audio Consultation Observation Tool (Audio-COT) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	873	95.7%	912	966	82.3%	1,174
Please indicate the extent to which you agree or disagree that Learning Event Analysis (LEA) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	848	93.0%	912	768	65.4%	1,174
Please indicate the extent to which you agree or disagree that Placement planning meeting is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	568	62.3%	912	804	68.5%	1,174
Please indicate the extent to which you agree or disagree that Leadership Activity is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	609	66.8%	912	804	68.5%	1,174
Please indicate the extent to which you agree or disagree that Quality Improvement Activity (QIA) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	737	80.8%	912	604	51.4%	1,174
Please indicate the extent to which you agree or disagree that Clinical Case Reviews are valuable in developing a trainee’s learning and contributing to a global judgement on your performance	863	94.6%	912	965	82.2%	1,174
Please indicate the extent to which you agree or disagree that Supporting Documentation are valuable in developing a trainee’s learning and contributing to a global judgement on your performance	615	67.4%	912	818	69.7%	1,174
The assessment of performance statements (i.e. where I am asked to rate the trainee’s overall competence at which they have shown they are performing) are helpful to determine a trainee’s progress and development needs	744	81.6%	912	871	74.2%	1,174
I find it helpful being required to make a global grade judgement on individual assessments	712	78.1%	912	885	75.4%	1,174
The different learning log options provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement of the trainee within the Educational Supervisor Report (ESR)	706	77.4%	912	870	74.1%	1,174
The different assessment tools provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement of the trainee within the Educational Supervisor Report (ESR)	754	82.7%	912	883	75.2%	1,174
The format of the clinical case reviews supports effective reflection on clinical encounters using the ‘what will I maintain, improve and stop’ format	674	73.9%	912	722	61.5%	1,174
A broad range of Care Assessment Tools (CATs) gives a more accurate assessment of a trainee’s ability than using Case-based Discussions (CbDs) alone	678	74.3%	912	732	62.4%	1,174
The leadership activity, Leadership Multi-source Feedback (Leadership MSF) and the leadership, management and professionalism learning log allows a trainee to demonstrate leadership skills	519	56.9%	912	555	47.3%	1,174
The Quality Improvement Project (QIP)/Quality Improvement Activity (QIA) helps a trainee to understand the methods behind quality improvement theory and adds value to their development	692	75.9%	912	707	60.2%	1,174
The overall range of WPBA facilitates early identification of a trainee’s developmental needs	599	65.7%	912	775	66.0%	1,174
The nine Clinical Experience Groups are a true representation of the spectrum and range of patients in general practice	577	63.3%	912	776	66.1%	1,174
The current range of WPBA provides sufficient opportunity to detail a trainee’s performance against the capabilities	727	79.7%	912	837	71.3%	1,174
The Trainee Portfolio is user friendly	573	62.8%	912	805	68.6%	1,174
There is sufficient information about the requirements for WPBA on the RCGP website (including the WPBA handbook)	719	78.8%	912	841	71.6%	1,174
There is sufficient information within an interim Educational Supervisor Report (iESR), without having to rate the trainee against each capability, to determine a global judgement of the trainee	688	75.4%	912	883	75.2%	1,174

*Agree or strongly agree.

Table 5. Multivariable regression model showing respondent attributes associated with a positive attitude towards WPBA.

		B	95% confidence interval		P-value
Trainer/Trainee	Trainer	0 (Reference)
	Trainee	−0.48	−0.54	−0.42	<0.001
Sex	Female	0
	Male	−0.15	−0.068	0.038	0.57
	Other	−0.72	−1.54	−0.096	0.08
	Prefer not to say	−0.24	−0.45	−0.036	0.02
Ethnic group	White British Irish	0
	White other	0.13	−0.27	0.010	0.07
	Asian	−0.0071	−0.088	0.073	0.86
	Black	0.062	−0.052	0.17	0.29
	Mixed and other	−0.13	−0.26	−0.001	0.05
	Prefer not to say	−0.29	−0.45	−0.13	<0.001
Country of Primary Medical Qualification	UK	0
	European Economic Area	0.13	−0.12	0.27	0.07
	Rest of the World	0.29	0.20	0.37	<0.001
	Prefer not to say	0.23	−0.0048	0.46	0.06

Qualitative findings

Of the 912 trainer respondents, there were 360 free-text responses elaborating on how trainees with protected characteristics might be disadvantaged and 316 free-text responses on gaps not currently addressed by the assessments. Of the 1,176 trainee respondents, the corresponding figures were 336 and 165 respectively. Qualitative analysis revealed that while trainees and trainers were generally positive about the WPBA, there were concerns about its implementation. These concerns focused on: the validity, relevance, gaps and areas for improvement in WPBA; the assessment burden and support available; fairness of WPBA; and perceptions of individual assessments. These are discussed below and summarised in Appendix 1.

Validity, relevance, gaps and areas for improvement

Familiarity in medical training assessing routinely activities and progress

Many respondents felt that WPBA was familiar to trainees and assessed activities that they already routinely undertook and their progress.

The WPBA assesses activities that the trainee would routinely perform anyway.

(Respondent 218, Trainee, Black African Female, No disability, Rest of the World graduate, LTFT)

I think, it gives some idea how a trainee is developing skills.

(Respondent 495, Trainer, Pakistani Female, No disability, Rest of the World graduate, LTFT)

Inconsistent ratings and subjectivity may mean E-portfolio does not reflect a trainee’s true abilities

Others felt that the eportfolio did not always represent a trainee’s capabilities, because of varying supervisor attitudes and subjectivity, inconsistency in rating, and the effect of the trainer-trainee relationship.

A trainee may be good at what they do but may find it hard to reflect in eportfolio thus not an actual reflection of their abilities.

(Respondent 354, Trainee, Black African Female, Rest of the World graduate, FT)

Gaps in assessing specific clinical areas, consultation settings, leadership, management, and empathy

Out-of-hours (OOH) working is routine for many GPs, as are home visits. Since COVID-19, remote consultation, via video or telephone, has become increasingly important.

Out of hours shifts can be difficult to access yet it remains mandatory which doesn’t feel like we are supported. Also, I don’t see the reason for creating separate capabilities for OOH entries.

(Respondent 798, Trainee, Black African Male, No disability, Rest of the World graduate, FT)

Remote consultation is a cornerstone of modern post-covid general practice. You should recommend a minimum of at least one or two audio CEPS to assess trainees’ ability to do telephone triages etc.

(Respondent 646, Trainee, White British Male, No disability, UK graduate, FT)

Some respondents felt that the WPBA did not adequately assess leadership and management skills. This could be important for those who want to set up their own practice.

Leadership and management, health inequalities, planetary health, supervising additional roles.

(Respondent 593, Trainee, White British Male, No disability, UK graduate, LTFT)

Practice management, finances and contracting obligations e.g. QOF, IIF.

(Respondent 191, Trainee, White British F, No disability, UK graduate, LTFT)

Managing workload was a key aspect of being a GP which some felt was not adequately assessed.

Overall, during training, we are not assessed on consultation technique, how to manage a clinic, managing workload, how to manage a ‘duty’ shift. This is the most pressing issue facing general practice today yet is not assessed at all.

(Respondent 558, Trainee, Asian Other Male, UK, FT)

Patient-doctor communication was felt to be important, in addition to the clinical skills that GPs had to demonstrate.

Empathy, compassion (for ourselves, and for others)…all the immeasurable things that get ignored and diminished because they can’t be quantified.

(Respondent 1116, Trainee, White British Female, No disability, UK graduate, FT)

Importance of feedback

Supervised consultations and feedback were felt to be vital formative aspects of WPBA:

Not enough supervised consultations - different being recorded and watching later on video screen to having instantaneous feedback. Can be difficult to access supervisor for this and means you do not immediately capitalise on feedback.

(Respondent 996, Trainee, White British Female, Physical disabilities, UK graduate, LTFT)

Assessment burden and support

Insufficient time to complete WPBA reduces time for learning

Some respondents felt that they had insufficient time to complete the WPBA, within the context of their other GP duties, which reduced time for learning.

Yes, because sometimes while in GP surgery we don’t get enough time to complete WBPA.

(Respondent 768, Trainee, Pakistani Female, No disability, Rest of the World graduate, FT)

If LTFT, there is less opportunity for patient exposure. Educational time is reduced and often more taken up by formal educational activities instead of time to engage with portfolio and complete WPBAs. Nevertheless, an ESR still needs to be completed six monthly with a high administrative burden.

(Respondent 1016, Trainee, Mixed Ethnicity, Female, No disability, UK graduate, LTFT)

Stress of WPBA assessments

The pressure to complete WPBA within a defined timeframe, especially within the context of other clinical demands, contributed to stress for trainees.

There is a lot of stress in completing required number of CBDs and COTs before each appraisal. This pressure does not help with learning and progressing instead trainees should be given adequate time to prepare for exams.

(Respondent 376, Trainee, Specific learning, other details prefer not to say)

External factors affecting assessment

Trainee performance might be affected by external work or personal factors, such as a busy shift contributing to fatigue.

Sometimes, external factors govern how a trainee would perform on a particular day. Just for an example: I had a very hectic shift and the CBD following it had been much low in feedback than the other 8 CBDs I had.

(Respondent 874, Trainee, Pakistani Male, Rest of the World graduate, FT)

Support available

These burdens were felt to be mitigated by the support available from educators and the work setting.

As far as I have observed, there is a lot of support available around to basically cover the areas.

(Respondent 862, Trainee, Pakistani Female, Rest of the World graduate, FT)

Fairness of work-place based assessment

Respondents varied in their perceptions of fairness of WPBA with some feeling that the WPBA was inherently fair, including to those with protected characteristics, while others disagreed.

Importance of good trainer-trainee relationships to how assessments are conducted

Positive trainee-trainer relationships were seen as critical to fairness, with some trainees even expressing reluctance to be assessed if an assessor was perceived as being unfair.

Achieving WPBAs rely entirely on positive relationships between trainer and trainee. If there are any cultural/physical barriers that exist between trainer and trainee then barriers also exist to getting sign-offs.

(Respondent 115, Trainee, White Other Male, Disability: prefer not to say, UK graduate, FT)

Hesitation to approach colleagues for assessment. Being evaluated differently from others.

(Respondent 994, Trainee, Pakistani Female, No disability, Rest of the World graduate, FT)

Varying perceptions of fairness and discrimination for those with protected characteristics

Many respondents felt that the WPBA was fair, including those trainees with protected characteristics.

I believe everyone is given equal opportunities to learn and participate. There could be small hurdles (e.g. post-pregnancy, if you have small children) but that could be overcome by the support from the rest of the team members.

(Respondent 797, Trainee, Pakistani Female, No disability, Rest of the World graduate, FT)

While it is easy to see how the process could become used as a tool to victimise candidates, I believe my colleagues to be above board and believe the assessments given to be fair minded.

(Respondent 367, Trainee, Black African Male, No disability, Rest of the World graduate, FT)

Others were aware that WPBA could discriminate against those with protected characteristics, and trainers could help support these individuals to successfully complete WPBA.

Yes. Although this [discrimination] has not happened to me personally, I have had colleagues who have complained of this.

(Respondent 600, Trainee, Asian Other Female, Disability unknown, No Rest of the World graduate, FT)

Some patients will discriminate based on protected characteristics positively or negatively.

(Respondent 859, Trainer, White British Female, More than one disability, UK graduate, FT)

They should not be disadvantaged. It is important as a trainer to support trainees individually depending on any protected characteristics they may hold.

(Respondent 323, Trainer, White British Female, No disability, UK graduate, LTFT)

Unconscious and conscious bias from assessors and patients

Some trainees and trainers felt that the WPBA had potential for bias and subjectivity from both assessors and patients.

All people are subject to prejudice, with some assessors doing so to a greater extent than others. This question is very vague and general.

(Respondent 27, Trainee, White British Male, UK graduate, FT)

Some respondents felt patients sometimes showed prejudice towards certain trainees with protected characteristics.

Patients may interact differently with them due to any of their characteristics, and this may not be understood by their trainer.

(Respondent 593, Trainee, White British Male, No disability, UK graduate, LTFT)

Some patients can discriminate against the trainee.

(Respondent 1006, Trainee, White – Irish Female, No disability, UK graduate, FT)

Within each protected characteristic, respondents raised concerns about potential discrimination. For example, female patients requesting female GPs may limit access to learning opportunities for male colleagues, those on maternity leave may need additional time to adjust. Discrimination may be overt or hidden, for instance, trainees returning from maternity leave may need more lead-in time before undertaking, and extra time to complete, the WPBA.

An assessor may not feel at ease with someone older than themselves.

(Respondent 361, Trainee, White British Male, No disability, EEA graduate, FT)

If they come back from maternity leave or any form of break, they may need time to adjust before assessments.

(Respondent 683, Trainee, Asian Other Female, Specific learning difficulty, EEA graduate, FT)

Disadvantages due to disability and importance of reasonable adjustments

There were many different ways that trainees with a disability could be disadvantaged. Those with learning disabilities can take longer to complete the WPBA. Reasonable adjustments could mitigate these.

It generally takes me longer to complete e-learning and portfolio tasks than my non-dyslexic colleagues, and last-minute shift coverage, professional exams, life stressors and high levels of professional burnout are likely to highly disadvantage anyone who has any form of disability.

(Respondent 1115, Trainee, White British Female, No disability, UK graduate, FT)

Race, religion, ethnicity, culture, international medical graduate status

There was a perception that race, religion, culture, and IMG status might disadvantage trainees or increase the risk of discrimination when undertaking the WPBA.

Some trainees experienced direct or indirect racial discrimination during WPBA. For example, some patients with whom trainees interacted were felt to discriminate on racial or ethnic grounds. There was also a perception of discrimination by some assessors.

There is the possibility of discrimination - based on who the assessor is.

(Respondent 512, Trainee, White British Female, No disability, UK graduate, FT)

Language, while not a specific protected characteristic, is linked to race, ethnicity, and culture. Communication issues may relate to being a non-native English speaker. Culture could also be related as some trainees, such as IMGs, may not be used to how the NHS worked or be familiar with English colloquialisms.

I think had you come from a different training background e.g., IMG and were not so familiar with this way of learning/assessing then you could be disadvantaged/find it quite demoralising or challenging.

(Respondent 608, Trainee, White British Female, No disability, LTFT)

I think non-UKs can be disadvantaged through a lack of understanding of the NHS system, and differences in cultural factors which influence health.

(Respondent 603, Trainee, Indian Female, No disability, UK graduate, FT)

Perceptions of individual assessments

There was detailed feedback on individual assessments, which generally related to at least one of the issues described above.

The care assessment tool (CAT)

CATs were viewed as offering a less stressful, but more comprehensive trainee assessment. For trainees with a disability, CATs were seen to offer better choice than CBDs alone.

I think the CATs offer options for a more comprehensive assessment of the trainee without increasing the stress levels for the trainee. As the CATs are not mandatory each. If any trainee has a disability that prevents them from achieving a particular CAT, they can just avoid it and do another one instead. But the choice I think it’s good, better than just the CBDs.

(Respondent 1026, Trainee, White Other Female, No disability, EEA graduate, LTFT)

Consultation observation tool (Cot)/audio consultation observation tool (audio-COT)

While there was strong support for the COT, some trainees felt that it could disadvantage some trainees with language barriers.

COT assessment is occasionally challenging due to language/accent difficulties. Also very examiner dependent and may introduce individual biases.

(Respondent 912, Trainee, Black African Female, No disability, Rest of the World graduate, FT)

While there was strong support for the COT, some trainers expressed concerns about its impact on those with language barriers in terms of stress, tone, and speech inflection. Linguistic nuance can be a particular problem for non-native English speakers who were not used to the complexities of British culture.

Assessing, for instance a COT, often involves assessing consultation skills. The nuances of the English language is such that if you didn’t train/grow up learning these nuances, it can be disadvantageous to you.

(Respondent 42, Trainer, White British Male, No disability, UK graduate, LTFT)

Issues with audio COT for those for whom English is not their first language such as stress, tone, and inflection in speech.

(Respondent 162, Trainer, White British Male, No disability, UK graduate, LTFT)

Multi-source feedback (MSF)

Most trainees felt that leadership MSF enabled them to demonstrate leadership, while some felt that the MSF did not truly assess trainees’ leadership skills. Leadership MSF was considered challenging for trainees with protected characteristics around issues, such as competing workloads and race. This could impact upon how they are assessed.

Leadership - the MSF doesn’t truly assess leadership.

(Respondent 243, Trainee, White British, Female, No disability, UK graduate, LTFT)

MSF is challenging with LTFT working (proxy for illness/maternity) where there are more contentious issues around working hours/proportionate workload and assessment which can cause issues within team and appear as reduced availability rather than a fair reflection of function appropriate to individual circumstances. It is more prone to stigma.

(Respondent 264, Trainee, White British Female, No disability, UK graduate, LTFT)

The MSF can be used against a trainee if the colleagues do not agree with a protected characteristic. This could be things like race for example where negative feedback can be given simply because the trainee happens to be from a different background.

(Respondent 585, Trainee, Pakistani Male, No disability, Rest of the World graduate, FT)

Exemplifying the strong support for the MSF in developing a trainee’s learning and contributing to a global judgement on their performance, some trainers suggested embedding it into placements longer than three months.

MSF should be done in every placement that is over three months long.

(Respondent 809, Trainer, White other Female, No disability, EEA graduate, LTFT)

Educational supervisor report (ESR)

While trainees felt that different learning logs provided sufficient evidence to trainers in their overall judgement of them within the ESR, they were considered to be a poor reflection of clinical competence in GPs. The full ESR was certainly considered administratively burdensome by this trainer and trainee respondent. For trainees who are LTFT, there is greater time pressure given that they work fewer hours. In working fewer hours, doing the WPBA means that there is less time for patient contact. There is also less time for educational activities that are crucial to career progression.

All of the case-based discussions/learning logs are a poor reflection of actually clinical competence as a medical practitioner.

(Respondent 1060, Trainee, White British, Male, No disability, UK graduate, FT)

I think there is far too much emphasis on hoops like leadership QIP, MSFs, QIP, QIA etc., etc., leaving too little time to focus on strategies for managing work in a failing system. Although the ESR is not time consuming, a lot of time is spent by trainee and trainer trying to ensure that all these extras are covered. It puts me off training and discourages GP trainees at a time when we should be doing all we can to support new recruits. Need to know they have good knowledge (AKT) and consulting skills, and a few CATS, not all the other stuff.

(Respondent 478, Trainer, White British Female, Hearing impairment, UK graduate, LTFT)

Trainee portfolio

Trainees accepted that colleagues with protected characteristics might find the portfolio challenging. The eportfolio requires a lot of writing and reading, which may disadvantage trainees with learning disabilities. The nature and extent of the disadvantage may depend on how engaged the supervisor is.

Dyslexia - difficult as there is a lot of writing and reading on the eportfolio.

(Respondent 223, Trainee, Asian Other Female, No disability, UK graduate, FT)

People from immigrant backgrounds have little to no experience with the UK portfolio system. Chasing supervisors, senior doctors - consultants and registrars for signatures and write ups are things new to them. As such they may seem deficient in things they really are very proficient in.

(Respondent 468, Trainee, Black African Male, No disability, Rest of the World graduate, FT)

I imagine that the ‘overhead’ of ePortfolio work disproportionately affects those working LTFT and those who don’t have an engaged or organised supervisor (or who can’t connect well with them). And I imagine those with protected characteristics are disproportionately represented in those groups.

(Respondent 1116, Trainee, White British Female, No disability, UK graduate, FT)

While trainers agreed that the Trainee Portfolio was user-friendly, they acknowledged that trainees with protected characteristics, especially those with learning disabilities, could find it problematic.

EPortfolio, as a whole, favours certain personality and learning types. This further exacerbates differences in cultural training and learning experiences.

(Respondent 676, Trainer, White British, Male, No disability, UK graduate, FT)

My trainee has been unable to do any eportfolio work due to mental health problems.

(Respondent 138, Trainer, White British Female, No disability, UK graduate, LTFT)

Dyslexia. IMGs-need prior training as to the significance of e-portfolio.

(Respondent 498, Trainer, White British Female, No disability, UK graduate, LTFT)

Discussion

Main findings

Trainers and trainees were generally positive towards most elements of WPBA. Trainers were more, or as positive, as trainees for all the assessments, which may have reflected their role as assessors. Trainees reported problems that they faced in undertaking the assessments. Despite concerns about bias, IMGs trained outside the EEA were significantly more positive towards WPBA and there were no significant differences by ethnicity.

Comparison with other literature

The positive attitudes towards WPBA suggests that trainers and trainees value this form of assessment in capturing the complexities of general practice [15]. The variation in response and negative comments from some trainees, that WPBA did not reflect actual performance, has been found in previous studies in other specialities, for example in a survey of paediatric trainees [16].

The finding that certain assessments were rated more highly (e.g. COT and audio-COT) by respondents than others (e.g. the leadership MSF), builds on previous studies comparing stakeholder perceptions [17], and offers opportunities to develop or modify tools or training in their rationale and use. This has to be balanced against previous findings that the MSF from colleagues and patient feedback on consultations were tools that were likely to provide a reliable opinion of workplace performance [18]. The MSF, in the right context and with appropriate feedback, has been shown to be likely to improve performance [19] or lead to perceived positive effects on practice [20].

Survey studies of practising medical staff in the United States [21] as well as survey [22] and qualitative studies of doctors in training in the UK [23] suggest widespread perceptions of direct and indirect discrimination against certain groups, such as women, IMGs and minority ethnic doctors. These perceptions, although present in the free-text responses to this survey, varied between trainees and trainers. IMGs trained outside the EEA were significantly more positive to WPBA than UK graduates, with no significant differences between women and men or between ethnic minority and White British Irish respondents. This is despite IMGs performing significantly less well on the ARCP [5].

Areas for improvement included better feedback, which is an important aspect of WPBA as a formative assessment and educational tool [24]. In a qualitative study in The Netherlands, the authors expressed that the content of feedback, how it is provided, and how it is incorporated in trainees’ learning are vital to its success [25].

Implications for educational policy, practice and research

WPBA is an important component of assessment of GP trainees and contributing to both formative with feedback and summative judgements of performance. Further development and refinement of WPBA may be seen as a path towards programmatic assessment [26].

Of particular importance to trainees were issues around time needed to complete the WPBA, as well as potential patient and assessor bias, which may disadvantage trainees with protected characteristics. Respondents felt that the WPBA should closely replicate the realities of being a GP, e.g. out-of-hours work, remote consultations, interpersonal skills or practice management.

Why differences exist between trainees and trainees, and between assessments need to be explored to find ways of narrowing these differences, either by modifying assessments or the way they are administered. Providing training for both on the different assessment types, the reasons for including them, and their reliability and validity could also narrow these differences. Allowing extra time to complete the WPBA could mitigate disadvantages faced by some trainees with protected characteristics. Further research should explore WPBA using qualitative methods such as interviews or focus groups and also examine the reliability of WPBA assessments individually and overall, particularly where changes in assessment tools are being implemented.

Strengths and limitations

The questionnaire was designed and administered without prior qualitative work or psychometric evaluation, but it showed high validity, for example, a high response and low levels of missing data. There were many detailed responses to free-text questions which allowed triangulation with quantitative findings. This was an anonymised online survey so we could not determine the true response rate, but there was good response from both GP trainees and trainers with a broad spread of demographic characteristics which allowed us to compare responses from different groups.

Conclusion

This study shows that trainers and trainees were positive towards most aspects of WPBA, and contrary to concerns of bias, there were no differences by sex or ethnicity and, indeed, IMGs from outside the EEA were significantly more positive towards WPBA. Further work needs to be conducted to examine the reliability of WPBA assessments individually and overall.

Abbreviations

AKT	=	Applied Knowledge Test
ARCP	=	Annual Review of Competence Progression
Audio-COT	=	Audio Consultation Observation Tool
CBD	=	Case-Based Discussions
CEPS	=	Clinical Examination and Procedural Skills
COT	=	Consultation Observation Tool
CSR	=	Clinical Supervisors Report
ESR	=	Educational Supervisor Report
IMG	=	International Medical Graduate
MiniCEX	=	Mini Consultation Exercise
MRCGP	=	Membership of the Royal College of General Practitioners
MSF	=	Multi-Source Feedback
PSQ	=	Patient Satisfaction Questionnaire
RCA	=	Recorded Consultation Assessment
WPBA	=	Workplace-Based Assessment

Ethical approval

The University of Lincoln granted ethical approval to analyse anonymised data: Lincoln Ethical Approval System (LEAS) reference 13,903.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14739879.2024.2379525

Additional information

Funding

AS, KE and TA are funded by the Royal College of General Practitioners.