Perceptions and experiences of trainers and trainees of UK workplace-based assessment for general practice licensing: a mixed methods survey

Figures & data

Box 1. Formative components of WPBA.

WPBA	Definition and purpose
Audio Consultation Observation Tool (Audio-COT)	The Audio-COT uses the same methodology and process of completing the assessment as the COT (see below) but is used in a different setting. It provides an additional tool to assess trainees’ telephone consultation skills.
Case-based Discussion (CBD)	In CBD, trainees are expected to select a case for discussion and demonstrate how they demonstrated their capabilities in that case. Trainers are expected to review the case, establish what the trainee did in the case, and whether the case was of sufficient complexity.
Consultation Observation Tool (COT)	Trainees are expected to review the consultations in various settings (GP surgery, home visits, etc.). with their trainer, who grades their performance and provides them with feedback. It is an in-person assessment, in contrast to the Audio-COT (see above). This can be via video or directly observed.
Mini-Consultation Exercise (miniCEX)	A real-life observation of a patient-doctor consultation. The assessment critiques clinical skills, attitudes and behaviours. Assessments should be spread out evenly in the duration of the post and last no longer than 15 minutes.
Multisource Feedback (MSF)	MSF collects feedback from colleagues about trainees’ clinical performance and professional behaviour. The Educational Supervisor (ES) has access to the feedback once MSF closes.
Patient Satisfaction Questionnaire (PSQ)	Gives trainees feedback from patients on the extent to which they have shown empathy and sought to build a relationship with patients.
Clinical Examination and Procedural Skills (CEPS)	Trainers must be satisfied through observed evidence or documented evidence from others that trainees are generally competent. Trainees are assessed on five intimate examinations and a range of non-intimate examinations.
Clinical Supervisors Report (CSR)	This is a report that is completed by the trainee’s clinical supervisor in each post. Trainees receive feedback about their performance.
Educational Supervisor Report (ESR)	These reports provide feedback based on evidence within the portfolio on performance within the 13 capabilities and identifies areas for further improvement.

Table 1. Respondent characteristics.

		GP trainees		GP trainers
		N	%*	N	%	Chi square
Gender	Female	701	(60.0)	539	(59.6)	0.45
	Male	435	(37.2)	343	(37.7)
	Other	0	(0.0)	2	(0.2)
	Prefer not to say	33	(2.8)	26	(2.9)
Ethnicity
	White British or Irish	333	(34.6)	629	(65.4)
	White Other	55	(4.7)	39	(4.3)
	Asian	406	(34.5)	178	(19.5)	<0.001
	Black African Caribbean	232	(19.7)	8	(0.9)
	Mixed/Other	90	(7.7)	21	(2.3)
	Prefer not to say	60	(5.1)	36	(4.0)
Country of Primary Medical Qualification
	UK	466	(39.6)	771	(62.3)
	EEA	65	(5.5)	25	(2.8)	<0.001
	Rest of the world	618	(52.6)	709	(34.1)
	Prefer not to say	27	(2.3)	14	(1.6)
Practice type
	Urban	523	(44.5)	506	(55.5)
	Rural	154	(13.1)	109	(12.0)	<0.001
	Urban/Rural mixed	459	(39.0)	285	(31.3)
	Prefer not to say	40	(3.4)	12	(1.3)
UK nation of training
	England	973	(82.9)	683	(75.6)
	Scotland	113	(9.6)	130	(14.4)	0.001
	Wales	48	(4.1)	51	(5.6)
	Northern Ireland	30	(2.6)	33	(3.7)
	Prefer not to say
Full time or less than full time
	Full time	865	(73.7)	384	(42.4)
	Less than full time	297	(25.3)	510	(56.4)	<0.001
	Prefer not to say	12	(1.0)	11	(1.2)

*Percentage of those responding.

Table 2. Time spent engaging with the educational portfolio or completing the educational supervisor review report.

		GP trainees		GP trainers
		Number	Percent	Number	Percent	Chi square
Time engaging with portfolio (hours per week)	0–1	199	(16.9)	273	(29.9)
	1–2	450	(38.3)	473	(51.9)
	2–3	299	(25.4)	124	(13.6)	<0.001
	3–4	131	(6.3)	31	(1.5)
	>4	97	(8.2)	11	(1.2)
Time to complete an ESR report (hours)
	0–1	155	(13.2)	8	(0.9)
	1–2	351	(29.8)	184	(20.2)
	2–3	268	(22.8)	334	(36.6)	<0.001
	3–4	181	(15.4)	247	(27.1)
	>4	221	(18.8)	139	(15.2)

Table 3. Item statistics for scaled responses.

	Mean	Std. Deviation
The Care Assessment Tool (CAT) would be an appropriate assessment of ability for ST1/2 in General Practice	3.47	1.39
Please indicate the extent to which you agree or disagree that the Patient Satisfaction Questionnaire (PSQ) contributes to a global judgement on your performance	3.41	1.60
Please indicate the extent to which you agree or disagree that the Multi-Source Feedback (MSF) contributes to a global judgement on your performance	4.17	0.95
Please indicate the extent to which you agree or disagree that the Leadership Multi-Source Feedback (Leadership MSF) contributes to a global judgement on your performance	2.78	1.63
Please indicate the extent to which you agree or disagree that the Quality Improvement Project (QIP) contributes to a global judgement on your performance	3.52	1.13
Please indicate the extent to which you agree or disagree that the Care Assessment Tool (CAT) contributes to a global judgement on your performance	3.69	1.43
Please indicate the extent to which you agree or disagree that the Consultation Observation Tool (COT)/Audio Consultation Observation Tool (Audio-COT) contributes to a global judgement on your performance	4.21	1.18
Please indicate the extent to which you agree or disagree that the Learning Event Analysis (LEA) contributes to a global judgement on your performance	3.88	1.02
Please indicate the extent to which you agree or disagree that the Placement planning meeting contributes to a global judgement on your performance	3.64	1.01
Please indicate the extent to which you agree or disagree that the Leadership Activity contributes to a global judgement on your performance	3.26	1.36
Please indicate the extent to which you agree or disagree that the Quality Improvement Activity (QIA) contributes to a global judgement on your performance	3.62	1.08
Please indicate the extent to which you agree or disagree that the Supporting Documentation contributes to a global judgement on your performance	3.72	0.96
The assessment of performance statements are helpful to determine progress and development needs	3.81	0.87
I find a global grade judgement on individual assessments helpful	3.82	0.89
I find it useful in assessing trainee progression to have a judgement of level of required supervision within the Clinical Supervisor Report (CSR)	3.92	0.94
The different learning log options provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement within the Educational Supervisor Report (ESR)	3.78	0.88
The whole package of assessment tools provides sufficient information to allow an Educational Supervisor (ES) to develop an overall judgement within the Educational Supervisor Report (ESR)	3.84	0.87
The format of the clinical case reviews supports effective reflection on clinical encounters using the ‘what will I maintain, improve and stop’ format	3.58	1.02
A broad range of Care Assessment Tools (CATs) gives a more accurate assessment of ability than using Case-based Discussions (CbDs) alone	3.58	1.28
The leadership activity, Leadership Multi-source Feedback (Leadership MSF) and the leadership, management and professionalism learning log allows a trainee to demonstrate leadership skills	3.02	1.49
The Quality Improvement Project (QIP)/Quality Improvement Activity (QIA) helps understand the methods behind quality improvement theory and adds value to development	3.56	1.05
The overall range of WPBA facilitates early identification of developmental needs	3.58	.97
The nine Clinical Experience Groups are a true representation of the spectrum and range of patients in general practice	3.57	.97
The current range of WPBA provides sufficient opportunity to detail performance against the capabilities	3.75	.87
The Trainee Portfolio is user friendly	3.59	1.09
There is sufficient information about the requirements for WPBA on the RCGP website (including the WPBA handbook)	3.74	.99

Table 4. Comparing trainers’ and trainees’ views of assessment tools.

	Trainers			Trainees
	Number agreeing*	% Agreeing	Total	Number agreeing	% Agreeing	Total
The Care Assessment Tool (CAT) would be an appropriate assessment of ability for ST1/2 in General Practice	683	74.9%	912	742	63.3%	1,173
Please indicate the extent to which you agree or disagree that the Patient Satisfaction Questionnaire (PSQ) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	743	81.5%	912	653	55.6%	1,174
Please indicate the extent to which you agree or disagree that the Multi-Source Feedback (MSF) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	853	93.5%	912	938	79.9%	1,174
Please indicate the extent to which you agree or disagree that the Leadership Multi-Source Feedback (Leadership MSF) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	497	54.5%	912	443	37.7%	1,174
Please indicate the extent to which you agree or disagree that the Quality Improvement Project (QIP) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	670	73.5%	912	661	56.3%	1,174
Please indicate the extent to which you agree or disagree that the Care Assessment Tool (CAT) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	777	85.2%	912	824	70.2%	1,174
Please indicate the extent to which you agree or disagree that the Consultation Observation Tool (COT)/Audio Consultation Observation Tool (Audio-COT) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	873	95.7%	912	966	82.3%	1,174
Please indicate the extent to which you agree or disagree that Learning Event Analysis (LEA) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	848	93.0%	912	768	65.4%	1,174
Please indicate the extent to which you agree or disagree that Placement planning meeting is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	568	62.3%	912	804	68.5%	1,174
Please indicate the extent to which you agree or disagree that Leadership Activity is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	609	66.8%	912	804	68.5%	1,174
Please indicate the extent to which you agree or disagree that Quality Improvement Activity (QIA) is valuable in developing a trainee’s learning and contributing to a global judgement on your performance	737	80.8%	912	604	51.4%	1,174
Please indicate the extent to which you agree or disagree that Clinical Case Reviews are valuable in developing a trainee’s learning and contributing to a global judgement on your performance	863	94.6%	912	965	82.2%	1,174
Please indicate the extent to which you agree or disagree that Supporting Documentation are valuable in developing a trainee’s learning and contributing to a global judgement on your performance	615	67.4%	912	818	69.7%	1,174
The assessment of performance statements (i.e. where I am asked to rate the trainee’s overall competence at which they have shown they are performing) are helpful to determine a trainee’s progress and development needs	744	81.6%	912	871	74.2%	1,174
I find it helpful being required to make a global grade judgement on individual assessments	712	78.1%	912	885	75.4%	1,174
The different learning log options provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement of the trainee within the Educational Supervisor Report (ESR)	706	77.4%	912	870	74.1%	1,174
The different assessment tools provide sufficient evidence to allow a trainer/supervisor to develop an overall judgement of the trainee within the Educational Supervisor Report (ESR)	754	82.7%	912	883	75.2%	1,174
The format of the clinical case reviews supports effective reflection on clinical encounters using the ‘what will I maintain, improve and stop’ format	674	73.9%	912	722	61.5%	1,174
A broad range of Care Assessment Tools (CATs) gives a more accurate assessment of a trainee’s ability than using Case-based Discussions (CbDs) alone	678	74.3%	912	732	62.4%	1,174
The leadership activity, Leadership Multi-source Feedback (Leadership MSF) and the leadership, management and professionalism learning log allows a trainee to demonstrate leadership skills	519	56.9%	912	555	47.3%	1,174
The Quality Improvement Project (QIP)/Quality Improvement Activity (QIA) helps a trainee to understand the methods behind quality improvement theory and adds value to their development	692	75.9%	912	707	60.2%	1,174
The overall range of WPBA facilitates early identification of a trainee’s developmental needs	599	65.7%	912	775	66.0%	1,174
The nine Clinical Experience Groups are a true representation of the spectrum and range of patients in general practice	577	63.3%	912	776	66.1%	1,174
The current range of WPBA provides sufficient opportunity to detail a trainee’s performance against the capabilities	727	79.7%	912	837	71.3%	1,174
The Trainee Portfolio is user friendly	573	62.8%	912	805	68.6%	1,174
There is sufficient information about the requirements for WPBA on the RCGP website (including the WPBA handbook)	719	78.8%	912	841	71.6%	1,174
There is sufficient information within an interim Educational Supervisor Report (iESR), without having to rate the trainee against each capability, to determine a global judgement of the trainee	688	75.4%	912	883	75.2%	1,174

*Agree or strongly agree.

Table 5. Multivariable regression model showing respondent attributes associated with a positive attitude towards WPBA.

		B	95% confidence interval		P-value
Trainer/Trainee	Trainer	0 (Reference)
	Trainee	−0.48	−0.54	−0.42	<0.001
Sex	Female	0
	Male	−0.15	−0.068	0.038	0.57
	Other	−0.72	−1.54	−0.096	0.08
	Prefer not to say	−0.24	−0.45	−0.036	0.02
Ethnic group	White British Irish	0
	White other	0.13	−0.27	0.010	0.07
	Asian	−0.0071	−0.088	0.073	0.86
	Black	0.062	−0.052	0.17	0.29
	Mixed and other	−0.13	−0.26	−0.001	0.05
	Prefer not to say	−0.29	−0.45	−0.13	<0.001
Country of Primary Medical Qualification	UK	0
	European Economic Area	0.13	−0.12	0.27	0.07
	Rest of the World	0.29	0.20	0.37	<0.001
	Prefer not to say	0.23	−0.0048	0.46	0.06