‘Team observation’: a six-year study of the development and use of multi-source feedback (360-degree assessment) in obstetrics and gynaecology training in the UK

Figures & data

Table

Practice points
•Multi-source feedback (360-degree assessment) is an important tool in assessing the behaviours of doctors in training.
•We have developed a simple four domain Team Observation tool that has been used widely within obstetrics and gynaecology training within the UK.
•The Team Observation tool may be used for identification of performance regarded as poor and for congratulating performance thought to be excellent in interpersonal behaviours.
•Multi-source feedback (360-degree assessment) seems to measure a very limited number of attributes—and the main construct that is measured in the Team Observation system is interpersonal behaviour.

Table 1.  Number of doctors assessed

Assessment	Number of doctors	Percentage	Number of forms	Percentage
First	113	52	839	49
Second	67	31	578	34
Third	20	9	156	9
Fourth	12	5	95	6
Fifth	5	2	37	2
Sixth	2	1	14	1
Total	219	100	1719	100

Table 2.  Numbers and percentages of responses in each of the four categories in all 1719 forms

Category	Needs serious attention	Progress needed	Fine—no problem	Outstanding	Nil	Total
Relationships with colleagues	16	110	899	692	2	1719
	(0.9%)	(6%)	(52%)	(40%)
Relationship to patients	11	66	919	664	59	1719
	(0.7%)	(4%)	(55%)	(40%)
Information gathering and notekeeping	5	80	875	637	122	1719
	(0.3%)	(5%)	(55%)	(40%)
Time management and diligence	13	104	929	653	20	1719
	(0.3%)	(6%)	(55%)	(38%)
Totals	45	360	3622	2646	203	6876
	(0.7%)	(5.2%)	(53%)	(38%)

Table 3.  Analysis of the free text comments

Types of free text comments	First assessment	All assessments
Negative	99	183
	(12%)	(11%)
Both positive and negative	109	222
	(13%)	(13%)
Neutral	43	62
	(5%)	(4%)
Positive	327	686
	(39%)	(40%)
None	261	566
	(31%)	(33%)
Totals	839	1719
	(100%)	(100%)

Table 4.  Principal components of the four domains of assessment

Category	Principal component number 1	Principal component number 2	Principal component number 3	Principal component number 4
Relationships with colleagues	0.87	0.37	0.16	0.28
Relationship to patients	0.89	0.29	−0.20	−0.30
Information gathering and notekeeping	0.86	−0.36	−0.33	0.16
Time management and diligence	0.86	−0.31	0.38	−0.14
Variance	3.02	0.45	0.32	0.21
Percentage of total variance	76%	11%	8%	5%

Table 5.  Spearman's correlation coefficients and p-values between the first and the second mean scores

Category	First and second mean scores (n = 67)	p-value
Relationships with colleagues	0.78	Less than 0.001
Relationship to patients	0.78	Less than 0.001
Information gathering and notekeeping	0.66	Less than 0.001
Time management and diligence	0.65	Less than 0.001
Principal component 1	0.77	Less than 0.001
Principal component 2	0.61	Less than 0.001

Table 6.  Spearman's correlation coefficients between mean scores of the different domains in 1719 assessments

Category	Relationships with colleagues	Relationship to patients	Information gathering and notekeeping	Time management and diligence
Relationships with colleagues	1	0.90	0.71	0.83
Relationship to patients	0.90	1	0.82	0.84
Information gathering and notekeeping	0.71	0.82	1	0.82
Time management and diligence	0.83	0.84	0.82	1

Note: All of the correlations are significant at p < 0.001 level.

Table 7.  Power of each domain to add discernment to the most discerning domain, in picking up the 124 with unsatisfactory scores

Category	Number detected as each domain is added	Percentage picked up in all (cumulative)
Relationship with staff	69 identified	56
Plus Information gatheringand notekeeping	31 more identified	81
Plus Time managementand diligence	17 more identified	94
Plus Relationship topatients	7 more identified	100

Table 8.  Demonstration of correlation between raters: intraclass correlations

Variable	Intraclass correlation	p-value
Relationships with colleagues	0.26	<0.001
Relationship to patients	0.27	<0.001
Information gathering and notekeeping	0.25	<0.001
Time management and diligence	0.24	<0.001
Score of four domains ‘Ability’	0.34	<0.001
Second attribute	0.04	0.04

Table 9.  Demonstration of the increasing generalizability of intraclass correlation coefficients, by increasing number of raters

Putative number of raters	Calculated intraclass correlation coefficients
1	0.34
2	0.54
3	0.60
4	0.67
5	0.72
6	0.75
7	0.78
8	0.80
9	0.82
10	0.84
11	0.85
12	0.86
18	0.90