Triangulation of written assessments from patients, teachers and students: Useful for students and teachers?

Figures & data

Figure 1. The StudentPEP project. Note: This overview was used when informing students and teachers about the project.

Table

Practice points

Formative assessment is important in clinical clerkships. Students want more feedback on their clinical performance. Multiple sources of feedback are recommended. The StudentPEP method of triangulation of written feedback from patients, teachers and students received mixed responses from students and teachers. The teacher's attitude towards the project strongly influenced the student's perception.

Table 1.  Responses from students and teachers on the StudentPEP project evaluation questionnaire, sorted by principal component analysis on student answers

	Students, n = 79 (Teachers, n = 71)
	Negative (%)	Neutral (%)	Positive (%)	Spearman's rho	Factor loads	Variance
Opinions of effort versus benefit	28	34	39			36.60
Participation in the StudentPEP project is time consuming^a	33 (28)	33 (33)	34 (39)	0.25*	−0.841
I think StudentPEP helped me to learn how to use a quality improvement tool (the StudentPEP questionnaire)	20	39	41		0.751
It was useful to evaluate myself/read the students’ self-assessment in the triangulated evaluations	25 (22)	27 (20)	48 (58)	0.19	0.615
The StudentPEP project is worthwhile	31 (32)	36 (31)	33 (37)	0.44*	0.526
Teacher's oral and written feedback	21	22	58			10.82
My teacher gave me feedback on my strengths and weaknesses throughout the clerkship period	25	37	38		0.899
I feel satisfied with the total feedback I got in my clerkship	8	12	80		0.870
I got/gave concrete feedback on my/the students weaknesses in the triangulated evaluations	22 (21)	24 (15)	54 (65)	0.21	0.539
My teacher and I had a discussion concerning the feedback provided in the StudentPEP material	27 (24)	15 (21)	58 (56)	0.29*	0.533
Attitude to patient evaluations	11	30	58			8.99
I was interested in what the patients thought of my/my student's consultations	11 (18)	11 (19)	78 (63)	0.08	0.848
Information prior to the StudentPEP project was good enough	11 (14)	27 (30)	62 (56)	0.42*	0.704
Patient feedback increased my self-confidence	8	41	51		0.613
I think the patients gave their honest opinion on the questionnaires	15	41	44		0.434
I believe patient surveys are helpful for general practitioners in collecting feedback	12	31	57		0.314
Usefulness of patient evaluations	32	28	41			4.85
It was difficult to translate patient feedback into something I could use in my practical work^a	31	34	35		0.823
Patient feedback increased my insight in my own skills	32	26	42		−0.667
Feedback from my patients was useful	28 (25)	26 (27)	46 (47)	0.11	−0.662
The StudentPEP questionnaires gave me concrete feedback from my patients	31	25	44		−0.641
Effect on teacher	30	24	46			4.45
My teacher's feedback was more structured in the triangulated evaluations than generally/The teacher forms helped me to structure the assessment	45 (44)	27 (25)	28 (31)	0.33*	0.797
I think StudentPEP helped me to ensure teacher observation during my consultations	21	20	59		0.726
I think StudentPEP helped me to get more feedback in my clerkship	26	29	45		0.610
I think StudentPEP helped me to get better feedback in my clerkship	37	24	39		0.482
It was useful to have my teacher's assessment in the triangulated evaluations	21	20	59		0.394
Helpful for reflection	26	29	46			3.80
StudentPEP helped me to assess my communication with the patient	31	20	49		0.692
I think StudentPEP helped me to ensure that I got the opportunity to have independent consultations	48	32	20		0.651
I was motivated to attend	24 (24)	34 (28)	42 (48)	0.22	0.635
I think StudentPEP helped me to confirm that I can handle the doctor role	12	28	60		0.629
My teacher helped me/I helped the student to make the most of StudentPEP	25 (31)	37 (38)	38 (31)	0.23	0.574
I think StudentPEP helped me to reflect upon my patients’ views	21	20	59		0.511
I think StudentPEP helped me to increase my learning about how I function in my role as a doctor	20	29	51		0.433

Notes: Questions translated from Norwegian.‘Positive’ refers to scores 4–5, ‘neutral’ to 3 and ‘negative’ to 1–2 on a 5-point Likert scale.^aThe questions had negative factor loads and the subscale means are calculated with the inverted value on these items.*Correlation between students’ and teachers’ responses on corresponding questions was significant with p < 0.05.

Table 2.  Logistic regression analysis showing items associated with student agreement on the item ‘The StudentPEP project was worthwhile’, among 5th year Norwegian medical students in general practice clerkship

	Bivariate analysis			Multivariate analysis
Questions	OR	(95% CI)	p-value	OR	(95% CI)	p-value
Clinical teachers
Participation in the StudentPEP project is time consuming
Agree (ref)	1.0			1.0
Neutral	1.1	(0.3–4.1)	0.94	1.7	(0.3–8.3)	0.52
Disagree	4.0	(1.1–14.6)	0.04	14.2	(2.2–90.5)	0.005
I was interested in what the patients thought of my student's consultations
Disagree (ref)	1.0			1.0
Neutral	7.5	(0.9–65.1)	0.13	26.7	(1.4–500.1)	0.028
Agree	5.5	(0.5–65.1)	0.43	18.9	(1.6–226.1)	0.02

Note: n = 79 students, 71 teachers. Questions translated from Norwegian.

CI, confidence interval.

Table 3.  Examples of written statements on the StudentPEP project evaluation questionnaire from medical students

Positive patient feedback was found either encouraging or untrustworthy

Learnt what I did well and where I could improve. (47)

That I am good at giving information, listening well, asking simple questions, appearing trustworthy and interested, making thorough examinations. (04)

Learnt a lot about how I communicate with the patients, how the way I express myself and my body language are experienced by others. (84)

That the patients mainly cared if they were listened to and taken seriously. (07)

I learnt in a way to listen more broadly, so that worries and any other concerns did not get lost in the consultation. (55)

After trying and failing, I came nearer my role as a doctor, there were no real surprises when it came to relating to patients. (07)

I mainly got positive feedback. That was nice. (14)

That I work well in a doctor–patient relationship. That the patients dared open up to me. (37)

They were exclusively positive and I suspect this was independent of my achievement. (61)

Triangulation enhanced learning

That one stresses different things, that the degree of relevance is different for student/doctor/patient. (74)

That the patient and the doctor are preoccupied with consultation skills and find these satisfactory, while I myself was more concerned whether my knowledge  is sufficient and with my lack of knowledge. (77)

I have learnt something from thinking critically about my consultation afterwards. (66)

How the doctor considered my behaviour. More specifically that I had to be more structured, focused and efficient. (11)

Got direct feedback from my tutor and ideas for further progress. (67)

I think the patients’ answers are biased because they think/know that we read what they have written afterwards and therefore they are not quite honest,  but ‘kind’. (24)

Objections regarding methods and implementation

Fixed questions that do not fit many of the consultations (impossible to answer). (16)

Both myself and my tutor ought to have been more motivated. (23)

StudentPEP takes too much time, in relation to what one can learn from it. And then it does not always fit into a busy general practice. (13)

He [my tutor] did not allow enough time for it [the project] and wanted me to use maximum 15 min for the consultation plus the evaluation. (75)

Time – The questionnaires were more of a bother and an obstacle for the good talks we had for each patient consecutively. (09)

StudentPEP's overall impact on the clerkship

I learnt more from what they patients told me directly than from their scores on the questionnaire. (25)

I think the StudentPEP-project is quite all right. (31)

Note: Students’ numbers are given in brackets.