Improving optometry student interpersonal skills through online patient, clinician and student evaluation and feedback

ABSTRACT

Clinical relevance

Interpersonal skills are crucial for successful clinician-patient interactions. To prepare future optometrists for clinical practice, pedagogical evaluation is important to support the implementation of new strategies for teaching and evaluating interpersonal skills.

Background

Optometry students largely develop their interpersonal skills through in-person patient interactions. Telehealth is increasing, yet strategies to develop the interpersonal skills of students for teleconsulting have not been explored. This study aimed to assess the feasibility, effectiveness and perceived usefulness of an online, multisource (patients, clinicians and students) evaluation and feedback program for developing interpersonal skills.

Methods

Via an online teleconferencing platform, optometry students (n = 40) interacted with a volunteer patient, observed by a teaching clinician. Patients and clinicians evaluated the interpersonal skills of the student in two ways: (1) qualitative written feedback, and (2) quantitative rating (Doctors’ Interpersonal Skills Questionnaire). All students received written patient and clinician feedback after the session, but not their quantitative ratings. A subset of students (n = 19) completed two sessions, self-ratings, and were provided with their written feedback and an audiovisual recording from their first interaction before completing the second session. All participants were invited to complete an anonymous survey at program completion.

Results

Patient and clinician overall interpersonal skills ratings were positively correlated (Spearman’s r = 0.35, p = 0.03) and showed moderate agreement (Lin’s concordance coefficient = 0.34). Student self-ratings did not match patient ratings (r = 0.01, p = 0.98), whereas there was moderate agreement between clinician and student ratings (Lin’s concordance coefficient = 0.30). Ratings improved at the second visit (p = 0.01). Patient ratings were higher than clinicians (p = 0.01) and students (p = 0.03). All participants agreed that the program was feasible, useful and effective at fostering good interpersonal skills.

Conclusion

Multisource feedback about interpersonal skills contributes to improvement in student performance. Patients and clinicians can evaluate and provide useful feedback to optometry students about their interpersonal skills using online methods.

KEYWORDS:

• Communication
• education
• multisource feedback
• professionalism
• telehealth

Introduction

Interpersonal skills are the verbal and non-verbal social skills used to communicate, interact and build relationships with other people. For all health professionals, good interpersonal skills enable relationships built on ‘respect, trust and effective communication’ and ‘practitioners to work in partnership with patients’.¹ Entry-level competency standards² require optometrists to incorporate the patient’s perspective to provide integrated, respectful and responsive people-centred eye care.³ Accordingly, significant curricular development has focused on teaching and evaluating interpersonal skills of health professionals, predominantly in medicine^4,5 but increasingly recognised in optometry.^6–8

In the four-year, masters-level Doctor of Optometry course at The University of Melbourne, students are introduced to interpersonal skills and communication strategies from their first year of study. Understanding how to take a history is taught via lectures in Year 1 and practised and assessed in a pre-clinical environment amongst their peers. In Year 2, students begin to explore the specific history questions required for different presenting complaints and how to approach management discussions. This is complemented by final-year student consultation observations in the clinic, where Year 2 students practise recording the history-taking they observe. Year 3 students begin to see patients individually in a clinical environment and attend 16 hours of seminars covering topics such as understanding self in communication, communication across cultures and communicating bad news. By final year, students are able to formally demonstrate their competency using qualitative information about their in-person interactions through clinical feedback systems, self-evaluation and a final-year portfolio.

However, there have been no reported opportunities in any optometry degree for students to develop their interpersonal skills in a telehealth environment. This is critical to address in contemporary health professional education,⁹ as telehealth is increasingly used in optometry.¹⁰ Furthermore, the important contribution of the ‘patient voice’ to communication and interpersonal skills training has long been recognised in medicine,^11–16 and more recently, in optometry.⁷

Thus, this study aimed to develop a novel, online activity to evaluate and provide feedback about the teleconsulting interpersonal skills of students from three perspectives (multisource, or also called 360-degree, feedback) – the patient, the clinician, and the student. The key research questions addressed by this study were: (1) Is there agreement between patients, clinicians, and students in rating and feedback about the interpersonal skills of students? (2) Do the interpersonal skills of students improve with a repeated patient interaction, following the provision of written feedback about an initial patient interaction? (3) Do patients, clinicians and students perceive the activity to be a feasible, useful and effective teaching and learning exercise for developing the interpersonal skills of students?

Because the new learning activity was added as a voluntary exercise for students in their final year of the Doctor of Optometry course at The University of Melbourne, no formal changes were made to the curriculum and no additional content or theory about interpersonal skills specifically in a telehealth setting or more broadly in a general setting was provided to students. Being final year optometry students, with at least one year of prior clinical exposure seeing patients under supervision, we expected that the student participants would be able to draw on their varying individual in-person experiences in the clinic and from their common learning and practice of communication and interpersonal skills built up progressively across the course to complete the activity.

Methods

Participants

Ethics approval was granted by the Human Research Ethics Committee of The University of Melbourne (ID 14513) and the study protocol adhered to the tenets of the Declaration of Helsinki. Recruitment was through word-of-mouth and email invitations sent to final-year Doctor of Optometry students at The University of Melbourne and teaching clinicians (registered optometrists with experience supervising optometry students) at The University of Melbourne, Australian College of Optometry, or private independent practice.

Volunteer patients were recruited through online advertisements posted at The University of Melbourne (e.g. Staff News), local senior community group mailing lists (e.g. University of the Third Age), or from a database of previous volunteers who had consented to being contacted for research opportunities.

All participants provided written informed consent and were required to have English language proficiency and access to an electronic device with microphone and camera. Patients were ‘real’ (not actors) for authenticity¹⁵ and were at least 50 years of age  to ensure that the mock scenarios would be age-appropriate. In total, 40 optometry students, 33 patients and 21 clinicians participated between October 2020 and March 2022.

Study sessions

Study sessions were conducted using an online teleconferencing platform (Zoom Video Communications Inc., San Jose, CA, USA) and involved one patient (de-identified by renaming), one student (identifiable) and one clinician (anonymous: muted and off camera). One study investigator (BN or JN) hosted each session to provide instructions, time management and technical assistance. For logistical reasons, most sessions lasted one hour to enable one patient and one clinician to evaluate two students (~30 mins each patient-student interaction).

A written and verbal prebriefing was provided to familiarise participants with the format and tasks. Students took a history (max 15 mins), assuming they had never met the patient before, and then discussed a fictional scenario and management plan (max 15 mins, without any digital visual aids such as screen sharing). The host chose a plausible scenario (from Supplementary Material A as likely to be encountered in clinical practice),¹⁷ based on the patient’s history (e.g. if a patient reported previous cataract surgery, the ‘cataract’ scenario was not chosen). During the patient-student interaction, patients were allowed to ask questions to the student optometrist, or seek clarification (e.g. ask student to repeat information, or rephrase), at any time if needed.

After each session, patients were provided with an information brochure (from Optometry Australia, the key professional organisation for optometrists in Australia) about the scenario discussed and offered a non-compulsory debrief session with the study investigators. Students were allocated to one of two participant groups: (1) group 1 (n = 19), who only received feedback from the patient and clinician; and (2) group 2 (n = 21), who received feedback from the patient and clinician and an audiovisual recording of their patient interaction, completed a self-evaluation (before any clinician or patient feedback was provided, to avoid biasing their self-evaluation), and were offered an opportunity to participate in a second session.

Surveys to evaluate and provide feedback about the patient-student interaction

Two types of data were collected using an online survey platform (Qualtrics, Provo, Utah, USA): (1) quantitative evaluation data to rate the interpersonal skills of students, and (2) qualitative comments about the student’s interpersonal skills (see Supplementary Material B). The first part of the survey was a modified version of the Doctors’ Interpersonal Skills Questionnaire (DISQ)¹⁸ previously used by patients to evaluate in-person the interpersonal skills of students.⁷

Each question in the 12-item DISQ is rated on a 5-point Likert scale (1 = poor, 2 = fair, 3 = good, 4 = very good, 5 = excellent). The twelve item ratings were summed to calculate an overall score (range 12–60), with higher scores associated with better interpersonal skills. The ratings questionnaire also included a reliability check, and question order was randomised. The second part of the survey consisted of two open-text feedback questions: ‘What two things did the student do well?’ and ‘What two things could the student improve?’.

Survey about the teaching and learning initiative

At program completion, participants received a link to an online, anonymous survey to evaluate the activity as per Schmid et al.⁷ Patients and clinicians were asked about their experience providing feedback to students (see Supplementary Material C), while students were asked about their experience receiving patient and clinician feedback (see Supplementary Material D).

Data analysis

Quantitative DISQ rating data were tested for normality using a Kolmogorov-Smirnov test. A repeated-measures analysis of variance (RM-ANOVA) was used to compare data between two factors (e.g. student groups, sessions). Post-hoc Bonferroni pairwise comparisons determined the statistical significance of a main effect with more than two factors (e.g. multiple feedback sources). To evaluate the relationships between patient, clinician and student ratings, Spearman’s rank correlations and Lin’s concordance correlation coefficient¹⁹ (CCC), and 95% confidence intervals (CI), were calculated to assess absolute agreement between two measures.²⁰ A correlation coefficient < 0.20 was considered poor and > 0.80 was considered excellent agreement.²¹ A chi-square test of proportions was used to compare percentage responses between groups. A p < 0.05 was considered statistically significant.

For the qualitative data (written feedback), a thematic analysis was undertaken. Deductive coding first identified the prominent themes of the twelve DISQ domains (e.g. time given to patient, opportunity for patient to express concerns or fears). Inductive coding was subsequently used as more themes and sub-themes emerged. Two authors (BN and JN) coded the written feedback in NVivo 12 (QSR International Pty Ltd, Doncaster, Victoria, Australia). Discrepancies in coding were resolved by consensus (BN and JN), and the extracted themes were verified by author MC who did not code the data.

Results

All questionnaires (quantitative DISQ ratings, qualitative written feedback, and program evaluation surveys) had complete data and therefore all these data were included in the analysis.

Patient and clinician quantitative evaluations of the interpersonal skills of students

When comparing patient and clinician ratings of students in group 1 (n = 19) and group 2 (n = 21), patients gave higher overall DISQ scores than clinicians (RM-ANOVA main effect of feedback source: F(1,38) = 7.40, p = 0.01, partial η² = 0.16), but there was no difference between group ratings (main effect of group: F(1,38) = 2.09, p = 0.16, partial η² = 0.05). Hence, groups 1 and 2 were combined (n = 40) to determine the level of agreement between patient and clinician ratings of ‘first-time’ patient-student interactions.

Figure 1A depicts the weak positive correlation between patient and clinician overall DISQ scores (Spearman’s r = 0.35, p = 0.03). There was moderate agreement between patient and clinician overall ratings of the interpersonal skills of students (Lin’s CCC = 0.34, 95% CI = 0.07–0.56). Given the Bland-Altman plot shown in Figure 1B, patient and clinician ratings are expected to differ on average by 4 points on the DISQ, and to be within approximately ± be within approximately 17 points of each other (95% limits of agreement).

Figure 1. Relationship between Doctors’ Interpersonal Skills Questionnaire (DISQ) overall scores evaluating the interpersonal skills of students at their first patient-student interaction by (A) patients and clinicians (n = 40 evaluations), (C) patients and students (n = 21 evaluations), and (D) clinicians and students (n = 21 evaluations). Panel B is a Bland-Altman plot, showing the difference in overall DISQ score between patients and clinicians as a function of average ratings of the interpersonal skills of students. The bias (average difference, horizontal dashed line) was negative (3.75 points), indicating that on average patient ratings were higher than ratings by clinicians. The 95% limits of agreement were from -20.5 to 13 points (horizontal dotted lines) .

Table 1 shows the patient and clinician ratings, and agreement thereof, for each of the 12 DISQ domains. Warmth of greeting, concern for patient as a person, explanation skills, and ability to be reassuring showed the highest agreement between patient and clinician ratings (Table 1; Lin’s CCC = 0.32–0.40, demonstrating moderate agreement).²¹ All other domains showed poor agreement (i.e. clinicians typically gave lower scores than patients), with patients and clinicians disagreeing the most when evaluating their confidence in the ability of students, respect shown to patient, and time given to patient (Table 1; Lin’s CCC = 0.16, demonstrating poor agreement).²¹

Table 1. Doctors’ Interpersonal Skills Questionnaire (DISQ) ratings by patients and clinicians from 40 ‘first-time’ patient-student interactions. Median and range are listed. Each domain could have a maximum score of 5, the overall score could have a maximum score of 60. Lin’s concordance correlation coefficient (CCC) and 95% confidence intervals are given to indicate agreement between patient and clinician ratings. Asterisks indicate significant agreement.

DISQ Item	Patients	Clinicians	Lin’s CCC
Overall satisfaction	4 (3–5)	4 (2–5)	0.24 (−0.04–0.49)
Warmth of greeting	4 (3–5)	5 (3–5)	0.40 (0.10–0.62)*
Ability to really listen	4 (2–5)	4 (2–5)	0.24 (−0.08–0.51)
Explanation skills	4 (2–5)	4 (1–5)	0.32 (0.07–0.53)*
Ability to be reassuring	4 (2–5)	4 (1–5)	0.32 (0.06–0.53)*
Confidence in student’s ability	4 (2–5)	4 (2–5)	0.16 (−0.15–0.44)
Opportunity to express concerns or fears	4 (2–5)	4 (2–5)	0.19 (−0.11–0.45)
Respect shown to patient	5 (3–5)	5 (3–5)	0.16 (−0.15–0.43)
Time given to patient	5 (2–5)	4 (2–5)	0.16 (−0.14–0.44)
Consideration for patient’s personal situation in deciding a treatment or advising the patient	4 (2–5)	4 (1–5)	0.18 (−0.31–0.38)
Concern for patient as a person	4 (2–5)	4 (1–5)	0.33 (0.04–0.57)*
Recommendation to others about the student	4 (3–5)	4 (3–5)	0.27 (−0.02–0.52)
TOTAL SCORE	52 (30–60)	47 (27–59)	0.34 (0.07–0.56)*

Student self-evaluations of their interpersonal skills

Figure 1C,D show the relationships between patient, clinician and student ratings based on the overall DISQ score. Student self-ratings (n = 21) did not match that of patients (Figure 1C; Spearman’s r = 0.01, p = 0.98; Lin’s CCC = 0.01, 95%CI = −0.31–0.33), with students tending to underscore themselves relative to patient ratings. Similarly, Spearman’s rank correlations and Lin’s concordance between clinician and student overall DISQ ratings did not reach statistical significance (Figure 1D; Spearman’s r = 0.29, p = 0.20; Lin’s CCC = 0.30, 95%CI = −0.11–0.62).

Effect of second session

Of the 21 students in group 2, one student could not attend a second visit and one patient did not complete an evaluation and was lost to follow-up. Thus, in total there were 19 repeat sessions (Figure 2). Overall ratings of the interpersonal skills of students by patients, clinicians and students were higher at the second visit (RM-ANOVA main effect of session: F(1,54) = 7.76, p = 0.01, partial η² = 0.13). There was also a main effect of feedback source (F(2,54) = 5.39, p = 0.01, partial η² = 0.17). Post-hoc Bonferroni pairwise multiple comparisons demonstrated that patients rated the interpersonal skills of students higher than clinicians (p = 0.01) and students (p = 0.03) across all visits.

Figure 2. Overall DISQ scores from the A: First session (n = 19 evaluations), and B: Second session (n = 19 evaluations), as rated by patients, clinicians and students. Boxplots show the median, 25th and 75th percentiles, with the whiskers depicting the 10th and 90th percentiles. All outliers are shown as individual symbols. Horizontal dotted lines indicate the minimum and maximum overall DISQ scores possible (i.e. 5 and 60, respectively). Overall ratings of the interpersonal skills of students by patients, clinicians and students were higher at the second visit (RM-ANOVA main effect of session: F(1,54) = 7.76, p = 0.01, partial η² = 0.13). Asterisks indicate that the post-hoc Bonferroni pairwise comparison was statistically significant at p < 0.05.

Thematic analysis of written feedback

As described in Table 2, the thematic analysis revealed 6 key themes (‘body language’, ‘communication techniques’, ‘confidence in student ability’, ‘manner’, ‘patient-centred approach’ and ‘relationship building’) to describe the feedback provided by patients, clinicians and students in response to the question: ‘What did the student do well?’ Under these 6 key themes, 27 sub-themes were identified. The most commonly encountered sub-themes under each key theme were: ‘smile’ and ‘eye contact’ (under ‘body language’), ‘good explanations or instructions’ (under ‘communication techniques’), ‘thorough’ (under ‘confidence in student ability), ‘warm and friendly’ (under ‘manner’), ‘answered questions’ and ‘opportunity for patients to ask questions or express concerns’ (under ‘patient-centred approach’) and ‘built rapport’ (under ‘relationship building’).

Table 2. Key themes (italics) and sub-themes (indented) identified across the three sources of written positive feedback about the interpersonal skills of students in response to the open-ended question: ‘What did the student do well?’. Theme totals (italics) are the aggregate of sub-themes.

Theme	Sources
	Patient	Clinician	Student
Body language	9	7	9
Eye contact	3	4	3
Gestures	0	0	3
Open	1	1	0
Smile	5	2	3
Communication techniques	40	56	22
Adjusted communication to suit patient	3	5	2
Clear and concise	14	11	2
Conversational	3	4	1
Good approach to breaking bad news	3	0	1
Summarising and paraphrasing	3	17	2
Good explanations or instructions	14	19	14
Confidence in student’s ability	24	39	13
Competent	10	16	5
Knowledgeable	9	5	1
Thorough	5	21	8
Manner	52	39	15
Calm and confident	12	15	4
Genuine	6	1	0
Professional	4	9	0
Respectful	6	4	4
Warm and friendly	24	10	7
Patient-centred approach	33	47	36
Answered patient’s questions	9	8	5
Involved patient in developing management plan	2	2	4
Consideration of patient’s individual situation	4	8	2
Empathy	4	9	3
Listened to patient	7	7	7
Opportunity for patients to ask questions or express concerns	4	9	9
Time given to patient	3	5	6
Relationship building	12	29	21
Built rapport	6	18	13
Reassuring and made patient feel comfortable	6	11	8

Table 3 summarises the thematic analysis of the feedback provided in response to the question: ‘What could the student improve?’ Overall, 8 key themes were identified, most of which were similar to the key themes reported in Table 2 (‘body language’, ‘communication techniques’, ‘confidence in student ability’, ‘language and information delivery’, ‘manner’, ‘patient-centred approach’, ‘relationship building’, ‘no improvement needed’). Of the 32 sub-themes, the most commonly encountered sub-themes under each key theme were: ‘poor eye contact and note taking approach’ (under ‘body language’), ‘poor explanations or instructions’ (under ‘communication techniques’), ‘did not fully explore patient history’ (under ‘confidence in student ability’), ‘use of jargon or acronyms’ (under ‘language and information delivery’), ‘not calm or confident’ (under ‘manner’), ‘did not consider the individual situation of the patient’ (under ‘patient-centred approach’), and ‘did not build rapport’ (under ‘relationship building’).

Table 3. Key themes (italics) and sub-themes (indented) identified across the three sources of written negative feedback about the interpersonal skills of students in response to the open-ended question: ‘What could the student improve?’. Theme totals (italics) are the aggregate of sub-themes.

Theme	Sources
	Patient	Clinician	Student
Body language	9	1	3
Closed	0	1	0
Did not smile	1	0	0
Distracting	1	0	0
Poor eye contact and note taking approach	7	0	3
Communication techniques	24	54	28
Conversation did not flow	3	7	4
Did not adjust communication to suit patient	0	0	2
Did not summarise or paraphrase	2	6	0
Not clear and concise	3	14	7
Poor approach to breaking bad news	8	3	4
Poor explanations or instructions	8	24	11
Confidence in student’s ability	7	45	14
Did not fully explore patient history	0	19	5
Knowledge gap	1	14	7
Made incorrect assumptions	3	1	0
Missed things	3	13	2
Language and information delivery	31	43	13
Casual or inappropriate language	3	15	0
Order of information	1	11	6
Speed of delivery	10	3	6
Too much or too little information	8	7	0
Use of jargon or acronyms	9	9	1
Manner	16	6	10
Not calm or confident	13	5	10
Nor warm or friendly	2	0	0
Unprofessional	2	1	0
Patient-centred approach	21	34	30
Did not answer patient’s questions	0	5	1
Did not check for patient understanding	4	3	1
Did not involve patient in developing management plan	0	1	5
Did not consider patient’s individual situation	4	14	8
Did not give opportunity for patient to ask questions or express concerns	6	5	4
Did not listen to patient	3	2	0
Insufficient time given to patient	2	3	9
Lack of empathy	2	3	2
Relationship building	12	5	3
Did not build rapport	11	3	2
Not reassuring or did not make patient feel comfortable	1	3	1
No improvement needed	17	2	1

Evaluation of learning and teaching activity

The anonymous program evaluation survey was completed by 31 students (78% response rate), 30 patients (91% response rate) and 17 clinicians (81% response rate). Overall, all participants perceived the activity to be useful, and agreed that patient and teaching clinician feedback is effective at fostering good interpersonal skills in the clinic learning environment (Figure 3). Fewer students agreed that feedback from clinicians was effective at fostering good interpersonal skills (82%), compared to feedback from patients (94%), but these proportions did not differ (Figure 3A; χ²(1, N = 53) = 1.23, p = 0.27).

Figure 3. Reported effectiveness of feedback for fostering good interpersonal skills in the clinic learning environment, from the perspectives of A: Student optometrists, when evaluating feedback from patients (black bars) and clinicians (grey bars), and B: Patients (black bars) and clinicians (grey bars), when evaluating their own feedback.

Figure 4 shows the student perspectives about the activity. Most students agreed that feedback from patients (97%) and clinicians (93%) was useful (Figure 4A), and reported that they reflected on and used the feedback from patients (100%) and clinicians (93%) to improve their clinical competence (Figure 4B). Example quotes of how students demonstrated behaviour change are listed in Supplementary Material E. Students felt anxious knowing they were being evaluated by patients and clinicians similarly (Figure 4C, chi-square test of proportions: χ²(1, N = 53) = 2.29, p = 0.13). Students also self-reported trying harder to positively interact with the patient, knowing they were being evaluated by the patient and clinician (Figure 4D, χ²(1, N = 51) = 0.22, p = 0.64). All patients and clinicians felt that providing feedback helped the student learn and that it was easy to give constructive comment about how the student interacted. Only 3% of patients felt anxious about providing feedback.

Figure 4. Frequency of student optometrists who self-reported A: That the feedback from patients and clinicians was useful, B: That they reflected on the feedback from patients and clinicians and used it to improve their clinical competence, C: Anxiousness knowing that feedback would be given from patients and clinicians, and D: That they tried harder to positively interact with the patient because they knew they were being evaluated by patients and clinicians. In all panels, black bars represent patients and grey bars represent clinicians.

Discussion

This study demonstrates the feasibility and usefulness of an online program to support the development of interpersonal skills of optometry students. Building on previous work examining patient feedback from in-person interactions,⁷ a novel multisource (patient, clinician and student) evaluation and feedback program that was wholly online was developed. The strengths of teleconferencing technology were capitalised on, to enable remote participation (to enhance inclusivity), de-identification and anonymity (to minimise patient and clinician discomfort in providing feedback), and audiovisual recording (as an additional feedback mechanism).

Participants were instructed on how to complete an evaluation (quantitative) and provide feedback (qualitative) in their own time using online surveys. This ensured that common barriers to effective feedback for health professional trainees were addressed, such as insufficient time,²² lack of instructions regarding feedback,²² indirect feedback mechanisms,²² unsafe or inappropriate environment for feedback,²² fear of awkward future interactions,²² concern that scoring poorly would be detrimental to a student’s reputation, academic standing or educational progress,²³ and lack of direct observation of student-patient interactions.²⁴ Students had a unique opportunity for explicit, focused observation of their interpersonal skills that enhanced credibility and specificity of feedback. Such focused opportunities are less frequent for interpersonal skills specifically and have not been reported for optometry teleconsulting.

Is there agreement between patients, clinicians, and students in their evaluation of and feedback about the interpersonal skills of students?

Consistent with general practitioner¹⁸ and medical resident²⁵ trainees, clinician ratings of the interpersonal skills of students were generally lower than, but moderately correlated with, ratings from patients. Differences of up to 17 points in the overall DISQ score between clinicians and patients (see Figure 1B) may not be acceptable in some evaluation settings, such as for final assessments of student performance. Nevertheless, the level of concordance between patient and clinician ratings of the interpersonal skills of students reported here (Lin’s CCC = 0.34) and in other health professions (medical residents, Lin’s CCC = 0.50)²⁵ implies that both patient and clinician assessments are equally valid and useful for training purposes.

Both the quantitative and qualitative analysis revealed that patients and clinicians do not provide the same evaluation of, and feedback to, students about their interpersonal skills. Because clinicians regularly assess students for the content and clinical knowledge exchanged during patient-student interactions, their evaluations of interpersonal skills could be influenced by the (wrong) clinical information presented by the student.²⁶ Indeed, the thematic analysis showed that patients thought students were ‘knowledgeable’ and had confidence in what the student was saying, whereas clinicians were more likely to pick up on inaccurate, misrepresented information. This lends further support to the known advantages of multisource feedback (e.g. teachers, patients, family members of patients),^27–29 as recommended by the Accreditation Council of Graduate Medical Education Outcome Project.³⁰ Different perspectives capture more information and likely provide a more complete and complementary account of student skills.³¹ It is therefore recommended that future implementations of the program continue to procure feedback from multiple sources, including the patient voice.³¹

Similar to previous reports in medical trainees, student self-ratings poorly matched patient ratings of interpersonal skills (Lin’s CCC = 0.11).²⁵ This is in contrast to a previous report of optometry students overestimating their interpersonal skills compared to masked graders.²³ On the other hand, patients rated students more highly than clinicians, suggesting a possible leniency in patient evaluations. Higher ratings by patients have also been reported in medical trainee evaluations,^18,25 in contrast to other work focusing on medical specialists (e.g. anaesthesiologists)²⁶ where non-blinded teaching staff assigned higher scores than blinded, external evaluators. None of the patients in this study asked for a debrief, but future work could benefit from compulsory debriefings to investigate the reasoning behind individual patient and clinician ratings and possible reluctance to give poor marks.²³

Interestingly, while clinician ratings were lower on average compared to patient ratings, it cannot be ascertained from this study whether the grades of clinicians were primarily driven by a lack in the clinical knowledge of students, which was reasonably expected to contribute to their confidence in the ability of students (Item 6 in the DISQ, see Supplementary Material B). While there is a positive correlation between low overall clinician grades and a lack of confidence in the ability of students (Spearman r = 0.76, p < 0.001), this relationship also exists for patient ratings (Spearman r = 0.90, p < 0.001). Thus, it can only be concluded from the current data that a high overall rating of interpersonal skills is, at least partly, driven by increased confidence in the ability of students, regardless of whether the patient perceives, or the clinician assesses, that the student has the requisite and correct clinical knowledge.

Do the interpersonal skills of students improve with a repeated patient interaction?

This study found increased quantitative ratings of the interpersonal skills of students at the second visit, which implies an improvement in performance. A significant proportion of students reported trying harder to positively interact with the patient because they knew they were being evaluated. Awareness of being evaluated could lead to the oft-cited ‘Hawthorne effect’ (despite limited evidence thereof).³² It is not possible in this study to disentangle a possible Hawthorne effect or some kind of ‘participant reactivity’³² from other factors that could contribute to improved student performance (i.e. multisource feedback, active student self-reflection, an audiovisual recording as additional feedback, a practice effect).

Improvement in interpersonal skills with time is unlikely to be attributed to a single feedback mechanism.^23,33 Nevertheless, given the depth of student self-reflections and associated self-reported behaviour change (Supplementary Material E), providing feedback did help students improve their general interpersonal skills,²³ where the ‘correct’ behaviour could not be learnt (unlike hand washing for hygiene, for example)³² because each session involved a different clinical scenario and patient.

Note that students were not explicitly asked in their self-evaluation to comment on whether their performance improved at the second session, if applicable, and neither was it revealed to clinicians and patients whether it was the first or second session for the student. This was the rationale for pooling the qualitative feedback from both sessions in the thematic analysis. Nevertheless, in a small number of student responses (n = 6), it was clear that their self-reflection was written in the context of what happened previously. This suggests that students actively tried to address their previous feedback.

Given the improvement in the interpersonal skills ratings at the second visit, there is both qualitative and quantitative data to support the overall suggestion that students were motivated to reflect and act on feedback to improve their interpersonal skills. We do not know, however, which feedback methods in particular were useful as no specific reference was made by students in the anonymous survey to the written feedback or audiovisual recording as feedback. Future investigations would benefit from a more structured evaluation of each component of feedback to ascertain their relative usefulness.

Do patients, clinicians and students perceive the activity to be a feasible, useful and effective teaching and learning exercise for developing the interpersonal skills of students?

The study results indicate that patient and clinician feedback is feasible and is perceived to contribute positively to the learning experiences of students. Consistent with a previous study by Schmid et al.,⁷ most patients agreed that providing feedback about interpersonal skills was a worthwhile learning activity for students. Similarly, students self-reported that the activity was useful and that they used the information to improve their interpersonal skills (Supplementary Material E). Perhaps unsurprisingly given their chosen vocation, teaching clinicians considered their evaluation and feedback to be useful for student learning. Note that the clinicians in this study participated voluntarily and outside their usual teaching allocations and clinical commitments, which may not be feasible for embedding into an optometry student curriculum without the wider support of a network of clinicians with supervisory experience.

While the main aim of this study was to demonstrate feasibility and positive uptake of a novel teaching activity, the results provide some preliminary insight into potential anxiety-provoking situations for optometry students. The bimodal distribution in Figure 4C suggests two categories of students, based on whether awareness of being evaluated causes anxiety. It is not possible from this study to explore whether the anxiety stems from a general anxiety about all clinical and skill evaluations or from the novelty of this online patient interaction. Albeit scarce, there is some literature in other health professions (e.g. nursing) that corroborates our finding that clinical experiences involving instructor observation and evaluation are anxiety-producing for students, whether real or perceived.^34,35 In future, debriefing with students might therefore be useful to better understand the student experience and how best to support students.

Potential future improvements to the online evaluation and feedback program

Patients were free to provide information as they saw fit, rather than being given scripted information. It must be acknowledged that there are advantages of using trained, simulated patients (e.g. actors, laypersons, people with lived patient experience) for consistency in teaching and evaluation purposes. For example, simulated patients have been considered more effective than teaching communication skills to medical students by conventional lectures.^36,37 On the other hand, there is conflicting evidence for whether students prefer contact with real versus simulated patients for practicing their interpersonal skills.^11,15

Another potential program modification is blinded evaluations, given the potential for biases from previous interactions between teaching clinicians and students.²⁶ Teachers might develop preconceived opinions and impressions from prior evaluations that could influence their present evaluation of a student.³⁸ To address this, there is future potential to send video recordings of patient-student interactions for subsequent review by external evaluators.²⁶ The approach adopted in this study can also be used effectively for in-person evaluations, using real-time video monitoring from a separate room²⁵ or remotely.

This study was not designed to evaluate whether the improvement in interpersonal skills was retained and has lasting effects on student performance, which would require a longitudinal experimental design, longitudinal experimental design (e.g. surveying and evaluating optometrists in practice at different post-degree timepoints). The literature on whether short-term training and practice leads to long-term benefits is conflicting,^39,40 but nonetheless, future studies could address the longer term impact of experiential learning, specific to interpersonal skills.

To control for consistency and reduce variability in overall ratings, possible modifications are compulsory participant training and debriefing and more structured checklists for feedback provision (based on the themes identified in Tables 2 and 3). Neither was this study designed to evaluate a specific intervention, such as the provision of particular content or theory about adapting interpersonal skills in a telehealth setting. Hence, the baseline level of student knowledge about setting up for online interactions (e.g. ideal camera positions, being attentive to eye contact, checking microphone) or whether students proactively thought about such details in preparation for the online interaction remain unknown.

To explore these aspects further using an intervention to develop these online skills, future work could take advantage of new short courses and modules that provide specific guidance on telehealth and the digital setting. One such example of broader training for health professionals is a recent ‘telehealth foundations’ online learning experience designed specifically for entry-to-practice health professions students,⁴¹ to understand the ethical standards, legal responsibilities, and aspects of privacy, communication and professionalism that are relevant to the digital environment.

Conclusion

This study reports a novel, wholly online, multisource feedback and evaluation program to develop the interpersonal skills of optometry students – an area previously limited to in-person interactions. The program was perceived to be useful for students, feasible for patients and clinicians, and led to self-reported positive behaviour change and improved skills (as evaluated by patients, clinicians and students). Implementation of the program may help better prepare future optometrists for successful patient-student interactions and form part of the health professionalism curriculum to prepare clinicians for in-person and telehealth consultations.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary Data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/08164622.2023.2195049

Additional information

Funding

This work was supported by a University of Melbourne Faculty of Medicine Dentistry and Health Sciences Learning and Teaching Initiative Seed Funding grant to authors BN, MP, JN and AC. The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.