Remote and onsite scoring of OSCEs using generalisability theory: A three-year cohort study

Abstract

Introduction: Onsite scoring is common in traditional OSCEs although there is the potential for an audience effect facilitating or inhibiting performance. We aim to (1) analyze the reliability between onsite scoring (OS) and remote scoring (RS); and (2) explore the factors that affect the scoring in different locations.

Methods: A total of 154 students and 84 raters were enrolled in a single-site during 2013–2015. We selected six stations randomly from a 12-station national high-stakes OSCE. We applied generalisability theory for the analysis and investigated the perceptions that affected RS scoring.

Results: The internal consistency reliability Cronbach’s α of the checklists was 0.92. The kappa agreement was 0.623 and the G value was 0.93. The major source of variance comes from the students themselves, but some from locations and raters. The three-component analysis including Technical Feasibility, Facilitates Wellbeing, and Observational and Attention Deficits explained 73.886% of the total variance in RS scoring.

Conclusions: Our study has demonstrated moderate agreement and good reliability between OS and RS ratings. We validated the factors of facility operation and quality for RS raters. Remote scoring can provide an alternative forum for the raters to overcome the barriers of distance, space, and avoid the audience effect.

Practice points

• Onsite scoring is common in traditional OSCEs with potential audience effect.

• Audience effect means the presence of audience possibly facilitating or inhibiting performance.

• Facility operation and quality are the affecting factors in remote scoring.

• Remote scoring provides an alternative forum for assessment.

• The optimization of facility is important between the financial consideration and facility operation.

Introduction

Since its introduction in 1975 (Harden et al. 1975), the OSCE has become a popular tool in medical education to assess the core competencies of medical students (Short et al. 2009). Carefully planned content, known as blueprinting, and assessments can test curricula learning objectives (Dauphinee et al. 1994; Wass et al. 2001) and enable students to show how according to Miller’s pyramid of competence (Miller 1990). Traditionally, teachers rated students in close physical proximity—by their side, in the same room, or from behind a one-way mirror. From the perspective of social facilitation theory and the audience effect, the mere presence of an audience can lead to better performance by individuals in some cases (e.g. simple performance accuracy or well-trained tasks) but worse in others (e.g. if the individual is performing a complex task) (Bond and Titus 1983; Aiello and Douthitt 2001). Given the variability of students’ skills (a simple task for some might be perceived as a complex one for others) alongside the variability across OSCE stations (some are simply more complex than others), we cannot predict the impact of close proximity raters on the performance of students and standardized patients (SPs).

In the 1970s, one-way mirrors were used to observe the performance of students during OSCEs, which led to an examination of their effect on student performance. Despite psychological research suggesting that the performance of the task can be affected by an audience (Cohen and Davis 1973), research in medical education settings has reported no significant differences between physicians graded by an examiner in the same room and those graded by an examiner behind a one-way mirror (Corley and Mason 1976). Nevertheless, research suggests that one-way mirrors have some disadvantages, including intimidating participants, providing a limited view, and increasing noise and light pollution (Ford 2008; Sauro 2016). Additionally, the audio quality from behind the mirror can be too poor for the raters to score effectively, and adequate space is needed for their construction; two aisles to separate raters and students are necessary to prevent contact between them. In the past few years, high technology for more distant observations (Chan et al. 2014)—such as video-based (Sturpe et al. 2010), camera-assisted, or web-based scoring (Novack et al. 2002)—has been used to rate students’ performance.

Furthermore, one of the most important factors for OSCE scoring is reliability. Brannick et al. reviewed the reliability of OSCEs systematically in a total 188 alpha values from 39 studies and reported that the overall alpha across stations was 0.66, the overall alpha within stations across items was 0.78 and the generalisability coefficient was 0.49 (Brannick et al. 2011). Sturpe et al. (2010) examined the issue of intrarater reliability between real-time and video-based observation. Even though the same raters were used for each condition, 13.3% of the students that were rated as passing based on real-time observation were rated as failing based on video observation. In addition, 3.3% of students that were rated as failing in real-time observation passed when rated by video observation (Sturpe et al. 2010). Reliability is very important for the assessment, especially for critical decision-making in the high-stakes OSCE.

The challenge of large-scale, multisite examinations

Since the application of OSCEs as end-of-course assessments to their use today as part of the certification and licensing process (Reznick et al. 1996; Whelan 2000; Dillon et al. 2004; Trewby 2005), the issue of large-scale and multisite OSCEs has been debated and investigated. Research has examined issues such as the recruitment and training of raters and SPs, differences between facility settings across different sites, and the reliability of raters (Rahayu et al. 2016).

Although research is beginning to address the issue of reliability and validity of multisite OSCE examinations, few studies have investigated the reliability and validity between remote and onsite OSCE scoring systems. Chan et al. reported a preliminary study comparing OSCE scores for on-site (same room) and remote-site (webcam in another building) scorers (Chan et al. 2014). Although technical issues led to inconsistent data collection, they found high correlations for three of the six stations—history taking, physical examination, and management—using both systems, but global ratings varied greatly (Chan et al. 2014). They concluded that remote examination might be a feasible and acceptable way of assessing students’ clinical skills despite technological issues. However, this study reported preliminary data with insufficient evidence to clarify the inter-rater reliability and the validity to identify the confounding factors in a high-stakes OSCE.

In summary, the challenge is for raters to observe the students’ performance closely and clearly without interrupting their performance. The aims of this study are to (1) analyze the reliability between OS and RS raters, and (2) explore the factors that affect the accuracy of scoring in OS and RS locations. The research question is: Is there a difference in the assessment of medical student performance on the OSCE when undertaken via OS rating in the examination rooms and via RS rating out of the rooms?

Methods

Context

The national high-stakes OSCE for seventh year medical students has been held in Taiwan since 2013. It takes place across multiple sites during a six-day period annually. It consists of a twelve-station track that includes the broad areas of history taking, clinical reasoning, skills, management, communication, and counseling. Two circuits of students take the OSCE test in a day. Specific tasks at the 12 stations differ day by day, and are set by the Taiwan Society of Medical Education (TSME). The scoring sheet consists of a checklist including 10–15 items (scale 0-1-2) to rate specific skills at each station and a global rating score (1–5 points) for the overall score. All raters are qualified according to the requirements of the TSME, who also regulate the OS raters to ensure equality across sites. The scores and the decision for a pass or fail rating for each task are also determined by the TSME.

Study design

A three-year, quasi-experimental cohort study was undertaken. The Institutional Review Board Ethical Committee of Chang Gung Medical Foundation approved the study. Participants were seventh year medical students who participated in the national high-stakes OSCE at one hospital during the period of 2013–2015.

Six stations were randomly selected among the 12-station OSCE every day. Twelve raters were divided into two paired groups, one OS and one RS. The OS raters scored the students in the examining rooms and RS raters scored the same students out of the examining rooms observing them through two real-time video cameras in the central control room. Both cameras had zoom-in/zoom-out functions with 360° adjustment. A total of 154 students (male 63.6%) and 84 raters were enrolled (Table 1). During the study period, we analyzed 924 pairs of ratings.

Table 1. Participant demographics and locations.

Demographic data
			Raters locations
Year	Participants (n)	Male	RS	OS	Ds	Cs
2013	48	39(70%)	6	6	2	4
2014	46	25(54%)	6	6	2	4
2015	60	39(65%)	6	6	3	5
Total	154	103(66.9%)	18	18	7	13

C: Circuit; D: Day; n: numbers; OS: onsite scoring; RS: remote scoring.

We developed a questionnaire to examine perceptions by RS raters of the factors that affected their scoring. These 10 items focused on the performance and effectiveness of the hardware and raters’ experiences of their rating task using a 1–5 point Likert scale from complete disagreement to complete agreement.

Statistics and analysis

We calculated the results using SPSS.12.0 with students’ t-test and the chi-square test results and presented them with mean ± standard deviations (SD). The reliability of the checklists was calculated by Cronbach’s alpha. The agreement of the raters between the two groups for pass or fail was tested by the kappa value. We also analyzed the correlation of the checklist scores and global ratings. We applied G study by EDU-G for the three-year cohort to examine the reliability of the following facets: participants, circuits, days, raters, and OS-RS locations. We used the results from this to inform our D study (decision) for optimization of the process.

Principle component analysis (PCA) with varimax rotation was performed to analyze the factors affecting scoring as measured by the questionnaire. The intragroup and intergroup correlation between checklist scores and global ratings was tested by Pearson’s correlation. Significance was set at p < 0.05.

Results

OSCE ratings

No significant differences were found between OS and RS checklist scores; three-year mean checklist scores were 17.70 ± 4.22 in the OS group and 17.64 ± 4.42 in the RS group without significance (p = 0.42). Overall, the internal consistency reliability of the checklists was Cronbach’s α = 0.92. A significant difference was found for the mean global ratings: 3.59 ± 0.87 for the OS group and 3.51 ± 0.91 for the RS group (p < 0.01). The overall percentage of agreement for pass/fail ratings of OS and RS raters was 91.8% (848/924) and the kappa value was 0.623 (0.375–1.000).

The overall intragroup Pearson’s correlation of checklist scores and global ratings in each group was significant (p < 0.001, r = 0.642). It also showed significant intergroup Pearson correlation of checklist scores (p < 0.001, r = 0.851) and global rating scores (p < 0.001, r = 0.571).

Generalisability study by EDU-G

The analysis of variance indicated that the corrected components of variance for the three-year cohort data comprised: participants 10.844, raters 0.002, locations 0.003, participant-rater 1.578, and participant-rater-locations 6.594. The components of variance consisted of participants (57%), participant-raters (8.3%), and participant-raters-locations (34.6%) (Table 2). In the G study of differentiation, the variance of participants (10.844) had an absolute error variance for locations: 0.002 (0.2%); participant-raters: 0.263 (32.2%); and participant-raters-locations: 0.549 (67.3%) (Table 3). The three-year coefficient G value was 0.93 (0.95, 0.84, and 0.96 for each year, respectively). For the D-study, we fixed the participants and locations. Following this, we optimized the numbers of raters by increasing them from 6 to 11. The relative/absolute coefficient G values were 0.946/0.945, 0.841/0.838, 0.963/0.963, but could be achieved to 0.970/0.969, 0.906/0.904, 0.980/0.980 when extrapolating to 11 raters (Table 4).

Table 2. Analysis of variance among participants, raters and locations of scoring.

	Variance components
Year	n	P	R	L	PR	PRL
2013	48	13.914 (61.9%)	−0.161	0	0.508 (2.3%)	5.948 (26.5%)
2014	46	7.413 (37.2%)	−0.1	−0.032	4.503 (22.6%)	7.713 (38.7%)
2015	60	12.113 (69.4%)	0.002	−0.012	0.186 (1.1%)	4.071 (23.3%)

L: location of scoring; n: numbers of participants; P: participant; R: rater.

Table 3. Generalisability analysis of variance in the cohort period.

		Absolute error variance (%)							Coef_G
Year	DV							SD	Rel.	Abs.
2013	P:C	R	L	PR:C	PRL:C	CR	CRL
	13.914	0	0	0.085 (10.5%)	0.496 (61.6%)	0.1 (12.4%)	0.112 (13.8%)	3.13	0.95	0.95
2014	P:D	R	L	PR:D	PRL:D	DR	DRL
	7.413	0	0	0.751 (52.5%)	0.643 (45%)	0	0	2.723	0.84	0.84
2015	P:C	R	L	PR:C	PRL:C	CR	CRL
	12.113	0	0	0.031 (6.7%)	0.339 (73.7%)	0	0.09 (19.5%)	3.48	0.96	0.96
Total	10.844	0	0.002 (0.2%)	0.263 (32.2%)	0.549 (67.3%)			3.293	0.93	0.93

Abs.: absolute; C: Circuit; Coef_G: coefficient G; D: Day; DV: differentiation variance; L: location; P: participant; SD: standard deviation; R: rater; Rel.: relative.

Table 4. Optimization analysis by the increment of raters.

		G-study		Option 1		Option 2		Option 3		Option 4		Option 5
Year		Lev.	Univ.	Lev.	Univ.	Lev.	Univ.	Lev.	Univ.	Lev.	Univ.	Lev.	Univ.
	P:C	12	INF	12	INF	12	INF	12	INF	12	INF	12	INF
	C	4	INF	4	INF	4	INF	4	INF	4	INF	4	INF
	R	6	INF	7	INF	8	INF	9	INF	10	INF	11	INF
2013	L	2	INF	2	INF	2	INF	2	INF	2	INF	2	INF
	Observ.	576		672		768		864		960		1056
	G Rel.	0.946		0.953		0.959		0.963		0.967		0.97
	G Abs.	0.945		0.953		0.958		0.963		0.966		0.969
	P:D	23	INF	23	INF	23	INF	23	INF	23	INF	23	INF
	D	2	INF	2	INF	2	INF	2	INF	2	INF	2	INF
	R	6	INF	7	INF	8	INF	9	INF	10	INF	11	INF
2014	L	2	INF	2	INF	2	INF	2	INF	2	INF	2	INF
	Observ.	552		644		736		828		920		1012
	G Rel.	0.841		0.86		0.875		0.887		0.897		0.906
	G Abs.	0.838		0.858		0.873		0.886		0.896		0.904
	P:C	12	INF	12	INF	12	INF	12	INF	12	INF	12	INF
	C	5	INF	5	INF	5	INF	5	INF	5	INF	5	INF
	R	6	INF	7	INF	8	INF	9	INF	10	INF	11	INF
2015	L	2	INF	2	INF	2	INF	2	INF	2	INF	2	INF
	Observ.	720		840		960		1080		1200		1320
	G Rel.	0.963		0.968		0.972		0.975		0.978		0.98
	G Abs.	0.963		0.968		0.972		0.975		0.978		0.98
	P	154	INF	154	INF	154	INF	154	INF	154	INF	154	INF
	R	6	INF	7	INF	8	INF	9	INF	10	INF	11	INF
Total	L	2	INF	2	INF	2	INF	2	INF	2	INF	2	INF
	Observ.	1848		2156		2464		2772		3080		3388
	G Rel.	0.93		0.94		0.947		0.952		0.957		0.961
	G Abs.	0.93		0.939		0.947		0.952		0.957		0.961

Abs.: absolute; C: Circuit; G: coefficient G; D: Day; DV: differentiation variance; INF: infinite; L: location; Lev.: level; Observ.: observation; P: participant; SD: standard deviation; R: rater; Rel.: relative; Univ.: universe.

Questionnaire

Fifty-eight questionnaires were mailed to the RS raters and 51 were returned for analysis. (See Table 5 for details.) Factor analysis with significant Bartlett’s sphericity test (p < 0.001) was noted and Kaiser–Meyer–Olkin (KMO) value was 0.703. Ten items could be divided into three groups as a three-component model with eigenvalues >1.0 (ranging from 1.632 to 3.150).

Table 5. Questionnaire items, means (SD) examining remote scorers’ perceptions of the factors that affect their scoring.

No.	Items	Mean ± SD
1.	Easy operation of hardware.	4.47 ± 0.50
2.	Video-quality is sufficient for scoring.	4.39 ± 0.60
3.	Audio-quality is sufficient for scoring.	4.30 ± 0.71
4.	Camera setting is sufficient for scoring.	4.24 ± 0.71
5.	Easy distraction in RS.	3.02 ± 1.03
6.	Fewer interruption to students in RS.	3.82 ± 0.87
7.	Fewer interruption to SPs in RS.	3.82 ± 0.95
8.	Less psychiatric stress in RS.	3.98 ± 0.91
9.	Difficult to observe technical skills in RS.	3.45 ± 0.97
10.	Difficult to observe interview skills in RS	3.80 ± 0.85
	Total	3.93 ± 0.41

RS: remote scoring; SP: Standardized patient.

The three-component model is as follows: Component 1) Technical Feasibility, consisting of ease of operation (0.898), video-quality (0.891), camera setting (0.871), and audio-quality (0.773); Component 2) Facilitates Wellbeing, consisting of fewer interruptions to students for RS (0.908), fewer interruptions to Standardized patients (SPs) for RS (0.890), and less stress at RS (0.818); Component 3) Observational and Attention Deficits, consisting of difficulties in observing technical skills at RS (0.893), difficulty in observing interview skills at RS (0.877), and being easily distracted (0.545). The percentage of components 1, 2, and 3 that accounted for the variance was 30.412%, 24.381%, and 19.093%, respectively. The three-component analysis explained 73.886% of the total variance (Table 6).

Table 6. Principle component analysis with Varimax rotation.

	Initial Eigenvalues			Rotated Sums of Squared Loadings
Component	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	3.150	31.502	31.502	3.041	30.412	30.412
2	2.606	26.058	57.561	2.438	24.381	54.793
3	1.632	16.325	73.886	1.909	19.093	73.886
4	0.696	6.962	80.848
5	0.599	5.990	86.838
6	0.394	3.936	90.773
7	0.325	3.253	94.026
8	0.261	2.611	96.637
9	0.179	1.786	98.422
10	0.158	1.578	100.000

Discussion

We analyzed data from a three-year cohort study to ascertain the reliability and feasibility of remote and onsite scoring in a national high-stakes OSCE. Our findings build on the work undertaken by Chan et al. (2014), who demonstrated significant correlations between three (of six) OS and RS examiners’ checklist scores with more variable correlations between OS and RS examiners’ global ratings. Unfortunately, their preliminary study did not provide data on intrarater or inter-rater reliability. Moving the field forward, our study suggests that good reliability between remote and onsite scoring can be achieved not only between individual stations but also between global rating scores for intrarater and inter-rater groups. We further found that the major source of variance comes from the students themselves, with some also coming from the rating location. However, we found very little variation from the raters.

Testing knowledge and performance is difficult and complicated. We need the right measurement to differentiate the valid data and the confounders. Generalisability theory allows us to separate noise from signal, identify sources of noise, and devise ways to reduce its contribution to final results. The application of the theory provides a way to analyze the results of psychometric tests, such as the OSCE (Bloch and Norman 2012). Brannick et al. reviewed the reliability of OSCEs systematically and reported variable of the alpha values and the generalisability coefficient (Brannick et al. 2011). The results of high-stakes OSCE will be criticized if there is no good reliability. In our study, we selected six tasks from a twelve-station track to compare the differences between remote and onsite scoring. We obtained good G values and were able to extrapolate estimated G values if we had selected more than six tasks by optimization through a D study.

In this national OSCE test, the raters scored the students, but did not decide to pass or fail each of them immediately. The scores for passing or failing each task were determined by the TSME using the borderline group method with regression. The raters were trained and qualified by the regulations and requirements of TSME. Among the twelve task stations, TSME randomly assigned six raters from other testing sites to avoid the halo effect (Iramaneerat and Yudkowsky 2007). Our results suggest a good correlation between the checklist scores and the global ratings. The results showed very little variation from the raters and provided a way for us to review the recruitment and training program of the raters.

Facility operation and technological quality were factors that concerned the RS raters. We have therefore demonstrated how the use of high technology, remote video observation can provide an alternative forum for OSCE assessment, instead of OS observation. The most important aspect of concern regarding technology-assisted scoring for the raters concerns hardware feasibility. The advantages of RS scoring include less psychiatric stress for the raters, fewer interruptions to the students and standardized patients. Besides, the raters can observe detail steps such as suturing skills by the cameras. The most important aspect of concern for the raters concerns hardware feasibility. Even in the conventional two-aisle setting observation, the problem of audio quality from behind the mirror still exists to be concerned for the scoring but the mirror can provide at least one observational view to avoid the problem of camera-related video quality. We can increase the numbers of camera setting or set both cameras and mirrors if we want to provide more views to observe the students. However, it will increase the cost of hardware and the problem of facility operation for the raters. The optimization of facility setting is also an important issue for the scoring effectiveness between the financial consideration and the feasibility of facility operation.

Study strengths and limitations

As with all research, our study has limitations. Despite this being a national OSCE with many testing sites throughout the country, our study occurred at a single-site. Although our finding that RS appears as reliable as OS, further large-scale and multisite studies are needed to validate the advantages and disadvantages of remote scoring. Despite this limitation, our study has strengths, including the use of rigorous data collection methods that enabled us to randomize OSCE stations and systematically collect data over multiple time periods. Furthermore, we had sufficient data to undertake a G study to examine inter-rater reliability and a subsequent D study to optimize the ratio of raters to the number of stations.

Future research

Although research has addressed some of the issues, we believe that raters’ reliability remains an area for further research to unpack the wider range of factors underlying successful scoring. Additionally, training for raters is required to minimize rating errors and help them reach a consensus for grading to ensure fairness. Furthermore, given that we have demonstrated some of the benefits of remote assessment during OSCE, we have not yet addressed the financial feasibility of this method. Remote video observation and scoring requires expensive high-technology hardware and software that in turn requires regular maintenance. Finally, the security and safety of the cables and the Internet remains a concern. As such, a cost-benefit study should be conducted to better understand when and where OS and RS ratings are beneficial.

Conclusions

Our study has demonstrated moderate agreement and good reliability between OS and RS ratings. Remote scoring can provide a way for raters to dissolve the barriers of distance and space, and also avoid the interruption and bias of an audience effect because of the presence of the teachers or raters. The expanded use of reliable, high technology-assisted scoring can also provide training practice to junior raters without interruption to the students and standardized patients.

Glossary

Remote rating/scoring: Provides a way for the raters by the application of reliable user-friendly high technology to rate students’ performance, such as video-based, camera-assisted, or web-based scoring for more distant observations to overcome the barriers of distance and space, and also avoid the interruption and bias of an audience effect.

Audience effect: An audience effect comes from psychology and states that the mere presence of an audience can lead to better performance by individuals in some cases (e.g. simple performance accuracy or for well-trained tasks) but worse in others (e.g. if the individual is performing a complex task).

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Funding

This work was supported by Kaohsiung Chang Gung Memorial Hospital, Chang Gung Medical Foundation, Taiwan [grant number CDRPG8E0033].

Notes on contributors

Te-Chuan Chen

Te-Chuan Chen, MD, Division of Nephrology, Department of Internal Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan. School of Medicine, Chang Gung University College of Medicine, Tao-Yuan, Taiwan. Chang Gung Memorial Hospital Linkou Branch, Chang Gung Medical Education Research Centre, Tao-Yuan, Taiwan.

Meng-Chih Lin

Meng-Chih Lin, MD, School of Medicine, Chang Gung University College of Medicine, Tao-Yuan, Taiwan. Division of Pulmonary Medicine, Department of Internal Medicine, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan.

Yuan-Cheng Chiang

Yuan-Cheng Chiang, MD, School of Medicine, Chang Gung University College of Medicine, Tao-Yuan, Taiwan. Department of Plastic and Reconstructive Surgery, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan.

Lynn Monrouxe

Lynn Monrouxe, PhD, Chang Gung Memorial Hospital Linkou Branch, Chang Gung Medical Education Research Centre, Tao-Yuan, Taiwan.

Shao-Ju Chien

Shao-Ju Chien, MD, Division of Pediatric Cardiology, Department of Pediatrics, Kaohsiung Chang Gung Memorial Hospital, Kaohsiung, Taiwan. School of Traditional Chinese Medicine, Chang Gung University College of Medicine, Tao-Yuan, Taiwan.