An integrable, web-based solution for easy assessment of video-recorded performances

Abstract

Assessment of clinical competencies by direct observation is problematic for two main reasons the identity of the examinee influences the assessment scores, and direct observation demands experts at the exact location and the exact time. Recording the performance can overcome these problems; however, managing video recordings and assessment sheets is troublesome and may lead to missing or incorrect data. Currently, no existing software solution can provide a local solution for the management of videos and assessments but this is necessary as assessment scores are confidential information, and access to this information should be restricted to select personnel. A local software solution may also ease the need for customization to local needs and integration into existing user databases or project management software. We developed an integrable web-based solution for easy assessment of video-recorded performances (ISEA).

Keywords:

• education assessment
• assessment software
• video-based assessment

Background

Clinical performance can be assessed by expert observation using assessment sheets, but this is problematic for two main reasons. Firstly, the identity of the examinee can influence the assessment.^1,2 This can be due to personal relations and prejudices,¹ which can be a large problem in smaller countries or within smaller medical specialties. Age and job title have also been shown to influence assessment scores.² Secondly, assessments are time-consuming and need experts to be at a specific location and at a specific time. A feasible way of addressing these problems is by video recording the clinical performance.

Recording videos can be unproblematic if designed carefully.³ However, the assessment procedure can easily become problematic and lead to logistic problems. Typically, hard drives or flash drives with videos and assessment sheets are sent to all examiners. When these are returned back to the data manager, some sheets may be incomplete and in some cases, may even be forgotten, requiring that the data manager go through a tiresome process to obtain all assessment scores. Finally, when all the cases are fully assessed, paper sheets are transcribed into a database in duplicate to avoid any typing error. This nonautomated assessment procedure is costly, time-consuming, and error-prone.

Existing software enable an automated process^4,5 but are not optimized for video-recorded assessments and are problematic since data are stored at external institutions; instead, data should be stored locally as assessment scores are confidential information, and access to such should be restricted to course personnel (or study personnel, in the case of research).^6,7 We developed an integrable web-based solution for easy assessment of video-recorded performances (ISEA).

Design

The design of ISEA is based on two user interfaces: one for the administrator and one for the examiner. The administrator manages projects, examiners, videos, and assessment sheets. A “project” is defined as a group of examiners who assess videos based on the same assessment sheets. The assessment sheet is composed of questions, each having a number of scoring items. Within a project, examiners are assigned one or more videos that, if necessary, may differ between the examiners. Results from all assignments are available from the administrator interface. User profiles and results are stored in a local database. Depending on the legal requirements in each project, videos can either be uploaded to an external video storage platform, such as Vimeo,⁸ or stored internally within an institution, and ISEA is then provided with a link to the uploaded video. Examiners have their own personal page with a list of assigned videos. An assigned video is seen side-by-side with the assessment sheet, to enable simultaneous assessment (Figure 1). When the assignment is completed, it is locked and scores cannot be changed. The results are available for the administrator and can be exported in a semicolon-separated or a comma-separated file format (.csv).

Figure 1 Simultaneous view of the video and the assessment sheet, as seen by the examiner.

We used PHP: Hypertext Preprocessor (The PHP Group) and My Structured Query Language (MySQL) as the server-side scripting language and database, respectively, as these are the most popular and commonly used^9,10 and may facilitate an easier implementation and integration to existing software. We wanted the examiners to be able to use our web solution in any digital device anywhere, so the interfaces were developed using very simple Hyper Text Markup Language (HTML) and Cascading Style Sheets (CSS) that are compatible with a large range of internet browsers, including smart phones and tablets. No add-on software is needed for ISEA. This clean and simple interface may ease integration to existing platforms. Adobe Dreamweaver CS5 (Adobe Systems, Inc., San Jose, CA, USA) was used for development. A download link for ISEA is available in the references.¹¹

Discussion

Video-based assessment of clinical performance may help eliminate bias related to the identity of the examinee by masking the identity, which can be achieved by adjusting the camera-angle or by editing the video afterwards.^1,12,13 However, the identity is rarely perfectly hidden – the face may be blurred and the voice may be distorted, but the identity may still be guessed if the examiner is familiar with the examinee and is aware of his or her specific characteristics, such as manner of speech, preferences of clothes, or characteristic body figure or posture. Also, in some cases, facial expression, eye contact, or the finer aspects of the voice, such as the tone, may be relevant for the assessment itself. A step toward better blinding and the reduction of biases emerging from personal relations may therefore be achieved by, not only use of videos but also, collaboration with examiners in other institutions and other countries. Video-recorded assessments provide flexibility for the examiners as they no longer need to be at a specific location at a specific time but may be at any location and assess the clinical performance at any time.¹² Implementing video-based assessments is, theoretically, not a problem, but practically, this is a tiresome process with potential pitfalls. Few studies have compared direct observation with video-based assessments.^13,14 Ma et al assessed central venous catheterization using both direct observation and video recordings and found no differences in ratings, and they reported that importance should be given to practical aspects.¹³ Kneebone et al explored the feasibility of a video-based assessment procedure with remotely placed examiners, and they also reported that a future improvement would be an automated approach for easy assessment of video-recorded performances.¹⁴ We developed ISEA, an integrable web-based solution for easy assessment of video-recorded performances that provides a format that can be stored locally, is customizable into existing systems, does not require any add-on software, and that is compatible with a large range of web browsers. ISEA may ease the assessment process of video-recorded performances and collaboration with examiners from other institutions. ISEA was used in the assessment process of a recent project,¹⁵ and the general comment from the examiners was that ISEA was easy to use. Also, ISEA can be customized to local systems, for integration to an existing project or user management systems that are used to administer courses and research. A few limitations should be noted. ISEA is an integrable application and not an “out of the box”, stand-alone application, and the implementation and customization may thus require some technical assistance. Customization, however, can be advantageous and even necessary for some types of education and research projects.⁷ We did not describe the implementation process in this paper as the implementation process is more technical and may vary greatly between institutions, depending on the software used. In its simplest form and without much effort, ISEA can be implemented using only a user database for the administrator and the examiners. In our experience, implementation has been unproblematic, and the limited technical assistance needed to implement ISEA is a small price to pay for a smooth and easy assessment process.¹⁵

Acknowledgments

We thank the University of Copenhagen for an education research grant (Undervisningskvalitetspuljen) for covering our publication costs.

Disclosure

The authors report no conflicts of interest in this work.