Innovative electronic exams with voice in- and output questions in medical terminology on a high taxonomic level

Abstract

Background: Computer-based assessments can improve quality and economy of university exams. Using the ‘Online Examination System’ (OES), for the first time questions can be implemented which are specifically designed to test the examinees’ voice-related knowledge and skills in electronic exams of History, Theory and Ethics of Medicine on all taxonomy levels.

Aims: To meet the specific learning objectives and to increase exam quality and economy with regard to testing student pronunciation and listening skills, the OES was to be prepared to handle sound output and vocal input.

Method: The usage of headsets reduces the inevitably occurring acoustical disturbances caused by sound file replay. Problems of acoustic disturbances during vocal recordings can be overcome by introducing synchronized micro-exams at the beginning of the assessments.

Results: The majority of participating students accept these types of electronic exams. Evaluation procedures of the respective items still require human intervention, but are increased in objectivity due to anonymity. Accelerated exam compilation, less staff interaction and assessments on a high taxonomic level increase the exam efficiency and quality.

Conclusions: The integration of sound in- and output facilities in electronic assessment systems is capable of increasing both test economy and objectivity.

Introduction

Computer-based assessments gain increasing importance in university education. This development is supported by at least the following factors: (1) affordable computer equipment (even for several hundred installations), (2) easy to use, powerful and safe software solutions, (3) acceptance of the computer as appropriate pedagogic device on both the teachers’ and learners’ side (Ogilvie et al. 1999), (4) the possibility to save money and time during planning, performing and evaluating examinations and (5) the understanding that a sensible use can increase assessment quality.

At the RWTH Aachen University, a modular software solution is being developed since 2001 and has been used for approximately 100,000 internet-based individual assessments within companies nationwide and several thousand university student assessments since then. The OES is open to all faculties of the RWTH Aachen University, and therefore is installed in a central lecture hall equipped with 200 identical installations of computer systems that possess a network connection to a dedicated server. For security reasons, the latter is, for the time the assessments last, disconnected from any outgoing and incoming internet travel. In the work in progress reported here, this ‘Online Examination System’ (OES) is launched in the Institute for History, Theory and Ethics of Medicine of the Medical Faculty as standard undergraduate assessment system, to replace former paper exams.

Table

Practice points

Electronic questions with sound capabilities are useful in disciplines that require a proper use of language, pronunciation and social interaction. Learning goals that include pronunciation and listening comprehension of medical terms can be tested with electronic assessments on a high level of reproducibility. The realization of voice output to recognize proper terminology use is comparatively simple (standard sound file formats, headsets). Assessment questions that make use of voice input require additional organization (recording synchronization during the exam, possibility to repeat input and audio recording software integrated in exam software). The automated evaluation of voice input questions is not possible up to now.

In medical settings oral communication has to be quick and precise. The correct application, pronunciation and listening comprehension of medical terms are necessary skills required for medical professionals. Therefore, the learning goals and objectives for the targeted learners (Kern et al. 1998) include the pronunciation and listening comprehension of medical terms, which can be broken down to the following specific educational objectives, which, of course, are at the same time assessment topics:

1. Knowing and applying the special pronunciation rules of latin and greek terms, prefixes, word roots and endings in medical terminology

2. Ability to identify medical terms in listening comprehension

3. Ability to pronounce and use correctly medical terms

4. Basic facts of medical history

5. Bioethical rules and principles.

However, for the first three objectives, adequate questions cannot be asked in written exams in a sensible manner. Due to limited staff resources, it is impossible to individually test the oral skills of about 320 examinees each year. Therefore, the decision was made to increase the technical effort and combine the advantages of oral and written exams in an electronic exam, preceded by two test runs. This decision was based on electronic exam quality and economy:

• In comparison to standard paper tests (largely multiple choice, fill-in the blanks and free text), the variance of question types is greatly increased. The OES offers more than 10 question types with several subtypes each and automatic evaluation.

• The usage of pictures is not restricted to static grayscale printer quality, but may incorporate animations, colours, context sensitive annotations, etc. in high quality. The integration of flash-based routines enables interactive add-ons, such as virtual microscopy.

• The screen layout and design are optimized to gain maximum student alertness during the exam.

Sensible usage of media

Another three aspects prove extremely useful for the testing of sound-related knowledge and competences of the students and increase the taxonomy level (Krathwohl 2002) of the respective questions:

• Sound files, impossible to use in paper exams and at least difficult to use in oral exams, can be integrated easily. Questions on the lower taxonomy level ‘knowledge and comprehension’ can thus be formulated using voice output. Example: ‘Please listen to the following four pronunciations of the word ductus epididymidis and check the sample that reflects the correct pronunciation’ (Four sound files can be played at will and can be individually marked by a mouse click.) (Figure 1).

• Also impossible in paper exams is an innovative type of question using a microphone for voice input. Example: ‘Please use your microphone and speak the phrase recessus costodiaphragmaticus with correct pronunciation’. This type mainly targets at the medium taxonomy level ‘application and analysis’.

• Video clips that correlate optical and acoustical information are used for questions on the upper taxonomy level ‘synthesis and evaluation’. Example: ‘Please watch the video clip of a patient interview. Give a reasoning in the text box why the orthopaedic surgeon has not succeeded in gaining the patient's confidence’ (short videoclip shows the surgeon actor sitting opposite to the patient actress, without eye contact, using almost only greek terminology, and calling the next patient without a proper termination of the interview).

Playing unsynchronized sound from sound files or video clips during an exam can cause disturbance. This is avoided by the use of headsets, which at the same time provide a microphone for voice input to be digitized and saved. A drawback is that voice inputs can cause disturbing acoustical noise and may be used to provide hints for students sitting in the neighbourhood. To overcome this, we implement a synchronization procedure for all voice input questions, the number of which is restricted to a maximum of three per exam. This synchronization requires that the respective questions are bundled into a separate micro-exam that is performed by all students at the beginning of the whole exam. This micro-exam is programmed to present these questions synchronized to all students. Each of the questions possesses a structure based on the voice input example question given above: ‘In this question, you are asked to speak the phrase recessus costodiaphragmaticus with correct pronunciation. Please watch the countdown timer display. After it counted to zero, use your microphone and clearly speak that phrase once. The recording will last exactly four seconds. After a pause of 5 s (see timer display), clearly speak that phrase once again’. Since students practice on these types of question in preliminary tutorials, they become used to both the instructing text and the process. By implementing the synchronized micro-exam, all students must talk at identical time intervals, which avoids acoustical disturbance. Statistical methods of collusion detection (Ercole et al. 2002) do not apply. Therefore, it is important to install different methods to prevent cheating. Since the lengths of the recording intervals are adapted to the speaking time of the requested voice input (plus a tolerance of about 25%), it is impossible for the students to start an attempt of deception by listening to the neighbour's pronunciation and repeating it. And since the recording is offered twice in each of the micro-exam's questions, students may correct a possibly bad first try. Finally, because during both of the recording sessions students have to enter a voice input, attempts of deception like ‘silently listening during the first recording session and repeating what was heard during the second one’ are also prevented. However, we find it necessary to offer two recording intervals and will evaluate these by always selecting the better sample.

Figure 1. The screenshot shows the typical OES question layout, here with an example of a voice output question. Each click on one of the blue play buttons will start the sound file replay in the student's headset. This can be repeated at will. In the example shown here, the choice made can be undone and changed at will, and the question can be revisited as often as needed. The upper area of the screen gives some information on the test status and the remaining time, and the bottom area provides buttons for navigation and user interaction with the system. Since the main target group of the OES are German speaking students, the interaction buttons with the system are in German. Personal data have been rendered unrecognizable.

The audio player, which we use to play the recorded files, is administrated in the resource section of the e-assessment system. The technology we use to realize the play sound and record sound interaction tasks is ‘Adobe Flash CS4’ (Adobe Systems Inc., San Jose). The interface of the player is designed in Flash and the source code is implemented in action script. The player works as a container for the audio files and can be modified to play also high-definition movies or animations.

Evaluation aspects

The OES is able, as most computer-based assessment systems are, to evaluate answers to the majority of all question types automatically and to assign the respective points based on a pre-defined calculus. This does not apply to the question types ‘free text entry’ and ‘voice input’, both of which must inevitably be evaluated by a skilled person. In order to maximize efficiency and objectivity, the evaluation procedure for free text entries is as follows: After the exam, the person in charge of the evaluation is being presented all text-box inputs one-by-one on a computer display. After comparing with the sample solution, the respective points are granted. This procedure is highly objective because it is impossible to identify the students’ identities, and it is time-saving since there is no page turning and no guessing of bad handwritings.

As for voice input questions, a similar procedure applies because an automated speech recognition (e.g. Nagy et al. 2008) does not reflect the learning aims. However, there is less anonymity because individual students could be identified by their voices. Since identification, which is always given in oral exams, is greatly reduced here by the amount of several hundred voice records, we estimate this small disadvantage as negligible. The record may be used in arguments during post-exam reviews for objective evidence on how the respective answer was given.

The acceptance and usability of the OES was evaluated by participating students by means of a questionnaire as follows: 75.0% possess a positive attitude towards electronic assessments with the OES, 19.6% had no preference to either electronic or conventional assessments and 5.4% stated that they would prefer conventional paper-and-pencil assessments in the future.

Discussion

The OES exams executed so far have revealed a great acceptance among students and teachers throughout all ages. Student nervousness when dealing with computers in exams and possible user interface battles has been reduced by offering voluntary test exams under real conditions. After a start-up phase in which teachers must learn the system handling and have to take care of establishing a pool of questions in the system's database, the usage of online exams bears advantages in economy and test-quality. Costs will be reduced by accelerated compilation of exams from pool questions, by fast and less staff-intensive exam evaluation, and still others. Both the possibility to implement motivating and relevant questions on a high taxonomy level and the use of objective evaluation procedures increase the test quality.

The OES's new modes of sound-based questions mainly address disciplines that require a proper use of language, pronunciation and social interaction. Therefore, the OES was implemented and tested for assessments of the Institute for History, Theory and Ethics of Medicine, which are offered to students of medicine and dentistry. The new question types can combine advantages of written and of oral exams in electronic exams. The fact that these questions cannot be evaluated automatically is, to our opinion, a small price to be paid for an objective and economic exam.

Acknowledgements

Dr Patrick Blum (inside Business Group Germany, Aachen, and Institute for Medical Statistics, RWTH Aachen University) supported the development and the operation of the OES. The helpful discussions with the ‘Master of Medical Education’ (MME) team, Germany, are greatly appreciated.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.

Additional information

Notes on contributors

Martin Baumann

MARTIN BAUMANN is an assistant professor at the Helmholtz Institute.

Jan Steinmetzer

JAN STEINMETZER is an assistant lecturer at the Institute for History, Theory & Ethics of Medicine.

Mazdak Karami

MAZDAK KARAMI is computer science student, co-worker at the Helmholtz Institute.

Gereon Schäfer

GEREON SCHÄFER is scientific assistant and teaching coordinator at the Institute for History, Theory & Ethics of Medicine.