Access to Mathematics for Visually Disabled Students Through Multimodal Interaction

Abstract

Mathematics relies on visual forms of communication and is thus largely inaccessible to people who cannot communicate in this manner because of visual disabilities. This article outlines the Mathtalk project, which addressed this problem by using computers to produce multimodal renderings of mathematical information. This example is unusual in that it is essential to use multiple modalities because of the nature and the difficulty of the application. In addition, the emphasis is on nonvisual (and hence novel) modalities. Crucial to designing a usable auditory interface to algebra notation is an understanding of the differences between visual and listening reading, particularly those aspects that make the former active and the latter passive. A discussion of these differences yields the twin themes of compensation for lack of external memory and provision of control over information flow. These themes were addressed by: the introduction of prosody to convey algebraic structure in synthetically spoken expressions; the provision of structure-based browsing functions; and the use of a prosody-based musical glance based on algebra earcons. The addition of prosody, when compared to a traditional method of presenting spoken algebra, was experimentally shown to increase the recovery of algebraic structure, enhance the retention of content, and reduce mental workload. These three factors can be said to compensate for the lack of an adequate external memory. Evaluations showed that the browsing functions and associated command language gave the fast and accurate control over information flow that is necessary for active reading. The algebra earcon was experimentally shown to convey the presence, location, and size of algebraic constructs within an expression in a manner that might be used as a rapid glance. Finally, an evaluation of the integrated components showed that the design principles derived from the Mathtalk program can give a more usable, active reading of algebra notation than that possible with traditional methods.