Human-computer multimodal interface to internet navigation

Abstract

Purpose

Propose and develop new multimodal interfaces that allow people with motor impairments to control mass use applications in a natural way through gestures and voice.

Methods

A multimodal interaction interface was developed for using Google Chrome, Gmail and Facebook applications through gestural and verbal commands. The interface activates mouse and keyboard commands from the processing of voice signals and videos of the user’s head movement. The interface does not disable traditional keyboard and mouse functions; moreover, it only requires a webcam and a microphone, which are usually built into portable computers.

Results

The tests were performed on three groups of people: young adults, older adults and people with motor impairments. The verbal interaction was tested on a total of 189 voice commands with an average performance of 75.7% and a total of 105 dictations. The dictations had an average of 13 words; the system had a performance of 81.1%. Moreover, the gestural interaction activated 126 commands without errors using a drop-down menu; a click was activated 84 times with a success rate of 70.2%.

Conclusion

The motor impairments group especially valued the option of using Google Chrome, Gmail and Facebook without physically manipulating a mouse and keyboard. This group showed a greater preference for verbal control than for gestural control. An adaptation period is required for the adults group to acquire greater skill in using the interface. The young adults group preferred the types of interactions they are accustomed to due to their familiarity with Information and Communication Technologies (ICT); they considered the interface fun.

Implication for rehabilitation

• Computers and their applications were conceived with unnatural interaction mechanisms, such as the keyboard and mouse, which prevent their use by people with psychophysiological limitations or digital literacy. For this reason, the need arises to design new natural interfaces commanded by gestures and voice.

• It is necessary to develop low-cost interfaces that can control mass-use applications such as Google Chrome, Facebook and Gmail, which do not require additional hardware using webcams and microphones, which are usually integrated into laptops.

• The development of these multimodal interfaces improves the quality of life of people with motor impairments, allowing them to have access to Information and Communication Technologies (ICT).

Keywords:

• Human-computer interaction
• speech recognition
• computer vision
• internet vavigation

Acknowledgment

The authors thank the Graduate Program of the School of Electrical and Electronic Engineering and the Perception and Intelligent Systems (PSI, in Spanish) research group of the Universidad del Valle.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was financed with resources of the Internal Call to Support the 2017 Graduate Students of the Universidad del Valle [code CI-2940].