Inside pandora’s box: a systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs

Abstract

Introduction: People with disabilities or special needs can benefit from AI-based conversational agents (i.e., chatbots) that are used for competence training and well-being management. Assessing the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understanding their potential long-term benefit. This in turn will help to increase adherence to their use, thereby improving the quality of life of the large population of end-users that they are able to serve.

Methods: Following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) methodology, we systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots using the from Scopus and the Web of Science electronic databases. Using the Boolean operators (AND/OR) the keywords chatbot*, conversational agent*, special needs, disability were combined.

Results: Revealed that only 15 of 192 papers on this topic included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the perceived quality of interactions with chatbots.

Conclusion: Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The current lack of reliable tools and systematic methods to assess chatbots for people with disabilities and special needs is concerning, and ultimately, it may also lead to unreliable systems entering the market with disruptive consequences for people.

Implications for rehabilitation

• Chatbots applied in rehabilitation are mainly tested in terms of clinical effectiveness and validity with a minimal focus on measuring the quality of the interaction

• The usability and interactive properties of chatbots applied in rehabilitation are not comparable as each tool is measured in different way

• The lack of a common framework to assess chatbots exposes people with disability and special needs to the risk of using unreliable tools

Keywords:

• Chatbots
• conversational agents
• assistive technology
• people with disability
• people with special needs
• quality of interaction
• usability
• user experience

Disclosure statement

The authors report that they have no conflicts of interest.

Additional information

Funding

This study was supported by DGTCSI-ISCTI – Directorate General for Management and Information and Communications Technology, Superior Institute of Communication and Information Technologies, Ministry of Economic Development, Rome, Italy, under grant Project “eGLU-box PRO” on October 6, 2019.