Kinect-for-windows with augmented reality in an interactive roleplay system for children with an autism spectrum disorder

ABSTRACT

Children with autism spectrum disorder (ASD) often find it difficult to communicate using body language and facial expressions to respond appropriately socially. This negatively affects the quality of their everyday social life. Children with ASD can learn to socially interact with virtual three-dimensional (3D) characters in animated games enhanced with augmented reality (AR). They can learn about body language and facial expressions through roleplay. This teaching method can effectively improve the interactive social skills of children with ASD and reduce the fear and anxiety which they typically experience when they face real people. However, most current virtual roleplay models focus only on predefined body language and facial expressions limited by the training system. These systems lack immediacy and a sense of genuine interaction for children with ASD. Therefore, we wanted to create a system that will allow trainers and therapists to control 3D virtual characters that are able to produce in real time the full range of natural body language and facial expressions appropriate for males, females, and even special cartoon characters. Based on the multiple baseline design across subjects, the data indicated that real-time Kinect Skeletal Tracking (KST) System can help the children with ASD improved their social interaction.

KEYWORDS:

• Autism spectrum disorder
• augmented reality
• interactive learning environment
• real-time Kinect Skeletal Tracking System
• Wizard of Oz experiment

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

I-Jui Lee

Dr. I-Jui Lee is a designer, artist and researcher with expertise in ergonomics and design methods, including product design, furniture carpentry, interaction design, and vocational education and training research. His research interests are in the areas of Augmented Reality, tangible user interfaces, human-computer interaction, computer-supported collaboration and design for children. A central theme of his research is exploring design spaces of novel interactive technologies that support people's needs by expanding ways they perceive, interpret and interact with the physical world.