Innovative computer technology in music-based interventions for individuals with autism moving beyond traditional interactive music therapy techniques

Abstract

Individuals with autism spectrum disorders (ASD), who exhibit developmental limitations in social-emotional interaction and communication, are widely reported to respond positively to music therapy interventions that incorporate active and improvisational techniques. The fundamental elements of music have been shown to bypass both cognitive and language impairments to help facilitate communication skills, form social relationships and promote cognitive progression. In recent years, computer technologies such as mobile devices and interactive motion capture systems are being developed to provide new approaches to music interaction. When included into new or existing music interventions, digital technology can enhance the music playing experience and motivate children to interact differently with their environment through providing novel and engaging approaches to music interaction and creativity. This review highlights the core symptoms of ASD and presents evidence that supports the use of music therapy techniques within the autistic population, underpinned by neurological studies reporting on benefits of musical intervention. Further to this, it is explained how new technologies such as mobile devices and multi-sensory computer systems can move beyond traditional music instruments to add additional dimensions to music therapy sessions. Finally, suggestions of future research are made alongside considerations in designing new and novel auditory interventions.

Keywords:

• Autism
• music therapy
• sound therapy
• auditory processing
• assistive technology
• tools for therapy
• multi-sensory

PUBLIC INTEREST STATEMENT

Individuals with autism spectrum disorder (ASD) are characterised as having developmental limitations in social-emotional interaction and communication, alongside displaying repetitive behaviours and interests. Autism is a lifelong condition and so the application of effective therapies to develop these core impairments is essential to promoting future independence and improving quality of life. Those with ASD have been widely reported to respond well to interactive and improvisational music therapy techniques, helping facilitate communication and emotional interactions that promote cognitive progression and develop social relationships. In recent years, computer technologies have been developed to provide new forms of music interaction, enhancing music playing experience, creativity and motivating those with autism to interact differently with their environment. This review highlights how technology can be used to move beyond traditional musical instruments to add additional dimensions to music therapy sessions. Finally, suggestions of future research are made alongside considerations in designing novel auditory interventions.

Autism spectrum disorders are characterised through impaired development in social interaction, communication and repetitive behaviours and interests (American Psychiatric Association, 2013). The condition is heterogeneous, and despite these symptoms being commonly identified with ASD, the extent of these difficulties are unique to the individual (Lord, Cook, Leventhal, & Amaral, 2000). Commonly, children with autism can experience complications in tactile, visual and auditory processing, with over 96% of children with ASD experiencing hypersensitivities in multiple domains. Sensory processing dysfunction is not specifically limited to individuals on the autistic spectrum; however, it does appear to more prevalent within this population when compared to other developmental disabilities (Marco, Hinkley, Hill, & Nagarajan, 2011; Pfeiffer, Koenig, Kinnealey, Sheppard, & Henderson, 2011). Sound sensitivity is an especially poignant and common issue for those with ASD. Research conducted by Greenspan and Wieder in 2000 found that all 200 participants with ASD displayed evidence of auditory processing impairments (Greenspan & Wieder, 1997). These receptive abnormalities can provoke atypical self-regulatory behaviours which may be observed as aggressive or autonomic fear responses such as covering ears, crying and self injury from blows to the ears (Stiegler & Davis, 2010). Unfortunately these profound aversions are reported to be provoked by common environmental sounds (Koegel, Openden, & Koegel, 2004). By avoiding challenging acoustic environments, children with ASD will experience increased isolation and further impairment in natural and social communications.

Early intervention is essential for children with ASD, with evidence supporting intervention for children as young as 3 years old (Narzisi, Costanza, Umberto, & Filippo, 2014; Webb, Jones, Kelly, & Dawson, 2014). Early brain development not only includes the emergence of associated abnormal neural connections and an exponential rate of brain growth, but also a period of increased brain plasticity. Early therapy can therefore influence the development of brain systems implicated in social, emotional and communication behaviours (Rogers, 1996; Sullivan, Stone, & Dawson, 2014). As well as pharmaceutical (Santosh & Singh, 2016) and dietary (Sathe, Andrews, McPheeters, & Warren, 2017) treatments, there are a wide range of interventions that include speech/language therapy (Chenausky, Norton, Tager-Flusberg, & Schlaug, 2016), occupational therapy (Bumin, Huri, Salar, & Kayihan, 2015) and sensory integration (Case-Smith, Weaver, & Fristad, 2015).

Despite the complications with auditory processing, music therapies remain a popular form of intervention for individuals with autism. In 2017, there were approximately 13,000 music therapists in Europe and the United States combined. There is extensive evidence within literature that supports the use of music in promoting communication, social-emotional, motor and neurological development (de Boer-Ott et al., 2004; Watson, 2007). Brown (1994) presented four distinct points as to why music, when used as a therapeutic tool, is vital when working with autistic children. Firstly, musical structures exhibit similarities with both the creative and organisational elements required for a person’s functional interaction with the world. In addition to this, musical therapies can be used to promote the social relationships those with autism find difficult to cement. Thirdly, the therapists can take advantage of fundamental musical components such as pitch and rhythm to establish emotional communication with the child. Finally, Brown suggests that the emotional qualities of music form a direct connection with a child that can completely bypass cognitive and language impairments. It is also these qualities that have led to music therapy being used as a means of diagnosis for the autistic population. The non-verbal nature of music is a significant tool of assessment for those with communication disorders, or when working with pre-verbal communication systems (Wigram, 1999).

Traditional music therapy techniques involve the use of acoustic instruments such as guitar, piano, pitched and non-pitched percussive instruments and singing (Magee, 2006). However, traditional instruments can be difficult to use for those with cognitive or physical disabilities and this may impose limitations upon therapy sessions (Thaut & Hoemberg, 2014). The introduction of music technology has provided novel approaches to music therapy within not only the autism population but for a large number of other neurological and physical disabilities (Burland & Magee, 2014). Technology such as digital instruments using musical instrument digital interface (MIDI) (Nagler, 1998) and alternative input devices such as switches (Magee & Burland, 2008) are effective in therapeutic applications for individuals with complex physical or neurological needs. Empirical research has observed that music technology not only enhances musical creativity (Magee, 2006) but also aids in the improvement of physical rehabilitation (Paul & Ramsey, 1998). Within the past decade, new approaches to music therapy and technology, such as fabric-based audio/visual control surfaces (Vazquez, Cardenas, Cibrian, & Tentori, 2016), interactive haptic devices (Loureiro, Amirabdollahian, Topping, Driessen, & Harwin, 2003) and motion capture sensors (Ichinose et al., 2016), emphasise the importance of integrating multi-sensory stimuli to enhance the music playing experience and to motivate children to interact differently, promoting motor control, body awareness, social-engagement and creativity (Hillier, Greher, Queenan, Marshall, & Kopec, 2016; Tam et al., 2007). But with all the possibilities new technology provides for the discipline of music and sound therapy, it also presents new challenges in both design and application and poses some interesting questions: is the technology easy enough to use but also challenging enough for prolonged engagement? Are the controls intuitive and understandable? Which core ASD impairments/symptoms does this address? Can it fit into pre-existing therapy frameworks or is there a need for new novel intervention techniques?

The aim of this article is to highlight the specific advantages which new computer assisted technology may offer, supporting interactive music therapy in addressing key symptoms of autism. The review begins with an examination of how traditional interactive music therapy techniques address deficits in social, communication and behavioural development. In support of questionnaire based evaluation of intervention methods, neurological evidence is presented which demonstrates changes in brain activity resulting from music therapy. Next, an in-depth investigation into how new technologies are able to replace traditional music instruments and enhancing therapeutic sessions is presented. Finally, the implications of current research are explored with suggestions for both future investigations and important design considerations for new music therapy interventions.

0.1. Inclusion of literature for review

Using the key words: Autis*, Asperger*, Music*, Intervention*, Therap* Computer*, Technology, Assistive, Multi Sensory, we conducted a search in the following online databases: Google Scholar, PubMed and Web of Science. Relevant findings in English were taken from peer reviewed investigations. For research exploring new technologies in autism spectrum disorders (ASD) therapy, articles were published between 2007 and 2017. Reference lists of search results were also checked for additional relevant articles, employing both quantitative and qualitative methodologies, including case study reports. Commentaries and editorials were excluded.

1. Background of autism and interactive music therapy

Interactive music therapy (IMT) aims to exploit the emotional, relational and motivational elements of music in order to create a viable medium for social communication. Its personalised and systematic approach provides an opportunity for the therapist to establish a meaningful relationship with the patient through shared music making (Alvin & Warwick, 1997). This section will present evidence for how interactive music therapy facilitates development in communication, social-emotional and behavioural skills in children with ASD. We also review research which identifies the neurological benefits of music-based interventions, promoting multimodal communication between parts of the human brain that are typically underutilised for those with autism.

1.1. Enhancing language and communication

From birth, infants exhibit an innate predisposition for musicality which is critical for pre-verbal communication, with their vocal sounds containing elements of pitch, melody, rhythm and timbre. These elementary sounds, which are used to communicate feelings of hunger, need for sleep, etc. will eventually be extended to speech as the child undergoes the cognitive processes associated with communication development (Darnley-Smith & Patey, 2003). Moreover, parents will communicate with an infant through familiar music-making techniques such as timing and pitch and timbre regulation, a process named by Trevarthen and Malloch as “Communicative Musicality”, which applies to the characteristics of human communication that are exploited in music (Trevarthen & Malloch, 2000), see also Dissanayake (Dissanayake, 2000). Children with ASDs experience significant communication deficits which are thought to be implicated in other core autistic symptoms of social and emotional impairments (Malloch, 1999). These theories, coupled with the similarities between music and language, form the rationale for music therapy applications for individuals with autism whose deficiencies in language are a defining characteristic of their disorder.

The ability to imitate another person’s actions and translate into ones own is considered a requirement of language development (Hadjikhani, 2007). However, studies have observed in children with autism a reduced ability to imitate when compared to typically developed (TD) control groups (Perra et al., 2008; Vanvuchelen, Roeyers, & De Weerdt, 2007). Buday (1995) investigated the imitation effects of music therapy in facilitating the memory for signed and spoken words, comparing musical and rhythmic approaches for 10 children ASD. In the music group, the target words were sung over music, whilst in the rhythm condition the same words were rhythmically spoken over the same song. It was observed that the amount of correctly imitated words in the music condition were significantly higher than their rhythm counterparts. One explanation of these findings is the possibility that music was able to increase the attention of the children. A similar use of those imitation techniques has been utilised for a 6-year-old nonverbal female with low functioning autism who imitated a trombone through grunting. Intervention sessions comprised of the therapist singing words in tune with the trombone. Upon completion of the intervention, the subject had progressed from grunting to verbalising words that had previously been sung to her (Hoelzley, 1993).

It has been observed that children with ASD exhibit a preference to song over spoken words (Molnar-Szakacs & Heaton, 2012). Furthermore, investigations focusing on the neural mechanisms associated with speech and song in those with ASD have found evidence of a distinct difference between song and speech perception, despite overlapping activity between the key cerebral regions implicated in processing both sets of sounds, such as the frontal and temporal neocortex regions (Sch¨On et al., 2010; Sharda, Midha, Malik, Mukerji, & Singh, 2015; Wan, Rüüber, Hohmann, & Schlaug, 2010). This shared distribution of activity within an overlapping network may therefore act as a viable mechanism that compensates for a lack of engagement with spoken word, and stimulates the language processing systems with song based therapies (Sharda et al., 2015). Empirical research has found that when compared to neurotypicals, autistic participants presented with speech stimuli display a decrease of functional response in the left inferior frontal gyrus, an area of the brain associated with speech processing (Lai, Pantazatos, Schneider, & Hirsch, 2012; Poldrack et al., 1999). In addition, Lai et al. (2012) also discovered the same ASD group showed an increase in left inferior frontal gyrus activity when listening to song. In similar work, intensive musically based speech therapies for individuals with Brocas aphasia have shown to engage growth of the arcuate fasciculus, the frontal-temporal tract that connects the Wernickes speech area, which is responsible for speech perception (Geschwind, 1972), and the left inferior frontal gyrus (Schlaug, Norton, Overy, & Winner, 2005).

Developed in the early 1970s, melodic intonation therapy (MIT) is a treatment that improves language by transposing the musical elements of speech (melody and rhythm) into sung melodic intonational patterns. Functional and simple phrases are sung during therapy, with the musical components formed as closely as possible to the normal inflection patterns and verbal utterances of normal speech. The technique is thought to engage the overlapping regions of the brain that are involved with music and language, with some evidence suggesting that undertaking in long-term MIT reactivates the speech processing regions in the left hemisphere (M. Thaut & Hoemberg, 2016). The majority of research into the benefits of MIT focuses on severe non-fluent aphasia in individuals with stroke (Draper, 2016; Van Der Meulen, Van De Sandt-Koenderman, Heijenbrok, Visch-Brink, & Ribbers, 2016); however, there are a limited number of studies applying the technique to ASD. An early study in 1973 tested MIT on a 3 year-old nonverbal male with autism (Miller & Toca, 1979), combining the intoned stimulus with signed language. Following the intervention, the subject was observed to make spontaneous intoned attempts at verbal communication based upon the musical and imitative components of the MIT program.

Recently an intonation-based speech therapy called Auditory-Motor Mapping Training (AMMT) has been introduced. An adaptation of melodic intonation therapy which aims to activate shared motor, auditory and visual neural regions through a combination of intoning simple phrases whilst tapping a pair of drums tuned to the same pitches of the intoning targets (Chenausky et al., 2016). The effectiveness of AMMT for children with ASD has been noted in investigations, all with participants experiencing varying degrees of verbal abilities (Chenausky et al., 2016; Wan et al., 2011). In the most recent work by Chenausky, Norton, and Schlaug (2017), a randomised control study found one AMMT participant experienced a greater improvement in spoken language than a control group receiving speech repetitive training (SRT), a therapy that does not use singing. It was also suggested that after the 28 week program, the AMMT group found it easier than the SRT control group to generalise developments in language skills to settings outside of therapy.

Research has shown that music training for neurotypical children leads to auditory and cognitive progression that extend further than abilities for musical perception. Compared to controls, children with prolonged active engagement in musical training display a larger corpus callosum (Rose, Jones Bartoli, & Heaton, 2017) alongside structural changes in the brain development in frontal, temporal, and parieto-occipital brain systems (Hyde et al., 2009), resulting in enhanced processing in language and speech domains (Kraus & Chandrasekaran, 2010; Magne, Sch¨On, & Besson, 2006; Schlaug et al., 2005). Comprehensive investigations have been made into understanding the cognitive impact of music education on ASD. Investigations observed a large number of musically trained individuals with ASD who exhibit abilities of perfect pitch discrimination, whilst also exhibiting impairments in both the cognitive and language domains (DePape, Hall, Tillmann, & Trainor, 2012).

However, research conducted by Heaton, Hudry, Ludlow, and Hill (2008) examined the capacity to discriminate pitch in speech and non-speech stimuli in 20 autistic children and 29 controls either typically developed or with minor learning difficulties. Results showed that when compared to controls, the children with ASD displayed exceptional sensitivity to the non-semantic pitch information of speech. Interestingly, the same study found some evidence that enhanced pitch perception may contribute to language impairment in the early stages of development. This is also supported in research that reported a group of individuals with ASD displaying both accurate auditory discrimination and early language delays (Jones et al., 2009). An intriguing argument for this is the reduced engagement with linguistic stimuli and a biased attention towards the musical non-speech aspects, resulting in an over-specialisation of pitch-processing mechanisms (Bonnel et al., 2010; Heaton, Williams, Cummins, & Happ´E, 2008; Kuhl, Coffey-Corina, Padden, & Dawson, 2005; Lepist¨O et al., 2005). However, despite supporting evidence of a correlation between enhanced pitch perception and language impairments, these results belong to a particular sub-group and do not represent the entire autistic spectrum. In fact it has been proven that children with Asperger’s Syndrome do not share equal levels of higher musical perception and processing (Bonnel et al., 2010). Despite some research indicating that differences in perceptual tuning could impede early language acquisition, the understanding of this area still remains incomplete. Literature suggests that language delays in ASD are impacted by social attention impairments such as joint-attention and social engagement. These are characteristics of autism which can be addressed through musically based interventions. (Dawson et al., 2004; Kim, Wigram, & Gold, 2009; Loveland & Landry, 1986; Mundy, Sigman, & Kasari, 1990).

1.2. Social, emotional and behavioural developments

Investigations into the neurological correlates of emotion recognition in music in those with high functioning autism have found ASD groups like TD control groups to be able to correctly recognise the implied emotional expressions of musical stimuli (Caria, Venuti, & de Falco, 2011; Gebauer, Skewes, Westphael, Heaton, & Vuust, 2014; Kopec, Hillier, & Frye, 2014). Specifically, brain imaging by Gebauer et al. (2014) found that when listening to music, both TD controls and nineteen participants with high-functioning autism exhibited an increase of neural activity within the brain regions responsible for processing the emotional content of music. For individuals with ASD, the ability to recognise emotional connotations in music regardless of difficulties in comprehending emotions communicated through speech (Heaton, Hermelin, & Pring, 1999), could present opportunities to develop their emotional understanding.

Interventions designed by Katagiri (2009) successfully taught 12 children with autism to recognise the four emotions of happiness, sadness, anger and fear. This was achieved through a combination of verbal instructions and emotionally specific background music or specially composed songs; however, the most significant developments were observed within the background music group. Further to this, a randomised control study by Kim et al. (2009) measured the emotional synchronicity between ten children with ASD and therapist when taking part in interactive music therapy. The 12-week intervention program led to improvements social-emotional timings, more specifically in expressing and identifying positive emotional expressions of joy. It was noted that these observations could be the result of the development of interaffectivity through musical attunement.

The process of musical attunement relies heavily upon the non-verbal and multi-modal components of music, with a typical session including vocal and instrument exchanges, facial expressions, eye contact and synchronous movement and gestures (Kim et al., 2009). Furthermore, group music making sessions can provide important social learning experiences through turn taking, joint attention and empathy. (Overy & Molnar-Szakacs, 2009). The development of these social behaviours is integral to education and daily functioning. Therefore, interventions such as music therapy that involve collaborative singing, moving, creative play and listening, help establish social connections and enhance the emotional understanding of autistic children (Walton & Ingersoll, 2013).

A child’s family plays a significant role in the intervention framework, often providing important individualised information in the design process, but also in delivering the intervention itself. Despite children with ASD having social and communication deficits, mothers experience increased levels of relationship closeness with their children when compared with mothers in the TD population, and this strong parent-child relationship can be harnessed to facilitate early social development (Karst & Van Hecke, 2012; Mahoney & Wiggers, 2007; Montes & Halterman, 2007). There are a number of advantages to providing family based interventions. Not only are they cost effective, but they also create enriched and responsive environments for parents who wish to deliver additional support. Further to this, such interventions will move away from the clinic and increase the child’s time spent in their more natural home or community environment, resulting in the increased motivation and generalisation of newly acquired skills to post-therapy contexts (Koegel, Koegel, Harrower, & Carter, 1999; Schreibman & Koegel, 1996).

There are a number of family based therapies such as TEACHH (Virues-Ortega, Julio, & Pastor-Barriuso, 2013) and Joint attention focused parent training (Rocha, Schreibman, & Stahmer, 2007). A systematic review explaining the benefits of parent-implemented interventions is provided by McConachie and Diggle (2007). Family driven music therapy can also support the child’s acquisition of new social, emotional and behavioural skills, and the development of a high-quality relationship with the family (Oldfield, Bell, & Pool, 2012). By adopting a relationship-based system involving turn taking, reciprocal interactions and affect sharing, therapists have been able motivate an autistic child to socially interact with others using the non-verbal aspects of music making (Kim, Wigram, & Gold, 2008). A randomised controlled study by Thompson, McFerran, and Gold (2014), which involved 23 children with severe ASD reported a significant improvement in the quality of social interactions after the 16 week family centred music therapy program. This is not only beneficial for the child but it also grants the parents valuable insight into the limitation of their child’s social skills and the therapeutic process.

Joint attention is the ability to participate in shared experiences and interests and is fundamental in early child development. Without appropriate joint development skills, higher functions such as language, communication and social interaction will be impeded (Kim et al., 2008). There are various successful intervention methods aimed to improve joint attention in children with autism spectrum disorders, these can include joint attention-mediated learning, functional training with preferred toys and robot-mediated interaction (see (Paparella & Freeman, 2015) for a review). Music therapy has shown to be a promising approach to interventions dealing with joint attention impairments and effective techniques have included singing (Vaiouli, Grimmet, & Ruich, 2015). Another important reason behind the success of IMT in promoting social and emotional skills in ASD is that it is often conducted in groups. Koegel suggests that peers are significant factors in social development in children with ASD (Koegel& Koegel, 1995). It is these group activities that not only facilitate learning of joint attention but also are key in the development of social skills such as turn taking, social reciprocity and imitation (Overy & Molnar-Szakacs, 2009).

Literature has shown that music-based activities provide an effective context for reciprocal interactions and shared experiences that enhance both social and emotional developments for those with ASD. In addition to this, children with autism enjoy music. This can not only be explained by recognising its emotional content but also its predictability and structured approach. These help create a sense of security which encourages a child to move outside of their usual social constraints and explore new interactions with others (Ghasemtabar et al., 2015).

2. Computer-assisted technology in music interventions

Since the early 2000s there has been an exponential growth in the use of electronic assistive technologies supporting research and the development of children with ASDs in social, communicative, language and educational domains (Ploog, Scharf, Nelson, & Brooks, 2013). This has in part been due to technological developments in computer based multimedia technology such as mobile devices and motion capture sensors (e.g. Microsoft KINECT®), which has brought advanced and novel approaches to computer interaction into the public domain, moving far beyond the traditional desktop computer.

When applied to music therapy, new technologies that provide alternative control that enables users with complex needs the accessibility to play the technology as they would a conventional musical instrument (Hunt, Kirk, & Neighbour, 2004). This plays a significant role in delivering the creative freedom which is often central to the key outcomes of interactive music intervention. Digital musical instruments are not new to therapy and so this section will focus on the use of computer technology which has been adapted to be included in music intervention programs for ASD, and how new avenues of control and interaction can improve upon the benefits of music therapy outlined in the previous section of this review.

2.1. Motion sensor devices

Children with autism often experience difficulties in motor skills such as balance (Ament et al., 2015), fine motor delays and gait deviations (Harris, 2017; ShetreatKlein, Shinnar, & Rapin, 2014). Although interventions such as sensory integration therapy are a common method in dealing with motor impairments (Karim & Mohammed, 2015), music therapy can also be used as an effective approach to motor rehabilitation. The rhythmic structure of music can engage areas of the brain responsible for movement such as the dorsal premotor cortex and supplementary motor area (Bengtsson et al., 2009; Chen, Penhune, & Zatorre, 2008). Despite a large amount of studies linking music therapy to improvement of motor efficiencies (Bradt, Magee, Dileo, Wheeler, & McGilloway, 2007), there seems to be very little investigations into its application to developing gross motor skills in ASD. To date only one randomised trial researching music interventions in the treatment of motor deficits for children with autism can be found. Compared to a control group who took part in non-musical movement activities. Eleven children with low functioning autism who received musically synchronised movement exercises over an 8-week period showed significant improvements in motor skills, such as balance and upper limb coordination as measured by the Bruininks-Oseretsky test of motor proficiency (Atigh, Akbarfahimi, & Zarei, 2017). This study therefore suggests that musical accompaniment during physical exercises enhances motor skill learning of children with autism. Further to this, a randomised controlled trial involving 36 children with autism who displayed significant behavioural and language impairments, received an integrated 10 week rhythm and movement program. Following the intervention, participants demonstrated significant improvements in gross motor performance and interpersonal synchrony skills (Srinivasan et al., 2015).

Motion sensing technology has been important in enabling access to music making in conjunction with promoting movement since the 1990s with the development of Soundbeam, a device that captures movement via ultrasonic beams and converts them into MIDI based music (Swingler, 1998). Soundbeam has proven success with individuals with profound mental or physical disabilities, enabling meaningful creativity and expression using music and sound with a control interface that requires only limited interaction. However, the ultrasonic sensors that this device relies upon, requires movement within a limited area and involves specific motions.

In recent years, consumer grade video camera technology alongside video image processing techniques can detect features of the human body and track minute movements and spatial positions across a large interaction environment (Magrini, Carboni Salvetti, & Curzio, 2015). The Microsoft KINECT® game controller has been implemented in a number of educational and therapeutic programs for children with ASD (Bartoli, Corradi, Garzotto, & Valoriani, 2013). For example, Rogli´C et al. (2016) conducted an investigation that incorporated the Kinect into serious game design. It was found that the physical interaction created by the device was effective in developing cognitive, motor and reflex skills of one autistic boy and on autistic girl. Liu et al. (2016) also used the Kinect to address motor dysfunctions through movement imitation.

With regard to the use of motion sensing devices in interactive music therapy, a system developed by Ichinose et al. (2016) named Cymis combines auditory cause and effect with the Kinects ability to enhance physical and motor interactions. The system was deployed in four music therapy sessions lasting 15 min for two children, one with ASD, and the second not yet formally diagnosed but who exhibited cognitive and social symptoms associated with autism. Each session consisted of four tasks set to music which would be completed if the subject held a particular pose or imitated movements defined by the music therapist. Over the course of interactions the examiners observed improvements in controlled motor interactions with both the system and the therapist, whilst also facilitating on-task concentration and enjoyment. Despite results indicating that Cymis could be an effective music therapy tool the small group size and lack of formal motor proficiency testing lead to a lack of definitive conclusions. It was however, included in this review as it is a positive example of how motion capture technology and interactive music therapy can be integrated.

2.2. Mobile technology

Recent developments in mobile technologies have led to an increase in multimedia apps (iOS and Android) available for both entertainment and educational forums, a number of which are specifically designed for the special needs population (Hillier et al., 2016; Howard & MacCalla, 2014; MacCalla, Xu, & Howard, 2015; Weng, 2015). Devices such as the iPad® are a popular intervention technology for those with autism due to their portability (Shah, 2011) and multi-sensory outputs (Hillier et al., 2016). Furthermore, the large touch screen layout provides the opportunity for those with verbal communication impairments to express understanding through gesture and visual representations (Flewitt, Kucirkova, & Messer, 2014).

Hillier et al. (2016) investigated the effectiveness of the iPad within a music therapy programme for 23 adolescents with high functioning autism. Participants could control the application simultaneously via the large touch screen, engaging in group musical composition activities and creative discussions, therefore promoting essential social and cognitive skills. Additionally, the intuitive design of the apps gave musically naive participants the creative freedom to safely explore music making and experience the benefits of traditional IMT techniques. This was supported by a reported reduction in both stress and anxiety at the end of the programme, when compared with pre-intervention testing. In support of this study, a review conducted by Putnam and Chon (Putnam & Chong, 2008) concluded that relaxed and natural interactions with multi-touch devices deliver a simple interaction mechanism that is especially beneficial to those on the autistic spectrum. 

The use of mobile tablets in music therapy has notable advantages and can provide affordable and compact alternatives to previously expensive digital music instruments. However, there is a lack of software developed specifically for this purpose, and applications used in music therapy are often developed for the wider consumer market (Knight, 2013). Although there could be benefits from research and development of specific therapy apps for autism, this lack is not generally a disadvantage. By using the same instruments as their typically developing peers, a therapist removes the stigma attached to using the technology and could also foster social interaction and collaboration with neurotypical children (Mechling, 2011); therefore, increasing motivation to engage with therapy.

2.3. Multi-sensory surfaces

Interventions controlled with interactive surfaces are not limited to touch screen devices; some approaches also augment traditional musical instruments with interactive visuals. Andantino (Xiao, Puentes, Ackermann, & Ishii, 2016) assists children learning to play music by projecting animated figures on top of a piano. BendableSound (Cibrian, Pen˜a, Ortega, & Tentori, 2017) creates a multisensory experience through a deformable fabric surface providing visual-motor control via Microsoft KINECT®. These displays allow user control through gestures, but the highly flexible characteristics of the elastic surfaces allow for more natural and empowering control through pulling, twisting and warping (Müller, Gründer, & Groh, 2015). In the case of BendableSound, the varied haptic modality of interaction permits the user to define their own auditory experience, controlling musical elements such as pitch, volume and rhythm. This allows for a more pragmatic approach to therapy sessions. The limited prompts and increased self-exploration enable children to apply newly acquired skills to real-life environments (Schaaf et al., 2014). The intervention provides a novel approach to musical therapy, exploring the relationship between music and movement to not only develop physiologically cognitive and sensory processes, but also improve upon significant sensorimotor skill impairments often exhibited by children with autism (Staples & Reid, 2010; Thaut & Hoemberg, 2016). For children with lower functioning autism, cognitive and motor dysfunctions can restrict interactions and creativity, often resulting losing attention and motivation to complete interventions (Burland & Magee, 2014). The elastic multisensory surface of BendableSound therefore addressed these issues, giving children with autism a multi-modal intervention resulting in sensorimotor, cognitive and behavioural developments (Cibrian et al., 2017).

By amalgamating audio, visual and haptic feedback into a singular therapy approach it is possible to prolong engagement and increase interest in intervention sessions. Loureiro et al., 2003 combines haptic based technology and 3D soundscapes to enhance the users experience with an interactive painting. The aim of the project is to improve motor control and emotional well-being of children with ASD through audio and visual exploration. Delivered over 12 loudspeakers, the 3D soundscapes matched the visual aquatic theme with the spatial positioning and individual sounds being dependent upon the position and velocity of the haptic device. For instance, if the subject interacted with the underwater section of the picture then the audio would replicate the sound of bubbles. A case study using seven participants with autism observed that audio played a significant role in determining how the children would interact with the painting. If they preferred a certain sound then they would interact with the corresponding section for a prolonged amount of time. The sound also dictated movement, with one user making circular movements because the sound would resemble their actions. One important outcome from the project was the communication and interaction between participants. As it was a group activity, it was essential that children engaged with one another in order to negotiate time spent on the device.

3. Opportunities for technology for music and audio interventions

The development of new affordable technology such as mobile touch screen devices, motion capturing and computer systems, provides new opportunities to construct novel approaches to the interaction of music and sound for individuals with autism. They are able to facilitate the established music therapy outcomes, alleviating core communication, social-emotional and perceptuo-motor deficits, whilst providing an individualised experience that engages on a cognitive and multi-sensory level. However, with the perpetual progression of consumer technology, there are some recommendations and considerations to be made.

For an autistic individual that might find the world confusing and unpredictable (Brown, 1994), computers are free from social constraints and provide an experience that delivers consistent and predictable responses that will maintain interest and increase motivation levels in autistic children (Iovannone, Dunlap, Huber, & Kincaid, 2003). In addition, computer assisted interventions such as serious games, can construct a safe environment through graphically displayed cues that promote multisensory exploration, which can be modified and customised to maximise experience towards the individuals specific needs (Sansosti, Powell-Smith, & Cowan, 2010). Through a combination of digital game play and an intervention framework, these computer-based therapies are being used for cognitive rehabilitation (Aresti-Bartolome & Garcia-Zapirain, 2015) and emotional development (FridensonHayo et al., 2017), all within a safe and non-threatening context which they can enjoy whilst practising and acquiring new skills (Zakari, Ma, & Simmons, 2014).

Gamification is recognised as the use of game like elements within non-game contexts, designed to encourage learning of targeted skills that are normally unrewarding or difficult (Deterding, Dixon, Khaled, & Nacke, 2011). Applying this to music therapy through computer game technology should be considered when designing new digital music interventions. Design parameters and mechanisms within serious games direct players to complete goals based around the acquisition of new skills, providing challenges and a reward system that reflects their progress (Richter, Raban, & Rafaeli, 2015). This theory does appear to influence the design structure of the Cymis system, in which the children had to perform certain tasks based on particular poses and movements. The combination of audio and visual feedback and a basic progression system, maintained the interest and motivation of the children to engage with the therapy (Ichinose et al., 2016).

Virtual Reality (VR) has become a popular platform used in ASD interventions and research has delivered successful results in areas including social and cognitive training exercises combined with exposure therapy (Bresnahan et al., 2016; Ip et al., 2016; Yang et al., 2017). These studies have noted that VR can reproduce realistic virtual scenarios in a controllable environment, allowing safe exploration and practise of social and occupational interactions. Added realism can also be achieved by interfacing the VR environment with motion capture devices that provide natural control to simulations (Adamovich, Fluet, Tunik, & Merians, 2009). Furthermore, with the ability to repeat social and cognitive training within the dynamically changing VR environment, there is a greater opportunity to generalise newly acquired skills into everyday life (Parsons & Mitchell, 2002).

Immersive virtual environments are a new medium in social and cognitive training and as yet there is no application to music or sound therapy for ASD. However, there is scope for development within this context. Singing within a choir can have psychological benefits for individuals with chronic mental health issues, with research indicating a reduction in anxiety and an increase in positive emotions and relaxation which translates to maintained and extended mental well being (Clift et al., 2010; Dingle, Williams, Jetten, & Welch, 2017). Research by Kearney et al. (2016) aimed to replicate group singing and its benefits through VR. Participants sung alongside a pre-recorded virtual choir located in a church, with audio being played over a multiple-loudspeaker array and accompanying 360-degree visuals provided by Oculus Rift. The combination of photorealistic video and accurate spatial and environmental audio representation produced an immersive experience that made singers feel that they were present in the original performance space.

An adaption of VR is augmented reality (AR), a technology in which interactive virtual objects are layered over real-world environments. An intervention based in AR would alleviate the need for physical instruments, instead supplying novel and interesting methods of controlling virtual instruments. This technology has already been used in the rehabilitation of children with profound physical disabilities. A virtual instrument system developed by Bloorview Kids Rehab (Hobbs & Worthington-Eyre, 2008), combines virtual shapes with video images of the user to create an environment where gesture recognition translates physical movements into musical sounds. A critical element of AR for ASD treatment is its ability to provide shared virtual experiences; this is a key difference to the mostly singular experience of VR. A carer and child can simultaneously interact with the virtual instruments and each other, remediating the core social and emotional impairments in autism. Finally, augmented objects can be customised both aesthetically and with sound to suit the subjective and cognitive needs of the child.

Research presented in this review has acknowledged the importance of family based interventions for autistic children. Therefore, the inclusion of parents or siblings should be an important consideration when developing a novel therapy system. In the context of family centred music therapy, mobile technology should be regarded as the primary platform on which to deploy new applications. This recommendation is largely based on the accessibility of mobile technology and mobile applications; with the vast majority of people in developed countries reporting to own mobile devices, and smartphone ownership in developing countries rapidly increasing (Poushter, 2016). With a large percentage of the population owning a touch-screen device capable of advanced multi-sensory outputs, interventions can be carried out in family centred environments where the child feels more comfortable and therefore can become more socially competent with a higher likelihood of generalisation (Karst & Van Hecke, 2012). Further to this is the lack of parental training needed to implement an effective therapy framework at home, with the majority of mobile music apps designed to be intuitive, offering maximum feedback whilst requiring no previous musical knowledge. This can lead to socially embedded activities revolving around creative and collaborative music making, taking advantage of the social-emotional benefits associated with IMT. Consequently being used in support of clinic based interventions enhancing skill learning, generalisation and maintenance. Finally, a large global trial conducted by Bieleninik et al. (2017) found no significant improvements in ASD symptoms when comparing improvisational music therapy to enhanced standard care. The authors suggest that these findings could be the result of a lack of uniformed practise between therapists and clinical assessors. Similar conclusions were also noted in a systematic review by Brondino et al. (2015), with difficulties in standardising IMT methodology generating negative affects on the outcomes of music based interventions. For those within the ASD population, repetition is critical to learning (Lovaas, 1987). The distribution and potential wide spread use of mobile applications with replicable means of musical interaction may provide opportunities to standardise approaches to music therapy. In addition, quantifiable data which maps user interaction and progression through therapy can be stored and used in empirical research that evaluates novel technology based interventions.

4. Future research suggestions

A number of key research suggestions can be taken from the evidence provided in this review. Firstly, similar to most technology research within the autistic population (Ploog et al., 2013), there is strong need for investigations into new therapy technologies which are based on larger and more representative participant samples. The majority of studies around intervention techniques and technology in this article are centred around either small groups or case studies. This may be due to a lack of viable participants who meet the research criteria or lack of facilities. However, developments in consumer technology and the potential for strategies to be embedded in the child’s natural environment, could possibly open up research to a much larger population. This could also present the opportunity to examine the effects of specific design features via randomised control group comparisons. By conducting these studies there is the opportunity to reach an optimal calibration in not only technology but also intervention protocols.

With the dynamic development of new consumer technology there should be continued research into any possible applications for music therapy for not only those with ASD but also the wider profound and multiple learning difficulties population. By investigating key design features, clinicians and researchers can design and make critical suggestions for future systems and integrated frameworks. The inclusion of new technology in ASD interventions will require a level of technical or programming expertise that clinicians may not possess. With this in mind, collaborative and multidisciplinary research between engineers, programmers and clinicians is essential in developing or modifying therapy tools.

However, an inclusive and collaborative design process should not be limited to clinicians and engineers. A participatory design (PD) process that includes children with physical or mental disabilities can provide a greater understanding and insight into their complex needs and preferences (Frauenberger, Good, & Keay-Bright, 2011). Commonly, children’s contributions are limited to aesthetics or small non-therapeutic aspects of an intervention (Malinverni et al., 2017). However projects such as Lands of Fog (Mora-Guiard, Crowell, Pares, & Heaton, 2016) aim to provide a deeper influence on its design elements and mechanics through PD. Five sessions were conducted in which children with ASD provided ideas which aided in the design of visuals, game mechanics and interactive content. However, conducting PD workshops with ASD children is often a challenging task as they require communication and social interactions. Malinverni, Mora-Guiard, and Pares (2016) applied multimodal evaluation of participatory design sessions, children were asked to perform exploration and drawing activities whilst investigators focused analysis of interactions based on gaze direction, position in the space, focus of attention and the children’s drawings.

Finally, generalisation and maintenance of new skills acquired through music therapy remain under explored. Within this review, no investigations evaluated generalisation after interventions. This is a difficult area in therapeutic approaches for autism, with many children showing improvements in the clinic environment but experiencing difficulty in maintaining and applying the new skills to everyday situations (Dautenyhahn, 2000). Although family centred therapy is suggested to enhance generalisation (Schreibman & Koegel, 1996), considerations need to be made not only when designing new intervention programs but also assessing when effectiveness in the months following completion.

5. Conclusions

Traditional interactive music therapy practices based on singing, music making, joint attention and social interactions have been shown to alleviate core impairments in social-emotional, communication and sensory motor skills in children with ASD. However, technology can be a significant tool in the therapy of persons with ASD, especially in the realm of music and sound. The recommendations made to introduce virtual and augmented reality to innovative music therapy protocols, could create engaging methods of creative control that are accessible to those with autism regarding their cognitive and motor abilities. Current mobile technology can not only support VR and AR applications but also provide accessible interventions that can move away from the clinic to the home, incorporating family members into therapy sessions. This allows them to gain a better understanding of their child’s behaviour and specific needs, whilst also building essential social and emotional relationships. It is important that research takes advantage of new advances in interactive audio technology and adapts it to the requirements of children with neurological disorders. This will deliver critical knowledge into the complex workings of the autistic brain which will aid in the innovation of more efficient intervention tools and frameworks. Furthermore, it will allow children with ASD similar enriched experiences enjoyed by their typically developed peers, creating enjoyment and empowerment. This might be argued to improve quality of life as much as psychological and neurological developments.

It is important however, that collaborative research between clinicians and engineers continues in order to investigate which design aspects are necessary in developing music and sound interventions that are both useful and appealing to children with ASD. Autism is a complex neurological disorder and occurs on a broad continuum of severity. Autistic individuals may react differently to music and music technology on a cognitive or social level, and the effectiveness of any intervention will be dependent on the individual. However, technology-based music interventions remain a powerful tool as they have the possibility to reduce the effects of the core symptoms of ASD by combining traditional IMT frameworks with engaging and motivating approaches to musical interaction and creativity.

Additional information

Funding

Funding was provided by a UK Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Award, via the Department of Electronic Engineering at the University of York, EPSRC Grant Number: EP/N509802/1.

Notes on contributors

Daniel Johnston

Daniel Johnston is a PhD student in Music Technology at the Department of Electronic Engineering at the University of York. His research is based on the application of interactive spatial audio systems into novel immersive interventions for individuals with autism, focusing on addressing difficulties in auditory processing.

Hauke Egermann

Dr. Hauke Egermann is an Assistant Professor in Music Psychology at the Music Science and Technology Research Cluster at the University of York and is the director of the York Music Psychology Group. His research is focused on the effects of music on recipients, embodied musical cognition and the implementation of these findings into music technology.

Gavin Kearney

Dr. Gavin Kearney is an Associate Professor in Audio and Music Technology at the Department of Electronic Engineering at the University of York. His research interests include spatial audio and surround sound, virtual and augmented reality, spherical acoustics, interactive audio systems, recording and audio post-production technique development.