Alertness, movement, and affective behaviour of people with profound intellectual and multiple disabilities (PIMD) on introduction of a playful interactive product: Can we get your attention?

ABSTRACT

Background: New technology may stimulate active leisure activities for people with profound intellectual and multiple disabilities (PIMD). We conducted a study of an interactive ball that responded to gross body movement, focus of attention, and vocalisations of users with PIMD. The aim was to increase alertness and body movement and elicit more expressions of positive, or fewer of negative affect.

Method: Nine participants with PIMD played during 8–10 sessions. The movement was analysed automatically. Alertness and affective behaviour were coded manually. We analysed the last 5 sessions for each participant and compared 15 min of interaction with 15 min of rest.

Results: Clearly positive effects were seen for three participants. Effects were seen in the unexpected direction for four participants. No strong effects were found for the remaining three participants.

Conclusions: Interactive technologies may provide suitable activities for people with PIMD but individual differences play an important role.

KEYWORDS:

• Leisure activities
• profound intellectual and multiple disabilities
• interactive play
• alertness
• affect
• movement

People with profound intellectual and multiple disabilities (PIMD) have severe to profound intellectual and physical disabilities and depend on others for their every-day activities (Maes, Lambrechts, Hostyn, & Petry, 2007; Munde, Vlaskamp, Maes, & Ruijssenaars, 2012; Nakken & Vlaskamp, 2007). They have limited amounts of suitable active leisure activities in their daily lives (Caltenco, Larsen, & Hedvall, 2012; van Delden et al., 2014; Vlaskamp, de Geeter, Huismans, & Smith, 2003) and are at risk of experiencing mainly somewhat passive activities (e.g., watching television, Zijlstra & Vlaskamp, 2005).

Recent developments in technology, including sensor technologies that respond to movement, allow for truly interactive systems that include a developing dialogue of actions and responses and can stimulate active leisure activities with more physical movement. The PhD thesis of Larsen (2015), which presents a series of prototypes developed for people with PIMD, clearly argues for the relevance and potential of these developments for people in this target group.

van Delden et al. (2014) performed an international catalogue search of interactive devices for people with PIMD (e.g., many multisensory spaces including Snoezelen^® objects). They concluded that only a limited number was suitable to stimulate active behaviour for this target group. Moreover, the devices they found lacked more elaborate interaction behaviour: multisensory spaces expose clients to various interesting stimuli, but interaction is usually limited to micro-switches that turn something ON or OFF (e.g., a lava lamp), which lacks the pattern of a “developing dialogue of actions and responses”. Instead, the truly interactive systems that we aim for combine gradual forms of feedback with several forms of input. Visual/auditory stimuli can increase and decrease in response to several types of non-verbal behaviour of the participant, potentially leading to richer patterns of action and interaction.

Inspiring examples in this context are the prototypes from the SID project (Larsen, 2015), which were developed to elicit interaction with people with PIMD. These prototypes had several forms (e.g., curtain, pillow, water bed, hug ball) and delivered visual, auditory, or motion stimuli, or any combination of these variables in response to tactile interaction such as pushing, pressing, and cuddling. In informal and anecdotal evaluations, the authors found that children with PIMD seemed to take initiative for interaction with the devices and seemed to enjoy themselves.¹ However, the systems were intended as tools for ideation rather than work towards systematic evaluation: the prototypes were meant to explore and illustrate the space of possibilities for designing such products. This explains why there was no focus on an effect study (Caltenco et al., 2012; Hedvall, Larsen, & Caltenco, 2013; Larsen, 2015; Larsen & Hedvall, 2012).

A few other activities focus on the use of music. The commercial SoundBeam system produces musical tones based on the movement of a user. Cappelen and Andersson (2012, 2016) created a pillow-like, tangible music-making system with sensors, music feedback, and a wall video projection. Meckin and Bryan-Kinns (2013) created a set of actuated, acoustic instruments for people with special needs to hear and feel sounds, played through an iPad. However, neither of these systems were used in a systematic effect study for our target group. In short, the few truly interactive systems that were created for people with PIMD show promise but have not yet been systematically evaluated for them.

In the present exploratory study, we created and evaluated an interactive ball that responded to the gross body movement, focus of attention, and vocalisations of users with PIMD. It was adapted to individual preferences and could make different sounds, wiggle, move from left to right, and emit coloured light. Based on discussions with care staff, literature about PIMD, and our striving for attainable and measurable outcomes that are considered desirable by stakeholders, we set out to design a ball of which the interactive behaviour would be associated with: (1) increased alertness; (2) improvement in affective behaviour; and (3) increased amount of body movement. This paper explains how we approached a systematic user study of the new interactive system, including results, and discusses the benefits and limitations of our approach. This paper is one of two complementary papers (also: Embregts et al., 2018) reporting on this new device in a small-scale study.

Method

Study design

Fitting current practice and due to large individual differences within the target group, we approached this exploratory study as a single case design with inter-case replication as well as within-case replication (Horner et al., 2005; Kratochwill et al., 2010). Nine subjects participated; every subject was exposed to multiple iterations (one session per day over the course of several days). Each single iteration was organised as an A-B-A reversal design, consisting of a 30-min contiguous session, with multiple measurements, divided into three phases. The phases were (A) a 7.5 min period of rest before the intervention, then (B) a 15 min intervention (introduction of the interactive ball), followed again by (A) a 7.5 min period of rest after the intervention (withdrawal of the interactive ball). This allowed us to explore the effect of introducing the interactive ball as compared to periods of not interacting with the ball directly before and afterwards. Every participant participated in 8–10 such sessions during 8–10 working days. Not all of these sessions were analysed as part of the study, though. The first three iterations were used as habituation sessions that allowed the participant to get used to the room and the interactive ball; the data from these sessions was not used in the analysis. These three sessions were furthermore used to adapt specific parameters of the ball to the participant’s individual preferences (see intervention section). Analyses were based on the results of the last five sessions of each participant. An overview of the design can be seen in Figure 1. For this study, we obtained approval from the Medical Ethical Committee Twente (study P14-08, NL48070.044.14) and the internal science advisory board of healthcare organisation Dichterbij. A separate comparison between the intervention and an alternative leisure activity (i.e., watching television) is reported in Embregts et al. (2018).

Figure 1. Graphical representation of the study design of this experiment. Due to large individual differences, comparisons have been done at participant level.

Participants

Nine people participated: 3 males and 6 females aged 24–62 years (M = 50, SD = 11). Individual psychologists confirmed that, according to the criteria of Nakken and Vlaskamp (2007), all participants had profound intellectual disabilities (i.e., developmental age up to 24 months) combined with profound to severe motor disabilities. All participants received support at the participating health care organisation; all but one participant had 24/7 residential support at Dichterbij, participant 6 received day support but lived at home. None of the participants had a complete hearing or vision impairment. Characteristics of individual participants are described in more detail in Table 1.

Table 1. Demographic characteristics of participants (fictitious names)^a.

P	Sex	Age	Medical diagnose(s) and disorders (besides profound ID)	#^b	Characteristic keywords used by staff
1	M	62	Cerebral Palsy (CP), psychomotor retardation, reduced vision, epilepsy	10	Social, happy, irritated at disliked activities
Marc
2	F	48	CP, Angelman syndrome, psychomotor retardation, epilepsy, able to crawl and do a few steps with assistance	10	Social, cheerful, happy, curious
Susan
3	F	52	CP, psychomotor retardation, microcephaly, reduced vision, epilepsy	10	Calm, content, sleepy, no cause-effect cognition
Joyce
4	F	47	Psychomotor retardation, microcephaly, able to toddle	10	Social, cheerful, opinionated, spontaneous, energetic, easily agitated, impatient, cause-effect cognition.
Linda
5	F	48	CP, psychomotor retardation, reduced vision, epilepsy	9^b	Peaceful, outdoorsy, happy, timid, sleepy
Dory
6	M	24	Reduced vision, epilepsy, spasm, short-breathed phlegm over-production	10	Happy, recognises “rituals” (recurring events), alert on sounds
Pete
7	M	57	Epilepsy	8^c,d	Fond of personal contact, cheerful, opinionated, seemingly understands simple spoken language
John
8	F	59	Spasm, encephalitis	9^d	Anxious, restless, agitated, cheerful and content at times
Lisa
9	F	56	Epilepsy	10	Cheerful, content, tired, fond of individual attention, seemingly understands sentences by staff that typically precede “recurrent activities”
Tessa

^aTable used with permission from [de-identified organisation]; descriptions based on participants’ psychologist, service coordinators, and individual support plans.

^bNumber of sessions.

^cSession dropped as participant had an epileptic seizure at start of session.

^dSession dropped as the ball stopped functioning appropriately.

Instruments

Alertness – alertness observation list (AOL)

To assess alertness, we adapted the observation list of Vlaskamp et al. (2003). Their system discerns four alertness levels: (1) Not alert (e.g., sleep, stare), (2) Alert, self-directed (e.g., touch own clothing, stereotypical behaviour), (3) Alert, directed at environment or other non-person stimuli (e.g., look around/explore room), (4) Alert, directed at people (e.g., make eye-contact). To this, we added a fifth category: (5) Alert, directed at the interactive ball. Using momentary time sampling (MTS), one alertness score was coded for every 10-second interval. After rehearsal, two raters independently scored 20% of all sessions. A detailed analysis (outside the scope of this paper, see Embregts et al., 2018) showed an overall sufficient interrater agreement of Cohen’s kappa = 0.795 (Krippendorff, 2004).

Affective behaviour – a tailored observation scheme

A tailored observation scheme was used to assess negative and positive affective behaviour. The scheme discerned two main scales. Negative affect had three categories: (a) aggressive/physically agitated behaviour (e.g., pushing, biting, negative facial expression), (b) physical non-aggressive behaviour (includes stereotypical behaviour), and (c) verbal agitated behaviour (e.g., screaming). Positive affect consisted of (a) facial expressions (e.g., smiling) and (b) vocalisations (e.g., laughing). To take into account the idiosyncratic behaviours of the user group (Hogg, Cavet, Lambe, & Smeddle, 2001), for each participant we also obtained individual indicators of affect from support staff, as suggested by Dillon and Carr (2007). The interrater agreement was considered good enough for further analysis (positive affect: Cohen’s kappa = 0.91; negative affect: Cohen’s kappa = 0.79) (Krippendorff, 2004). Full details, including the background and reliability analysis of the data collected using this instrument can be found in Embregts et al. (2018).

Movement – Simplified Motion Energy Analysis (SMEA)

To measure movement, we used computer vision (i.e., software for automatically analysing recorded video from a camera using a computer algorithm). We implemented a method similar to the motion energy analysis by Ramseyer and Tsacher (2011) and the motion history that was successfully used by Iwabuchi et al. (2014) for analysing responses to human interaction by tracking movement of people with severe and multiple disabilities. We adapted these two methods to a simpler method that measures overall movement, which we call Simplified Motion Energy Analyses (SMEA; see Figure 2 for a visual explanation of the main technical steps in this procedure). The SMEA, implemented with OpenCV 2.1, is based on calculating how many pixels are different in two successive video frames. Figure 2 shows how we derive the difference in pixels between two subsequent, cropped, video frames (step 3–5 in the Figure), followed by calculating the sum of greyscale differences per pixel (0–255) between frames (step 6). This yielded values in the range of 0–255 times the number of pixels per frame. Finally, for every phase in a session, the average overall movement in that phase was represented as a value relative to the average overall movement in the whole session.

Figure 2. Graphical representation of our SMEA procedure, using a colleague as example subject.

Interviews

We additionally carried out semi-structured interviews with involved staff members following the sessions. Interviews provided additional and mostly illustrative information and were not systematically analysed nor reported in depth. The interviews covered the staff’s interpretation of various topics, as mostly raised by the interviewer: (1) overall response of the client, changes in alertness, affective behaviour, and movement, (2) the extent to which staff expected that the client would appreciate the intervention, (3) suggestions for improvement, and (4) the added value of the ball.

Intervention

Design approach for development of interactive ball

In contrast to the inspirational designs by Larsen (Larsen, 2015, p. 30), we focused on interactions at some distance away from the user because it was known that some users in the target group do not like to be touched. Moreover, we wanted to steer the users towards having a more external focus of their attention as part of activating them more. This excluded the use of wearables (clothing with embedded technology as well as handheld or body-worn devices). Specifically for this study, we designed an interactive, body-controlled, physically present ball of about 50 cm in diameter (see Figure 3). For reasons of safety (both from electronic, mechanic, and hygiene point of view) direct contact with the ball was not used for input or feedback. Instead, to encourage active behaviour it responded to gross motor movement, the focus of attention, and vocalisations of the user by making different sounds, wiggling and moving from left to right.

Figure 3. The movable remote controlled interactive ball of 50 cm. The ball is shown emitting different colours, both with the painted (blue yellow / high contrast) and a more transparent white outer shell.

Originally, we set out to make a fully automatic system that would allow the participant to interact with the ball independently. During a series of pilot sessions it became apparent that it was too technically challenging, with the current state of the art, for the ball to respond automatically and accurately to the wide variety of inputs for this target group. Instead, one of the researchers controlled the ball remotely based on a predefined interaction protocol (explained below) that was based on user behaviours likely to be automatically recognisable in the foreseeable future. Responses of the ball were activated by the researcher to occur immediately after the corresponding trigger from the participant.

The interactive ball as presented during the sessions

Participants were exposed to the interactive ball. The ball was operated by the researcher in response to the behaviour of participants so that it moved to the right and to the left, playing different sounds/beeps, and/or showing LEDs in different colours. Sounds (e.g., virtual instruments, animal sounds, and bells) were played using standard PC speakers in front of the participant. It was possible to adapt the parameters of these responses to individual preferences. The ball was painted in yellow with blue figures to create a higher contrast. For one participant, we additionally tailored the ball by using a white ball cover as she seemed to react positively to a more transparent cover, which showed the coloured LEDs better.

The responses of the interactive ball that were triggered during interaction depended on the behaviour of the participant, as recorded in a protocol (shown in Table 2). This protocol mapped the possible actions of the participant to a fitting response of the ball, in order to standardise the interaction patterns across the sessions.

Table 2. Interaction protocol, developed over several pilot sessions with 5 participants.

Action participant	Response ball system	Action participant	Response ball system	Action participant	Response ball system
Reach towards ball	Non-short sound^a RTYU away from hand while hand out	Leans back	Closest end (keep bumping at end)	Stretches feet, (up some time)	Non-short sound RTYU green leds and change colour 5
Move both hands	Non-short sound RTYU to the end of both hands^b turn on blue led lights 1	Leans forward (longer and slower)	Non-short sound QWE, last direction (switch direction at end) turn on yellow leds 3	Kick feet	Short sound per kick QWE half way to end of that foot's, next time to end, wiggle if both
Grab wheel of chair	To the end that is grabbed	Move forward/ backward	Short sound QWE	Wobble leg	Wiggle ball purple leds and change colour 6
Touch wheelchair	A sound QWERTYU	Shoulders up/ sits up	Non-short sound RTYU, wiggle, turn on cyan leds 4	Ball in focus	Towards biggest space left, return at end play a sound QWERTYU
Move R/L hand up	Wiggle ball, turn on blue led 1	Leans head on arm	(Every 7s) short sound QWE, towards that end	Ball in focus long >2 min	Add: purple leds and change colour 6
Move R/L hand toward right or left	Short sound QWE, that direction (keep bumping at end)	Leans head and body to R/L side	Short sound QWE further towards that direction (keep bumping at end)	NO interaction/ out of focus for >20s (repeat +−7s)	1st : towards focus or wiggle in middle if no detectable focus 2nd /60s; move + play ball's tune 3rd /90s: move + tune + green leds and change colour 5 4rd /120s: move+ tune + leds + A sound QWERTYU
Move R/L hand down	Short sound QWE, turn on leds in red colour 2	Vocalisations (if >3 min no interaction)	Interactive, QWERTYU^c,d add: towards focus or middle if no detectable focus + beep from ball
		Laughs	Play a sound QWERTYU this triggers a feedback loop

^aFor all sounds the letters represent categories of sounds. When 4x no “reaction” of participant is shown then play next sound in the same [category] and if multiple [categories] of sounds are available for this action then first go to next category;

^bWiggle is a small left and right movement of the ball, it moves but shouldn’t really roll.

^cIf not rolling in same direction roll back and forth half of the section.

^dUse of interactive sounds means same vocalisation results in playing same sound.

Procedure

Nine participants were selected by staff and researchers. With help of the service coordinators, legal representatives as well as involved support staff members were approached, given research information letters, and asked to give written consent. After receiving consent forms, support staff provided individual behavioural indicators of positive and negative affect.

All sessions took place at the participants’ day-support centre. Six participants had to postpone their final session to the next week (n = 6) due to technical problems and a group outing. As 4 individual sessions had to be dropped (ball was defective and one participant had an epileptic seizure) the number of sessions with the interactive ball varied between 8 (n = 1), 9 (n = 2), and 10 (n = 6) sessions/participant. We also received consent from support staff who were present at the sessions. We interviewed support staff twice about each participant’s interaction with the ball: once at the end of the first week, and once after finishing all the sessions. As well as this we also received informal feedback from staff members at other moments. All sessions were video recorded from three angles and combined in one video file for manual annotation. Video recordings from only the frontal camera were used to measure the amount of movement. In total, we had 86 sessions (6 × 10 sessions, 2 × 9, 1 × 8), a total of 2580 min of video recordings, 1290 min used for quantitative analyses.

After tailoring the protocol to a participant in the first three sessions, no more changes were made and it remained fixed for the remaining (five to seven) sessions. We could tailor sounds, the speed of movement, ball cover, and use of LEDs. For example, some sounds were excluded for one participant who reacted anxiously to those sounds.

Raw data were archived within a protected environment in accordance with the guidelines of Tilburg University and the University of Twente and subsequently annotated. Ratings were analysed using Microsoft Office Excel 2013 and IBM SPSS Statistics 22. For the SMEA we started by manually indicating the screen area to be analysed, and used Matlab R2012a to filter out noise (manual inspection: motion artefacts in camera or environment) and visualise the results.

Data analyses

The data have been analysed and presented on an individual basis, fitting the heterogeneous character of the group. Furthermore, we report descriptive statistics only. Impressions are based on visual inspection, comparison of the values, and discussions with staff members and the research team. If relevant and possible, quantitative impressions were illustrated with qualitative materials from the interviews at the end of the results section.

Results

Besides the numbering of the participants, we use fictitious names in this section to improve readability. For all participants, the numeric average of alertness, affective behaviour, and movement is presented in Tables 3–5 respectively. Figure 4 shows a graphical representation of the movement results for all sessions for P1. Marc, P2. Susan, and P4. Linda. These graphs exemplify how they helped to understand in a visual way whether the measured movement is noticeably higher during the intervention, as compared to directly before and after.

Figure 4. SMEA movement graphs of P1. Marc (no effect), P2. Susan (more movement), and P4. Linda (decrease in movement).

Table 3. Observed alertness averaged over last 5 ball sessions.

Client		Not Alert			Alert, self-directed			Alert, ball			Alert, environment			Alert, person
		B	D	A	B	D	A	B	D	A	B	D	A	B	D	A
1	M	15	34	27	39	33	30		9		39	21	36	7	3	8
	SD	20	36	43	9	21	21		4		21	18	27	5	2	8
2	M	0	0	0	24	8	43		38		15	3	8	61	51	49
	SD	0	0	0	19	2	16		11		12	4	7	23	14	18
3	M	11	22	46	68	54	37		10		20	14	16	1	0	0
	SD	13	36	31	9	23	24		11		13	12	13	2	0	1
4	M	0	0	0	44	21	52		46		31	22	35	26	11	13
	SD	0	0	0	15	9	21		4		12	8	22	6	4	12
5	M	41	71	94	48	22	4		2		11	5	2	0	0	0
	SD	13	30	7	15	21	7		4		9	5	4	0	0	0
6	M	31	18	63	48	38	27		26		21	18	11	0	0	0
	SD	19	10	21	13	18	13		17		13	7	10	0	0	0
7	M	31	43	44	60	49	51		7		8	1	5	0	0	0
	SD	33	31	32	35	21	26		12		8	2	10	1	0	0
8	M	49	15	7	38	61	66		6		8	17	25	5	2	2
	SD	37	13	7	23	11	17		2		8	6	15	9	3	5
9	M	45	37	58	31	35	25		13		23	15	16	1	0	0
	SD	47	45	53	24	25	31		13		25	14	24	2	0	1

Note: Showing mean and standard deviation as percentage of interval scores B:before (7.5 min) D:during intervention (15 min) and A:after intervention (7.5 min). Colours indicate recurring positive (dark grey) and negative (light grey) results for comparison of the conditions.

Table 4. Observed affective behaviour per participant (P.) before (B), during (D), and after (A) sessions.

		Positive facial expression			Positive vocalisation			Negative vocalisation			Negative facial expression			Idiosyncratic behaviour 1*			Idiosyncratic behaviour 2*			Idiosyncratic behaviour 3*
P		B	D	A	B	D	A	B	D	A	B	D	A	B	D	A	B	D	A	B	D	A
1	M	10	5	15	1	2	4	0	0	0	0	2	0
	SD	13	9	19	2	3	4	0	1	0	0	1	0
2	M	22	78	6	7	60	4
	SD	24	16	4	7	16	4
3	M	0	3	0										28	44	0	24	36	0	7	12	2
	SD	0	4	0										15	32	1	14	25	0	7	9	5
4	M	50	69	11	36	56	4							16	4	7	0	0	2
	SD	16	5	11	19	9	7							6	6	5	0	0	5
6	M	0	29	2	0	5	0				1	3	0
	SD	0	15	0	0	5	0				2	4	2
8	M							7	35	37	9	48	48
	SD							7	27	28	8	29	33
9	M	3	4	0										2	6	1	29	38	40
	SD	6	7	1										5	7	3	42	35	54

^aThe idiosyncratic behaviour (ISB): for P3. Joyce ISB 1: tong/mouth movement, ISB 2: vocalisations and ISB 3: rubbing hands; for P4. Linda ISB 1: upper body movement (rocking) and ISB 2: sad vocalisation (different from agitation); for P9. Tessa ISB 1: sighing while rotating her head, and ISB 2: folding her hands (post-hoc indicated as positive).

Note: Represented as means and standard deviation over last 5 session in percentages. Colours indicate recurring positive (dark grey) and negative (light grey) results for comparison of the conditions.

Table 5. Measured amount of movement.

Participant and condition		M	SD
P 1	Before	396.25	784.80
	Intervention	−215.92	349.42
	After	18.63	1284.22
P 2	Before	−331.34	688.77
	Intervention	753.37	386.90
	After	−1259.51	616.21
P 3	Before	156.59	249.70
	Intervention	313.65	281.92
	After	−785.80	333.46
P 4	Before	2467.08	1907.47
	Intervention	−822.66	886.43
	After	−924.10	2109.08
P 5	Before	102.28	129.41
	Intervention	93.02	63.71
	After	−265.52	204.64
P 6	Before	−37.88	148.53
	Intervention	93.01	62.67
	After	−132.33	45.93
P 7	Before	124.36	173.64
	Intervention	22.88	56.77
	After	−170.24	134.89
P 8	Before	−539.10	349.66
	Intervention	171.83	163.25
	After	260.99	213.44
P 9	Before	−49.84	169.70
	Intervention	121.57	105.00
	After	−195.20	128.83

Note: Represented in session normalised, filtered Simplified Motion Energy Analyses values. Colours indicate recurring positive (dark grey) and negative (light grey) results for comparison of the conditions.

For every measure, we present the results in more detail for only those participants who seem to show an effect, either in the hypothesised direction (more alertness, movement, and positive affect; less negative affect) or in the opposite direction. That means that some participants are not mentioned when they appeared not to respond to the ball in any of the measures.

When presenting results for a specific participant, we report the mean values over all sessions for that participant, for instance (M = 9%) is reported as (9%) unless indicated differently. After describing these core quantitative results regarding alertness, affective behaviour, and movement of the participants, we summarise the most relevant remarks and feedback from the interviews with support staff to better contextualise the meaning of the results with regard to individual differences, and possible shortcomings of the device and study.

Alertness measures

Higher levels of alertness were observed for three participants (P2. Susan, P4. Linda, and to some extent P6. Pete), when compared to the baseline directly before and after the intervention. In contrast, for P1. Marc, there was a slightly overall negative tendency (although the results differed considerably per day). For the remaining participants, the differences shown in Table 3 were not strong enough to merit separate discussion.

We will first look at the participants with higher levels of alertness in more detail. During the intervention, Susan showed a reduced amount of self-direct alertness (8%) and was instead often focused on the ball (38%). Most types of alertness were similar before and after the session; however, after the intervention, she tended to be more self-directed (43%) compared to measurements before the intervention (24%). Linda also showed less self-directed behaviour during the intervention (21%) than before (44%) or after (52%). Instead, during the intervention, there was more focus on the ball (46%). She showed alertness towards people mainly before the session (26%) and less during (11%) and after (13%). She also showed more alertness to the environment after (31%) and before (35%) than during the intervention (14%). During the intervention, Pete showed attention towards the ball regularly (26%). The change in alertness was not associated with a reduction in alertness towards the environment, the combination (ball + environment) was thus also higher during the intervention (44%) than before (21%) or after (11%). He also slept more after the intervention (63%) and before (31%) than during intervention (18%).

One participant showed clearly reduced levels of alertness. Marc did show alertness to the ball but only for a limited amount of time and in a limited number of occasions (9%). During the intervention, he showed a decrease of alertness towards the environment and towards people (21%; 3%) when compared to measurements before (39%; 7%) and after the intervention (36%; 8%), not compensated for with the additional focus on the ball. In contrast, Marc slept more during the intervention (34%) as compared to measurements before and after the intervention (14%; 27%) and was less alert to the environment.

Affective behaviour

For P2. Susan, P4. Linda, and P6. Pete, their affective behaviour improved during the intervention; those participants also showed higher levels of alertness. For P1. Marc, affective behaviour showed the opposite effect, similar to his alertness levels. For P8. Lisa, the measured affective behaviour indicated diminishment. For P3. Joyce, there was an increased quantity of her idiosyncratic stereotypical behaviour, which is regarded by care staff in this field in general as an increase of negative affective behaviour. The remaining participants either showed no affective behaviour at all (P5. Dory and P7. John), or there were no clear differences to be seen (P9. Tessa), see Table 4.

We will first look in more detail at the participants who showed improvement in their affective behaviour. For Susan, the percentages of positive affect were highest during the intervention for positive facial expressions (78%) and positive vocalisations (60%). Before the intervention, she showed more positive affect (22%; 7%) than after the intervention (6%; 4%). No signs of negative affect were observed. For Linda, self-regulatory behaviour was seen more often before (16%), and less during interaction (4%) and after the intervention (7%). During the intervention, she showed the most positive affect, both in the form of positive vocalisations (56%) and positive facial expression (69%). Before the intervention started she already showed many signs of positive affect, (50%; 36%), after the intervention the number of occurrences diminished (11%; 4%). Pete showed more positive facial expressions during the intervention (29%) than before (0%) or after (2%). He produced some positive vocalisations during the interaction (5%) and none before or after the intervention.

We will now look in more detail at the participants whose affective behaviour seemed to change for the worse with respect to our hypotheses. For Marc, the average percentage of positive facial expressions was lowest during the intervention (5%), compared to measurements before (10%) and after (15%) the intervention. For Joyce, the indicators for positive vocalisations, negative vocalisations, and facial expression were almost never seen, too rare to annotate. She showed only some positive facial expressions during the intervention (3%) and none after (0%). There was a clearer increase of her idiosyncratic behaviours during the intervention (44%, 36%,12%) compared to directly before (28%, 24%, 7%) and after (0%, 0%, 2%). The increased amount of self-regulatory behaviour (vocalisations) might have been an indicator of positive affect for Joyce, according to remarks by some staff members in the course of the study, although this was not indicated clearly when compiling the annotation scheme. For Lisa, the number of expressions of negative affect, both in negative vocalisations (35%) and negative facial expressions (48%) was higher during than before (7%; 9%), and was comparable to after (37%; 48%).²

Movement

There were two participants for whom the amount of movement changed during the intervention, compared to directly before and after (see Table 5; low values indicate less movement during a phase; high values indicate relatively more movement during a phase³). For P2. Susan, there was an increase in the measured amount of movement during the interaction with the ball. For P4. Linda, there was a decrease in the measured amount of movement. For the other participants the amount of movement remained similar, or the difference between conditions alternated between sessions. Therefore, these latter participants have not been analysed in further detail regarding their movement. To provide a clearer image on the variability of the measured movement we also plotted the measured movement as shown in Figure 4. This figure shows P1. Marc, P2. Susan, and P4. Linda. For Marc, it shows similar measurements between the conditions, whereas for Susan and Linda there is a visible effect over all sessions.

For Susan, a clear increase can be seen regarding movement during the intervention (753) when compared to measurements before (−331) or after (−1260) the intervention. The amount of movement was also more limited after the intervention compared to during and before the intervention.

Linda moved very intensively before, during, and after the intervention. She moved less during the intervention (−822) than before (2467), or after (−924). The type of movement often differed between conditions. During the intervention she was often moving towards the ball (leaning forward), or moving towards the people. Before the intervention (related to affective measures) she showed more stereotypical body-rocking (16%) than during (4%), or after (7%), and had more moments in which she played with her toy hanging from her wheelchair (43%) than during (33%), and after (33%).

Insights of staff members

Staff members provided various feedback during the interviews and spontaneous conversations with the researchers. This mainly concerned the varying appropriateness of the ball for individual participants, primarily regarding the appropriateness of the chosen modalities; how the ball relates to social interaction; and relevant day-to-day contextual differences for participants.

Interaction modalities

According to several staff members, the interactive ball might be beneficial for some clients, but would not be appropriate for all due to the type of intervention and how well it fits the interests of the participant. For instance, John’s staff member (John showed no striking effects in any of the measures) suggested “He might be more interested in a tangible device that he could hold and touch, like his fake remote control”. Also for some other participants, support staff suggested that other modalities and means of interaction might better fit their interests.

Social interaction

Several staff members suggested that for some participants a lack of social interaction and a lack of social physical contact could also play a role in the limited effect of the ball. John used to be alert and move when he was triggered with social interaction by his supporting staff member after the sessions. According to his staff, the ball simply did not suit his needs. He did notice the ball and followed it with his eyes, but seemed to have no interest in it. However, it was interesting to see that participants Susan and Linda did show alertness to people throughout the sessions. This behaviour was interpreted as social interaction by one staff member as if they “wanted to share their enjoyment with us” while playing with the ball.

Day to day differences

Finally, some participants exhibited day-to-day differences in their general behaviour, reporting this does not fit within the length of this paper. These day-to-day differences might impact their sessions with the ball. This sometimes seemed to depend on the day they had had so far. When asking the staff members about participants who showed strikingly different behaviour during a session, several times staff referred to other activities and events the clients had experienced earlier that day, such as: took a shower just before the session, went to a Jacuzzi, just woke up, slept too little, had a seizure that day, had interactions with their favourite staff member, or had an unusually prolonged walk to the location of the session. The explanations of staff clarified that contextual factors, taking place before the intervention, might have influenced the results for particular sessions.

Discussion

We investigated the effect of an interactive ball for people with PIMD on alertness, affective behaviour, and movement. The ball was developed to be responsive (react to movements, actions and vocalisations of participants) as well as to pro-actively elicit responses from the participants by displaying “unprovoked” actions (i.e., when ball was not in focus for longer time, or ball was in focus but the participant showed no actions) and by offering a certain “build-up” in responses over the course of multiple actions from the participant (as can be seen in the interaction protocol, it took into account a state of history, e.g., “long in focus”, and the response, e.g., “next sound for no reaction”). The combination of the three can be seen as (more) interactive (cf. Table 2). We compared measurements made during an intervention (playing with the ball) to measurements made directly before and after the introduction of the ball. The present study was exploratory and based on a rather small sample size. Several staff members indicated that the intervention might be beneficial for a selection of clients, but might add little for others. The results presented are in line with this remark. The impact of the interactive ball on our three outcome measures clearly differed between participants. For three participants, clearly positive effects (that is, in the hypothesised direction of more alertness, movement, or positive affect) were seen on one or more of the outcome measures, and each dimension was successfully targeted with the ball at least once. For four participants effects were seen in an unexpected direction; for three of them, individual factors seemed to play an important role in this. For the remaining three participants, no strong effects were found in either direction. These results need to be interpreted with caution but several lessons can be taken regarding the experimental design and regarding the development of future interventions.

Heterogenous user group with mixed results

The target user group is very heterogenous in personal characteristics, preferences, and abilities, which might partially explain the large differences in effects we have seen. Especially when clients cannot communicate well enough to explicitly indicate their (dis)preference for activities, effort must be made to “establish what the individual brings to the situation in terms of personality and sensitivities that makes Snoezelen beneficial, a nonevent, or positively detrimental” (Hogg et al., 2001, discussing multisensory rooms). This holds no less for the truly interactive systems that we investigate. In addition, it is not just about whether the activity does something for a particular client, but also about whether its specific effects fit the individual goals that have been set by care staff for that client. For instance, some clients might benefit from moving more, whereas for restless clients that tend to move a lot it might be an unsuitable goal to trigger movement. Influencing clients towards more external alertness might be good for some clients, but less relevant for clients already looking for social contact or clients who are vulnerable regarding the regulation of arousal.

In general, we remark that although we set out to trigger certain responses by offering the interactive ball, for some people in the target group it is conceivable that they show engagement through lessened response instead. One of the reviewers of this manuscript mentioned a young man they worked with who showed engagement by “stilling” (i.e., by reduced movement); we think this type of response might in fact not be uncommon in this population. Taking such individual variety into account, we will look in more detail at the results of the four participants whose measurements changed in the opposite direction from what we targeted with our intervention. For P1. Marc, the ball indeed seems to be a somewhat unsuitable leisure activity as it does not seem to trigger any of the targeted responses from him. For the three other participants, it is likely that individual factors also affected the results. For P3. Joyce, specific self-regulatory behaviour (vocalisations) was initially interpreted as negative affective behaviour (during the personalisation of measurements). Based on the doubts of staff members and researchers during the sessions, however, the possibility was offered that this behaviour in her case might be an indicator of positive affect. For P4. Linda, the diminishment of movement during the ball sessions was accorded a positive interpretation by staff, contrary to the assumptions underlying our setup, as in general she showed over-active and restless behaviour. For P8. Lisa, her behaviour appeared to depend on how long she was left in a certain context, regardless of activity, thus rather than being an effect of the ball. A more detailed interpretation of the types of behaviour displayed by participants might help towards such more nuanced analyses; however, the number of movement episodes in our data is not large enough to support an annotation procedure with a useful number of categories for this.

Furthermore, support staff indicated that activities outside of the sessions could influence how the session of a certain day went. We used a day-based comparison to counter such effects. Yet, it is likely that such contextual factors for some participants influenced the results. When a participant was very tired it seemed there was simply not enough energy to interact with the ball, and thus this could also reduce the chance of finding any effects from this interaction.

Reflection on study design

Given the above observations, we think that our study design was appropriate. Per-session, within-client, comparisons of the measures over a somewhat longer series of sessions (∼10 in 2 weeks) allowed us to account somewhat for the individual and day-to-day variability of mood, level of alertness and activity. Using the habituation period for the individualised tailoring of the interaction protocol ensured that the intervention had a potentially better fit with the idiosyncrasies and preferences of each client, while keeping the general interaction protocol similar across clients. Using individually tailored observation schemes for affective behaviour turned out to be crucial for interpreting the outcome of our observational measures. Finally, we obviously needed major input from the clients’ daily care staff to get a grip on all of these factors of variability. They know best what might be the right tailoring of intervention and observation schemes; furthermore, they can help with contextualising the possible meaning of the results. However, their input on the latter might conceivably be biased and subjective, and furthermore was not collected or analysed systematically enough, so we did not use this input as the basis for our main results and conclusions.

Implementation in practice and inventing future interventions

The motivation behind the present study is to work towards interactive (entertainment) products that can improve the quality of life for people with PIMD. Ideally, interventions such as the interactive ball can be rolled out in care organisations. In line with what Hogg et al. suggested regarding Snoezelen^® (Hogg et al., 2001), the present results lead us to believe that the decision to offer interactive devices should also be made on an individual basis, and should fit the goals that were originally set out for this person. Thompson and Mcferran (2015) adjusted their music therapy to musical preferences of their participants, and such personalisation is probably necessary for any kind of new product for this target group. This implies that care staff should be trained to not only determine the suitability of the device for a specific client (with preference tests, repeated habituation sessions, and some tailoring) but also to set out individualised interaction protocols, including lists of behaviour that the device should respond to.

Furthermore, there are two factors that in hindsight should be implemented better in future products for this target group: tangible interaction and addressing social connectedness. These product design implications are discussed in more detail in van Delden (2017).

Conclusion

We have performed a single case study with inter-case replication as well as within-case replication: nine people with PIMD were each exposed to an intervention in one session per day over the course of several days. Based on video recordings from three angles, we measured differences in alertness, expressions of positive and negative affect, and movement. The intervention was an interactive ball that responded to the users’ behaviour.

As can be expected, this interactive ball is probably not beneficial for all people with PIMD. We have seen differences for several participants that might be partially explained by external factors. The results are also not uniformly positive. For just a few participants the ball had positive effects, for one participant on all targeted dimensions, for two others only with respect to alertness and affect. Yet, the ball might add a new source of entertainment for several of our participants that could improve their alertness, affective behaviour, or movement. Especially in comparison to more passive activities that are often offered to this target group (such as watching TV, or a relaxation-oriented use of multi-sensory rooms), we believe that interactive products such as this ball offer potential for more active leisure activities, with or without involvement of others, for a user group that currently can only be offered a limited amount of suitable leisure activities.

Acknowledgements

The authors thank all the clients, parents, and staff, and especially Carla, Jan, Luciënne, and Gerryan, for their participation, input, and passionate enthusiasm.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

R. W. van Delden http://orcid.org/0000-0002-6592-1199

D. Reidsma http://orcid.org/0000-0002-7503-573X

Additional information

Funding

This project was supported by Dutch national research program COMMIT / [Rijksdienst voor Ondernemend Nederland, FES2011.1], health care organization Dichterbij additionally sponsored both conducting research and development of the intervention, for which the company Kitt Engineering was hired. During study preparation and data collection, two of the authors were partly employees of Dichterbij.

Notes

1 We refer to the project website for more details: http://sid.desiign.org/

2 This mainly seemed to be a duration dependent factor more than a response on the ball, according to staff members she also showed a similar increase of agitation in other situations.

3 For clarity, a negative amount of movement means that the participant has moved less than average in that phase. Counter intuitively the three values do not have to sum to zero, as the number of values used per condition will differ due to the removal of noise.