Go-with-the-Flow: Tracking, Analysis and Sonification of Movement and Breathing to Build Confidence in Activity Despite Chronic Pain

Figures & data

FIGURE 1. The three parts of the Go-with-the-Flow framework.

FIGURE 1. (Continued).

Note. SES = sonified exercise spaces.

FIGURE 2. (Middle) Device attached to person’s back for sonifying trunk movement during the forward reach exercise (Left). (Right) Examples of SESs for a forward reach exercise: The Flat sound is a repetition of the same tone played between the starting standing position and the maximum stretching position. The Wave sound is a combination of two tone scales (phrases), an ascending one ending at the easier stretching target and a descending one to the final more challenging target. The reaching of the easier target is marked by the highest tone (Singh et al., 2014).

FIGURE 3. Final design of the wearable device: (a) Architecture of the breathing module built using Arduino UNO. (b) Breathing sensors. (c) Front and back views of the tabard with integrated breathing sensors, button held in the person’s hand for calibration, the smartphone in its pocket on the back of the trunk and the breathing sensing module shown outside its corresponding pocket. (d) Smartphone interface for selecting sonified exercise spaces and visualization of the tracked signals.

FIGURE 4. Description of the Implemented Sonified Exercise Spaces (SESs) for the Forward Reach Exercise.

SESs for Forward Reach and for Sit-to-Stand (Phase 1)
Sound Name	Sonification of Trunk Bending	Anchor S: Standing Position	Anchor C: Point of Comfortable Stretch	Anchor M: Today’s Maximum Stretch	SES Boundaries
Flat	11 repetitions of the same tone from a piano instrument.	Sound starts as soon as bending forward starts	None	Sonification stops	Kinect: The sonification occurs only between S-M
(Figure 1 top)	These were spaced equidistant from each other between S and M.				Wearable: The sonification continues on the lowest tone after M
Wave	Formed by 2 major scales separated by the anchor point C: 7 equidistant ascending tones before C and 4 equidistant descending tones after C.		Highest tone	Lowest tone of the descending scale	The sonification is played in reverse as the person moves backwards
(Figure 1 bottom)	The piano instrument was selected			Note: In phase 2 of sit-to-stand a tone from the clarinet was played
Water	Continuous sound of moving water with a sound of a splash suggesting the body enters the water		Splash	None	The sound is continuously played until the person returns to S
Windchimes	Continuous sound of windchimes		None	None

FIGURE 4. (Continued).

SESs for Forward Reach and for Sit-to-Stand (Phase 1)
Sound Name	Sonification of Trunk Bending	Anchor S: Standing Position	Anchor C: Point of Comfortable Stretch	Anchor M: Today’s Maximum Stretch	SES Boundaries
A walk in the forest	Soundscape: stepping on leaves, birdsong, a river, sheep bleating, nightingale.		River sound	Nightingale	Beyond M, sound played at M continues.
	The list of sounds is played sequentially in the order above as the amount of stretching increases.				The sonification is played in reverse as the person moves backwards
A song	Active Music Listening: interactive orchestration of the song (“I’ll Be There for You”).		Lead vocal track	All instrument tracks together	Note: In phase 2 of sit-to-stand the sound at M is continuously played.
	Instrument tracks (shakers, percussion, lead vocal, piano, string) are incrementally added as the amount of stretching increases

Note. These SESs were also used to sonify the bending forward (till C) of the sit-to-stand first phase. The description of how movement was tracked is provided in Appendix A.

FIGURE 5. Layout of room for evaluation study for both devices.

FIGURE 6. Description of Independent and Dependent Variables for each Part of the Study.

Experiment	Device	Exercise	Independent variables	Dependent Variables
Familiarization	Both devices	Both exercises	n.a.	n.a
			(all sounds)	(Think Aloud)
Part 1: Effect of sonification paradigms	Kinect-based device	Both exercises	3 SESs: wave sound, naturalistic soundscapes, active listening	—Actual amount of movement measured by the device
				—Perceived amount of movement
				—Awareness of movement
				—Motivation in moving
				—Confidence in moving
				—Relaxation
Part 2: effect of sonification information: (2.a) without breathing; (2.b) with breathing	Wearable device	Forward reach	4 SES: no sound, flat sound, wave sound, water sound
Part 3: Effect of sonification in transferring skills	Wearable device (without breathing)	Forward reach with and without target	4 SES: no sound, flat sound, wave sound, water sound
Part 4: Efficacy of sonification alterations on awareness of avoidance strategies and movement exploration	Both devices with alteration option on		n.a.: explorative study all alteration techniques presented in Box 3 of Figure 1	n.a: explorative study (Think Aloud)

Note. Within each part the sound conditions were randomized. SES = sonified exercise space.

FIGURE 7. Mean (± SE) perceived and actual bend angle for all four sound conditions in the study with the wearable device.

Note. The perceived angles were obtained by translating the 1 to 5 ratings to the respective range of angles centred at 15°, 30°, 45°, 60°, 75°. *Significant differences, p < .05.

FIGURE 8. Mean (± SE) ratings (0 = worst to 6 = best) on awareness, performance, motivation, and relaxation for all four sound conditions in the study with the Kinect (left) and the wearable device (right).

Note. For sake of conciseness, the significant differences are reported in the text and in Figures B.2–B.5 in Appendix B.

FIGURE 9. Distribution of results for all the sounds.

Note. Nature-bin refers to both Kinect nature sound and water sound from the Smartphone device. Tones-bin refers to both Smartphone and Kinect tone sound. The flat sound was never selected.

FIGURE A1. (Left) To detect forward reach movement and related guarding strategies, the system monitors the distances between shoulders, hips, and feet. (Right) The progress of the sit-to-stand movement is detected by monitoring the angles between trunk, thigh, and crus.

FIGURE B1. Results from Statistical Comparisons between the Sound Conditions (Different Amounts of Information) on Perceived and Actual Bend Angle During Forward Reach Exercising Using the Wearable Device. Bonferroni Correction was Applied to Multiple Comparisons (p = .008 Corresponding to a Significance Level of α = 0.05).

Effect of Information on Performances (Wearable Device)
Measure	Friedman Tests/ANOVA	Wave vs. No Sound	Water vs. No Sound	Flat vs. No Sound	Wave vs. Water	Wave vs. Flat	Water vs. Flat
Perceived bend angle	χ^2(3) = 14.59, p = .002	Z = −2.49, p = .0065	ns	ns	ns	Z = −2.81, p = .0025	ns
Actual bend angle	F(1.9, 26.5) = 4.65, p = .020	t(14) = −2.72, p = .008	ns	ns	ns	t(14) = −3.11, p = .004	ns

Note. Nonsignificant comparisons are marked as ns.

FIGURE B2. Statistical Comparison of Sonification Effects During Physical Activity using the Kinect-based Device on Awareness, Performance, Motivation and Relaxation.

Effect of Sonification Paradigms on Exercising (Kinect Device)
Measure	Friedman tests	Wave Sound vs. Active Listening	Wave Sound vs. Naturalistic Soundscape	Active Listening vs. Naturalistic Soundscape
Awareness	χ^2(2) = 11.17, p = .004	ns	Z = −2.81, p = .0025	Z = −2.20, p = .0014
Performance	χ^2(2) = 4.67, p = .097	—	—	—
Motivation	ns	—	—	—
Relaxation	ns	—	—	—

Note. Bonferroni correction was applied to multiple comparisons (p = .0017 corresponding to a significance level of α = 0.05). Nonsignificant comparisons are marked as ns.

FIGURE B3. Statistical Comparison of Sonification Effects During Forward Reach with the Wearable Device on Awareness, Performance, Motivation and Relaxation.

Effect of Sonification Paradigms on Exercising (Wearable Device)
Measure	Friedman Tests	Wave vs. No Sound	Water vs. No Sound	Flat vs. No Sound	Wave vs. Water	Wave vs. Flat	Water vs. Flat
Awareness	χ^2(3) = 33.025, p < .001	Z = −3.24, p = .0005	Z = −2.96, p = .0015	Z = −2.40, p = .008	Z = −3.30, p = .0005	Z = −3.20, p = .0005	ns
Performance	χ^2(3) = 32.819, p < .001	Z = −3.23, p = .0005	Z = −3.13, p = .002	ns	Z = −3.10, p = .001	Z = −3.21, p = .0005	ns
Motivation	χ^2(3) = 25.935, p < .001	Z = −3.08, p = .001	Z = −2.83, p = .0025	ns	ns	Z = −3.07, p = .001	ns
Relaxation	χ^2(3) = 18.892, p < .001),	Z = −2.54, p = .0065	Z = −2.53, p = .0065	ns	ns	Z = −2.54, p = .065	ns

Note. Bonferroni correction was applied to multiple comparisons (p = .008 corresponding to a significance level of α = 0.05). Nonsignificant comparisons are marked as ns.

FIGURE B4. Results from the Comparisons of the Effects Between Sound Conditions with and Without Target on Awareness, Performance, Motivation, and Relaxation.

Effect of Target Presence on Transferring From Exercise to Functioning (Wearable Device)
Conditions	Measure	Flat	Wave	Water
Target vs. no target	Awareness	ns	Z = −2.04, p = .041	ns
	Performance	ns	Z = −2.04, p = .041	ns
	Motivation	Z = −2.21, p = .027	Z = −2.04, p = .016	Z = −1.98, p = .047
	Relaxation	ns	ns	ns

Note. Nonsignificant comparisons are marked as ns.

FIGURE B5. Results from Statistical Comparisons Between the Sound Conditions (Different Amounts of Information) Using the Wearable Device and Performing Movement Aimed Towards a Target.

Effect of Sonification Type on Functioning (Wearable Device)
Measure	Friedman Tests	Wave vs. No Sound	Water vs. No Sound	Flat vs. No Sound	Wave vs. Water	Wave vs. Flat	Water vs. Flat
Awareness	χ^2(3) = 34.80, p < .001	Z = −3.32, p = .0005	Z = −2.96, p = .0015	Z = −2.57, p = .0065	Z = −3.22, p = .0005	Z = −3.22, p = .0005	ns
Performance	χ^2(3) = 33.18, p < .001	Z = −3.23, p = .0005	Z = −3.13, p = .001	ns	Z = −3.10, p = .001	Z = −3.21, p = .0005	ns
Motivation	χ^2(3) = 32.14, p < .001	Z = −3.22, p = .0005	Z = −3.13, p = .001	Z = −2.39, p = .008	Z = −2.99, p = .0015	Z = −3.20, p = .0005	ns
Relaxation	χ^2(3) = 13.08, p = .004	ns	ns	ns	ns	ns	ns

Note. For each type of measure, Friedman’s test between the three conditions and planned pairwise comparisons using Wilcoxon (Bonferroni correction p = .008 corresponding to a significance level of α = 0.05) are presented. Nonsignificant comparisons are marked as ns.

Singh, A., Klapper, A., Jia, J., Fidalgo, A., Tajadura-Jimenez, A., Kanakam, N., … Williams, A. (2014). Motivating people with chronic pain to do physical activity: Opportunities for technology design. Proceedings of the CHI 2013 Conference on Human Factors in Computer Systems. New York, NY: ACM.

 Google Scholar