Effects of visual feedback on motion mimicry ability during video-based rehabilitation

Abstract

The motion mimicry ability of patients facilitates execution of therapy moves based on visual observation of rehabilitation exercise videos, which can help speed up the recovery process. This study investigates the effects of visual feedback on the mimicking ability of human subjects in video-based rehabilitation. Inertial Measurement Unit (IMU) sensors was used, which provide a portable system to detect human motion tracking, allowing for experiments to be conducted without space restrictions and provide a greater variety of actions that can be tested. In the experiment, healthy subjects were shown a video of an instructor performing a certain movement task and had to mimic actions to the best of their ability. A real-time visual feedback system, based on input data from IMU sensors, was introduced to inform subjects of the accuracy of their mimicking actions. Subjects were tested with and without feedback and the relevant joint angle data was collected to determine the individual’s mimicking ability. Our results showed a significant improvement in subject’s mimicking ability from “no feedback” to “feedback” condition. The key implication of the findings is that visual feedback provides an extrinsic source that allows patients to better synchronize their hand-eye coordination during mimicry. Potential prospective works will investigate the relevance of motion mimicry mechanism in home-based rehabilitation.

Keywords:

• Inertial Measurement Unit
• video-based rehabilitation
• mimicry
• visual feedback

Public Interest Statement

The motion mimicry ability of patients allows them to perform physiotherapy exercises based on visual observation of rehabilitation exercise videos, which can help speed up their recovery process. This study investigates the effects of visual feedback on the mimicking ability of human subjects in video-based rehabilitation. The mimicking ability can be measured as the combined accuracy and speed of the motion relative to the demonstration. Our experimental studies showed significant improvements in the subject’s mimicking ability from “no feedback” to “feedback” condition. The key implication of the findings is that visual feedback provides an extrinsic source that allows patients to better synchronize their hand-eye coordination during mimicry.

Competing Interests

The authors declare no competing interest.

1. Introduction

Telerehabilitation is fast gaining popularity in our society and it involves the patient mimicking a series of video-based exercises for rehabilitative purposes (McGarry & Russo, 2011), especially in the motor performance of stroke patients (Michaelsen, Dannenbaum, & Levin, 2006; Salbach et al., 2004; Whitall, Waller, Silver, & Macko, 2000). The basis of mimicking comprises action observation and performance of the observed task. The ability of patients to mimic the exercises accurately is paramount to the recovery process as well as to reduce potential injuries from misuse (Pasch, Bianchi-Berthouze, van Dijk, & Nijholt, 2009). The mimicking ability also decreases with increasing complexity of the task (Neo, Low, Krishnan, Barraza, & Yeow, 2013) and there is a need for a feedback system for safety reasons. Augmented feedback, or external feedback, is used to enhance task-intrinsic feedback. The use of augmented feedback systems (verbal, visual or physical guidance) in rehabilitation can greatly enhance a motor learning in subjects (van Vliet & Wulf, 2006), by providing information regarding the performance and quality of movement (Muratori, Lamberg, Quinn, & Duff, 2013).

The mirror neuron system (MNS) is defined as motor-related areas that are activated when performing an action and observing another person’s movement (Yutaka, Eizaburo, Naoki, Naoyuki, & Shin-ichi, 2013). Action observation activates certain motor-related areas, like the supplementary motor area, premotor area and anterior parietal area (Grafton, Arbib, Fadiga, & Rizzolatti, 1996; Rizzolatti, Fadiga, Gallese, & Fogassi, 1996). MNS is thought to contribute to imitation of an action by extracting motor parameters conveyed by visual information acquired by observing movements (Buccino, Binkofski, & Riggio, 2004; Oouchida & Izumi, 2012). Celnik et al. (2006) conducted a study to show that the observation of movements when incorporated with physical execution of the observed action enhances the effects of motor training. This was similarly concurred by Buccino, Solodkin, and Small (2006) and Ertelt et al. (2007). The concept of combining the observed action with physical execution of the task has strong basis in the MNS where mirror neurons are more activated during the action execution than action observation (Iacoboni & Mazziotta, 2007).

This paper aims to compare and quantify the mimicking ability of subjects, with and without a visual feedback system, for future applications in tailoring and assessing rehabilitation programmes. The feedback device integrates Inertial Measurement Unit (IMU) sensors, which are orientation sensing units that provide a portable system for human motion tracking. This allows for experiments to be conducted without space restrictions, providing a greater variety of outdoor actions that can be tested. IMU sensors has been used in the rehabilitation field for upper (Cifuentes et al., 2013) and lower limb (Giggins, Kelly, & Caulfield, 2013) rehabilitation exercises, home-based telerehabilitation (Zheng, Black, & Harris, 2005) and rehabilitation robotics (Cifuentes et al., 2012). Joint kinematic data obtained from IMU will be used to correlate and quantify the mimicking ability of subjects. An experiment involving 10 healthy participants will be conducted. Basic rehabilitation tasks such as flexing of elbows, raising of legs and stretching of shoulders, were adapted from the Shoulder Surgery Exercise Guide, Knee Arthroscopy Exercise Guide and Rotator Cuff and Shoulder Conditioning Program of the American Academy of Orthopaedic Surgeons. The results obtained showed significant improvements in mimicking ability with visual feedback and this will have future implications for rehabilitation of individuals after surgery, stroke and Parkinson patients, the elderly and potential for home-based rehabilitation programmes. We have been developing and experimenting on medical devices for rehabilitation (Yap, Goh, & Yeow, 2014, 2015; Yap, Lim, Nasrallah, Goh, & Yeow, 2015; Yap, Nasrallah, et al., 2015) and prosthetic applications (Chua & Chui, 2016; Chua, Chui, Chng, & Lau, 2013; Chua, Chui, & Teo, 2015; Chua, Chui, Teo, & Lau, 2015).

Section 2 describes the methods and materials which includes motion sensor unit, experimental set-up and data analysis. Section 3 presents the results of the experiment while Section 4 discusses the results and limitations of the study.

2. Methods and materials

2.1. Motion sensor unit

The sensor unit comprises an Arduino Pro Mini (3.3 V, 8 MHz) microcontroller (SGBotics, Singapore) with ATmega328, 9 degrees-of-freedom MinIMU-9 v3 IMU (Pololu, USA) and a XBee 2 mW PCB Antenna Series 2 wireless module (SGBotic, Singapore). In a previous calibration study, the accuracy of the sensor unit was compared with an optical motion capture system Vicon Nexus 1.8.3 and was found to have a comparable accuracy of 96.1–96.5% in the y-axis. The sensor unit was encased in a customized belt for easy donning onto the human subjects (see Figure 1).

Figure 1. Sensor unit encased in a customized belt with adjustable width.

2.2. Experimental set-up

2.2.1. Participants

Ten healthy individuals (mean age 23.4 ± 2.3 years, with a range of 20–29 years, five males and five females) were recruited for this study. The exclusion criteria were: a history of joint injuries or deformities or colour blindness. Informed consent was obtained from all subjects, in accordance with the approval from the National University of Singapore Institutional Review Board (NUS-IRB).

2.2.2. Procedure

Subjects were asked to perform three different simple movement tasks in a randomised sequence (Figure 2). The three movement tasks comprise: (A) elbow flexion (flexing the elbow joint), (B) leg raise (flexing the hip joint) and (C) shoulder stretch (shoulder abduction). Subjects were asked to perform test actions with the dominant hand and leg. The sensor unit was secured in a customized belt with adjustable width to accommodate a range of limb sizes. It was strapped on at the respective positions for each corresponding tasks: (A) mid-forearm, (B) mid-thigh and (C) on upper arm (next to elbow joint).

Figure 2. Example photos of instructor performing movement tasks (A) elbow flexion, (B) leg raise, and (C) shoulder stretch.

There are two test conditions in this study: (1) no feedback and (2) feedback. “No feedback” and “feedback” were defined as the conditions where the subjects copied the instructor’s action without and with accuracy feedback from the sensor respectively. All three movement tasks were performed in both test conditions. Each of the movement tasks was repeated three times in each test condition. Subjects were given at least 5 min of rest between test conditions. Both the instructor’s video and feedback system were shown on a parallel screen in front of the subject. The “no feedback” condition aimed to test the natural mimicking ability of a healthy individual and acted as a control. At the beginning of each task, subjects were shown a video of the instructor performing the task and were asked to imitate them. The “feedback” condition aimed to test the effect of a real-time visual feedback system on the mimicking ability of a healthy individual. Similar to the “no feedback” condition, subjects were shown the same video of the instructor performing the task and were asked to imitate them. They were also shown a visual accuracy feedback system (a range of lights with different colours) to give an indication on how well they were mimicking the task. Subjects were asked to aim for the “Best” region (Figure 3).

Figure 3. Feedback system interface showing the “Best” region switched on.

Notes: (From left to right) The percentage range for each region are: Poor (0–49%), Average (50–69%), Good (70–89%), Best (90–110%), Good (111–130%), Average (131–150%), Poor (151–200%).

2.2.3. Visual feedback system

The interface of the feedback system was created with Processing 2.0, based on real-time input from the sensor unit. It consists of seven different coloured regions to indicate the mimicking ability of the subject. A specific coloured region will be illuminated when the sensor reading falls into its assigned percentage accuracy range.

The target angle for each mimicking attempt is equivalent to the angle of raise and flexion performed by the instructor. A 100% score was achieved by the subject if he or she reaches the same angle as the instructor. At the beginning of the action, the regions are switched on from the leftmost region (Poor: 0–49% of the target angle). As the action progresses to its maximum point (where the subject’s action is similar to that of the instructor’s), the regions are progressively illuminated from left to right (Figure 3). If the subject exceeds the angle of motion set by the instructor, the score decreases and the highest percentage subsequently recorded. As the action returns back to the start, the illumination returns back to the leftmost region. For example, for a subject who was able to perform the action within the “Best” region, the regions will be illuminated as follows: Poor (0–49%)—Average (50–69%)—Good (70–89%)—Best (90–110%)—Good (70–89%)—Average (50–69%)—Poor (0–49%).

The time taken by the subject to mimic the instructor is also a critical component measured. Similar to the angle of motion, the subjects are required to reach the target time taken by the instructor. The scoring of the time taken by the subject relative to the instructor is as follows: Poor (0–49%)—Average (50–69%)—Good (70–89%)—Best (90–110%)—Good (111–130%)—Average (131–150%)—Poor (151–200%). The overall mimicking score will be computed as the average of both scorings of angle and time taken.

2.3. Data analysis

The data of the subjects were compared against the instructor data (instructor data collected while filming video in a previous setting) to determine the mimicking ability of the subject in Equation (1). The mimicking ability of the subject is defined by how closely the individual is able to imitate the target angle and time taken for the movement task performed by the instructor. The terms Angle _subject, Angle _instructor, Time _subject and Time _instructor in Equation (1) refers to the measured output values for the subject and instructor respectively. As the three movement tasks have different range of motion (ROM) and speed, the subject output data is made proportional for each task accordingly.

For each movement task, there are six data sets (three data sets from “no feedback” test condition, three data sets from feedback test condition). Each data-set consists of 3 repetitive actions. A paired t-test was performed between “no feedback” and “feedback” conditions (Table 1).

(1) $$\text{Mimicking}\text{Ability}=\left[\left(\frac{Angl{e}_{\text{subject}}}{Angl{e}_{\text{instructor}}}\times 100\right)+\left(\frac{Tim{e}_{\text{subject}}}{Tim{e}_{\text{instructor}}}\times 100\right)\right]÷2$$(1)

Table 1. Summary of mimicking ability of subjects for each movement task

Movement tasks	Average mimicking ability (%)		p-value
	No feedback	Feedback
Elbow flexion	118.3 ± 9.4	105.8 ± 2.7	<0.001^*
Leg raise	121.4 ± 22.5	103.1 ± 4.8	<0.05^*
Shoulder stretch	111.8 ± 10.7	98.4 ± 5.7	<0.001^*

^* p-value < 0.05. Hence, there is significant difference in the “No feedback” and “Feedback” conditions for a particular movement.

3. Results

For all subjects, there was a significant improvement in mimicking ability for the “feedback” test condition, as compared to the “no feedback” condition. (Figure 4). For task A, 9 out of 10 subjects show a significant difference between “no feedback” and “feedback” conditions. For task B, 6 out of 10 subjects show a significant difference between “no feedback” and “feedback” conditions. For task C, 7 out of 10 subjects show a significant difference between “no feedback” and “feedback” conditions.

Figure 4. Results for each movement task shown with error bars (standard deviation across three trials) (A) elbow flexion, (B) leg raise, and (C) shoulder stretch.

Note: The “no feedback” and “feedback” conditions are depicted by the light and dark grey bar respectively.

The “Best” region in the feedback system is assigned a range of 90–110% mimicking ability and is highlighted in Figure 5 as a 100% line for visual comparison against subject’s performance. The graph is plotted against 100% action cycle, where the action cycle is normalized to the total amount of time taken to complete three repetitive actions (Figure 5). For task A, the average mimicking ability of the subjects, 108.85–127.73% (Figure 5), fall in the “Best” to “Good” region (90–110% and 111–130%) for the “no feedback” condition. For task C, the average mimicking ability of the subjects, 101.02–122.50% (Figure 5), fall in the “Best” to “Good” region (90–110% and 111–130%) for the “no feedback” condition. Additionally, the maximum points of the curve for the “no feedback” condition show greater standard deviation as compared to the “feedback” condition. A similar trend is seen in standard deviation values reported in Table 1.

Figure 5. Average graph for each movement task for all 10 subjects (representative of 1 data-set) (A) elbow flexion, (B) leg raise, and (C) shoulder stretch.

Notes: The “no feedback” condition is depicted by a black line with gray shaded area (standard deviation). The “feedback” condition is depicted by a black dashed line with light gray shaded area (standard deviation). The 100% line provides a visual comparison of the maximum points for the “no feedback” and “feedback” conditions.

A paired t-test was performed to compare the mimicking ability for each movement task between “no feedback” and “feedback” conditions. The p-value for all movement tasks is less than 0.05 (Table 1). Therefore, all movement tasks show a significant improvement in mimicking ability from “no feedback” to “feedback” condition.

4. Discussion

4.1. Enhancement of mimicry via external feedback

This study aims to investigate the mimicking ability of subjects, with and without feedback, in video-based rehabilitation using a custom-built wearable IMU-based sensor device. The key findings are (a) significant improvement in mimicking ability from “no feedback” to “feedback” condition, (b) feedback system provides subject with control of motion and (c) subjects have natural mimicking ability of about 90–130% for upper limb tasks.

The significant improvement in mimicking ability from “no feedback” to “feedback” condition is expected of healthy study recruited in this study. Healthy subjects are not limited by high muscle tones or spasticity of muscles, pain after surgery or neurological impairment when performing the movement tasks. Some implications of this study concern rehabilitation programmes tailored for stroke rehabilitation (Whitall et al., 2000), rehabilitation for patients after surgery and home-based rehabilitation. By providing patients with a visual indication of their performance with respect to the goals set by physiotherapists, patients can work towards improvement. Also, this study provides a quantitative method for physiotherapists to track progress and set new goals for patients. For home-based rehabilitation, the feedback system can be integrated with online portal where physiotherapists can to monitor performance of patients over time and set new goals for rehabilitation. The effect of existing rehabilitation training programmes can also be evaluated through a quantitative method.

Another observation made during the testing process is that subjects tend to perform the action slower during the feedback test condition in order to reach the “Best” region. The result of this observation can be seen from smaller standard deviation in the “feedback” condition (Table 1 and Figure 5) suggesting that subjects were able to achieve greater control of the task at hand.

4.2. Sensory focus shift

During the “feedback” condition, subjects were shown both the instructor video and feedback system. We observed that subjects have the tendency to focus on the feedback system once they visually register the motions performed by the instructor, instead of looking at both the instructor video and the feedback system. While subjects show improvement in mimicking ability during the “feedback” condition, over-reliance on the feedback system is not encouraged in rehabilitation (Salmoni, Schmidt, & Walter, 1984; Schmidt, 1991). However, van Vliet & Wulf (2006) explains that feedback can be categorized as “intrinsic” and “extrinsic”, where intrinsic feedback refers to the individual’s innate representation (based on sensory perceptions) of the action to be performed and extrinsic feedback as external feedback provided in addition to intrinsic feedback. In this study, the intrinsic feedback represents the “no feedback” condition and the extrinsic feedback represents the “feedback” condition. According to van Vliet & Wulf (2006), extrinsic feedback for learning in healthy subjects, promotes a more automatic type of control and frequent feedback does not constrain the learning process. Therefore, the use of the feedback system with frequent feedback is acceptable to promote learning in healthy subjects only. The feedback system only serves enhance the mimicry of the subjects, but cannot replace an instructor in displaying the exercises since there is a requirement for visual demonstrations for precise mimicking. More work has to be done to investigate if the same principles apply to stroke patients since their intrinsic feedback systems may be undermined, making it difficult for them to determine how much has to be done in order to improve performance (van Vliet & Wulf, 2006).

4.3. Limitations

There is a need to extend the subjects to include stroke patients in future and to vary the complexity of the exercises since the complexity of the movement tasks in this study may not be complex enough, for healthy subjects, to assess their mimicking ability. Current results for healthy subjects do not necessarily translate to similar results in stroke or handicapped patients and the complexity of the task should be matched to the physical capability of the subjects in future works.

This study determines the mimicking ability of the subject by examining only a single plane (i.e. sagittal or frontal) of motion. Since the movement tasks spans throughout all three planes, the IMU data in all three axes can be used to increase the accuracy of computing the mimicking ability.

Another observation is that healthy subjects have a natural mimicking ability of 90–130%. This is expected for healthy subjects with functioning intrinsic feedback systems and not hindered by pain during movement of limbs. In Figure 4, some subjects showed no significant difference between the “no feedback” and “feedback” conditions in two out of the three movement tasks. This may be due the varying levels of ability between subjects. As the sample size for this study is small, further prospective research with larger samples is required to determine the probable reason.

5. Conclusion

The key finding of this study is the significant improvement in mimicking ability from “no feedback” to “feedback” condition. The feedback system provides subjects with control of motion and subjects were found to have a natural mimicking ability of about 90–130% for upper limb tasks. Some implications of this study concern rehabilitation programmes tailored for stroke patients, patients after surgery, strength training in elderly and home-based rehabilitation. By providing patients with a visual indication of their performance with respect to the goals set by physiotherapists, patients can work towards improvement. Also, this study provides a quantitative method for physiotherapists to track progress and set new goals for patients.

Funding

This work was supported by the Fédération pour la Recherche sur le Cerveau [grant number R-397-000-231-112].

Additional information

Notes on contributors

Vanessa Wei-Lin Mak

The authors belong to the Evolution Innovation Lab at NUS, headed by Dr Raye Yeow Chen-Hua. The lab focuses on bio-inspired engineering research. Together with his team of engineers, students and clinical collaborators, their vision is to study the mechanisms of nature and to develop medical technology that target important clinical unmet needs, particularly in the aspect of smart assistive devices, rehabilitation, surgical robotics, soft robotics, biomechanics and wearable sensors. The ultimate aim is to translate these technologies into commercialized products that fulfil real-world medical unmet needs.