Intra- and Inter-Rater Reproducibility of the Face-to-Face and Tele-Assessment of Timed-up and Go and 5-Times Sit-to-Stand Tests in Patients with Chronic Obstructive Pulmonary Disease

Abstract

Timed-Up and Go (TUG) and 5-Times Sit-to-Stand (5STS) are frequently used in clinical practice for patients with chronic obstructive pulmonary disease (COPD). The aim of the study was to investigate the intra- and inter-rater reproducibility of the TUG and 5STS as both face-to-face and tele-assessment tests in patients with COPD. Forty-four patients with diagnosed COPD were included. Evaluations were carried out face-to-face and tele-assessment (synchronized and asynchronized). Inter-reliability between face-to-face and tele-assessment was excellent for TUG (ICC = 0.977) and 5STS (ICC = 0.970). Inter-reliability between two tele-raters was also excellent for TUG (ICC = 0.995) with the SEM = 0.04, SEM95% = 0.08, and SDC95% = 0.10 s, and 5STS (ICC = 0.990) with the SEM = 0.06, SEM95% = 0.12, and SDC95% = 0.18 s. Intra-rater reliability of the tele-assessment (synchronized) was excellent for TUG (ICC = 0.976) and 5STS (ICC = 0.964). The SEM, SEM95%, and SDC95% values were computed as 0.08, 0.16, and 0.22 s for TUG, and 0.11, 0.22, and 0.31 s for 5STS, respectively. The TUG and 5STS tests are reproducible tele-assessment measures in patients with COPD with excellent intra- and inter-rater reproducibility. The authors recommend these tests as practical assessment tools in patients with COPD at home for tele-health interventions. The reported SEM, SEM95%, and SDC95% values can be used as a minimum change that needs to be observed to be confident that the observed change is real and not, potentially, a product of measurement error.

Keywords:

• Tele-assessment
• chronic obstructive pulmonary disease
• timed-up and go test
• 5-times sit-to-stand test
• reliability

Introduction

Chronic obstructive pulmonary disease (COPD) is marked by airway inflammation and airflow limitation [1]. As a leading cause of morbidity and mortality, COPD has an ever-increasing social and economic burden [2]. Pulmonary rehabilitation (PR) is an important treatment for individuals with COPD [3]. However, PR programs face challenges such as poor participation, low compliance, and high rates of drop-out [4, 5]. In an attempt to make PR available to more patients and increase their adherence, alternative approaches such as tele-rehabilitation are recommended [6]. Telehealth innovation is considered promising assistance with in-home PR as it facilitates monitoring the patients and maintaining rehabilitation programs [7].

The fundamental component of an effective PR program is a rigorous assessment. Various assessments have been used to assess the response to PR in patients with COPD [8]. In order to assess functionality and lower limb capacity, practical tests such as 5-Times Sit-to-Stand (5STS) and Timed-Up and Go (TUG) are commonly used in patients with COPD as these tests require short time and small space and can be easily performed at home [9, 10]. The TUG and 5STS tests can evaluate the functional capacity of the lower limbs in patients with COPD [11]. Moreover, the 5STS is valid, reliable, and responsive in patients with COPD. The 5STS is an objective and practical, functional outcome measure appropriate for use in most healthcare settings [10]. The five-repetition sit-to-stand test is a predictive factor of severe exacerbations in COPD [12]. Additionally, the TUG, which can be adopted in both research and clinical settings, is responsive to PR [9]. The TUG has a potential role in assessing COPD patients’ balance [13].

Face-to-face PR programs have been forced to reduce enrollment significantly or in some cases completely shut down reason for COVID-19 pandemic [14]. Given the COVID 19 crisis, the importance of “tele-assessment and tele-rehabilitation” applications is sharply rising. Additionally, there is a need for reliable remote follow-up evaluation methods to assess the effectiveness of tele-rehabilitation programs. Although the TUG and 5STS tests are frequently used in clinical practice for patients with COPD, psychometric properties of them as tele-assessment tests have not yet been investigated in COPD. Therefore, in this study we investigated the intra-rater and inter-rater reliability of the TUG and 5STS as both face-to-face and tele-assessment tests in patients with COPD.

Methods

Participants and study design

This study was carried out between November 2020 and March 2021. Convenience sampling was used in the study. During the data collection period, 47 patients were interviewed, and 44 of them met the inclusion criteria who were clinically stable patients with COPD who had the same drug regimen over the past three weeks and were not taking any antibiotics. The diagnosis of COPD was made by a specialist (>25 years of experience in COPD) according to the global initiative for chronic obstructive pulmonary disease (GOLD) guidelines based on medical history, current symptoms, and pulmonary function tests [15].

We ensured that all participants had access to smartphones and a good-quality internet connection. Exclusion criteria were; exacerbation of COPD (within the four weeks), limitations on walking and performing the tests due to neurological or musculoskeletal disorders such as hemiplegia and injury or pathology at lower limbs, and unstable cardiac condition (within the four months).

Ethics approval

All study procedures were in accordance with the ethical standards and with the 1964 Helsinki declaration and its later amendments. The study protocol was approved by Selçuk University Faculty of Health Sciences Ethics Committee (Report number: 2020/1470). Informed consent was obtained from all participants.

Data collection

Participant’ characteristics

Demographic data and clinical characteristics of all participants were recorded. In line with the guidelines of the American Thoracic Society (ATS) and European Respiratory Society (ERS), a spirometer (Quark- SPIRO spirometer, COSMED, Roma, Italy) was used to evaluate the participants’ pulmonary function. Forced expiratory volume in 1 s (FEV1) and forced vital capacity (FVC) values were assessed, and Tiffeneau-Pinelli index (FEV1/FVC) was calculated for each participant [16].

To assess current symptoms and health status of the participants, the COPD Assessment Test (CAT) was used [17].

Measurements

Timed up and go test (TUG)

This functional test measures the time that it takes an individual to stand up from a standard chair, walk 3 m at their normal pace, turn around, walk back to the starting point, and sit down. The TUG is shown to be valid and responsive in COPD [9]. Using a stopwatch, we recorded our participants’ test scores in seconds as the primary outcome [18]. In accordance with the standards for home measurements, the dimensions of the chairs to be used for the TUG test were specified. The chair sizes required for the evaluations were given to the participants. Chairs suitable for optimal dimensions were determined according to the returns of the participants.

5-Times sit-to-Stand test (5STS)

An armless chair with a straight back and the hard seat was stabilized against a wall. The seat was 48 cm above the floor. The participants were asked to sit on the chair and come forward on the seat until their feet were flat on the floor. With their arms crossed on their chest, the participants were asked to perform a “stand-up and sit-down” once. The test was terminated for those who were unable to complete the task or needed help. Those who completed the initial task successfully were asked to perform five consecutive “stand-up and sit-down” as fast as possible. The timing was started on the command ‘go’ and stopped at the end of the fifth stand. We recorded our participants’ test scores in seconds using a stopwatch [19]. The validity, reliability, and responsiveness of the 5STS test for COPD were reported in the literature [10].

Assessment procedure

Evaluations were carried out face-to-face and tele-assessment (synchronized and asynchronized). Clinical and home measurements were made on different consecutive days (24–48 h apart).

Three trials were performed for the TUG, and the best trial was used for analysis. Participants were allowed to rest between the trials for recovery. At least 4 min of seated rest occurred between each TUG test. Sensations of breathlessness and fatigue (using the Borg scale), hemodynamic values (SpO₂ and heart rate) were assessed before and after the tests. The test was repeated when the initial values were returned [20, 21].

The minimum value of three trials of the 5STS was considered the participant’s score. Participants were allowed to rest between the trials for recovery. Patients performed three 5STSs with the appropriate number of intervals (>5 min). Sensations of breathlessness and fatigue (using the Borg scale), hemodynamic values (SpO2 and heart rate) were assessed before and after the tests. The test was repeated when the initial values were returned [22].

For the tele-assessments, the ‘WhatsApp’ phone application was used. The evaluations were made in the following order:

• Rater 1 primarily carried out all evaluations, face-to-face and in the clinic.

• The same evaluations were performed remotely (tele-assessment) and synchronously by rater 1.

• The records of the remote evaluations (made by rater 1) were evaluated asynchronously by rater 2.

• As a retest, rater 1 repeated the evaluation remotely and synchronously.

• The records of the retest evaluation (made by rater 1) were evaluated asynchronously by rater 2.

The diagram of the assessment procedure is presented in Figure 1.

Figure 1. Assessment procedure.

Statistical analysis

Statistical analyses were conducted with IBM SPSS Statistics for Windows (Version 25.0. Armonk, NY: IBM Corp.) and Microsoft 365 Excel (Microsoft Corporation, Redmond, WA, USA). The normality of the data was evaluated using the Kolmogorov-Smirnov test and investigating histograms. We determined intra-rater and inter-rater reliability using the intraclass correlation coefficients (ICCs), and Bland-Altman plots. Two-way mixed-effects models were used to assess intra-rater reliability, and two-way random-effects models were used for inter-rater reliability [23]. To calculate the standard error of measurement (SEM), a variable was computed with the difference between the score obtained during the first and second trial. The standard deviation (SD) of the differences was calculated, and this value was multiplied by $\sqrt{1}-\mathit{ICC}$. SEM95% was calculated as “SEM × 1.96”. The smallest detectable change (SDC95%) with 95% confidence interval (CI) was calculated using the formula; “$\mathit{SEM}95\mathrm{\%}\times 1.96\times \surd 2$” [24]. Repeated measures of ANOVA were used to investigate the systematic bias (error) in the repeated testing (i.e. synchronized tele-assessment) [25].

Sample size calculation

With the correlation coefficient of 0.95 between face-to-face and technology-based remote assessments, 90% power, and 95% confidence interval, the smallest sample size of the study.

Results

Two patients did not volunteer to participate in the study because they were living alone, and one patient had a stroke with a lower limb sequel. Therefore, in total, forty-four patients’ data were analyzed. Clinical and demographic data of the study sample are presented in Table 1. The patients evaluated were mostly male (n = 41, 93.2%), and the mean age was 62.27 (SD = 8.41) years. Table 2 shows the range of data of all assessments, including means and SDs.

Table 1. Clinical and demographic data of the patients with COPD.

Characteristic	Mean (SD)
Sex, male, n (%)	41 (93)
Age, years	62.27 (8.41)
BMI, kg/m^2	27.73 (5.25)
GOLD Spirometry Classification (I/II/III/IV) (%)^a	16/55/23/7
GOLD COPD group (A/B/C/D) (%)^a	41/30/7/23
Smoking (pack-year)	36.63 (14.33)
Pulmonary function test
FVC predicted (%)	72.64 (19.18)
FEV1 predicted (%)	59.66 (18.58)
FEV1/FVC	62.89 (6.90)
CAT score (0–40)	12.41 (9.22)
Comorbidities (%)
Hypertension	43
Heart failure	27
Diabetes mellitus	20
Other respiratory disease (asthma, obstructive sleep apnea syndrome)	9

a Percentages are not total 100 due to rounding.

Values are presented as mean (SD) unless specified.

COPD: Chronic Obstructive Pulmonary Disease; GOLD: global initiative for chronic obstructive pulmonary disease; SD: standard deviation; BMI: Body Mass Index; FVC: forced vital capacity; FEV₁: Forced expiratory volume in 1 s; CAT: COPD Assessment Test.

Table 2. Range of data of all assessments of TUG and 5STS tests.

	Face-to-face assessment in clinic Rater 1	Tele-assessment in home (synchronized) Rater 1	Retest tele-assessment in home (synchronized) Rater 1	Assessment in home from video records (asynchronized) Rater 2	Retest assessment in home from video records (asynchronized) Rater 2
TUG, s	8.50 (2.56)	8.50 (2.40)	8.37 (2.53)	8.35 (2.37)	8.25 (2.45)
5STS, s	11.41 (2.60)	11.38 (2.59)	11.0 (2.53)	11.19 (2.62)	10.92 (2.57)

Data are presented as mean (standard deviation).

TUG: timed up and go test; 5STS: 5-times sit-to-stand test.

Inter-reliability

All participants completed all trials of TUG and 5STS both in clinic and home, indicating that there was no floor effect for the TUG and 5STS. Results obtained from two-way random-effects models indicated that inter-reliability between face-to-face and tele-assessment was excellent for TUG (ICC = 0.977) and 5STS (ICC = 0.970). Inter-reliability between two tele-raters was also excellent for TUG (ICC = 0.995) with the SEM = 0.04, SEM95%=0.08, and SDC95%=0.10 s, and 5STS (ICC = 0.990) with the SEM = 0.06, SEM95%=0.12, and SDC95%=0.18 s. Furthermore, a visual inspection of the Bland-Altman plots (Figures 2 and 3) revealed no significant trend toward improving or worsening with regards to tests’ performance. Table 3 presents the inter-reliability results of the TUG and 5STS tests, including ICC with 95%CI, SEM, SEM95%, and SDC95%.

Figure 2. Bland-Altman plots for the Timed and Go Test. The means on the x axis are the average of two trials for the Timed and Go Test, and the differences between Timed and Go Test scores are in the y axis. (A) Inter-reliability between face-to-face and tele-assessment. (B) Inter-reliability between two tele-raters. (C) Intra-rater reliability of the tele-assessment (synchronized). (D) Intra-rater reliability of the tele-assessment from videorecords (asynchronized). The 95% limits of agreement are depicted (dashed line). SD, standard deviation.

Figure 3. Bland-Altman plots for the 5-Times Sit-to-Stand Test. The means on the x axis are the average of two trials for the 5-Times Sit-to-Stand Test, and the differences between 5-Times Sit-to-Stand Test scores are in the y axis. (A) Inter-reliability between face-to-face and tele-assessment. (B) Inter-reliability between two tele-raters. (C) Intra-rater reliability of the tele-assessment (synchronized). (D) Intra-rater reliability of the tele-assessment from videorecords (asynchronized). The 95% limits of agreement are depicted (dashed line). SD, standard deviation.

Table 3. Inter-rater reliability results of the TUG and 5STS tests.

	Inter-reliability between face-to-face and tele-assessment				Inter-reliability between two tele-raters
	ICC (95%CI)	SEM	SEM95%	SDC95%	ICC (95%CI)	SEM	SEM95%	SDC95%
TUG, s	0.977 (0.958, 0.987)	0.08	0.16	0.23	0.995 (0.978, 0.998)	0.04	0.08	0.10
5STS, s	0.970 (0.946, 0.984)	0.11	0.22	0.31	0.990 (0.974, 0.996)	0.06	0.012	0.18

ICC, intraclass correlation coefficient; CI, confidence interval; SEM, standard error measurement; SDC, smallest detectable change; TUG, timed up and go test; 5STS, 5-times sit-to-stand test.

Intra-rater reliability

Results obtained from two-way mixed-effects models indicated that intra-rater reliability of the tele-assessment (synchronized) was excellent for TUG (ICC = 0.976) and 5STS (ICC = 0.964). The SEM, SEM95%, and SDC95% values were computed as 0.08, 0.16, and 0.22 s for TUG, and 0.11, 0.22, and 0.31 s for 5STS, respectively. Intra-rater reliability of the tele-assessment from videorecords (asynchronized) was also excellent for both TUG (ICC = 0.976) and 5STS (ICC = 0.964). A visual inspection of the Bland-Altman plots (Figures 2 and 3) revealed no significant trend toward improving or worsening with regards to tests’ performance. In addition, the repeated measures of ANOVA revealed that there was no significant difference between the two synchronized tele-assessments of TUG [F(1,43)=2.567, p = 0.116]. However, the repeated measures of ANOVA revealed that there was a significant improvement in the synchronized tele-assessment retest trial of 5STS [F(1,43)=16.617, p < 0.001]. Intra-rater reliability results of the TUG and 5STS tests are presented in Table 4.

Table 4. Intra-rater reliability results of the TUG and 5STS tests.

	Intra-rater reliability of the tele-assessment (synchronized)				Intra-rater reliability of the tele-assessment from video records (asynchronized)
	ICC (95%CI)	SEM	SEM95%	SDC95%	ICC (95%CI)	SEM	SEM95%	SDC95%
TUG, s	0.976 (0.956 to 0.987)	0.08	0.16	0.22	0.976 (0.957 to 0.987)	0.03	0.06	0.08
5STS, s	0.964 (0.888 to 0.984)	0.11	0.22	0.31	0.964 (0.925 to 0.981)	0.06	0.012	0.17

ICC: intraclass correlation coefficient; CI: confidence interval; SEM: standard error measurement; SRD: smallest detectable change; TUG: Timed Up and Go Test; 5STS: 5-Times Sit-to-Stand Test.

Discussion

The current study is the first to investigate tele-assessment psychometrics of the TUG and 5STS tests in patients with COPD. According to the results, the TUG and 5STS tests can be reliably used as tele-assessment measures in patients with COPD with excellent intra-and inter-rater reproducibility.

In recent years, the globalization of health care systems has yielded an increasing need for novel telehealth services to reduce health care costs. However, the effectiveness of these systems needs to be validated concerning traditional evaluation methods. While in the 1990s, the high cost of the necessary technical equipment was an obstacle to the availability of telehealth [26], recent developments in technology and communication have provided a wide range of cost-effective options for eHealth [27]. In the present study, WhatsApp software was used for video calls between the participants and the evaluator to conduct tele-assessments. This cheap, simple, and user-friendly application is an effective way of communication within the health sector [28].

Simple functional tests, such as the TUG and 5STS, are commonly used in patients with COPD [9]. As a simple, inexpensive, and reliable test, the TUG is a valid tool for evaluating functional performance in this population [9]. There is evidence for the responsiveness of the TUG to PR [9]. To our knowledge, the psychometric properties of the TUG as a tele-assessment method are not investigated in patients with COPD. Our results indicated excellent inter-reliability between face-to-face and tele-assessment of the TUG. Similarly, there was excellent intra-rater reliability of synchronized and asynchronized tele-assessment. The TUG test requires minimal time, space, and equipment so that it can be easily administered in home settings.

As a frequently used maneuver during activities of daily living, sit-to-stand can be an indicator of lower extremity muscle function and balance [29, 30]. The 5STS test is reported to have good test-retest and inter-rater reliability in COPD [10]. Moreover, the test is responsive to PR and has a minimum clinically important difference, indicating its potential utility as an outcome measure in COPD [10]. As the first study to investigate tele-assessment psychometrics of the 5STS in patients with COPD, the present study revealed excellent inter-rater reliability (face-to-face and tele-assessment) and intra-rater reliability (synchronized and asynchronized) for the 5STS test as a method of tele-assessment. These results indicate that the 5STS test can be used as a practical functional outcome measure at home.

The absolute reliability of outcome measures helps know the precision of repeated test scores. The SEM and SDC95% have clinical utility compared to the ICC values because they are expressed in the same units as the original measurement tools. SEM is a parameter for the amount of measurement error present in a measurement tool and indicates reliability. SEM can provide a range around the observed value within which the theoretical "true" value lies. The SDC95% provides a value for the minimum change that must be observed to ensure that the observed change is real and is not potentially a product of measurement error in the tool. The SDC95% can be used in clinical practice to label individuals in a study sample as changed or unchanged. Clinicians and researchers can use the reported values for SEM, SEM95%, and SDC95% in our study as a yardstick in interpreting the TUG and 5STS scores. For example, SEM95% for the TUG and 5STS, when conducted in synchronized tele-assessment manner, was 0.16 and 0.22 s in our study. Thus, for a patient with scores of 8 s on the TUG and 10 s on the 5STS, 95% of the time, named patient’s true scores will lie between 7.84 and 8.16 s on the TUG, and between 9.78 and 10.22 s on the 5STS [31]. If a patient completed the TUG in 10 s, our SDC95% value for the synchronized tele-assessment suggests that 95% of the time, a score on the repeated test of between 9.78 and 10.22 s would indicate a truly unchanged performance. On the other hand, below 9.78 s can be interpreted as true improving, whereas above 10.22 s can be interpreted as true improving. One should note that these values are not the minimal clinically important difference values, and they give information about the measurement error.

The repeated-measures ANOVA results indicated that there was no statistically significant systematic error in the data of TUG. However, a systematic error was present for the 5STS. Since the participants significantly improved their 5STS score in the retest trial, it seems that a learning effect is present for the 5STS. Therefore, performing more than one trial for the 5STS is recommended to compensate for the bias. However, future research is warranted to determine how many trials are needed to eliminate the learning effect. Nevertheless, since the systematic error does not lower the reliability, our results for the 5STS are still valid [25].

We have a patient sample that reflects the COPD population in general, with age (patients over 60 years old), sex (majority men in Turkey), and disease severity (moderate severity). Since we excluded patients with an exacerbation of COPD (within the four weeks), limitations on walking and performing the tests due to neurological or musculoskeletal disorders such as hemiplegia and injury or pathology at lower limbs, and unstable cardiac condition (within the four months), all our patients had relatively good functional capacity. Thus, all patients could complete all TUG and 5STS trials. However, a study reported that the 5STS had a floor effect, with up to 15% of the COPD cohort unable to attempt or complete the test [10]. Hence, that study suggests that the 5STS may have increased value in better functioning patients as a functional outcome measure. In addition, due to the cultural norms of Turkey, the number of patients living alone is exceedingly small. Therefore, all patients in our study had a significant one to help them perform the tests. However, especially in western cultures, such help cannot be provided as high as in Turkey. Especially when considering stable patients with COPD living in Turkey, the external validity of our results seems reasonable. Nevertheless, the sample may not represent all age groups and sex due to the convenience sampling method used in our study. In addition, we should note that the generalizability of the results to western cultures, unstable patients with COPD, females, and patients with a limited functional capacity is limited. It is a subject for further research.

This study is subjected to some limitations. First, although our sample size was larger than the minimum required sample size, the number of women and COPD sub-groups were relatively low, and all patients had a good functional capacity to complete the study outcome measures, which limits the generalizability of the results. Second, using recorded videos in asynchronized assessment of the patients may have limited ecological validity.

In conclusion, the TUG and 5STS tests are reliable tele-assessment measures in patients with COPD with excellent intra- and inter-rater reliability. The authors recommend these tests as practical assessment tools in patients with COPD at home for tele-health interventions.

Funding

This study was not funded from anywhere.

Declaration of interest

The authors declare that there is no conflict of interest.