Reliability and validity of the public transportation use assessment form for individuals after stroke

Abstract

Purpose

To examine the reliability and validity of the public transportation use assessment form (PTAF), which was developed for assessing the performance of tasks during public transportation use.

Materials and methods

Fifty consecutive patients admitted after a stroke to a convalescent rehabilitation hospital and received field-based practice for public transportation use were enrolled. A physical therapist (PT) and an occupational therapist (OT) independently evaluated actual participant performance using the PTAF. Its internal consistency, inter-rater reliability, and construct validity were examined against other clinical measures related to the ability to use public transportation.

Results

Cronbach’s coefficient alpha for the internal consistency for overall PTAF was 0.84 and 0.88 for PTs and OTs, respectively. Cohen’s weighted κ coefficient for the inter-rater reliability for each item ranged from 0.61 to 0.83. Intraclass correlation coefficients for the inter-rater reliability for the mean scores of the items comprising the PTAF were 0.90 for all 14 items, 0.76 for items required for train use only, and 0.88 for items required for bus use only. The correlation coefficients for the construct validity between PTAF and clinical measures ranged from 0.38 to 0.59 (p < 0.05).

Conclusions

The PTAF showed sufficient internal consistency, intra-rater reliability, and construct validity.

Implications for rehabilitation

• This study illustrated the inter-rater reliability of the public transportation use assessment form (PTAF), indicating that the PTAF can be used for reliable assessment independent of the rater.

• The PTAF showed good internal consistency, indicating that each item in the PTAF consistently assessed the ability of patients with stroke to use public transportation.

• The PTAF correlated with assessment tools such as walking ability, balance, motor paralysis, intelligence, and activities of daily living, indicating that it reflects the functions and abilities necessary to use public transportation.

Keywords:

• Cerebrovascular disorders
• instrumental activities of daily living
• task performance
• validity
• reliability

Introduction

For community-dwelling individuals, the availability of transportation, such as cars and public transport, is important for moving outdoors and participating in society [1,2]. After a stroke, individuals are often unable to drive a car due to physical and cognitive deficits [3,4], and public transportation is one of their primary modes of transportation. Therefore, using public transportation is one of the important rehabilitation tasks for individuals post-stroke to expand their living spaces [5].

The use of public transportation, which includes various processes, is difficult for individuals post-stroke [3,6–9]. Studies have shown that after a stroke, many individuals have difficulty using trains and buses [3,10,11]. For example, a study in Sweden investigating the frequency of train and bus use before and after stroke onset reported that 28 out of 55 (51%) individuals decreased or stopped using public transportation after stroke onset [3]. Difficulty in using public transportation for individuals post-stroke is a barrier to activities and participation in the community [12,13]. Therefore, support is needed through rehabilitation provided by health care professionals [14], and appropriate intervention begins with assessing performance.

However, few established assessment tools are focused on the use of public transportation by individuals after stroke. Therefore, we developed the public transportation use assessment form (PTAF), a tool consists of four categories and 15 subtasks necessary for individuals post-stroke to use public transportation and assesses the degree of independence of each subtask at three levels [15]. Before using the PTAF in clinical settings and making an intervention program, the reliability and validity of the assessment form need to be examined [16]. For example, confirmed inter-rater reliability will allow reliable assessment of patient performance without depending on the rater. Moreover, demonstrated validity will verify that the assessment results using the PTAF can appropriately reflect the performance of individuals post-stroke when using public transportation. Therefore, confirming the reliability and validity will facilitate the interpretation of the results of the PTAF assessment in clinical practice and contribute to considering interventions based on more accurate results. In a previous study, we confirmed the content validity of the PTAF, but its reliability and other validities were not verified [15]. The international standard for scale development, consensus-based standards for the selection of health measurement instruments (COSMIN), provides nine measurement properties [17], of which PTAF has only validated one [15]. The purpose of this study is to examine the inter-rater reliability, construct validity, and internal consistency of the PTAF in clinical use for individuals after stroke.

Materials and methods

Study setting

The study setting was a 160-bed subacute rehabilitation hospital in Japan. Patients are eligible for admission to the Kaifukuki Rehabilitation Ward (KRW) within two months of stroke onset. KRW is the main facility providing subacute intensive rehabilitation covered by Japan’s medical insurance system, and patients can stay up to six months after admission [18]. Since 2017, rehabilitation outside the KRW institution has been covered by medical insurance when the rehabilitation is related to activities of daily living [19]. In the hospital, training in public transportation use has been conducted during hospitalization according to the patient’s needs.

The training was conducted in Narashino City (population of 172 000 in 2017 [20]) and Funabashi City (population of 632 000 in 2017 [20]) in northwestern Chiba Prefecture, which is adjacent to Tokyo. It takes 30–40 min to reach Tokyo by train; trains and buses account for 30% of the travel transportation modes used [21]. The daily number of passengers at each station used in training ranged from 11 000 to 137 000 [22,23].

The study conforms to the COSMIN [17] and is examined for reliability and validity using classical test theory. A completed COSMIN checklist is provided in the online Supplemental Table 1. The study protocol was approved by the ethics review committee at Tokyo Bay Rehabilitation Hospital (approval number: 170). Written informed consent was obtained from all participants prior to the study.

Assessment form

The PTAF we developed is an assessment form to assess individual subtasks that comprise using public transportation for inpatients post-stroke by observing their behavior in clinical settings (Figure 1) [15]. The form was developed primarily for patients who can walk with or without ambulatory assistive devices. The PTAF includes 15 items, including the item “other problems observed”; the other 14 items are classified into four categories (plan for going out, mobility, using trains, and using buses), which are necessary for patients to use public transportation. Each subtask is scored as follows: 3, independent (the individual can complete the task without requiring any help from therapists); 2, requires supervision or verbal assistance (the individual can complete the task with supervision or needs verbal assistance from therapists); 1, requires assistance (the therapists need to assist the individual physically or the therapist needs to manipulate equipment to complete the task); and N, not applicable (the individual does not perform the task, e.g., items in the category “Using trains” do not apply to those who use only buses). The mean score was calculated by dividing the total score by the number of items (excluding the items marked as not applicable [N]).

Figure 1. Public transportation use assessment form.

The actual form of the public transportation use assessment form.

Participants

A total of 50 consecutive individuals post-stroke, who were admitted to the hospital from May 2017 to July 2018 and received field-based practice for public transportation use during hospitalization, were enrolled. Those who practiced using public transportation met all the following criteria: (1) the individual was expected to use public transportation after discharge from the hospital, (2) the individual or their family wanted to practice, and (3) the physician and therapists in charge judged that the practice was required during hospitalization. The participants’ background data, including the Stroke Impairment Assessment Set (SIAS) motor items [24,25] and the Functional Independence Measure (FIM) [26,27], are presented in Table 1. The SIAS motor items evaluate the upper limbs (two items) and lower limbs (three items) on the hemiparetic side, with six grades ranging from 0 (total paralysis) to 5 (normal) for each item [24,25]. The FIM score was evaluated by a nurse within one week before or after the practice. As shown in Table 1, the participants had relatively high scores on the FIM and SIAS motor items. A large percentage of participants (72%) could walk without walking aids or lower extremity orthosis. Nine participants (18%) were diagnosed with aphasia by their rehabilitation physicians, and four (8%) were diagnosed with spatial neglect.

Table 1. Participants’ characteristics (n = 50).

Sex, male/female, n	34/16
Age, years, mean (SD)	63.0 (14.0)
Type of stroke: hemorrhage/infarction/subarachnoid hemorrhage, n	27/21/2
Side of hemiparesis, right/left/bilateral/none, n	24/23/1/2
Days after stroke onset, mean (SD)	89.4 (37.4)
Days after admission to rehabilitation hospital, mean (SD)	59.5 (34.7)
Stroke Impairment Assessment Set: motor function, median (IQR)
Knee-mouth	4.5 (1)
Finger-function	4 (3)
Hip-flexion	5 (1)
Knee-extension	5 (1)
Foot-pat	4 (1)
Functional Independence Measure, median (IQR)
Motor score	86 (8)
Cognitive score	32 (5)
Total score	118 (11)
10-meter walk test, mean (SD)	10.0 (3.7)
Berg Balance Scale, median (IQR)	54 (6)^a
Kohs Block Design test, median (IQR)	94 (53)^b
Non-motor symptoms, n (%)
Aphasia	9 (18)
Spatial neglect	4 (8)
Mobility aid used, n (%)
None	36 (72)
Cane and ankle-foot orthosis	7 (14)
Cane	4 (8)
Ankle-foot orthosis	3 (6)

SD: standard deviation; IQR: interquartile range.

Missing in ^a4 and ^b2 cases. Stroke Impairment Assessment Set, each item is scored 0–5 points. Functional Independence Measure: motor score ranges 0–91 points, cognitive score ranges 0–35 points, and total score ranges 0–126 points.

Procedure

Participants practiced using public transportation while accompanied by a physical therapist (PT) and an occupational therapist (OT). The type of transportation used was chosen according to participant needs before practicing in the following three courses (Figure 2): course 1, use the train only; course 2, use the bus only; and course 3, use the train and the bus. Participants received information from the therapists before the day of the practice regarding the courses to be followed and the time limit for the practice. During the practice, the participants were instructed to go around the course using the train and/or bus. The therapist supported via assistance and/or encouragement, but only on request from the participants or if the therapists judged it necessary to accomplish the tasks safely. Before the practice, the accompanying PT and OT were instructed by the researchers to independently score the participant’s actual performance using the PTAF immediately after the practice. The therapists briefly recorded the participant’s performance during the exercise to ensure safety, such as preventing the patient from falling. The scores were filled in on an assessment sheet after completing the practice. For examination of validity, the 10-meter walk test, Berg Balance Scale (BBS), SIAS motor items, Kohs Block Design test, FIM motor items, and FIM cognitive items were assessed within one month of the practice.

Figure 2. Courses of training for public transportation use in the hospital.

The three courses that participants practiced using public transportation in this study.

Instruments

The 10-meter walk test is an assessment of walking ability, in which the rater measures the time required to walk at a comfortable speed on a 10-m straight-line walking path, with a 3-m runway at each end. We adopted the 10-meter walk test because we considered that the ability to walk, including walking speed, is necessary when moving outdoors, such as on crosswalks, passing through ticket gates, and in crowds. The reliability and validity of this measure have previously been confirmed in patients with stroke [28,29].

The BBS is an assessment of balance ability comprising 14 items related to activities of daily living, and each item is scored 0–4 points, with higher scores indicating better balance ability [30]. We adopted it since a high level of balance is necessary when moving outdoors and getting on/off trains and buses. The scale has been previously confirmed as reliable and valid in individuals post-stroke [31].

The SIAS motor items assess motor function, and a score of 0–25 was used as the total score for the five subscales [24,25]. We adopted it since motor function is required when using public transportation, such as mobility to walk outdoors and bilateral arm movements used to manipulate money. The SIAS has been previously verified for reliability and validity in individuals post-stroke [24,25,32].

The Kohs Block Design test is a test of intelligence in which participants replicate sample designs with colored blocks, and scoring is based on the difficulty of the task and the time to correct answers [33]. We adopted it since sufficient cognitive function is necessary to accomplish a series of tasks efficiently using the proper procedure. This test has been previously validated [33].

The FIM is a widely used assessment of activities of daily living in the rehabilitation field [26,27]. Although the use of public transportation is classified in the instrumental activities of daily living (IADLs) [34], the FIM was adopted as a comparison because IADLs are a superordinate concept of activities of daily living and have a strong relationship [35–37]. The reliability and validity of this measure have been previously confirmed in individuals post-stroke [38,39].

Analysis

Internal consistency, inter-rater reliability, and construct validity were examined based on PTAF scores assessed by the PT and OT. For internal consistency and construct validity, calculations were separately performed with PT and OT assessments, since there were two assessments for each patient. In the calculation of the whole assessment form, the mean scores of the PTAF only for the participants who chose course 3 (train and bus), wherein all items of the PTAF were assessed, were adopted because those who used only the train or only the bus underwent assessment of different tasks and different interpretations of the total score. The number of patients enrolled was determined to be 50, which is considered an adequate sample size in COSMIN [17].

To clarify the consistency of the PTAF in assessing a patient’s ability, the internal consistency for each category and the whole assessment form were examined using Cronbach’s alpha coefficients for PTs and OTs, respectively. In the calculation for each category, the category of “plan for going out” was excluded because more than one item was required to calculate the coefficients.

To clarify how well the assessments agreed between the two raters, the inter-rater reliability of each item was examined using Cohen’s weighted κ coefficients. Items rated “N: not applicable” by either the PT or OT were excluded from the calculation. The inter-rater reliability for the mean score of PTAF given by each assessor was also examined using the intraclass correlation coefficient (ICC) [1,2]. The reliability of the mean scores of all items in the PTAF was examined for all participants. The reliability of the mean scores for the three categories of “plan for going out,” “mobility,” and “using trains,” which are required when using trains, was examined for participants who used trains (i.e., all participants except those who practiced on course 2, “bus only”). The reliability of the mean scores for the three categories of “plan for going out,” “mobility,” and “using buses,” which are required when using buses, was examined for participants who used buses (i.e., all participants except those who practiced on course 1, “train only”).

To clarify whether the PTAF reflects the ability to use public transportation, construct validity (convergent validity) was verified by examining the correlation with other assessments considered relevant to the use of public transport, including the 10-meter walk test, BBS, SIAS motor items, Kohs Block Design test, FIM motor items, and FIM cognitive items. All correlational analyses between the mean scores of PTAF items, excluding “N: not applicable,” and each assessment were performed using Spearman’s rank correlation coefficients for PTs and OTs, respectively.

All statistical analyses were performed using R package version 3.3.2 (R Foundation for Statistical Computing, Vienna, Austria). All p values of <0.05 were considered statistically significant. Cohen’s weighted κ coefficients and ICC were interpreted as follows: slight, 0.00–0.20; fair, 0.21–0.40; moderate, 0.41–0.60; substantial, 0.61–0.80; almost perfect, 0.81–1.00 [40]. Spearman’s rank correlation coefficients were interpreted as follows: slight almost negligible relationships, 0.00–0.20; low correlation, 0.20–0.40; moderate correlation, 0.40–0.70; high correlation, marked relationship, 0.70–0.90; and very high correlation, very dependable relationship, 0.90–1.00 [41].

Results

A total of 50 patients participated in the study, with 33 PTs and 28 OTs accompanying the practices. The number of patients using the different practice courses was nine (18%) patients for course one (train only), one (2%) for course 2 (bus only), and 40 (80%) for course 3 (train and bus). Practices were conducted for 1–2 h between 9 am and 5 pm, with the most common time for practice being 3 pm to 5 pm (19 [38%] patients), followed by 1 pm to 3 pm (10 [20%] patients), 10 am to 12 am (8 [16%] patients), and 9 am to 11 am (4 [8%] patients). For the subtask of “operating ticket machines,” more patients were judged as “N” than for the other subtasks because they did not need to use the ticket machine due to using prepaid transportation cards.

For PTs and OTs, respectively, Cronbach’s coefficient alpha value for the overall PTAF was 0.84 and 0.88, and for each category were 0.64–0.82 and 0.70–0.84. The results indicated that overall, PTAF and its categories had at least substantial internal consistency (Table 2).

Table 2. Inter-rater reliability and internal consistency in the PTAF.

Category	Item	Cohen’s weighted κ coefficient	Cronbach’s coefficient alpha		Number of participants
			PT	OT
Plan for going out			–	–
	Creating and understanding plans	0.65			50
Mobility			0.64	0.84
	Movement over long time	0.62			50
	Walking in crowds	0.79			49
	Using outdoor stairs	0.61			50
	Using escalators	0.76			50
Using trains			0.82	0.83
	Operating ticket machines	0.83			31
	Passing through ticket gates	0.78			49
	Selecting departure time and platform	0.68			49
	Getting on and off trains	0.73			49
	Movement in trains	0.81			49
Using buses			0.74	0.70
	Selecting departure time and bus stop	0.73			41
	Getting on and off buses	0.73			41
	Movement in buses	0.73			41
	Paying fares	0.71			41

PT: physical therapist; OT: occupational therapist.

Cronbach’s coefficient alpha for the category “plan for going out” was not calculated because this category comprised only one item.

Cohen’s weighted κ coefficient value for each item ranged from 0.61 to 0.83, indicating that items in the PTAF had at least substantial inter-rater reliability (Table 2). The ICC of the mean score of all PTAF items was almost perfect, with an ICC of 0.90 (95% confidence interval [CI], 0.81–0.95), and the ICC of the mean score of the three categories required for using trains was substantial, with an ICC of 0.76 (95% CI, 0.59–0.86). The ICC of the mean score of the three categories for using buses was almost perfect, with an ICC of 0.88 (95% CI, 0.80–0.93).

All correlations between the mean scores of the PTAF and the scores of each assessment reflecting the ability to use public transportation were significant, and the absolute value of Spearman’s rank correlation coefficients was 0.38–0.59 (p < 0.05), indicating moderate correlation (Table 3).

Table 3. Construct validity (convergent validity) of the PTAF (N = 40).

	Physical therapists		Occupational therapists
	Spearman’s rank correlation coefficient	p Value	Spearman’s rank correlation coefficient	p Value
10-meter walk test	–0.52	0.001	–0.51	0.001
BBS	0.59	<0.001	0.53	0.001
FIM motor domain	0.57	<0.001	0.45	0.003
FIM cognitive domain	0.50	0.001	0.40	0.010
SIAS motor items	0.51	0.001	0.53	<0.001
Kohs Block Design test	0.38	0.019	0.40	0.013

BBS: Berg Balance Scale; FIM: Functional Independence Measure; SIAS: Stroke Impairment Assessment Set.

Discussion

The PTAF, a new assessment tool for public transportation use, showed sufficient internal validity, intra-examiner reliability, and construct validity. Cronbach’s alpha coefficients were 0.84 and 0.88 for the overall PTAF and 0.64–0.84 for each category, showing at least substantial internal consistency. The values were comparable with Cronbach’s alpha coefficients in other commonly used IADL assessment tools, which were 0.78–0.87 for the overall assessment tools [36,42] and 0.73–0.97 for their subscales [35,43]. These results indicate that the mean score of all PTAF items is sufficiently valid for assessing the use of public transportation, and the mean of the categories “using trains” and “using buses” can also be used separately, e.g., it can be used for individuals post-stroke who use only buses or trains.

Regarding inter-rater reliability, κ coefficients of each item ranged from 0.61 to 0.83, and the ICC values of the mean of PTAF items were 0.76–0.90. In several previous studies examining the inter-rater reliability of the Frenchay Activities Index (FAI), a common IADL assessment tool used in individuals post-stroke, κ coefficients of items were 0.21–0.80 [44], 0.41–0.90 [45], and 0.71–1.00 [46]. Variations were seen among these studies; some items have low-reliability coefficients. In the present study, the coefficient for the item with the lowest value was 0.61, indicating substantial reliability. This result indicates that it is possible to interpret the assessment of all items with reliability, which is a strength of the PTAF. In addition, the ICC of PTAF was comparable with that of the total score of FAI, which was reported to be 0.90 [45], indicating that the PTAF shows sufficient reliability to interpret the mean score of all items. In this study, several therapists in the hospital (33 PT and 28 OT) with no specific training in the use of assessment tools assessed patients’ performance; hence, the PTAF enables objective assessment and is suitable for use in clinical settings. Since sufficient inter-rater reliability was confirmed, we believe that the PTAF can assess patient performance regardless of the rater and is useful for understanding patient status and provides information for the medical professionals and family members.

The construct validity results (convergent validity) showed significant fair to moderate correlations between all of the assessment scores. A previous study reported that those whose frequency of public transportation use decreased after stroke onset had significantly deteriorated upper and lower limb function and decreased balance ability than those whose frequency did not change [3]. In another study, the subjective experience of individuals post-stroke reported a high physical load when getting on/off the bus due to higher steps and greater distance [6]. Considering the physical functions necessary for the use of public transportation, it was reasonable that there were moderate correlations between the PTAF and 10-meter walk test, BBS, FIM motor score, and SIAS motor items in the present study. The present study also found a correlation between the PTAF and the cognitive assessments, FIM cognitive score, and Kohs Block Design test. A previous qualitative study reported that the use of public transportation includes serial and complex tasks and time pressure that make it difficult for individuals post-stroke to use it independently [9], indicating that there is a relationship between public transport use and cognitive function. Therefore, the relationship we found is reasonable, and we believe it shows the validity of the PTAF.

The present study showed that the mean score of the whole PTAF has significant reliability and validity, indicating that the PTAF can measure the degree of independence of individuals post-stroke when using public transportation. The PTAF can also be used to judge whether the patient is independent in using public transport. In addition, each item of the PTAF could be assessed with reliability. In clinical practice, by identifying the items that are not independent, we can clarify the tasks to be targeted for intervention, which will contribute to the planning of rehabilitation programs.

Limitations

This study has several limitations. Although reliability and validity were confirmed in the present study, they did not meet the COSMIN requirements at a high level. Furthermore, the study did not cover all items listed in the COSMIN. For example, we did not examine the intra-rater reliability considering participant burden and could not refer to the intra-rater reliability or calculate the standard error of the mean and statistically driven minimal clinically important difference of the total score. Furthermore, we did not examine the concurrent validity because there was no tool to assess actual performance regarding individuals using public transportation post-stroke. In addition, this study investigated the relationship between the PTAF and BBS and between the PTAF and FIM but did not compare them with tools for assessing IADL or clarify whether they reflect the ability to perform tasks with many processes and variations that are characteristic of IADLs. Furthermore, the study was conducted in an area where the public transportation system was well developed, and the sample was limited to subacute inpatients with relatively fewer impairments after stroke. In particular, more than two-thirds of the participants in this study were able to ambulate independently. Therefore, generalization of the results of this study should be performed with caution, and a further reliability and validation study is needed in different regions and countries, including individuals post-stroke at different phases and severity of impairments.

In addition, the number of courses used in the present study were limited to only three. This may have contributed to increasing the reliability and validity of the present study. Furthermore, it is unclear whether those judged to be independent on the PTAF can use public transportation independently in the area where they would live after discharge from the hospital.

Future directions

Future studies should include a larger sample size and investigate the intra-rater reliability and responsiveness of the PTAF by conducting multiple assessments on the same patients to demonstrate its usefulness in determining the effectiveness of interventions and measuring changes over time. Regarding the validity of the PTAF, the future study would also examine the concurrent or construct validity of the PTAF against other tools, such as one capable of assessing the ability of using transport more broader sense, namely using transport as a driver or passenger [47] and a simulation-based tools for patients with developmental cognitive disability [48] and autistic spectrum disorder [49], as well as other IADL scales that assess the performance of complex tasks. In addition, to make the PTAF more usable, it may be necessary to examine its reliability and validity in real situations, i.e., on the actual public transportation routes that patients use after discharge from a hospital. Furthermore, to clarify whether the use of public transportation after discharge from the hospital can be predicted, a further study with post-discharge follow-up will ascertain the predictive validity and draw the cut-off values for the prediction.

Conclusions

The PTAF showed sufficient internal consistency, inter-rater reliability, and construct validity. Although further investigation of reliability and validity is warranted, the PTAF will contribute to the planning of intervention for individuals post-stroke who can ambulate and offer an objective measure for the degree of independence of public transportation use.

Disclosure statement

The authors report no conflicts of interest.