The role of theory to develop and evaluate a toolkit to increase clinical measurement and interpretation of walking speed and distance in adults post-stroke

Abstract

Purpose

The iWalk study showed that 10-meter walk test (10mWT) and 6-minute walk test (6MWT) administration post-stroke increased among physical therapists (PTs) following introduction of a toolkit comprising an educational guide, mobile app, and video. We describe the use of theory guiding toolkit development and a process evaluation.

Materials and methods

We used the knowledge-to-action framework to identify research steps; and a guideline implementability framework, self-efficacy theory, and the transtheoretical model to design and evaluate the toolkit and implementation process (three learning sessions). In a before-and-after study, 37 of the 49 participating PTs completed online questionnaires to evaluate engagement with learning sessions, and rate self-efficacy to perform recommended practices pre- and post-intervention. Thirty-three PTs and 7 professional leaders participated in post-intervention focus groups and interviews, respectively.

Results

All sites conducted learning sessions; attendance was 50-78%. Self-efficacy ratings for recommended practices increased and were significant for the 10mWT (p ≤ 0.004). Qualitative findings highlighted that theory-based toolkit features and implementation strategies likely facilitated engagement with toolkit components, contributing to observed improvements in PTs’ knowledge, attitudes, skill, self-efficacy, and clinical practice.

Conclusions

The approach may help to inform toolkit development to advance other rehabilitation practices of similar complexity.

Implications for Rehabilitation

• Toolkits are an emerging knowledge translation intervention used to support widespread implementation of clinical practice guideline recommendations.

• Although experts recommend using theory to inform the development of knowledge translation interventions, there is little guidance on a suitable approach.

• This study describes an approach to using theories, models and frameworks to design a toolkit and implementation strategy, and a process evaluation of toolkit implementation.

• Theory-based features of the toolkit and implementation strategy may have facilitated toolkit implementation and practice change to increase clinical measurement and interpretation of walking speed and distance in adults post-stroke.

Keywords:

• Toolkit
• stroke rehabilitation
• walking
• standardized assessment
• process evaluation
• theory
• knowledge translation

Introduction

Stroke remains a major health concern and the second highest cause of disability worldwide [1–3]. Stroke rehabilitation guidelines are widely available [4–7] and implementation of recommended practices is expected to improve the quality of care and patient outcomes [8,9]. Existence of recommendations, however, does not ensure consistent application in clinical practice [10–14]. In fact, guidelines have been criticized for lacking a comprehensible structure, utility, and local applicability [15,16]. The field of knowledge translation [17] has emerged to narrow the gap between what is known from research and knowledge synthesis and application of this knowledge by decision-makers to improve healthcare delivery and outcomes. Knowledge translation interventions designed to facilitate guideline implementation in stroke rehabilitation to date have been costly, requiring funds to support and train local clinical facilitators or clinicians to enable practice change [14,18–20]. Interventions have also targeted the implementation of multiple recommended practices simultaneously [14,19,21], making it challenging for stroke teams to prioritize practices [22].

Toolkits, defined as “a packaged grouping of multiple knowledge translation tools and strategies that codify explicit knowledge,” [23] may be an alternative low cost knowledge translation intervention, aimed at supporting widespread implementation of a set of inter-related clinical practice guideline recommendations. According to a systematic review of 39 toolkit evaluations [23], toolkits may include a variety of components, such as educational materials; templates (e.g., checklists, preprinted letters); visual reminders (e.g., posters); pocket cards or electronic tools (e.g., calculator on personal digital assistant), to educate and facilitate behaviour change. To improve the reporting of toolkit evaluations, review authors recommended clearly describing the purpose of and rationale for each toolkit component; the evidence supporting the effectiveness of each toolkit component; and guidance on the implementation process [23]. None of the studies included in this review described toolkits targeting rehabilitation professionals or healthcare services for people post-stroke [23]. Nonetheless, individual tools [20] or toolkits [14,21] have been previously included in multi-component knowledge translation interventions targeting the implementation of best practices in stroke rehabilitation.

Toolkits are considered to be complex interventions, as they may target a large number of components, behaviours, groups, organizational levels and outcomes, or they may require significant tailoring during implementation [24]. Implementation science experts recommend using theories, models and frameworks to design, tailor, and implement complex interventions, as this enables us to better understand how each intervention component is expected to help achieve the desired outcomes [24–26]. Numerous theories, models, and frameworks exist [27], however, making their choice and application challenging. Nilsen [28] classified theories, models and frameworks used in implementation science as either: (1) process models, which aim to describe or guide the process of translating research into practice; (2) determinant frameworks, which specify types of determinants that influence implementation outcomes; (3) classic theories, which identify variables and relationships between variables to explain causal processes underlying human behaviour; (4) implementation theories, which explain aspects of implementation; or (5) evaluation frameworks that guide plans to evaluate implementation. While authors commonly cite theories, models and frameworks used to develop knowledge translation interventions [27], there are few examples [29] of how to use them to guide knowledge translation intervention design, methods of evaluation, and interpretation of results in the rehabilitation context.

Recently, we conducted a single-group before-and-after study to evaluate the impact of implementing a novel toolkit (“iWalk”) for stroke rehabilitation [30]. The details of this “impact evaluation” will be available in an upcoming publication (manuscript in preparation). Briefly, the objective of the iWalk toolkit is to promote an evidence-informed approach to using the 10-meter walk test (10mWT) and the 6-minute walk test (6MWT) post-stroke among physical therapists (PTs). Following iWalk toolkit implementation, PTs were significantly more likely to administer the two walk tests post-stroke based on documentation in health records. The highest increase was observed in outpatient rehabilitation settings, followed by inpatient rehabilitation and acute care settings. Compared to the 6MWT, implementation of the 10mWT improved to a greater extent.

In this paper, we describe how we used a process model, a determinant framework, and two classic theories to guide the design and process evaluation of the implementation of the iWalk toolkit. These findings will help us to understand the results of the impact evaluation and advance the science of developing toolkits as knowledge translation interventions.

Methods

We used the knowledge-to-action (KTA) framework [31] (Figure 1) to guide the overall research process. The KTA framework is widely used [32] and promotes ongoing collaboration between researchers and decision-makers (or “end-users” e.g., patients, clinicians, managers, policy-makers) in the research process to optimize relevance and applicability of findings [33]. While the KTA framework provides broad guidance, we used self-efficacy theory [34], a guideline implementability framework [35], and the transtheoretical model [36] to better understand determinants of human behaviour change. The role of these theories, models, and frameworks within the overall KTA paradigm is illustrated in Figure 2. Below, we (i) describe these individual theories, models, and framework in greater detail; and (ii) outline each of the KTA steps. The specific process evaluation of the iWalk toolkit implementation aligns with Step 7 (“Implement intervention & monitor knowledge use”) and is the focal point of the present study.

Figure 1. Knowledge-to-action framework (taken from Graham et al. [31] with copyright permission).

Figure 2. Use of the knowledge-to-action framework, self-efficacy theory, a guideline implementability framework, and the transtheoretical model to guide the development and process evaluation of the toolkit and implementation strategy.

Overview of theories, models, and frameworks

Knowledge-to-action framework

The KTA framework [31] is a meta-framework based on the synthesis of planned action theories. As a process model, it outlines a series of steps for the creation and application of knowledge. As shown in Figure 1, the knowledge creation phase is represented by a funnel to highlight the importance of filtering and synthesizing knowledge from individual studies to develop knowledge products and tools, such as guidelines, designed to guide clinical decision-making. The action cycle of the KTA framework outlines a series of steps that can be taken in variable order to apply the knowledge products of the knowledge creation phase. The knowledge creation funnel can spin to inform any step in the action cycle.

Self-efficacy theory

Self-efficacy theory [34] is a classic theory that can assist with the interpretation, modification and prediction of behaviour. Self-efficacy theory highlights the importance of building positive self-efficacy beliefs and outcome expectations to facilitate behaviour change. As shown in Figure 2, self-efficacy, defined as the level of belief in one’s ability to organize and execute given types of performances, is expected to predict clinician behaviour. Performance accomplishments, vicarious experience, verbal persuasion, and emotional arousal are four mechanisms for increasing self-efficacy. Outcome expectations, defined as judgments of the likely consequences a behaviour will produce, can provide incentive or disincentive to perform a desired behaviour.

Guideline implementability framework

The guideline implementability framework [35] is a determinant framework that identifies 27 guideline design elements, grouped into nine domains shown in Figure 2, expected to improve guideline use.

Transtheoretical model

The transtheoretical model [36] is a classic theory developed in the field of psychology. It posits that people progress through five stages when attempting to change behaviour either on their own or with intervention: (1) precontemplation – not intending to take action within the next 6 months; (2) contemplation – intending to take action within the next 6 months; (3) preparation – intending to take action in the next 30 days; (4) action – made overt changes less than 6 months ago; and (5) maintenance – made overt changes more than 6 months ago.

Use of theory to design the toolkit, implementation strategy, and process evaluation

The iWalk toolkit was developed and evaluated through an eight-step process based on the knowledge creation funnel and action cycle of the KTA framework, with integration of self-efficacy theory, the guideline implementability framework, and transtheoretical model (Figure 2). Preliminary descriptions of steps 1 and 2 have been reported [37]. We applied an integrated knowledge translation approach [33] throughout by collaborating with end-users, including physical therapy clinicians, professional leaders (PLs), educators, medical directors, and regional stroke system representatives.

Step 1: Identify problem & Assess barriers to knowledge use

Despite recommendations in stroke rehabilitation guidelines to use standardized assessment tools [4–7], our survey [13] showed that PTs do not consistently use valid measures to evaluate or monitor change in walking, formulate a prognosis, or judge readiness for discharge. Questionnaire items were designed to assess factors, including self-efficacy and outcome expectations, influencing knowledge use. Results from our survey and other research identified barriers, including insufficient knowledge of valid measures and normative values [13,38]; limited time and self-efficacy to access and appraise the literature to select optimal measures [39–41], administer and score tests [13,42]; and low outcome expectations due to scepticism regarding the ability of tests to capture change and relate to the home or community environment [13].

Step 2: Knowledge creation

To address these problems, we conducted four systematic reviews [43–46] to synthesize evidence of measurement properties, reference values, and protocols, for standardized measures of walking speed and distance post-stroke. We focused the reviews on measures of walking speed and distance, as they were recommended in emerging guidelines and had demonstrated sensitivity to change [47,48].

Step 3: Identify, review, select knowledge

We used systematic review findings to develop an evidence-informed approach to using the 10mWT and 6MWT for the purposes of evaluation, monitoring change, patient education, and goal setting. These tests were recommended and had optimal quality and interpretability [43–46]. This approach was designed to support three Canadian Stroke Best Practice Recommendations to use valid assessment tools, provide patient, family and caregiver education, and involve patients and families in their management, goal setting and transition planning [49].

Selection of walk test protocols and reference values

Because walk test protocol features (e.g., distances [50,51]; walking aid used [52]; encouragement [53]) can influence performance, a single protocol for each walk test was developed to enable comparison across care settings. We used criteria (see Supplementary Material 1) to select and adapt protocols that would optimize measurement accuracy, applicability to people undergoing stroke rehabilitation, and implementation feasibility for hospital settings in terms of cost, space, and time. For example, tests permitting use of minimal physical assistance were selected to broaden clinical utility for patients post-stroke. A supplemental file contains the final 10mWT and 6MWT protocols (Supplementary Material 2). From the systematic review findings [43–46], we used criteria established a priori (Supplementary Material 3) to select published reference values for age- and sex-specific norms [54,55]; maximum mean distances to walk at community locations [45]; and crosswalk speed of 1.2 m/s based on municipal [56] and federal [57] documents; and derived minimal detectable change values at the 90% confidence level (MDC₉₀) [58] from a study evaluating similar walk test protocols [59].

Step 4: Assess barriers and facilitators to knowledge use

We conducted two qualitative studies with PTs to identify barriers and facilitators to knowledge use, using self-efficacy theory to inform interpretation of results [15,60,61]. Findings highlighted difficulty administering tests in patients with moderate-to-severe motor deficits and aphasia as a barrier, and developing skill and self-efficacy to administer walk tests, and a smartphone application as potential enablers.

Step 5: Develop intervention and implementation strategy & Adapt knowledge to local context

The iWalk toolkit, consisting of an educational guide, mobile app, and educational video [62], was developed to codify the evidence-informed approach created in Step 3. The purpose of the toolkit was to help PTs administer stroke-specific protocols for the 10mWT and the 6MWT, interpret test performance, educate patients about test performance, set goals for each test, and select treatments with the potential to improve walking speed and distance in acute care, and inpatient and outpatient stroke rehabilitation. In alignment with the KTA process [31], toolkit components were designed to address barriers and incorporate facilitators to walk test use in clinical practice. Table 1 outlines the purpose and rationale for each toolkit component.

Table 1. Purpose and rationale for iWalk toolkit components to address barriers to walk test use.

Toolkit component	Purpose	Rationale and supporting literature
Guide, app, and video	↑Knowledge of how to set up, administer, score, and interpret walk tests for patient education and goal setting	Guide and app provide written stroke-specific walk test protocols and diagrams for walkway set-up Guide provides a glossary of terms, instructions for scoring/interpretation App References section describes reference values Video shows a physical therapist administering and scoring tests with a person post-stroke [38,60,64]
Learning sessions (implementation strategy)	↑Skill and self-efficacy to administer, score, and interpret walk tests for patient education and goal setting	Guide includes agendas for three group learning sessions to review guide, video, and app; practice walk tests, and practice interpreting performance, educating patients, and setting goals using case scenarios Successful practice in a group setting expected to ↑skill and self-efficacy to perform tasks via performance accomplishment, vicarious experience, emotional arousal [34,61,64]
Guide, app, and video	↑Belief in clinical relevance of walk tests	Guide has top 10 reasons to use tests, evidence of reliability, validity, and interpretability, related guideline recommendations, and “Did you know” text boxes highlighting supporting research evidence, that facilitate use of assessment tools App quickly provides information tailored to patient linked to capacity for community ambulation Walk test protocols allow assistance and include pictorial instructions for people with aphasia in alignment with guideline recommendations to maximize applicability to stroke population [61,78,79]
App and lanyard	↓Time to administer, score and interpret tests for patient education and goal setting	Tool for screening 6MWT contraindications Audio-recorded and written walk test instructions Integrated 10mWT stopwatch and 6MWT timer and length counter to automate scoring Vibration prompt and display of 6MWT encouragement phrase given each minute Automated algorithms to score and interpret test performance for patient education and goal setting Lanyard would enable handsfree test administration for patient safety [60,82,83]

6MWT, 6-min walk test; 10mWT, 10-meter walk test.

Based on the guideline implementability framework [35], we incorporated design elements into the toolkit to increase its clinical adoption, as outlined in Table 2. For example, to address the applicability domain in the framework, we used case scenarios, a recommended adult learning strategy desired by PTs [60,63], to illustrate how to apply recommended practices to individual patients. Finally, we proposed small group learning sessions with homework activities as a strategy to implement the toolkit. Review of the guide and practice of the clinical activities targeted in the guide was expected to increase knowledge and skill, and self-efficacy to administer tests, and interpret and document test performance and positive outcome expectations about the clinical relevance of the walk tests [34,61,64]. Self-efficacy was expected to increase through the mechanisms of performance accomplishments, emotional arousal, and vicarious experience [34].

Table 2. Reviewer ratings of guideline implementability features of the iWalk toolkit.

Implementability domain	How toolkit incorporates implementability domain	Implementability design element	Element	Domain^a
			Reviewer ratings (mean score)
Adaptability: Guide is available in a variety of versions for different users or purposes	Guide in print and pdf version, content partially in app and video Users include physical therapists and others who assess walking	Sources	4.6	4.4
		Versions	4.8
		Users	4.0
Usability: Content is organized to enhance ease with which the guideline can be used	Table of contents Narrative and tabulated evidence iWalk recommendations tabulated	Navigation	4.6	4.7
		Evidence	4.8
		Recommendations	4.6
Validity: Evidence is summarized and presented such that its quantity and quality are apparent	CSBPR provided with levels of evidence Evidence levels were defined 3 iWalk recommendations provided	# references	4.3	4.5
		Evidence graded	4.6
		# recommendations	4.6
Applicability: Information is provided by which to interpret and apply the guidelines for individual patients	Guide describes walk test indications, case scenarios, clinical tips/notes, interpretation based on age/sex, walking speed	Clinical considerations	4.8	4.8
Communicability: Resources for providers or patients to inform, educate, support, and involve patients	Guide describes approach to patient education and goal setting with example statements and form for patients	Inform, educate, support	4.9	4.9
		Decision making	4.9
Relevance: The focus or purpose of the guideline is explicitly stated	Guide describes purpose and users of guide, how guide addresses needs of users and patients targeted	Objectives	4.9	4.9
		Stakeholders	4.9
		Needs	4.8
Resource implications: Anticipated changes, resources, and competencies required to adapt and accommodate guideline utilization are identified	Guide describes equipment and related costs, evaluator qualifications, clinical processes for administering tests, interpreting performance, patient education and goal setting, selecting treatments and evaluating practice	Technical	4.6	4.8
		Regulatory	N/A
		Human Resources	4.8
		Professional	4.9
		Workflow	4.8
		Costs	4.8
Implementation: Processes for planning and applying local strategies to promote guideline utilization are described	Guide describes known implementation barriers and strategies to overcome, equipment and space requirements, walkway set up, readiness checklist, agendas for 3 learning sessions	Identify barriers	4.9	4.9
		Tailor guidelines	4.9
		Integrated tools	4.9
		Promote utilization	4.9
Evaluation: Processes for evaluating guideline implementation and utilization are described	Guide describes audit and feedback process and 4^th meeting to reflect on practice change	Implementation	4.7	4.7
		Utilization	4.7

CSBPR: Canadian Stroke Best Practice Recommendations.

^a Mean domain scores represent the mean of scores on domain elements.

iWalk guide

The printed guide consisted of eight modules. Module 1 outlined the “Top 10” reasons to use the walk tests post-stroke; iWalk practice recommendations, and related Canadian Stroke Best Practice Recommendations. Modules 2–5, respectively, provided guidance on test administration, interpretation, patient education and goal setting, and treatment selection. These modules included clinical tips and “Did you know?” text boxes to highlight relevant research supporting each walk test. Module 6 described a process for using audit and feedback [65–67] to advance practice change and strategies for overcoming challenges PTs might encounter. Module 7 presented case scenarios and learning activities to help integrate the information in the guide. Module 8 provided information to guide three 1-h small group learning sessions scheduled at 2-week intervals to implement the toolkit, including instructions and agendas, equipment and space checklists, printable test protocols (including aphasia-friendly procedures), data collection and goal setting forms, a quick reference value guide, a patient education tool, and breakdown of equipment and training costs to implement the walk tests. We recommended identifying someone to coordinate and facilitate learning sessions. PTs were asked to review relevant modules prior to each session. Session 1 involved reviewing Module 1, discussing walk tests, viewing the educational video, and practicing the tests. Session 2 involved entering data into the app to interpret test performance based on case scenarios, and documenting results. Session 3 involved role-playing to practice goal setting based on a case scenario, and discussing feasibility of using audit and feedback to monitor implementation. Each session, PTs discussed implementation and were asked to practice what they learned (i.e., test administration, documentation, interpretation, patient education and goal setting) with a patient prior to the next session.

App and video

The iWalkAssess app [62] was developed for iOS and Android platforms. The purpose of the app was to (i) increase awareness of stroke-specific 10mWT/6MWT protocols and reference values; and facilitate (ii) administration and scoring of 10mWT and 6MWT post-stroke; (iii) interpretation of test performance at point of care using normative, community ambulation, and MDC values; (iv) goal setting; and (v) use of the 6MWT to evaluate exercise intensity. The app developer followed user-centred design principles [68] to develop a prototype based on a template proposed by author NMS, draft content, and a “think aloud” session during which a PT verbally described her thought process during walk test administration, patient education and goal setting with a simulated patient. Authors NMS (researcher and PT) and SV (PT in clinical practice) provided feedback on app prototypes. Once walk test protocols and reference values were finalized, we developed scripts and engaged a PT and a patient to produce an 10-min educational video consisting of a PT administering walk tests with a person post-stroke [69]. Individuals who were videorecorded signed a photo/video release form.

Step 6: Tailor intervention

We conducted a series of studies to evaluate the sensibility of the guide among experts and PTs; and the clinical feasibility of recommended practices and usability of the app among PTs. Ten researchers from Canada (n = 4), the USA (n = 3), Australia (n = 2) and Sweden (n = 1), with expertise in measurement (n = 9), stroke (n = 7), and/or gait assessment (n = 7), provided written feedback on content validity of the guide (accuracy and comprehensiveness), rating the above-mentioned 27 design elements as 1-Inadequate, 1-poor, 2-fair, 3-average, 4-good, or 5-excellent. Mean ratings across design elements and domains ranged from 4.0 to 4.9 (Table 2). Clinical reviewers (i.e., end-users) of the iWalk guide consisted of 18 PTs with an average of 14 years of clinical experience in acute (n = 7), inpatient (n = 7), outpatient (n = 6), and homecare (n = 3) stroke rehabilitation practice settings. Reviewers were asked to identify information that should be added or removed, and rate the format, organization, readability, content validity of the guide, and the clinical feasibility of implementing the iWalk recommendations across clinical settings as 1-Inadequate, 1-poor, 2-fair, 3-average, 4-good, or 5-excellent. Reviewers received a $50 gift card in acknowledgment. Mean ratings of format, organization, readability, and content validity were 4.5, 4.6, 4.7, and 4.8, respectively. Mean ratings of the feasibility of implementing the walk tests in acute care, inpatient rehabilitation, outpatient rehabilitation, and in homecare were 3.0, 4.1, 4.5, and 2.8, respectively.

In beta testing of the app, three PTs [70] in acute care, inpatient and outpatient rehabilitation used the app to administer walk tests with colleagues or patients and provided feedback on clinical utility, usability, content, function, and format in a 30-min interview. Feedback guided final improvements to the app.

Step 7: Implement intervention & Monitor knowledge use

As noted previously, the present paper focuses on a process evaluation of the iWalk toolkit implementation, which aligns with steps in the KTA action cycle. We used the theories, models, and frameworks discussed earlier to develop the objectives, hypotheses, and methods of the process evaluation described here to help interpret main study findings. We expected that perceived toolkit quality together with implementation fidelity and engagement with the intervention, would increase self-efficacy and readiness to measure outcomes in the causal pathway towards performance of recommended practices. Process evaluation objectives were to: (1) describe implementation fidelity and engagement of PTs and PLs with the toolkit; (2) describe toolkit quality, specifically toolkit implementability, and app quality; and (3) evaluate effects of providing the toolkit on PTs’ readiness to measure outcomes, self-efficacy to perform recommended walk test practices, and app use for learning and practice (i.e., intermediate study outcomes).

Overview of process evaluation

The process evaluation of the single-group before-and-after impact evaluation study incorporated quantitative and qualitative methodologies. Table 3 outlines the process evaluation framework. An online questionnaire was administered to participants in clinical practice pre-intervention (August 2015) and five months later (February 2016). Site-specific focus groups with PTs, and interviews with PLs were held after completion of learning sessions. Figure 3 provides an overview of data collection for the primary analysis and process evaluation. The health sciences research ethics board (REB) at the University of Toronto and REBs of each participating healthcare organization approved the study protocol. The TIDIeR checklist [71] was used to guide reporting.

Figure 3. Overview of data collection for the primary analysis (impact evaluation) and process evaluation.

Table 3. Process evaluation framework: constructs, variables, and data collection approach.

Construct	Variable	Response Scale	Data Collection
			Method	Timing
Implementation fidelity	Implementation of learning sessions 1, 2, and 3	Yes/no	Email communication with PLs/PPLs	0–5 months
		N/A	Interviews with PLs/PPLs	Post-learning sessions
	Time to complete learning sessions	# months	Interviews with PLs/PPLs	Post-learning sessions
Engagement with intervention	Attendance at learning session 1, 2, and 3	Yes/no	E-survey of PTs/PPLs	5 months
	Review of guide modules	Yes/no
	Review of video	Yes/no
	Practice of each walk test	Yes/no
	Completion of case-based activities	Yes/no
	Duration of app use	# months
Toolkit quality	Toolkit implementability: Adaptability Usability Validity Applicability Communicability Relevance Resource implications Implementation Evaluation	N/A	Interviews with PLs/PPLs Focus groups with PTs	Post-learning sessions
	App quality using Mobile App Rating Scale Engagement Functionality Aesthetics Information quality	1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent	E-survey of PTs/PPLs	5 months
	Likelihood of app to improve user: Knowledge Attitudes Intentions to change Behaviour change	1-Strongly disagree, 2-Disagree, 3-Neutral, 4-Agree, 5-Strongly agree	E-survey of PTs/PPLs	5 months
	Recommendations for toolkit improvement	N/A	Interviews with PLs/PPLs Focus groups with PTs	Post-learning sessions
Impact of toolkit on PTs/PPLs’: -readiness to measure outcomes; -self-efficacy to perform walk test practices; -use of app for learning and practice	Readiness to measure outcomes using Clinician Readiness for Measuring Outcomes Scale Pre-contemplation Contemplation Preparation Action Maintenance	% in each stage	E-survey of PTs/PPLs	0 months 5 months
	Self-efficacy to: Identify appropriate patients for administering the test Safely administer the test Interpret test performance Educate patients about test performance Set patient-centred goals to improve performance	11-point scale (0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100% confidence)	E-survey of PTs/PPLs	0 months 5 months
	Use of app to: Increase provider knowledge of walk tests Screen for 6MWT Administer a test Interpret test performance Share information with the patient Share information with healthcare team	None, a little, some, most, or all of the time	E-survey of PTs/PPLs	5 months

PL: professional leader; PPL: professional practice leader; PT: physical therapist; N/A: not applicable; 6MWT: 6-minute walk test.

Eligibility and recruitment

Hospitals providing acute care, inpatient or outpatient rehabilitation services for people post-stroke were eligible. PTs registered with the provincial regulatory body and providing walking rehabilitation to ≥10 patients post-stroke per year could participate. Professional leaders and professional practice leaders (PPLs), responsible for advancing evidence-based practice and ensuring attainment of professional practice standards, were invited to participate. PPLs, but not PLs, carried a caseload. We recruited from six general and three rehabilitation hospitals in Ontario (n = 7) and Nova Scotia (n = 2) to understand toolkit effects across the care continuum and to optimize external validity. Eligible individuals (PTs, PLs, and PPLs) expressing interest in participating provided written informed consent.

Intervention

The main intervention of this study is the iWalk toolkit. The intervention content was developed as part of Step 5 (“Develop intervention and implementation strategy & Adapt knowledge to local context”) of the KTA cycle and was thus described previously. In September 2015, each site was mailed printed copies of the iWalk guide; Android smartphones for 22 PTs who did not own a smartphone or were unwilling to use their personal smartphone; and lanyards for smartphones to enable safe, hands-free administration of the walk tests. An electronic copy of the iWalk guide and a private YouTube^TM link to the iWalk video was emailed participants. In case of questions, participants could email or telephone a PT expert with 24 years of experience treating people post-stroke. Communication with the clinical expert was tracked. Sites were asked to complete learning sessions within a 5-month intervention period.

Quantitative data collection

An online questionnaire developed using Fluid Survey v3.0 was administered pre- and post-intervention. Table 3 outlines the variables, response scales, and timepoints for data collection. PTs’ engagement with the toolkit was evaluated by asking about completion of learning activities, attendance at learning sessions, and duration of app use. To evaluate app quality, we included the Mobile App Rating Scale [72] (MARS) in the post-intervention questionnaire. MARS is a 23-item self-report questionnaire with four subscales: engagement, functionality, aesthetics, and information quality. In each subscale, items are rated on a 5-point scale (1-inadequate, 2-poor, 3-acceptable, 4-good, 5-excellent), and ratings are averaged to produce a subscale score. Evidence of validity and excellent inter-rater reliability of total and subscale scores among raters with expertise related to the topic of the app has been reported [72]. Supplementary MARS items designed to evaluate the likelihood of the app to improve user knowledge, attitudes, intentions to change, and behaviour change using a 5-point Likert scale were included to further evaluate app quality. To evaluate readiness to measure outcomes, the 26-item Clinician Readiness for Measuring Outcomes Scale [73] (CReMOS) was included in the pre- and post-intervention questionnaire. CReMOS scores are used to classify individuals in one of the five categories of readiness to change (i.e., pre-contemplation, contemplation, preparation, action, and maintenance). Evidence of validity and excellent test–retest reliability of the CReMOS among rehabilitation professionals, including PTs, has been reported [73]. To evaluate self-efficacy to perform recommended walk test practices, we developed 6 questionnaire items per test (see Table 3) based on guidelines for developing self-efficacy scales [74] and included them in the pre- and post-intervention questionnaire. For each item, participants were asked to indicate their level of confidence to perform the described activity on an 11-point scale that ranged from 0% (no confidence) to 100% (completely confident). To evaluate the nature of app use for learning and practice, we developed questionnaire items to determine the frequency of app use to increase personal knowledge, administer and interpret each test, and share results (Table 3). Data on sociodemographic and practice characteristics (age, sex, highest degree attained, years of clinical experience, number of patients post-stroke on caseload, setting, preferred methods for evaluating walking, and entry-to-practice education on use of the walk tests) were collected pre-intervention using pre-tested items [39]. Questionnaire items developed by the research team were piloted with two PTs with expertise in neurological rehabilitation and revised based on their feedback. Supplementary Material 4 contains all questionnaire items.

Qualitative data collection

At each site, focus groups were conducted with PTs using a semi-structured guide developed to explore their experiences using the iWalk toolkit, participating in learning sessions, and applying the evidence-informed approach to using the walk tests in clinical practice, and suggestions for toolkit improvement. In Ontario, focus groups were held onsite, whereas in Nova Scotia, focus groups were conducted by telephone with an onsite facilitator. PLs and PPLs were interviewed by phone using semi-structured interview guides developed to explore their experiences with organizing walkways and learning sessions, the features of the toolkit that facilitated these activities, and recommendations for improvement. Additionally, PPLs were asked about their experiences using the walk tests in clinical practice.

Focus groups and interviews were scheduled for one hour. The interviewer held a Master’s degree and had previous experience conducting qualitative research. The interviewer was not a PT and was not familiar to study participants. This may have enabled study participants to speak openly about their experiences without fearing judgment and describe clinical scenarios in detail when the interviewer asked for clarification. Ideas raised by participants in one focus group or interview were raised in subsequent sessions. After each session, the interviewer wrote reflective notes describing key messages, and relating them to comments from previous sessions. Focus groups and interviews were audio-recorded and professionally transcribed. The interviewer compared transcripts to audio-recordings to verify accuracy.

Data analysis

Categorical data were summarized using frequencies and percentages. Percentages were interpreted as low (0%–30%); moderate (31%–60%); and high (61%–100%). MARS subscale scores used to evaluate app quality were summarized using mean, standard deviation and range if normally distributed (otherwise median and 25th/75th percentile) for PTs who used the app for ≥1 month. Wilcoxon signed rank test was used to evaluate change from 0 to 5 months in item-level ratings of self-efficacy to perform recommended practices. McNemar’s test was used to evaluate change from 0 to 5 months in the percentage of PTs in the action or maintenance stage to measure outcomes. Adjustment for clustering of PTs within sites was not considered in the analysis, as study site did not exert a clustering effect in the main analysis [30]. An alpha value of 0.05 determined statistical significance.

Qualitative data and email communication with PLs/PPLs were used to determine implementation fidelity, specifically walkway set-up, and the timing and conduct of the learning sessions comprising the implementation strategy. To explore the implementability of the toolkit as a component of toolkit quality, a directed content analysis [75] of the qualitative data was performed. Each domain of the guideline implementability framework was defined in a codebook and used for initial coding categories. One author (N. M. S.) and a research assistant used the codebook to independently code four transcripts using the predetermined codes. New codes were developed for text that could not be categorized. After, coders met to discuss the results [75], reconcile differences and modify the codebook. The research assistant coded the remaining transcripts using NVivo 11.0 software.

Sample size

A sample size of 30 PTs was targeted to obtain a paired sample size of 150 patients post-stroke (5 patients/PT in pre- and post-intervention periods) for the main analysis of the impact evaluation [30]. For the process evaluation, a sample size of 30 PTs provided 80% power (2-sided α = 0.05) to detect an increase of 32% or greater in the percentage of therapists in the action or maintenance stage (given a 3% drop from the action/maintenance stage) [76], and an effect size of ∼0.5 for the mean change in self-efficacy ratings, pre- to post-intervention [77]. For the qualitative analysis, a sample size of 30 PTs was considered sufficient given that we reached theoretical saturation with 23 participants in a related study [15].

Step 8: Sustain knowledge use

Following completion of the process evaluation described in the present paper, the iWalk toolkit was made freely available at www.iwalkassess.com and launched using a comprehensive dissemination plan that targeted stroke and research networks, physical therapy schools worldwide, and members of the World Confederation of Physical Therapy.

Results

Figure 4 describes the flow of participants in the study. Three PPLs and 4 PLs overseeing practice at 9 hospital centres from 6 healthcare organizations participated. Hospitals provided acute care (n = 6), in-patient rehabilitation (n = 5), and out-patient rehabilitation (n = 6). Supplementary Material 5 outlines hospital characteristics. Pre-intervention, 1 PPL in an acute care hospital became unavailable, and 1 PL at a rehabilitation hospital was promoted and replaced by a PPL and together they facilitated study implementation. A total of 46 PTs (35 from Ontario, 11 from Nova Scotia) and 3 PPLs completed the online questionnaire at baseline and 34 PTs and 3 PPLs completed the online questionnaire post-intervention. Of the 37 PTs with pre- and post-intervention data, 57% held a Bachelor’s degree, and delivered acute care (43%), inpatient (35%) and outpatient (22%) rehabilitation. The percentage of PTs who had learned to administer the 10mWT and the 6MWT during their professional training, was 27% and 78% respectively. Preferred methods for evaluating walking included observational gait analysis (92%); Functional Independence Measure (62%); Timed “Up and Go” (62%); 2MWT (60%); 6MWT (49%); Community Balance and Mobility Scale (41%); and 10mWT (32%). We conducted nine focus groups with 33 PTs from all study sites, and interviewed 3 PPLs and 4 PLs. Table 4 shows that characteristics of PTs completing the questionnaire and focus groups were similar. Throughout the study, the clinical expert received two questions from participants about confirming the need to evaluate blood pressure pre- and post-6MWT and saving data in the app. An algorithm error for calculating 6MWT norms for men was detected and corrected partway through the study.

Figure 4. Flow of participants in the study.

Table 4. Participant and practice characteristics.

Characteristic	Therapists Completing Pre-Intervention Questionnaire (n = 49)	Therapists Completing Pre- and Post-Intervention Questionnaire (n = 37)	Therapists Participating in Focus Groups (n = 33)
Age
20–29 years	6 (12%)	3 (8%)	3 (9%)
30–39 years	25 (51%)	23 (62%)	19 (58%)
40–49 years	10 (20%)	6 (16%)	6 (18%)
50+ years	8 (16%)	5 (14%)	5 (15%)
Female	45 (92%)	33 (89%)	29 (88%)
Highest degree
Certificate/diploma	1 (2%)	1 (3%)	1 (3%)
Bachelor’s	28 (57%)	21 (57%)	20 (61%)
Entry-level master’s	16 (33%)	12 (32%)	10 (30%)
Applied/research master’s	4 (8%)	3 (8%)	2 (6%)
Years of clinical experience
1–5	8 (16%)	5 (14%)	5 (15%)
6–10	12 (24%)	11 (30%)	8 (24%)
11–15	14 (29%)	13 (35%)	11 (33%)
16–20	5 (10%)	2 (5%)	2 (6%)
>20	10 (20%)	6 (16%)	7 (21%)
No. patients post-stroke per week
<2 patients	8 (16%)	4 (11%)	5 (15%)
2–5	18 (37%)	16 (43%)	14 (42%)
6–10	13 (27%)	11 (30%)	8 (24%)
>10 patients	10 (20%)	6 (16%)	6 (18%)
Primary type of care delivered
Acute care, inpatient	17 (35%)	16 (43%)	13 (39%)
Rehabilitation, inpatient	20 (41%)	13 (35%)	13 (39%)
Outpatient	12 (24%)	8 (22%)	7 (21%)
Learned to administer 10mWT during professional program	12 (24%)	10 (27%)	8 (24%)
Learned to administer 6MWT during professional program	35 (71%)	29 (78%)	24 (73%)

Data, number (percentage). No.: number; 10mWT: 10-meter walk test; 6MWT: 6-minute walk test.

Implementation fidelity

Walkways were marked completely (n = 6), partially (i.e., 10 m section but not acceleration/deceleration distances; n = 2); or not at all (n = 1). All sites conducted learning sessions: 7 sites (78%) completed three learning sessions within 3 (n = 3), 4 (n = 2), or 5 (n = 2) months; 1 site completed the recommended learning activities in two sessions within 6 months; and 1 site, delayed by an influenza outbreak, completed two of the three learning sessions within 7 months.

Engagement with the toolkit

Table 5 presents results on completion of learning activities. Attendance at learning sessions 1, 2, and 3 was 78%, 61%, and 50%, respectively. The module reviewed most and least frequently was Module 2 describing test administration (97%), and Module 6 describing evaluation using audit and feedback (62%), respectively. The educational video was viewed by 83%. Percentage of PTs who practiced the 10mWT and 6MWT with colleagues was 73% and 54%, respectively. Duration of app use (n = 37) was: 0 months (16%), 1 month (27%), 2 months (16%), 3 months (11%), 4 months (8%), and 5 months (22%).

Table 5. Completion of learning activities by physical therapists (n = 37).

Learning activity	n	n (%)
Review/read iWalk Guide Module 1: Introduction	37	34 (92%)
Review/read iWalk Guide Module 2: Performing the Tests	37	36 (97%)
Review/read iWalk Guide Module 3: Interpreting Test Performance	37	32 (86%)
Review/read iWalk Guide Module 4: Educating and Setting Goals	37	29 (78%)
Review/read iWalk Guide Module 5: Selecting Treatments	37	25 (68%)
Review/read iWalk Guide Module 6: Evaluating Practice using Audit and Feedback	37	23 (62%)
Review/read iWalk Guide Module 7: Putting it All Together with Case Scenarios	37	26 (70%)
View iWalk YouTube^TM video on administering the 10-meter and 6-minute walk tests	35	29 (83%)
Practice 10-meter walk test with colleagues	37	27 (73%)
Practice 6-minute walk test with colleagues	37	20 (54%)
Complete Activity 1: Interpreting Test Performances described in iWalk Guide Module 7: Putting it All Together with Case Scenarios	36	23 (64%)
Complete Activity 2: Educating and Setting Goals described in iWalk Guide Module 7: Putting it All Together with Case Scenarios	36	22 (61%)
Meet and complete Learning Session 1 with physical therapy colleagues	37	29 (78%)
Meet and complete Learning Session 2 with physical therapy colleagues	36	22 (61%)
Meet and complete Learning Session 3 with physical therapy colleagues	36	18 (50%)
Review/read iWalk User Guide to using the iWalkAssess app	37	34 (92%)

App quality

Table 6 presents MARS item ratings. Mean (standard deviation) score for engagement, functionality, aesthetics, and information subscales and total score were 2.9 (0.6), 3.5 (0.6), 3.4 (0.6), 4.0 (0.8), and 3.5 (0.5), respectively. Table 7 presents level of PTs’ agreement on the likelihood that the app will improve knowledge, attitudes, intentions, and behaviour for each walk test (n = 30). The following indicates the percentage of PTs who agreed or strongly agreed the app would likely increase: (i) knowledge to administer the walk test: 10mWT: 90%, 6MWT: 83%; (ii) knowledge to interpret test performance: 10mWT: 100%, 6MWT: 87%; (iii) attitude toward using the walk tests: 10mWT: 73%, 6MWT: 73%; (iv) intentions/motivation to administer the walk tests: 10mWT: 77%, 6MWT: 67%; and (v) walk test use: 10mWT: 73%, 6MWT: 67%.

Table 6. Mobile App Rating Scale scores (n = 31)^a.

Domain and items	Median^b (P25, P75)
Engagement
Entertainment	2.0 (2.0, 3.0)
Interest	3.0 (2.0, 4.0)
Customization	3.0 (2.0, 3.0)
Interactivity	3.0 (2.0, 3.0)
Appropriateness for target group^c	4.0 (3.0, 4.0)
Function
Performance	3.0 (3.0, 4.0)
Ease of Use	3.0 (3.0, 4.0)
Navigation	3.0 (3.0, 4.0)
Gestural design	3.0 (3.0, 4.0)
Aesthetics
Layout	4.0 (3.0, 4.0)
Graphics	3.0 (3.0, 4.0)
Visual appeal	3.0 (3.0, 3.0)
Information
Accuracy of app description	4.0 (3.0, 4.0)
Goals	4.0 (3.0, 4.0)
Quality of information	4.0 (4.0, 5.0)
Quantity of information	4.0 (4.0, 5.0)
Visual information	4.0 (4.0, 5.0)
Credibility	4.0 (2.0, 4.0)
Subjective quality^d	n (%)
Intention to recommend app to those who might benefit
Not anyone	0
Very few people	3 (10)
Several people	17 (55)
Many people	4 (13)
Everyone	7 (23)
Expected frequency of use of app in next 12 months if relevant
0 times	1 (3)
1–2 times	3 (10)
3-10 times	11 (35)
11–50 times	13 (42)
>50 times	3 (10)

P₂₅: 25th percentile; P₇₅: 75th percentile.

^a 5-point scale: 1-inadequate, 2-poor, 3-acceptable, 4-good, 5-excellent.

^b Scores are median and 25th and 75th percentile unless otherwise noted.

^c Physical therapists and other health professionals who evaluate walking post-stroke.

^d 45% and 32% endorsed a 3-star and 4-star rating of the app, respectively.

Table 7. Likelihood of iWalkAssess app improving users’ knowledge, attitudes, intentions, and behaviour (n = 30).

Area of impact	Strongly disagree or disagree	Neutral	Agree or strongly agree
	n (%)
Knowledge of how to administer 10mWT post-stroke	2 (7%)	1 (3%)	27 (90%)
Knowledge of how to administer 6MWT post-stroke	1 (3%)	4 (13%)	25 (83%)
Knowledge of how to interpret 10mWT	0	0	29 (100%)
Knowledge of how to interpret 6MWT	0	4 (13%)	26 (87%)
Attitude toward using 10mWT	2 (7%)	6 (20%)	22 (73%)
Attitude toward using 6MWT	1 (3%)	7 (23%)	22 (73%)
Intention to use 10mWT	1 (3%)	6 (20%)	23 (77%)
Intention to use 6MWT	2 (7%)	8 (27%)	20 (67%)
Behaviour change: likely to increase 10mWT use post-stroke	1 (3%)	7 (23%)	22 (73%)
Behaviour change: likely to increase 6MWT use post-stroke	2 (7%)	8 (27%)	20 (67%)

10mWT: 10-meter walk test; 6MWT: 6-minute walk test.

Toolkit implementability

Table 8 summarizes the qualitative findings outlining the barriers and facilitators to using the iWalk toolkit in each implementability domain. Relevance: PTs noted that the guide clearly communicated how to not only administer the walk tests but also use the tests results in clinical practice. Adaptability: Although the guide was designed as the primary educational resource, some PTs expressed a preference for learning content from the app. Similarly, some PTs preferred to carry and refer to the entire guide while delivering patient care, even though the app, and protocol and reference value summaries in the guide appendix were designed as portable tools to facilitate clinical application. Usability: The extensive length of the guide made it difficult to carry and quickly access information in the clinical setting. No comment was made regarding challenges with carrying mobile phones to use the app. Supplementary Material 6 outlines participants’ recommendations related to improving the content of the guide and the app and presenting the information in different formats to facilitate learning. Despite suggestions for alternate learning resources and strategies, there was consensus that the toolkit was a sufficient resource for learning and application to practice.

Table 8. Qualitative findings describing iWalk toolkit implementability.

Domain	Barrier	Facilitators	Quotes
Relevance	PTs did not use the module on treatment selection	Guide clearly supported CSBPR	“The actual booklet is great in terms of understanding not just how to use the test but what to use the test for.” (Hospital 4, PL, acute care setting)
Adaptability	Video redundant if familiar with tests Practice of some tasks (e.g., taking HR/BP, setting goals) unnecessary as these reflect expected clinical competencies	App useful alternative to written guide Guide is particularly good for less experienced clinicians Video helps to visualize test in action	“And then just the video, to be able to kind of see how it's done in other facilities, was helpful, just to be able to put it into action, into real life.” (Hospital 1, PPL, in- and out-patient rehabilitation setting)
Usability	Review of guide for learning requires careful attention During clinical practice, guide not portable due to size and not quick to consult Overwhelming content Inconsistent functioning of 6WMT length counter in app particularly in the lanyard	Guide is useful ongoing resource Guide content is clear and organized Photos and diagrams in guide facilitated learning Walk test approach useful across patient populations	“So going through it the first time was a little bit cumbersome just because there's so much information in here. But once you get through it then you can use it as a reference guide…Essential? The protocol obviously for both tests. I like the charts with the values on there. We could always go back and check if we needed to. I like the diagrams in it. Diagrams make it a lot easier to kind of understand what’s going on with the protocol.” (FG5, Hospital 5B, PT, inpatient rehabilitation setting)
Validity	None	Evidence (“Top 10”, CSBPR) motivated therapists who desired evidence based practice Evidence supported relevance of tests	“I think it’s great that you have that link back to best practices and the best practice guidelines. You know, it was just from a personal point of view, that’s something that I already review on a consistent basis. I didn’t need that extra information. But I think it’s a great thing to have there as part of it because it does set it out nicely in terms of there’s a reason why we’re doing this and why.” (FG3, Hospital 6 A, PT, acute care setting)
Applicability and Communicability	Belief across settings that only patients who can walk 6 min continuously can perform the 6MWT decreased its perceived applicability to clientele Guide lacked patient scenarios from acute care Walk test approach challenging to implement in acute care due to: low ambulation levels and medical issues; minimal time to conduct tests and follow up on goals due to brief LOS Traffic/equipment in halls in acute care decreases validity of test result Too early in recovery in acute care to discuss walking performance Increasing walk test distance or speed did not align with setting patient-centred goals, thus patient report not relevant 6MWT norms don’t apply to people aged > 85 years Accuracy of community distances questioned	Walk tests appropriate for in- and out-patient rehabilitation Reference values supported relevance of tests, and increased buy-in to change practice, and knowledge of normal ambulation Walk test instructions and calculation feature in app saved time, were convenient, and useful to set walk test goals Aphasia-friendly instructions appreciated %norm easier than gait speed for patients to understand Case scenarios outlining how to communicate test results useful PTs used reference values with patients to motivate, educate about change over time, initiate conversations about discharge, education/prepare to walk in community	“The only advantage I can see of doing the test in acute care is so that [PTs in] rehab would have that number for use when they get there. Having said that, if they’re going to do the test the day that the patient gets there then it probably doesn’t make a difference.” (FG9, Hospital 2, PT, acute care setting) “I found the normative values really helpful to just kind of frame my own thinking about okay, this is what at this age should be happening.” (FG8, Hospital 3B, PT, out-patient setting)
Resource Implications and Implementation	1 PPL overseeing 2 sites described walkway set-up, review of guide/app, and learning session facilitation as “a lot of work”; they felt PTs would expect them to be the expert Difficulty implementing learning session homework of finding an appropriate patient to practice walk tests within 2 weeks Difficulty finding a 30-meter distance for 6MWT due to insufficient hallway length, stretchers, and equipment Hospital doesn’t permit floor marking	Space and equipment checklist, walkway diagrams, instructions useful to set-up walkways Instructions, goals, and agendas for learning sessions useful Consensus that toolkit was a sufficient resource to adopt new tests like 10mWT and 6MWT without need for training session with an external expert Some experienced PTs felt guide was a sufficient learning tool; others felt learning sessions were essential Learning sessions could be done in ≤ 1 h 2 PPLs developed slide decks for learning sessions 2 PPLs covered material for 3 sessions in 2 sessions	“I tried my best obviously to follow the agenda that they set out. When I set up the appointments, I would send out reminders as to what to bring, what to do ahead of time, based on the iWalk guide…so I used it as a bit of a lifeline.” (Hospital 1, PPL, in- and out-patient rehabilitation setting) “Without the learning sessions, it wouldn't have been sufficient… I’m glad you gave us … some practical sessions … But after that, [the iWalk guide] was sufficient.” (FG5, Hospital 5B, PT, acute care and inpatient rehabilitation setting)
Evaluation	Audit and feedback would require manual review of charts and cause an unacceptable workload for the PPL Audit and feedback inappropriate as: (1) few ambulatory patients in acute care and they might be discharged before PTs can administer tests; (2) sites already had to achieve targets for implementing quality-based procedures and PL didn’t want to add more targets; and (3) PTs were experienced and did not require it	Preferred option of meeting to discuss reflections on practice change instead of conducting audit and feedback	“With the high turnover of patients and not necessarily being able to get to the tests on all the patients…You might try to audit five people and you’d have none that would have been done.” (Hospital 3 A, PL, acute care setting with out-patient services)

PT: physical therapist; CSBPR: Canadian Stroke Best Practice Recommendations; PL: professional leader; HR: heart rate; BP: blood pressure; PPL: professional practice leader; 6MWT: 6-minute walk test; FG: focus group; LOS: length-of-stay; 10mWT: 10-meter walk test.

Validity: PTs expressed valuing the incorporation of research evidence in clinical practice and acknowledged the clear list of clinical practices that the toolkit aimed to promote, and links to supporting research evidence and guideline recommendations in the guide. Applicability and communicability: PTs expressed an appreciation for the pictorial instructions for people with aphasia, quick access to interpretations of walk test performance using normative, MDC, and community ambulation values, and case-based examples of how to communicate walk test results, as this information enhanced their knowledge and enabled them to apply test results to individual patients for patient education and goal setting. Focus groups revealed a misconception that the 6MWT is only for patients who can walk continuously for six minutes. These beliefs limited the perceived and actual clinical applicability of the test, particularly in the acute care setting. PTs noted additional factors that decreased the feasibility of toolkit implementation. Organizational factors related to the challenge of finding a 30 m 6MWT walkway length free of traffic, and hospital policy prohibiting taping of floors and walls. Toolkit factors included the lack of a case scenario reflecting the acute care setting; perceptions by some that goal-setting for improved walk test performance was not patient-centred or relevant to acute care; and inconsistent function of the app’s 6MWT length counter.

Resource implications: PLs and PPLs described referencing the guide’s equipment checklist and instructions and diagrams for setting up walkways, but not the list of equipment and training costs as sites either had access to the equipment needed to administer the tests or had no budget to purchase new equipment.

Implementation: PLs and PPLs consistently described using instructions and agendas in the guide for setting up and running learning sessions. At one site where the PPL became unavailable, administrative staff scheduled learning sessions. PTs followed the agendas during the sessions, but did not describe completing learning activities in advance as there was no one to provide reminders to do this. Evaluation: Most PL/PPLs and PTs indicated that limited resources to perform an audit prohibited the use of audit and feedback to monitor documentation of walk test use. They preferred the alternative proposed in the guide of scheduling a group meeting to reflect on and discuss practice change.

Effects of toolkit

The percentage of 37 PTs in each stage of readiness to measure outcomes at 0 versus 5 months was: pre-contemplation, 0% versus 0%; contemplation, 0% versus 0%; preparation, 5% versus 8%; action, 73% versus 65%; and maintenance, and 22% versus 27%. The percentage of PTs in the action or maintenance stage decreased from 95% to 92% between 0 and 5 months; this change was not significant (McNemar’s S = 0.33, p = 0.56).

Table 9 presents self-efficacy ratings pre- and post-intervention. Mean changes in self-efficacy ratings to perform six 10mWT practices ranged from 17 to 30 percentage points and were significant (p ≤ 0.004). Mean changes in self-efficacy ratings to perform six 6MWT practices ranged from 2-9 percentage points and were not significant (p ≥ 0.112). The two largest increases in self-efficacy for both tests related to interpreting performance, and educating patients about performance.

Table 9. Self-efficacy to apply walk test practices pre- and post-intervention (n = 37).

	Self-efficacy percentage points, mean (SD) range
Item	Pre	Post	Change	p-value
Identify patients with stroke appropriate for 10mWT	68 (35) 0–100	87 (17) 20–100	20 (32) −20 to 100	0.000
Safely administer 10mWT post-stroke	71 (37) 0–100	89 (18) 20–100	17 (33) −30 to 100	0.004
Interpret 10mWT performance	48 (37) 0–100	78 (19) 20–100	30 (36) −40 to 100	0.000
Educate patients about 10mWT performance	51 (37) 0–100	79 (21) 20–100	28 (34) −30 to 100	0.000
Set patient-centred goals to improve 10mWT performance	54 (36) 0–100	76 (26) 10–100	22 (35) −50 to 100	0.001
Apply treatments known to improve 10mWT performance	56 (37) 0–100	76 (25) 0–100	20 (34) −30 to 100	0.001
Identify patients with stroke appropriate for 6MWT	83 (18) 40–100	86 (18) 10–100	4 (18) −30 to 40	0.234
Safely administer 6MWT post-stroke	84 (21) 30–100	85 (21) 10–100	2 (20) −50 to 60	0.564
Interpret 6MWT performance	68 (25) 10–100	77 (20) 10–100	9 (28) −50 to 80	0.116
Educate patients about 6MWT performance	70 (24) 10–100	78 (21) 10–100	9 (28) −50 to 80	0.112
Set patient-centred goals to improve 6MWT performance	71 (24) 20–100	76 (22) 10–100	5 (26) −50 to 60	0.263
Apply treatments known to improve 6MWT performance	72 (26) 10–100	77 (24) 0–100	5 (23) −60 to 60	0.157

10mWT: 10-meter walk test; 6MWT: 6-minute walk test.

Table 10 outlines the impact on app use. Approximately a third of PTs reported using the app most or all of the time to interpret performance on the 10mWT and 6MWT, and to share 10mWT and 6MWT information with the patient.

Table 10. Nature of iWalkAssess app use during intervention period (n = 31)^a.

Type of use	None or little of the time	Some of the time	Most or all of the time
Improve provider knowledge of 10mWT	18 (58%)	5 (16%)	8 (26%)
Administer 10mWT	18 (58%)	5 (16%)	8 (26%)
Interpret 10mWT performance using reference values	18 (58%)	4 (13%)	9 (31%)
Share 10mWT information with patient	19 (61%)	3 (10%)	9 (31%)
Share 10mWT information with healthcare team	19 (61%)	9 (31%)	3 (10%)
Improve knowledge of 6MWT	18 (58%)	5 (16%)	8 (26%)
Screen patient for 6MWT	20 (65%)	5 (16%)	6 (19%)
Administer 6MWT	17 (55%)	5 (16%)	9 (31%)
Interpret 6MWT performance using reference values	16 (52%)	4 (13%)	11 (35%)
Share 6MWT information with patient	18 (58%)	2 (6%)	11 (35%)
Share 6MWT information with healthcare team	21 (68%)	6 (19%)	4 (13%)

^a Values are n (%).

Discussion

Implementation fidelity and engagement with the toolkit

The universal completion of learning sessions, and the moderate-to-high rates of attendance at learning sessions, review of the guide and video, and practice of walk tests, provide some evidence linking the toolkit intervention with improved walking assessment practice observed in the main analysis. The moderate percentage of PTs who used the app for 3–5 months does not necessarily imply these were the only individuals who attempted to use the walk tests because PTs’ preferences for using the smartphone versus the printed guide varied. This finding aligns with previous research [60].

Toolkit quality (toolkit implementability and app quality)

Findings related to toolkit quality provide insight as to whether the intervention addressed known barriers across practice settings to enable practice change. Qualitative findings related to toolkit validity, applicability, and communicability, triangulated with MARS ratings suggest the intervention increased knowledge, and fostered beliefs in the clinical relevance (i.e., outcome expectations) and implementation of recommended practices, except in acute care hospitals. This helps to explain why documented walk test administration was lowest in acute care settings. The app was a particularly effective tool for enabling quick and convenient access to test instructions and reference values. PTs are generally positive about using a personal digital device to support evidence-based practice [60]. Few PTs in the current study expressed previously reported concerns about visual accessibility, given the small device size, or using a digital device in front of patients [60].

Belief in clinical relevance is an important factor influencing best practice implementation [78,79]. There is little guidance, however, on how to change negative attitudes. Theoretical Domains Framework [64] developers recommend providing feedback on behaviour, including self-monitoring and comparison with peers, to help change PTs’ beliefs about consequences. Our findings suggest that persuasive research-based arguments and guideline recommendations supporting the recommended practices, case scenarios reflecting the targeted clinical contexts, and information on how to interpret scores for individual patients can improve attitudes towards recommended assessment practices, thereby facilitating practice change.

Effects on readiness to measure outcomes, self-efficacy, and app use

The toolkit and implementation strategy could not increase PTs’ readiness for measuring outcomes because 95% of PTs were already in the action or maintenance stage at baseline. Although the scale used to determine readiness to change was not specific to the 10mWT and 6MWT, these results may indicate PTs were actively involved with modifying or maintaining their approach to using these walk tests as a result of the toolkit intervention [73]. In contrast, self-efficacy to perform all recommended 10mWT practices significantly increased, with the largest gains observed for interpreting performance and educating patients about performance. A similar pattern of smaller but non-significant improvements in self-efficacy for 6MWT practices was observed. Because self-efficacy beliefs are considered to predict behaviour [34], these results may partly explain why a larger increase in documented 10mWT compared to 6MWT administration was found in the primary analysis. Compared to the 10mWT, a higher percentage of PTs reported learning the 6MWT in their professional training, identified the 6MWT as their preferred approach to measuring walking capacity, and consistently rated their self-efficacy for 6MWT practices higher at baseline, leaving less room for improvement. Moreover, difficulty administering the 6MWT due hospital traffic was a universal challenge that toolkit resource could not overcome. Conducting the 6MWT using a 15-meter walkway, found reliable in a recent study [80], may address some of the issues associated with the 30-meter walkway.

Limitations

Self-reported methods of data collection in this study were vulnerable to social desirability bias [81] and may have led to an overestimation of effects on study outcomes, such as knowledge, self-efficacy, attitudes, and behaviour. As noted previously, this before-and-after study did not have a control group. Thus, a causal relationship between the toolkit and measured change in study outcomes cannot be assumed. Given only a third of PTs reported using the 10mWT at baseline, however, it is likely that the toolkit and implementation strategy contributed to the positive changes in self-efficacy to perform recommended 10mWT practices observed.

Conclusions

This study contributes novel understanding of how a process framework, a determinant framework, and classic theories can be used to guide the development and process evaluation of a toolkit and implementation strategy to promote recommended practices in rehabilitation. We used theory to plan the overall research process, identify determinants of behaviour, incorporate strategies to enable behaviour change in the intervention, and identify intermediate study outcomes and hypotheses guiding qualitative and quantitative data analysis. The approach may help to inform toolkit development to advance other rehabilitation practices.

Acknowledgements

The authors acknowledge Jonathan Lung for his work in developing the iWalkAssess app.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data gathered and used for this analysis are not publically available because of the inclusion of identifying information of the participants and potentially sensitive case information.

Additional information

Funding

Funding for this project was provided by the Government of Ontario, the Canadian Institutes of Health Research, and the Heart & Stroke Foundation.