A preliminary analysis of physical therapist agreement regarding the perceived impairments in cases of runners with knee pain

ABSTRACT

Background

There is a scarcity of evidence describing how physical therapists use data from clinical examinations to inform the treatment of runners with knee pain.

Objective

Our purpose was to examine the between physical therapist agreement on the selection of perceived impairments in runners with knee pain.

Methods

Twelve physical therapists reviewed two cases of runners with knee pain. The cases included clinical subjective information, objective data, and review of videos of each participant running. Each rater selected up to three perceived impairments (from a list of eight) that each physical therapist would address at the next physical therapy session. Percent agreement was calculated to determine the between rater agreement on each individual perceived impairment selection and Fleiss Kappa was calculated for each unique combination of three perceived impairments per case.

Results

Twelve raters with 51 (18–156) months of clinical experience participated. Percent agreement ranged from 8%-100% for both cases for individual impairments. When assessing the unique combination of three impairments selected, inter-rater agreement was less than what is expected due to chance alone (κ = −0.09, p = .92; κ = −0.09, p = .98) for both cases.

Conclusion

The 12 physical therapists demonstrated poor to excellent levels of agreement when selecting an individual perceived impairment. Agreement was worse than chance when selecting a combination of three unique impairments.

KEYWORDS:

• Reliability
• gait analysis
• running related injuries
• clinical examination

Introduction

Running related injuries (RRIs) occur in up to 79% of runners and are common for runners of all skill levels (Franke, Backx, and Huisstede, 2019; Kluitenberg, van Middelkoop, Diercks, and van der Worp, 2015; Van Gent et al., 2007; Videbæk, Bueno, Nielsen, and Rasmussen, 2015). While RRIs occur at various joints, the prevalence of RRI at the knee ranges from 16.1 to 50% and is the most common joint to experience RRIs (Franke, Backx, and Huisstede, 2019; Linton and Valentin, 2018; Van Gent et al., 2007). The impact of RRIs and persistent pain with running could be associated with long-term joint pain, development of osteoarthritis, and poor joint health (Alentorn-Geli et al., 2017; Bullock et al., 2020). Despite this, up to 86% of runners continue to run while experiencing pain from an injury (Linton and Valentin, 2018). This highlights an opportunity for physical therapists to have a positive impact on these athletes and potentially alter the trajectory of these individuals while promoting long-term health and fitness (Roos and Arden, 2016).

Running-related injuries are multifactorial, with each body region having potentially differing factors that contribute to RRIs. Movement impairments are one (of many) contributing factors (Bramah, Preece, Gill, and Herrington, 2018; Dingenen et al., 2019; Dugan and Bhat, 2005; Neal, Barton, Birn-Jeffery, and Morrissey, 2019). Movement impairments may consist of aberrant motions at the involved or regional joints that directly provoke pain or are thought to provoke pain with progressive loading over time (Wainner, Whitman, Cleland, and Flynn, 2007). To better understand the potential impact of movement-related impairments on RRIs, it is important to examine data elements from multiple tools during an orthopedic examination including physical impairment-based (i.e. muscle performance, range of motion, muscle extensibility), movement-based (i.e. dynamic postural control), and running-specific assessments (i.e. joint angle measurements at different phases of running gait cycle). This data may be utilized to guide interventions focused on reducing pain and injury risk, while potentially improving running performance (Chumanov, Wille, Michalski, and Heiderscheit, 2012; Heiderscheit et al., 2011; Napier, Cochrane, Taunton, and Hunt, 2015; Willy and Davis, 2011).

A comprehensive clinical orthopedic examination of a runner with knee pain may include a medical history, detailed subjective interview, physical impairment-based measures (i.e. range of motion and strength) as well as a movement-based assessment of function in task-specific positions (Reiman and Thorborg, 2014; Willy et al., 2019). Each component of the clinical exam provides insight into different potentially contributory factors. For example, the medical history and subjective interview may identify a previous injury at the same joint or a training error. However, the understanding of running-related mechanics that are contributory to the underlying pathology may be limited without the use of running specific assessments (Bramah, Preece, Gill, and Herrington, 2018; Dingenen et al., 2019; Neal, Barton, Birn-Jeffery, and Morrissey, 2019; Rees, Younis, and MacRae, 2019). The inclusion of running specific assessments such as two-dimensional video running gait analysis in a comprehensive clinical examination of a runner may reveal and reliably quantify other contributory movement impairments during running that otherwise may have been missed (Bramah, Preece, Gill, and Herrington, 2018; Dingenen et al., 2019; dos Santos, Nakagawa, Serrão, and Ferber, 2019; Neal, Barton, Birn-Jeffery, and Morrissey, 2019; Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Rees, Younis, and MacRae, 2019; Wille et al., 2014). It is important to note that two-dimensional running analyses will provide an understanding of frontal and sagittal plane running mechanics (McGinley, Baker, Wolfe, and Morris, 2009). However, understanding transverse plane joint motion, which can be an important component of running-related pain especially at the knee, will be limited, and only hypothetical inferences can be made regarding their potential influence on running mechanics (McGinley, Baker, Wolfe, and Morris, 2009).

Although a comprehensive examination of a runner may identify a set of impairments that are amendable to clinical interventions, there is a scarcity of evidence that describes how physical therapists use examination findings to prioritize which impairments to address and how the prioritization of impairments may influence clinical decisions (Chumanov, Wille, Michalski, and Heiderscheit, 2012; Heiderscheit et al., 2011; Napier, Cochrane, Taunton, and Hunt, 2015; Willy and Davis, 2011). To highlight this uncertainty, clinical practice guidelines have been written to provide physical therapists with a comprehensive framework for the evaluation and treatment of a variety of diagnoses (Martin et al., 2018; Willy et al., 2019). However, the implementation of such recommendations is limited (Côté, Durand, Tousignant, and Poitras, 2009; Lin et al., 2018; Poitras, Durand, Côté, and Tousignant, 2012; Stander, Grimmer, and Brink, 2019; Zadro, O’Keeffe, and Maher, 2019). Most notably, it has been shown that only 58% of clinicians perform recommended interventions for patients with knee pain/osteoarthritis and 98% of clinicians perform at least one intervention that has no recommendation (Zadro, O’Keeffe, and Maher, 2019). This misalignment of interventions for a particular patient may be a critical contributing factor that results in unnecessary variations in practice. These variations may lead to overtreatment, unnecessary healthcare costs, and/or worse outcomes (Traeger, Moynihan, and Maher, 2017). These negative findings associated with variations in practice have been shown in patient populations including low back pain, knee osteoarthritis, and knee arthroplasty (Flynn, Smith, and Chou, 2011; Lin et al., 2018; Naylor, Hart, Harris, and Lewin, 2019; Oatis et al., 2014; Traeger, Moynihan, and Maher, 2017; Zadro, O’Keeffe, and Maher, 2019). It is plausible that other patient populations (i.e. runners and active individuals) are not immune from these issues.

The lack of clear consensus of how physical therapists interpret a comprehensive orthopedic examination in a runner with a RRI and determine which impairments are present and clinically important to address requires investigation. This inquiry is vital in the effort to reduce unnecessary practice variation and optimize care and outcomes for runners. The purposes of this preliminary analysis were to: 1) understand the inter-rater agreement on individual impairment selection per case; and 2) understand the inter-rater agreement on unique impairment combinations selected per case. We hypothesized that physical therapists would demonstrate moderate agreement on individual impairment selection, and moderate agreement when selecting a combination of three impairments.

Methods

This study protocol was reviewed and permission to conduct the study was granted by the Ohio State University Institutional Review Board (IRB Study Number: 2019H0481). This study represents a preliminary analysis of data obtained from a video running gait joint angle reliability study assessing multiple physical therapist raters. A convenience sample of twelve licensed physical therapists employed at a hospital-based, outpatient sports-medicine focused clinic (Jameson Crane Sports Medicine Physical Therapy Clinic, Ohio State University Wexner Medical Center) who had a clinical interest, expertise, and actively sought out opportunities to treat patients with RRIs were recruited for this study. The raters provided their written informed consent to participate, and duration of licensed physical therapy experience was collected.

All included physical therapists had completed an internally created competency program pertaining to endurance athlete medicine and video running gait analysis. The formal competency consisted of an internally generated written and practical examination, which would qualify the physical therapist to operate a ‘running injury clinic’ which includes clinical orthopedic evaluation and video running gait analysis (as warranted) independently in a fee-for-service model of care. The formal competency was created by the Endurance Medicine Team leadership group, consisting of three clinicians with extensive experience working with RRIs. The information included in the competency was derived from multiple continuing education courses and an extensive review of the literature. The competency includes topics related to RRI incidence, underlying etiologic theory, and evidence-based treatment based on current available best evidence (Heiderscheit et al., 2011; Lenhart, Thelen, and Heiderscheit, 2014; Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Souza, 2016; Wille et al., 2014). The competency exam involves a written component and practical examination on two live patients (i.e. physical examination, marker placement, camera set up, test administration, and interpretation of results) with > 80% score necessary to pass. The competency undergoes review and is updated every two years.

Determination of up to three perceived impairments

Two patient cases of recreational runners with knee pain (anterior and lateral respectively) were selected for the physical therapist raters to review from a larger, unpublished study examining runners. The cases included in this secondary analysis were the only two from the primary study with pathology localized to the knee joint. The knee joint-specific cases were chosen for this exploratory analysis due to the commonality of RRIs to the knee joint (Franke, Backx, and Huisstede, 2019). The cases included a written narrative of the subjective and objective physical therapy examination, as well as previously recorded standardized videos of the patients running on a treadmill. Each rater was first presented with the written case narrative, which corresponded to the videos they were asked to assess. The case narratives are described in Appendices A and B. The standardized protocol used to record the video of the patients running on a treadmill is described in Appendix C. Video recordings included a posterior and lateral view of the runner on the treadmill and were 6–10 seconds in length, in each position (Figure 1).

Figure 1. Screenshots from video running gait analysis.

(a) Posterior View(b) Lateral View.

After reading the case narrative and viewing the previously recorded running videos, the physical therapist raters were asked to measure sagittal and frontal plane joint angles on the iPad 2 (2011, Apple Inc, Cupertino, CA), using the V1 Golf Motion Analysis iPad application (V1 Sports, Plymonth, MI). The iPad 2 and V1 Golf Motion Analysis application were chosen to increase the external reproducibility of this study. The iPad is a readily available tool in all clinical settings for a moderate cost and the application was free to use and allowed joint angle measurements.

The angles of interest are listed in Table 1 and were used across all eight cases (multiple involved joints). The joint angles measured provide a wide variety of possible joints that suspected deviations could occur in runners, are noted in the literature as being associated with RRIs and/or shown to have moderate to excellent measurement reliability (Intraclass correlation coefficient above 0.70) (Bramah, Preece, Gill, and Herrington, 2018; Chumanov, Wille, Michalski, and Heiderscheit, 2012; Damsted, Nielsen, and Larsen, 2015; Dingenen et al., 2019; Heiderscheit et al., 2011; Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Reinking et al., 2018). The joint angle measurement inter-rater reliability was calculated bilaterally when able and ranged from poor to moderate for continuous angles (Intraclass correlation coefficient: 0.16 to 0.58) and categorical variables (Kappa: 0.15–0.55) (Portney and Watkins, 2000) (Table 2a,b).

Table 1. Joint angle measurements of interest.

Posterior View	Ankle STJ Eversion- Mid Stance*
	Knee Position (varus/neutral/valgus) – Mid Stance**
	Trendelenburg- Mid Stance^#
Lateral View	Ankle DF- Loading Response^$
	Knee Flexion- Initial Contact^&
	Knee Flexion- Mid Stance^ψ
	Hip Extension- Pre-swing^Ζ

STJ = Subtalar Joint; DF = Dorsiflexion; *measured: line bisecting the heel and a line bisecting the tibia, center of the ankle joint as the axis; **measured: line bisecting the tibia and a line bisecting the femur, center of the knee joint as axis, greater than 5° from neutral (180 degree) line in either direction; ^#measured: angle between a line parallel to the ground and a line created by both PSIS (as marked by sticker/dots); ^$measured: angle between a line parallel to the ground (the treadmill belt) and a line from heel to 1^st or 5^th met head where the sole of shoe meets the bottom of the shoe covering; ^&measured: line bisecting the tibia and a line bisecting the femur, center of the knee joint as the axis. ^ψmeasured: line bisecting the tibia and a line bisecting the femur, center of the knee joint as the axis; ^Ζ “Neutral pelvis” line is established by creating a line between the anterior pelvic crest and posterior pelvic crest. Then a line bisecting the trunk (perpendicular to neutral pelvis line) and a line bisecting the femur with the greater trochanter as the axis.

Table 2. Inter-rater reliability for joint angle measurement.

(a) Posterior view joint angle between rater reliability
Joint Angle Measurement	Left (ICC, 95%CI)	Right (ICC, 95%CI)	Total (ICC, 95%CI)
Ankle STJ Eversion* (Midstance)	0.066 (−0.06–0.99)	−0.85 (−0.90–0.84)	0.26 (0.04–0.86)
Knee Position** (Midstance)	0.13	0.15	0.15
Trendelenburg* (Midstance)	0.15 (−0.07–0.99)	0.56 (0.14–0.10)	0.55 (0.23–0.95)
ICC = Intraclass Correlation Coefficient, *Intraclass Correlation Coefficient calculated at 95% confidence interval, CI = confidence interval, **measured as percent agreement
(b) Lateral view joint angle between rater reliability
Joint Angle Measurement	Left (ICC, 95%CI)	Right (ICC, 95%CI)	Total (ICC, 95%CI)
Ankle Dorsiflexion* (Loading Response)	0.82 (−0.06–0.99)	0.31 (0.01–0.10)	0.16 (0.00–0.79)
Knee Flexion* (Initial Contact)	0.42 (0.06–0.10)	0.77 (0.34–0.10)	0.58 (0.26–0.95)
Knee Flexion* (Midstance)	0.38 (0.04–0.10)	0.17 (−0.03–0.10)	0.31 (0.07–0.88)
Hip Extension*^# (Pre-Swing)	−0.044 (−0.08–0.98)	Not measured	Not applicable

ICC = Intraclass Correlation Coefficient, *Intraclass Correlation Coefficient calculated at 95% confidence interval, CI = confidence interval, #Unable to obtain right side due to visual obstruction from the treadmill.

Following completion of joint angle measurements, the raters were provided a list of eight impairments and asked to “select up to three impairments that you would want to address at their next session” based on all aspects of the patient case. The list of potential running-specific impairments (Table 3) was based on previous evidence identifying these impairments as common in runners or having an association with injured runners (Bramah, Preece, Gill, and Herrington, 2018; Chumanov, Wille, Michalski, and Heiderscheit, 2012; Dingenen et al., 2019; Heiderscheit et al., 2011; Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Souza, 2016). Although physical and movement-based impairments are also present in patients with knee pain, only running-specific impairments were included in this analysis to prevent heterogeneity in responses and to ensure the data could be analyzed. ‘Perceived impairments’ were defined as potential problems in body function or structure such as a deviation or loss that may be perceived as clinically significant to the patient as a runner (Fox, Krahn, Sinclair, and Cahill, 2015). Several perceived impairments are noted as “inappropriate” thus, the value could be either too much of the noted motion or too little and the interpretation was at the discretion of the physical therapists. The choices were constrained to three options to force the physical therapist raters to prioritize what they perceived as clinically impactful to address. We did not gather information regarding how the physical therapist would address the selected impairment (i.e. running gait re-training, manual therapy, neuromuscular control exercise, or therapeutic strengthening exercise).

Table 3. Perceived impairment selection list.

Impairment Number	Movement Impairment	Impairment Cut-Offs
1	Trendelenburg gait	Males: Too much: ≥6° Too little: ≤2° Females: Too much: ≥8° Too little: ≤3°
2	Inappropriate Ankle DF with landing	Too much ankle PF: ≥6° Too much ankle DF: ≥11°
3	Inappropriate Ankle STJ Eversion with landing	Too much: ≥9° Too little: ≤5°
4	Knee Valgus/Varus Positioning	> 5 degrees from neutral (180 degree) line in either direction
5	Inappropriate knee flexion values	Initial contact Too much: ≥26° Too little: ≤19°
		Midstance Too much: ≥46° Too little: ≤39°
6	Inappropriate trunk rotation	Asymmetry based on visual assessment from posterior view Rotation of the trunk Movement of the arms backward (toward the camera) Movement of the shoulders in the transverse plane
7	Narrow Stepping	Narrow stepping When the landing foot crosses the center of mass line
8	Inadequate Hip Extension	Too much: ≥9° Too little: ≤4°

DF = Dorsiflexion, STJ = Subtalar Joint, PF = Plantarflexion

Data analysis

Median (and range) of licensed physical therapy experience were calculated. Raters were able to select up to three perceived impairments, leading to a possibility of 56 unique combinations of three perceived impairments per case. Each unique combination of up to three was coded into a single number (i.e. unique combination of impairments 1, 5, 6 is coded as combination 1). If a rater did not select three perceived impairments (i.e. only selected 2), then the third impairment would be noted as a selection of “no perceived impairment” since they did not feel there were either any other perceived impairments or they felt any other impairments were not critical to address first.

Percent agreement was calculated for each individual impairment selected for each case. For consistency, the numerator (number of exact agreements between raters) is the number of agreements where the perceived impairment was selected. However, if zero raters selected a specific perceived impairment, percent agreement would be reported as 100%, as all raters agreed that the perceived impairment was not present or clinically meaningful to address. To account for the influence of chance within percent agreement analysis, Fleiss Kappa (κ) at 95% confidence interval (CI), a-priori alpha ≤ 0.05, was calculated to determine the inter-rater agreement for each unique combination of three perceived impairments for each case. Kappa results were rated as, excellent (≥ 0.80), substantial (0.60–0.79), moderate (0.40–0.59) or poor (< 0.40) (Portney and Watkins, 2000). Statistical analyses were performed using STATA 16 (StataCorp. 2019. Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC).

Results

The 12 included physical therapists had a median of 51 months (18–156) duration of licensed experience. In case one, percent agreement between physical therapist raters ranged from 8%-100%, with 50% of the impairments having >80% agreement (Table 4). Inappropriate ankle subtalar joint eversion with landing (impairment 3) and inappropriate trunk rotation (impairment 6) were identified as not present or not clinically important to address by all 12 raters (100% agreement). Inappropriate knee flexion (impairment 5) was identified as present and clinically important by all 12 raters (100% agreement).

Table 4. Inter-rater agreement per perceived impairment in case one.

Impairment	Total Selections (n)	% agreement
1	10	83
2	2	17
3	0	100
4	5	42
5	12	100
6	0	100
7	1	8
8	4	33

In case two (Table 5), percent agreement ranged from 8%-100% with only 25% of the impairments having >80% agreement. Trendelenburg gait (impairment 1) was identified as present and clinically important by 11 raters (92% agreement) and inappropriate ankle dorsiflexion at landing (impairment 2) was identified as not present or not clinically important by all 12 raters (100% agreement).

Table 5. Inter-rater agreement per perceived impairment in case two.

Impairment	Total Selections (n)	% agreement
1	11	92
2	0	100
3	3	25
4	7	58
5	4	33
6	3	25
7	1	8
8	5	42

In case one, ten of twelve raters selected three perceived impairments, with two raters selecting two. Of 56 possible unique combinations of three perceived impairments, six were selected. Two unique combinations were selected at the same frequency (n = 4 each). Between rater agreement when selecting a unique combination of three impairments was κ = −0.09, p = .92. In case two, ten of twelve raters selected three perceived impairments, with two raters selecting two. Of 56 possible unique combinations of three perceived impairments, nine were selected. Three unique combinations were selected at the same frequency (n = 2 each). Between rater agreement when selecting a unique combination of three impairments was κ = −0.09, p = .98.

Discussion

This study was a secondary analysis of a dataset from a video running gait analysis reliability study. The purpose of this study was to examine the inter-rater agreement on individual impairment selection and the unique combination of three perceived impairments in two cases of runners with knee pain. Our results suggest that raters tend to have better agreement when selecting a single perceived impairment compared to their unique combination of multiple impairments. This study is the first attempt to understand the agreement between physical therapists when identifying what they perceive as clinically important impairments to address in runners with knee pain.

We hypothesized that raters would have a moderate level of agreement; however, our results suggest there may be much higher variability and inconsistency between raters despite having the same clinical data to interpret. The between rater percent agreement for any one individual impairment ranged from poor to excellent (8%-100%) across both cases. However, it should be noted that percent agreement can be interpreted in both directions and we used the numerator as the number of “yes” votes only for consistency in interpretation. The inconsistencies observed in the current study share similarities with reported inter-rater agreement values for body posture assessment and alignment measurement using video running gait analysis on injured runners (κ = 0.00–0.85) (Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016). This means that raters were inconsistent in their agreement on the categorization of the runner’s body alignment (i.e. excessive, mild, or appropriate trunk side bending), which could be viewed as classifying potential impairments. Our analysis further adds that physical therapist’s ability to interpret and prioritize perceived impairments identified from a comprehensive physical examination is inconsistent.

Of specific interest, raters demonstrated both low agreement for identifying knee position (valgus/varus/neutral) during the video running assessment (κ = 0.15) and for selecting knee valgus/varus as a perceived impairment (i.e. both cases, percent agreement = 42% and 58%). A considerable amount of literature has focused on the potential clinical importance of knee valgus as a possible cause of, or contributory factor in knee pain, especially patellofemoral pain (Graci and Salsich, 2015; Holden, Boreham, Doherty, and Delahunt, 2017). Within the case narrative, both runners demonstrated valgus during dynamic tasks (i.e. single-leg squat and single-leg hop). However, a recent meta-analysis determined that there were 34 variations of the ‘single-leg squat test,’ with test performance and scoring methodology differing considerably amongst the 34 variants (Ressman, Grooten, and Rasmussen Barr, 2019). Each different scoring criteria will place more weight on different aspects of the test (pain reproduction vs. mechanics vs. side-to-side comparison). Within the current study, the interpretation of the case narrative was based on the individual bias of the raters. It may be inferred from the results that raters have different perceptions as to what constitutes valgus (visually) and its clinical relevance as an impairment in runners with knee pain (Simon, Parizek, Earl-Boehm, and Bazett-Jones, 2018).

An important confounding factor that could influence the current study’s results are the observed inter-rater joint angle measurement reliability and the differences in reliability between right and left sides (Table 2a,b). Inter-rater agreement ranged from poor to fair. It is possible that poor to fair agreement for kinematic assessment while running heavily influenced the raters perceived impairment choices. The raters in the current study had to determine the frame to use and then measure the joint angles from the self-selected frame compared to a recent study that used the average angles across seven consecutive steps (Dingenen et al., 2019). This methodological difference may have contributed to the reduced inter-rater agreement (Dingenen et al., 2019). The raters in the current study could have struggled more with identifying the frame to measure, which would have resulted in an error in perspective, differing joint angle measurements, and perceptions of a suspected impairment. The number of raters in the current study (12) could also have impacted agreement values compared to the number of raters used in most of the literature (Dingenen et al., 2019; Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016).

Further, the joint angles chosen for measurement in the current study may have contributed to the lower inter-rater reliability. Transverse plane motion (ankle eversion) is challenging to measure using two-dimensional video systems (McGinley, Baker, Wolfe, and Morris, 2009). Additionally, identifying knee position (valgus, varus, neutral) from a posterior view may have led to an altered perspective compared to an anterior view. Measuring hip adduction angle without internal rotation may explain the conflicting inter-rater agreement observed.

Raters in the present study were asked to “select up to three impairments that you would want to address at their next session.” This was to force the raters to prioritize their clinical thoughts although we did not ask them to rank the impairments. We found a non-statistically significant overall agreement that is worse than what would be expected by chance for both cases (κ = −0.09, p = .92, and κ = −0.09, p = .98). The frequency of selected impairments across cases may explain these findings (Tables 4 and 5). Two impairments (i.e. Trendelenburg gait and inappropriate knee flexion) were selected by 83% and 100% of raters in case one, with four other impairments receiving votes. This suggests that raters agreed on at least two of the three impairments, but the third was more difficult to determine. In case two, 7 of 8 perceived impairments were selected, with only one (impairment 1) over 90% agreement. In the context of a runner who is seeking care due to a knee injury, they may be provided a different treatment approach by each clinician they may see. Treatment variability is an inherent positive aspect of physical therapy, as there will always be more than one exercise or technique that can assist in resolving a patient’s impairments. However, previous research would caution us that treatment variability may not always follow best practice guidelines (Zadro, O’Keeffe, and Maher, 2019). Perpetual inconsistency of agreement between clinicians on what are the perceived clinically important impairments to address during physical therapy may lead to unnecessary practice variation, overtreatment, and increased financial burden on our healthcare system (Flynn, Smith, and Chou, 2011; Lin et al., 2018; Naylor, Hart, Harris, and Lewin, 2019; Oatis et al., 2014; Traeger, Moynihan, and Maher, 2017; Zadro, O’Keeffe, and Maher, 2019).

Limitations

There are several limitations to note when interpreting the findings from this study. Further, the raters did not perform the clinical exam themselves and there may have been additional testing they would have done to further understand the patient’s impairments. However, all of the 12 raters used the same data to generate their responses. This was necessary to assess agreement accurately. The use of standardized information allowed us to have a high number of raters (12) which increases the generalizability of our results. The clinical examination was not performed in real-time with raters observing a live patient. This eliminated the raters’ ability to understand any contextual factors of the patient besides what was included in the case narrative. However, this may have created an additional source of bias that could increase the variance of agreement. We only included two cases for raters to assess. It is possible that the addition of a third case would have assisted in identifying a clearer trend between rater agreement. The provided list of impairments is not all-encompassing of the totality of factors that could lead to a RRI (at any joint) and may introduce the biases of the research team into the results. Future studies should include an enhanced list that covers possible kinematic, spatio-temporal and subjective/training history factors should be included. It is also important to build the list of impairments with clinician partners to ensure appropriate phrasing for all to interpret and to not constrain the number of impairment choices. Finally, the between rater reliability of joint angle measurements during the video running gait analysis ranged from poor to fair and may have been influenced by the iPad’s recording speed (30 frames/second, Appendix C) or the number of raters. We recognize that the recording speed of 30 frames/second is lower than what is currently recommended, however, this was only one data source that the raters used to make their decisions (Souza, 2016).

Future research

Future research is necessary to build on the methods used in this preliminary analysis. Specifically, the same orthopedic exam findings (encompassing subjective, objective, and video running gait analysis findings) should be provided to the physical therapist raters to judge. Information should be collected on why the physical therapist chooses the perceived impairment and what specific findings led them to that conclusion to further understand how physical therapists prioritize clinical decisions based on clinical exam findings. An additional question examining what clinicians believe compared to what is reported in the literature regarding impairments may be interesting to understand. This study should be repeated with runners of different skill levels (e.g. recreational and elite) to identify if this influences clinical decisions, with runners experiencing different involved joints, and different patient populations (e.g. shoulder pain and back pain). Last, the examination of clinical decision-making and agreement in real-time would be important to better discern variations in clinical decision-making.

Conclusion

Our preliminary analysis reveals that the physical therapist raters demonstrated poor to excellent levels of agreement when selecting individual impairments based on a comprehensive clinical orthopedic examination for two cases of runners with knee pain. However, between rater agreement was worse than chance when selecting a combination of impairments.

Acknowledgments

We would like to thank the physical therapists who participated in this study.

Disclosure statement

We report no potential conflicts of interest.

Appendix A.

Case one

The patient is a 19-year-old female who reports a five-month history of bilateral (B/L) knee pain (right knee is more painful than left). She has a small right knee medial meniscus tear found with recent magnetic resonance imaging but also has persistent left patellar pain. Pain is 1–2/10 on average but can increase to 6/10 on the right with running more than 1 mile. Goals are to decrease B/L knee pain for return to half marathon training, which she signed up for her first half marathon. It will occur in 5 months.

Objective Findings

Gait: Hard impact, moderate pronation B/L

Single-Leg Squat: knee valgus and femoral internal rotation B/L with loss of balance

Single-Leg Hop: knee valgus and femoral internal rotation B/L, pain in left knee only and only able to perform one repetition

Manual Muscle Testing:

Hip Flexion: 5/5 B/L

Hip Abduction: 4/5 B/L

Hip Adduction: 5/5 B/L

Hip External rotation: 4/5 B/L

Hip Internal Rotation: 5/5 B/L

Hip Extension: Right 4/5, Left 4/5

Cadence: 156 steps/minute

Louder Landing Volume with Left lower extremity on a treadmill

University of Wisconsin Running Injury and Recovery Index: 19/36

Lower Extremity Functional Scale: 64/80

Appendix B.

Case two

The patient is a 24-year-old female with right lateral knee pain (medical diagnosis of right iliotibial band pain) that began 3-months ago after running a 10 K event without appropriate training per her report. Now, she can run 10–15 minutes before pain onset. Pain is a 0/10 at rest and 6/10 with running.

Objective Findings

Bilateral (B/L) Hip and Knee range of motion is within normal limits, pain-free

Double Limb Squat: B/L anterior knee translation and valgus. No pain

Single Limb Squat: Increased B/L anterior knee translation and valgus

Positive Obers Ober’s on right lower extremity

Manual Muscle Testing:

Hip Flexion: Right 4/5, Left 4+/5

Hip Abduction: Right 4+/5, Left 5/5

Hip Adduction: 5/5 B/L

Hip External rotation: 4+/5 B/L

Hip Internal rotation: 5/5 B/L

Knee Extension: 5/5 B/L

Knee Flexion: 5/5 B/L

Cadence: 172 steps/minute

Louder Landing Volume with Left lower extremity on the treadmill

University of Wisconsin Running Injury and Recovery Index: 17/36

Lower Extremity Functional Scale: 70/80

Appendix C.

Two dimensional-video running gait analysis standardized protocol

Video running gait analysis used a standardized protocol. The patient had colored flat stickers placed over a set of standardized anatomic landmarks that were obtained through examiner palpation (Sinclair et al., 2012). Landmarks included: bilateral posterior sacral iliac spine, posterior knee joint lines at the midline of the knee, bilateral greater trochanters, and bilateral lateral knee joint line (Sinclair et al., 2012). These were chosen based on the best currently available evidence (McGinley, Baker, Wolfe, and Morris, 2009; Sinclair et al., 2012). Markers were placed on bare skin when able. However, to maintain patient modesty in the clinical environment, markers on the sacrum and greater trochanters were placed over formfitting clothing. After visual inspection of the applied stickers, the runner was asked to step onto a treadmill (WOODWAY USA, Waukesha, WI) and was given five minutes to achieve a self-selected running pace at which they felt comfortable and was similar to their usual training pace (Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Souza, 2016). The runner would signal that they have reached their desired running pace and video recording began at that time.

The video was recorded using an iPad 2 (2011, Apple Inc, Cupertino, CA), which was held by the administering clinician to record the runner from both a posterior and lateral view. The lateral view was chosen based on the side of the pathology and limitations of the treadmill (obstructed view). The iPad distance was placed 10-feet away from the treadmill in each direction. This distance was chosen to ensure a full view of the runner’s entire body within the video frame. Each view was recorded for a duration of 6–10 seconds (Figure 1a,b) at 30 frames per second (Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016). This duration of time has been shown to capture variability in a person’s running gait (Pipkin, Kotecki, Hetzel, and Heiderscheit, 2016; Souza, 2016). This was the fastest frame rate of the iPad instruments at the time of the study.