‘Trustworthy’ systematic reviews can only result in meaningful conclusions if the quality of randomized clinical trials and the certainty of evidence improves: an update on the ‘trustworthy’ living systematic review project

What is the ‘trustworthy’ living systematic review project?

The ‘trustworthy’ living systematic review (SR) project began in 2022 with the ambitious goal of synthesizing verifiable moderate- to high-quality certainty evidence that could be confidently translated into strong clinical practice recommendations. Relying upon this type of evidence: 1) prevents strongly discordant clinical recommendations based totally or partially on low-quality evidence [1]; 2) breaks the cycle of the grossly overused phrase, ‘These results, however, should be interpreted with caution [2]’; and 3) builds a foundation for synthesizing living, ‘trustworthy’ research for the practicing clinician that can be periodically updated [3]. Our methods involved establishing the prospective validity of the identified randomized clinical trials (RCTs) (i.e. ensuring the studies were prospectively registered, conducted, and reported consistent with the registry). Establishing prospective validity through a prospective registry is critical as authors fail to follow the established rules after collecting the data and obtaining a result 73% of the time [2]; this is a form of potentially inappropriate research behavior known as post-randomization bias [4].

Post-randomization biases create an environment where researchers report what they create from the data instead of how the methods and the data answer the research question(s) of interest [4]. Research questions should be asked a priori, not once data collection is underway or completed. Likewise, the data analysis plan is connected to the testable hypotheses established at the outset and, therefore, cannot be a ‘wait and see’ response to the data collected. These post-hoc modifications to prospective intent are often used intentionally or non-intentionally to create type I research errors (i.e. false positive findings) where deviating from the game’s rules after playing the game creates statistically significant findings where none exist [5]. One of the side effects of these behaviors is that they render the tools used to assess the quality of RCTs, certainty of RCTs, confidence in SRs, and clinical applicability useless because they are predicated on the assumption of prospective validity [6].

Additionally, our protocol ensured that the studies were externally valid (i.e. the inclusion and exclusion criteria were identified, at least moderately internally valid with a Physiotherapy Evidence Database (PEDro) score of 6 or higher [7], and had a moderate- to low-risk of bias on the Cochrane Collaboration Risk of Bias tool 2 (RoB 2 tool) [3].

What did we find?

After completing three ‘trustworthy’ SRs using the parameters we outlined in our published protocol [7], we identified [8] a single ‘trustworthy’ RCT [9] guiding the use of manual therapy for treating patients with non-radicular cervical spine impairments and identified three trustworthy RCTs [9–11] when investigating the effects of manual therapy on quantitative sensory testing and patient-reported outcome measures in participants with musculoskeletal impairments [12]. Our most recent effort reviewed RCTs investigating the use of manual therapy to treat patients with shoulder dysfunction [13]. Unfortunately, we could not identify any ‘trustworthy’ RCTs after evaluating prospective validity, external validity, internal validity, and risk of bias, which prevented a further assessment regarding the certainty of the evidence using the GRADE criteria [13].

What are the GRADE criteria, and how are they used to establish the certainty of randomized clinical trials?

The protocol of the ‘trustworthy’ living SR project [3] was designed to ensure that downgrading did not occur as the RCT data were synthesized into the certainty of the observed effect sizes and practice recommendations using the GRADE criteria. The GRADE criteria are used to establish if the certainty of the synthesized effect is accurate. Although a thorough discussion of the GRADE criteria is beyond the scope of this paper, further reading can be found in the Evidence-based medicine (EBM) toolkit the British Journal of Sports Medicine created [14].

As the GRADE criteria are applied, the certainty of the evidence starts high if all included studies in the SR are RCTs [15]. This certainty may then be downgraded by assessing the included studies for: 1) risk of bias on the RoB 2; 2) imprecision (accuracy of the 95% confidence interval (CI)); 3) inconsistency (i.e. overlapping CIs or heterogeneity that crosses zero in the meta-analysis, a.k.a. heterogeneity); 4) indirectness (i.e. patients or outcomes are different than the recommendation made in the RCTs); and 5) publication bias (i.e. assessed through statistical methods).

Why is it essential not to synthesize RCTs that result in downgrading?

GRADE certainty ratings can be found in Table 1 [14].

Table 1. GRADE certainty ratings.

Certainty	What it means
Very low	The true effect is probably markedly different from the estimated effect.
Low	The true effect might be markedly different from the estimated effect.
Moderate	The authors believe the true effect is probably close to the estimated effect.
High	The authors have a lot of confidence that the true effect is similar to the estimated effect.

The certainty ratings mean that downgrading the certainty of the data based on any two or one of the five above criteria creates very-low to low-certainty of the synthesized RCTs on the GRADE. This translates into an observed effect that is probably or might be wrong and ‘ … should be interpreted with caution [2]. ’ The question then becomes, should this level of certainty in the RCT evidence be translated into any clinical practice recommendations when synthesized in an SR? When this happens, we cannot assume that an intervention is or is not effective [16]. When reliable evidence does not exist, the absence of this evidence cannot be translated into any discordant clinical practice recommendations [1]. A conclusion based on this absence of reliable evidence cannot provide data-driven evidence of any effect [16]. The most honest interpretation when this occurs should be not to change clinical practice by not recommending the treatment [16].

What do we know about the methodological quality and confidence in the conclusions of physical therapy SRs?

A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR 2) is a critical appraisal tool used to assess SRs that use RCTs of healthcare interventions [17]. The AMSTAR 2 has seven critical domains. These domains are: Protocol registered before the commencement of the review (item 2); Adequacy of the literature search (item 4); Justification for excluding individual studies (item 7); Risk of bias from individual studies being included in the review (item 9); Appropriateness of meta-analytical methods (item 11); Consideration of risk of bias when interpreting the results of the review (item 13); and Assessment of presence and likely impact of publication bias (item 15) [17]. Table 2 includes the criteria for how the AMSTAR 2 is rated.

Table 2. Rating overall confidence in the results of the review [17].

Rating	How the rating is determined	How the rating should be interpreted
Critically low	More than one critical flaw, with or without non-critical weaknesses	The review has more than one critical flaw and should not be relied on
Low	One critical flaw with or without non-critical weaknesses	The review has a critical flaw and may not provide an accurate and comprehensive summary of the available studies that address the question of interest
Moderate	More than one non-critical weakness*	The systematic review has more than one weakness but no critical flaws. It may provide an accurate summary of the results of the available studies that were included in the review
High	No or one non-critical weakness	The systematic review provides an accurate and comprehensive summary of the results of the available studies that address the question of interest

*Multiple non-critical weaknesses may diminish confidence in the review, and it may be appropriate to move the overall appraisal down from moderate to low confidence.

Beyond the certainty of the synthesized RCTs included within SRs, the SR’s methodological quality impacts how confident clinicians can be in the SR’s conclusions. When the confidence in the SR results is critically-low or low, the interpretation of the SR’s findings is that the SR should not be relied on and may not provide an accurate and comprehensive summary of the available studies that address the question of interest, respectively. Again, this situation is where the synthesis of RCTs using SR methodology creates unreliable evidence. When reliable evidence does not exist, the most honest interpretation is that this evidence cannot be translated into clinical practice recommendations [16]. Again, the most honest interpretation when this occurs should be not to change clinical practice by not recommending the treatment [16].

What have we learned about heterogeneity (variability) in published research?

The GRADE criteria include imprecision and inconsistency, which involve the accuracy of the 95% CI and discuss the overlapping of CIs in RCTs or heterogeneity that crosses zero in the meta-analysis, respectively, as reasons to downgrade the certainty of the RCT evidence. Both criteria are related to the statistical heterogeneity or accuracy of the data. It is known that the false positive rate of p-values is approximately 30% [18]. Researchers, therefore, should assess the size of the average treatment effect (ATE) (i.e. is it large enough to be clinically meaningful) and the variability of the effect [5]. The heterogeneity measures inform the reader of the variability of ATE and provide a range of where that ATE could land if the RCT or SR were repeated. The standard measure of heterogeneity in RCTs is the 95% CI, and the most frequently used measures of heterogeneity in SRs include the I² statistic and the prediction interval. High levels of statistical heterogeneity are the primary factor that limits generalizability [19] and the clinical relevance of research findings.

It is known that if the 95% CI overlaps when assessing differences between and within groups, although statistically significant findings may be present and the effect size may be clinically meaningful, the accuracy of that finding is not precise enough to rely upon. This is a strong indicator that if the study was repeated, the effect size obtained could land anywhere within the CI, challenging the clinical utility and reproducibility of the study’s findings.

SR heterogeneity is commonly expressed via the I² statistic. The I² statistic assesses the variability between the individual study’s effects and the synthesized effect of the meta-analysis [20]. The I² is reported as a percentage, with 25% considered low, 50% considered moderate, and 75% considered high [21]. Heterogeneity is expected when comparing results across studies [19], but strict guidelines on meta-analysis based solely on heterogeneity using the I² statistic are lacking [22]. It has been suggested that the clinician consumer determines if the heterogeneity levels in a meta-analysis are appropriate [22]. This is problematic on several levels.

1. A meta-analysis should be a prospectively registered decision. A meta-analysis should not be performed if the data do not meet this prospectively derived heterogeneity threshold.

2. The I² as a measure of heterogeneity is a mathematical point estimate that should be reported with the 95% CI so that the reader can determine the estimate’s accuracy. This is a problem in published SRs as these 95% CIs are often not reported, and it is known that even I² point estimates of 0% (i.e. suggesting no statistical heterogeneity) can have 95% CIs that range from 0 to 79% [23]. When this occurs, it strongly suggests that the point estimate of heterogeneity and the variability are too variable to be trusted. In other words, the variability strongly suggests that the meta-analysis should not be performed. This is even more challenging considering that the I² can be imprecise and biased in small SRs with few included studies [24], which is often the case. Bias in I² can lead to over- or under-estimating heterogeneity in small SRs, with positive bias present when heterogeneity is truly small and negative bias present when heterogeneity is substantial [24].

3. A more accurate means of expressing heterogeneity is the prediction interval [20,25]. The prediction interval statistically determines the variability across the synthesized studies. Unfortunately, the prediction interval is not reported in most SRs. When reported, they are wide and cross zero even when the 95% CI of the synthesized effect does not [26]. This is important as the prediction interval helps to identify imprecision and inconsistency when applying the GRADE and, in this case, would downgrade the rating on the GRADE from high to low based on these two criteria.

What conclusions can we draw from the certainty of RCTs and confidence in SRs?

In 2023, it was identified that 95% of the SRs published in the International Society of Physiotherapy Journals Editors (ISPJE) member journals indexed in MEDLINE were rated as ‘critically low’ on the AMSTAR 2 [27]. Additionally, it was identified that 87.5% of the ISPJE member journals do not require prospective SR registration [27]. An SR that is not prospectively registered is downgraded on the AMSTAR 2 by one level. These findings grossly erode the evidence base of physical therapy and highlight the urgency of improving the quality of RCTs and SRs. There is no evidence-based practice without our ability to interpret findings accurately [28].

The findings of these ‘trustworthy’ SRs do not suggest that manual therapy is ineffective. The findings indicate that there is an absence of moderate to high certainty evidence that can confidently translate into strong practice recommendations through the SR process. There are similar uncertainty and confidence challenges related to other interventions used by physical therapists related to heterogeneity and conflicts of interest that include: 1) blood flow restricted training [29]; 2) cognitive behavioral therapy [30]; 3) foam rolling [31]; 4) instrument assisted soft tissue mobilization [32]; 5) kinesiotaping [33]; 6) movement system impairments [34,35]; 7) pain neuroscience education [36]; 8) stratified interventions using the STarT Back Screening Tool [37]; 9) therapeutic exercises [38,39]; and 10) trigger point dry needling [40] to name a few. Additionally, these are not challenges unique to physical therapy interventions, as medical interventions also display unknown levels of certainty and confidence in observed effects [41–44].

The findings (or gross lack thereof) signal two potential future paths. First, if the profession fails to respond accordingly by not generating trustworthy published evidence, then there will be no future need to rely upon living SRs or to consider this sector of the profession evidence-based. In contrast, if the profession chooses the alternate path and begins to generate published evidence consistent with a strict adherence to professional and ethical standards of performing, reporting, and publishing research, then the idea of a living SR project can assist with translating future published evidence into clinical practice. Given the current lack of certainty surrounding the evidence, publishers, editors, and authors should be scrutinized for appropriate clinical practice recommendations based on RCTs and the SRs that synthesize them in the published research. Again, an honest interpretation of critically low to low certainty RCT evidence synthesized into SRs with critically low to low confidence should not be used to make clinical recommendations as no reliable evidence can or should be used to make any decisions. The question then becomes, are we as a profession more interested in making grossly overused, cautious, and discordant practice recommendations, always finding an effect, or forging the difficult path ahead to establish a solid base of high-quality evidence and meaningful practice recommendations?