Why do ‘Trustworthy’ Living Systematic Reviews Matter?

In this issue, the Journal of Manual and Manipulative Therapy is publishing a novel approach to systematic reviews termed ‘living’ systematic reviews (SRs) that continually update and incorporate relevant new evidence as it becomes available [1]. This methodology was first proposed in the medical literature in 2014 [2], with publications in 2015 [3] and the first such publication in the physical therapy-related literature in 2020 [4]. Living SRs methodology includes continual surveillance for new research evidence through ongoing or frequent searches and the timely inclusion of relevant new information into the review so that the SRs’ findings remain current. Information changes, and it is expected to change over time. This methodology can help clinicians stay abreast of the most recent updated information published through the Journal and in the near future with the Duke Center for Excellence in Manual and Manipulative Therapy https://sites.duke.edu/cemmt/.

The median time from a randomized clinical trial (RCT) publication to when it is synthesized into an SR is 2.5 to 6.5 years [5]. When reading SRs, it is easy to determine the time from the last search date until the SR is published, which often can take at least one year [6]. Once published, it can take several years for SRs to be updated [7]. This creates a progressively increasing timeline from when the data are collected, to the primary RCTs publication, the RCTs synthesis into an SR, to the ultimate SR publication and potential clinical practice recommendations. This inability to keep synthesized literature such as SRs up to date may lead to progressive inadequacies of synthesized published research.

Two years after the publication of synthesized research, more than 23% of this literature needs to include new primary research that substantively changes the findings regarding treatment effects and potential harm of interventions [8]. A methodologically rigorous living SR that is updated regularly could be essential in disseminating trustworthy, accurate, up-to-date pre-processed resources for the practicing clinician.

A foundational construct involved in this process is that the trustworthiness of the primary literature must be established for the living SR to live an honest life. Once the foundation of trustworthiness is established, the living SR can be updated informally regularly. When it appears that the accumulation of added research would change the effects or harms of the interventions, the formal protocol can be executed with the results submitted to a peer-reviewed journal to ensure a peer review process is complete before potential publication.

This editorial aims to provide a brief overview of what trustworthiness in published research is, how confidence in that research is created, why this confidence is essential when synthesizing literature in living SRs, and to discuss the responsibilities of researchers, publishers, and consumers of research based on established professional and ethical standards.

What is trustworthiness in research?

Trustworthy published research is required to translate evidence into strong clinical practice recommendations. For example, high confidence in estimated effects is essential when interpreting primary research and creating pre-processed literature such as SRs [9]. However, there has been a progressive evidence-informed narrative on the problem with, and prevalence of, both known and unknown post-randomization biases [10–13]. These biases occur when research is published inconsistently with the prospectively established research record (i.e. inconsistent with a prospectively registered protocol). In addition, there are unknowns related to unregistered, retrospectively registered, and unclearly registered trials, which hinder trustworthiness when establishing confidence in the estimated treatment effects of published research. Unfortunately, despite the documented narrative surrounding these issues, progress toward a resolution has yet to occur.

One challenge may be the total scope of the problem. Since this is not only a challenge with published physical therapy research and everybody is doing it across healthcare professions, the known challenges in consuming published research for the practicing clinician may be considered acceptable [12,14]. An editorial written by Cook and Garcia on ‘Post-randomization bias’ [15] described a definite need for healthcare sciences research to improve. It is clear from their editorial that this issue is not just one of adequate reporting but also one of honest and transparent reporting. A solution proposed by Cook and Garcia is for journals not to accept randomized clinical trials (RCTs) that have not been prospectively registered [15] while also suggesting that readers make this distinction when consuming literature [15]. While it seems reasonable for this to occur during the peer-review process, we would ask, is it fair or ethical for a journal to expect clinicians to determine if the trial was registered, registered prospectively, and if the results were reported consistent with the author’s prospective research question(s)?

How are trustworthiness and confidence in published research established?

Trustworthiness and confidence in estimated treatment effects start with a prospective clinical trial registry. The prospective registry allows publishers and consumers of research the ability to verify that the research findings and interpretation were consistent with the prospectively established research question(s), using a prespecified outcome measure when reporting the results, discussion, and conclusions of the manuscript. In addition, adhering to prospectively registered intent minimizes post-randomization biases. While discussed in detail elsewhere, these biases occur by generating a hypothesis after the results are known, reporting and discussing data that support a hypothesis, running statistical analyses until statistical significance is found, and searching for associations between variables just because the data exist [16]. Finally, the confidence in the estimated effect is established through statistical significance, a clinically meaningful estimated effect, and by ensuring the precision of the effect through 95% confidence intervals [13] in the presence of prospective registration and reporting that matches the published protocol.

Why is trustworthiness in the research necessary, and who is professionally responsible?

Evidence-based practice is a process that starts with a clinical question. Its application is a blend of the best available evidence with the clinician’s experience and the patient’s values [17]. The process is assessed based on the patient’s response and repeated [18]. If you can’t trust the evidence, you can’t trust the process.

In 2005 Alan Jette wrote an editorial titled, ‘Without Scientific Integrity, There Can Be No Evidence Base [19], ’ a concept that is echoed in the American Physical Therapy Association (APTA) Position Statement on Research [20]. The position statement describes research integrity as using ‘honest and verifiable methods’ when proposing, performing, and evaluating research results while adhering to rules, regulations, and guidelines while following commonly accepted professional codes or norms [21].

Researchers, publishers, and consumers of research should expect prospective registration to be the norm in RCTs. The CONsolidated Standards Of Reporting Trials (CONSORT) statement lists clinical trial registration names and numbers as a requirement of this reporting guideline [22]. As of 1 July 2005, the International Committee of Medical Journal Editors (ICMJE) considers prospective clinical trial registration in a publicly available source as a requirement for RCTs [23]. As of June 2007, the ICMJE adopted the World Health Organization’s (WHO’s) definition of a clinical trial as ‘any research study that prospectively assigns human participants or groups to one or more health-related interventions to evaluate the effects on health outcomes.’ [23] The ICMJE also reports that retrospective trial registration serves neither the intent nor purpose of prospective registration [23]. The Journal of Manual and Manipulative Therapy adopted prospective trial registration guidelines for the submission of manuscripts in 2015 in an attempt to raise the quality of the submitted manuscripts (https://www.tandfonline.com/action/authorSubmission?show=instructions&journalCode=yjmt20).

Clinical trial registration was initiated with the intent to help achieve transparency in research reporting [24], and it makes more sense to verify a manuscript’s adherence to professional practices once (i.e. during review) than it does to do so repeatedly (i.e. each reader independently). Unfortunately, it is known that this is not presently happening for many International Society of Physiotherapy Journal Editors (ISPJE) member journals. Of note, prospective registration should allow reviewers to understand whether researchers were honest and thorough in their data reporting, which helps determine whether or not post-randomization bias occurred. However, this review activity is often lost in preference for the anonymity of the double-blinded review. Therefore, having someone perform this task during the review process is essential. Post-randomization behaviors such as the selective reporting of results, unreported data exclusions, and other eliminations or modifications of experimental intent [25] may have as significant an impact as more blatant misconduct, including plagiarism and intentional falsification [26]. The pervasive nature of these problems within scientific publications has been noted repeatedly [24,27–29]. The physical therapy community should seek to avoid perpetuating these scientific shortcomings and ensure the prospective fidelity of RCTs and the SRs that synthesize them with their prospective intent. Journals should follow their established professional and ethical standards, provide training to staff, editors, and reviewers on what those professional and ethical standards are, and assess this process to ensure that what is being done is consistent with those professional and ethical standards. This scenario requires researchers to be aware of the standards prior to commencing research projects so that they generate research that adheres to contemporary standards.

How are we doing with research integrity?

There is an ongoing conversation regarding research trustworthiness outside and within the physical therapy profession. In 2016, Nature published a paper titled ‘1,500 scientists lift the lid on reproducibility.’ [30] The paper identified that 70% of published research might not be reproducible, 52% of those surveyed believed this is a crisis, yet almost 31% believed there is nothing wrong with the published research [30]. While the true prevalence of scientific misconduct is unknown, methodological issues are a problem, and the publication process appears to be more interested in new findings than in verifying previous findings [31]. As a result, it has been suggested that published research is flat-out ‘untrustworthy’ [31]. In the physical therapy research published in ISPJE journals, less than 1/3 of RCTs published could be confirmed as following their prospective question(s) [10]. Additionally, the number of verifiable RCTs contained in SRs has been identified as a lackluster 9% [11]. A 2023 study identified that 61.8% of prospectively registered physical therapy RCTs were published inconsistently with the established research record, suggesting that the rate of post-randomization bias was much higher than previously anticipated in physical therapy research [12].

What is our professional responsibility as physical therapy researchers, publishers, and consumers?

The professional accountability of researchers, publishers, and consumers is quite clear if we follow the National Institutes of Health standards regarding falsification of research data and misconduct [21], the APTA Position Statement on Research Conduct [20], the Committee on Publication Ethics (COPE) guidelines for ethical publication [32], and the ICMJE standards regarding prospective clinical trial registration [33] and handling of post-publication challenges [34].

The first step in solving any problem is to recognize that it is a problem. Post-randomization biases are a known crisis within and outside the physical therapy profession that: (1) threatens the prospective validity of research; (2) challenges the validity of tools used to assess literature quality; (3) endangers the reproducibility of published research; (4) creates unwarranted variation in clinical practice; and (5) wastes valuable time and resources. It is time to stop the false narrative that ‘everybody is doing it’; therefore, it does not warrant significant immediate attention. This, by definition, is a bandwagon logical fallacy. Independent of your beliefs, ‘Right is right if no one is doing it; wrong is wrong even if everybody is doing it’- Saint Augustine.

Second, authors should follow the APTA Position Statement on Research integrity as using ‘honest and verifiable methods’ when proposing, performing, and evaluating research results while adhering to rules, regulations, and guidelines while following commonly accepted professional codes or norms [20,21]. Prevention of post-randomization biases can be facilitated by ensuring that all research involving interventions to human participants is prospectively registered, verified, and published consistent with the ICMJE guidelines. Senior researchers should model these behaviors to positively influence doctoral candidates, post-docs, and junior faculty beginning their research careers.

Third, it is essential to realize that when an RCT is published inconsistent with the established research record, this may be considered a falsification of research data and misconduct based on the National Institute of Health definition [35]. These inconsistencies, however, have essential caveats. First, they do not include honest mistakes or differences of opinion [35]. It is all right to make mistakes when reporting research findings. However, these mistakes warrant clarification, correction, and/or potential retraction when identified according to ICMJE standards [34,36].

Lastly, it is OK to criticize questionable research. There is nothing wrong with seeking clarification and potential correction. Our professional guidance suggests that it is our ethical and professional responsibility.

Conclusion

Many authors may have knowingly or unknowingly introduced post-randomization bias into their work as they sought publication [26]. This may be motivated by the pressures to publish positive results, and it may be unreasonable to expect this pressure to subside anytime soon. Ideally, authors should be confident in submitting null results obtained through rigorous, trustworthy methodologies for publication. Unfortunately, statistical significance is not equivalent to impact. We firmly believe that any journal is only as strong as the submitted manuscripts it receives and the reviews they undergo. Above, we call upon authors to be rigorous in their processes and fully transparent in their reporting.

Additionally, we implore reviewers to be equally rigorous when considering submitted works and for journals to structure the review process in such a way as to maximize the likelihood that all published papers are trustworthy. Journal editors must act as the first step in assessing the prospective fidelity of a manuscript. The double-blind wall of peer review does not allow reviewers to accurately evaluate a manuscript relative to the prospective research record. Quite simply, it is not enough for a journal to require prospective registration if no one checks the adherence to the registry. Ultimately, it is unfair for journals to expect their clinician readers to wade through the muddy waters of questionable clinical research. As a profession, let’s make sure they don’t have to.

Disclosure statement

The primary author Sean P. Riley is an Deputy Editor at the Journal of Manual and Manipulative Therapy (JMMT). He also contributes to the Duke Center for Excellence in Manual and Manipulative Therapy. Chad E Cook is funded by the NIH, the Director of the Duke Center for Excellence in Manual and Manipulative Therapy, and the APTA Foundation. Dr. Cook also is a consultant for Zimmer, Hawkins Foundation, and Revenite. Jean-Michel Brismée is the Editor in Chief of JMMT. The authors otherwise report no competing interests to declare.

Additional information

Notes on contributors

Sean P. Riley

Dr. Sean P. Riley is an Assistant Professor in the Doctor of Physical Therapy Program and a faculty member in the orthopaedic physical therapy residency at the University of Hartford. He is board certified in orthopaedics and a Fellow of the American Academy of Orthopaedic Manual Physical Therapists. Dr. Riley’s research interests include symptom modification approaches to evaluating and treating neuromusculoskeletal disorders, evidence-based practice, research methodology, and clinical reasoning.

Brian T. Swanson

Dr.Brian T.Swanson is an Associate Professor at the University of Hartford. He serves as Chair of the Department of Rehabilitation Sciences, Director of the Doctor of Physical Therapy program, and co-director of the University of Hartford/HHCRN orthopedic physical therapy residency program. He is board certified in orthopaedics and a Fellow of the American Academy of Orthopaedic Manual Physical Therapists. Dr. Swanson’s research interests include validating tests and measures in orthopedic manual physical therapy, developing a further understanding of the mechanisms of manual physical therapy interventions, and evidence-based practice/research methodology.

Stephen M. Shaffer

Dr. Stephen M. Shaffer is a residency and fellowship-trained clinical specialist, educator, and scientist with nineteen years of experience in the physiotherapy profession. He has worked primarily in orthopaedic settings, is an adjunct professor at the University of Hartford, and is a Fellow of the American Academy of Orthopaedic Manual Physical Therapists and the Canadian Academy of Manipulative Physiotherapy. Dr. Shaffer has co-authored numerous peer-reviewed scientific papers and has presented at local, state, national, and international venues.

Daniel W. Flowers

Dr. Daniel W. Flowers is an Assistant Professor in the Doctor of Physical Therapy and PhD in Rehabilitation Sciences Programs at LSU Health Shreveport. He also serves as program director of the orthopaedic residency. He is board-certified in orthopaedic physical therapy. His research interests include modifying the gait impairments of patients with knee osteoarthritis, post-traumatic rehabilitation, and educational outcomes of physical therapy students.

Chad E. Cook

Dr.Chad E. Cook is a neuromusculoskeletal clinical researcher with foci in diagnosis, interventions, and outcomes assessment. As a productive researcher, he has published over 250 peer-reviewed papers and has an H index of 37. His earlier work involved validating outcome measures, including item response theory. Dr. Cook has completed several studies investigating commonly used tests and measures for orthopedic diagnoses. Recent investigations have included trials designed to investigate the effectiveness of manual therapy-based interventions of the spine and shoulder. His work has led to international recognition and collaborations in numerous countries.Jean-Michel Brismée.

Jean-Michel Brismée

Jean-Michel Brismée, PT, ScD is Professor in the Doctor of Philosophy and Doctor of Science in Rehabilitation Sciences Programs at Texas Tech University Health Sciences Center. Dr. Brismée teaches at TTUHSC in the areas of neuromusculoskeletal care and is involved in clinical research in Orthopaedics, Manual Therapy and Movement Sciences.