Low reproducibility of randomized clinical trials methodology related to sampling: a systematic methodological review

ABSTRACT

Objectives: The reporting of sampling methods in Randomized Clinical Trials (RCTs) allows for research quality assessment, determination of sampling bias, and assures the presence of details necessary for reproducibility in future trials. The purpose of this study was to: (1) determine if sampling methodology was reproducible in RCTs related to musculoskeletal physical therapy (MSKPT) interventions to treat non-specific low back pain (NSLBP) and (2) establish if there was a relationship between sample reproducibility and established measures of research quality.

Methods: Data were collected through a systematic review by a professional librarian. The identified RCTs were assessed for methodological quality by two blinded individual reviewers. Data analysis was performed by a third, blinded researcher; additional comparisons were made based on Journal Impact Factor and PEDro score.

Results: Ninety-nine published peer-reviewed RCTs were identified that met inclusion criteria. Only 29% of the articles were judged to be reproducible based on the reported sampling methodology. There were meaningful correlations between two out of ten of the sampling reporting criteria and the judgement made if the sample was reported in significant detail to allow for replication. There was no relationship between sampling reporting criteria, Journal Impact Factors (JIFs), and Physiotherapy Evidence Database (PEDro) scores.

Discussion: The reporting of sampling methodology needs to be considered to ensure reproducibility and avoid sampling bias. Despite the proliferation of measures of research quality, the overall reporting quality of RCTs continues to be inadequate to allow widespread reproducibility of trials.

Level of Evidence: 1a

KEYWORDS:

• Journal Impact Factor
• reproducibility of results
• selection bias

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

No funding was provided for the development of this manuscript. I certify that neither I nor the co-investigators on this research project have any conflicts of interest related to this work.

Notes on contributors

Sean P. Riley

Sean P. Riley, PT, DPT, ScD is an Assistant Professor in the Physical Therapy program at Sacred Heart University in Fairfield, CT, USAand faculty in the Sacred Heart University Orthopedic Physical Therapy Residency program. He graduated from the UConn Physical Therapy Program in 2004 and completed his Doctorate in Physical Therapy at Simmons College in 2006. He completed his Doctor of Science from Texas Tech University Health Science Center in 2013. Sean became board certified in Orthopedics through the American Board of Physical Therapy Specialties in 2007 and completed his sports certification in 2009. He completed his Fellowship in Manual Therapy from Regis University in 2018.

Brian T. Swanson

Brian T. Swanson, PT, DSc is an Assistant Professor of Physical Therapy in the Department of Rehabilitation Sciences at the University of Hartford in West Hartford, CT, USA and faculty in the Select Medical Orthopedic Physical Therapy Residency program. Dr. Swanson graduated from Quinnipiac University with a Bachelor of Science degree in Physical Therapy in 1996 and earned a Doctorate of Science in Physical Therapy from Andrews University in 2012. Brian became board certified in Orthopedics through the American Board of Physical Therapy Specialties in 2005. He completed two-year residency and Fellowship in the Kaltenborn-Evjenth Norwegian Approach to Orthopaedic Manual Physical Therapy at the Institute of Orthopaedic Manual Therapy, Woburn, MA in 2006.

Jean-Michel Brismée

Jean-Michel Brismée, PT, ScD is professor in the Doctor of Science Program in Physical Therapy at Texas Tech University Health Sciences Center in Lubbock, Texas. Dr. Brismée graduated from the Catholic University of Louvain-la-Neuve in Belgium with Bachelor of Science degrees in Physical Education and Physical Therapy in 1985. He graduated from Texas Tech University in 1996 with a Master of Sciences in Sports Health and earned a Doctor of Science degree in Physical Therapy in 2003. He is the Chair of the Research Committee and Director of the Fellowship program in Orthopaedic Manual Therapy of the IAOM-US, Editor-in-Chief of the Journal of Manual and Manipulative Therapy, and maintains clinical practical at University Medical Center in Lubbock, Texas.

Steven F. Sawyer

Steven F. Sawyer, PT, PhD, is professor and chair of the Department of Rehabilitation Sciences, Associate Dean for Faculty Development, and director of the Clinical Postural Control Research Laboratory in the Center for Rehabilitation Research, all within the School of Health Professions at Texas Tech University Health Sciences Center. His PhD in Neuroscience was earned from the University of California at San Diego in 1989. He graduated from PT school at the Texas Tech University Health Sciences Center in 1997, and has been on the faculty since then. He received certification in Vestibular Rehabilitation from Emory University in 2001.

Elizabeth J. Dyer

Elizabeth J. Dyer is Associate Dean of Library Services and a Research and Teaching Librarian at the University of New England in Portland, Maine where she liaisons with the Doctor of Physical Therapy program. She has a BA from Cornell University, an MLIS from University of South Carolina, and is a Senior Member of the Medical Library Association's Academy of Health Information Professionals (AHIP). She attended the University of Pittsburgh’s Systematic Review Workshop in 2016 and was a 2015 fellow at the National Library of Medicine’s Biomedical Informatics Course. She is Chair-Elect of the North Atlantic Health Sciences Libraries chapter of the MLA.