Using bench protocol platforms to improve compliance with systematic review guidance documents and reporting checklists: Proof of concept

Abstract

Context

A significant number of guidance documents and reporting checklists have been published to support researchers in planning, doing, and writing up scientifically rigorous systematic reviews (SRs). However, compliance of researchers with SR guidance and reporting checklists remains a significant challenge, with the majority of published SRs lacking in one or more aspects of the rigour of methods and transparency of reporting.

Objective

To explore how bench protocol development platforms might be repurposed for improving compliance of SRs with conduct guidance and reporting checklists.

System design

We developed a proof-of-concept technology stack based around a general-purpose, guidance- and checklist-compliant SR protocol that was built in protocols.io. We used the protocols.io platform to create an integrated research planning and data collection process for planning guidance-compliant SRs. We used our own custom code and the mustache templating language to automatically create checklist-compliant first-draft SR protocol documents in Microsoft Word

Discussion

Creating the operational process for SR protocol planning and the technology stack for automated documentation allowed us to develop our theoretical understanding of how such a system may improve compliance with research conduct and reporting standards. This includes the potential value of algorithmic rather than heuristic approaches to conducting and reporting research studies, positioning of labelled data rather than a study manuscript as the primary product of the research process, and viewing the process of developing research standards as being analogous to the development of open software. Our study also allowed us to identify a number of technological issues that will need to be addressed to enable further development and testing of our proposed approach. These include limitations in templating language, especially when working in Microsoft Word, and the need for more data labelling and export formats from protocols.io.

Keywords:

• Automation
• systematic review
• reporting checklists
• research standards

Acknowledgements

The authors would like to thank: Lancaster University for funding; EBTC’s Scientific Advisory Council for pre-submission review and comments; and all 26 members of the Meta-Data Enhanced Study Templates discussion group, as chaired by PW, for ideas and discussions that inspired or influenced this manuscript, in particular Mónica González-Márquez, Emma Ganley (Protocols.io), Peter Murray-Rust (Cambridge), and Joel Chan (University of Maryland).

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials. The data and materials are openly accessible at https://osf.io/zjevp/ and https://github.com/StephenWattam/TARPD and https://osf.io/zjevp/. To obtain the author’s disclosure form, please contact the Editor.

Authors contributions

Created using Tenzing app (Holcombe et al. 2020).

Conceptualization: Paul Whaley. Data curation: Stephen Wattam. Funding acquisition: Paul Whaley and John Vidler. Investigation: Paul Whaley and Stephen Wattam. Methodology: Paul Whaley and Stephen Wattam. Project administration: John Vidler. Software: Stephen Wattam. Supervision: John Vidler. Validation: Stephen Wattam. Visualisation: Paul Whaley and Stephen Wattam. Writing – original draft: Paul Whaley. Writing – review & editing: Paul Whaley, Stephen Wattam, Anna Mae Scott and John Vidler.

Ethics statement

Ethical approval and informed consent were not required for this research, as it involved neither human nor animal participants.

Preregistration

This study was not preregistered.

Disclosure statement

PW declares the following financial interests: consultancy services for the Evidence-Based Toxicology Collaboration (EBTC) at Johns Hopkins University Bloomberg School of Public Health; Lancaster University, mainly around securing impact of his PhD research (completed 2020); the University of Central Lancashire (UCLAN), which has involved developing furniture fire safety models relevant to reducing the impact of the use of chemical flame retardants in ensuring fire safety for the UK Government Agency OPSS; and for Yordas Group, to deliver training in SR methods for EU JRC. PW also receives an Honorarium as Systematic Reviews Editor for the journal Environment International. In general, PW’s consultancy work is around development and promotion of systematic review methods in environmental health research, delivering training around these methods, and providing editorial services. He is applying for funding to develop evidence surveillance and automated SR documentation methods through Lancaster University, which if successful will arrive within the next two years.

PW also declares the following non-financial interests: a historical, public commitment to high quality systematic review methods in support of environmental health policy and decision-making; active involvement in a range of projects intended to improve research standards in evidence synthesis. These include development of SR conduct standards (e.g., COSTER), reporting checklists (e.g., ROSES), and critical appraisal tools (e.g., CREST_Triage, IV-CAT) to support the production of high quality systematic reviews, evidence maps, and other study designs. PW is an active member of the GRADE Working Group, has a senior role at EBTC that involves promoting SR and other evidence-based methods in toxicology and environmental health, and is a member of an informal UK NGO network promoting good environmental policy in the UK, and several Brussels-based environmental policy advocacy groups including HEAL and EEB. In general, PW has a strong interest in supporting precautionary, evidence-based approaches to environmental health policy.

SW declares providing scientific & technical consultancy services as a subcontractor to PW on a UK OPSS-funded study into the use of chemical fire retardants in furniture. SW has also provided scientific consultancy services in unrelated fields, and states that publication of work in any area impacts chances of future employment.

AMS declares that, as part of her role at Bond University, she provides training in: evidence-based medicine and systematic review methodology (including automation tools), to Australia-based academics, clinicians and students. Those workshops charge a registration fee. AMS is part of the Systematic Review Accelerator team at Bond University, which develops and makes publicly available automation tools to accelerate completion of systematic reviews, as well as conducting research in this area (all tools are available free of charge). AMS is an Associate Editor for Systematic Reviews, a BMC/Springer Nature journal (non-remunerated).

JV is a Senior Research Associate at Lancaster University and declares having no interests that may compromise the integrity of the research described in this manuscript.

Data availability statement

All data is available from the accompanying OSF project record at https://doi.org/10.17605/OSF.IO/ZJEVP and GitHub repository at https://github.com/StephenWattam/TARPD

Additional information

Funding

This work was supported by Lancaster University Impact Acceleration Award: ‘TARPD Prototype Tool for Automated Research Project Documentation’.