Using bench protocol platforms to improve compliance with systematic review guidance documents and reporting checklists: Proof of concept

Figures & data

Figure 1. Operationalisation into the general protocol of COSTER Recommendation 1.1.2: ‘identify information management practices for each stage of the review, including reference and knowledge management tools, systematic review software, and statistics pakages’.

Table 1. Cross-walk of the top-level data labels and steps of the general-purpose protocol with items in the conduct standards and reporting checklists with which the protocol is designed to facilitate compliance.

Labels	Meaning of Labels	Step of General Protocol	Related conduct guideline or reporting checklist items
			COSTER	PRISMA-P	PRISMA-S	ROSES	PROSPERO (human SRs)
osf_record	Title and URL of accompanying Open Science Framework record	1	–	1a, 1b	–	1	1
Competencies	Key competencies of the SR team	2	1.1.1	3b (partial)	16	–	6, 11
info_manage	Information management practices for the SR	3	1.1.2	11a	–	–	26
Funding	Information about funding of the SR	4	–	5a, 5b, 5c	–	–	12
interests	Conflicts of interest management policy	5	1.1.3, 1.1.4	–	–	38	13
Need	Explanation of need for the SR	6	1.2.1	6	–	5	–
rationale	The theoretical framework that supports the choice of question and use of SR methods to answer it.	7	1.2.2	6	–	5	–
question	The question being answered by the SR, characterised as a PECO statement	8	1.2.3	7, 12, 13	–	7, 8	15, 24, 25
eligibility	The eligibility criteria for evidence to be included in the SR	9	1.3	8	–	20	18-23
screening	The approach to screening evidence for eligibility for the SR	10	3.1	11b	–	18-20	26
multi_pubs	The strategy for handling multiple publications from the same eligible study	11	3.4	–	–	–	–
Search	The search strategy for the SR	12	1.4.1, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6	9, 10	1–6, 8, 10, 11	9, 10, 15	16, 17
rerun_search	The plans for re-running the search	13	2.7	–	12	17	16
Rob	The risk of bias assessment methods for the SR	14	1.4.3, 5.3, Section 5	14	–	22, 23	27
other_appr	Other evidence appraisal methods to be used	15	7.8	–	–	22	–
char_incl	The planned characteristics of included studies table	16	1.4.2	–	–	–	–
extract_forms	The forms to be used for data extraction	17	1.4.7	12	–	–	–
Synth	The methods to be used for synthesising the included evidence	18	1.4.4	15–16	–	30–34	28, 29
certainty	The methods to be used for assessing certainty in the results of the synthesis	19	1.4.5	17	–	23, 35,	–
integration	The methods to be used for integrating multiple streams of evidence	20	1.4.6	–	–	–	–
pilot_screening	Instruction to pilot the screening methods	21	1.4.7
pilot_extraction	Instruction to pilot the data extraction forms	22	1.4.7	11c	–	25, 26	–
pilot_rob	Instruction to pilot the risk of bias assessment methods	23	1.4.7	–	–	–	–
pilot_certainty	Instruction to pilot the certainty assessment methods	24	1.4.7	–	–	–	–
Write	Instruction to create a draft protocol document from the above steps	25	–	–	–	–	–
registration	Instructions and location data for registration of the protocol document	26	1.5.1	2	–	–	–
peer_review	Instructions for peer-review of the protocol document	27	1.5.2	–	14	–	–
publication	Instructions and location data for publication of the peer-reviewed protocol document	28	1.5.3	–	–	–	34
Items not covered in general template		PRISMA-P		Not covered: 2, 3a, 3b, 4. 12 is covered by rationale (protocol step 7)
		PRISMA-S		Not covered: 7, 16. 9 is covered by eligibility criteria. 13, 15 not relevant for a protocol
		ROSES		Not covered: 2, 3, 4, 6, 11, 12, 13, 14, 15, 16, 21, 27, 29, 36, 37. 16 (search validation) should be added in a future update. 19, 22, 28: COSTER recommends duplicate screening and appraisal so consistency checking is implicitly covered
		PROSPERO		Not covered: 2, 3, 4, 5, 7, 8, 9, 10, 14, 30, 31, 32, 33, 35, 36, 37, 38, 39, 40. Mainly administrative data for PROSPERO

For a complete list of labels and definitions, see https://osf.io/vw4e3.

Figure 2. The process for automating the documentation of a protocol run. Arrows in blue represent interfaces secured using the Oauth2 authentication standard (https://oauth.net/2/).

Figure 3. A truncated sample of structured data in JSON format, as extracted from the protocol run.

Figure 4. An example of how a user can comment on a step of the protocol to suggest a feature or improvement. A user could also generate a fork of the protocol to add steps specific to their own requirements.

Figure 5. Illustration of how the general-purpose protocol operationalises conduct standards into practical steps generating labelled data, which can be assembled into checklist-compliant protocol manuscripts. We believe this flow should be applicable to any study design, not just SR protocols.

Data availability statement

All data is available from the accompanying OSF project record at https://doi.org/10.17605/OSF.IO/ZJEVP and GitHub repository at https://github.com/StephenWattam/TARPD