Assessing the reporting quality of randomized controlled trials on COVID-19 vaccines: a systematic review

Figures & data

Figure 1. A flowchart of the screening process for randomized controlled trials (RCT) articles of the COVID-19 vaccine.

Table 1. Overall quality of reporting: rating using items based on the 2010 CONSORT statement (n = 22)

Item	Criteria	Description	No. of positive trials	%	Cohen’s k coefficient
1	Title	Identification as a randomized trial in the title	13	59	0.91
2	Abstract structure	Structured summary of trial design, methods, results and conclusions	21	95	1
3	Background	Adequate description of the scientific background and explanation of rational	22	100	NA*
4	Objectives	Description of the specific objectives or the scientific hypotheses in the introduction	21	95	0.89
5	Trial design	Description of trial design, including allocation ratio	20	91	0.81
6	Participants	Description of the eligibility criteria for participants	20	91	0.83
7	Settings and location	Description of the settings and locations where the data were collected	10	45	1
8	Interventions	Details of the interventions intended for each group	20	91	1
9	Outcomes	Definition of primary and secondary outcome measures, including how and when they were assessed	18	82	0.94
10	Sample size	Description of sample size calculation	15	68	0.92
11	Randomization, sequence generation	Definition of the method used to generate the random allocation sequence	15	68	0.95
12	Randomization, restriction	Description of the type of randomization details of any restriction	12	55	0.79
13	Allocation concealment	Description of the mechanism used to implement the random allocation sequence to assure concealment until interventions were assigned	18	82	1
14	Implementation	Description of who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions	9	41	0.85
15	Blinding	Whether or not participants, those administering the interventions, or those assessing the outcomes were blinded to group assignment; if relevant, description of the similarity of interventions	18	82	0.88
16	Statistical methods	Description of the statistical methods used to compare groups for primary and secondary outcomes	18	82	0.98
17	Ancillary analysis, method	Description of the methods for additional analyses, such as subgroup analyses and adjusted analyses	5	23	0.83
18	Diagram	A CONSORT diagram was presented to show the flow of participants	19	86	0.86
19	Participant flow	Details on the flow of participants through each stage of the trials (number of patients randomly assigned, receiving intended treatment, and were analyzed for the primary outcome)	13	59	0.96
20	Recruitment	Dates defining the periods of recruitment and follow-up	15	68	1
21	Baseline data	A table showing baseline demographic and clinical characteristics for each group	22	100	NA*
22	Intent-to-treat analysis	Number of patients in each group included in each analysis and whether patients were analyzed according to the group to which they were randomly assigned	4	18	0.77
23	Outcomes measures	For each primary and secondary outcome, a summary of results for each group, the estimated effect size and its precision (eg, 95% CI) are provided	12	55	0.87
24	Ancillary analyses	Results of subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory	7	32	1
25	Adverse event classification	Description of all important adverse events in each group, with classification	20	91	0.74
26	Registration	Presentation of the registration number and name of trial registry	22	100	NA*
27	Protocol	Where the full trial protocol can be accessed	20	91	0.73
28	Funding	Sources of funding and other support	19	86	0.86

Abbreviations: CI: confidence interval; CONSORT: Consolidated Standards of Reporting Trials; NA: not available.

* Cohen’s k indices could not be calculated because the positive rates awarded by the 2 investigators were both 100% for these items.

Table 2. The basic characteristics of clinical experiments

NO.	Author	Year	Score	Region in which trials were conducted	Journal	Blinding	Type of vaccine	Sample size	Phase	Endpoints
1	Peter Richmond	2021	21	Australia	Lancet	triple blind	SCB-2019	148	phase 1	Safety (adverse reactions) and immunogenicity
2	Laurence Chu	2021	24	US	Vaccine	triple blind	mRNA-1273	600	phase 2	Safety (adverse reactions) and immunogenicity
3	Kathryn E. Stephenson	2021	20	USA	JAMA	triple blind	Ad26.COV2.S	25	phase 1	immunogenicity
4	Hong-Xing Pan	2021	23	China	Chinese Medical Journal	double blind or double-masked	KCONVAC	560	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
5	L.R. Baden	2021	23	USA	The new england journal of medicine	triple blind	mRNA-1273	30420	phase 3	effectiveness of the vaccine
6	V.Shinde	2021	21	South Africa	The new england journal of medicine	triple blind	NVX-CoV2373	4406	phase 2	Safety (adverse reactions) and effectiveness of the vaccine
7	Katherine R W Emary	2021	12	UK	Lancet	unblinded or open label	ChAdOx1nCoV-19(AZD1222)	520	Phase 2 and phase 3	effectiveness of the vaccine
8	J.Sadoff	2021	19	Belgium and the United States	The new england journal of medicine	double blind or double-masked	Ad26.COV2.S	810	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
9	Pedro M Folegatti	2020	24	UK	Lancet	double blind or double-masked	ChAdOx1nCoV-19	1077	Phase 1 and phase 2	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
10	Denis Y Logunov	2021	24	Russia	Lancet	double blind or double-masked	Gam-COVID-Vac	21 977	phase 3	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
11	Yanjun Zhang	2021	22	China.	Lancet Infect Dis	double blind or double-masked	CoronaVac	744	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
12	Mark J. Mulligan	2020	15	USA	Nature	single blind or single-masked	BNT162b1	45	Phase 1 and phase 2	Not included
13	Maheshi N Ramasamy	2020	21	UK	The Lancet	single blind or single-masked	ChAdOx1 nCoV-19	560	phase 2	Safety (adverse reactions), immunogenicity and effectiveness of the vaccine
14	Edward E. Walsh	2020	16	the United States	The new england journal of medicine	unblinded or open label	BNT162b1	195	phase 1	Safety (adverse reactions) and immunogenicity
15	Shengli Xia	2020	21	China	JAMA	double blind or double-masked	inactivated COVID-19 vaccine	320	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
16	Fernando P. Polack	2020	20	United States, Argentina, Brazil, South Africa, Germany and Turkey,	The new england journal of medicine	unblinded or open label	BNT162b2	43548	Phase 2 and phase 3	Safety (adverse reactions) and effectiveness of the vaccine
17	Shengli Xia	2020	22	China	Lancet Infect Dis	triple blind	BBIBP-COV	640	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
18	C. Keech	2020	17	Australia	The new england journal of medicine	unblinded or open label	NVX-CoV2373	131	Phase 1 and phase 2	Safety (adverse reactions) and immunogenicity
19	S.A. Madhi	2021	18	South Africa	The new england journal of medicine	double blind or double-masked	ChAdOx1 nCoV-19	2026	Phase 1 and phase 2	effectiveness of the vaccine
20	Jing Pu	2021	16	China	Vaccine	double blind or double-masked	inactivated SARS-CoV-2	192	phase 1	Safety (adverse reactions) and immunogenicity
21	Feng-Cai Zhu	2020	25	China	The Lancet	triple blind	Ad5-vectored COVID-19	508	phase 2	Safety (adverse reactions) and immunogenicity
22	Raches Ella	2021	24	India	Lancet Infect Dis	triple blind	BBV152	375	phase 1	Safety (adverse reactions) and immunogenicity

Table 3. Trial characteristics

Characteristic	No. of studies (n = 22)	%
Year of publication
2020	9	41
2021	13	59
Region in which trials were conducted
Asia	7	32
Europe and North America	10	45
Others	5	23
Journal
The new england journal of medicine	7	32
The Lancet	6	27
Journal of the American Medical Association	2	9
Lancet Infectious Diseases	3	14
Vaccine	2	9
Nature	1	5
Chinese Medical Journal	1	5
Journal impact factor
<30	6	27
30–80	9	41
>80	7	32
Sample size
Median(range)	560(25–43548)
<500	8	36
500–1000	8	36
>1000	6	27
Sources of trial funding
Government/foundation	11	50
Completely funded by industry	4	18
Partially funded by industry	7	32
Type of vaccine
inactivated vaccine	6	27
Nucleic acid vaccine (DNA, mRNA)	7	32
Adenovirus vector vaccine	8	36
Protein subunit vaccine	1	5
Phase
1	14	43
2	15	45
3	4	12

Table 4. Publication characteristics associated with 2010 overall reporting quality

Publication characteristic	Mean CONSORT adherence scores (95% CI)	Estimate (95% CI)*	p
Year of publication
2020	20.3 (17.44, 22.79)	Reference	0.783
2021	20.7 (18.38, 22.70)	−0.17 (−0.99, 0.65)
Region in which trials were conducted
Asia	22.4 (19.16, 24.55)	Reference	0.351
Europe and North America	20 (16.95, 23.05)	0.61 (−0.11, 1.32)
Others	19 (17.04, 20.96)	−0.33 (−2.48, 1.81)
Journal impact factor
<30	22.5 (18.69, 24.98)	Reference	0.395
30–80	20.3 (17.03, 23.64)	−0.28 (−1.29, 0.73)
>80	19.1 (16.91, 21.37)	0.07 (−1.84, 1.99)
Sample size
<500	18.8 (16.08, 21.42)	Reference	0.248
500–1000	21.5 (17.60, 24.40)	0.54 (−0.18, 1.27)
>1000.	21.7 (19.12, 24.21)	0.14 (−1.16, 1.44)
Sources of trial funding
Government/foundation	21.5 (18.75, 23.43)	Reference	0.58
Completely funded by industry	20.3 (14.99, 25.51)	0.14 (−0.30, 0.58)
Partially funded by industry	19.3 (15.84, 22.73)	0.08 (−0.27, 0.44)

Abbreviation: CONSORT adherence scores rated on a scale of 0 to 28. *The estimates indicate the benefit observed compared with the reference. Any positive value indicates incremental benefit compared with the reference, whereas any negative value indicates detriment compared with the reference.

Figure 2. Percentage of literature that meets the 28-item 2010 Consolidated Standards of Reporting Trials (CONSORT) standardized evaluation checklist.

Data availability statement

Data are available upon reasonable request.