Evaluation of Global Post-Outbreak COVID-19 Treatment Interventions: A Systematic Review and Bibliometric Analysis of Randomized Controlled Trials

Abstract

Purpose

The outbreak of COVID-19 has led to a global pandemic with millions of cases and deaths. Many randomized controlled trials (RCTs) were conducted to establish effective therapies. However, the methodological quality of these trials is paramount, as it directly impacts the reliability of results. This systematic review and bibliometric analysis aim to assess the methodological approach, execution diversity, global trends, and distribution of COVID-19 treatment RCTs post-outbreak, covering the period from the second wave and onward up to the present.

Methods

We utilize articles from three electronic databases published from September 1, 2020, to April 1, 2023. Inclusion and exclusion criteria were applied to identify relevant RCTs. Data extraction involved the collection of various study details. Risk of Bias (RoB) 2 tool assessed methodological quality, while implementation variability was evaluated against registration information. Bibliometric analysis, including keyword co-occurrence and country distribution, used VOSviewer and Tableau software.

Results

Initially, 501 studies were identified, but only 22 met the inclusion criteria, of which 19 had registration information. The methodological quality assessment revealed deficiencies in five main domains: randomization process (36%), deviations from intended interventions (9%), missing outcome data (4%), measurement of the outcome (18%), and selection of reported results (4%). An analysis of alignment between research protocols and registration data revealed common deviations in eight critical aspects. Bibliometric findings showcased global collaboration in COVID-19 treatment RCTs, with Iran and Brazil prominently contributing, while keyword co-occurrence analysis illuminated prominent research trends and terms in study titles and abstracts.

Conclusion

This study offers valuable insights into the evaluation of COVID-19 treatment RCTs. The scarcity of high-quality RCTs highlights the importance of enhancing trial rigor and transparency in global health emergencies.

Keywords:

• COVID-19
• randomized controlled trials
• methodological quality
• diversity in execution
• keyword trends
• geographic distribution

Introduction

The outbreak of a novel and highly contagious COVID-19 disease, caused by the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2 virus), began in Wuhan, China in December 2019.¹ The disease has put public health systems under pressure because of its rapid and intense transmission.² SARS-CoV-2 symptoms can include fever, coughing, myalgia, lethargy, pneumonia, and SARS, which can be fatal.^3,4 As of September 06, 2023, the total number of COVID-19 cases reported globally has exceeded 770.43 million, and over 6.95 million deaths worldwide.⁵ Despite the efforts that were taken such as practicing social distancing, staying at home, wearing masks, and regularly washing hands, the world is still actively working to combat the disease.⁶ However, the effectiveness of treatment interventions currently being used to treat COVID-19 has a high level of uncertainty as there is not enough clinical experience associated with these interventions.⁷ Thus, further research is needed to evaluate their potential efficacies against the virus.⁸

Randomized Controlled Trials (RCTs) are frequently regarded as the gold standard for assessing the efficacy of novel interventions or treatments.⁹ Several RCTs exploring different COVID-19 treatment interventions have been conducted.¹⁰ These include antiviral treatments designed to impede viral replication directly, recombinant neutralizing monoclonal antibodies aimed at preventing viral entry into host cells, supplementary therapies targeting the host immune response (such as anti-inflammatory), anti-fibrotic formulated to prevent angiotensin-converting enzyme 2 (ACE2) downregulation, thereby averting the establishment of a pro-fibrotic microenvironment as well as herbal and non-pharmacological interventions.^11,12 Antiviral compounds like remdesivir, along with corticosteroids such as dexamethasone, have consistently demonstrated enhanced survival, while the interleukin 6 receptor antagonist tocilizumab and the Janus kinase 1/2 inhibitor baricitinib have also exhibited reduced mortality in COVID-19 patients.¹³ Moreover, anti-fibrotic medications like Pirfenidone and Nintedanib are emerging as potential treatment options for post-COVID-19 interstitial lung disease, particularly when administered early in the clinical course of the disease.¹⁴ By September 2023, the WHO platform had published more than 726,000 articles related to COVID-19, including 86,067 records of RCTs.¹⁵ ClinicalTrials.gov has registered 4315 clinical trials, including 2543 interventional studies.¹⁶ The COVID-19 pandemic has generated an unprecedented volume of data and knowledge.¹⁷ In response to this, certain studies have undertaken systematic reviews. These reviews aim to compare the impacts of different therapeutic interventions for COVID-19,¹⁸ or appraise the evidence concerning drugs employed in the treatment of COVID-19.¹⁹ However, there has been a scarcity of studies that systematically assess the methodological aspects of RCTs.²⁰ While earlier studies have assessed the methodological aspects of RCTs during the early stages of the COVID-19 pandemic, our study offers a unique contribution by focusing on a more recent time frame. This timeframe encompasses the time from the onset of the second wave and extends to the present as of the writing of this review. This approach significantly augments our understanding of the evolving landscape of RCT methodology in the later phases of the COVID-19 pandemic. Therefore, it is important that we promptly collect, critically review, assess, and effectively implement this information to ensure generating unbiased results that lead to effective and efficient treatments.

To bridge this gap and provide a comprehensive overview, this paper applies a systematic review and bibliometric analysis of RCTs focused on COVID-19 treatment interventions post-outbreak. Our objectives are to assess the methodological quality of these RCTs, analyze the diversity in their execution, and create interactive visualizations for clustering keywords and mapping research distribution across countries. This would help advance the current knowledge on potential treatments and provide a guide for future research, thereby aiding healthcare systems by reducing hospitalizations and Intensive Care Unit admissions (ICU). This, in turn, ensures the allocation of essential resources to other patients’ needs.

Materials and Methods

Eligibility and Search Strategy

We adhere to the guidelines outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) during the process of extracting data and subsequent reporting.²¹ The search for relevant studies was conducted across multiple electronic databases, including Web of Science, PubMed, and ProQuest. Our search strategy involved the use of terms related to coronavirus and RCTs spanning from September 1, 2020, through April 1, 2023 covering the early stages of the COVID-19 pandemic up to the specified end date. We combined these terms using both “OR” and “AND” operators. Full details about searching strategy in each database are provided in Table S1.

Study Selection

We included all studies that met our following inclusion criteria: human participants diagnosed with COVID-19, any treatment intervention studies for COVID-19, with no restrictions on the type of control group, conducted between September 1, 2020, and April 1, 2023, utilizing RCT design, and available in the English language. The exclusion criteria comprised studies focusing on COVID-19 epidemiology, diagnosis, and prevention, as well as any study types other than RCTs (eg, cohort studies, case-control studies, observational studies, theoretical analyses, literature reviews, systematic reviews, or trial protocols), duplicate studies, and those with incomplete or insufficient information. The study selection process involved the use of Mendeley 1.19.8 software to organize collected records, initially eliminating duplicates through both automated and manual methods. Subsequently, we conducted a review of the titles and abstracts of the remaining studies to determine their compliance with the inclusion criteria. Finally, potentially relevant studies were read in full. The selection process involved two researchers (MA and NZ), with any disagreements being resolved through discussion involving all researchers (MA, NZ, AAS, and ZS).

Data Extraction

A digital data form was used to extract relevant information including the following title, interventions, duration of treatment, dose, author/s, country, registration number, ethic approval, level of severity, inclusion criteria, exclusion criteria, sample size, blinding, sites, funding, and date of release. To ensure efficient data management, Microsoft Office Excel (Microsoft Office 2016) was utilized as the tool for organizing and categorizing the extracted data.

Evaluating Diversity in Execution

The evaluation of execution diversity involved examining the congruence between reports of RCTs and their corresponding registration details. For this evaluation, only those studies with assigned registration numbers were considered, as their methodologies could be cross-checked against the initial registration data to establish coherence. A total of eight critical elements were subjected to comparison, encompassing inclusion and exclusion criteria, sample sizes, research outcomes, sites of recruitment, interventions, and procedures related to blinding.

Assessing the Methodological Quality

Evaluating the methodological rigor of the studies included in the analysis was carried out independently by two researchers (MA and NZ). The assessment was performed using the Risk of Bias (RoB) 2 tool, a revised version of the Cochrane risk-of-bias tool developed for randomized trials.²² The evaluation encompassed the following five domains: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results.²³ Through consensus discussions among researchers (MA, NZ, AAS, and ZS), the domains were assessed and recorded as low risk, high risk, or some concerns.

Co-Occurrence of Keywords

We have conducted an analysis of keyword frequency to delineate the internal composition of our study sample. Keywords offer a suitable representation of the content within articles, and assessing the co-occurrence frequency of keywords is valuable for comprehending the conceptual framework and developments within a particular field.²⁴ This involves quantifying the intensity of relationships among representative keywords in the selected documents, a phenomenon known as “co-occurrence”, which signifies when two keywords appear together within documents. Keyword co-occurrence analysis was conducted using VOSviewer software, version 1.6.19.^25,26

Geographic Distribution of Studies

We analyzed the distribution of included studies among various countries using a visualization tool, Tableau (version 8.1.10).²⁷ The visualization map generated through Tableau allows us to gain valuable insights into the geographic representation of the studies in our analysis. Additionally, it provides a comprehensive overview of the countries contributing to our research. This approach not only enhanced the understanding of the geographical diversity of the literature but also facilitated the identification of potential regional trends or disparities within the dataset.²⁸

Data Analysis

RCTs that met the inclusion criteria and were available in the English language published from September 1, 2020, to April 1, 2023, were categorized by their interventions. For each intervention, dose, recruitment site, registration information, ethical approval, level of disease severity, sample size, blinding, and funding were documented. The date of the first online publication was defined as the “time of first release” for each study.

The analysis of RCTs also involved the examination of PICO information, encompassing participant categories, interventions, control groups, and findings. Patients diagnosed with confirmed COVID-19 were described by various groups, including mild, moderate, severe, mild to moderate, or unspecified. Interventions utilized in these RCTs were categorized into broader groups, such as antivirals, antibiotics, etc. Control groups were divided into interventional control and placebo control. Interventional control entailed a controlled group of patients who received treatments potentially effective for treating COVID-19, which included comparisons among different interventions or the same intervention with varying doses. Primary and secondary findings were additionally grouped into more comprehensive categories, encompassing clinical events, laboratory tests, vital signs, and other assessment scales like exercise intensity rating, fatigue, mental state, tolerability, patient satisfaction, and severity of dyspnea.

The assessment of the methodology in the included RCTs involved the utilization of RoB graphs and RoB summary figures. The presentation of consistency in comparisons is expressed through frequency and percentage. Descriptive statistical analysis was carried out utilizing Microsoft Office Excel. The creation of RoB graphs and RoB summary figures was accomplished using the Robvis tool.²⁹ Graphs illustrating the co-occurrence of keywords and the distribution of studies among different countries were generated using VOSviewer and Tableau.

Results

A total of 501 studies were initially identified for review. After removing duplicate records, we screened the titles and abstracts of 466 studies. Subsequently, the full text of 27 studies was carefully reviewed. In the end, twenty-two studies were found to meet our inclusion criteria.^30–52 A complete representation of this study selection process is shown in Figure 1.

Figure 1 PRISMA diagram of the study selection process of the randomized controlled trials (RCTs) included in the analysis for the treatment of COVID-19.

Notes: PRISMA figure adapted from Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. Creative Commons.²¹

Demographic and Clinical Characteristics of the Included Studies

The earliest included RCT was published on October 20, 2020,³⁸ while the most recent study was published on February 17, 2023.³⁰ Notably, more than half (12/22, 55%) of studies enrolled participants from a single location,^{30,32–35,37,39,40,42,43,47,48} with the highest number of recruitment sites being 56.⁵² All studies (22/22, 100%) were approved by ethical committees. The sample sizes across the studies varied considerably, ranging from as low as 33⁴⁶ to as high as 1372.⁵² The largest proportion of studies (10/22, 46%) had sample sizes between 101 and 200,^{30,31,33,34,36–40,44} followed by studies (7/22, 32%) with sample sizes in the range of 33–100 participants.^{32,35,41–43,46,47} However, it is important to mention that only 8 (8/22, 36%) studies implemented blinding in their study design.^{31,32,34,42–44,50,52} The duration of treatment in the studies spanned from 1 to 14 days. The most common treatment duration observed in the studies was 10 days, which was employed in 7 (7/22, 32%).^{32,34,37,40,43,50,51} In terms of funding sources, 5 (5/22, 23%) studies received financial support from industry sources,^{32,35,41,42,50} while 8 studies (8/22, 36%) received grants from public organizations.^{31,33,34,36,38,44,46,47} Additionally, 3 (3/22, 14%) studies were funded by both private and non-private organizations,^49,51,52 and 6 studies (6/22, 27%) reported no funding.^{30,37,39,40,43,48} The demographic and clinical characteristics of the included studies are summarized in Table 1.

Table 1 The Demographic and Clinical Characteristics of the Randomized Controlled Trials (RCTs) Included in the Analysis for the Treatment of COVID-19

Intervention	Dose	Author/s	Duration of Treatment	Registration No.	Ethics Approval	Sample Size (ITT Population)	Blinding	Date of Release
Remdesivir	200 mg on day 0 and 100 mg on days 1 through 9	Karim Ali et al^Citation51	10 days	NCT04330690	Y	1282	Open label	19/01/2022
Favipiravir	1800 mg BID on Day 1, followed by 800 mg BID on Days 2–10	Yoav Golan et al^Citation50	10 days	NCT04600895	Y	1231	Double-blinded	6/9/2022
Umifenovir	Hydroxychloroquine 400 mg BID on the first day followed by Umifenovir 200mg TID 7 to 14 days based on the severity of the disease	Marzieh Nojomi et al^Citation47	7 to 14 days based on the severity of the disease	IRCT20180725040596N2	Y	100	Open label	14/12/2020
Hydroxychloroquine	400 mg BID for 1 d or HCQ 200 mg BID for seven days	Cheng-Pin Chen et al^Citation46	7 days	NCT04384380	Y	33	Open label	2/12/2020
	Hydroxychloroquine 400 mg BID on the first day, 400 mg once daily for a total of seven days	Álvaro Avezum et al^Citation52	7 days	NCT04466540	Y	1372	Double-blinded	31/03/2022
	400 mg BID for day 1 followed by 200 mg 12 hourly for the next five days.	Sultan M. Kamran et al^Citation48	6 days	NCT04491994	Y	540	Open label	30/03/2021
Hydroxychloroquine + Azithromycin	Two 200 mg Hydroxychloroquine capsules BID for a total course of 7 days and one 500 mg Azithromycin capsule taken on day 1, followed by one 250 mg Azithromycin capsule daily for the next 4 days	Cristhieni Rodrigues et al^Citation42	7 days	30,413,020.8.0000.0008	Y	84	Double-blinded	25/08/2021
Ivermectin + Doxycycline	Ivermectin 200 μgm/kg single dose + Doxycycline 100mg BID for ten days	Abu Taiub Mohammed Mohiuddin Chowdhury et al^Citation40	10 days	NP	Y	116	a	25/02/2021
Ivermectin + Metronidazole	Ivermectin 0.2 mg/kg of body weight (max. 12 mg) as a single dose, Metronidazole 8 mg/kg q6hr for five days	Mohammad Reza Heydari et al^Citation44	5 days	IRCT20180612040068N1	Y	135	Triple-blinded	19/11/2022
Methylprednisolone	Methylprednisolone (2 mg/kg/day) over 60 min, and tapered to half dosage every five days	Keivan Ranjbar et al^Citation43	10 days	IRCT20200204046369N1	Y	93	Triple-blinded	11/5/2021
	Intravenous methylprednisolone 10 mg/kg per dose once daily for 3 days, maximum dose 1000 mg per day	Tatjana Welzel et al^Citation41	3 Days	NCT04826588	Y	76	Open label	3/2/2023
Tocilizumab	Intravenous infusion of tocilizumab (8 mg/ Kg, with a starting dosage of 400 mg and a maximum dose of 800 mg, followed by a second dose 12–24 h later	Al Shaimaa Ibrahim Rabie et al^Citation30	7 days	NCT0487185	Y	110	Open label	17/02/2023
	Tocilizumab intravenously within 8 hours from randomization at a dose of 8mg/kg up to a maximum of 800 mg, followed by a second dose after 12 hours.	Carlo Salvarani et al^Citation38	One day	NCT04346355	Y	126	Open label	20/10/2020
Convalescent plasma	5 mL/kg body weight of convalescent	Manuel E. Baldeón et al^Citation31	One day	NP	Y	158	Double-blinded	9/1/2022
Vitamin D	A single oral dose of either 400,000 IU or 50,000 IU cholecalciferol	Ce´dric Annweiler et al^Citation49	One day	NCT04344041	Y	260	Open label	31/05/2022
Vitamin C	50 mg/kg/day	Poona Kumari et al^Citation39	NP	NP	Y	150	Open label	30/11/2020
L-Arginine	100 mL solution containing 5 g powdered formulation, 3 g of L-Arginine, followed by a 100 mL water rinse, 10 days or until the patients were off O2 support	Jananee Muralidharan et al^Citation32	10 days or until the patients were off O2 support	CTRI/2022/01/039549	Y	74	Triple-blinded	26/09/2021
Sentinox	Sentinox TID for Arm I and Sentinox 5 times/day for Arm II, for 5 days	Donatella Panatto et al^Citation35	5 days	NCT04909996	Y	57	Open label	12/5/2022
Nigella sativa oil	Oral Nigella sativa oil 500 mg BID postprandial for 10 days	Abdulrahman E. Koshak et al^Citation37	10 days	NCT04401202	Y	183	Open label	15/08/2021
Nilavembu Kudineer + Kaba Sura Kudineer	60 mL of Nilavembu Kudineer for Arm I and 60 mL of Kaba Sura Kudineer for Arm II, BID post morning and evening meals for a maximum of 10 days	Anurag Srivastava et al^Citation34	10 days	2020/08/027286	Y	125	Double-blinded	28/08/2021
Lianhua Qingwen	Lianhua Qingwen capsule (0.35 g per capsule), 4 capsules TID for 14 days	Ling Zhang et al^Citation36	14 days	ChiCTR2100042066	Y	120	a	14/11/2022
Qigong exercise and acupressure rehabilitation program	BID at 10 am and 4 pm from the first day for the therapists until the participants were deemed competent.	Shu-ting Liu et al^Citation33	From the first day for the therapists until the participants were deemed competent.	ChiCTR2000029994	Y	128	a	29/10/2021

Note: ^aBlinding not implemented.

Abbreviations: NP, not provided; Y, yes; N, no; ITT, intention-to-treat; q, every; BID, twice daily; TID, three times daily.

PICO Characteristics

Among the studies included in our analysis, 59% (13/22) of the studies classified patients into various subgroups based on the severity of their condition, encompassing mild, mild to moderate, moderate, and severe cases. However, 9 (9/22, 41%) studies did not provide specific details about the severity level of the patients included in their research.^{33,36,38,41,43,44,47,49,51} Concerning the interventions utilized in these studies, three studies centered around antiviral medications,^47,50,51 and three studies included the antimalarial Hydroxychloroquine as their sole intervention,^46,48,52 with one of them combining it with the antibiotic Azithromycin.⁴² Additionally, two studies employed the anthelmintic Ivermectin in combination with antibiotics,^40,44 two studies utilized the corticosteroid Methylprednisolone,^41,43 and two studies employed the immunomodulatory agent Tocilizumab.^30,38 Only one study focused on immunotherapy involving convalescent plasma.³¹ Furthermore, three studies incorporated supplementation strategies, including the use of vitamin C, vitamin D, and amino acid L-Arginine,^32,39,49 while one study utilized the acid-oxidizing solution Sentinox.³⁵ Additionally, three studies explored herbal medicine interventions, including Nigella sativa oil, Nilavembu Kudineer with Kaba Sura Kudineer, and Lianhua Qingwen,^34,36,37 and one study investigated non-pharmacological interventions such as Qigong exercise and acupressure.³³ In terms of control groups, the most frequently employed control was standard care therapy, which exhibited variations across studies, accounting for 50% (11/22).^{30,31,33,35,37–39,44,46,48,51} The interventional control was used in 23% (5/22) of the studies,^{40,41,43,47,49} which was the same proportion as those employing a placebo control (23%, 5/22).^{32,34,42,50,52} Approximately 82% (18/22) of the studies reported both primary and secondary outcomes, with additional details available in Table S2, more information about the most commonly outcome measures used are summarized in Table 2.

Table 2 Frequently Used Finding Measures in the Randomized Controlled Trials (RCTs) Included in the Analysis for the Treatment of COVID-19

Category	Finding Measure/s	Specifics	Number (%)
Finding*	Laboratory tests	PCR	1/22 (4.5)
	Clinical events	Recovery and disappearance of symptoms	1/22 (4.5)
		Hospitalization time	1/22 (4.5)
		Need for ventilation	1/22 (4.5)
		Mortality	1/22 (4.5)
Primary finding	Laboratory tests	RT-PCR, lymphocyte levels, inflammatory markers (IL6), nucleic acid negative time and rate	4/22 (18.1)
	Clinical events	Recovery, the disappearance of symptoms (eg shortness of breath and complete cessation in O2 support), and viral clearance	8/22 (36.4)
		Hospitalization time	4/22 (18.1)
		Clinical deterioration, progression of the disease, survival and mortality	7/22 (31.8)
	Vital signs	Respiration rate, pulse rate, blood pressure, blood oxygen levels, and temperature	1/22 (4.5)
	Severity of dyspnea	Modified Medical Research Council dyspnea scale	1/22 (4.5)
Secondary outcome	Laboratory tests	RT-PCR, PCR, routine blood tests, and biochemical indicators.	5/22 (22.7)
	Non-laboratory tests	CT scan	1/22 (4.5)
	Clinical events	Recovery, the disappearance of symptoms (eg shortness of breath and complete cessation in O2 support) and viral clearance	9/22 (40.9)
		Hospitalization time and admission to ICU	13/22 (59.1)
		Clinical deterioration, progression of the disease, mechanical ventilation use, and all-cause mortality	13/22 (59.1)
		Safety, tolerability, adverse events and adverse reactions to treatment	8/22 (36.4)
	Vital signs	Respiration rate, pulse rate, blood pressure, blood oxygen levels, and temperature	1/22 (4.5)
	Exercise intensity rating	Modified Borg dyspnea scale	1/22 (4.5)
	Fatigue assessment	Fatigue Scale-14	1/22 (4.5)
	Mental assessment	Patient health questionnaire-9 scale	1/22 (4.5)
	Tolerability	Visual analog scale	1/22 (4.5)
	Patient satisfaction	5-point anchored Likert scale	1/22 (4.5)

Note: *Finding encompasses findings without specifying whether they are primary or secondary findings,

Abbreviations: RT-PCR, reverse transcription polymerase chain reaction; O2, oxygen; CPR, cardiopulmonary resuscitation; IL6, interleukin-6; ICU, intensive care unit.

Methodological Assessment of Included Studies

The assessment of bias in the included studies reveals varying levels of methodological rigor. In terms of randomization processes, a majority of the studies (14/22, 64%) exhibited a low risk of bias in their randomization processes. Nonetheless, certain studies raised some concern due to limited available information regarding the concealment of the allocation sequence.^{30,39,40,46,51} Furthermore, three studies were deemed to have a high risk of bias because the concealment of the allocation sequence was not implemented.^35,38,41 Regarding deviations from intended interventions, the majority of studies (20/22, 91%) showed low risk of bias, suggesting that the interventions were carried out as planned. Nevertheless, two studies raised some concerns, due to the limited information regarding deviations from the intended intervention due to trial context in the study,⁴⁰ and the intervention revision from what was initially assigned in the study protocol, which could potentially impact the outcome in the study.⁵¹ Regarding missing outcome data, nearly all studies (21/22, 96%) were characterized by a low risk. However, one study stood out due to incomplete outcome data caused by patient dropouts, resulting in an imbalance between the experimental and control groups, and this study was noted for having a high risk of bias.⁴⁴ Most of the studies (18/22, 82%) exhibited a low risk of bias in their measurement of the outcome. However, some studies raised high risk due to differences in the measurement of the outcome between the intervention groups,^44,51 and awareness of study participants’ interventions by outcome assessors, as well as the potential influence of knowledge about the intervention received on the assessment of the outcome.^33,35 In terms of selection of the reported results, most of the studies (21/22, 96%) were characterized by a low risk. However, one study deviated from this pattern by failing to report one of the secondary outcomes in the results section. Consequently, this study was deemed to have a high risk of bias.⁴³ The summary of findings indicates that among the 22 studies, 11 exhibited a low risk of bias, while 5 studies raised some concerns, and 6 studies were categorized as having a high risk of bias. The RoB graph for all 22 studies is provided in Figure 2, and the RoB summary covering all the studies is available in Figure S1.

Figure 2 Risk of bias assessment of the randomized controlled trials (RCTs) included in the analysis for the treatment of COVID-19.

Evaluating RCT Reports and Registration Data

In the analysis of the 22 studies included, registration information was available for 19 of them. These registrations were associated with the following registries: ClinicalTrials.gov (n = 10),^{30,35,37,38,41,46,48–52} Iranian Registry of Clinical Trials (n = 3),^43,44,47 Clinical Trials Registry of India (n = 2),^32,34 Chinese Clinical Trial Registry (n = 2),^33,36 and Brazilian Registry of Clinical Trials (n = 1).⁴² Notably, one study yielded no results when searching for information using the provided registration number.³⁰ Further, about 14% of studies (3/22) did not state whether they were registered.^31,39,40 The most frequent inconsistency, observed in approximately 36% (5/14) of the studies, revolved around guideline modifications, including secondary outcomes. Following closely, the discrepancies between the published sample size and the intended target, coupled with reduced exclusion criteria. The third most prevalent disparity was linked to modifications made to the interventions, with further elaboration provided in Table 3.

Table 3 Comparison of Reports with Registration Information of the Randomized Controlled Trials (RCTs) Included in the Analysis for the Treatment of COVID-19

Category	Content	Number and Percentage of Studies in Content [n (%)]	Number and Percentage of Studies in Category [n (%)]
Inclusion criteria	Items reduced	2/18 (11.1)	18/18 (100)
	Item content revised	1/18 (5.6)
	Consistent with registration information	15/18 (83.3)
Exclusion criteria	Items increased	1/18 (5.5)	18/18 (100)
	Items reduced	4/18 (22.2)
	Item content revised	1/18 (5.6)
	Consistent with registration information	12/18 (66.7)
Sample size	Sample size increased	2/18 (11.1)	18/18 (100)
	Sample size reduced	4/18 (22.2)
	Consistent with registration information	12/18 (66.7)
Primary finding	Items reduced	1/18 (5.6)	18/18 (100)
	Consistent with registration information	17/18 (94.4)
Secondary finding	Items increased	1/18 (5.6)	17/18 (94)
	Items reduced	2/18 (11.1)
	Item content missing	2/18 (11.1)
	Item content revised	5/18 (27.8)
	Consistent with registration information	8/18 (44.4)
Sites	Locations increased	1/18 (5.6)	18/18 (100)
	Locations reduced	2/18 (11.1)
	Consistent with registration information	16/18 (88.9)
Intervention	Intervention revised	3/18 (16.7)	18/18 (100)
	Consistent with registration information	15/18 (83.3)
Blinding	Blinding not implemented	2/18 (11.1)	18/18 (100)
	Consistent with registration information	16/18 (88.9)

Key Trends in the Included Studies

When analyzing the most frequently utilized keywords within the included studies, as represented in Figure 3, it becomes evident that certain terms, such as “covid-19”, “randomized control trial”, “sars-cov-2”, “azithromycin”, “efficacy”, “hydroxychloroquine”, “clinical trial”, and “ivermectin”, appeared with larger fonts and circles, representing the high frequency of these keywords appearing in the titles and abstracts of the studies. Figure 3 further illustrates the presence of distinct clusters within the various topic areas. For instance, there is a distinct cluster in the red area that includes keywords like “azithromycin”, “ivermectin”, and “doxycycline”. This implies a deep relationship between them.

Figure 3 Keyword co-occurrence network of the randomized controlled trials (RCTs) included in the analysis for the treatment of COVID-19.

Distribution of Studies Among Countries

The studies were categorized based on the locations where they were conducted. The majority of studies originated from Iran,^43,44,47 and Brazil,^42,50,52 with each country accounting for three studies. This was followed by two studies each from China,^33,36 India,^32,34 Italy,^35,38 and Pakistan.^39,48 Additionally, individual studies were conducted in the United States,⁵⁰ Canada,⁵¹ Ecuador,³¹ Egypt,³⁰ France,⁴⁹ Mexico,⁵⁰ Saudi Arabia,³⁷ Switzerland,⁴¹ Taiwan,⁴⁶ and Bangladesh.⁴⁰ A visual representation of the geographic distribution of the studies conducted in various countries is shown in Figure 4.

Figure 4 The global distribution of the randomized controlled trials (RCTs) included in the analysis for the treatment of COVID-19.

Discussion

Our systematic review and bibliometric analysis of RCTs centered on COVID-19 treatment interventions, which were conducted during the period beyond the initial outbreak. We assess the methodological quality of these trials, elucidating both their strengths and areas of concern. Furthermore, we explore the diversity in the execution of these RCTs, underscoring the vital importance of consistency between study reports and registration details to ensure scientific rigor. We also conduct keyword co-occurrence analysis to pinpoint essential COVID-19 treatment research trends and assess the geographical dispersion of these studies, emphasizing worldwide cooperation in the pursuit of effective COVID-19 treatments.

Assessing the methodological quality of the included RCTs is crucial for understanding the reliability of their findings. Previous research has highlighted the scarcity of well-structured RCTs in publications related to urgent situations. Xie et al noted that RCTs were the most common design choice in 51 pilot studies. However, only 10 of these studies had incorporated blinding into their research methodology.⁵³ It is essential to recognize that RCTs characterized by suboptimal designs may not serve the best interests of patients or healthcare practitioners and can potentially raise significant ethical and practical concerns. Rather than allowing for exceptional cases, researchers should cooperate and strive to maintain the gold standard of RCTs, thereby making a valuable contribution to the ongoing development of the clinical evidence base for COVID-19.⁵⁴ A notable example of this is the case of hydroxychloroquine, whose utilization surged following the rapid publication of an open-label, non-randomized, small-scale trial claiming its efficacy, only to be later demonstrated as no more effective than a placebo and consequently withdrawn based on a well-designed RCT with large sample size.^55,56 This underscores the importance of raising the standard of research and registered clinical research programs while strictly adhering to the regulations for clinical trials.^57–59

Considering the methodological assessment conducted in our study, it becomes apparent that there is a range of rigor in the design and execution of research within this field. Although most studies demonstrated a low risk of bias, some raised certain concerns or were categorized as having a high risk of bias in particular domains. These concerns primarily revolved around issues such as concealment of the allocation sequence, incomplete outcome data, deviations from the intended interventions, and discrepancies in reporting secondary outcomes. These findings highlight the importance of transparent reporting and adherence to predefined protocols in RCTs, as deviations from the original plan can introduce bias and affect the validity of study results. Further, blinding, an essential component of RCTs to minimize bias, was implemented in only a minority of the included studies. This raises questions about the potential impact of knowledge about the intervention on outcome assessments in non-blinded studies.⁶ The sample size represents another crucial aspect of research, as trials without adequate statistical power tend to miss significant effects. Studies conducted with limited sample sizes are predisposed to Type II errors.⁶⁰ For instance, previous research highlights a prevalent issue where a considerable number of trials with negative outcomes are marred by insufficient participant numbers, leading to inaccurate conclusions and inefficient resource allocation in research.^61,62 In our study, we observed that a majority of the included studies had relatively small sample sizes. This underscores the need for researchers to carefully consider and plan for appropriate sample sizes to ensure the reliability and validity of their findings.

The analysis of execution diversity within RCTs is essential for understanding the consistency between study reports and registration details. In our study, registration information was available for most included studies, and most were registered with well-recognized clinical trial registries. However, discrepancies between the published study reports and registration information were identified in a significant number of cases. These discrepancies in registration and reporting may raise concerns about transparency and rigor within the field of clinical research. We believe that obtaining reliable and high-quality evidence in the near future may be challenging. The primary reasons for the low quality of registered clinical trial protocols could include a lack of clinical research expertise among researchers and their limited experience in handling unexpected health events.⁶³

The key trends identified within the included studies reveal a substantial focus on topics directly related to the COVID-19 pandemic. Unsurprisingly, terms such as “covid-19”, “sars-cov-2”, and “randomized clinical trial” prominently featured in the keywords, indicating the prevailing interest in understanding and addressing the novel coronavirus. These keyword trends offer valuable insights into the prevailing themes and priorities in COVID-19 treatment intervention research.⁶⁴ Moving beyond thematic trends, the distribution of studies among different countries reveals a global effort to contribute to the understanding of COVID-19 treatment. Despite the fact that all countries are working to create successful treatment plans, countries with high incomes have stood out for their audacious efforts in this field of study due to their abundant financial resources, higher experience, and infrastructure.⁶⁵ Our finding highlights Iran and Brazil, both middle-income nations, as prominent contributors to our study. This is supported by a study that revealed that low- to middle-income countries have published a smaller proportion of research related to SARS-CoV-2 testing, potentially reflecting their emphasis on research areas constrained by limited resources, such as treatment interventions.⁶⁶

Improving upcoming RCTs requires robust support from diverse stakeholders such as public agencies, research academic institutions, medical companies, and other health partners. The main focus should be on strengthening research methods and developing adaptable clinical trial designs to improve the reliability of therapeutic evidence in rapidly changing healthcare crises.⁶⁷ Trials with adaptive designs, which can be adjusted in terms of sample size and randomization based on evolving circumstances, hold promise for improving the efficiency of clinical trials, aligning with the complexities and uncertainties encountered in the context of the COVID-19 pandemic.⁶⁸ Embracing a broader spectrum of research, including observational and interventional studies, in real-world settings is also crucial. Real-world research, which involves patients from medical institutions, households, and communities, offers greater flexibility and can reflect actual treatment processes.⁶⁹

This study carries significant implications for various stakeholders. Researchers are reminded of the critical importance of methodological rigor and transparent reporting in COVID-19 treatment intervention studies,⁷⁰ while healthcare practitioners benefit from a heightened awareness of potential biases and methodological issues when assessing specific treatments.⁷¹ Policymakers can utilize the study’s insights to target resources, advocate for transparency among researchers, and tailor interventions to regions with varying research trends.⁷² Future research priorities include addressing trial heterogeneity, exploring clinical significance, and fostering international collaboration. Additionally, transparent reporting and acknowledgment of limitations in public health communication can foster public trust and realistic expectations regarding COVID-19 treatments.⁷³ Together, these implications advance evidence-based care and informed decision-making in the ongoing battle against the pandemic.

This study is not without limitations. Firstly, our research exclusively focused on the analysis of RCTs for the treatment of COVID-19. By limiting our scope to English-language publications, there may be a potential language bias, which could result in the exclusion of valuable research published in languages other than English. Furthermore, a more comprehensive understanding could be achieved by incorporating additional databases and utilizing free-text searches. The identification of inconsistencies between RCT reports and registration information highlights potential transparency and reporting bias issues, though this study does not delve into the reasons behind these discrepancies. Additionally, our study primarily emphasizes methodological aspects and conducts bibliometric analysis, offering a relatively limited discussion on the clinical significance of the interventions under examination. Lastly, we did not explicitly address or analyze the heterogeneity among the included studies.

Conclusion

In conclusion, as the COVID-19 pandemic continues to evolve, the need for high-quality research on treatment interventions remains essential. This study contributes to the understanding of the current landscape of COVID-19 treatment RCTs and identifies strengths and weaknesses in their methodologies through RoB analysis, underscoring the need for improved study design and execution. Additionally, the bibliometric analysis reveals trends and global collaboration in COVID-19 treatment studies, aiding researchers, and policymakers in prioritizing future investigations.

Data Sharing Statement

All the datasets generated during this review are provided within this manuscript.

Ethical Approval

This systematic review and bibliometric analysis incorporated only publicly accessible papers, and as no human subjects were part of the study, there was no necessity for ethics approval.

Disclosure

The authors declare no conflicts of interest.

Additional information

Funding

This study is supported by a grant from Universitas Padjadjaran (1549/UN6.3.1/PT.00/2023).