A systematic scoping review of adherence to reporting guidelines in health care literature

Abstract

Background

Reporting guidelines have been available for the past 17 years since the inception of the Consolidated Standards of Reporting Trials statement in 1996. These guidelines were developed to improve the quality of reporting of studies in medical literature. Despite the widespread availability of these guidelines, the quality of reporting of medical literature remained suboptimal. In this study, we assess the current adherence practice to reporting guidelines; determine key factors associated with better adherence to these guidelines; and provide recommendations to enhance adherence to reporting guidelines for future studies.

Methods

We undertook a systematic scoping review of systematic reviews of adherence to reporting guidelines across different clinical areas and study designs. We searched four electronic databases (Cumulative Index to Nursing and Allied Health Literature, Web of Science, Embase, and Medline) from January 1996 to September 2012. Studies were included if they addressed adherence to one of the following guidelines: Consolidated Standards of Reporting Trials (CONSORT), Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), Quality of Reporting of Meta-analysis (QUOROM), Transparent Reporting of Evaluations with Nonrandomized Designs (TREND), Meta-analysis Of Observational Studies in Epidemiology (MOOSE) and Strengthening the Reporting of Observational Studies in Epidemiology (STROBE). A protocol for this study was devised. A literature search, data extraction, and quality assessment were performed independently by two authors in duplicate. This study reporting follows the PRISMA guidelines.

Results

Our search retrieved 5159 titles, of which 50 were eligible. Overall, 86.0% of studies reported suboptimal levels of adherence to reporting guidelines. Factors associated with better adherence included journal impact factor and endorsement of guidelines, publication date, funding source, multisite studies, pharmacological interventions and larger studies.

Conclusion

Reporting guidelines in the clinical literature are important to improve the standards of reporting of clinical studies; however, adherence to these guidelines remains suboptimal. Action is therefore needed to enhance the adherence to these standards. Strategies to enhance adherence include journal editorial policies endorsing these guidelines.

Keywords:

• scoping
• systematic
• review
• adherence
• reporting
• guidelines

Background

The medical literature is an integral component of clinical care, education, and research, as it has a serious impact on our understanding of health and disease. There are thousands of medical journals that publish articles related to clinical interventions, prognosis, diagnosis, and risks – among others – with an influence on health and life in general. For example, a quick glance at PubMed shows over 22 million citations for biomedical literature.¹ It is therefore a challenge to try to assimilate data presented in the literature and make evidence-based informed decisions. Attempts to summarize these data using systematic reviews are commendable as these reviews aim to provide a summary of the state of knowledge on a specific topic and address the inconsistent findings from single studies. However, these reviews are exponential in number and may report disparate findings. Searching for systematic reviews on depression resulted in 30,038 articles,² and in cancer resulted in 323,633.³

One way to assimilate and disseminate knowledge that can influence decision-making and provide an understanding of a certain condition is to perform a systematic review of reviews. The past few decades have given rise to a handful of such studies in several clinical areas including lifestyle interventions,⁴ interventions to improve mental health,⁵ homeopathy,⁶ medical education,⁷ spinal manipulation,⁸ sleep medicine,⁹ and cancer,¹⁰ among others. Each of these reviews of reviews is focused on a specific clinical question. There is a paucity of systematic reviews that assess the quality of reporting of clinical studies across different clinical areas, and that use different reporting guidelines. The EQUATOR (Enhancing Quality and Transparency in Health Research) network is an international initiative that supports the development and dissemination of such guidelines.¹¹ The EQUATOR website provides guidelines for the minimum information required to report research methods and findings for various kinds of medical research.¹²

The evidence that is presented in the clinical literature can carry substantial weight in informing professionals and users of health care on multiple aspects of health risks, disease, health care outcomes, and delivery. However, readers of the literature are faced with conflicting results presented in various formats and styles, making interpretations and conclusions challenging even for the most informed readers. For this reason, a consensus on reporting such evidence is needed to establish the quality of such studies. It is also important to ensure that a more uniform method is used by researchers to enable the combination of results from multiple studies and reach more standardized summaries and conclusions; this can minimize heterogeneity, which often complicates meta-analyses in future studies. Furthermore, poorly reported research can cause harm to patients and lead to the use of scarce resources on ineffective treatments.¹³

To address the concern over the quality of reported studies and ensure transparency in reporting clinical studies, the Consolidated Standards of Reporting Trials (CONSORT)¹⁴ statement was produced as a collaborative effort to provide a checklist and flow diagram for authors to have as a guide to prepare reports on randomized controlled trials (RCTs) for publication. The CONSORT Statement was further updated in 2010 based on new evidence and an added focus on specific designs of RCTs.¹⁵ The CONSORT is a widely accepted and adopted statement that is well described in many freely accessible publications and websites. In brief, the CONSORT Statement provides a 25-item checklist describing the required criteria for inclusion when reporting RCTs. Such items include the study design, the participants, interventions, outcomes, and sample size among others. It also recommends the inclusion of a flow diagram, accounting for recruitment, randomization, allocation of interventions, and retention in the study.¹⁶ Since the introduction of the CONSORT, several extensions and modifications of the original statement have been established to improve the quality of reporting of various study types, including observational studies, systematic reviews, and meta-analysis. Despite the availability of such guidelines for reporting, the quality of reporting of clinical studies has remained suboptimal with several manuscripts in a number of clinical areas missing key items as described in the CONSORT.^16–23

Evidence suggests that the use of the CONSORT criteria is associated with improved standards of reporting.^24,25 However, it is not clear what the current level of adherence to reporting guidelines is, what factors are associated with improving the reporting of clinical literature, and how the results from different studies on reporting standards can be compiled to provide an overall conclusion on the current state of reporting standards.

We therefore undertook a systematic scoping review evaluating systematic reviews addressing the adherence standards to reporting guidelines published since the introduction of the CONSORT Statement in January 1996 to September 2012.

Study aims

In this study, we aimed to examine the extent of adherence to reporting guidelines in published clinical research since the introduction of the CONSORT Statement in 1996. The purpose of this systematic scoping review is to inform researchers, guideline developers, journal editors, and evidence users on the profile of reporting the existing literature and the current state of knowledge in the application of these guidelines. In particular, we will endeavor to address the following questions: (1) what is the current adherence to the reporting standards that include the CONSORT,²⁶ Strengthening the Reporting of Observational Studies in Epidemiology (STROBE),²⁷ Quality of Reporting of Meta-analysis – (QUOROM),²⁸ Transparent Reporting of Evaluations with Nonrandomized Designs (TREND),²⁹ Meta-analysis Of Observational Studies in Epidemiology (MOOSE),³⁰ and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines?³¹ (2) What are the factors that are associated with adherence to reporting standards in medical literature? And (3) what guidance can we provide based on the current state of knowledge on adherence to reporting standards? More specifically the objectives of this review are to:

1. Report the levels of adherence to the above reporting guidelines in clinical research;

2. Determine the key factors associated with adherence to good reporting; and

3. Provide recommendations to enhance adherence to reporting guidelines for future studies.

We preselected the six guidelines above because they are among the oldest and the most popular, spanning through a wide range of study designs and clinical areas, and are therefore likely to be reported in systematic reviews, thus potentially generating a number of reviews to be included in this study.

Methods

We adopted a “systematic” scoping review approach – which is a combination of a scoping review methodology – to ensure the inclusion of broad areas of research and study designs, and a systematic review of reviews methodology.^32,33

A scoping review is a relatively new type of study providing an assessment of available evidence from the literature in a broad area of research such as the compliance in the reporting of clinical studies to established guidelines. It also serves to identify gaps in the field and provide recommendations for implementation.³² The methodology of scoping reviews was first described in detail by Arksey and O’Malley³² in their pivotal paper published in 2005, which provided a foundation for carrying out a scoping review. This framework was further operationalized, and five stages were proposed to be followed when conducting a scoping review, including: (1) the identification of a research question; (2) finding the relevant studies; (3) the selection of studies to be included in the review; (4) data extraction from the included studies; and (5) assembling, summarizing, and reporting the results of the review.³⁴

The methods of conducting a systematic review of systematic reviews follow a similar approach, but include the provision of guidelines and suggestions for clinical practice, education, and research.³³ The aim of the methods and search strategy here is to ensure that the systematic review of reviews is comprehensive, thorough, and objective. We will report the results using the PRISMA (formerly QUOROM) reporting guidelines for systematic reviews.³⁵ A protocol was specifically designed for this study outlining the study design, search strategy, and selection criteria.

Data sources and search strategy

Electronic literature databases including Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science, Embase, and Medline (from January 1996 [date of CONSORT] to September 2012) were searched using a comprehensive search strategy designed with the assistance of a librarian who is experienced in conducting systematic reviews. The reference lists of identified articles were also reviewed for additional studies, and a manual search of key journals like BioMed Central systematic reviews, BioMed Central Research Methodology, and the Cochrane Library was conducted to avoid missing relevant reviews. Such search strategies are well supported for this type of systematic search and retrieval of relevant studies.^36,37 The databases were searched for the following key search terms: (Systematic reviews OR reviews OR quality of reporting OR completeness of reporting) AND (CONSORT OR STROBE OR QUOROM OR QUORUM OR PRISMA OR TREND OR MOOSE) OR adherence. For Web of Science, we also performed a forward citation search of the publications pertaining to reporting guidelines, whose acronyms might have other meanings, such as TREND and QUORUM. This helped us to decrease the occurrence of false positives in our search.

Initially, no language limits were set to identify the number of non-English reviews; however, a limit was then set for English language reviews only (which was necessary due to the lack of resources required to translate reviews from other languages). We also set the limits to “human” and “published complete systematic reviews.”

Inclusion criteria

1. Systematic reviews of clinical studies addressing the quality of reporting of the studies based on at least one of the six preselected reporting guidelines: CONSORT for RCTs; TREND for non-RCTs; STROBE for observational studies; and PRISMA (formerly QUOROM) or MOOSE for systematic reviews of RCTs or observational studies, respectively.

2. The systematic reviews must be complete (not abstracts only), reported in English, and investigating the quality of reporting in human studies of all age groups using one of the above guidelines.

3. The quality of reporting guidelines must be the primary focus of the systematic review.

Exclusion criteria

Systematic reviews were excluded if they were published as abstract only; the primary focus of the review was not on the quality of reporting; the quality of reporting was based on the standards of reporting that were different from the ones stated above, or if they were a duplicate publication of existing reviews (commentaries, letters, and editorials).

Selection of systematic reviews

Two independent reviewers examined the titles and abstracts of all citations identified in the literature search. Articles were selected for full-text review if the inclusion criteria were met and if both reviewers considered the citation potentially relevant. Disagreement at any stage of study selection was resolved by discussion and consensus between the two reviewers. If agreement could not be reached a third author was recruited to determine eligibility. Initial agreement between the two reviewers was calculated using the kappa statistic.³⁸

Each reviewer independently:

• Assessed retrieved titles and abstracts for relevance and duplication;

• Screened full text articles deemed eligible for inclusion;

• Decided on including or excluding articles;

• Extracted relevant data using specifically designed data abstraction forms;

• Appraised the quality of the included reviews. A PRISMA flow diagram of included/excluded studies is provided (Figure 1).³⁵

  Figure 1 Flow diagram for study selection.

  Abbreviations: CINAHL, Cumulative Index to Nursing and Allied Health Literature; n, number.

  

Quality assessment of systematic reviews

The quality of each systematic review that met the inclusion criteria for the study was assessed using a modified version of the assessment of multiple systematic reviews (AMSTAR, a validated tool to assess the methodological quality of systematic reviews).^39–41 Certain items of AMSTAR are not relevant to this type of review and cannot be assessed (eg, item 9, “Were the methods used to combine the findings of studies appropriate?”), as pooling of data may not be feasible in all systematic reviews of methodological quality, and should relate to the study question. In addition, item 10 (“Was the likelihood of publication bias assessed?”) is irrelevant to this review, which is focused on the quality of reporting of published studies. Both of these items were omitted from the quality assessment.

We also used the modified version of the enhanced Overview Quality Assessment Questionnaire (OQAQ) to assess the quality of systematic reviews included in this study.⁴² In addition to these tools, we assessed the quality of the reviews based on the following criteria: the use of explicit criteria to assess individual study quality using the guidelines checklist; explicit definition of the research question using a flow diagram to explain study selection; and a formal sample size calculation for the assessment of association.

Data abstraction

A spreadsheet was created to record the following items from the selected reviews: authors, year of publication, number of primary studies included in the review, study location, study type, primary outcomes of the study, outcomes measures, and the overall results and conclusions. Two authors independently piloted the data extraction form for this review and modifications were made when necessary before reaching the final data abstraction forms used for this study. Data abstraction disagreements were resolved by discussion and consensus, and a third author extracted the data if an agreement could not be reached. Data collected from each systematic review included the study primary question, primary outcome, number of the studies included in the review, the statement investigated, quality assessment, the factors associated with adherence, the journal of publication, and whether the journal endorsed the statement in question.

Analysis

The level of agreement between raters was estimated using the kappa statistic. The adherence to reporting standards was summarized, and key determinants of adherence were identified in a narrative manner.

Results

Study selection

Our search retrieved 5159 articles from the four electronic databases searched. Following searching through the title, abstract, and full text screening, 50 articles were selected and included for data extraction and quality assessment (Figure 1). The strength of agreement between two independent raters on abstract screening was substantial (Kappa = 0.65; 95% confidence interval [CI] 0.53, 0.76; P < 0.001), and almost perfect for full text screening (Kappa = 0.94; 95% CI 0.85, 1.00; P < 0.001). Agreement was also substantial for the quality assessment using the modified OQAQ/AMSTAR checklist (Kappa = 0.63; 95% CI 0.42, 0.85; P < 0.001).

Study characteristics

Forty-one studies (82.0%) assessed RCTs using the CONSORT Statement, five (10.0%) studies used the QUOROM checklist, and two studies (4.0%) used the PRISMA tool. The final three systematic reviews (6.0%) consisted of two reviews assessing both RCTs and observational studies using the CONSORT and STROBE guidelines, and the last study used both the QUOROM and PRISMA guidelines. The systematic reviews were published in a wide variety of journals and were led by authors from many different countries (Table 1). The median and interquartile range of the number of studies included in each review were 78 and 80.5, respectively.

Table 1 Characteristics of included studies

First author	Year	Journal	City/country	Statement assessed	Number of studies
Al-Namankany^Citation54	2009	International Journal of Pediatric Dentistry	London, UK	CONSORT	173
Areia^Citation21	2009	Endoscopy	Coimbra, Portugal	CONSORT	120
Augestad^Citation44	2012	Journal of the American Medical informatics Association	Tromso, Norway	CONSORT	32
Balasubramanian^Citation45	2006	Annals of Surgery	Sheffield, UK	CONSORT	69
Bath^Citation55	1998	Stroke	London, UK	CONSORT	114
Bereza^Citation56	2008	Annals of Pharmacotherapy	Toronto, ON, Canada	QUOROM	16
Bian^Citation43	2006	Journal of Chinese Integrative Medicine	Hong Kong, People’s Republic of China	CONSORT	66
Bousquet^Citation57	2010	Journal of Allergy and Clinical Immunology	Montpelier, France	CONSORT	94
Capili^Citation58	2010	Clinical Journal of Pain	New York, NY, USA	CONSORT	10
Cavadas^Citation59	2011	International Urogynecology Journal	Porto, Portugal	CONSORT	41
Chowers^Citation60	2009	Journal of Antimicrobial Chemotherapy	Kfar Saba, Israel	CONSORT	49
Cook^Citation61	2011	Medical Education	Minnesota, USA	STROBE	130
de Vries^Citation62	2010	Archives of Diseases in Childhood	Leeuwarden, Netherlands	CONSORT	107
Ethgen^Citation63	2009	BMC Medical Research Methodology	Paris, France	CONSORT	132
Eyawo^Citation64	2008	Trials	Burnaby, BC, Canada	CONSORT	47
Farrokhyar^Citation65	2007	Canadian Journal of Surgery	Hamilton, ON, Canada	CONSORT	50
Froud^Citation48	2012	Community Dentistry and Oral Epidemiology	London, UK	CONSORT	23
Fung^Citation49	2009	Ophthalmology	San Francisco, CA, USA	CONSORT, STROBE	36
Gagnier^Citation66	2006	American Journal of Medicine	Toronto, ON, Canada	CONSORT	206
Halpern^Citation67	2004	International Journal of Obstetric Anesthesia	Toronto, ON, Canada	CONSORT	99
Hemels^Citation68	2004	Current Medical Research and Opinion	Paris, France	QUOROM	32
Herdan^Citation69	2011	Gynecological Surgery	Bamberg, Germany	CONSORT	37
Junhua^Citation70	2007	The Journal of Alternative and Complementary Medicine	Tianjin, People’s Republic of China	QUOROM	107
Kiehna^Citation46	2011	Journal of Neurosurgery	Charlottesville, VA, USA	CONSORT	27
Kober^Citation71	2006	Journal of the National Cancer Institute	North Lyneham, Australia	CONSORT	142
Ladd^Citation25	2010	Addictive Behaviors	Albuquerque, NM, USA	CONSORT	127
Li^Citation72	2011	Evidence-Based Complementary and Alternative Medicine	Baltimore, MD, USA	CONSORT	42
Lu^Citation73	2011	Expert Review of Anticancer therapy	Guangzhou, People’s Republic of China	CONSORT	46
Ma^Citation74	2011	PLoS One	Lanzhou, People’s Republic of China	PRISMA	369
Marshman^Citation75	2010	Community Dental Health	Sheffield, UK	CONSORT	48
Moberg-Mogren^Citation76	2006	American Journal of Occupational Therapy	Cleveland, OH, USA	CONSORT	14
Moher^Citation47	2002	BMC Pediatrics	Ottawa, ON, Canada	CONSORT	251
Montané^Citation77	2010	BMC Clinical Pharmacology	Barcelona, Spain	CONSORT	92
Montgomery^Citation50	2011	Trials Journal	Bristol, UK	CONSORT	76
Norton-Mabus^Citation78	2008	OTJR: Occupation, Participation and Health	Toledo, OH, USA	CONSORT	30
Parsons^Citation79	2011	Journal of Bone and Joint Surgery. British Volume	Coventry, UK	CONSORT, STROBE	100
Piggott^Citation80	2004	Palliative Medicine	London, UK	CONSORT	93
Plint^Citation51	2006	Medical Journal of Australia	Ottawa, ON, Canada	CONSORT	8
Rios^Citation81	2008	Journal of Clinical Endocrinology and Metabolism	Hamilton, ON, Canada	CONSORT	89
Shea^Citation82	2006	The Journal of Rheumatology	Amsterdam, Netherlands	QUOROM	57
Strech^Citation83	2011	Journal of Clinical Psychiatry	Hannover, Germany	CONSORT	105
Thabane^Citation84	2007	International Journal of Obesity	Hamilton, ON, Canada	CONSORT	63
Vigna-Taglianti^Citation85	2006	Annals of Oncology	Torino, Italy	QUOROM	80
Walleser^Citation86	2011	Journal of Clinical Epidemiology	Renens, Switzerland	CONSORT	106
Wangge^Citation52	2010	PLoS One	Utrecht, Netherlands	CONSORT	232
Weir^Citation87	2012	International Journal of Medical Informatics	Salt Lake City, UT, USA	PRISMA, QUOROM	13
Willis^Citation88	2011	BMC Medical Research Methodology	Manchester, UK	PRISMA	236
Zhong^Citation89	2011	European Journal of Integrated Medicine	Chengdu, People’s Republic of China	CONSORT	153
Zintzaras^Citation53	2010	Clinical Therapeutics	Larisa, Greece	CONSORT	18
Ziogas^Citation90	2009	Annals of Epidemiology	Larisa, Greece	CONSORT	261

Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; QUOROM, Quality of Reporting of Meta-analysis; BMC, BioMed central; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; PLoS, Public Library of Science; PRISMA, preferred reporting items for systematic reviews and meta-analyses; OTJR, Occupational Therapy Journal of Research.

Adherence to reporting guideline

The adherence of the studies included in the systematic reviews to their respective guidelines, and the author’s conclusions, are shown in Table 2. Forty-three (86.0%) of the included studies concluded that the adherence to reporting guidelines was inadequate, poor, medium, or suboptimal, or that some improvement was needed. No combined, quantitative result was generated from the 50 systematic reviews due to differences in the measurement tools used by the individual reviews.

Table 2 Description of the studies’ findings

Guideline	First author	Measure of quality	Results mean (SD)/% count	Authors’ conclusion
CONSORT	Al-Namankany^Citation54	Modified 34-item CONSORT checklist	Compliance varied across items and articles. Good compliance of articles to CONSORT for introduction sections (96%–98%), discussion sections (96%–98%). Poor reporting in randomization methods (5%–9%), description of sample size calculation (4%), intention-to-treat analysis (1%)	Quality of reporting of RCTs in pediatric dental journals was generally poor, with negligible improvement after the publication of CONSORT Statement
	Areia^Citation21	Application of CONSORT/STARD	15.7 (2.2)	Level of adherence is medium for quality of reporting in diagnostic endoscopy
	Augestad^Citation44	CONSORT adherence, Jadad	30.75 (4), 40% of the trials had a Jadad score of ≥3 points	Level of adherence is low for quality of reporting for RCTs of disease specific clinical decision support
	Balasubramanian^Citation45	Modified CONSORT score, allocation concealment as assessed by Schulz et al, Jadad score	Medians of the modified CONSORT score were 85.45 (interquartile range 81.09–86.13) and interquartile range 68.97 (62.89–73.1 1) for RCTs from medical and surgical journals, respectively; 13% clearly explained allocation concealment; 37.7% of RCTs had a Jadad score of ≥ 3	Quality of reporting of surgical RCTs was suboptimal, and reporting in surgical journals was inferior to surgical trials in medical journals “We found that the quality of reporting of general surgical RCTs leaves considerable room for improvement”
	Bath^Citation55	In all, 33 criteria of the CONSORT Statement and 53 additional factors relevant to acute stroke or trials in general. Trial quality was also assessed with a seven-point scale	Median total report quality was 40/86 (range 15–61). Median CONSORT criterion was 19/33 (9–29)	Poor quality for acute stroke RCTs “We believe that authors should follow the CONSORT guidelines and that referees and editors should ensure this happens”
	Bian^Citation43	63-item revised CONSORT checklist designed for Chinese Herbal Medicine clinical studies	Median score of overall reporting quality was 32% (8%)	Overall quality of reporting of CHM RCTs was poor. Need to improve reporting in clinical trials in this area. “To improve the quality of reporting of RCTs of CHM, we recommend adopting a revised CONSORT checklist that includes items specific to CHM. We also recommend that editors of CHM journals require authors to use a structured approach to presenting their trials as a condition of publication”
	Bousquet^Citation57	Reporting of procedure, randomization, dropouts, strict conduct of intention-to-treat analysis, sample size calculation, which was assessed with eight items of the CONSORT Statement	Four of the 94 studies met the eight items of the CONSORT Statement criteria	RCTs in subcutaneous immunotherapy and sublingual immunotherapy had poor reporting quality. Encourage more use of CONSORT Statement
	Capili^Citation58	Adherence to CONSORT/presence of harms guidelines in RCTs	I7(range: 14–21) for CONSORT; 7/22 = 77.3% for harm	Level of adherence is bad for quality of reporting, for RCTs on acupuncture, for pain reduction
	Cavadas^Citation59	A 25-item 2010 revised CONSORT Statement	No combined data: only a few items were reported in less than 50% of the studies; some items were reported in more than 90% of the studies	“RCTs in [pelvic organ prolapsed] are scarce. The quality of reporting is suboptimal in many aspects and has not improved in recent years”
	Lu^Citation73	Percentage of articles that reported each applicable item of the CONSORT checklist	Sample size: only one (2.2%) of the papers mentioned sample size calculation. Randomization: 12 studies (26.1 %) were deemed to have authentic randomization. Blinding: 36 papers (78.3%) provided no information about blinding of either participants or investigators. Reporting of baseline characteristics: 39 papers (84.8%) reported the details of the baseline characteristics of participants. Length of follow-up: 22 papers (47.8%). There was no information provided on the length of time for which participants were followed. Loss-to-follow-up: a total of 36 studies (78.3%) failed to report dropout rates. Statistical reporting: only one paper (2.2%) did not report what statistical methods they had used	Findings indicate that the reporting quality of RCTs needs improvement for RCTs on the treatment of cancer pain in People’s Republic of China
	Chowers^Citation60	CONSORT guidelines for adverse events, adjusted to the design of the HAART trial	No combined score: harms were reported in only 24% of trials, 1/49 reported on adverse events collection method	Large variability and a lack of standard reporting of adverse events between trials; many trials did not adhere to CONSORT recommendations
	de Vries^Citation62	Adequate reporting of adverse drug reactions	Mean of 3, and 18% of articles scored 6 or higher	Insufficient reporting quality in adverse event reporting in RCTs of children
	Ethgen^Citation63	CLEAR NPT – a checklist to evaluate RCTs of nonpharmacological treatments	Most studies failed to report 8/12 quality indexes in the checklist. Reporting of generation of allocation sequence was adequate in 38.8% of studies, treatment allocation in 26.3%, intention-to-treat analysis in 70.0%	Inadequate reporting amongst trials involving stents. “The current reporting of results of RCTs testing stents needs to be improved to allow readers to appraise the risk of bias and the applicability of the results”
	Eyawo^Citation64	Revised CONSORT checklist to assess reporting of each items on the checklist in counts (percentage)	14/16 items ranged from 2%–7%; the other two items, sample size determination, and reporting of masking were reported in 72% and 75% of the articles	Deficiencies in the design, planning, and reporting of noninferiority and equivalence trials in ophthalmology literature
	Farrokhyar^Citation64	Modified CONSORT Statement and added factor relevant to surgical trials and CABG surgery	51.7 out of 105 (11.5)	The total reporting quality of trials in this review varied substantially between publications (35–96 out of a possible max score of 105). The results showed that there is a need for improvement in quality of reporting
	Froud^Citation48	Number and percentage of studies satisfying the revised 1l-item consort checklist	Most items were reported in an adequate percentage of studies; 5/11 reported in 78%–100% of the studies	Their results suggest that cluster randomized trials in oral health are of reasonable quality with respect to the key criteria of accounting for clustering in the design and analysis
	Fung^Citation49 (CONSORT and STROBE) Gagnier^Citation66	Presence or absence of CONSORT (maximum score of 37 points) statement indicators Mean CONSORT score based on 42 items and the percentage of items reported	CONSORT: median and mean values of 89% and 83%, respectively 18.92 out of 42 (5.54), and 45% of items were reported across all trials	Overall level of reporting is acceptable and has improved since the creation of CONSORT and STROBE “We found that reports of RCTs of herbal medicine interventions reported less than half of the necessary information in their published results” Overall adherence is low
	Halpern^Citation67	Percentage of articles that reported each applicable item of the modified CONSORT checklist and count of articles complying with modified CONSORT items	In the 23 articles in Anesthesia and Analgesia, the median percentage of correct CONSORT items was 63%	Poor – total number of items that are inadequately reported is high in the current RCT literature with obstetric anesthesia
	Herdan^Citation69	22-item CONSORT checklist expressed	On average 87.4% of the CONSORT items were reported	The reporting quality has improved significantly in the period after dissemination of the CONSORT Statement; however, reporting of adverse events needs attention
	Kiehna^Citation46	Quality of reporting score using CONSORT (max score of 44) and Jadad score out of 5 points	26.4 out of 44 (range: 17–38)/67% of studies had no description or the prestudy sample size calculation, 63% did not describe whether subjects, treatment providers or assessors/analysts were blinded	The quality of reporting of RCTs in neurosurgical journals remains suboptimal
	Kober^Citation71	CONSORT criteria based on a 14-item questionnaire	75% of studies reported only six of the 13 items; only 14% reported randomization process; only 13% provided details about concealment of allocation; only 13% provided a statement on study power; only 12% used intention-to-treat analysis	Articles of Hodgkin’s lymphoma published after 1996 do not conform to the CONSORT recommendations
	Ladd^Citation25	Assessment of 36 of the items from the CONSORT Statement based on a score out of 36	24.43 out of 36 (3.27)	The overall level of adherence to CONSORT has improved since 1994, and continues to remain highest among studies that have been published within journals that have adopted the CONSORT guidelines
	Li^Citation72	Score out of 40 based on a 40-item modified checklist based on the CONSORT Statement	42% of the studies included explained how sample size was determined; 14% of studies described whether or not outcome assessors were blinded	The reporting quality of these trials is suboptimal and substantial improvement is required to meet the CONSORT guidelines. Almost 50% of the trials we reviewed did not satisfy more than half of the criteria in the modified CONSORT checklist, and only 23% of RCTs provided adequate details of Tai Chi intervention used in the trials
	Marshman^Citation75	56 criteria based on the CONSORT Statement	27/56, with variation between journals (23.2 to 27.7)	Poor adherence to the CONSORT checklist in RCTs in dental health
	Moberg-Mogren^Citation76	Average NMECI score (0–201 subitems scale)	104.2(32.9)	Less than half of the articles met criteria of these subitems in selected RCTs relevant to occupational therapy
	Moher^Citation47	CONSORT checklist, frequency of unclear allocation concealment, and a five-point quality assessment instrument (Jadad)	12.7/32 of the CONSORT checklist included; 81.3% unclear allocation concealment; 1.9/5 for the Jadad assessment scale	Overall, there was no difference in the PedCAM RCTs and conventional medicine quality, with both types achieving 43% of their maximum possible outcome
	Montané^Citation77	Revised CONSORT checklist, 22 items	10.5(2.7)	Quality was good in 23 (25%) of the articles and poor in 69 (75%) of the reports for RCTs on the efficacy of analgesic drugs in postoperative pain after TOS
	Montgomery^Citation50	Qualitative look	N/A	Varying level of reporting quality factorial trials of complex interventions in community settings
	Norton-Mabus^Citation78	NMNECI (2l2subitems)	119.5(25.48)	Article consistency with CONSORT Statement was less than 60%. Occupational therapy RCT had higher consistency with the instrument, scoring higher than articles in speech therapy
	Parsons^Citation79 (combined CONSORT and STROBE guideline)	Overall compliance calculated as the weighted mean of the compliance rates for the seven selected journals, using a previously made questionnaire	59% (CONSORT)	Very few papers fulfilling all criteria; general lack of statistical rigor
	Piggott^Citation80	Compared RCTs of three different time period cohorts, with the CONSORT (condensed, 13-item) checklist	Quality of reporting variable; 30% of trials or less used true randomization, allocation concealment, intention-to-treat analysis, and power calculations	Quality of reporting over time cohorts was variable, no consistent improvement over time. Quality of reporting remains poor for RCTs in specialized palliative care literature
	Plint^Citation51	22-item checklist from the CONSORT Statement	Standardized mean difference between CONSORT-adopting journals and nonadopters was 0.83 (95% CI, 0.46–1.19)	Journal adoption of CONSORT is associated with improved reporting of RCTs
	Rios^Citation81	Overall quality score, which is a 15 point overall reporting quality score made from CONSORT checklist	10(2.03)	Suboptimal reporting quality in an endocrine journal
	Strech^Citation83	A checklist based on the CONSORT Statement	There are 72 items on the checklist; 42% were reported adequately and 25% were reported inadequately	While some trial-related information is well reported, a good part of the reporting quality of RCTs in bipolar disorder falls well below the required and practically feasible level for many aspects essential for the adequate interpretation of methodological quality and clinical relevance. Authors should be further encouraged to follow the CONSORT criteria. No consistent trend could be shown for improvement in the quality of reporting over time, or for reporting essential methodological items differently. There is a consistent trend toward better reporting in journals that endorse the URM
	Thabane^Citation84	Percentage of studies satisfying each of the 44 CONSORT criteria	26.25 (4.51) and 60% adherence for reporting criteria: 90% satisfied criteria for the introduction; 19% for the methods; 75% the study protocol, 70% for the results	Overall, the quality of reporting is suboptimal in RCTs of weight loss intervention. Key reporting criteria that may impact the validity and generalizability of the results were adequately reported
	Walleser^Citation86	CONSORT-CRT	34% inadequately reporting on more than half of the CONSORT-CRT criteria	The quality of reporting in CRTs needs improvement. This will hopefully improve implementation and planning
	Wangge^Citation52	Extension of the CONSORT Statement for noninferiority and equivalence trial	No blinding in 34.0%, noninferiority margin in 97.8%, with only 45.7% reporting the method of determining the margin	Adherence improved slightly after CONSORT for noninferiority trials
	Zhong^Citation89	Number of studies describing each of the 38 modified consort items	Of the 38 CONSORT items, only five items were described in more than 80% of the 153 included	Adherence was suboptimal for two-group parallel randomized controlled clinical trials of multiherb formulae
	Zintzaras^Citation53	17-item CONSORT checklist	17 CONSORT checklist items were reported in 7/18 studies, and 9/17 CONSORT checklist items were reported in all 18/18 studies	Proper assessment of the credibility and generalizability of the results can be ensured by reporting quality
	Ziogas^Citation50	24-item questionnaire based on the CONSORT checklist	75% of the studies addressed 13 out of the 24 items of the CONSORT Statement	Reporting on myeloid malignancies remains unsatisfactory and requires further improvement to properly assess the validity of clinical research
PRISMA	Ma^Citation74	Adherence to PRISMA checklist items (27 items)	Title, introduction, limitations, and conclusions were reported well in 90% or more of the studies. Most other items varied from 30%–70% of the studies	Compliance with PRISMA reporting guidelines is low for systematic reviews on TCM published in Chinese journals
	Weir^Citation87 (combined PRISMA and QUOROM guideline)	An integrated score consisting of the number of items completed over the total numbers of items on both the PRISMA and QUOROM criteria, resulting in cored ranking from 0% to 100% (excluding the items focused on in the abstract)	Mean = 63% (range 45%–81%) on a scale of 0%–100%	Systematic reviews of empirical computerized provider order-entry research had moderate quality
	Willis^Citation88	Adherence to the 27-item PRISMA checklist	Of the 236 meta-analyses included following selection: 1% reported the study protocol; 25% reported the searches used; 32% reported the results of a risk of bias assessment; and 35% reported the abstract as a structured summary	Compliance with the PRISMA statement was generally poor; none of the review completely adhered to all 27 checklist items for the published meta-analyses of diagnostic tests
QUOROM	Bereza^Citation56	18-item QUOROM checklist, ten-item checklist OQAQ used for scientific quality	61% ± 19% (median 60%, range 39%–94%) for the QUOROM checklist. 58% ± 28% for OQAQ	“Reporting/scientific quality was considered less than fair-to-good. Stakeholders should strive for higher scientific quality of meta-analyses”
	Hemels^Citation68	18-item QUOROM checklist	On average 50.2% of the CONSORT items were reported	The overall quality of reporting in the meta-analysis of RCTs in major depressive disorder was marginally acceptable
	Junhua^Citation70	18-item QUOROM checklist, ten-item checklist OQAQ used for scientific quality	No combined score; methodological and reporting flaws in more than half of the review articles. Flaws were mainly in the literature search, characteristics of included and excluded studies, quality assessment of primary trials, and data merging	Methodology and reporting quality are poor in both systematic reviews and meta-analysis of TCM published in journals in the People’s Republic of China
	Shea^Citation82	18-item QUOROM checklist, ten-item checklist OQAQ used for scientific quality	All systematic reviews were found to have good overall quality. OQAQ mean score was 5.02 (95% CI 3.71–6.32)	Reporting quality of Cochrane musculoskeletal systematic reviews was generally good, with room for improvement
	Vigna-Taglianti^Citation85	QUOROM-based checklist (score out of 50)	29.9/50	“Oncologists should be aware that they could be relying on poor underlying documents. Writing groups should be aware of methodological problems, and should consult the existing manuals for the preparation of guidelines”
	Weir^Citation87 (combined PRISMA and QUOROM guideline)	An integrated score consisting of the number of items completed over the total numbers of items on both the PRISMA and QUOROM criteria, resulting in cored ranking from 0% to 100% (excluding the items focused on in the abstract)	63% (range 45%–81%) on a scale of 0%–100%	Systematic reviews of empirical computerized provider order-entry research were of moderate quality
STROBE	Cook^Citation61	Quality of reporting, methodological quality, and the association between methodological quality and effect size	253 (90)	Reporting the quality of experimental research on health profession education was found to be generally suboptimal
	Fung^Citation45 (combined CONSORT and STROBE guideline)	STROBE (maximum score of 37 points) statement indicators	STROBE mean and median: 70% and 71%, respectively	Overall level of reporting is acceptable and has improved since the creation of CONSORT and STROBE
	Parsons^Citation79 (combined CONSORT and STROBE guideline)	Weighted mean of the compliance rates for the seven selected journals, using a previously made questionnaire	58% (strobe)	Very few papers fulfilling all criteria, general lack of statistical rigor

Abbreviations: SD, standard deviation; CONSORT, Consolidated Standards of Reporting Trials; RCT, randomized controlled trial; CHM, Chinese herbal medicine; HAART, highly active antiretroviral treatment; CLEAR NPT, checklist to evaluate a report of a nonpharmacological trial; CABG, coronary artery bypass surgery; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; NMECI, Nelson-Moberg Expanded CONSORT Consolidated Standards of Reporting Trials Instrument; PedCAM, Pediatric Complementary and Alternative Medicine Research and Education Network; TOS, thoracic outlet syndrome; N/A, not applicable; NMENCI, Nelson-Moberg, Norton Expanded Consolidated Standards of Reporting Trials Instrument; CI, confidence interval; URM, uniform requirement for manuscript; CRT, consolidated standard of reporting trial; NI, noninferiority; PRISMA, preferred reporting items for systematic reviews and meta-analyses; QUOROM, Quality of Reporting of Meta-analysis; OQAQ, Overview Quality Assessment Questionnaire; TCM, traditional Chinese medicine.

CONSORT Statement

The adherence of RCTs to the CONSORT Statement was assessed with different versions of the CONSORT checklist. These checklists ranged from eight to 63 items, except for two studies that used the 212 subitem, Nelson–Moberg–Norton Expanded CONSORT instrument and the 201 subitem Nelson–Moberg Expanded CONSORT instrument. The revisions of the CONSORT Statements were usually based on the specific field of the RCT, and the applicability of the items on the CONSORT checklist to that field. For instance, Bian et al⁴³ used a revised 63-item CONSORT checklist designed for Chinese Herbal Medicine clinical trials. In addition to the CONSORT checklist, four studies (Augestad et al,⁴⁴ Balasubramanian et al,⁴⁵ Kiehna et al,⁴⁶ and Moher et al)⁴⁷ also used the five-point Jadad instrument to assess the quality of the individual RCTs.³⁹

Of the 41 systematic reviews assessing RCTs reporting adherence to the CONSORT Statement, 33 (80%) of them concluded that some improvement was needed, or that the reporting quality was inadequate, poor, medium, or suboptimal (Table 3). Furthermore, the authors recommended the use of the CONSORT Statement as a guideline to improve the quality of reporting of RCTs. Eight studies did not report inadequate reporting quality of RCTs. Froud et al⁴⁸ concluded that cluster randomized trials in oral health had a reasonable quality. Fung et al⁴⁹ reported that the overall level of reporting was acceptable and the reporting quality has improved since the creation of CONSORT and STROBE statements. Ladd et al²⁵ also concluded that the overall reporting quality had improved since 1994 and the articles published in journals that endorse the CONSORT Statement had the highest levels of adherence to reporting guidelines. Moher et al⁴⁷ only compared the quality of pediatric complementary and alternative medicine RCTs and reported 40% of the CONSORT checklist items were included in these RCTs. Montgomery et al⁵⁰ evaluated the RCTs qualitatively and found that there was a varying level of reporting quality in factorial trials of complex interventions in community settings. Plint et al⁵¹ compared RCTs from CONSORT-endorsing and nonendorsing journals, and their results suggested some improvement in the quality of reporting when the CONSORT checklist is used. Wangge et al⁵² suggested that adherence to reporting guidelines for noninferiority trials have improved slightly since the CONSORT Statement has been published. Lastly, Zintzaras et al⁵³ did not comment directly on an overall quality of reporting and concluded that adhering to reporting standards can ensure proper assessment of the results.

Table 3 Studies’ conclusions

Type of guideline	Total number of studies	Number of studies concluding that “some improvements are needed, reporting inadequate, poor, medium, suboptimal, etc”
CONSORT	41 (two combined study with both CONSORT and STROBE)	33 (80%)^Citation21,Citation43^–^Citation46,Citation54,Citation55,Citation57^–^Citation60,Citation62^–^Citation67,Citation69,Citation71^–^Citation73,Citation75^–^Citation81,Citation83,Citation84,Citation86,Citation89,Citation90
PRISMA	3 (one combined study with both PRISMA and QUOROM)	3 (100%)^Citation74,Citation87,Citation88
QUOROM	6 (one combined study with both PRISMA and QUOROM)	3 (50%)^Citation56,Citation70,Citation87
STROBE	3 (two combined studies with both CONSORT and STROBE)	2 (67%)^Citation61,Citation79
All guidelines	50 (distinct studies)	43 (86.0%)^Citation21,Citation43^–^Citation46,Citation54^–^Citation67,Citation69^–^Citation81,Citation83,Citation84,Citation86^–^Citation90

Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; QUOROM, Quality of Reporting of Meta-analysis.

PRISMA, QUOROM, and STROBE statements

Three studies examined adherence to the PRISMA guidelines, and all concluded that the adherence of the assessed systematic reviews was poor or moderate. Ma et al⁷⁴ and Willis et al⁸⁸ used the 27-item PRISMA checklist to assess the level of adherence. Ma et al⁷⁴ found that systematic reviews on traditional Chinese medicine published in Chinese journals had low adherence to the PRISMA checklist. Willis et al⁸⁸ also concluded that adherence to the PRISMA checklist was generally poor for published meta-analyses of diagnostic tests. Weir et al⁸⁷ used an integrated score consisting of both the PRISMA and QUOROM criteria and found that systematic reviews of empirical computerized provider order-entry research were only of moderate quality.

The assessment of studies’ adherence to the QUOROM guideline was done with the 18-item QUOROM checklist coupled with a ten-item OQAQ checklist in three studies. Bereza et al⁵⁶ and Junhua et al⁷⁰ reported that there was a need to improve the quality of reporting of reviews, while Shea et al⁸² concluded that the quality of Cochrane musculoskeletal systematic reviews was good. Hemels et al⁶⁸ used only the QUOROM checklist and they concluded that the quality of meta-analyses in studies on major depressive disorder was marginally acceptable. Vigna-Taglianti et al⁸⁵ used the QUOROM checklist with a specific weighting system for each of the headings and the average score was 29.9/50. No conclusions concerning adherence were made, although the authors did recommend the use of manuals to prepare guidelines for the management of breast and colon cancers. Lastly, as described in the previous paragraph, Weir et al⁸⁷ used an integrated score containing both PRISMA and QUOROM criteria.

The studies by Fung et al⁴⁹ and Parsons et al⁷⁹ assessed the adherence of both RCTs and observational studies to their respective guidelines. Parsons et al⁷⁹ found there was a general lack of statistical rigor.

Factors associated with adherence to reporting guidelines

Although we included systematic reviews assessing the adherence of research articles to four different guidelines, only systematic reviews related to the CONSORT Statement reported on the factors that were associated with adherence to the guideline (Table 4). The exception was Hemel et al,⁶⁸ who concluded that the overall quality of reporting of meta-analyses using the QUOROM guidelines did not significantly change over time, and that the year of publication was not associated with change in adherence. From the CONSORT-related studies, the following are the factors that were reported to be significantly associated with an increase in adherence to the CONSORT Statement or to the quality of reporting of RCTs, as well as the number of studies reporting these factors: publication in CONSORT-endorsing journals (3); declared funding source (1); high impact factor (3); industrial funding (1); multicenter studies (1); non-Chinese reports (compared to those published in mainland China) (1); number of authors (1); reporting of allocation concealment (1); reporting in a medical journal (1); reporting method of sequence generation (1); sample size (3); trial quality (1); type of intervention (pharmacologic intervention versus nonpharmacologic intervention); and year of publication (before and after CONSORT) (9). These factors are summarized in Table 4. Having a positive outcome in RCTs (compared to a neutral or negative outcome) was the only factor reported to be significantly associated with a decrease in adherence to the CONSORT Statement (Spearman correlation = −0.192; 95% CI, −0.351 to −0.011).⁵⁵ Other factors that reported but did not reach statistical significance for an association with adherence to the CONSORT Statement are also summarized in Table 4.

Table 4 Factors associated with reporting quality of articles using the CONSORT guideline

First author	Sample size	Factors associated with adherence ↑↓
Al-Namankany^Citation54	173	1. Year of publication (↑)
Areia^Citation21	120	1. Publication in CONSORT-endorsing journals (↑) 2. Year of publication (↑)
Balasubramanian^Citation45	69	1. Number of authors (↑)* 2. Multicenter studies (↑)* 3. Declared funding source (↑)* 4. Reporting in medical journals (↑)*
Bath^Citation55	114	1. Trial quality (↑)* 2. Trials with positive outcome (↓)* 3. Year of publication (↑)*
Capili^Citation58	10	1. Journal requiring the use of CONSORT (↑)
Chowers^Citation60	49	1. Industry sponsored trials (industry sponsored versus nonindustry sponsored trial) (↑) 2. Year of publication (↑)*
de Vries^Citation62	107	1. Sponsoring (↑)
Ethgen^Citation63	132	1. Impact factor (↑)* 2. Publication in CONSORT-endorsing journals (↑)*
Farrokhyar^Citation65	50	1. Sample size (↑)* 2. Year of publication (more recent publication year [up to 2005] [2001, P = 0.822; 2002, P < 0.001; 2003, P = 0.204; 2004, P < 0.001; 2005, P < 0.001]) 3. Location of the study (UK, P = 0.900; Scandinavia, P = 0.002; Other, P = 0.003) 4. Source of funding (↓) 5. Type of primary outcome in the study-categorical (↓)
Herdan^Citation69	37	1. Year of publication (↑)*
Kiehna^Citation46	27	1. Publication in CONSORT-endorsing journals (↑)*
Ladd^Citation25	127	1. Year of publication (↑)*
Moberg-Mogren^Citation76	14	1. Year of publication (↑)*
Montané^Citation77	92	1. Year of publication (↑)* 2. Impact factor (↑)* 3. Studies with placebo control group (↑)
Montgomery^Citation50	76	1. Year of publication (↑)*
Plint^Citation51	8	1. Reporting method of sequence generation (↑)* 2. Allocation concealment (↑)* 3. Overall consort items (↑)
Rios^Citation81	89	1. Industrial funding (↑)* 2. Journal of publication (publication in JCEM) (↑)* 3. Sample size (↑)*
Thabane^Citation84	63	1. Sample sizes (↑)* 2. Year of publication (↑)* 3. Type of intervention (pharmacologic intervention versus nonpharmacologic intervention) (↑)*
Zhong^Citation89	153	1. Non-Chinese reports (compared to those published in mainland China) (↑)* 2. Publication in CONSORT-endorsing journals (↑)*
Ziogas^Citation90	261	1. Year of publication (↑)* 2. Impact factor (↑)*

Notes:

* Statistically significant increase/decrease, P ≤ 0.05;

(↑) positively associated with adherence;

(↓) negatively associated with adherence.

Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; JCEM, The Journal of Clinical Endocrinology and Metabolism.

Quality of included studies, measured by the modified OQAQ/AMSTAR checklist

The global score of each of the studies is listed in Table 5. The mean global score of the 50 included studies was 16.6 ± 2.4. Twenty-one (42%) out of the 50 studies had a global score of 17 or more. The items with the lowest scores were question 5, “Was information on included and excluded studies provided?” and question 6, “Were the characteristics of included studies provided?” with only 16% and 32% of the studies reporting each of these items correctly, respectively.

Table 5 Reporting quality of the 50 included systematic reviews, assessed by the modified AMSTAR/OQAQ (ten items, score out of 20)

First author	Global score
Al-Namankany^Citation54	15
Areia^Citation21	18
Augestad^Citation44	20
Balasubramanian^Citation45	16
Bath^Citation55	16
Bereza^Citation56	20
Bian^Citation43	15
Bousquet^Citation57	18
Capili^Citation58	15
Cavadas^Citation59	17
Lu^Citation73	18
Chowers^Citation60	12
Cook^Citation61	18
de Vries^Citation62	14
Ethgen^Citation63	13
Eyawo^Citation64	18
Farrokhyar^Citation65	19
Froud^Citation48	16
Fung^Citation49	17
Gagnier^Citation66	16
Halpern^Citation67	14
Hemels^Citation68	19
Herdan^Citation69	15
Junhua^Citation70	13
Kiehna^Citation46	16
Kober^Citation71	17
Ladd^Citation25	19
Li^Citation72	18
Ma^Citation74	19
Marshman^Citation75	14
Moberg-Mogren^Citation76	16
Moher^Citation47	14
Montané^Citation77	15
Montgomery^Citation50	17
Norton-Mabus^Citation78	10
Parsons^Citation79	17
Piggott^Citation80	14
Plint^Citation51	18
Rios^Citation81	20
Shea^Citation82	19
Strech^Citation83	18
Thabane^Citation84	19
Vigna-Talianti^Citation85	15
Walleser^Citation86	19
Wangge^Citation52	12
Weir^Citation87	20
Willis^Citation88	20
Zhong^Citation89	17
Zintzaras^Citation53	18
Ziogas^Citation90	15

Abbreviations: AMSTAR, assessment of multiple systematic reviews; OQAQ, Overview Quality Assessment Questionnaire.

Discussion

We undertook a systematic scoping review of systematic reviews to investigate the adherence to reporting guidelines that included the CONSORT, PRISMA, QUOROM, TREND, MOOSE, and STROBE statements. Our systematic review included 50 studies that fulfilled our inclusion criteria, most of which originated from North American and European countries (43/50 studies). Despite the widespread acceptance of the CONSORT Statement and its subsequent extensions, the standards of reporting of clinical studies remained suboptimal. Our study showed that 86.0% of the systematic reviews included in this study concluded that there was a suboptimal quality of reporting across multidisciplinary clinical research topics using different study designs including RCTs and observational studies. The adherence of the assessed studies to reporting standards were not specific to any field of clinical research, but rather spanned across various disciplines including diagnostic procedures, interventions, cancer trials, and alternative medicine, implying the widespread lack of adherence to reporting guidelines in the medical literature. Despite the availability of guidelines and operational definitions of how to use these guidelines to improve reporting and transparency of clinical literature (including providing checklists, flow diagrams, and explicit methods of recruitment and allocation¹²), the uptake of these guidelines remained low. Several shortcomings of the reporting standards of clinical literature include inadequate reporting of the methods, selective reporting of the results, or misinterpretation of the results.⁹¹ Studies have shown that the use of these guidelines was associated with better reporting of studies of acupuncture trials,⁹² and only minimal improvement in the adherence to reporting guidelines of studies that investigated diagnostic accuracy.⁹³ It is possible that the lack of adherence may relate to the narrow focus of these guidelines on specific clinical areas or study designs, and therefore further guidelines need to be developed. Such new guidelines can be developed based on sets of tools and criteria, as proposed previously.⁹⁴ The poor adherence to reporting guidelines seen in the clinical literature is also seen in other settings including the failure to follow the National Institute of Health guidelines for reporting sex and ethnicity in clinical trials.⁹⁵ Efforts to address the gap between the standards set by the guidelines and the actual standards of the published literature are therefore needed.

The most striking observation from our study was the lack of consistency in methods of recording the adherence to the reporting guidelines, and therefore it was not possible to combine the results to provide a summary statistic. This highlights the need for a consensus statement on the reporting of methodological quality of studies addressing the adherence to CONSORT and other statements.

Despite the suboptimal adherence to reporting guidelines in most of the studies reviewed, we observed that RCTs have a better adherence to reporting standards than non-RCTs. In addition, studies published in journals endorsing the CONSORT Statement have higher adherence to reporting standards. Not surprisingly, studies published after the introduction of CONSORT showed a better reporting quality and adherence to reporting guidelines. These findings are encouraging and provide a platform to disseminate knowledge generated by this study to multiple disciplines in health research to stress the need for improvement in adherence to reporting guidelines.

The strengths of our study are that we conducted a rigorous systematic review and included studies investigating the quality of reporting across various clinical areas of research, thus adding a scoping review methodology to a systematic review. We have also extracted relevant data and attempted to provide a summary statistic; however, the diversity of the findings did not allow for the computation of results.

Our study results are limited by the lack of reviews addressing adherence standards to other guidelines (MOOSE, TREND, QUOROM), the inability to combine the overall study findings, and the unavailability of tools designed to assess the quality of systematic reviews investigating methodological quality. Furthermore, the design, conduct, analysis, and reporting of the results of the reviews including definitions of outcomes (and predictor variables) varied substantially within and between the guidelines. This is mainly due to the lack of an established framework or standard for the conduct and reporting of reviews assessing the adherence to guidelines.

The study findings are nonetheless important for educators, authors, editors, sponsors, health consumers, and research ethics boards.

Summary and recommendations

Factors that are associated with reporting standards can be grouped into four categories:

• Study design: Better reporting standards were seen in studies with large sample sizes; RCT design; transparency in reporting randomization, adverse events, and secondary outcomes; and studies of drug interventions.

• Timing of publication: Studies that were published more recently were associated with better quality of reporting.

• Study sponsor: Studies with an industrial sponsor were also associated with a better quality of reporting.

• Journal: Journals with a high impact factor and those endorsing the CONSORT Statement and its extensions tended to publish studies with better adherence to reporting standards.

Recommendations for educators

Educators are at the forefront of teaching research methodology and applications in clinical settings, and therefore they play an important role in improving the reporting standards of clinical literature. Educators need to emphasize the importance of reporting standards and incorporate the guidelines in research training. They also need to provide ongoing training through workshops at professional meetings, and highlight the factors shown to improve the quality of reporting to foster improved reporting standards of the clinical literature.

Recommendations for authors

Authors should use the reporting standards appropriate to the study design as a guide to planning and reporting studies, and provide a flow diagram and checklist that will not only improve the reporting standard and adherence to guidelines, but will also help with transparency and reproducibility of the study. The use of the guidelines will also help to minimize reporting bias. For resources on using reporting standards, see the EQUATOR Network website.¹²

Recommendations for editors

Studies published in journals endorsing the CONSORT and its extensions were described as having better reporting quality and increased adherence to guidelines. Therefore, editors must endorse the reporting standards as part of their journal editorial policy.

Furthermore, inclusion of the respective guideline checklist must also be part of the editorial policy. Editors need to consider assessing the adherence to reporting guidelines as a requirement for peer review, and they should revise the peer review process to incorporate these assessments.

Recommendations for sponsors

Sponsors can ensure that the quality of the study methodology and transparency are meeting these standards by requesting adherence to the respective reporting guidelines appropriate for the study design.

Recommendations for research ethics boards

Institutional Review Boards or Research Ethics Boards have a substantial responsibility to ensure ethical and sound methodological quality of clinical studies. Therefore, we recommend that Institutional Review Boards/Research Ethics Boards require that protocols be submitted for ethical approval to clearly state what reporting standards the study will be using based on the study design, and that reporting guidelines checklist are part of the application for ethics approval.

Recommendation for health consumers

In accordance with the general principles of evidence-based health care practice,⁹⁶ we encourage consumers or health care users to be actively involved in their health care by discussing their care options with their providers. Understanding information presented in published studies can be an important ingredient in these discussions. We suggest that health care users consider the evaluation of the quality of the information presented in the literature by looking for a guideline statement and a checklist to ensure the study reporting followed a certain standard that is appropriate for the particular study design.

Lastly, one element that all parties need to take into consideration is the importance of conducting large studies. Large studies have been shown to have a better quality of reporting.^81,84,97 Large studies are also less prone to problems of bias and have better precision.

Conclusion

Reporting guidelines help to improve the quality and transparency of clinical studies and allow for systematic reviews and meta-analyses to provide evidence worthy of changing practice, improving knowledge, and better management of health and disease. The current reporting standards and adherence to guidelines are poor and are in need of major improvement. Steps need to be taken by all involved in the conducting and reporting of clinical research in order to achieve better standards of reporting, thus minimizing bias and providing reproducible studies that can be combined to reach conclusive evidence.

Acknowledgments

Special thanks to Neera Bhatnagar for the detailed discussion on literature search strategies. This study was funded in part by funds from the CANNeCTIN program.

ZS designed and wrote the first draft of the protocol and current manuscript; LM, DK, VBD, RD, SZ, VF, BD, and MB contributed to the protocol design and writing of the manuscript; LT conceived of the idea for the study and contributed to the design and writing of the manuscript. All authors read and approved the final draft.

Disclosure

The authors report no conflicts of interest in this work.