Quality Assessment of Systematic Reviews Performed As An Undergraduate thesis in A Peruvian University: A Systematic Review

Abstract

Objective

To determine the quality of systematic reviews submitted as a thesis in the Medical School of Ricardo Palma University.

Methods

We conducted a systematic review. We included systematic reviews submitted as theses from Ricardo Palma University, and we excluded narrative reviews, editorials, clinical experiments, and those with incomplete data. We performed a structured search on EMBASE, PubMed, Scopus, and Institutional Repository from the Ricardo Palma University and RENATI. The risk of bias assessment was performed through the AMSTAR-2 and the modified AMSTAR-2 tools. The primary outcome was review quality. A qualitative synthesis of the information was performed.

Results

One thousand four hundred eighty-seven theses were identified, and exclusion criteria were applied, whereby 11 theses were selected for review and thorough consultation. Of the 11 selected theses, and through the AMSTAR-2 and modified AMSTAR-2 tools, the findings reached were that 90.9% of the included theses presented critically low quality that was not modified even when the quality was reevaluated after its publication as a scientific article.

Conclusion

The systematic reviews presented as undergraduate thesis in the Medical School of Ricardo Palma University showed low and critically low quality. Improvement in systematic review training is required for both students and institutional advisors.

Keywords:

• thesis
• systematic review
• medical school

Introduction

Systematic reviews (SR) consist of straightforward and schematic summaries of information available oriented towards answering specific clinical questions.¹ They represent high level of evidence due to the multiple sources of information included. Additionally, systematic reviews count on transparent and comprehensible descriptions from the drafting process to critically collect, select, evaluate, and summarize all available evidence regarding new treatment, diagnosis, and prognosis.¹ For this, they use explicit and systematic methods selected to minimize bias in order to provide more reliable results from where to extract conclusions and make critical clinical decisions.² At the Ricardo Palma University of Lima-Peru, medical students, within the educational curriculum, take a course called “Thesis Preparation Workshop” during their fifth year of medical school, intended mainly for planning and monitoring the Development of their Thesis Project. In the last year of the undergraduate degree, students receive personalized advice for the execution and delivery of their thesis.

The quality is evaluated through A Measurement Tool to Assess Systematic Reviews (AMSTAR) tool, which was developed to evaluate SR of randomized trials, which allows for a more detailed evaluation of SR.^2,3 Although a more extensive validation has not been carried out for other study designs, this instrument follows critical steps in comprehensively undertaking SR.

These days, in our faculty, we are witnessing a notable increase in the production of systemic reviews submitted as undergraduate degree theses. However, it is worth noting that despite their undeniable potential, they still face a moderate reception in the academic field when presented for this purpose.⁴ It is crucial to comprehensively evaluate these systematic reviews to obtain a situation analysis that leads to future improvements in our research processes using this methodology. Accordingly, our study aimed to evaluate the quality of systematic reviews submitted as undergraduate thesis at the Medical School of Ricardo Palma University.

Methods

General Design

We conducted a systematic review following the criteria established in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) statement. Additionally, the protocol was registered in PROSPERO (CRD42023435277).

Eligibility Criteria

We included undergraduate theses written as systemic reviews from the Medical School at Ricardo Palma University. There was no language restriction, and clinical experiments, observational studies, and dissertations were excluded.

Information Sources

A comprehensive systematic search was carried out, which began in the Institutional Repository of Ricardo Palma University and the National Theses Registry (RENATI) of Peru, to find submitted theses. Subsequently, using the keywords derived from the theses found, we searched its published versions in known databases such as EMBASE, MEDLINE, SCOPUS, and Scielo, from its inception to present day (July 10, 2023).

Search Strategy

For the search we used the following terms: Systematic Review and Medical School in the Institutional Repository and RENATI. The details are presented in (Annex 1).

Selection of Studies

Two researchers (ALCM and AACA) independently reviewed and selected the studies. Those duplicated or presented insufficient data for the qualitative synthesis were excluded. If there was any discrepancy between the authors (ACLM and AACA), a third researcher (RPR) made the final decision.

Data Extraction Process

Two researchers (CGR and ERT) independently carried out the data review and extraction of the studies. In case there was any discrepancy among them, we counted on the assistance of a third author (RPR) to make the final decision. The extracted data was registered in a base designed based on the objectives, including information such as primary author, year of publication, search period, total number of studies and patients included, population characteristics, and instrument used to evaluate the quality of the studies selected.

Quality Assessment

To evaluate the quality of the systematic review, we used the instrument AMSTAR-2. (Annex 2) For the systematic review of observational studies, we used AMSTAR-2 modified according to that reported by Santos-Marques et al.² AMSTAR-2 gives us the following categories as a result:

• High: No or one non-critical weakness. The SR provides an accurate and comprehensive summary of the results of the available studies.

• Moderate: No critical weakness and more than one non-critical weakness. The SR has weaknesses but no critical flaws and may provide an accurate summary of the results of the available studies.

• Low: Up to one critical flaw, with or without non-critical weaknesses. The SR may not provide an accurate and comprehensive summary of the available studies.

• Critically Low: More than one critical flaw with or without non-critical weaknesses. The SR is not reliable.

Quality was reevaluated after its scientific publication.

We chose this instrument given that it allows us to evaluate Quality objectively in comparison to other instruments, reducing subjectivity of the evaluation (Joanna Briggs, CAPS).

Synthesis of the Results

Given the review’s focus, we opted to exclusively conduct a qualitative information synthesis.

Results

Selection of Studies

We found a total of 1488 theses. We finally included 11 theses available^5–15 (Figure 1).

Figure 1 Flow chart.

Characteristics of Studies

Three SRs only included clinical trials, and eight were observational studies.^5–15 (Table 1)

Table 1 General Characteristics of the Studies Included in the Review

N°	Reference	Publication Year	Type of Review	Search period	Number of studies included	Number of patients included	Population characteristics (patients’ diagnosis)	Report	Instrument used for the quality evaluation of the chosen studies
	[5]	2022	Systematic review with meta-analysis of observational studies	Not specified	5 studies	30,017	Obesity	Vitamin D with obesity according to body mass index (OR=1.36; 95% CI 1.04 to 1.77) / Obesity according to blood pressure (OR = 1.74; 95% CI 1.26 to 2.40)	Modified New Castle Ottawa risk of bias tool
	[6]	2023	Systematic review with meta-analysis of observational studies	Not specified	7 studies	4129	Anemia / Helicobacter pylori infection	Odds Ratio (OR) of 2.20 and a confidence Interval of 95% from 1.21 to 3.99.	Adapted Newcastle Ottawa quality assessment scale
	[7]	2023	Systematic review and meta-analysis of randomized clinical trials	October 2021 (randomized clinical trials published during the period 2011–2021.)	5 randomized clinical trials	420 patients	Schizophrenia	General symptoms (DME = −0.69, CI 95%: −1.07 to −0.31; p = 0.0004; I2 = 0%), negative symptoms (DME = −0.49, CI 95%: −0.87 to −0.11; p = 0.01; I2 = 0%) and visual care (DME = 0.32, CI 95%: 0.11 to 0.52; p = 0.002; I2 = 0%),	“Cochrane Collaborations Tool for Assessing Risk of Bias” and GRADE methodology.
	[8]	2021	Systematic review and meta-analysis of observational studies	Studies performed worldwide during the period of 2011–2020	8 studies	Not reported	Sepsis / Neonates	The masculine sex (OR: 1.97; CI 95%: 0.26–14.59; p=0.03), prematurity (OR: 2.48; CI 95%: 1.13–5.45; p=0.04), use of central venous catheter, (OR:3.83; CI 95%: 1.07–13.71; p<0.01) and mechanical ventilation (OR: 2.83; CI 95%: 1.42–5.68; p<0.01) were associated with the development of late-onset neonatal sepsis	Newcastle-Ottawa Scale (NOS) for observational studies
	[9]	2022	Systematic review of randomized clinical trials with meta-analysis.	Up to October 2021	7 randomized clinical trials	8716 women	Pre-eclampsia	The risk of pre-eclampsia with supplementation with Vitamins C and E had a prevalence of 3.1% up to 41.6%. While the risk of pre-eclampsia with placebo had a prevalence of 4.1% up to 41.3%.	Risk of bias (RoB 2) tool
	[10]	2020	Systematic review of randomized clinical trials.	During January 2019	6 randomized clinical trials	2530 patients	Depression	No effect was found between the physician’s facilitation in decision making (CI95%: - 4.37 to 7.18), therapeutic adherence (CI95%: −0.31 to 0.71) or depressive symptoms (CI95%: −0.22 to 0.09).	“Cochrane Collaborations Tool for Assessing Risk of Bias” and the GRADE methodology
	[11]	2022	Systematic review of randomized with meta-analysis of analytical observational studies.	Not specified	6 studies	25,814	HTN / Hypertriglyceridemic waist	Statistically significant association between HTN and hypertriglyceridemic waist (OR:1.36; IC 95% 1.07 to 1.71)	Newcastle-Ottawa Scale (NOS) for cross-sectional studies and NCO for cohort studies
	[12]	2021	Systematic review of analytical studies, cohorts.	Not specified	5 studies	6821	Pregnant women/ Subclinical hypothyroidism	The main outcomes that were identified were miscarriage (P= 0.03; OR 0.77; CI 95%: 0.61 to 0.97), pre-term labor (P=0.46; OR:1.11; CI 95%: 0.85 to 1.44) and placental abruption (P=0.56; OR:1.60; CI 95%: 0.33 to 7.66) upon evaluation of the effectiveness of medical treatment, the only favorable outcome was against miscarriages.	Newcastle-Ottawa Scale (NOS)
	[13]	2023	Systematic review of case-control and cohort studies.	Up to October 2021	4 studies	16,478	Maternal complications/ Teenage pregnancy	We evidenced a low risk of pre-eclampsia in pregnant adolescents (OR = 0.93, CI 95% 0.69–1.25) and in postpartum hemorrhage (OR = 0.86, CI 95% 0.74–0.99).	Newcastle-Ottawa Scale (NOS)
	[14]	2023	Systematic review of observational studies.	Up to October 2021	9 studies	7617	Polytraumatized patients / Lactate measurement	A significant association was found between elevated lactate during admission and mortality (OR: 1.80; CI 95% 1.11 to 2.91) and mortality within 72 hours (OR: 1.24; CI 95% 1.02 to 1.50).	Newcastle-Ottawa Scale (NOS)
	[15]	2022	The study is a systematic review of cross-sectional and cohort studies with a meta-analysis of its results of random effects.	October 18–24, 2021	11 studies	5210	Hypertensive retinopathy / Coronary cardiopathies	An association was found between the presence of hypertensive retinopathy and coronary heart disease (P=0.005; OR 1.83; CI 95%: 1.20 to 2.79).	Newcastle-Ottawa Scale (NOS)

The systematic reviews with meta-analysis of randomized clinical trials of observational studies included in the review were published between the years 2020 and 2023, representing the latest theses from graduates of the Medical School from Ricardo Palma University.^5–15 Only four theses did not present a registration in PROSPERO.^6–8,14 The years that had the most publications were 2022 and 2023.

The reviews included a median of six studies, and clinical trials included a total of 18 studies.^7,9,10 Most studies included over 1000 patients, only one had a total of 420 participants,⁷ and another review did not report the number of participants.⁸ Only two reviews used the GRADE (Grade of Recommendation, Assessment, Development, and Evaluation) methodology to evaluate and stratify the evidence quality (Table 1).

Only two SRs included a meta-analysis.^7,15 Seven theses were published in a scientific journal, of these two are published in a local journal.

Quality Assessment

Only Two systematic reviews^7,15 presented low quality. On the other hand, the remaining studies presented critically low quality.^5,6,8–14 (Table 2)

Table 2 Results of the Quality Evaluation of the Instruments AMSTAR II and Modified AMSTAR II for Observational Studies

Item 1*	Item 2*	Item 3*	Item 4*	Item 5*	Item 6*	Item 7*	Item 8*	Item 9*	Item 10*	Item 11*	Item 12*	Item 13*	Item 14*	Item 15*	Item 16*	Evaluation
Systematic reviews of clinical trials
Yes	Yes	No	No	Yes	Yes	Yes	Partial Yes	Yes	No	No	No	No	No	No	No	Critically low
Yes	Yes	No	No	Yes	Yes	No	No	No	No	No	No	No	No	No	No	Critically low
Yes	Yes	No	Partial Yes	Yes	Yes	Yes	Partial Yes	Yes	Yes	No	Yes	Yes	No	Yes	No	Low
Systematic reviews of observational studies
Yes	Partial Yes	No	Partial Yes	Yes	Yes	No	Partial Yes	Yes	No	Yes	No	No	Yes	No	No	Critically low
No	No	No	Partial Yes l	Yes	Yes	No	Partial Yes	Yes	No	Yes	No	No	No	No	No	Critically low
Yes	Partial Yes	No	Partial Yes l	Yes	Yes	No	Partial Yes	Yes	No	Yes	No	No	Yes	No	No	Critically low
Yes	Partial Yes	No	Partial Yes	Yes	Yes	Yes	Partial Yes	Yes	No	Yes	No	No	Yes	No	No	Critically low
No	Partial Yes	No	Partial Yes	Yes	Yes	Yes	Partial Yes	Yes	Yes	Yes	No	No	Yes	No	No	Critically low
No	Partial Yes	No	Partial Yes	Yes	Yes	Yes	Partial Yes	Yes	No	Yes	No	No	Yes	No	No	Critically low
No	Partial Yes	No	Partial Yes	Yes	Yes	No	Partial Yes	Yes	No	Yes	No	No	Yes	No	No	Critically low
No	Partial Yes	No	Partial Yes	Yes	Yes	Yes	Partial Yes	Yes	No	Yes	No	No	Yes	Yes	Yes	Low

Note: *View Annex 2: AMSTAR II components.

Clinical Trials

All the reviews of clinical trials had a negative response on items 3, 11, 14 and 16.^7,9,10 Only items 1 and 2 had an affirmative response in all reviews.^5–15 Most items had at least two negative responses in three systematic reviews of clinical trials. Only items 7 and 9 had an affirmative response in at least one systematic review.^5–15

Observational Studies

In the systematic review of observational studies, only items 5, 6, 9, and 11 had affirmative responses in all the reviews.^5–15 Items 3, 12, and 13 had negative responses in all the reviews, and items 4 and 8 had a partial yes in all the reviews.^5–15 Most reviews had at least one negative response in items 1, 2, 7, 10, 14, 15 and 16.

Reevaluation of the Quality of Theses Prior to the Scientific Publication

We did not observe modifications in any critical component (item 2,4,7,9,11,13,15) of AMSTAR-2 during the reevaluation of the systematic reviews following its publication as a scientific article, therefore there were no changes in the pre-publication and post-publication AMSTAR 2 scores. We only noted text language, structure, and style modifications.

Discussion

With the growing number of theses produced as systematic reviews in the Medical School of Ricardo Palma University, it is essential to evaluate the quality of its production and report positive feedback for students and professors, in order to establish a situational analysis that allows future improvement of the production of these with this methodology. According to our results, 90.9% of theses included presented critically low quality. This differs from the results of Santos-Marques et al,² where the quality of systematic reviews of COVID-19 was critically low in only 27% of the studies evaluated. Likewise, in a study carried out in Korea, 41% of the systematic reviews evaluated were of low quality. Since our last search, no studies have reported the quality of systematic reviews presented as undergraduate theses, thus presenting the first evidence in our field.¹⁶

It is essential to mention that during our study, we observed that the AMSTAR-2 instrument had very restrictive characteristics in its different items and little flexibility for different situations at the time of evaluation of systematic reviews that were already previously reported by other authors,¹⁷ which would explain the low and critically low quality of the theses evaluated. However, the tool allows us an objective assessment, in comparison to other instruments. Additionally, this instrument did not present a valid version for systematic reviews of observational studies, only counting on the adjustments made to the instrument, limiting the possibility of a more comprehensive evaluation of these types of studies.¹⁸ Taking item 4 (“Did the review authors use a comprehensive literature search strategy?”) as an example, it is scarcely applicable to observational studies. To obtain a “yes”, one should have conducted a search in clinical trial registry databases. However, for observational studies, this requirement would be nonsensical. In a systematic review of this type of study, the highest attainable rating for this critical item would be a ‘partial Yes’, requiring modification if we aim to conduct a comprehensive quality review.¹⁸ Thus, we would be reducing the quality of all systematic reviews of observational studies to low or critically low since it compromises a critical item, which is unacceptable since this is only due to a limitation in the instrument’s instructions.¹⁸

The most important strength was the ready availability of the full-text undergraduate theses in the institutional portals of the university and the national registry of theses. On the other hand, among the main limitations, our study did not count on a valid instrument to evaluate the quality of systematic reviews of observational studies and had little flexibility when evaluating the systematic reviews of clinical trials. Likewise, another relevant limitation was the low quantity of theses produced as systematic reviews. We recommend reevaluating the instrument and proposing validation with new items and instructions for systematic reviews of observational studies and clinical trials. It is essential to replicate this study in the future to reevaluate the quality of theses with a new instrument. Although students take thesis preparation courses, there is no deeper focus on learning how to prepare systematic reviews; however, training in systematic reviews is currently being implemented for students and helping increase the rigor of evaluation by the advisor professors at the time of its evaluation.

Conclusions

The systematic reviews presented as undergraduate theses in Medical School of Ricardo Palma University demonstrated low and critically low quality. It is imperative to reevaluate the evaluation instruments for these reviews, considering the inclusion of new items and instructions, culminating in a validation process. We recommend focusing its efforts on strengthening the training of its students and professors in systematic reviews and intensifying the rigor in evaluating these projects.

Disclosure

There are no conflicts of interest to declare.

Data Sharing Statement

The data, codes for analysis, and complementary materials will be available upon request of the researcher. Please contact the corresponding author.

Additional information

Funding

Self-financing.