Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?

ABSTRACT

Objectives

Artificial intelligence-powered tools, such as ASReview, could reduce the burden of title and abstract screening. This study aimed to assess the accuracy and efficiency of using ASReview in a health economic context.

Methods

A sample from a previous systematic literature review containing 4,994 articles was used. Previous manual screening resulted in 134 articles included for full-text screening (FT) and 50 for data extraction (DE). Here, accuracy and efficiency was evaluated by comparing the number of identified relevant articles with ASReview versus manual screening. Pre-defined stopping rules using sampling criteria and heuristic criteria were tested. Robustness of the AI-tool’s performance was determined using 1,000 simulations.

Results

Considering included stopping rules, median accuracy for FT articles remained below 85%, but reached 100% for DE articles. To identify all relevant articles, a median of 89.9% of FT articles needed to be screened, compared to 7.7% for DE articles. Potential time savings between 49 and 59 hours could be achieved, depending on the stopping rule.

Conclusions

In our case study, all DE articles were identified after screening 7.7% of the sample, allowing for substantial time savings. ASReview likely has the potential to substantially reduce screening time in systematic reviews of health economic articles.

KEYWORDS:

• Artificial intelligence
• systematic review
• simulation
• accuracy
• efficiency
• stopping rule
• asreview

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Open science statement

Analysis was performed in R [13] and all scripts and data will be published upon publication of the manuscript.

Earlier presentations

Part of the work has been presented in the form of a poster on the 8 November at ISPOR Europe 2022 in Vienna

Supplementary Material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/14737167.2023.2234639

Additional information

Funding

This paper was not funded.