Flaws (and quality) in research today: can artificial intelligence intervene?

ABSTRACT

The existing flaws in both conducting and reporting of research have been outlined and criticized in the past. Weak research design, poor methodology, lack of fresh ideas and poor reporting are the main points to blame. Issues have been continually raised on the types of results published, review process, sponsorship, notion, ethics, and incentives in publishing, the role of regulatory agencies and stakeholders, the role of funding, and the cooperation between funders and academic institutions and the training of both clinicians and methodologists or statisticians. As a result, there is loss of the utmost goal: the production of robust research to form recommendations to support pragmatic decision in a real-world context. We propose the construction of a model based on artificial intelligence that could assist stakeholders, clinicians, and patients to guide conducting the best quality of research. We briefly describe the levels of the workflow, including the input and output data collection, the feature extraction/selection, the architecture, and parameterization of the model, along with its training, operation, and refinement.

KEYWORDS:

• Artificial neural network
• artificial intelligence
• research
• study design

Introduction

The optimal delivery of health care requires four elements: research, practice, evaluation, and policy development. The quality of the former has been constantly the target of scientific interest, where Randomized Controlled Trials (RCTs) and systematic reviews (SRs) have both occupied the top of the hierarchy of evidence. But, both of them are problematic: the former are extremely time and money consuming, while the latter are numerous, often carrying the questioned quality of the original trials: different approaches through the combination of production of primary data, teamwork, replication, and meta-analyses, have been proposed (Panagiotou et al. 2013).

The existing flaws in both conducting and reporting have been outlined and criticized in the past (Ioannidis 2005; Goodman and Greenland 2007), while, later on, potential solutions of future prospects have been recorded (Ioannidis 2014; Wang 2016). The fearing statement of these and other reports was that as much as half of the scientific literature may be wrong. Weak research design, poor methodology, lack of fresh ideas to drive innovation and poor reporting often are the main points to blame (Ioannidis 2005, 2014; Goodman and Greenland 2007; McElreath and Smaldino 2015; Wang 2016). Also, the procedures of evaluating both research and researchers are passing through lengthy CVs and social pressure to paper count (Smaldino and McElreath 2016): quantitative metrics, such as publication rates and impact factors have been the norm (van Diest et al. 2001).

In parallel, the number of publications is rising (Larsen and von Ins 2010): there are more opportunities for collaboration and multi-author papers, an intense competition for academic positions, publication of the largest possible number of results instead of the most rigorous ones, and an endless competition for grants, promotions, and prestige (van Diest et al. 2001; Larsen and von Ins 2010; Ioannidis et al. 2014).

In addition, issues have been continually raised on the types of results published (publishing everything vs. selective publishing with respect to cost-benefit and time), review process, sponsorship, notion, ethics, and incentives in publishing, access of patients to published and unpublished results, amount of research published in other than English languages and not indexed in the basic databases, the role of regulatory agencies and stakeholders, the role of funding, and the cooperation between funders and academic institutions and the training of especially clinicians and practicing scientists on one hand and methodologists or statisticians on research practices and real-world study aims and final endpoints, on the other (van Diest et al. 2001; Ioannidis 2005, 2014; Goodman and Greenland 2007; Larsen and von Ins 2010; Cyranoski et al. 2011; McElreath and Smaldino 2015; Wang 2016).

As a result, there is loss of the utmost goal: the production of robust research to form recommendations to support pragmatic decision in a real-world context.

The utilization of some rapidly evolving fields, such as computer-based prediction models and Artificial Intelligence (AI) systems conforms to the tendency toward automation of the procedures performed in any modern medical field, replacing the use of the human brain in specific tasks. Such systems are already implemented in the modern scientific era with numerous examples, including cancer research, neurology, cardiology, imaging, drug design, and immune therapies, among others (Siristatidis et al. 2010; Hamet and Tremblay 2017). These models/systems have demonstrated usefulness in performing associations between the analyzed factor/s [or causative source(s)] and results [such as treatment outcome(s)]. Artificial Neural Networks (ANNs), the basic components of AI, exhibit remarkable information processing abilities pertinent mainly to nonlinearity, high level of parallelism, some fault tolerance capability, as well as learning capability followed by generalization and self-adapting prospects (Hamet and Tremblay 2017). Their special characteristics lie through a ‘learning through training’ model, resembling the capacity of the brain to learn, assimilate and recall this knowledge in anticipation of a future event, while via trained learning, they self-adapt and change their structural characteristics to accumulate the knowledge (Goodman and Greenland 2007; Hamet and Tremblay 2017). The reduction of cost, the combination of hardware specifically designed for AI implementation, the improvement of evaluation, at the stages of data acquisition, interpretation, and decision-making are important features, among others (Monroe 2018).

Description of methods

A workflow of such a system could comprise the following levels (Figure 1):

Figure 1. A proposed workflow of the ANN approach.

Level 1: Input data collection

Data from the bibliographic research of major databases, such as the Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE, and CINAHL, Thomson Reuters Web of Science, along with trial registers for ongoing and registered trials, conference abstracts in the Web of Science and Google Scholar for gray literature, professional societies, organizations, and individuals will be collected. Of them, features that can lead to the evaluation of their quality will be extracted, serving as the ANN inputs (Eysenbach 2011; Levitt and Thelwall 2011; Cottrell EK 2015; Chavda and Patel 2016; Maggio et al. 2017; Braithwaite et al. 2019; Higgins et al. 2019; Leatherdale 2019; Sterne et al. 2019). In order to condense them into a fine-structured set, algorithms will be used. A list of characteristics with the potential to serve as ANN inputs is presented in Table 1.

Table 1. List of characteristics with the potential to serve as ANN inputs.

Bibliometrics, such as citation-based metrics, composite metrics of the weighted sum of impact factors and the number of citations, author contribution measurements, downloads or online views received

Altmetrics, such as web-based metrics with a priority on social media features (Altmetric donut), activity on social media or scientific platforms, such as Mendeley, Research Gate, Academia, Publons or Twitter and Facebook)

H-index, immediacy index indicative of the author(s) reputation

Study type/design (RCTs, quasi-RCTs, systematic reviews, meta-analyses, and cohort, observational, experimental studies)

Use of Quality Assessment Tools, such as the Newcastle-Ottawa Quality Assessment Scale tool to assess risk of bias; Quality Assessment of Diagnostic Accuracy Studies-2; Assessment of the Methodological Quality of Systematic Reviews; Cochrane tools, such as RoB2 and the Enhancing the Quality and Transparency Of health Research (EQUATOR) Network.

The type/number of patients/interventions/comparisons

Type of institutions performing the research

The number of products created based on the specific research (either through pilot or main studies) and patents filed

Record of funding by national or private agencies

Evaluation of research by algorithms (e.g. PageRank)

The existence of peer reviewing

Cost-effectiveness data

The function of discussion platforms, for the number of positive and negative comments

Journal ranking/circulation and coverage/acceptance/rejection rates

Level 2: Output data collection

The ANN system outputs are quality scores, i.e., measures of quality indicators (for example, using a scale from 1 to 100). Notably, these data will be dependent on the target population; in specific, these outputs and/or scores are different according to the target population, i.e., clinicians, patients, or stakeholders, and will be used only for the initial ANN construction phase (Table 2). These data will be collected through Internet publications and relevant reports or personal or web-based questionnaires depending on the issue raised or the populations implicated (stakeholders, clinicians, or patients). Of note, the ANN could predict either all metrics separately or in groups, according to the issue raised and its programming.

Table 2. List of outputs and/or scores according to the target population, used for the initial ANN construction phase.

Stakeholders (employees, customers, suppliers, shareholders, regulatory or government agencies, boards of directors, and business owners)	To enhance and promote evidence-informed policymaking
	To ensure the best evidence of the effectiveness and side effects of a new application, either pharmacological or interventional
	To improve the understanding of a specific pathological symptom and/or illness
	To ensure a robust framework for the oversight and conduct of clinical trials
	To ensure the best healthcare policy
	To support network activities among developers
	Globalization of clinical research
	To safely guide funding/ensuring an investment in a logical and cost-effective outcome
	Further proposals for the best accomplishment of future targets
	To schedule new experimental research designs ones, guide research and, informing current and emerging public health priorities, stimulate political interest
Clinicians	To raise their quality care, affordability, quality of life and best practices
	Quickly access detailed drug or intervention information on dosing, availability, formulations, side effects and contraindications
	To gain confidence in their clinical decisions
	To get the best knowledge/management strategies/up-to-date recommendations for clinical practice
	Protect them in law from malpractice
	To stay current and ensure their ongoing professional development
Patients	Get the right information
	Make it easier to make decisions
	To get the most informed and accurate option for a treatment, for example success metrics, accurate information for possible side effects, quality of life imprints
	Side effects: frequency and severity

Level 3: Feature extraction/selection

Mathematical models, such as information gain, correlation coefficient scores, chi-square tests, and recursive feature elimination algorithms, among others, will be applied in the data of the two previous levels. The aim of this process is to identify the most important characteristics to be used as inputs for ANN construction (Setiono and Liu 1997; Kwak and Choi 2002). As this step is related to the training of the ANN, the data processed are those relevant to training, which are characterized as ‘important’. Notably, these data are associated, for example, with high-quality metrics. In addition, even the ‘non-important’ input features can be selected. Markedly, ANNs have the capability to isolate and avoid using or using only a part of such data that are not related to the set output.

Level 4: Architecture and parameterization

In this step, the ANN architecture is being selected and parameterized. A different ANN will be used for each individual target group (patients, clinicians, stakeholders/organizations/scientific communities). Furthermore, the parameterization will be dependent on the ANN model/s of choice, while ANN characteristics will be selected, such as size, training algorithms, iterations, and definition of criteria for the stop of training.

Level 5: Training

This is performed using data from Levels 1 and 2, being selected at Level 3. Firstly, during this process, the ANN ‘learns’ from the data inserted, while, simultaneously, can incorporate knowledge on unknown data, i.e., data not included in the training set, a characteristic so-called ‘generalization’ (Haykin 1999). Secondly, the proposed ANN would have a ‘cyclic nature and function’, i.e., the system should be able to be trained again to accumulate new knowledge and/or adjust existing knowledge from previous training processes, a process so-called ‘continual learning’ (Parisi et al. 2019; Zeng et al. 2019).

Level 6: Operation – refinement

The trained ANN system will be used in an operational manner (routine) for evaluating research quality, separately for each individual target group. In addition, data collection will be continually renewed via automated searches, with subsequent extraction of new data, which will – probably replace the initial selected, based on low-performance indices of the ANN and new groups interested – and inserted. Thus, new training cycles can be performed during the operational period, in an attempt to develop and refine the system. Importantly, in this level, the user input is expected to ‘judge’ the ANN performance, serving as a guide to refine its training (https://doi.org/10.1097/MD.0000000000006612). In addition, further information will be gathered from the patients themselves, as a supplementary form of feedback, in an effort to refine the accuracy of the system. At this point, standardized criteria of effectiveness should be applied in order to avoid subjective interpretations and contradictory results.

Discussion

In the field of research, a construction of such an AI model, based on a supervised learning ANN could assist stakeholders, clinicians, and patients to guide conducting the best quality of research. For example, an adequate input data set could be composed of all the above-mentioned parameters and be complemented by feature extraction and selection algorithms. Both selection of the appropriate ANN model and definition of its parameters and characteristics, along with the evaluation of the system performance, would subsequently depend on the target population that the information will be addressed to. Thus, the inputs are the characteristics gathered from various sources indicated, such as bibliometrics, altmetrics, and quality assessment tools and the outputs are the final target addressed for stakeholders, clinicians, and patients. Finally, the training and self-adaptation of the system would depend on the target group that will be used for and the questions posed. The final aim is to obtain a robust AI-assisted way of judging and thus leading research.

A typical example of such a system was reported by our team, some months ago. The constructed ANN was based on statistically significant variables with the outcome of live birth, representing a stable and efficient system with increased performance indices. Validation of the system allowed an insight of its clinical value as a supportive tool in medical decisions, and overall provides a reliable approach in the routine practice of IVF units in a user-friendly environment. Of note, the rather small standard deviation in the performance indices between the training and test sets throughout the validation process indicated a stable performance of the constructed ANN (Vogiatzi et al. 2019).

Of course, limitations exist and lie on a ‘non human way’ of leading human acts, the lack of reasoning, since ANNs have no mechanism to justify their output, the possible lack of qualitative assessment and the objectivity of the collection of the primary data sets, on which the ANN would base its outcome, along with the clear definition and choice of the quality metrics, named here as outputs.

Compared to the existing relevant approaches, our proposed ANN system displays numerous advantages, such as the quality metrics linked to the need of specific target groups, the capability for continuous and augmenting learning and adaptation, through the inclusion of new data and the users’ feedback, and the lack of need to design human understandable algorithms and metrics, to obtain a quality score: instead, the ANN, operating as a black box, self-adapts and provides quality metrics according to the user needs.

In conclusion, this ANN system is not just a re-iteration of parameters already used to assess research and researchers. We propose the combination of high validity current quality metrics adapted to the individual needs of patients, clinicians, and stakeholders. To the authors’ belief, the implementation of such a system, apart from being feasible, offers the capability of combining current quality inputs and outputs, targeting to the goal of the production of robust research. In this context, features that could add to its refinement and improvement are the technologies of big data, the cloud, and the interoperability offered by operators of systems, which could help not only to the data collection but also to its operation, as a product.

Author contributions

Conceived the idea and drafted the paper: CS; co-drafted the paper and wrote the technical parts: AP.

Acknowledgments

The authors wish to thank Paraskevi Vogiatzi for her ideas for writing the manuscript.

Disclosure statement

Authors have no competing interest.