Protocol for designing INVITES-IN, a tool for assessing the internal validity of in vitro studies

Abstract

This protocol describes the design and development of a tool for evaluation of the internal validity of in vitro studies, which is needed to include the data as evidence in systematic reviews and chemical risk assessments. The tool will be designed specifically to be applied to cell culture studies, including, but not restricted to, studies meeting the new approach methodology (NAM) definition. The tool is called INVITES-IN (IN VITro Experimental Studies INternal validity).

In this protocol, three of the four studies that will be performed to create the release version of INVITES-IN are described. In the first study, evaluation of existing assessment tools will be combined with focus group discussions to identify how characteristics of the design or conduct of an in vitro study can affect its internal validity. Bias domains and items considered to be of relevance for in vitro studies will be identified. In the second study, group agreement on internal validity domains and items of importance for in vitro studies will be identified via a modified Delphi methodology. In the third study, the draft version of the tool will be created, based on the data on relevance and importance of bias domains and items collected in Studies 1 and 2. A separate protocol will be prepared for the fourth study, which includes the user testing and validation of the tool, and collection of users’ experience.

Keywords:

• Cell culture
• NAMs
• next generation risk assessment
• risk of bias

Author contributions

Conceptualization: Camilla Svendsen, Paul Whaley, Gunn E. Vist, Trine Husøy, and Gro H. Mathisen. Funding acquisition: Camilla Svendsen and Gro H. Mathisen. Methodology: Camilla Svendsen, Paul Whaley, Gunn E. Vist, Trine Husøy, and Gro H. Mathisen. Project administration: Camilla Svendsen and Gro H. Mathisen. Supervision: Paul Whaley, Anna Beronius, Ingrid Druwe, Thomas Hartung, Sebastian Hoffmann, Carlijn Hooijmans, Joshua F. Robinson, Erwin Roggen, Andrew A. Rooney, and Mathieu Vinken. Visualization: Paul Whaley and Gro H. Mathisen. Writing – original draft: Camilla Svendsen, Paul Whaley, and Gro H. Mathisen. Writing – review and editing: Camilla Svendsen, Paul Whaley, Gunn E. Vist, Trine Husøy, Anna Beronius, Emma Di Consiglio, Ingrid Druwe, Thomas Hartung, Vasiliki I. Hatzi, Sebastian Hoffmann, Carlijn Hooijmans, Kyriaki Machera, Joshua F. Robinson, Erwin Roggen, Andrew A. Rooney, Nicolas Roth, Eliana Spilioti, Anastasia Spyropoulou, Olga Tcheremenskaia, Emanuela Testai, Mathieu Vinken, and Gro H. Mathisen.

Disclosure statement

Completed declaration of interest forms for each author are at the preprint registration https://doi.org/10.5281/zenodo.7633626. The authors declare no potential conflicts of interest with respect to the research, authorship and/or publication of this protocol.

Disclaimer

The author Ingrid Druwe is employed at the U.S. Environmental Protection Agency. The views expressed in this article are those of the authors and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency.

The author Andrew Rooney is employed at the U.S. National Institute of Environmental Health Sciences. The views expressed in this article are those of the authors and do not necessarily represent the views or policies of the U.S. National Institute of Environmental Health Sciences.

Definitions

Bias are systematic errors, or deviations from the truth, in results or inference (Cochrane Collaboration 2005). For in vitro studies, systematic errors may be introduced in the study design, conduction and/or analysis and cause the result to be an overestimate or underestimate.

Bias domains are themes such as study performance, analysis and reporting, under which bias items can be organised/grouped.

Bias items are study properties that may be relevant for introduction of bias in results and/or their interpretation. Criteria are the issues that have to be fulfilled for bias to be avoided. In the guidance document for the INVITES-IN tool there will be criteria for reaching risk-of-bias judgements for each signalling question.

Internal validity is the extent to which the design and conduct of a study are likely to have prevented bias (Cochrane Collaboration 2005).

In vitro (‘in the glass’) tests means that it is done outside of a living organism and it usually involves isolated tissues, organs or cells (ECHA 2023).

NAMs does not have a standard definition yet. However, there seems to be a general agreement that the term ‘NAMs’ include in chemico, in silico and in vitro studies. One established definition is that NAMs includes any technology, methodology, approach or combination that can provide information on chemical hazard and risk assessment without the use of animals, including in silico, in chemico, in vitro and ex vivo approaches (ECHA 2016; EPA 2018).

Risk of bias is a measure for systematic errors. Risk of bias tools are used for evaluation of the extent to which the design and conduct of a study are likely to have prevented bias (the degree of systematic errors).

Signalling questions are the questions that the users of the tool answer in order to determine whether the criteria have been fulfilled.

Validity is the degree to which a result (of a measurement or study) is likely to be true and free of bias (systematic errors) (Cochrane Collaboration 2005).

Additional information

Funding

This work was supported by HORIZON-HLTH-2021-ENVHLTH-03, Grant [101057014] (the authors Camilla Svendsen, Paul Whaley, Gunn E. Vist, Trine Husøy, Emma Di Consiglio, Vasiliki I. Hatzi, Kyriaki Machera, Eliana Spilioti, Anastasia Spyropoulou, Olga Tcheremenskaia, Emanuela Testai and Gro H. Mathisen) and the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract no. 22.00230 (the author Nicolas Roth).