Cross-mapping of terms used in chemical risk assessment with those used in systematic review: research protocol

Abstract

The focus on implementation of systematic review (SR) principles in chemical risk assessments (CRAs) is growing as it has the potential to advance the rigour and transparency of the CRAs. However, the SR and CRA communities use their own specific terminologies. Understanding the meaning of core SR and CRA terms and where they overlap is critical for application of SR methods and principles in CRAs. Moreover, it will increase the possibility for cross-sectorial collaboration, avoid misunderstandings, and improve communication among risk assessors, researchers, and policy makers. We present a process for the cross-mapping of core CRA terms and core SR terms. Core terms for study appraisal, evidence synthesis and integration used in the SR and CRA communities will be included. The outcome will be an overview of how core SR terms map onto core CRA terms and vice versa, and a description of the relationship and conceptual overlap between the terms. The cross-mapping is divided in three phases, where in the first phase the core SR and CRA terms will be identified. In the second phase, existing SR and CRA definitions will be mapped. In the third phase, descriptions of the relationship and conceptual overlap between the terms will be derived. The third phase will include weekly 1-h online meetings for SR and CRA experts.

Keywords:

• Conceptual overlap
• cross-mapping
• definitions
• interoperability
• terminology

1. Introduction

Chemical risk assessments (CRAs) should be evidence-based, which means that they are grounded in a comprehensive and rigorous, transparent and objective analysis of all evidence relating to the assessment task. Applying systematic review (SR) principles in CRAs has become an established methodology for achieving this goal, from its first practical introduction in 2013–14 (NTP OHAT 2015; Woodruff and Sutton 2014), its popularisation in the following few years (Hoffmann et al. 2017; Whaley et al. 2016), and its wider uptake by national and international risk assessment agencies including US EPA (EPA 2022, 2023), EFSA (EFSA 2010; Hardy et al. 2017a), and WHO (WHO 2021).

Because SR and CRA methodologies were developed independently of each other, the SR and CRA communities use their own specific terminologies and language. Numerous SRs performed as part of CRAs have shown that these terminologies often are analogous to each other or overlapping, but rarely the same or directly translatable. Also, often a reporting checklist, like the PRISMA checklist, have not been included. It can therefore be difficult to understand which SR method (or to what level/extent) is applied in the CRAs (i.e., whether a given framework or application is sufficiently rigorous to be described as “systematic”) and may be impeding the understanding and therefore potentially slowing the uptake of SR methods in the CRA community.

In this project, we will analyse conceptual overlap and differences between the core CRA terms of SR and CRA. This way, we aim to increase the interoperability of SR and CRA terminologies by improving the understanding of the meaning of and the relationships between the core terms of the respective domains.

1.1. Project governance

This project is a part of the “Next generation risk assessment in practice” project (VKM 2023) which is included in the European Partnership for the Assessment of Risks from Chemicals (PARC; Project 101057014)”. The participants in this project include the members of the research team and the members of the scientific advisory group (SAG). A project group (PG) has been established with the responsibility for drafting the protocol and performing the study.

2. Methods

2.1. Study design

A cross-mapping of core SR and CRA terms will be performed to explore the relationship between the terms and to identify conceptual overlaps. By “core” we mean terms denoting key concepts in the study appraisal, evidence synthesis, and evidence integration steps of systematic reviews and chemical risk assessments. For CRAs and SRs, the scope is limited to methods for quantitative systematic review of health effects of interventions and exposures and methods for evidence synthesis. The project is divided into three phases as shown in Figure 1.

Figure 1. Overview of the three phases and the timeline for the cross-mapping between core systematic review and core chemical risk assessment terms. CRA, chemical risk assessment; SR, systematic review.

The timeline for the project and estimated duration for each phase are shown in Figure 1. Phase 3 will include weekly 1-h online meetings for the discussion of the descriptions of SR and CRA term relationships and conceptual overlap, and the anticipated duration of this phase is 3 to 5 months.

2.2. Phase 1: Cataloguing core terms in SR and CRA terminologies

The objective is to catalogue core SR and CRA terms that are used for study appraisal, evidence synthesis and integration.

2.2.1. Creating longlists of SR and CRA terms

A selection of handbooks, guidance documents and glossaries from different sources will be used to identify core SR and CRA terms (Table 1). Two persons from the PG will independently go through the documents listed in Table 1 and extract terms that are used for study appraisal, evidence synthesis or evidence integration. All extracted terms will be included in the longlist. The longlist of terms will be computed into Excel and will be available as supplementary materials.

Table 1. The document collection for identification of core SR and CRA terms.

Document	References
SEVCO (Scientific Evidence Code System)	FEvIR Platform Version 0.80.0 (06.12.2022)
Finding What Works in Health Care: Standards for Systematic Reviews	Institute of Medicine Committee on Standards for Systematic Reviews of Comparative Effectiveness (2011)
JBI Manual for Evidence Synthesis	Aromataris and Munn (2020)
Application of systematic review methodology to food and feed safety assessments to support decision making	EFSA (2010)
Methodological Expectations of Cochrane Interventions Reviews (MECIR)	Higgins et al. (2023)
Glossary of Terms in The Cochrane Collaboration, Version 4.2.5	Cochrane Collaboration (2005)
Handbook for Conducting a Literature Based Health Assessment Using OHAT Approach for Systematic Review and Evidence Integration	NTP OHAT (2019)
Framework for the use of systematic review in chemical risk assessment	WHO (2021)
Guiding Principles and Key Elements for Establishing a Weight of Evidence for Chemical Assessment	OECD (2019)
ORD Staff Handbook for Developing IRIS Assessments	EPA (2022)
Guidance on the assessment of the biological relevance of data in scientific assessments	Hardy et al. (2017b)
PRISMA 2020	Page et al. (2021)
EFSA glossary	EFSA (2024)

It was originally intended that the longlists would be machine generated using the Term Frequency-Inverse Document Frequency (TF-IDF) method. However, the PG did a pilot test of the TD-IDF method on the document collection using the English Web corpus enTenTen21 (Sketch Engine 2023) as the comparator corpus, and it was found to be less efficient than expected, with challenges on corpus selection, data cleaning, and optimisation of term abstraction and rank ordering. We have therefore determined that manual identification of terms will be done instead.

2.2.2. Creating the final shortlist of core SR and CRA terms

An expert group, consisting of a minimum four experts from the PG and SAG, who have experience within SR, CRA or both fields will be created.

The expert group members will individually screen the longlist which contains both CRA and SR terms in Excel. The experts can also add additional terms they believe should be included but are not on the longlist. For each term, the expert group members will rate their level of agreement using a 5-point Likert scale (5 indicates strong agreement, whereas 1 indicates strong disagreement) towards two statements: 1) This term is a core CRA term for study appraisal, evidence integration and/or evidence synthesis, 2) This term is a core SR term for study appraisal, evidence integration and/or evidence synthesis.

Terms scored “4” or “5” on the Likert scale by one or more members of the expert group will be considered as core terms and included in the shortlist.

The shortlists will be presented and discussed in PG and SAG meetings to identify i) terms on the shortlist that are not related to study appraisal, evidence synthesis and integration process, and ii) additional terms that should be included. The final shortlists will be prepared by the PG and be available as supplementary materials. Information on the cumulative score (Likert scale scoring) will also be available.

In the case that many core terms are identified and there is a need for prioritisation, the cumulative scores (Likert scale) will be used to prioritise terms to be included in the cross-mapping.

2.3. Phase 2: Preparing a list of existing definitions of SR and CRA terms

The objective of this phase is to prepare a list of a representative range of definitions of the core SR and CRA terms.

The definitions for the SR and CRA terms will be primarily collected from glossaries, guidance’s and handbooks listed in Table 1. However, if the definitions are not listed in those documents other sources might be used, which will be identified by an Internet search or brainstorming within the PG and SAG. The definitions of the core SR and CRA terms will be extracted by one PG member and checked by another PG member. Each extracted definition will be tagged with from which source it was extracted from. The table will be made available as supplementary materials.

2.4. Phase 3: Identifying conceptual overlap between CRA and SR terms

The objective is to identify areas of conceptual overlap and difference between CRA terms and SR terms. The cross-mapping will be done via a consensus process involving an expert group with PG and SAG members and additional experts (see Section 2.6) self-identifying as having relevant CRA and/or SR expertise. The main steps in the cross-mapping are shown in Table 2. A more detailed overview of the steps is included in the Supplementary materials (Table S1). The approach is adapted from the consensus process used in the development of the SEVCO methods ontology and the GRADE Ontology (Alper et al. 2021, 2024).

Table 2. The four main steps in the cross-mapping process.

Step	What
1. Identification of relationships between core CRA terms and core SR terms	PG will map which terms are related
2. Assembling an expert group with PG members, SAG members, and additional experts, all self-identifying as having relevant SR and/or CRA expertise	The relationship between terms is discussed in the expert group. Descriptions of the relationships between the terms will be drafted according to the discussion
3. Identification of agreement on discussed descriptions (approval of the draft statements from step 2)	The experts vote (anonymously, online) “agree” or “not agree” on approval of the draft statement from step 2 The criteria for agreement are: i) at least 5 members voted AND ii) all votes were for “agree” If an agreement is reached, stop here If no agreement is reached, proceed to step 4
4. Suggestion of changes to the descriptions where no agreement was reached	The result of the vote is discussed in the expert group and redrafted when needed. Participants will vote (online) “agree” or “not agree” on each discussed statement. The criteria for agreement for descriptions of relationships between the terms that are discussed for the second time are: i) at least five members voted AND ii) at least 80% of the votes were for “agree” If an agreement is reached, stop here If no agreement is reached, proceed to step 3 Steps 3 and 4 are repeated a maximum of two times. If agreement is not reached, no statement of the relationship between the CRA and the SR terms will be created. However, information on the difficulties and suggested descriptions will be included as supplementary materials

2.5. The expert groups participating in Phase 3

The experts participating in the online meetings will be PG and SAG members and additional experts self-identifying as having relevant SR or CRA expertise. Recruitment of the additional experts will be done by PG and SAG members via their national and international SR and CRA networks. Anyone self-identifying as having the relevant expertise can sign up at any time. Expert group participants will be sent project updates, in particular notifications of when terms are open for vote, by email. Votes will be cast by email to the PG member tasked with facilitating the discussion and voting process. The facilitator will anonymise the votes to the rest of the PG and expert group.

Following the additional experts first participation, they will be asked to fill out a short questionnaire with questions about their affiliation, type of profession, country of residence, gender, age, and number of years of experience with SRs and CRAs (phase 3). Information on race or ethnicity will not be collected because in Norway this information falls under the category of “sensitive personal data” and on a general note this type of data is not legal to be collected or handled.

Everyone on the mailing list will receive meeting documents in front of the meetings. Everyone that participated in a meeting will be asked to participate in the voting after the meeting. The votes will not be anonymous for the PG but will be anonymised in the manuscript.

Expert Group members are eligible to be co-authors if they i) vote and/or comment on at least ten terms or cross-mappings in total, and ii) reviews the manuscript. Expert group members not eligible to be co-authors will be listed in the acknowledgements. No financial compensation or other incentives are offered for the participation as additional expert.

3. Anticipated results

In this section we describe how the result of the study will be presented in the results section of the finalised manuscript. All other results will be made available as supplementary materials.

3.1. Core SR and CRA terms

A list of core terms in the SR and CRA terminologies, and the Likert scale scoring, will be presented. A table illustrating a proposed way to present the results is included in the Supplementary materials (Table S2).

3.2. SR and CRA term definitions

The definitions of SR and CRA core terms and their synonyms will be presented. Tables illustrating the presentation of the catalogued definitions of SR and CRA terms are included in the Supplementary materials (Tables S3 and S4). The presentation of participant characteristics for the expert group participating in phase 3 is illustrated in the Supplementary materials (Table S6).

3.3. Conceptual overlap between core SR and core CRA terms

Descriptions of the relationship between CRA terms and SR terms will be presented. Tables illustrating how this information will be presented are included in the Supplementary materials. The proposed presentation of the cross-mapping of CRA terms on the SR terms is shown in the Supplementary materials (Table S5). The proposed presentation of the SR terms and the related CRA terms and the conceptual overlap is shown in the Supplementary materials (Table S6). The proposed presentation of participant characteristics for the expert group participating in phase 3 is shown in the Supplementary materials (Table S7).

4. Limitations

The methods described in this protocol are considered to provide a grounded process towards a common understanding for the meaning of these terms without being too time-consuming. A consequence may be that not all relevant terms from all SR and CRA communities will be included.

While we attempt to involve a broad and diverse group of experts from several institutions in this project, it is possible that we will not be able to recruit participants from all relevant institutions within the SR and CRA communities. However, being able to distribute information through the networks of both the PG and the SAG, we expect to recruit participants from several relevant institutions.

For the online group discussions in Phase 3, there is a risk for some persons to dominate the discussion. To reduce this, an experienced and neutral moderator (PW) will facilitate the discussions. Anonymous voting online should also allow for more ease at disagreeing with strong personalities and participants assumed higher in a social hierarchy. Because we are aiming at a high level of consensus, voting will give all participants the opportunity to contribute over several rounds of development of the cross-map, with the requirement that their comments are considered in each round of discussion. If any participants feel their comments are not responded to, they can vote “no” in the next round of voting. Also, we are planning for a diverse mix of experiences and backgrounds in the group.

Disclaimer

The author Ingrid Druwe is employed at the U.S. Environmental Protection Agency. The views expressed in this manuscript are those of the authors and do not necessarily represent the views or policies of the U.S. Environmental Protection Agency.

The author Andrew A. Rooney is employed at the U.S. National Institute of Environmental Health Sciences. The views expressed in this manuscript are those of the authors and do not necessarily represent the views or policies of the U.S. National Institute of Environmental Health Sciences.

Dissemination

The outcome of this project will be published in a scientific journal.

Definition

Core terms are in this project defined as terms denoting key concepts in the study appraisal, evidence synthesis, and evidence integration steps of systematic reviews and chemical risk assessments.

Authors contribution

Conceptualization: Camilla Svendsen, Gro H. Mathisen, Gunn E. Vist, Trine Husøy, and Paul Whaley. Funding acquisition: Camilla Svendsen and Gro H. Mathisen. Methodology: Camilla Svendsen, Gro H. Mathisen, and Paul Whaley. Project administration: Camilla Svendsen and Gro H. Mathisen. Supervision: Paul Whaley. Visualization: Camilla Svendsen, Gro H. Mathisen, and Paul Whaley. Writing - original draft: Camilla Svendsen, Gro H. Mathisen, Gunn E. Vist, and Paul Whaley. Writing - review & editing: Camilla Svendsen, Gro H. Mathisen, Gunn E. Vist, Trine Husøy, Heather M. Ames, Anna Beronius, Emma Di Consiglio, Ingrid Druwe, Thomas Hartung, Sebastian Hoffmann, Carlijn R. Hooijmans, Kyriaki Machera, Joshua F. Robinson, Erwin Roggen, Andrew A. Rooney, Nicolas Roth, Eliana Spilioti, Anastasia Spyropoulou, Olga Tcheremenskaia, Emanuela Testai, Mathieu Vinken, and Paul Whaley.

Ethical considerations

Application for ethical approval will be submitted to the Norwegian Institute of Public Health.

Abbreviations
CRA	=	Chemical risk assessment
PG	=	Project group
SAG	=	Scientific advisory group
SEVCO	=	Scientific evidence code system
SR	=	Systematic review

Disclosure statement

Completed declaration of interest forms for each author are available as supplementary materials. The authors declare no potential conflicts of interest with respect to the research, authorship, and/or publication of this protocol.

Additional information

Funding

Camilla Svendsen, Gro H. Mathisen, Gunn E. Vist, Trine Husøy, Heather M.R. Ames, Emma Di Consiglio, Kyriaki Machera, Eliana Spilioti, Anastasia Spyropoulou, Olga Tcheremenskaia, Emanuela Testai, and Paul Whaley were supported by HORIZON-HLTH-2021-ENVHLTH-03, Grant [101057014]; Nicolas Roth was supported by The Swiss State Secretariat for Education, Research and Innovation (SERI) under contract no. 22.00230.