Evaluation of QSAR models for predicting mutagenicity: outcome of the Second Ames/QSAR international challenge project

Figures & data

Table 1. Overview of the First and Second Ames/QSAR International Challenge Projects.

	First Project	Second Project
Aim	QSAR tool improvements
Date	2014–2017	2020–2022
Participants	12 teams (7 countries)	21 teams (11 countries)
	Mainly QSAR venders	QSAR venders/academia/non-commercial entities
Training dataset	Phase I: none	Chemicals used the First Project
	Phase II: 3,902 chemicals
	Phase III: 3,902 + 3,802 chemicals
Trial dataset	Phase I: 3,902 chemicals	1,589 new chemicals
	Phase II: 3,802 chemicals
	Phase III: 4,409 chemicals
Models analysed	Statistical and rule-based models	Statistical, rule-based and machine-learning models

Table 2. Teams participating in the Second Project.

Team no.	Team name	Country	Note*
1	Shanghai Institute of Organic Chemistry, Chinese Academy of Sciences (SIOC, CAS)	China
2	Altox Ltd.	Brazil
3	MN-AM	Germany/USA	(i)
4	Instem	USA	(i)
5	Istituto di Ricerche Farmacologiche Mario Negri IRCCS (IRFMN)	Italy	(i)
6	IdeaConsult Ltd.	Bulgaria	(i)
7	MultiCASE Inc.	USA	(i)
8	Lhasa Ltd.	UK	(i)
9	Istituto Superiore di Sanità (ISS)	Italy	(i)
10	Gifu University	Japan
11	Massachusetts Institute of Technology	USA
12	Simulations Plus, Inc.	USA	(i)
13	Chemotargets	Spain
14	LMC – Bourgas University	Bulgaria	(i)
15	The University of Sydney	Australia
16	Meiji Pharmaceutical University	Japan
17	Liverpool John Moores University	UK
18	Evergreen AI, Inc.	Canada
19	Politecnico di Milano	Italy
20	National Center for Toxicological Research U.S. Food and Drug Administration (NCTR/FDA)	USA
21	National Institutes of Biomedical Innovation, Health and Nutrition (NIBIOHN)	Japan

*(i): team also participated in the First Project.

Table 3. Number of chemicals in the Ames classes in the trial set for the Second Project.

Class A	Class B	Class C	Total
80 (5.0)	156 (9.8)	1,353 (85.1)	1,589 (100)

Expressed as numbers (%).

Table 4. Performance metrics used to evaluate Ames/QSAR model performance.

Performance metric	Calculation and description
A-Sensitivity (A-Sens.)	TPA/(TPA + FN)
	Measures the ability to correctly predict strongly Ames-positive compounds (class A).
Sensitivity (Sens.)	TP/(TP + FN)
	Measures the ability to correctly predict Ames-positive compounds (class A or B).
Specificity (Spec.)	TN/(FP + TN)
	Measures the ability to predict Ames-negative compounds (class C).
Accuracy (Acc.)	(TP + TN)/(TP + TN + FP + FN)
	Assesses overall prediction performance by returning the fraction of compounds that were correctly classified.
Balanced Accuracy (BA)	(Sens. + Spec.)/2
	Assesses overall model performance while giving each class equal weight.
Mathews Correlation Coefficient (MCC)	[(TP * TN) − (FP * FN)]/[(TP + FP) * (TP + FN) * (TN + FP) * (TN + FN)]^1/2
	Assesses overall model performance.
	Values can range from − 1 to 1, unlike the other metrics in this table that range from 0 to 1.
Coverage (Cov.)	(TP + TN + FP + FN)/Total
	Assesses the proportion of compounds for which the model can make positive or negative prediction.
Positive Prediction Value (PPV)	TP/(TP + FP)
	Indicates how frequently positive predictions are correct.
Negative Prediction Value (NPV)	TN/(TN + FN)
	Indicates how frequently negative predictions are correct.
F1-Score	2*(Recall * Precision)/(Recall + Precision)
	Indicates the harmonic mean of Recall and Precision, where Recall = Sensitivity and Precision = PPV.

TPA: True positive for class A (prediction is positive and Ames test result is class A); TP: True positive (prediction is positive and Ames test result is class A or B); TN: True negative (prediction is negative and Ames test result is class C). FN: False-negative (prediction is negative and Ames test result is class A or B); FP: False positive (prediction is positive and Ames test results is class C).

Table 5. Sample of training data provided to the participants of the Second Project*.

						−S9					+S9
Class	Chemical name	Chemical structure	Mol. weight	Purity (%)	Solvent	TA100	TA1535	WP2uvrA	TA98	TA1537	TA100	TA1535	WP2uvrA	TA98	TA1537
C			163.2
C			216.4	98.3	THF	C	C	C	C	C	C	C	C	C	C
B			420.1	99.4	Acetone	C	C	C	C	C	B	B	C	C	C
C			515.7
C			2366.2
B			148.0	99.7	DMSO	C	C	C	C	C	B	C	B	C	C
C			264.1
A			203.0	99.8	DMSO	C	C	C	C	C	A	C	C	B	C

*The chemical names and structures were provided to the participants under a confidentiality agreement. Mol. weight, purity and solvent indicate the molecular weight of the training chemical, the chemical purity of the sample used in the Ames test and the solvent used for the Ames test, respectively. −S9 and +S9 indicate that the five strains were tested without (–) and with (+) metabolic activation. Ames class A is strongly positive, class B is positive and class C is negative for mutagenicity.

Table 6. Curation of the training dataset used in the Second Project.

	Class A	Class B	Class C	Total
(I) Chemicals used in the First Project	672	1,085	10,383	12,140
(II) Ames study report missing	44	164	1,238	1,446
(III) Reports for curation (= I − II)	628	921	9,145	10,694
(IV) Curation completed on March 2020	628	921	16	1,565
Curation not completed (= III − IV)^*1	0	0	9,129	9,129
Number of modified reports	26	18	16
Change after modification^*2	−23	15		−6
Provided training dataset (March 2020)	649	1,100	10,385	12,134

^*1Curation was performed by the DGM/NIHS members. The participating teams were not provided with detailed information on the noncurated chemicals.

^*2Six chemicals were not assigned to class A, B, or C.

Table 7. Models selected by each team BEFORE access to Ames test results for the trial chemicals.

Model no.^*1	Team name	Before being informed of the results
1	SIOC, CAS	CISOC-PSMT (SIOC, CAS, China)
3	MN-AM	ChemTunes. ToxGPS Ames NIHS_v2
4	Instem	Leadscope 2nd QSAR Challenge Consensus Model
5–1	IRFMN	Mutagenicity (Ames test) CONSENSUS model (18k) version 0.9.1
6	IdeaConsult Ltd.	AMBIT DeepN v4.85
7	MultiCASE Inc.	PHARM_BMUT model version (1.8.0.0.17691.350)
8	Lhasa Ltd.	Sarah Nexus v.3.0.1 with 2068 NIHS chemicals
9^*2	ISS	in vitro Mutagenicity (Ames test) by ISS- modified2020
10	Gifu University	xenoBiotic 0.9q
11	Massachusetts Institute of Technology	Chemprop
12–1^*3	Simulations Plus Inc.	Average of S+MUT_NIHS_ABC model and S+MUT_NIHS_AC model
13	Chemotargets	CHMT_GBoostSC
14	LMC – Bourgas University	TIMES_AMES 17.17.3 (in domain TIMES model)
15–1	The University of Sydney	DRSpicySTiM-Ensemble
16–1	Meiji Pharmaceutical University	MMI-STK2
17–1	Liverpool John Moores University	DL
18	Evergreen AI, Inc.	Avalon
19	Politecnico di Milano	GCN
20-a	NCTR/FDA (one of two best models)	DeepAmes
20-b	NCTR/FDA (one of two best models)	Decision Forest
21	NIBIOHN	GNN(kMoL)_bestbalanced

^*1The model number encodes the team number shown in Table 2 and whether the model was selected before (1) or after (2) access to the Ames test results. Letters indicate that more than one model was selected by a team (see NCTR/FDA). Team no. 2 selected a model only after access to the Ames test results.

^*2in vitro Mutagenicity (Ames test) by ISS- modified2020 is not publicly available.

^*3Performance metrics were calculated as the average of values generated by two models (S+MUT_NIHS_ABC and S+MUT_NIHS_AC) among the three submitted models (S+MUT_NIHS_ABC, S+MUT_NIHS_AC and S+MUT_NIHS).

Table 8. Models selected by each team AFTER access to Ames test results for the trial chemicals.

Model no.^*1	Team name	After being informed of the results
1	SIOC, CAS	CISOC-PSMT (SIOC, CAS, China)
2–2^*2	Altox Ltd.	GeneTox-iS – Prototype - Decision tree core v.1.0
3	MN-AM	ChemTunes. ToxGPS Ames NIHS_v2
4	Instem	Leadscope 2nd QSAR Challenge Consensus Model
5–2	IRFMN	NCSTOXVega-0.18 version 0.18
6	IdeaConsult Ltd.	AMBIT DeepN v4.85
7	MultiCASE Inc.	PHARM_BMUT model version (1.8.0.0.17691.350)
8	Lhasa Ltd.	Sarah Nexus v.3.0.1 with 2068 NIHS chemicals
9^*3	ISS	in vitro Mutagenicity (Ames test) by ISS- modified2020
10	Gifu University	xenoBiotic 0.9q
11	Massachusetts Institute of Technology	Chemprop
12–2	Simulations Plus Inc.	S+MUT_NIHS
13	Chemotargets	CHMT_GBoostSC
14	LMC – Bourgas University	TIMES_AMES 17.17.3 (in domain TIMES model)
15–2	The University of Sydney	GreedyMBAK
16–2	Meiji Pharmaceutical University	MMI-VOTE1
17–2	Liverpool John Moores University	RF model2
18	Evergreen AI, Inc.	Avalon
19	Politecnico di Milano	GCN
20-a	NCTR/FDA (one of two best models)	DeepAmes
20-b	NCTR/FDA (one of two best models)	Decision Forest
21	NIBIOHN	GNN(kMoL)_bestbalanced

^*1Model number encodes team number shown in Table 2 and whether the model was selected before (1) or after (2) access to Ames test results. Letters indicate that more than one model was selected by a team (see NCTR/FDA).

^*2A model was selected only after access to the Ames test results.

^*3in vitro Mutagenicity (Ames test) by ISS- modified2020 is not publicly available.

Table 9. Performance metrics for the models selected BEFORE access to the Ames test results^*1.

Model no.	A-Sens. (%)	Sens. (%)	Spec. (%)	Acc. (%)	BA (%)	MCC	Cov. (%)	F1-score (%)	Training data^*2	Model category^*3
1	68.8	54.0	78.8	75.1	66.4	0.27	99.3	39.3	(1)	Rule-based + Statistical
3	89.3	76.9	80.0	79.5	78.5	0.45	80.9	53.8	(2)	Rule-based + Deep-learning (statistical)
4	88.5	62.7	84.6	81.4	73.7	0.40	97.9	49.7	(2)	Statistical
5–1	58.2	44.7	89.6	82.9	67.1	0.34	99.9	43.7	(2)	Rule-based + Statistical
6	60.0	40.7	90.5	83.5	65.6	0.31	82.2	40.8	(2)	Deep-learning (statistical)
7	71.4	50.8	91.5	85.9	71.2	0.42	86.8	49.7	(3)	Statistical
8	74.3	62.0	75.9	73.9	69.0	0.29	80.4	41.0	(3)	Statistical
9	60.0	47.5	78.1	73.6	62.8	0.21	100	34.8	(2)^*4	Rule-based
10	33.3	23.7	96.9	86.0	60.3	0.30	98.7	33.4	(3)	Statistical
11	48.8	32.2	96.3	86.8	64.3	0.38	100	42.0	(2)	Statistical
12–1	72.7	62.8	76.1	74.2	69.5	0.30	94.1	41.2	(3)^*5	Deep-learning (statistical)
13	50.0	33.5	95.0	85.9	64.3	0.35	100	41.4	(2)	Statistical
14	95.7	58.0	88.6	84.2	73.3	0.42	35.9	51.1	(1)^*6	Rule-based
15–1	52.5	35.6	91.7	83.4	63.7	0.30	100	38.9	(1)^*7	Statistical
16–1	71.3	53.4	90.7	85.1	72.0	0.43	100	51.6	(2)	Statistical
17–1	19.0	20.0	79.3	70.5	49.6	−0.01	99.5	16.8	(2)	Deep-learning (statistical)
18	76.3	58.1	85.8	81.7	71.9	0.38	100	48.5	(2)	No information
19	58.8	36.4	95.2	86.5	65.8	0.38	100	44.4	(2)	Deep-learning (statistical)
20-a	58.4	47.4	90.8	84.3	69.1	0.38	97.1	47.6	(3)	Statistical
20-b	45.0	31.8	86.5	78.4	59.2	0.18	100	30.4	(2)	Statistical
21	55.0	39.4	95.0	86.8	67.2	0.41	100	47.0	(3)	Deep-learning (statistical)

^*1Model no. is shown in Table 7. Abbreviations as in Table 4. All metrics range from 0% to 100% except MCC, which ranges from − 1 to 1.

^*2Training data category: (1) release model (excluding data provided by the Second Project), (2) release model + all NIHS data, (3) release model + selected NIHS data.

^*3Models were categorized as rule-based, statistical or deep-learning.

^*4In the case of (1), A-Sens. = 78.8%, Sens. = 58.5%, Spec. = 73.2%, Acc. = 71.1%, BA = 65.9%, MCC = 0.24, Cov. = 100.0%, F1-score = 37.5%.

^*5Values were the average of two models (S+MUT_NIHS_ABC and S+MUT_NIHS_AC) of the three models submitted (S+MUT_NIHS_ABC, S+MUT_NIHS_AC and S+MUT_NIHS).

^*6In the case of all chemicals, A-Sens. = 76.3%, Sens. = 52.1%, Spec. = 83.0%, Acc. = 78.4%, BA = 67.5%, MCC = 0.30, Cov. = 100.0%, F1-score = 41.8%.

^*7This team reported the training data as (1), but it appears to be (2) according to the additional comments entered into the model information sheet.

Table 10. Performance metrics for the models selected AFTER access to the Ames test results^*1.

Model no.	A-Sens. (%)	Sens. (%)	Spec. (%)	Acc. (%)	BA (%)	MCC	Cov. (%)	F1-score (%)	Training data^*2	Model category^*3
1	68.8	54	78.8	75.1	66.4	0.27	99.3	39.3	(1)	Rule-based + Statistical
2–2	83.1	57.9	87.2	82.9	72.6	0.40	77.8	50.0	(1)	Statistical
3	89.3	76.9	80.0	79.5	78.5	0.45	80.9	53.8	(2)	Rule-based + Deep-learning (statistical)
4	88.5	62.7	84.6	81.4	73.7	0.40	97.9	49.7	(2)	Statistical
5–2	75.0	61.4	74.9	72.9	68.1	0.28	99.9	40.2	(3)	Rule-based + Statistical
6	60.0	40.7	90.5	83.5	65.6	0.31	82.2	40.8	(2)	Deep-learning (statistical)
7	71.4	50.8	91.5	85.9	71.2	0.42	86.8	49.7	(3)	Statistical
8	74.3	62.0	75.9	73.9	69.0	0.29	80.4	41.0	(3)	Statistical
9	60.0	47.5	78.1	73.6	62.8	0.21	100	34.8	(2)	Rule-based
10	33.3	23.7	96.9	86.0	60.3	0.30	98.7	33.4	(3)	Statistical
11	48.8	32.2	96.3	86.8	64.3	0.38	100	42.0	(2)	Statistical
12–2	77.3	69.3	73.1	72.6	71.2	0.32	94.1	42.1	(1)	Deep-learning (statistical)
13	50.0	33.5	95.0	85.9	64.3	0.35	100	41.4	(2)	Statistical
14	95.7	58.0	88.6	84.2	73.3	0.42	35.9	51.1	(1)	Rule-based
15–2	75.0	64.4	75.8	74.1	70.1	0.31	100	42.5	(1)	Deep-learning (statistical)
16–2	82.5	72.0	82.0	80.6	77.0	0.44	100	52.4	(2)	Statistical
17–2	75.9	64.7	72.7	71.5	68.7	0.28	99.5	40.3	(3)	Statistical
18	76.3	58.1	85.8	81.7	71.9	0.38	100	48.5	(2)	No information
19	58.8	36.4	95.2	86.5	65.8	0.38	100	44.4	(2)	Deep-learning (statistical)
20a	58.4	47.4	90.8	84.3	69.1	0.38	97.1	47.6	(3)	Statistical
20b	45.0	31.8	86.5	78.4	59.2	0.18	100	30.4	(2)	Statistical
21	55.0	39.4	95.0	86.8	67.2	0.41	100	47.0	(3)	Deep-learning (statistical)

^*1Model no. is shown in Table 8. Abbreviations as in Table 4.

^*2Training data category: (1) release model (excluding data provided by the Second Project) (2) release model + all NIHS data (3) release model + selected NIHS data.

^*3Models were categorized as rule-based, statistical or deep-learning.

Table 11. Averages (and ranges) of performance metrics for all models in the Second Project versus the First Project^*1.

	Second Project			First Project^*2
	Selected models BEFORE access to Ames test results	Selected models BEFORE access to Ames test results (First Project participates only)^*3	Selected models AFTER access to Ames test results	Phase I	Phase II	Phase III
A-Sens. (%)	62.2 (19.0–95.7)	74.5 (58.2–95.7)	68.3 (33.3–95.7)	68.9 (51.4–82.8)	72.4 (55.3–89.5)	71.1 (42.7–85.7)
Sens. (%)	46.3 (20.0–76.9)	56.2 (40.7–76.9)	52.0 (23.7–76.9)	55.8 (38.6–70.0)	56.6 (41.6–72.1)	56.6 (31.7–70.4)
Spec. (%)	87.5 (75.9–96.9)	83.9 (75.9–91.5)	85.2 (72.7–96.9)	78.7 (62.5–91.5)	85.4 (64.9–93.5)	80.1 (60.7–93.0)
Acc. (%)	81.4 (70.5–86.8)	79.9 (73.6–85.9)	80.4 (71.5–86.8)	75.3 (63.6–83.9)	81.2 (65.8–87.7)	76.8 (61.1–87.3)
BA (%)	66.9 (49.6–78.5)	70.1 (62.8–78.5)	68.6 (59.2–78.5)	67.2 (62.1–72.5)	71.0 (64.0–78.9)	68.4 (62.0–74.4)
PPV (%)	42.5 (14.4–60.3)	38.8 (27.5–48.7)	41.7 (27.5–60.3)	32.3 (23.8–46.1)	42.6 (27.4–58.2)	34.1 (21.1–51.0)
NPV (%)	90.5 (85.0–95.0)	92.0 (89.5–95.0)	91.3 (87.9–95.0)	91.4 (89.4–93.4)	91.9 (88.1–94.2)	91.9 (89.1–93.6)
MCC	0.33 (−0.01–0.45)	0.35 (0.21–0.45)	0.34 (0.18–0.45)	0.28 (0.20–0.39)	0.38 (0.24–0.50)	0.31 (0.17–0.44)
Cover. (%)	93.0 (35.9–100)	84.2 (35.9–100)	92.3 (35.9–100)	86.7 (14.5–100)	85.5 (18.0–100)	86.0 (9.7–100)
F1-score (%)	42.2 (16.8–53.8)	45.1 (34.8–53.8)	43.7 (30.4–53.8)	40.0 (31.8–48.9)	47.8 (36.9–57.9)	41.5 (31.7–51.5)
Number of models	21	9	22	18	21	19
No. of chemicals	1,589	1,589	1,589	3,902	3,829	4,409

^*1Abbreviations as in Table 4. Selected model information (names and training data) before and after access to Ames test results are shown in Tables 7–10.

^*2All models from Tables 5–7 of Honma et al. [14].

^*3Nine teams that participated in both the First Project and the Second Projects are indicated in Table 2.

Figure 1. Receiver operating characteristic graphs of Ames mutagenicity prediction for the QSAR models. (a, b) Sensitivity to class A chemicals (A-sensitivity) versus specificity to class C chemicals. (c, d) Sensitivity to class A + B chemicals versus specificity to class C chemicals. Each black dot represents a single QSAR model from one of the participating teams. Red and yellow circles indicate the models selected before and after access to the Ames test data, respectively.

M. Honma, A. Kitazawa, A. Cayley, R.V. Williams, C. Barber, T. Hanser, R. Saiakhov, S. Chakravarti, G.J. Myatt, K.P. Cross, E. Benfenati, G. Raitano, O. Mekenyan, P. Petkov, C. Bossa, R. Benigni, C.L. Battistelli, A. Giuliani, O. Tcheremenskaia, C. DeMeo, U. Norinder, H. Koga, C. Jose, N. Jeliazkova, N. Kochev, V. Paskaleva, C. Yang, P.R. Daga, R.D. Clark, and J. Rathman, Improvement of quantitative structure–activity relationship (QSAR) tools for predicting Ames mutagenicity: Outcomes of the Ames/QSAR International Challenge Project, Mutagenesis 34 (2019), pp. 3–16. doi:10.1093/mutage/gey031.

 PubMed Web of Science ®Google Scholar