A methodology for the automatic evaluation of data quality and completeness of nanomaterials for risk assessment purposes

Figures & data

Figure 1. Example of data reporting template for an ecotoxicological study, partially filled with information related to a specific study.

Figure 2. Details and studies for a particular material (here TiO₂) in the eNanoMapper database. For this particular NM, data related to ecotoxicological studies (4 studies), physicochemical studies (157 studies) and toxicological studies (130 studies) are available. Data is structured according to the eNanoMapper data model (Jeliazkova et al. 2015; Kochev et al. 2020).

Table 1. State-of-the-art criteria for evaluating data quality and completeness.

Criteria \ Ref	Completeness	Reliability	Relevance	Adequacy	Additional criteria
(Klimisch et al. 1997)	NA	Four categories (‘Reliable without restrictions,’ ‘Reliable with restrictions,’ ‘Not reliable,’ and ‘Not assignable) based on 12 criteria for acute studies and 14 for chronic studies.	Covers the extent to which data and tests are appropriate for risk assessment.	Defines the usefulness of data for risk assessment purposes.	NA
(Card and Magnuson 2010)	List of 10 key physicochemical properties adequately characterizing NMs in human toxicity studies. Results in a score 0-10.	Base of ToxRTool. Generic questions related to 1) Test substance identification; 2) Test system characterization; 3) Study design description; 4) Study results documentation and 5) Plausibility design and results.	NA	NA	NA
(Lubinski et al. 2013)	NA	Based on Klimisch criteria, focused on quality for QSAR models	NA	NA	Usefulness of the data for developing computational models: The Klimisch criteria have been expanded and revised to take into account the specific nature of nanomaterials, which differs from bulk compounds in terms of size, shape, and surface properties, etc. (related to ‘quality’). In addition, the dataset size is considered as a fundamental criterion for assessing usefulness for computational modeling.
(Hristozov et al. 2014) (ENPRA/MARINA/SUN)	NA	As in Klimisch et al. (Klimisch et al. 1997).	NA	The two main types of toxicological studies (i.e. in vivo and in vitro) are considered and weighted differently, moreover such studies are subdivided by means of the route of exposure.	Statistical power: statistical significance and sample size. Toxicological significance: relevance of the dose used in the test system.
(Moermond et al. 2016)	NA	Concerns the intrinsic scientific quality of the study, regardless of the purpose for which it is assessed. It is determined by an assessment of the design, performance, and analysis of the experiment. Four categories: reliable without restrictions (R1), reliable with restrictions (R2), not reliable (R3), and not assignable (R4). Based on 20 criteria related to Test setup, test compound, test organism, exposure conditions, statistical design, and biological response.	Relevance depends on the purpose of the assessment and concerns the way the study will be used for a specific purpose. 4 relevance categories: relevant without restrictions (C1), relevant with restrictions (C2), not relevant (C3), and not assignable (C4). Based on 13 criteria related to biological relevance and exposure relevance.	NA	NA
(Hartmann et al. 2017)	NA	Reliability categorization of nanomaterial ecotoxicity based on 1) the appropriateness of the study design for the purpose of nanomaterial testing, 2) documentation provided on the study design, 3) data on nanomaterial properties and d) nanomaterial characterization in the test system and during the test. Revised version of Moermond et al. (Moermond et al. 2016), specific for NMs, where each criterion has different weights (1, low importance, to 3, critical importance)	Revised version of the criteria defined by Moermond et al. (Moermond et al. 2016), with nano-specific guidance on certain criteria. Not weighted, as opposed to reliability.	NA	NA
(Fernández-Cruz et al. 2018) (GUIDEnano)	Substance characterization score, based on a modified version of the work proposed by Card & Magnusson (Card and Magnuson 2010).	Based on a Modified version of the ToxRTool, reducing as much as possible the need of expert judgment. Applies for in-vivo and invitro data (Human Health) as well as ecotoxicological data. Questions are divided in three groups: test organism characterization, study design description and results documentation.	NA	NA	NA
(Krug et al. 2018) (DANA/NanoGRAVUR)	Minimum information checklist: criteria adopted in DANA database to evaluate scientific literature before inserting it in the knowledge base. It consists of a set of minimum information that must or might be provided (related to Physicochemical properties, sample preparation, testing parameters and other general aspects, i.e. if data evaluation/statistics are provided or whether a SOP was used). Also related to data reliability.	NA	NA	NA	NA
(Comandella et al. 2020)	Based on 11 physicochemical properties and 30 associated experimental techniques, a Completeness Score (CS) is computed for each entry of a template related to a study as the number of items reported divided by the number of items required. This then is averaged for each material to compute the CS for ‘a template’ and may be further generalized to compute a CS for a query.	NA	NA	NA	NA
(Basei et al. 2021) (partial implementation of the methodology described in this paper).	Same as Comandella et al. (Comandella et al. 2020), extended to also consider (eco)toxicological data reporting templates	Based on ToxRTool/ NanoCRED tool outcomes.	Based on the relevance of the adopted protocol with respect to the endpoint (protocols suggested by ECHA CLP guidance or IR&CSA guidance are prioritized)	In vivo data is considered more reliable than data obtained in vitro or in silico.	NA

NA = Not assessed, or not directly assessed.

Table 2. Comparison of the criteria for assessing reliability of in vivo data by means of the ToxRTool (Schneider et al. 2009) and corresponding mapping on the criteria adopted by the methodology for data quality and completeness.

	ToxRTool criteria (in vivo data)	Mapping in other criteria included in the data quality and completeness methodology
Criteria Group I: Test substance identification	1. Identification of the test substance	Substance names and CAS (Chemicals Abstract Source) number are included in every template. Composition is one of the 11 parameters considered when computing the CS for physicochemical characterization.
	2. Information on the purity	This information is included in the template Surface Chemistry.
	3. Information on the source/origin of the substance	Information on NM supplier is included in each data reporting template.
	4. Information on the nature and/or indispensable (for the test) physicochemical properties, such as particle size, physical state, and pH.	Key physicochemical properties are covered by the 11 parameters considered when computing the CS for physicochemical characterization.
Criteria Group II: Test organism characterization	5. Information on the test species	This information is included in toxicological templates.
	6. Information on the sex of the test organism	This information is included in toxicological templates
	7. Information on the strain of test animals	This information is included in toxicological templates.
	8. Information on age/body weight at the start of the study	This information is included in toxicological templates.
	9. Information on housing/feeding conditions (for dose–response studies only)	This information is included in dose–response toxicological templates.
Criteria group III: Study design description	10. Information on the administration route	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	11. Information on doses administered or concentration in application media	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	12. Information on frequency and duration of exposure, as well as time-points of observation	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	13. Are negative/positive controls included?	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	14. Information on the number of animals per group	This information is included in in toxicological templates.
	15. Details on the administration scheme	This information is implicitly derivable from the adopted protocol.
	16. Were achieved concentrations analytically verified, or was the stability of the test substance ensured or made plausible? (Inhalation and repeated dose toxicity only)	This information is only partially included in toxicological templates.
Criteria group IV: Study results documentation	17. Information on the study endpoint and method of determination.	This information is included in toxicological templates.
	18. Is the description of results for all endpoints investigated transparently and complete?	This information is directly related to the completeness of toxicological templates.
	19. Information/documentation on the statistical methods applied for data analysis	This information is implicitly derivable from the results Section of toxicological templates.
Criteria group V: Plausibility of study design and results	20. Information of the study design appropriateness for obtaining the substance-specific data	This information is directly related to the completeness of toxicological templates, and to the relevance of the adopted protocol.
	21. Are the quantitative study results reliable?	This information requires expert judgment, thus is only indirectly related to completeness and relevance.

Parameters highlighted in bold are considered essential to be categorized at least in Category 2 by the ToxRTool.

Table 3. Comparison of the criteria for assessing reliability of in vitro data by means of the ToxRTool (Schneider et al. 2009) and corresponding mapping on the criteria adopted by the methodology for Data Quality and Completeness.

	ToxRTool criteria (in vitro data)	Mapping in other criteria included in the data quality and completeness methodology
	Criteria Group I - Test substance identification: criteria 1-4 are the same as Table 2.	Considerations for criteria 1-4 are the same as Table 2.
Criteria Group II: Test system characterization	5. Description of the test system	This information is included in toxicological templates, and directly related to the adopted protocol (relevance).
	6. Information on source/origin of the test system	This information is included in toxicological templates.
	7. Information on test system properties, conditions of cultivation and maintenance	This information is included in toxicological templates and related to the adopted protocol (relevance).
Criteria group III: Study design description	8. Information on the method of administration	This information is included in the toxicological template, and implicitly derivable from the adopted protocol.
	9. Information on doses administered or concentration in application media	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	10. Information on frequency and duration of exposure, as well as time-points of observation	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	11. Are negative controls included?	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	12. Are positive controls included?	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	13. Are the number of replicates provided?	This information is included in toxicological templates.
	Criteria group IV - Study result: criteria 14-16 are the same as criteria 17-19 in Table 2.	Considerations for criteria 14-16 are similar to the considerations for criteria 17-19 in Table 2.
	Criteria group V - Plausibility of study design and results: criteria 17-18 are the same as criteria 19-20 in Table 2.	Considerations for criteria 17-18 are similar to the considerations for criteria 19-20 in Table 2.

Parameters highlighted in bold are considered essential to be categorized at least in Category 2 by the ToxRTool.

Table 4. Comparison of the criteria for assessing reliability of ecotoxicological data by means of the NanoCRED evaluation method for reliability (Moermond et al. 2016; Hartmann et al. 2017), and corresponding mapping on the criteria adopted by the methodology for data quality and completeness.

	NanoCRED criteria (ecotoxicological data)	Mapping in other criteria included in the data quality and completeness methodology
Criteria Group I: General information	1. Information on the (modified) guideline method used	This information is included in ecotoxicological templates and is directly related to the relevance criteria.
	2. Is the test performed under good laboratory practices conditions?	This information is indirectly related to the relevance criteria.
	3. Information on the validity criteria (e.g., control survival, growth)	This information is indirectly related to the relevance criteria and partially included in parameters of ecotoxicological templates.
	4. Information on the test controls (e.g., negative/positive controls, solvent)	This information is directly related to the relevance criteria and included in parameters of ecotoxicological templates.
Criteria Group II: Test compound	5. Information on the substance (CAS number or substance name)	This information is included in both psychochemical and ecotoxicological templates.
	6. Information on the purity of the substance	This information is included in the template Surface Chemistry.
	7. If a formulation is used or there are impurities, is an environmental effect known for such impurities/formulation? Is the amount known?	This information is partially included in templates Surface Chemistry and in information on the Composition.
Criteria group III: Study test organism	8. Information on test organism	This information is included in toxicological templates, and implicitly derivable from the adopted protocol.
	9. Are the test organisms from a trustworthy source and acclimatized to test conditions? Have the organisms not been pre-exposed to test compound or other unintended stressors?	This information is implicitly derivable from the adopted protocol. Information on the provider is included in ecotoxicological templates.
Criteria group IV: Exposure conditions	10. Information on the appropriateness of the experimental system (whether it takes into consideration physicochemical properties of the NM)	This information is strongly based on expert judgment; however, the physicochemical characterization is highly considered in the CS computation, and this information in principle is related to the relevance of the adopted protocol.
	11. Information on the test system and on the test conditions (including the stability of such conditions during the experiment)	This information is included in ecotoxicological templates.
	12. Information on the exposure concentration with respect to water solubility, as well as on the solvent used	This information is included in ecotoxicological templates.
	13. Information on the spacing between exposure concentrations	This information is directly related to the adopted test protocol (data relevance).
	14. Information on exposure duration	This information is directly related to the adopted test protocol (data relevance) and included in ecotoxicological templates.
	15. Information on the chemical analyses adopted to verify concentrations during the study	This information is partially included in ecotoxicological templates, and implicitly related to the test protocol (data relevance).
	16. Information on the appropriateness of biomass loading of organisms in the test system	This information is implicitly related to the adopted test protocol (data relevance).
Criteria group V: Statistical design and biological response	17. Information on the number of replicates, for the test and positive/negative controls	This information is implicitly related to the adopted test protocol (data relevance) and partially included in ecotoxicological templates.
	18. Information on the statistical methods used	Partially accounted in the results Section of ecotoxicological templates.
	19. Information on the existence of a dose–response relationship and on the statistical significance of the response	Partially accounted in the results Section of ecotoxicological templates.
	20. Information on the amount of data available to check the correctness of calculations on endpoints.	Partially accounted in the results Section of ecotoxicological templates, and directly related to the adopted test protocol (data relevance).

Table 5. Proposed setup for the selected criteria.

Criteria	Description	Score
Completeness	Data completeness is assessed following the methodology proposed by Comandella et al. (Comandella et al. 2020), extended in order to evaluate also completeness of (eco)toxicological templates. The CS is computed as described in Section 3.2.1. The final CS is the average of the CSs of physicochemical templates and the CS of the (eco)toxicological template. A possible improvement consists of providing weights to the parameters of each template, assigning higher weighs to parameters that are considered more important by the user, and/or computing the CS on each section of the template, weighing the score for each section, and then averaging the scores (cf. Section 3.2.1).	Corresponds to the final CS (i.e. the average of the CS related to physicochemical templates, and the CS related to (eco)toxicological template).
Relevance	Data is considered relevant to the task if they are derived using internationally recognized protocols and methods, as described in Section 3.2.2. This step was automatized by providing a list of protocols for the first two categories with respect to each endpoint of interests.	Category 1 (data derived following standard guidelines): 1 Category 2 (data derived following nanospecific guidelines, guidelines with modifications, or protocols that are candidate to become guidelines): 0.7 Category 3 (data derived following protocols that are not considered in Category 1 or Category 2): 0.1 Category 4 (data for which the adopted protocol is not reported): Data should be considered of insufficient quality
Adequacy	Data adequacy is evaluated based on the fact that in general in vivo data (Category 1) is considered to be more appropriate than in vitro (Category 2) and in silico (Category 3) data for risk assessment from a regulatory perspective, as described in Section 3.2.4. This was again automized by associating an adequacy score to the reported protocol.	Category 1 (in vivo): 1 Category 2 (in vitro): 0.6 Category 3 (in silico): 0.3

Table 6. Results of applying the methodology to publicly available data in eNanoMapper related to JRCNM03000a (NM-300K, Ag 16.7 nm). Overall completeness and quality scores for the studies are highlighted in bold. The traffic light indicator (last row of the table) highlights both the 'traffic light' and the corresponding meaning (i.e., 'green light' for data of high quality, 'yellow light' for data of sufficient quality, and 'red light' for data of insufficient quality).

NM	JRCNM03000a (NM-300K, Ag 16.7 nm)
Reference	NANoREG D4.12 (NANoREG 2016a)
Endpoint	LC50	EC50				LC10	EC20	EC10
Endpoint value (mg/L)	[4.57, 22.22]	[0.71, 0.83]	[0.006, 9.4]	0.01	[4.68, 9.82]	[2.68, 15.65]	[0.52, 0.63]	[0.001, 2.35]	[1.78, 8.26]
Test method	OECD TG 203	OECD TG 202	OECD TG 201	OECD TG 201	OECD TG 201	OECD TG 203	OECD TG 202	OECD TG 201	OECD TG 201
Relevance	1	1	1	1	1	1	1	1	1
Adequacy	1	1	1	1	1	1	1	1	1
Completeness ($C{S}_{\mathit{physchem}}$)	0.35	0.36	0.35	0.24	0.28	0.35	0.36	0.35	0.28
	(0.30)	(0.30)	(0.30)	(0.30)	(0.24)	(0.30)	(0.30)	(0.30)	(0.24)
Overall score	0.78	0.79	0.78	0.78	0.76	0.78	0.79	0.78	0.76
Traffic light indicator	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a	Data is of sufficient quality^a

^aEven though the overall score is greater than 0.7, data is of sufficient (not high) quality because the CS related to the physicochemical characterization is below the threshold of 0.5 (i.e. it is not sufficiently characterized).

Figure 3. Aggregate CSs related to physicochemical characterization of the selected data for the second case study (i.e. genotoxicity in vitro). Scores have been classified using the thresholds defined in Section 3.2.5 for the traffic light system into four categories (i.e. ‘No data,’ ‘Incomplete data,’ ‘Sufficiently complete data,’ and ‘Highly complete data’).

Figure 4. Aggregate CSs related to the overall physicochemical characterization, the toxicological test, and the overall CS for physicochemical data of the selected data for the second case study (i.e. genotoxicity in vitro). Scores have been classified using the thresholds defined in Section 3.2.5 for the traffic light system into three categories (i.e. ‘Incomplete data,’ ‘Sufficiently complete data,’ and ‘Highly complete data’).

Figure 5. Results of the evaluation of data quality and completeness for genotoxicity. Two studies resulted to be of high quality, while 42 studies resulted to be of sufficient quality.

Figure 6. Implementation of the data completeness and reliability criteria in the development version of eNanoMapper. When downloading information selecting as format ‘NANoREG template,’ completeness is computed, and the completeness score is highlighted using an heatmap from red (0, data fully incomplete) to green (1, data fully complete). The numbers between parentheses in the left menu of eNanoMapper represent the total number of data points associated to each category, after application of filters. The number of substances, instead, is shown on the top of the main window.

Figure 7. A list of Standard Operating Procedures is already provided in the eNanoMapper database when selecting a reporting template.

Figure 8. Screenshot of the initial version of a tool implementing the data quality assessment methodology. The results of the case study related to genotoxicity in vitro (cf. Section 3.3.2) are displayed. Currently, the tool computes data quality and completeness scores starting from the selected endpoint, the CSs of physicochemical parameters, and the classification into data reliability and adequacy categories, and then displays aggregate information on data quality and completeness of the dataset. In future versions, it will be able to directly query the eNanoMapper database.

Jeliazkova, N., C. Chomenidis, P. Doganis, B. Fadeel, R. Grafström, B. Hardy, J. Hastings, et al. 2015. “The eNanoMapper Database for Nanomaterial Safety Information.” Beilstein Journal of Nanotechnology 6 (1): 1609–1634. doi:https://doi.org/10.3762/bjnano.6.165.

 PubMedGoogle Scholar

Kochev, N., N. Jeliazkova, V. Paskaleva, G. Tancheva, L. Iliev, P. Ritchie, and V. Jeliazkov. 2020. “Your Spreadsheets Can Be Fair: A Tool and Fairification Workflow for the Enanomapper Database.” Nanomaterials 10 (10): 1908.

 Google Scholar

Klimisch, H. J., M. Andreae, and U. Tillmann. 1997. “A Systematic Approach for Evaluating the Quality of Experimental Toxicological and Ecotoxicological Data.” Regulatory Toxicology and Pharmacology: RTP 25 (1): 1–5. doi:https://doi.org/10.1006/rtph.1996.1076.

 PubMed Web of Science ®Google Scholar

Card, J. W., and B. A. Magnuson. 2010. “A Method to Assess the Quality of Studies That Examine the Toxicity of Engineered Nanomaterials.” International Journal of Toxicology 29 (4): 402–410. doi:https://doi.org/10.1177/1091581810370720.

 PubMed Web of Science ®Google Scholar

Lubinski, L., P. Urbaszek, A. Gajewicz, M. T. D. Cronin, S. J. Enoch, J. C. Madden, D. Leszczynska, J. Leszczynski, and T. Puzyn. 2013. “Evaluation Criteria for the Quality of Published Experimental Data on Nanomaterials and Their Usefulness for QSAR Modelling.” SAR and QSAR in Environmental Research 24 (12): 995–1008. doi:https://doi.org/10.1080/1062936X.2013.840679.

 PubMed Web of Science ®Google Scholar

Hristozov, D. R., A. Zabeo, C. Foran, P. Isigonis, A. Critto, A. Marcomini, and I. Linkov. 2014. “A Weight of Evidence Approach for Hazard Screening of Engineered Nanomaterials.” Nanotoxicology 8 (1): 72–87. doi:https://doi.org/10.3109/17435390.2012.750695.

 PubMed Web of Science ®Google Scholar

Moermond, C. T. A., R. Kase, M. Korkaric, and M. Ågerstrand. 2016. “CRED: Criteria for Reporting and Evaluating Ecotoxicity Data.” Environmental Toxicology and Chemistry 35 (5): 1297–1309. doi:https://doi.org/10.1002/etc.3259.

 PubMedGoogle Scholar

Hartmann, N. B., M. Ågerstrand, H. C. H. Lützhøft, and A. Baun. 2017. “NanoCRED: A Transparent Framework to Assess’ the Regulatory Adequacy of Ecotoxicity Data for Nanomaterials – Relevance and Reliability Revisited.” NanoImpact 6: 81–89. doi:https://doi.org/10.1016/j.impact.2017.03.004.

 Google Scholar

Fernández-Cruz, M. L., D. Hernández-Moreno, J. Catalán, R. K. Cross, H. Stockmann-Juvala, J. Cabellos, V. R. Lopes, et al. 2018. “Quality Evaluation of Human and Environmental Toxicity Studies Performed with Nanomaterials-the GUIDEnano Approach.” Environmental Science: Nano 5 (2): 381–397. doi:https://doi.org/10.1039/C7EN00716G.

 Google Scholar

Krug, H. F., N. Bohmer, D. Kühnel, C. Marquardt, K. Nau, and C. Steinbach. 2018. “The DaNa2.0 Knowledge Base Nanomaterials—an Important Measure Accompanying Nanomaterials Development.” Nanomaterials 8 (4): 204. doi:https://doi.org/10.3390/nano8040204.

 Web of Science ®Google Scholar

Comandella, D., S. Gottardo, I. M. Rio-Echevarria, and H. Rauscher. 2020. “Quality of Physicochemical Data on Nanomaterials: An Assessment of Data Completeness and Variability.” Nanoscale 12 (7): 4695–4708. doi:https://doi.org/10.1039/c9nr08323e.

 PubMed Web of Science ®Google Scholar

Basei, G., A. Zabeo, K. Rasmussen, G. Tsiliki, and D. Hristozov. 2021. “A Weight of Evidence Approach to Classify Nanomaterials according to the EU Classification, Labelling and Packaging Regulation Criteria.” NanoImpact 24: 100359. doi:https://doi.org/10.1016/j.impact.2021.100359.

 PubMedGoogle Scholar

Schneider, K., M. Schwarz, I. Burkholder, A. Kopp-Schneider, L. Edler, A. Kinsner-Ovaskainen, T. Hartung, and S. Hoffmann. 2009. “ "ToxRTool", a new tool to assess the reliability of toxicological data.” Toxicology Letters 189 (2): 138–144. doi:https://doi.org/10.1016/j.toxlet.2009.05.013.

 PubMed Web of Science ®Google Scholar

NANoREG. 2016a. Accumulation potential and aquatic toxicity of relevant groups of nanomaterials and product formula. https://www.rivm.nl/sites/default/files/2018-11/NANoREG%20D4.12%20FS%20Accumulation%20potential%20and%20aquatic%20toxicity%20of%20relevant%20groups%20of%20nanomaterials%20and%20product%20formula.pdf

 Google Scholar