Assessing the accuracy of sensitivity analysis: an application for a cellular automata model of Bogota’s urban wetland changes

Figures & data

Table 1. Data resources.

Data	Year	Type	Original resolution	Data resource
Images	1998, 2004, 2010	WMS resource	0.46 m	https://www.ideca.gov.co/recursos
Distance to the arterial road network		Vector	1:1000
DEM	2014	Raster	5 m
Urban geomorphology	2016	Vector	1:1000
Restricted Areas		Vector	1:1000
Population	2004, 2010	Document	Administrative	http://www.bogota.gov.co/
Cadastral information	1998, 2004, 2010	Vector	1:1000	https://serviciosgis.catastrobogota.gov.co/
Annual Mean Temperature		Vector	1:1500	https://datosabiertos.bogota.gov.co/
Annual Precipitation		Vector	1:1500
Annual Mean Precipitation		Vector	1:1500

Table 2. Area (ha) of each land use class.

Land use class	1998	2004	2010
Constructions	21 849.4	22 544.7	22 995.3
Crops and pastures	6 709.9	5 996.9	5 499.3
Quarries	254.1	243.9	244.3
Urban Green Spaces	1 592.1	1 659.5	1 758.7
Water	136.8	136.8	137.9
Wetlands	810.2	770.7	717.0

Figure 1. Study area of Bogota, Colombia.

Figure 2. Driving factors - a detailed map of urban geomorphology together with the legend is attached as Appendix (see Figure A.1).

Figure 3. Schematic diagram of the methodology.

Table 3. Metrics used for SA assessment.

Approaches	Formula	Description
Figure of merit (FoM)	$\mathit{FoM}= \frac{B}{A+B+C+D}$	The FoM is the intersection of observed and simulated changes divided by the addition of observed and predicted changes^(a).
Producer accuracy (PA)	$PA= \frac{n_{ii}}{n_{+k}}$	PA indicates the proportion of pixels that the model simulates well as change^(a).
Overall accuracy (OA)	$OA={\sum }_{i=1}^k \frac{n_{ii}}{n}$	OA provides the general agreement between the reference map and the simulated one^(a).
Area under the curve for the Total operating characteristic (TOC)	$\mathit{AUC}={\int }_0^1\mathit{TPR} d(\mathit{FPR})$	TOC metric indicates how well the model is forecasting change^(b).
Fuzzy Kappa statistic	$K=\frac{P-E}{1-E}$	It is used to demonstrate the overall agreement between what is observed and what is simulated^(c).
Quantity disagreement (QD)	$Q=\frac{{\sum }_{g=1}^J{q}_g}{2}$	Measures the disagreement between the number of cells in each category without considering the spatial location^(d).
Allocation disagreement (AD)	$A=\frac{{\sum }_{g=1}^J{a}_g}{2}$	Evaluates the amount of disagreement between the reference map and the simulation map regarding the spatial location of the cells in each category^(d).

^a (Varga et al. 2019).

^b (Azari et al. 2022).

^c (Hagen-Zanker 2009).

^d (Pontius and Millones 2011a).

Figure 4. Results from the SA assessment through cross-map method for 5 m spatial resolution. C (constructions), CP (crops and pastures), W (water), UGS (urban green spaces), Wet (wetlands).

Figure 5. Results from the SA assessment through the cross-map method for 30 m spatial resolution. C (constructions), CP (crops and pastures), W (water), UGS (urban green spaces), Wet (wetlands).

Table 4. Overall accuracy for cross-classification maps with wetlands mask.

Spatial resolution	Reference-Neighborhood size	Overall accuracy	Spatial resolution	Reference-Neighborhood size	Overall accuracy
5 m	3 x 3 vs 5 x 5	0.9893	30 m	3 x 3 vs 5 x 5	0.9701
	3 x 3 vs 7 x 7	0.9874		3 x 3 vs 7 x 7	0.9661
	5 x 5 vs 7 x 7	0.9871		5 x 5 vs 7 x 7	0.9639

Table 5. Landscape metrics for neighborhood sizes 3 x 3, 5 x 5, 7 x 7 for 5 m and 30 m spatial resolution.

Neighborhood size	NP	LPI (%)	TE (m)	MN (m^2)	SHAPE_AM	PAFRAC	AI (%)
5 m
Wetlands
Reference	27	0.66	117 311.533	26.54	3.18	1.45	98.02
3 × 3	121	0.63	137 602.474	6.04	3.36	1.24	97.71
5 × 5	334	0.63	160 272.939	2.19	3.74	1.27	97.32
7 × 7	470	0.63	180 588.875	1.55	4.04	1.31	96.97
30m
Wetlands
Reference	29	0.66	109 020	24.71	3.08	1.38	88.84
3 × 3	40	0.63	115 350	18.28	3.29	1.38	88.28
5 × 5	42	0.63	118 830	17.41	3.36	1.39	87.92
7 × 7	54	0.63	123 900	13.54	3.44	1.40	87.40

Table 6. SA assessment of the outcomes indicated by the OA, PA, and FoM.

Neighborhood size 5 m spatial resolution	PA**	OA*	FoM**	Neighborhood size 30 m spatial resolution	PA**	OA*	FoM**
3 × 3	0.864104	0.9401	0.1082	3 × 3	0.8714	0.935115	0.1173
5 × 5	0.864104	0.9401	0.1107	5 × 5	0.86	0.934225	0.1267
7 × 7	0.864101	0.9398	0.1132	7 × 7	0.8608	0.931885	0.1363

* for the whole map.

** for the wetland land use class.

Figure 6. AUC of the TOC - zoom of Figure A.1 with thresholds.

Figure 7. Graph of fuzzy Kappa index and similarity above each bar for wetlands land-cover.

Figure 8. Quantity disagreement, allocation disagreement.

Figure A1. Urban geomorphology.

Figure A2. Sensitivity analysis using TOC for wetlands land use class.

Table A1. Agreement and disagreement values for wetland land use class.

Resolution	Neighborhood	Change simulated as persistence	Correctly simulated change	Incorrectly simulated change	Persistence simulated incorrectly	Persistence simulated correctly
5 m	3 x 3	0.0035294605	0.0005201335	0.0001016665	6.562470e-04	0.01977552
	5 x 5	0.0035199717	0.0005315361	9.975273e-05	6.502666e-04	0.01978150
	7 x 7	0.0035100841	0.0005429387	0.0000982377	6.435686e-04	0.01978820
30 m	3 x 3	3.465304e-03	0.0005598461	0.0001090982	6.373633e-04	0.01983578
	5 x 5	3.419368e-03	0.0006000402	0.0001148402	6.029112e-04	0.01987023
	7 x 7	3.373432e-03	0.0006431053	0.0001177112	5.856852e-04	0.01988746

Table A2. Confusion matrix-5 m resolution 3x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	1781037	52939	367	13055	8	3501
CP	43684	377758	0	7023	41	3916
Q	2	0	19226	0	0	0
UGS	12777	3617	0	119491	3	396
W	1	14	0	0	10856	12
Wet	2363	5999	0	286	74	49756

Table A3. Confusion matrix-5m resolution 5x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	1781037	52939	367	13055	8	3501
CP	43684	377758	0	7023	41	3916
Q	2	0	19226	0	0	0
UGS	12777	3617	0	119491	3	396
W	1	14	0	0	10856	12
Wet	2363	5999	0	286	74	49756

Table A4. Confusion matrix-5m resolution 7x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	1780874	52578	317	13459	8	3401
CP	43801	377024	0	6985	38	3966
Q	7	0	19223	0	0	0
UGS	12610	4260	0	119730	3	394
W	0	9	0	0	11133	9
Wet	2387	6211	0	309	61	49405

Table A5. Confusion matrix-30m resolution 3x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	49304	1472	7	457	0	89
CP	1392	10300	0	196	1	106
Q	2	0	549	0	0	0
UGS	369	177	0	3292	0	11
W	0	0	0	0	300	0
Wet	72	163	0	5	1	1397

Table A6. Confusion matrix-30m resolution 5x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	49387	1479	5	461	0	108
CP	1390	10362	0	206	0	112
Q	1	0	516	0	0	0
UGS	306	255	0	3099	0	11
W	0	0	0	0	297	0
Wet	63	173	0	11	1	1419

Table A7. Confusion matrix-30m resolution 7x neighborhood size.

	C	CP	Q	UGS	W	Wet
C	49235	1456	6	564	0	92
CP	1453	10349	0	185	0	116
Q	0	0	539	0	0	0
UGS	310	326	0	3141	0	8
W	0	0	0	0	317	0
Wet	54	168	0	6	1	1336

Varga OG, Pontius RG, Singh SK, Szabó S. 2019. Intensity analysis and the figure of merit’s components for assessment of a cellular automata – Markov simulation model. Ecol Indic [Internet]. 101:933–942. [accessed 2021 December 20].

 Web of Science ®Google Scholar

Azari M, Billa L, Chan A. 2022. Multi-temporal analysis of past and future land cover change in the highly urbanized state of Selangor, Malaysia. Ecol Process [Internet]. 11(1):1–15. [accessed 2022 December 11].

 Web of Science ®Google Scholar

Hagen-Zanker A. 2009. An improved Fuzzy Kappa statistic that accounts for spatial autocorrelation. Int J Geograph Inform Sci [Internet]. 23(1):61–73. [accessed 2022 February 7].

 Web of Science ®Google Scholar

Pontius RG, Millones M. 2011a. Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int J Remote Sens [Internet]. 32(15):4407–4429. [accessed 2020 November 26].

 Web of Science ®Google Scholar