Repurposing a deep learning network to filter and classify volunteered photographs for land cover and land use characterization

Figures & data

Table 1. Scene categories from the Places205 project (Zhou et al. 2014).

abbey	construction_site	inn/outdoor	river
airport_terminal	corn_field	jail_cell	rock_arch
alley	corridor	kasbah	rope_bridge
amphitheater	cottage_garden	kindergarden_classroom	ruin
amusement_park	courthouse	kitchen	runway
aquarium	courtyard	kitchenette	sandbar
aqueduct	creek	laundromat	schoolhouse
arch	crevasse	lighthouse	sea_cliff
art_gallery	crosswalk	living_room	shed
art_studio	cathedral/outdoor	lobby	shoe_shop
assembly_line	church/outdoor	locker_room	shopfront
attic	dam	mansion	shower
auditorium	dining_room	marsh	ski_resort
apartment_building/outdoor	dock	martial_arts_gym	ski_slope
badlands	dorm_room	mausoleum	sky
ballroom	driveway	medina	skyscraper
bamboo_forest	desert/sand	motel	slum
banquet_hall	desert/vegetation	mountain	snowfield
bar	dinette/home	mountain_snowy	staircase
baseball_field	doorway/outdoor	music_studio	supermarket
basement	engine_room	market/outdoor	swamp
basilica	excavation	monastery/outdoor	stadium/baseball
bayou	fairway	museum/indoor	stadium/football
beauty_salon	fire_escape	nursery	stage/indoor
bedroom	fire_station	ocean	subway_station/platform
boardwalk	food_court	office	swimming_pool/outdoor
boat_deck	forest_path	office_building	television_studio
bookstore	forest_road	orchard	topiary_garden
botanical_garden	formal_garden	pagoda	tower
bowling_alley	fountain	palace	train_railway
boxing_ring	field/cultivated	pantry	tree_farm
bridge	field/wild	parking_lot	trench
building_facade	galley	parlor	temple/east_asia
bus_interior	game_room	pasture	temple/south_asia
butchers_shop	garbage_dump	patio	track/outdoor
butte	gas_station	pavilion	train_station/platform
bakery/shop	gift_shop	phone_booth	underwater/coral_reef
cafeteria	golf_course	picnic_area	valley
campsite	harbour	playground	vegetable_garden
candy_store	herb_garden	plaza	veranda
canyon	highway	pond	viaduct
castle	home_office	pulpit	volcano
cemetery	hospital	racecourse	waiting_room
chalet	hospital_room	raft	water_tower
classroom	hot_spring	railroad_track	watering_hole
closet	hotel_room	rainforest	wheat_field
clothing_store	hotel/outdoor	reception	wind_farm
coast	ice_cream_parlor	residential_neighborhood	windmill
cockpit	iceberg	restaurant	yard
coffee_shop	igloo	restaurant_kitchen
conference_center	islet	restaurant_patio
conference_room	ice_skating_rink/outdoor	rice_paddy

Table 2. Scene attributes from the Places205 project (Zhou et al. 2014).

sailing/boating	spectating	tiles	glossy
driving	farming	concrete	matte
biking	constructing	metal	sterile
transporting	shopping	paper	moist
sunbathing	medical	wood	dry
touring	working	vinyl	dirty
hiking	using tools	plastic	rusty
climbing	digging	cloth	warm
camping	business	sand	cold
reading	praying	rocky	natural
studying	fencing	dirt soil	man-made
training	railing	marble	open area
research	wire	glass	semi-enclosed area
diving	railroad	waves	enclosed area
swimming	trees	ocean	far-away horizon
bathing	grass	running water	nohorizon
eating	vegetation	still water	rugged
cleaning	shrubbery	ice	vertical components
socializing	foliage	snow	horizontal components
congregating	leaves	clouds	symmetrical
waiting	flowers	smoke	cluttered
competing	asphalt	fire	scary
sports	pavement	natural light	soothing
exercise	shingles	sunny	stressful
playing	carpet	electric lighting
gaming	brick	aged

Table 3. Rules used to assist in the classification of the photographs as useful, from Antoniou et al. (2016).

No.	Description
1	Land cover is only considered when it is within about 10 m of the photographer, to take into account positioning errors of the photograph. Thus, land cover types in the far distance should not be considered
2	If it is possible to see or infer with reasonable certainty what is at the photographer’s footprint (even when the footprint is not visible), and there is only one possible class from the nine classes considered, choose “yes”
3	If more than one of the classes above can be assigned to the photographer’s footprint vicinity (using the 10 m limit defined in rule 1), choose “maybe”
4	If there is no information about what may be at the photographer’s footprint, e.g. an aerial or panoramic view, then choose ‘no’
5	Individual trees are discounted regarding the dominant land cover (e.g. a tree in a grass field) unless one can infer from the photograph that there are many trees around
6	For vintage photographs, the answer is ‘no’, since the land cover may have changed (or the photograph may be incorrectly geo-tagged)
7	For snow that completely covers the surface (so it is unclear what the underlying land cover is), because the study area is in London, the answer should be ‘no’. Here context is used, not only the photograph, because in the city of London it is known that no permanent snow cover exists
8	For photographs taken underground, i.e. in a metro station, the answer is ‘no’. If the station is clearly above ground and there is no other land cover type within 10 m, then the answer is ‘yes’ (artificial surfaces)
9	Water frequently causes difficulties because in many cases it is not possible to unequivocally determine if the photograph was taken from a boat (then the answer should be ‘yes’), on a bridge, or at the water vicinity. Then, if the water is identified to be within 10 m of the photographer, the answer is ‘maybe’

Table 4. UA classes of levels 2 and 4 (European Union, 2011, https://cws-download.eea.europa.eu/local/ua2006/Urban_Atlas_2006_mapping_guide_v2_final.pdf).

Level 2		Level 4
Code	Class name	Code	Class name
11	Urban fabric	1110	Continuous urban fabric
		1120	Discontinuous urban fabric
		1121	Discontinuous dense urban fabric
		1122	Discontinuous medium density urban fabric
		1123	Discontinuous low-density urban fabric
		1130	Isolated structures
12	Industrial, commercial, public, military, private, and transport units	1210	Industrial, commercial, public, military and private units
		1222	Other roads and associated land
		1223	Railways and associated land
13	Mine, dump and construction sites	1340	Land without current use
14	Artificial non-agricultural vegetated areas	1410	Green urban areas
		1420	Sports and leisure facilities
20	Agricultural areas, semi-natural areas and wetlands	2000	Agricultural areas, semi-natural areas and wetlands
30	Forests	3000	Forests
50	Water	5000	Water

Figure 1. UA classes for the Paris study region, with locations of all photographs overlaid. The abbreviation “S.L” in the legend refers to the proportion of sealed surface which helps to define that class.

Figure 2. Example of 20, 50 and 100 m buffers around spots where photos were taken.

Table 5. Frequency of UA level 2 classes in the data set at the specific location of each photo (point) and in the area around it (buffer).

Class	Point (%)	20 m	50 m	100 m
		buffer (%)	buffer (%)	buffer (%)
11	28.3	27.9	28.4	26.9
12	37.6	41.3	38.3	33.3
13	1.0	1.1	1.2	2.2
14	9.1	8.4	9.4	10.9
20	10.2	8.0	9.2	10.9
30	3.1	3.2	3.6	4.2
50	10.6	10.1	9.9	11.6

Table 6. Frequency of UA level 4 classes in the data set at the specific location of each photo (point) and in the area around it (buffer).

Class	Point (%)	20 m	50 m	100 m
		buffer (%)	buffer (%)	buffer (%)
1110	10.8	10.7	10.7	10.9
1121	10.6	10.9	13.6	14.1
1122	4.6	4.6	4.3	5.7
1123	1.7	2.0	2.2	2.0
1130	0.6	0.3	0.3	0.2
1210	20.2	15	14.2	14.7
1221	0	0.2	0.4	0.5
1222	16.4	28.6	26.7	21.8
1223	1.0	2.0	1.6	1.5
1340	1.0	0.7	0.6	1.2
1410	6.9	5.6	5.6	6.4
1420	2.3	1.9	2.6	3.0
2000	10.2	6.5	6.9	7.2
3000	3.1	2.6	2.7	2.8
5000	10.6	8.3	7.4	7.7

Table 7. Confusion matrix, precision, recall and F1-score produced on a stratified test set, with number of photographs equal to 25% of all photographs in the Paris data set (the remainder were used for training the model) for the DT approach.

Actual class	Predicted class						Precision	Recall	F1-score
	b	bi	hf	hl	n	u
b	84	0	6	3	0	4	0.79	0.87	0.82
bi	4	10	0	0	0	0	0.71	0.71	0.71
hf	6	0	34	2	1	1	0.57	0.77	0.65
hl	9	0	7	31	0	2	0.86	0.63	0.73
n	0	0	9	0	4	0	0.67	0.31	0.42
u	4	4	4	0	1	12	0.63	0.48	0.55

Table 8. Average precision, recall, and F1-score for the DT approach.

	Unweighted	Weighted
Precision	0.70	0.73
Recall	0.63	0.72
F1-Score	0.65	0.72

Figure 3. Photographs mislabelled by the DT method for identifying human impact in a landscape.

Table 9. Confusion matrix, and the precision, recall, and F1-score for the “usefulness” prediction with the DT approach.

Actual	Predicted		Precision	Recall	F1-score
	No	Yes
No	118	76
Yes	53	351	0.82	0.87	0.84

Table 10. Confusion matrix, and the precision, recall, and F1-score for the ‘usefulness’ prediction with the DT approach by tuning the weights such that we limit the number of false positives (increasing precision at the expense of recall).

Actual	Predicted		Precision	Recall	F1-score
	No	Yes
No	168	26
Yes	128	276	0.91	0.68	0.78

Table 11. Confusion matrix, precision, recall and F1-score for the “usefulness” prediction with the UW approach.

Actual	Predicted		Precision	Recall	F1-score
	No	Yes
No	411	164
Yes	180	206	0.56	0.53	0.54

Table 12. Confusion matrix for UA level 2 and the class prediction based on the scene attributes.

Actual class	Predicted class							Precision	Recall	F1-score
	11	12	13	14	20	30	50
11	43	51	0	4	4	1	2	0.54	0.41	0.46
12	27	186	0	8	3	0	4	0.69	0.82	0.75
13	0	2	0	1	0	0	0	0	0	0
14	1	15	0	23	3	2	5	0.53	0.47	0.50
20	4	6	0	5	14	2	4	0.54	0.40	0.46
30	1	2	0	1	0	3	1	0.38	0.38	0.38
50	4	8	0	1	2	0	27	0.63	0.64	0.64

Notes:

Here, 11 = Urban Fabric, 12 = Industrial, commercial, public, military, private and transport units, 13 = Mine, dump and construction sites, 14 = Artificial non-agricultural vegetated areas, 20 = Agricultural areas, semi-natural areas and wetlands, 30 = Forests, 50 = Water.

Table 13. Prediction accuracy of land cover with the DT approach on the extraction of UA classes from photographs, considering level 2 classes (7 classes) and level 4 classes (14 classes). “Chance of getting the result at random” is simply the chance of randomly guessing the class, 1/7th and 1/14th respectively.

Predictor	Level 2	Level 4
	Accuracy (%)	Accuracy (%)
Scene categories	63.19	48.09
Scene attributes	62.77	50.43
Penultimate layer of the neural network	64.26	52.13
Chance of getting the result at random	14.29	7.14

Table 14. Average unweighted and weighted average of the precision, recall, and F1-score for the level 2 UA classes presented in Table 11.

	Unweighted	Weighted
Precision	0.47	0.61
Recall	0.44	0.63
F1-Score	0.45	0.62

Figure 4. The example of urban photographs whose geolocation indicates objects which is not the objects of interest.

Table 15. Prediction accuracy of land cover with the DT approach for levels 2 and 4 of UA classes within 20, 50, and 100 m buffers defined around the photographs’ locations.

Predictor	UA level 2			UA level 4
	20 m	50 m	100 m	20 m	50 m	100 m
	buffer (%)	buffer (%)	buffer (%)	buffer (%)	buffer (%)	buffer (%)
Scene categories	86.38	91.28	94.04	61.49	68.09	80.00
Scene attributes	85.53	90.43	94.04	64.89	70.85	80.43
Penultimate layer of the NN	85.96	90.21	92.98	66.38	73.19	82.13
Chance of getting the result at random	25.88	30.69	38.59	16.36	21.62	29.29

Zhou, B., A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. 2014. “Learning Deep Features for Scene Recognition Using Places Database.” Paper presented at Advances in Neural Information Processing Systems 27, Montréal, Canada, December 08−13.

 Google Scholar

Antoniou, V., C. Fonte, L. See, J. Estima, J. Arsanjani, F. Lupia, and S. Fritz. 2016. “Investigating the Feasibility of Geo-tagged Photographs as Sources of Land Cover Input Data.” ISPRS International Journal of Geo-Information 5 (5): 64. doi:10.3390/ijgi5050064.

 Web of Science ®Google Scholar