Deep learning models for spatial relation extraction in text

Figures & data

Figure 1. Overall architecture of CasREL.

Figure 2. Overall architecture of PURE.

Figure 3. Dataset is divided, and the original data set is manually labeled and divided into fine-tune and eval datasets, where the fine-tune data set is divided into training sets, validation sets, and test sets according to the Bootstrapping method.

Table 1. Location contains a comparison table of geographic entity types and samples.

Type	Sample
Natural landscape	Anning hot spring is located in Anning County, about 40 kilometers west of Kunming City. [安宁温泉位于昆明市西约40公里处的安宁县境内。]
Monuments	Zhangwu village in Anji County has the former residence of Wu Changshuo, a modern painting artist. [安吉县章吴村有近代书画艺术家吴昌硕故居。]
Administrative regions, countries and continents	Acheng City is located to the east of Harbin City, which is a famous ancient city in Heilongjiang Province. [阿城市位于哈尔滨市以东, 阿城市是黑龙江省著名古城。]
Latitude and longitude coordinates	Altun Mountain Nature Reserve located between 36°-37°38’ N and 87°10’ − 91°18’ E. [阿尔金山自然保护区介于北纬36°～37°38’, 东经87°10’～91°18’。]
Transport facilities	The urban area of Acheng City is along the Ashe River and the Binsui Railway. [阿城市市区在阿什河畔、滨绥铁路沿线。]
Artificial electrical, water, communications and mineral facilities	Taibai Gold Mine in Taibai County of Baoji City is the largest gold enterprise in northwest China. [宝鸡市太白县太白金矿是中国西北地区最大黄金企业。]
Points of interest	Baoding has universities such as Hebei University and Hebei Agricultural University. [保定市现有河北大学、河北农业大学等高等院校。]

Table 2. Spatial entity contains a comparison table of geographic entities with samples.

Type	Sample
Natural landscape without geographical names	Many lakes surround Dongting Lake. [许多湖泊围绕在洞庭湖四周。]
Buildings without geographical names	A new office building has been built on the left side of Xuanwu Lake Park. [玄武湖公园左侧新建了一座办公楼。]
Transport facilities without geographical names	Nanjing South Station is one of the hubs of the East China Railway, where many railways meet. [南京南站是华东铁路枢纽之一, 许多铁路在此交汇。]
Geographical phenomena	Typhoon Likima made landfall in Taizhou City, Zhejiang Province. [台风利奇马在浙江台州市登陆。]
Geographical area	Baoji City is located in the western Guanzhong Plain, north of the Loess Plateau. [宝鸡市位于关中平原西端, 北倚黄土高原。]

Table 3. Example of annotated records and their corresponding English translations.

样例1: {  “_id”:“J20211214202647”,  “text”:“北 镇 满 族 自 1治县 京 沈 铁 路 穿 过 县 境 。”,  “entities”: [   {“entityId”:“E20211214202629”,   “content”:“北镇满族自治县”,   “indices”: [0,6],“label”:“ADM”},   {“entityId”:“E20211214202634”,   “content”:“京沈铁路”,   “indices”: [7,10],“label”:“TRANS”}  ],  “relations”: [   {“relationId”:“R20211214202644”,   “subjectId”:“E20211214202634”,“subject”:“京沈铁路”,   “objectId”:“E20211214202629”,“object”:“北镇满族自治县”,   “relationType”:“traverse”}],  ] }

Sample 1: {  “_id”:“J20211214202647”,  “text”: “Beijing-Shenyang Railway passes through the Beizhen Manchu Autonomous County.”,  “entities”: [   {“entityId”:“E20211214202629”,   “content”:“Beizhen Manchu Autonomous County”,   “indices”: [0,6],“label”:“ADM”},   {“entityId”:“E20211214202634”,   “content”:“Beijing-Shenyang Railway”,   “indices”: [7,10],“label”:“TRANS”}  ],  “relations”: [   {“relationId”:“R20211214202644”,   “subjectId”:“E20211214202634”,“subject”:“Beijing-Shenyang Railway”,   “objectId”:“E20211214202629”,“object”:“Beizhen Manchu Autonomous County”,   “relationType”:“traverse”}],  ] }

Figure 4. Training losses (a) and F1 scores (b) at different max sequence length of CaREL on validation dataset.

Figure 5. Training losses (a) and F1 scores (b) at different learning rate of CaREL on validation dataset.

Figure 6. Training losses (a) and F1 scores (b) at different batch size of CaREL on validation dataset.

Table 4. Best predictive results of the CasREL and time cost under different hyperparameter settings.

No.	Hyperparameters			Best predictive results			Time cost
	Batch size	Learning rate	Max Sequence length	F1	P	R	Training time （h:min:second）
1	8	0.00001	50	0.431	0.581	0.342	00:36:41
2	8	0.00001	100	0.391	0.522	0.313	00:38:06
3	8	0.00001	200	0.478	0.664	0. 314	00:39:55
4	8	0.00002	200	0.426	0.575	0.338	00:39:07
5	8	0.000001	200	0.096	0.622	0.052	00:41:28
6	4	0.00001	200	0.477	0.582	0.404	00:40:43
7	16	0.00001	200	0.407	0.605	0.307	00:37:26

Table 5. Best predictive results of the PURE entity model and time cost under different hyperparameter settings.

No.	Hyperparameters			Best predictive results			Time cost
	Batch size	Learning rate	Context window	F1	P	R	Training time (h:min:second)
1	16	0.00001	0	0.863	0.869	0.857	00:27:49
2	16	0.00001	100	0.847	0.881	0.815	00:27:10
3	16	0.00001	300	0.864	0.867	0.862	00:28:05
4	16	0.0001	300	0.844	0.861	0.827	00:28:23
5	16	0.000001	300	0.855	0.860	0.850	00:28:56
6	8	0.00001	300	0.878	0.883	0.873	00:34:22
7	32	0.00001	300	0.861	0.880	0.844	00:28:37

Figure 7. Training losses (a) and F1 scores (b)at different max sequence length of PURE on validation dataset.

Figure 8. Training losses (a) and F1 scores (b) at different batch size of PURE on dev dataset.

Figure 9. Training losses (a) and F1 scores (b) at different learning rate of PURE on validation dataset.

Table 6. Best predictive results of the PURE relation model and time cost under different hyperparameter settings.

No.	Hyperparameters			Best predictive results			Time cost
	Batch size	Learning rate	Max Sequence length	F1	P	R	Training time （h:min:second）
1	32	0.00002	128	0.674	0.703	0.648	1:08:27
2	32	0.00002	128	0.671	0.701	0.644	1:43:39
3	64	0.00002	128	0.667	0.703	0.634	0:56:57
4	16	0.0002	128	0.679	0.706	0.654	0:58:48
5	16	0.00001	128	0.665	0.677	0.654	0:59:02
6	16	0.00004	128	0.684	0.695	0.673	0:59:02

Table 7. Comparative evaluation of jointly extraction and pipeline extraction methods.

No.	Method	Best predictive results
		F1	P	R
1	CasRel(jointly)	0.515	0.621	0.440
2	PURE(pipeline)	0.621	0.645	0.598

Table 8. Comparison of extraction results of spatial relationship by manual interpretation.

No.	Method	Best predictive results
		F1	P	R
1	CasRel(jointly)	0.672	0.810	0.574
2	PURE(pipeline)	0.767	0.798	0.739

Table 9. Comparative evaluation of spatial relations between joint and pipeline extraction by manual interpretation.

Method	Topological			Directional			Distance
	F1	P	R	F1	P	R	F1	P	R
CasRel	0.713	0.833	0.623	0.245	0.429	0.171	0.000	0.000	0.000
PURE	0.775	0.803	0.749	0.697	0.742	0.657	0.000	0.000	0.000
Rule-based	0.594	0.538	0.665	0.753	0.723	0.786	0.736	0.701	0.775

Table 10. The wrong extraction results of PURE.

Input text data	Wrong extraction results of PURE	Missed results of PURE
Qianshan county is located in southwest part of Anhui Province. 潜山县位于安徽省西南部。	[“Qianshan County”, “location”, “Anhui Province”, “location”, “south east”] [“潜山县”, “location”, “安徽省”, “location”, “south east”]	[“Anhui province”, “location”, “Qianshan county”, “location”, “south west”] [“安徽省”, “location”, “潜山县”, “location”, “south west”]
Qinglong Town is located in northeast part of Qingpu county, which is in west part of Shanghai City. 青龙镇位于上海市西部青浦县东北境。	[“Shanghai City”, “location”, “Qingpu County”, “location”, “north east”], [“Qinglong Town”, “location”, “Shanghai City”, “location”, “west”] [“上海市”, “location”, “青浦县”, “location”, “north east”], [“青龙镇”, “location”, “上海市”, “location”, “west”]	[“Qingpu County”, “location”, “Qinglong Town”, “location”, “north east”], [“Shanghai City”, “location”, “Qingpu County”, “location”, “west”] [“青浦县”, “location”, “青龙镇”, “location”, “north east”], [“上海市”, “location”, “青浦县”, “location”, “west”]
Benxi Water Cave is located in Xiejiaweizi, 20 kilometers east of Benxi City, with magnificent scenery. 本溪市东20公里谢家崴子有本溪水洞, 景色瑰丽。		[“Benxi City”, “location”, “Xiejiaweizi”, “location”, “distance”] [“本溪市”, “location”, “谢家崴子”, “location”, “distance”]
Qingcheng Mountain is backed by Qionglai Mountain and faces the Chengdu Plain. 青城山背靠邛崃山, 面向成都平原。		[“Qingcheng Mountain”, “location”, “Chengdu Plain”, “spatial entity”, “front”], [“Qingcheng Mountain”, “location”, “Qionglai Mountain”, “location”, “behind”] [“青城山”, “location”, “成都平原”, “spatial entity”, “front”], [“青城山”, “location”, “邛崃山”, “location”, “behind”]
Wuxi County is located in the southern part of Jiangsu Province, with Taihu Lake in the southwest. 无锡县位于江苏省南部, 西南滨太湖。		[“Wuxi County”, “location”, “Taihu Lake”, “location”, “south west”] [“无锡县”, “location”, “太湖”, “location”, “south west”]

Table 11. The wrong extraction results of CasREL.

Input text data	Wrong extraction results of CasREL	Missed result of CasREL
Qianshan County is located in the southwest of Anhui Province. 潜山县位于安徽省西南部。	[“Qianshan County”, “location”, “Anhui Province”, “location”, “south west”] [“潜山县”, “location”, “安徽”, “location”, “south west”]	[“Qianshan County”, “location”, “Anhui Province”, “location”, “south west”] [“潜山县”, “location”, “安徽”, “location”, “south west”]
Qinglong Town is located in the northeast of Qingpu County in the west of Shanghai City. 青龙镇位于上海市西部青浦县东北境。	[“Qinglong Town”, “location”, “Shanghai City”, “location”, “north east”] [“青龙镇”, “location”, “上海市”, “location”, “north east”]	[“Qingpu County”, “location”, “Qinglong Town”, “location”, “north east”], [“Shanghai City”, “location”, “Qingpu County”, “location”, “west”] [“青浦县”, “location”, “青龙镇”, “location”, “north east”], [“上海市”, “location”, “青浦县”, “location”, “west”]
Benxi Water Cave is located in Xiejiaweizi, 20 kilometers east of Benxi City, with magnificent scenery. 本溪市东20公里谢家崴子有本溪水洞, 景色瑰丽。	[“Benxi City”, “location”, “Xiejiaweizi”, “location”, “east”] [“本溪市”, “location”, “谢家崴”, “location”, “east”]	[“Benxi City”, “location”, “Xiejiaweizi”, “location”, “east”], [“Benxi City”, “location”, “Xiejiaweizi”, “location”, “distance”] [“本溪市”, “location”, “谢家崴子”, “location”, “east”], [“本溪市”, “location”, “谢家崴子”, “location”, “distance”]
Qingcheng Mountain is backed by Qionglai Mountain and faces the Chengdu Plain. 青城山背靠邛崃山, 面向成都平原。		[“Qingcheng Mountain”, “location”, “Chengdu Plain”, “spatial entity”, “front”], [“Qingcheng Mountain”, “location”, “Qionglai Mountain”, “location”, “behind”] [“青城山”, “location”, “成都平原”, “spatial entity”, “front”], [“青城山”, “location”, “邛崃山”, “location”, “behind”]
Wuxi County is located in the southern part of Jiangsu Province, with Taihu Lake in the southwest. 无锡县位于江苏省南部, 西南滨太湖。	[“Taihu Lake”, “location”, “Wuxi County”, “location”, “south west”] [“太湖”, “location”, “无锡县”, “location”, “south west”]

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.