Semantic-aware event link reasoning over industrial knowledge graph embedding time series data

Figures & data

Figure 1. The unified model and framework of industry semantic knowledge base.

(a) Time series data prediction model for obtaining semantic information; (b) The flow for building knowledge graph related to devices; (c) A dynamic event link reasoning model for mining the implicit information from the industrial temporal knowledge graph.

Figure 2. Integrated segmentation process of time series data of IoT perception.

The segmentation flow of time series data, showing time series data divided into different segments.

Figure 3. The data processing flow of the NLTK and D2R.

The data processing flow for building industrial knowledge graph, showing the upper section is using NLTK to process unstructured data, and the lower section is using D2R to process structured data.

Table 1. Description of semantic relationship.

Type	Name	Definition	Description
Inclusion relationship	Perform a weld	Inclusion relationship between device and workpiece	$◯⟹performaweld◯$
	Make an image detection	Inclusion relationship between DL-based model and workpiece image	$◯⟹\text{make an image detection}◯$
	Detect a feature	Inclusion relationship between workpiece image and detection result	$◯⟹\text{detect a feature}◯$
Hierarchical relationship	Sequence	Subordinate relationship of processing between device and device	$◯⟹Sequence◯$
	Parallel	Parallel relationship between device and device	$◯⟹Parallel◯$

Table

Algorithm 1: the unified semantic link algorithm between time series and knowledge graph
Input:  Array(E) is the set of events and attributes, Array(D) is the set of static device knowledge nodes.
Output:  DE is the set of quadruples with event knowledge, DK is the set of dynamic knowledge nodes.
1:	DK ←ϕ
2:	DE = Array(D)
3:	Ti = [t1, t2, t3, … ,ti, … , tn]// Ti is the time when event i occurred
4:	Rij = [Ri1,Ri2, Ri3, … ,Rij, … , Rin]
5:	foreach Ei in Array(E) // Traverse all events, Ei is the i-th event
6:	add Ei to DE
7:	foreach Aij in Ei // Aij is the j-th feature of the i-th event
8:	if Aij in DE
9:	Rij = [Ri1,Ri2, Ri3, … ,Rij, … , Rin]//Rij represents the relationship between Ei and Aij
10:	new Kt = (Ei,Rij,Aij,Ti) // Kt represents the quadruple knowledge with time series
11:	add Kt to DK
12:	end if
13:	else
14:	add Aij to DE   if Ai does not exist, add it to the knowledge base to complete the knowledge
15:	Rij = [Ri1,Ri2, Ri3, … ,Rij, … , Rin]
16:	new Kt = (Ei,Rij,Aij,Ti)
17:	add Kt to DK
18:	end if
19:	end foreach
20:	end foreach
21:	return DE, DK

Table

Algorithm 2: Reasoning algorithm for implicit information in time series data fusion with knowledge graph
Input: The triples for time series fusion with the knowledge graph Ti = (es,rt,eo)|Ti∈T, The local graph sequence{g1, g2, … , gt−1}, Number of sample N.
Output: Conditional distribution P((es,rt,eo)|g(Et+Δt)|g(E:t)), estimation triples Ti+Δt = (es+Δt,rt+Δt,eo+Δt)
1:	foreach {g1, g2, … , gt−1}←Ti  do
2:	Aggregate information g(Et) by Equation (10), Equation (11)
3:	Aggregated vectors z = {z1, z2, … , zn}
4:	end foreach
5:	t = {t1, t2, … , tn}∈ti
6:	foreach tm < ti+Δt, (m < i) do
7:	Compute the triples under aggregation $\mathcal{G}\left(E_t\right)$ at time tm  P((es,rt,eo)|g(Et+Δt)|g(E:t)) by Equation (12)
8:	Upgrade the triples based on Transformer by Equation (16), Equation (17), Equation (18)
9:	tm = tm+1
10:	return Ti+Δt = (es+Δt,rt+Δt,eo+Δt)
11:	end foreach

Figure 4. Information on industrial data types and corresponding statistics.

(a) A preview example of the welding time series data and its data types; (b) Types of defects for welding images and their statistics; (c) An example of temporal knowledge graph for welding data and the statistics in the graph.

Figure 5. Loss variation curves for the four models.

Four curves plotting the training loss of four models for predicting the quality of shell weld seam. Significantly higher prediction accuracy occurs in the green line.

Table 2. Performance comparison of the models’ predictions.

Model	RMSE	Accuracy
CNN-LSTM model	5.78	90.16%
CNN model	9.18	83.12%
LSTM model	11.22	77.13%
BP model	34.58	60.53%

Figure 6. Knowledge graph network of local temporal events for welding thin-walled shell workpiece.

Industrial welding temporal knowledge graph network (local), with red nodes and yellow edges, showing the degree of association between nodes and edges.

Figure 7. Thin-walled shell machining knowledge graph with fusion its welding temporal features.

Industrial welding temporal knowledge graph network fused static knowledge, with red nodes and edges, showing the association of temporal semantic characteristics and static knowledge between nodes and edges.

Table 3. Comparison of our models with several baseline models for space complexity.

Models	Scoring function	Space complexity
TransE	$\Vert e_s+r_p-e_o\Vert$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
DistMult	$<e_s,r_p,e_o>$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
ComplEx	$\mathrm{R}\mathrm{E}(<e_s,r_p,{\overline{e}}_o>)$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
ConvE	$f\left(\mathrm{v}\mathrm{e}\mathrm{c}\right(f\left(\right[\overline{{\mathbf{e}}_s};\overline{{\mathbf{r}}_r}]\ast \omega )\left)\mathbf{W}\right){\mathbf{e}}_o$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
RotatE	$\Vert e_s\circ r_p-e_o\Vert$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
R-GCN	$\Vert e_s+r_p-e_o\Vert$	$\mathcal{O}\left(\mathcal{B}K{d}^2+\mathcal{B}K\left|\mathcal{R}\right|\right)$
CompGCN	$\Vert e_s+r_p-e_o\Vert$	$\mathcal{O}\left(K{d}^2+\mathcal{B}d+\mathcal{B}\left|\mathcal{R}\right|\right)$
TTransE	$\Vert e_s+r_p+w_t-e_o\Vert$	$\mathcal{O}\left(n_{e}d+n_{r}d+n_{t}d\right)$
TA-TransE	$\Vert e_s+\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}\left(\left[r_p;t_{seq}\right]\right)-e_o\Vert$	$\mathcal{O}\left(n_{e}d+n_{r}d+n_{token}d\right)$
TA-DistMult	$<e_s,\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}\left(\left[r_p;t_{seq}\right]\right),e_o>$	$\mathcal{O}\left(n_{e}d+n_{r}d+n_{token}d\right)$
ATiSE	${\mathcal{D}}_{\mathcal{K}\mathcal{L}}\left({\mathcal{P}}_{e,t},{\mathcal{P}}_{r,t}\right)|$	$\mathcal{O}\left(n_{e}d+n_{r}d\right)$
TKEGN (Ours)	$<e_s,\text{Transformer}\left(\left[r_p;t_{seq}\right]\right),e_o>$	$\mathcal{O}(K{d}^2+\mathcal{B}d+n_{token}d$
		$+\mathcal{B}\left|\mathcal{R}\right|)$

Figure 8. Performance comparisons of the reasoning models on relationship ‘perform a weld’.

A four section bar graph in twelve models plotting the accuracy of reasoning ability of four evaluation metrics, showing the performance of different reasoning models on relationship ‘perform a weld’.

Figure 9. Performance comparisons of the reasoning models on relationship ‘make an image detection’.

A four section bar graph in twelve models plotting the accuracy of reasoning ability of four evaluation metrics, showing the performance of different reasoning models on relationship ‘make an image detection’.

Figure 10. Performance of the comprehensive reasoning ability over time in the welding datasets.

A three line graph plotting the Mean Reciprocal Rank of three models for the comprehensive reasoning ability. Significantly higher reasoning accuracy occurs in the green line.

Table 4. Comparison of prediction accuracy between the temporal models and the static models in welding temporal knowledge graph.

		Welding temporal knowledge graph
	Methods	MRR	H@10	H@3	H@1
Static models	TransE	.283	.353	.279	.194
	DistMult	.337	.438	.312	.238
	ComplEx	.341	.466	.390	.312
	ConvE	.480	.549	.458	.331
	RotatE	.362	.474	.353	.285
	R-GCN	.357	.483	.344	.273
	CompGCN	.498	.592	.476	.415
Temporal models	TTransE	.265	.334	.203	.177
	TA-TransE	.272	.360	.267	.186
	TA-DisMult	.404	.562	.430	.317
	ATiSE	.518	.656	.542	.446
	Proposed Method	.549	.673	.566	.482

Table 5. Examples of specific temporal knowledge link reasoning found by our model.

Relationships (edges)	Temporal knowledge links reasoning
Perform a weld, not specified below: 2021-06-20	Predict to weld width: 2021-06-20 →Make an image detection: 2021-06-22 → Give to the decision: 2021-06-23
Make an image detection, not specified below: 2021-07-04	Found an image defect: 2021-07-04 → Check for weld defects: 2021-07-04 → Inspecting of spinning machining quality: 2021-07-01

Data availability statement

The raw/processed data required to reproduce these findings cannot be shared at this time due to data privacy and security concerns restrictions in the aerospace company.