Exploring latent weight factors and global information for food-oriented cross-modal retrieval

Figures & data

Figure 1. The overall framework of the proposed method.

Table 1. The symbols and corresponding meanings in the current paper

Symbols	Meanings	Symbols	Meanings
$R$	the cooking recipe collection	${\mathit{x}}_1$,${\mathit{x}}_2$,${\mathit{x}}_3$	the vector space of title, ingredient, cooking instruction components
$I$	the food image collection	$\mathit{X}$	the constructed input tensor
$N$	the number of image-recipe pairs	$d_1$, $d_2$, $d_3$	dimensionality size of ${\mathit{x}}_1$, ${\mathit{x}}_2$, ${\mathit{x}}_3$
$x_i^R$	a recipe	$\mathcal{W}$	the weight tensor of a linear layer
$x_i^I$	a food image	$\mathit{b}$	the bias of a linear layer
$t^{ttl}$	recipe title component	$d_r$	the dimension size of the final recipe embeddings
$g^{ing}$	recipe ingredient component	$k$	the index of $d_r$
$s^{ins}$	cooking instruction component	${\mathit{W}}_k$	Order-3 weight tensor
${\mathit{e}}_t^{ttl}$	the textual embedding of title	$\widetilde{{\mathcal{W}}_k}$	low-rank weight tensor
${\mathit{e}}_g^{ing}$	the textual embedding of ingredient	${{\mathit{w}}_k}_{(j)}^{ttl}$	the weight vector of the title component
${\mathit{e}}_s^{ins}$	the textual embedding of cooking instruction	${{\mathit{w}}_k}_{(j)}^{ing}$	the weight vector of the ingredient component
${\mathit{E}}^R$	representation of a recipe	${{\mathit{w}}_k}_{(j)}^{ins}$	the weight vector of the ingredient component
${\mathit{E}}^I$	representation of a food image	$\mathrm{\Re }$	the minimum rank value
${\psi }_{img}$	the image encoder	$r$	the rank value of the weight tensor
${\mathit{e}}_{res}^I$	feature embedding of image $x_i^I$	${\mathit{w}}_{(j)}^{ttl}$	the low-rank factors for the title component
${\mathit{W}}_{I_q},{\mathit{W}}_{I_k},{\mathit{W}}_{I_v},{\mathit{W}}^{\mathbf{\prime }}$	trainable weight matrices	${\mathit{w}}_{(j)}^{ing}$	the low-rank factors for the ingredient component
$d_I$	the dimension size of a self-attention layer	${\mathit{w}}_{(j)}^{ins}$	the low-rank factors for the instruction component
$wd$	a word token	$\alpha$	the hyper-parameters for image-recipe pair loss
$t$	the number of word tokens in a sentence	$\beta$	the hyper-parameters for recipe component loss
$Sq$	a sentence	$m$	margin value
$\mathit{S}{\mathit{q}}^{ttl}$	the sentence embedding of the title	${\mathit{E}}_p^R$, ${\mathit{E}}_p^I$	positive samples of recipes and images
${\mathit{S}\mathit{q}}_j^{ing}$	the sentence embedding of the $j$th sentence in the ingredient component	${\mathit{E}}_n^R$, ${\mathit{E}}_n^I$	negative samples of recipes and images
${\mathit{S}\mathit{q}}_j^{ins}$	the sentence embedding of the $j$th sentence in the cooking instruction component	${\mathit{E}}_n^I$	negative samples of images
$M$	the number of sentences of ingredient component	$a,b$	the representations of any two recipe components
$P$	the number of sentences of cooking instruction component	${\mathit{a}}_p$, ${\mathit{b}}_p$	positive samples of $a$ and $b$
${\mathit{W}}_{tt{t}_q},{\mathit{W}}_{tt{t}_k},{\mathit{W}}_{tt{l}_v},{\mathit{W}}_c^{\mathrm{\prime }}$	the learned weight matrices	${\mathit{a}}_n$, ${\mathit{b}}_n$	negative samples of $a$ and $b$

Table 2. The comparison results of our proposed method and popular baseline methods. In this table, bold values represent the best scores.

			Image-to-Recipe				Recipe-to-Image
Size		Method	MedR↓	R1↑	R5↑	R10↑	MedR↓	R1↑	R5↑	R10↑
1K	consistent representation learning-based method	JE	5.2	24.0	51.0	65.0	5.1	25.0	52.0	65.0
		ATTEN	4.6	25.6	53.7	66.9	4.6	25.7	53.9	67.1
		AdaMine	2.0	39.8	69.0	77.4	1.0	40.2	68.1	78.7
		IMHF	1.0	53.2	80.7	87.6	1.0	54.1	82.4	88.2
		MSJE	1.0	56.8	84.7	90.9	1.0	56.2	84.9	91.1
		HTL	1.0	58.6	87.1	92.5	1.0	59.2	86.8	92.5
	modality alignment-based method	R2GAN	2.0	39.1	71.0	81.7	2.0	40.6	72.6	83.3
		MCEN	2.0	48.2	75.8	83.6	1.9	48.4	76.1	83.7
		ACME	1.0	51.8	80.2	87.5	1.0	52.8	80.2	87.6
		SCAN	1.0	54.0	81.7	88.8	1.0	54.9	81.9	89.0
		JEMA	1.0	58.1	85.8	92.2	1.0	58.5	86.2	92.3
		LCWF-GI	1.0	59.4	86.8	92.5	1.0	60.1	86.7	92.7
10K	consistent representation learning-based method	JE	41.9	-	-	-	39.2	-	-	-
		ATTEN	39.8	7.2	19.2	27.6	38.1	7.0	19.4	27.8
		AdaMine	13.2	14.9	35.3	45.2	12.2	14.8	34.6	46.1
		IMHF	6.2	23.4	48.2	58.4	5.8	24..9	48.3	59.4
		MSJE	5.0	25.6	52.1	63.8	5.0	26.2	52.5	64.1
		HTL	4.0	27.3	55.2	67.0	4.0	27.9	55.3	66.9
	modality alignment-based method	R2GAN	13.9	13.5	33.5	44.9	12.6	14.2	35.0	46.8
		MCEN	7.2	20.3	43.3	54.4	6.6	21.4	44.3	55.2
		ACME	6.7	22.9	46.8	57.9	6.0	24.4	47.9	59.0
		SCAN	5.9	23.7	49.3	60.6	5.1	25.3	50.6	61.6
		JEMA	4.2	26.9	54.0	65.6	4.0	27.2	54.4	66.1
		LCWF-GI	4.0	27.9	56.0	67.8	4.0	28.6	55.8	67.5

Table 3. The food-oriented cross-modal retrieval results of this proposed method fusing different recipe components. In this table, bold values represent the best scores.

		Image-to-Recipe				Recipe-to-Image
		MedR↓	R1↑	R5↑	R10↑	MedR↓	R1↑	R5↑	R10↑
1K	TL + IG	1.0	57.1	85.3	91.6	1.0	56.7	84.7	91.2
	TL + IS	2.0	47.8	79.6	87.9	1.9	48.1	79.2	88.0
	IG + IS	1.0	57.5	85.5	91.7	1.0	57.1	84.9	91.6
	TL + IG + IS	1.0	59.4	86.8	92.5	1.0	60.1	86.7	92.7
10K	TL + IG	5.0	25.5	52.4	64.3	5.0	25.5	52.4	64.2
	TL + IS	8.2	18.1	42.1	54.4	8.3	18.7	42.1	54.2
	IG + IS	5.0	25.6	52.8	64.8	5.0	25.9	52.7	64.5
	TL + IG + IS	4.0	27.9	56.0	67.8	4.0	28.6	55.8	67.5

Figure 2. The retrieval results of latent weight factor with different rank values in our proposed LCWF-GI method on Image-to-Recipe and Recipe-to-Image tasks on 1 and 10 K test set. (a) The Value of Recall@1 of Our LCWF-GI method for Image-to-Recipe and Recipe-to-Image Retrieval Tasks on 1 K Test Set; (b) The Value of Recall@1 of Our LCWF-GI method for Image-to-Recipe and Recipe-to-Image Retrieval Tasks on 10 K Test Set.

Figure 3. The example of Recipe-to-Image retrieval of our proposed method and HTL baseline method on 10 K test set. The first column represents that the query is a piece of recipe text. The second column is the ground truth of the recipe text. The final column is the top 5 retrieval result.

Figure 4. The results achieved by our LCWF-GI method and the best baseline method (HTL) on two retrieval tasks on 1 K test set when using the different training epochs. The solid curve represents our LCWF-GI method and the dashed curve means the HTL method. The solid and dashed curves with orange, green and blue colours represent the metrics of Recall@1, Recall@5 and Recall@10 respectively.

Table 4. The parameter counting and the time of each epoch of our LCWF-GI method and the best baseline method (HTL). M and s represent million and second.

Method	Parameter Count (Million)	Time (second)
HTL	93.95M	7,106s
LCWF-GI	100.25M	6,977s

Figure 5. The visualisation of attention mechanism in transformer for the cooking instruction component. (a) The self-attention visualisation of a sentence in the cooking instruction component; (b) The self-attention visualisation of sentence a and the next sentence in the cooking instruction component.