Context-aware and co-attention network based image captioning model

Figures & data

Figure 1. Intuition of our context-aware co-attention-based image captioning model. It consists of relation-aware attention, region-aware attention, transformer-based co-attention module, and $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$-based language decoders. The relation-aware attention module is used to generate interactive words based on the relationships between objects. The relation-aware attention module is used to focus on related image regions to generate entity words. Transformer based co-attention module is used to capture the intra-modal and inter-modal interaction between image regions and objects.

Table 1. Limitations of previous works.

Method	Limitations
EnsCaption [28]	Fails to identify the difference between visually similar images. Unable to detect some relevant objects in the images.
Up-down [19]	Sometimes focus on irrelevant image regions.
IDGAN [29]	Fails to produce semantically diverse image captions for visually similar images. Unable to detect few objects in the images that have no high-quality retrieved captions.
MTTSNet [30]	Errors due to visual ambiguity. Haziness may come from visually similar but dissimilar objects or geometric vagueness. Errors due to lighting, the model may describe the object with a distinct colour.
TSFNet [31]	Error in undetected objects and mistakes in identifying the objects.
High-Order [32]	Fails to generate the exact ground truth words.
Region-Aware [33]	Unable to discriminate the explicit objects from the noisy background.

Figure 2. An overview of our context-aware co-attention network-based image captioning model which consists of region-aware attention, relation-aware attention, $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ decoder pair, and transformer-based co-attention network modules.

Figure 3. Transformer model-based co-attention network with multi-head attention for relationship features and object features to generate context-aware global features.

Table 2. Hyperparameter values.

Hyperparameter used	Value
No. of detected object regions $\left(N\right)$	100
Region feature dimension	2048
No. of hidden units of GRU $\left(M\right)$	2048
Input embedding dimension $\left(d\right)$	2048
Relation-aware attention dimension $\left(d\right)$	256
Head number for relation-aware attention	2
Region-aware attention dimension	1024
Head number for region-aware attention	8
Optimizer	Adam
Learning Rate	0.0003
Attentions Heads	12
Decay Rate	0.8
Dropout (GELU)	0.4
Dropout (relation-aware attention)	0.5
Max. decoding steps	12
Batch Size	64
No. of iterations	32

Table 3. Ablation tests conducted on MSCOCO Karparthy test split.

Variant	B@4	M	R	S	C
Reg-aware Att w/o Att loss	39.2	28.3	47.9	21.6	120.6
Reg-aware Att w Att loss	40.9	29.1	48.9	22.4	124.2
Reg-aware Att + Rel-aware Att w loss	42.2	29.5	49.3	22.9	124.9
Reg-aware Att + Rel-aware Att w loss + Self-Att	43.1	30.1	50.6	23.2	125.6
Reg-aware Att + Rel-aware Att w loss + Self-Att + Hadamard Product	43.6	30.6	61.1	23.7	126.1

Table 4. Results of our context-aware co-attention based image captioning model and compared models on MSCOCO Karpathy test split with cross-entropy loss.

Reference method	B@1	B@4	M	R	S	C
BUTD [19]	77.2	36.2	27.0	56.4	20.5	112.5
SCST [14]	–	30.0	25.9	53.4	–	99.4
AoANet [24]	77.4	37.2	28.4	57.5	21.3	119.8
GCN-LSTM [18]	77.3	36.8	27.9	57.0	20.9	116.3
SGAE [15]	77.6	36.9	27.7	57.2	20.9	116.7
X-LAN [48]	78.0	38.2	28.8	58.0	21.9	122.0
CtxAdpAtt [49]	77.8	38.1	28.9	58.6	22.5	120.9
High-Order [32]	79.8	40.0	28.8	59.0	22.0	122.7
Ctx-Aw-Att [26]	75.9	35.8	27.8	56.4	–	111.3
Mul_Att [11]	79.9	42.4	29.4	59.7	23.2	125.7
GNN_VR [13]	80.9	43.4	30.3	60.7	–	124.6
Ours (Reg_Att + Rel_Att)	81.2	43.1	29.9	60.9	23.3	125.1
Ours (Reg + Rel + Co-attention)	82.1	43.6	30.6	61.1	23.7	126.1

Table 5. Results of our context-aware co-attention based image captioning model and compared models on MSCOCO Karpathy test split with CIDEr score optimization.

Reference method	B@1	B@4	M	R	S	C
BUTD [19]	79.2	36.3	27.7	56.9	21.4	120.1
SCST [14]	–	34.2	26.7	55.7	–	114.0
AoANet [24]	80.2	38.9	29.2	58.8	22.4	129.8
GCN-LSTM [18]	80.5	38.2	28.5	58.3	22.0	127.6
SGAE [15]	80.8	38.4	28.4	58.6	22.1	127.8
X-LAN [48]	80.8	39.5	29.5	59.2	23.4	132.0
CtxAdpAtt [49]	81.0	39.5	29.7	59.7	24.0	131.1
High-Order [32]	81.1	40.7	29.4	59.4	23.0	130.4
IDGAN [29]	81.3	38.5	28.5	58.8	–	123.5
SRPL [50]	77.8	37.7	28.1	57.4	21.1	117.6
EnsCaption [28]	81.7	39.2	29.4	59.0	–	125.5
M2-Transformer [23]	80.8	39.1	29.2	58.6	22.6	131.2
Ours (Reg_Att + Rel_Att)	81.5	43.5	30.2	61.4	23.8	126.4
Ours (Reg + Rel + Co-attention)	82.7	44.1	30.8	61.9	24.1	132.1

Note: Bold values represent the highest value of the previous models with which we have compared the proposed model.

Table 6. Results of our context-aware co-attention based image captioning model and compared models on the Flickr30k dataset.

Reference method	B@1	B@4	M	R	S	C
CapGen-VisAtt [51]	66.9	19.9	18.46	–	–	–
ImCap-SemAtt [52]	64.7	23.0	18.9	–	–	–
SCA-CNN [53]	66.2	22.3	195	44.9	–	44.7
AdapAtt [54]	67.7	25.1	20.4	46.7	14.5	53.1
ALT [55]	68.5	27.0	21.2	48.0	15.5	56.2
BUTD [19]	72.4	28.3	22.0	49.3	16.3	60.4
AoANet [24]	72.8	29.7	22.3	50.1	16.6	64.3
High-Order [32]	73.2	30.1	22.9	50.1	16.8	65.0
EnsCaption [28]	76.2	32.9	24.6	54.2	–	69.3
Ctx-Aw-Att [26]	69.8	27.7	21.5	48.5	–	57.4
IDGAN [29]	72.9	29.3	21.8	49.9	–	65.0
SRPL [50]	75.6	32.0	23.5	52.8	–	67.5
Mul_Att [11]	74.3	31.2	23.5	51.5	17.2	65.6
GNN_VR [13]	75.2	32.2	24.5	52.5	–	64.6
Ours (Reg_Att + Rel_Att)	75.6	32.7	24.4	53.1	16.9	65.1
Ours (Reg + Rel + Co-attention)	76.9	33.6	25.1	55.1	17.8	69.8

Note: Bold values represent the highest value of the previous models with which we have compared the proposed model.

Table 7. Comparison with the existing models in terms of the number of parameters in million (M), training time on GPUs, Flops, layers, width and MLP on MSCOCO.

Model	No. of parameters (M)	Training time	Flops	Layers	Width	MLP
BUTD [19]	52	960 h (M40)	4.8	24	384	1536
MA_RN [11]	115	48 h (V100)	9.5	12	768	3072
Ext_Know [12]	135	74 h (V100)	12.6	24	768	3072
GNN_VR [13]	43	43 h (GTX1080)	4.8	24	384	1536
Ours	41	39 h (GTX1080)	3.8	24	768	1072

Figure 4. Qualitative results of our model and other models on MSCOCO. Ours indicates the captions generated by our model. EnsCaption [28] and M2-Transformer [23] are the strong comparative models. For each image, we have shown one interaction and two entity words. Highest attention weights are shown in red colour.

Figure 5. Qualitative results of our model and other models on Flickr30k. Ours indicates the captions generated by our model. EnsCaption [28] and Mul_Att [11] are the strong comparative models. For each image, we have shown one interaction and two entity words. Highest attention weights are shown in red colour.

Figure 6. Visualization results of attended regions/objects of our context-aware co-attention module during the decoding phase. Higher attention weights are shown in the form of brighter regions. (Best viewed in colour and $200\text{\%}$).

Figure 7. Same caption is generated by the proposed model for the left and middle images, as these two images are visually similar. In the rightmost image, our model fails to identify “onion” and “orange” and thus generates the wrong caption.

Yang M, Liu J, Shen Y, et al. An ensemble of generation-and retrieval-based image captioning with dual generator generative adversarial network. IEEE Trans Image Process. 2020;29:9627–9640.

 Web of Science ®Google Scholar

Anderson P, He X, Buehler C, et al. Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018. p. 6077–6086.

 Google Scholar

Liu J, et al. Interactive dual generative adversarial networks for image captioning. In: Proc. AAAI; 2020. p. 11588–11595.

 Google Scholar

Kim DJ, Oh TH, Choi J, et al. Dense relational image captioning via multi-task triple-stream networks. IEEE Trans Pattern Anal Mach Intell. 2021;44(11):7348–7362.

 Web of Science ®Google Scholar

Hu N, Ming Y, Fan C, et al. TSFNet: triple-steam image captioning. IEEE Transactions on Multimedia. 2022:1–14.

 Google Scholar

Wang Y, Xu N, Liu AA, et al. High-order interaction learning for image captioning. IEEE Trans Circuits Syst Video Technol. 2021;32(7):4417–4430.

 Web of Science ®Google Scholar

Liu AA, Zhai Y, Xu N, et al. Region-aware image captioning via interaction learning. IEEE Trans Circuits Syst Video Technol. 2021;32(6):3685–3696.

 Web of Science ®Google Scholar

Rennie SJ, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2017. p. 7008–7024.

 Google Scholar

Huang L, Wang W, Chen J, et al. Attention on attention for image captioning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2019. p. 4634–4643.

 Google Scholar

Yao T, Pan Y, Li Y, et al. Exploring visual relationship for image captioning. In: Proceedings of the European conference on computer vision (ECCV); 2018. p. 684–699.

 Google Scholar

Yang X, Tang K, Zhang H, et al. (2019). Auto-encoding scene graphs for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. p. 10685–10694.

 Google Scholar

Pan Y, Yao T, Li Y, et al. X-linear attention networks for image captioning. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit.; 2020. p. 10971–10980.

 Google Scholar

Zhang Z, Wu Q, Wang Y, et al. Exploring pairwise relationships adaptively from linguistic context in image captioning. IEEE Trans Multimed. 2021;24:3101–3113.

 Web of Science ®Google Scholar

Wang J, Wang W, Wang L, et al. Learning visual relationship and context-aware attention for image captioning. Pattern Recognit. 2020;98:107075.

 Web of Science ®Google Scholar

Sharma H, Srivastava S. Multilevel attention and relation network-based image captioning model. Multimed Tools Appl. 2022:1–23.

 PubMed Web of Science ®Google Scholar

Sharma H, Srivastava S. Graph neural network-based visual relationship and multilevel attention for image captioning. J Electron Imaging. 2022;31(5):053022.

 Web of Science ®Google Scholar

Liu X, Li H, Shao J, et al. Show, tell and discriminate: image captioning by self-retrieval with partially labeled data In: Proc. Eur. Conf. Comput. Vis. (ECCV); 2018. p. 338–354.

 Google Scholar

Cornia M, Stefanini M, Baraldi L, et al. Meshed-memory transformer for image captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020. p. 10578–10587.

 Google Scholar

Xu K, et al. Show, attend and tell: neural image caption generation with visual attention. Comput Sci. Feb. 2015;2015:2048–2057.

 Google Scholar

You Q, Jin H, Wang Z, et al. Image captioning with semantic attention. In: Proc. CVPR; 2016 Jun. p. 4651–4659.

 Google Scholar

Chen L, et al. SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning. In: Proc. CVPR; 2017 Jul. p. 6298–6306.

 Google Scholar

Lu J, Xiong C, Parikh D, et al. Knowing when to look: adaptive attention via a visual sentinel for image captioning. In: Proc. CVPR; 2017 Jul. p. 3242–3250.

 Google Scholar

Ye S, Han J, Liu N. Attentive linear transformation for image captioning. IEEE Trans Image Process. 2018;27(11):5514–5524.

 Web of Science ®Google Scholar

Sharma H, Jalal AS. Incorporating external knowledge for image captioning using CNN and LSTM. Mod Phys Lett B. 2020;34(28):2050315.

 Web of Science ®Google Scholar