Context-aware and co-attention network based image captioning model

ABSTRACT

Previous captioning methods only rely on semantic-level information considering the similarity of features between image regions in visual space and ignoring the linguistic context incorporated in the decoder for caption generation. In this paper, a transformer-based co-attention network is proposed which uses linguistic information to capture the pairwise visual relationships among objects and significant visual features. We infer the entity words from the visual content of objects, during the caption generation process. Also, we infer the interactive words by focusing on the relationship between entity words, based on the relational context between words generated in the course of caption decoding. We use linguistic contextual information as a guiding force to discover the relationships between objects efficiently. Further, we capture both intra-modal and inter-modal interactions using the multilevel co-attention network. Our model attains 44.1/33.6 BLEU@4, 30.8/25.1 METEOR, 61.9/55.1 ROUGE, 132.1/69.8 CIDEr, and 24.1/17.8 SPICE scores on MSCOCO and Flickr30k datasets, respectively.

KEYWORDS:

• Image captioning
• co-attention network
• transformer
• context-aware
• visual relationships
• computer vision
• natural language processing
• semantic

Introduction

To describe everything that we see in life, we use natural language. To make the computer enable doing the same task is termed image captioning. But generating descriptions of the scene in a natural language is an extremely challenging job for a machine. Accomplishing this task needs to bridge the domains of image processing, natural language processing, and computer vision. This research area can benefit several applications like video captioning [1,2], visual question answering [3–5], and remote sensing [6,7]. Applications such as human–computer interaction, assisted driving and intelligent navigation for visually impaired people are also benefitted from image descriptions.

Based on the visual understanding of image contents, good quality generated words are desirable for caption generation. Visual understanding involves the relationship between objects (similar and dissimilar) which can be attended to using attention mechanisms. Visual region attention is a common approach to attend features for relating image regions and visual relationship attention attend the pairwise visual relationships of image object that focuses on interactive words. Thus, determining the visual relationships between objects and supplying them to the decoder is of great significance for obtaining good-quality image captions.

The semantic contextual information denotes the association between caption words which guides the process of producing image description. The semantic info contains significant meaningful content that emphasizes the visual relationship between different objects in the image. Better the relationship better the generated words that result in improved quality of captioning. Thus, the task to determine visual relationships is vastly associated with a language decoder for image captioning using the semantic context of the image. The selection of related objects depends on the dynamic semantical context that influences different relationships for producing relevant words. An attention selector attends to related visual relationships as per the dynamic semantics of the image where the relationship attention is used to generate caption words whereas region attention is used to generate image region-based words. It is challenging for the attention selector to switch between different attentions. To handle the issue, in this work we have used a transformer-based co-attention mechanism that takes input from relation-aware attention as well as region-aware attention and aggregates them to find the features in the global context. The global features are then fed into the $GRU$-$GELU$-based decoder to generate the caption words. Thus, we proposed an encoder-decoder-based co-attention mechanism for caption generation. Existing image captioning models mostly used $LSTM$ [8] based decoders. Although used by many models, the computational complexity of $LSTM$ is high due to a large number of network parameters. Instead, we used $GRU$ [9] decoder that gives an improved performance with few network parameters that ultimately reduced the computational complexity. The hidden layers of the $GRU$ decoder contain semantic features of the image to determine related visuals for the caption. The proposed decoder is called the $GRU$-$GELU$ model which contains the contextual information between words that produced improved captions where $GELU$ is used to obtain the features between words that represent the corresponding association. We have proposed $aGRU$-$GELU$ encoder-decoder-based image captioning model which is combined with a context-aware co-attention network module and Bottom-up attention mechanism to extract visual features of image regions. $GRU$-$GELU$ decoder is employed to encode and generate a variable-length image caption. Precisely, we have used $theGRU$ decoder and $GELU$ decoder in our model. First, the $GRU$ decoder is used to generate the semantic context which is computed on the global region feature, input word, and previous output of the $GELU$ decoder. Second, we used the Gaussian Error Linear Unit ($GELU$) [10] decoder that boosts the semantic context with the region-aware/relation-aware context given by the proposed context-aware co-attention network module that focuses on related image objects to generate the caption words. Figure 1 shows the caption generated by the proposed model using the attention mechanism and $GRU$-$GELU$ encoder-decoder.

Figure 1. Intuition of our context-aware co-attention-based image captioning model. It consists of relation-aware attention, region-aware attention, transformer-based co-attention module, and $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$-based language decoders. The relation-aware attention module is used to generate interactive words based on the relationships between objects. The relation-aware attention module is used to focus on related image regions to generate entity words. Transformer based co-attention module is used to capture the intra-modal and inter-modal interaction between image regions and objects.

The significant contributions of this paper are:

• A transformer-based co-attention network for selecting either the region-aware attention or relation-aware attention for generating caption words as a set of entity words or interactive words based on the linguistic information of the caption decoder.

• The proposed model captures the pairwise visual relationship between objects by focusing on intra-model and inter-model interactions between relationship features and region features.

• To reduce the computational complexity, $\mathrm{G}\mathrm{R}\mathrm{U}〈it〉-〈/it〉\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ decoders are employed by the proposed model. To the best of our knowledge, this is the first work that uses these decoders.

• Experiments on $\mathrm{M}\mathrm{S}\mathrm{C}\mathrm{O}\mathrm{C}\mathrm{O}$ and $\mathrm{F}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{r}30\mathrm{k}$ datasets validate that the proposed model attains improved results on almost all evaluation metrics.

Related works

The task of generating an image caption describes the scene content and detects the visual relationships among objects as well as objects and image regions. Fine-grained visual processing is needed to generate high-quality outputs using linguistic contexts. Consequently, visual attention mechanisms have been widely used in image description [11–14]. The existing works explored the visual relationship for image captioning using encoder-decoder systems [15–18], attention-based models [19–24], and relation-aware models [25–27].

Encoder decoder-based models: Visual relationships for image captioning using the encoder-decoder model have been explored by many researchers in the recent past. Yang et al. [15] proposed a model that incorporates an encoder-decoder framework using a scene graph that connects an object node to adjective and relationship nodes for caption generation. Differently, Hoxha et al. [16] used support vector machines (SVMs) based decoder where a network of SVMs was used instead of RNNs to decrypt the image info into an image caption, particularly with the limited amount of training samples. Al-Malla et al. [17] proposed an attention-based architecture using the Encoder-Decoder approach that used features extracted from a CNN model together with extracted object features using a positional encoding scheme. Yao et al. [18] proposed an encoder-decoder architecture that integrates both semantic and spatial relationships among objects with the region features using an attention mechanism for caption generation.

Co-attention-based models: Many researchers proposed attention-based models that explored visual relationships for image captioning. For example; Anderson et al. [19] proposed an attention mechanism that calculates the attention at the object level and image regions where image regions are associated with feature vectors using Faster R-CNN to determine feature weights for image captioning. Wang et al. [20] explored the associations between image attributes using semantic and region properties which adaptively fuse the region features with attained relationships for caption generation. Xiao et al. [21] proposed an LSTM-based attention model that attends relevant features like spatial, visual, and contextual content which are fused to generate improved image descriptions. Wang et al. [22] exploit the attention variation by integrating a parallel network to increase the model reliability and a balance mechanism to balance channel attention and region attention to generate the image description using a regularization penalty. Huang et al. [24] proposed a model that used attention mechanisms to generate an information vector and an attention gate which is further integrated with another attention to obtaining the attended information for caption generation.

Relational reasoning-based methods: Relational reasoning is crucial in visual understanding which is required along with semantic info of each region for improved image descriptions. It is necessary to combine multiple regions to obtain such a relationship. Zhou et al. [25] detect the relationships between image objects to improve the accuracy of image captioning using a relational network based on semantic context. Wang et al. [26] implicitly model the relationship among image regions using a graph neural network that learns relation-aware visual representations considering contextual information for caption generation. Guo et al. [27] introduced a relational network that implements a concepts-to-sentence memory translator through the fusion and recurrent memory mechanism to encode visual context using info of textual corpus.

Motivation: The above-discussed models encode visual relationships that lack guidance from the linguistic contextual content of the decoder as they still rely on the similarity of visual features of the object to define the relationship between objects. In Table 1, we have shown the limitations of the previous methods. In image captioning, similar objects hold a relationship that may not result in interactive words for captioning whereas different objects hold strong visual relationships resulting in interactive words. Consequently, strong guidance from the semantic context in the decoding phase should be used to explore the relationships between different objects. To generate interactive words for similar and dissimilar objects, we have used $aGRU$-$GELU$ language decoder in this work which is combined with the context-aware co-attention network that focuses on related image objects for generating interactive words using region-aware attention, relation-aware attention, and adaptive attention mechanism is used to decide which features to select and provided to language decoder. We have used the caption loss for training our captioning method.

Table 1. Limitations of previous works.

Method	Limitations
EnsCaption [28]	Fails to identify the difference between visually similar images. Unable to detect some relevant objects in the images.
Up-down [19]	Sometimes focus on irrelevant image regions.
IDGAN [29]	Fails to produce semantically diverse image captions for visually similar images. Unable to detect few objects in the images that have no high-quality retrieved captions.
MTTSNet [30]	Errors due to visual ambiguity. Haziness may come from visually similar but dissimilar objects or geometric vagueness. Errors due to lighting, the model may describe the object with a distinct colour.
TSFNet [31]	Error in undetected objects and mistakes in identifying the objects.
High-Order [32]	Fails to generate the exact ground truth words.
Region-Aware [33]	Unable to discriminate the explicit objects from the noisy background.

Proposed method

The proposed context-aware co-attention network-based image captioning model is shown in Figure 2. It is the combination of three major components: Bottom-up attention for extracting visual features, context-aware co-attention network module, and $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ language decoder.

Figure 2. An overview of our context-aware co-attention network-based image captioning model which consists of region-aware attention, relation-aware attention, $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ decoder pair, and transformer-based co-attention network modules.

The bottom-up attention mechanism is applied to extract a feature set with the size of $k\times 2048$, where $k$ represents the number of detected regions and each region is represented as $V_i$ . To be more specific, Faster R-CNN [34] is at the core of the bottom-up attention module to detect regions of interest as $R=\left\{R_1,R_2,\dots \dots ,R_k\right\}$. and $ResNet$ [35] is used to generate feature descriptions corresponding to these regions as $V=\left\{V_1,V_2,\dots \dots ,V_k\right\}$. The $k$-th object region is represented by $D_v$ dimensional visual features such that $V_k\in {\mathbb{R}}^{D_V}$.

Further, we have employed the $GRU$-$GELU$ decoder to encode and produce a variable length caption $S=\left\{s_1,s_2,\dots ..,s_T\right\}$ with $T$ words by using the obtained image region features $V$, where $s_t\in {\mathbb{R}}^{D_y}$ represents $D_y$ dimensional one-hot feature representation for the $t$-th word. We have used $theGRU$ decoder and $GELU$ decoder in our model. First, the $GRU$ decoder is used to generate the semantic context $h_t^1$ which is computed on global region feature $\overline{V}$, input word $x_t$, and previous output $h_{t-1}^2$ of $GELU$ decoder. Second, we used the Gaussian Error Linear Unit $\left(GELU\right)$ decoder that boosts the semantic context with the region-aware/relation-aware contexts $C_t$ given by the proposed context-aware co-attention network module.

Based on the dynamic linguistic/semantic context of the decoder, the proposed context-aware co-attention network mechanism focuses on related image objects or pairwise relationships between objects for generating entities or interactive words. To generate the caption words, the proposed model uses three different modules, such as region-aware attention, relation-aware attention, and adaptive attention. (1) The region-aware attention is used to generate entity words by focusing on related image objects and then combining object-based contextual features for language decoder. (2) The relation-aware attention is used to generate interactive words by focusing on related visual relationships between objects and combining relationship-based contextual features for generating these words. (3) The adaptive attention mechanism is used to decide which features to select and provide to the language decoder. The raw semantic information $h_t^1$ is employed as the semantic information for context-aware co-attention network mechanism to attend to the related visual content. We have used the caption loss for training our captioning method.

In subsequent sections, we will explain the $GRU$-$GELU$ language decoder and context-aware co-attention network module.

$\mathit{G}RU$-$GELU$language encoder

The proposed model uses the $GRU$-$GELU$language decoder to encode previously generated caption words $\left\{s_1,s_2,\dots ..,s_{t-1}\right\}$ to generate the next word $s_t$at current time step $t$. To be precise, at each time step $t$, the $GRU$ decoder gives raw linguistic content by using three inputs: global region feature $\overline{V}$ = ${\displaystyle \frac{1}{K}{\sum }_{k=1}^K{V}_k}$, features corresponding to input word $x_t$, and $h_{t-1}^2$ as the output states of $GELU$ decoder at previous time step $t-1$. We use the previously predicted word $s_{t-1}$ as the current input word $x_t$at time step $t$. The remaining preceding words $\left\{s_1,s_2,\dots ..,s_{t-2}\right\}$ are represented as the output states $h_{t-1}^2$ of $GELU$ encoder till time step $t-2$. Further, we provide the combination of these three features in the $GRU$ decoder as follows: (1) $h_t^1=GRU\left(\left[\overline{V};W_e{x}_t;h_{t-1}^2\right],h_{t-1}^1\right)$(1) Here, $W_e\in {\mathbb{R}}^{D_y\times D_e}$ represents the embedding matrix for a given word $x_t$ which is used to transform a large dimensional feature representation to a low dimensional dense feature representation. At time step $t$, the output $h_t^1\in {\mathbb{R}}^{D_h}$ represents the hidden states comprising raw content that can be effectively utilized to predict word $s_t$. Moreover, the proposed model uses $h_t^1$as the semantic context input for each module of the context-aware co-attention network module, which is used to generate effective interactive/entity words by capturing the related visual relationships/objects.

The proposed context-aware co-attention network module takes $h_t^1$ and $V$ as inputs and outputs a global contextual feature $C_t$ according to the varying semantic context contained in $h_t^1$. Precisely, contextual object features are captured by the region-aware attention for predicting the entity word, while a set of relation-aware contextual features is predicted by the relation-aware attention for producing the interactive word. The adaptive attention module decides which attention mechanism should be changed and merges related contextual features as the global contextual feature $C_t$ for the $GELU$ decoder. (2) $C_t=AdapAtt\left(h_t^1,V\right)$(2) As $h_t^1$ represents raw semantic content, the global contextual feature $C_t$ given by the context-aware co-attention network module functions as the complementary visual content that additionally improves $h_t^1$ with the significant relationship/object contextual feature which is powerfully associated to the semantic context. To attain this, we feed the concatenation of $C_t$ and $h_t^1$ into the$GELU$ decoder for enhancing the multi-modal information: (3) $h_t^2=GELU\left(\left[C_t;h_t^1\right];h_{t-1}^2\right)$(3)

We input the output states $h_t^2$ of $theGELU$ decoder into the word generator for predicting subsequent word $s_t$. Our $GELU$ decoder is an improved language decoder as compared to the $GRU$ decoder, with the knowledge of precise semantic contextual information and extremely associated visual contextual features for predicting interactive/entity words.

Further, our word predictor utilizes the meaningful and significant linguistic information in the hidden states $h_t^2$to predict the conditional probability over likely output words at time step $t$ as shown in Eq. ($4$). Finally, we compute the distribution overall likely output caption sentences using chain rule using Eq. ($5$) (4) $prob\left(y_t/y_{1:t-1}\right)=softmax\left(W_p{h}_t^2+b_p\right)$(4) (5) $prob\left(y_{1:T}\right)=\prod _{t=1}^{T}prob\left(y_t/y_{1:t-1}\right)$(5)

Context-aware co-attention network

In this section, we will describe our proposed context-aware co-attention network module, which operates between $GRU$-$GELU$ decoders. It has three major components: visual relation-aware attention, region-aware attention, and attention modulator. Specifically, the nodes are represented by the detected $K$visual object regions $R=\left\{R_1,R_2,\dots \dots .,R_K\right\}$, and the $K^2$ edges represent the visual relationships between all object regions. Further, we provide the visual features $V=\left\{V_1,V_2,\dots \dots .,V_K\right\}$ as inputs to this module and selectively focus on significantly correlated region-aware (visual object) features or relation-aware (visual relationship) features based on the semantic context of hidden states $h_t^1$.

Relation-aware attention

This attention mechanism aims to capture the visual relationship between the different object regions in a pairwise form, and then obtains visual features for the captured pairwise relationships. We provide the visual features $V=\left\{V_1,V_2,\dots \dots .,V_K\right\}$ corresponding to $K$ object regions as inputs to this module. To capture the visual relationship between object regions, a self-attention mechanism [36] is employed. Hadamard product-based low-rank bilinear pooling [37] is used to capture second-order interactions between visual object features. Using a bilinear self-attention mechanism, the proposed model performs complex reasoning over image region pairs which in turn represent the power of relation-aware features.

Furthermore, the hidden states $h_t^1$ of $theGRU$ decoder are used to capture the visual relationships, which are based on linguistic context. Our relation-aware attention mechanism has two sub-modules: (1) pairwise relation-aware attention maps (2) the generation of a relation-aware feature. Both sub-modules use the Hadamard product-based bilinear pooling approach. The attention map generation task is directed by the semantic context incorporated in the hidden states $h_t^1$.

Pairwise relationship attention map generation

For $K$ object region features represented as $V=\left\{V_1,V_2,\dots \dots .,V_K\right\}$, we have total $K^2$ visual relationship pairs. Further, we provide linguistic context $h_t^1$ together with the visual region features to generate a $K\times K$ relation-aware matrix $A_t\in {\mathbb{R}}^{K\times K}$to represent the attention maps for the captured visual relationships. We represent the object regions ($R_i,R_j$, $i,j\in \left[1,K\right])$ as the query nodes and represent all captured relationships for $R_i$ by a vector ${\overline{\alpha }}_{t,i}$. Further, we represent the softmax-normalized attention weight by ${\alpha }_{t,〈i,j〉}$, which depicts the relationship between the object region pair $〈R_i,R_j〉$. (6) $A_t=\{{\overline{\alpha }}_{t,1};\dots ..{\overline{\alpha }}_{t,i};\dots \dots ..{\overline{\alpha }}_{t,K})$(6) (7) ${\overline{\alpha }}_{t,i}=\left\{{\alpha }_{t,〈i,1〉},\dots ..,{\alpha }_{t,〈i,j〉},\dots \dots .,{\alpha }_{t,〈i,K〉}\right\}$(7) (8) ${\alpha }_{t,〈i,j〉}=softma{x}_j\left(e_{t,〈i,j〉}\right)={\displaystyle \frac{exp\left(e_{t,〈i,j〉}\right)}{{\sum }_{n=1}^{K}exp\left(e_{t,〈i,j〉}\right)}}$(8)

Finally, we compute the normalized relationship value $e_{t,〈i,j〉}$ by applying pairwise low-rank bilinear pooling followed by the injection of linguistic context $h_t^1$. First, we compute the outer product $p_{i,j}$ between feature pairs $〈V_i,V_j〉$ using the Hadamard product, which gives discriminative features to perform reasoning over the complex relationship. reasoning calculated based on the feature pairs $〈V_i,V_j〉$ and linguistic context feature $h_t^1$. Finally, we provide the linguistic context $h_t^1$ into the obtained mean-pooled features $p_{i,j}$ to generate the normalized relationship value $e_{t,〈i,j〉}$ (9) $p_{i,j}=W_P^{\mathrm{\prime }}T\left(\sigma \left(W{}_U^{\mathrm{\prime }}T{V}_i\right)\cdot \sigma \left(W{}_V^{\mathrm{\prime }}T{V}_j\right)\right)$(9) (10) $e_{t,〈i,j〉}=W_A\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(p_{i,j}+W_h^{\mathrm{\prime }}{h}_t^1\right)$(10) where $W_U^{\mathrm{\prime }}T\in {\mathbb{R}}^{D_V\times L^{\mathrm{\prime }}}$, $W_V^{\mathrm{\prime }}T\in {\mathbb{R}}^{D_V\times L^{\mathrm{\prime }}}$, $W_h^{\mathrm{\prime }}\in {\mathbb{R}}^{D_l\times D_h}$ represent the embedding matrices used to map region features to a low-rank dimensional feature space, and $W_P^{\mathrm{\prime }}T\in {\mathbb{R}}^{D_l\times L^{\mathrm{\prime }}}$ represents the pooling matrix, $\sigma$ represents $ReLU$ function, and $\circ$represents Hadamard product (element-wise multiplication). $W_A\in {\mathbb{R}}^{1\times D_l}$ represents the embedding matrix which projects the linguistic context-based feature into the relationship value $e_{t,〈i,j〉}$.

Visual relationship feature generation

In this module, we generate the features for the captured visual relationships which are denoted by the attention map $A_t\in {\mathbb{R}}^{K\times K}$. For given object region $R_i,i\in \left[1,K\right])$, we represent the relationship feature $C_{t,Ri}$, which is encoded using low-rank bilinear pooling for the associated attention map ${\overline{\alpha }}_{t,i}$. (11) $C_R=\left\{C_{t,R1},C_{t,R2},\dots \dots ,C_{t,RK}\right\}$(11) Further, we fuse the relationship features with the attention weights to obtain the context-based relationship features using the low-rank Hadamard product. (12) $C_{t,Ri}=W_P^T\sum _{j=1}^K{\alpha }_{t,〈i,j〉}\left(\sigma \left(W_U^T{V}_i\right)\cdot \sigma \left(W_V^T{V}_j\right)\right)$(12) Where $W_P^T$ represents the pooling matrix and $W_U^T$, $W_V^T$represents the embedding matrices. $\sigma$ represents the $ReLU$ function, and $.$represents the Hadamard product.

Region-aware attention

Our region-aware attention mechanism focuses on the related object regions for generating entity words at time step $t$ and outputs context-based object feature $C_{t,RO}$. Further, the proposed model explores the second-order relations between the context-based query $h_t^1$ and object features $V_k$using low-rank bilinear pooling. Figure 3 shows transformer model-based co-attention network with multi-head attention for relationship features and object features to generate context-aware global features. (13) $o_{t,k}=W_O^{\mathrm{\prime }}\left(\sigma \left(W_V^{\mathrm{\prime }}{V}_k\right).\sigma \left(W_h^{\mathrm{\prime }}{h}_t^1\right)\right)$(13) (14) $z_{t,k}=W_z^{\mathrm{\prime }}\sigma \left(o_{t,k}\right)$(14) (15) ${\text{β}}_{t,k}=softmax\left(z_{t,k}\right)={\displaystyle \frac{exp\left(z_{t,k}\right)}{{\sum }_{n=1}^{K}exp\left(z_{t,n}\right)}}$(15) (16) $C_O=W_O\sum _{k=1}^K{\text{β}}_{t,k}\left(\sigma \left(W_V{V}_k\right)\cdot \sigma \left(W_h{h}_t^1\right)\right)$(16)

Figure 3. Transformer model-based co-attention network with multi-head attention for relationship features and object features to generate context-aware global features.

Thus, we get raw attention value $z_{t,k}$ using activation function $\sigma$ and embedding matrices $W_V^{\mathrm{\prime }}$ and $W_h^{\mathrm{\prime }}$, which is then fed into $softmax$ function.

Attention adapter

Our relation-aware attention module is used to generate interactive words by exploring the visual relationships between object regions that are highly related to the linguistic context. Also, visual region-aware attention is needed to generate entity words. Apart from these two tasks, we need a module that selectively switches between these two attention modules based on the need of the language decoder. Thus, our model uses an adaptive attention module to achieve this task.

The relation-aware attention module obtains a set of relation-based features and the region-aware attention module produces context-based object features. Moreover, we fuse both these features to generate one set of complex visual features: (17) $\overline{V}=\left\{C_{R0},C_{R1},\dots \dots ,C_{RK}\right\}$(17) (18) $C_{R0}=W_{oe}{C}_O$(18) Finally, we fed $\overline{V}$ into the Sigsoftmax-based [38] adaptive-attention module to select the appropriate attention: (19) ${\hat{z}}_{t,k}=W_c\left(W_{cv}{C}_{Rk}+W_{ch}{h}_t^1\right)$(19) (20) $m_{t,k}=Sigsoftmax\left(z_{t,k}\right)={\displaystyle \frac{exp\left({\hat{z}}_{t,k}\right)\sigma \left({\hat{z}}_{t,k}\right)}{{\sum }_{n=0}^{K}exp\left({\hat{z}}_{t,n}\right)\sigma \left({\hat{z}}_{t,n}\right)}}$(20) (21) $C_t=\sum _{k=0}^K{m}_{t,k}{C}_{Rk}$(21)

Where $W_c,$ $W_{cv,}$ and $W_{cv}$ are transforming matrices. To improve the representation ability of Softmax function, Sigsoftmax has been introduced where sigmoid function acts as a gating function to make the function smooth.

Caption loss

To train our proposed model, we used both cross-entropy loss and CIDEr score function. First, we train the proposed captioning model by minimizing the cross-entropy loss of the output caption. In the second step, we consider the reward in terms of the CIDEr score and train the model to minimize the negative expected reward of randomly selected captions as the loss: (22) $L_{Cap}^{XE}\left(\theta \right)=-\sum _{t=1}^T\log \left(p_{\theta }\right({\overline{y}}_t/{\overline{y}}_{1:t-1}\left)\right)$(22) (23) $L_{Cap}^{RL}\left(\theta \right)=-E_{y_{1:T}^s\sim p_{\theta }}\left[\gamma \left(y_{1:T}^s\right)\right]$(23)

Performance analysis

Datasets and evaluation metrics

We have evaluated the proposed image captioning model on $\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{M}\mathrm{S}\mathrm{C}\mathrm{O}\mathrm{C}\mathrm{O}$ dataset [39] and the Flickr30k dataset [40]. $\mathrm{T}\mathrm{h}\mathrm{e}\mathrm{M}\mathrm{S}\mathrm{C}\mathrm{O}\mathrm{C}\mathrm{O}$ dataset contains $82,783$ images for training purposes, $40,504$ images for validation purposes, and $40,775$ images for testing purposes. For each image, it contains $5$ captions. We have used the ‘Karpathy’ data splits [41], for the offline assessment, comprising $113,287$ images for training, and 5000 images for both validation and testing purposes. Further, we set the maximum caption length to $20$. There are $10,331$ words in the word vocabulary, which contains only those words appearing more than $4$ times in the training caption set. The $\mathrm{F}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{r}30\mathrm{k}$ dataset contains $31,000$ images taken from $\mathrm{F}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{r}$. For each image, there are $5$ reference captions captured from human annotators, and we use $29,000$ images to train, 1000 for validation, and $1000$ to test the proposed model.

We have evaluated the proposed model on widely used evaluation metrics: $\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}@\mathrm{N}\left(\mathrm{B}@1,\mathrm{B}@2,\mathrm{B}@3,\mathrm{B}@4\right)$ [42], $\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}$ [43], $\mathrm{R}\mathrm{o}\mathrm{u}\mathrm{g}\mathrm{e}$-$\mathrm{L}$ [44], $\mathrm{S}\mathrm{P}\mathrm{I}\mathrm{C}\mathrm{E}$[45] and $\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}\mathrm{r}$ [46]. Moreover, we have computed these metric scores using the COCO captioning evaluation tool [39]. Out of these metrics, $\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}\mathrm{r}$ and $\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}$ scores convey the highest-level correlations with human-generated captions.

Hyperparameter settings

We set the input word dimension o $1024$. We fixed the hidden states in $\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{G}\mathrm{R}\mathrm{U}$ decoder to $1024$. The relation-aware attention is represented by a $512$-dimensional vector. We fixed the head number to$2$. The region-aware attention is represented by a $2048$-dimensional vector by setting the head number to $12$. The attention selector dimension is fixed to $2048$. We have used the $\mathrm{A}\mathrm{d}\mathrm{a}\mathrm{m}$ optimizer [47] by setting the initial learning rate to $0.0003$. Every epoch, we are decreasing the learning rate by a factor of $0.8$. Further, we used a dropout ratio of $0.4$ on the o/p states of $\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{G}\mathrm{R}\mathrm{U}$ decoder and set the dropout ratio to $0.5$for relation-aware attention. For the beam search decoding process, we used a beam size of $5$. Precisely, we initially optimize the entire architecture of the proposed model by adding the cross-entropy loss, similar to the previous models. Similar to existing works [56, 57], we employ self-critical sequence training strategy for optimizing our proposed model with CIDEr score during the second stage. Table 2 shows the values set for important hyperparameters in the proposed model.

Table 2. Hyperparameter values.

Hyperparameter used	Value
No. of detected object regions $\left(N\right)$	100
Region feature dimension	2048
No. of hidden units of GRU $\left(M\right)$	2048
Input embedding dimension $\left(d\right)$	2048
Relation-aware attention dimension $\left(d\right)$	256
Head number for relation-aware attention	2
Region-aware attention dimension	1024
Head number for region-aware attention	8
Optimizer	Adam
Learning Rate	0.0003
Attentions Heads	12
Decay Rate	0.8
Dropout (GELU)	0.4
Dropout (relation-aware attention)	0.5
Max. decoding steps	12
Batch Size	64
No. of iterations	32

Ablation study

This section shows the results of the ablation test conducted to show the role of each module of the proposed model. Table 3 shows the different variants of the proposed model and their performances. Our variant $1$ (Reg-aware Att w/o attention loss) is the baseline model which does not use any attention loss function for training purposes. Our variant 2 (Reg-aware Att w attention loss) is similar to variant $1$, however, this variant is trained explicitly with attention loss functions such as entity loss and interaction loss. Moreover, this variant is used to verify the role of attention losses in improving the performance of variant $1$. Also, we have checked whether region-aware attention is sufficient enough to focus on the relationships between different objects effectively. The gain in performance of model $2$ over model $1$ validates that the use of loss functions (either entity or interaction) is beneficial for facilitating the attention model to apply both attention mechanisms. Also, region-aware attention is not able to explore the contribution of relation-aware for interactive words, which leads to poor performances. Our variant $3$ (Reg-aware Att + Rel-aware Att w Att loss) uses a relation-aware attention mechanism in addition to the region-aware attention with attention losses. When we use a relation-aware attention module, the performance of the proposed model is improved significantly. Our variant $4$ (Reg-aware Att + Rel-aware Att w loss + Self-Att) can model intra-model interactions (word-to-word or region-to-region) with entity loss by using the transformer-based self-attention module. Our variant $5$ (Reg-aware Att + Rel-aware Att w loss + Self-Att + Hadamard Product) is used to model inter-model deep interactions (word-to-region) with interaction loss. It can be observed that entity loss is less important as compared to interaction loss. When we do not use entity loss, our model can acquire entity information subtly. When the model does not use interaction loss, it is difficult to learn object relationships by the model. Precisely, we initially optimize the entire architecture of the proposed model by adding the cross-entropy loss, similar to the previous models. Similar to existing works [56, 57], we employ self-critical sequence training strategy for optimizing our proposed model with CIDEr score during the second stage.

Table 3. Ablation tests conducted on MSCOCO Karparthy test split.

Variant	B@4	M	R	S	C
Reg-aware Att w/o Att loss	39.2	28.3	47.9	21.6	120.6
Reg-aware Att w Att loss	40.9	29.1	48.9	22.4	124.2
Reg-aware Att + Rel-aware Att w loss	42.2	29.5	49.3	22.9	124.9
Reg-aware Att + Rel-aware Att w loss + Self-Att	43.1	30.1	50.6	23.2	125.6
Reg-aware Att + Rel-aware Att w loss + Self-Att + Hadamard Product	43.6	30.6	61.1	23.7	126.1

Quantitative analysis

In Tables 4 and 5, we have shown the results obtained by our image captioning model and compared the results with the state-of-the-art model on MSCOCO dataset. We have integrated the proposed model with Cross-Entropy Loss and CIDEr Score Optimization settings. In Table 6, we have shown the results obtained by our image captioning model and compared the results with the state-of-the-art model on Flickr30k datasets, respectively. For fair comparisons, the proposed model uses similar hyperparameter values as considered by existing image captioning methods.

Table 4. Results of our context-aware co-attention based image captioning model and compared models on MSCOCO Karpathy test split with cross-entropy loss.

Reference method	B@1	B@4	M	R	S	C
BUTD [19]	77.2	36.2	27.0	56.4	20.5	112.5
SCST [14]	–	30.0	25.9	53.4	–	99.4
AoANet [24]	77.4	37.2	28.4	57.5	21.3	119.8
GCN-LSTM [18]	77.3	36.8	27.9	57.0	20.9	116.3
SGAE [15]	77.6	36.9	27.7	57.2	20.9	116.7
X-LAN [48]	78.0	38.2	28.8	58.0	21.9	122.0
CtxAdpAtt [49]	77.8	38.1	28.9	58.6	22.5	120.9
High-Order [32]	79.8	40.0	28.8	59.0	22.0	122.7
Ctx-Aw-Att [26]	75.9	35.8	27.8	56.4	–	111.3
Mul_Att [11]	79.9	42.4	29.4	59.7	23.2	125.7
GNN_VR [13]	80.9	43.4	30.3	60.7	–	124.6
Ours (Reg_Att + Rel_Att)	81.2	43.1	29.9	60.9	23.3	125.1
Ours (Reg + Rel + Co-attention)	82.1	43.6	30.6	61.1	23.7	126.1

Table 5. Results of our context-aware co-attention based image captioning model and compared models on MSCOCO Karpathy test split with CIDEr score optimization.

Reference method	B@1	B@4	M	R	S	C
BUTD [19]	79.2	36.3	27.7	56.9	21.4	120.1
SCST [14]	–	34.2	26.7	55.7	–	114.0
AoANet [24]	80.2	38.9	29.2	58.8	22.4	129.8
GCN-LSTM [18]	80.5	38.2	28.5	58.3	22.0	127.6
SGAE [15]	80.8	38.4	28.4	58.6	22.1	127.8
X-LAN [48]	80.8	39.5	29.5	59.2	23.4	132.0
CtxAdpAtt [49]	81.0	39.5	29.7	59.7	24.0	131.1
High-Order [32]	81.1	40.7	29.4	59.4	23.0	130.4
IDGAN [29]	81.3	38.5	28.5	58.8	–	123.5
SRPL [50]	77.8	37.7	28.1	57.4	21.1	117.6
EnsCaption [28]	81.7	39.2	29.4	59.0	–	125.5
M2-Transformer [23]	80.8	39.1	29.2	58.6	22.6	131.2
Ours (Reg_Att + Rel_Att)	81.5	43.5	30.2	61.4	23.8	126.4
Ours (Reg + Rel + Co-attention)	82.7	44.1	30.8	61.9	24.1	132.1

Note: Bold values represent the highest value of the previous models with which we have compared the proposed model.

Table 6. Results of our context-aware co-attention based image captioning model and compared models on the Flickr30k dataset.

Reference method	B@1	B@4	M	R	S	C
CapGen-VisAtt [51]	66.9	19.9	18.46	–	–	–
ImCap-SemAtt [52]	64.7	23.0	18.9	–	–	–
SCA-CNN [53]	66.2	22.3	195	44.9	–	44.7
AdapAtt [54]	67.7	25.1	20.4	46.7	14.5	53.1
ALT [55]	68.5	27.0	21.2	48.0	15.5	56.2
BUTD [19]	72.4	28.3	22.0	49.3	16.3	60.4
AoANet [24]	72.8	29.7	22.3	50.1	16.6	64.3
High-Order [32]	73.2	30.1	22.9	50.1	16.8	65.0
EnsCaption [28]	76.2	32.9	24.6	54.2	–	69.3
Ctx-Aw-Att [26]	69.8	27.7	21.5	48.5	–	57.4
IDGAN [29]	72.9	29.3	21.8	49.9	–	65.0
SRPL [50]	75.6	32.0	23.5	52.8	–	67.5
Mul_Att [11]	74.3	31.2	23.5	51.5	17.2	65.6
GNN_VR [13]	75.2	32.2	24.5	52.5	–	64.6
Ours (Reg_Att + Rel_Att)	75.6	32.7	24.4	53.1	16.9	65.1
Ours (Reg + Rel + Co-attention)	76.9	33.6	25.1	55.1	17.8	69.8

Note: Bold values represent the highest value of the previous models with which we have compared the proposed model.

Tables 4–6 demonstrate the effectiveness of the proposed model as it outperforms the state-of-the-art on almost all metrics. Therefore, it can be observed that the proposed model generates high-quality captions by explicitly focusing on the visual relationships between objects based on the robust semantic context. The proposed model explores the intra-model interactions (word-to-word or region-to-region) with entity loss by using the transformer-based self-attention module. Further, the improvement in the performance is achieved by capturing the inter-model deep interactions (word-to-region) with interaction loss. It can be observed that entity loss is less important as compared to interaction loss.

In Tables 5 and 6, we have shown the superiority of the proposed model, which uses a multilevel co-attention mechanism with region-aware and relation-aware attention mechanisms, over the existing models on almost all the evaluation metrics. On the MSCOCO dataset, the proposed model outperforms the $\mathrm{E}\mathrm{n}\mathrm{s}\mathrm{C}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$ [28] and $\mathrm{M}2$-$\mathrm{T}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}$ [23] models by 4.9% and 5.0% on $\mathrm{B}@4$, 1.4% and 1.6% on $\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}$, 2.9% and 3.3% on $\mathrm{R}\mathrm{O}\mathrm{U}\mathrm{G}\mathrm{E}$-$\mathrm{L}$, 6.6% and 0.9% in CIDEr metrics, respectively. On the Flickr30k dataset, the proposed model outperforms the $\mathrm{M}\mathrm{u}\mathrm{l}\mathrm{\_}\mathrm{A}\mathrm{t}\mathrm{t}$ [11] and $\mathrm{E}\mathrm{n}\mathrm{s}\mathrm{C}\mathrm{a}\mathrm{p}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}$ [28] models by 2.4% and 0.7% on $\mathrm{B}@4$, 1.6% and 0.5% on $\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}$, 3.6% and 0.9% on $\mathrm{R}\mathrm{O}\mathrm{U}\mathrm{G}\mathrm{E}$-$\mathrm{L}$, 4.2% and 0.5% on $\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}\mathrm{r}$-$\mathrm{D}$ respectively. Our model achieved improved results over all other models on the $\mathrm{S}\mathrm{P}\mathrm{I}\mathrm{C}\mathrm{E}$ metric with 24.1 and 17.8 scores on MSCOCO and Flickr30k datasets, respectively.

Using the MSCOCO dataset, we evaluate the computational complexity in terms of average training FLOPs and training time per image. Since training time and FLOPs have a positive correlation, we utilize the FLOPs metric to calculate complexity. In terms of the number of parameters, floating-point operations, training time, and other model parameters, Table 7 compares the proposed model with the existing methods.

Table 7. Comparison with the existing models in terms of the number of parameters in million (M), training time on GPUs, Flops, layers, width and MLP on MSCOCO.

Model	No. of parameters (M)	Training time	Flops	Layers	Width	MLP
BUTD [19]	52	960 h (M40)	4.8	24	384	1536
MA_RN [11]	115	48 h (V100)	9.5	12	768	3072
Ext_Know [12]	135	74 h (V100)	12.6	24	768	3072
GNN_VR [13]	43	43 h (GTX1080)	4.8	24	384	1536
Ours	41	39 h (GTX1080)	3.8	24	768	1072

Qualitative analysis

Qualitative results of our model on MSCOCO and Flickr30k dataset are shown in Figures 4 and 5 respectively. The model attends to the relation between visuals as well as image regions for generating image captions using semantic information about the image. For each test image, the regions and relations are detected for generating the caption words. For region-aware attention, weights are assigned to different identified regions that are further used to establish the relationship among image regions using relation-aware attention.

Figure 4. Qualitative results of our model and other models on MSCOCO. Ours indicates the captions generated by our model. EnsCaption [28] and M2-Transformer [23] are the strong comparative models. For each image, we have shown one interaction and two entity words. Highest attention weights are shown in red colour.

Figure 5. Qualitative results of our model and other models on Flickr30k. Ours indicates the captions generated by our model. EnsCaption [28] and Mul_Att [11] are the strong comparative models. For each image, we have shown one interaction and two entity words. Highest attention weights are shown in red colour.

For example, in Figure 4 (first row), region-aware attention attended regions ‘cat’ and ‘mirror’ that demonstrate the correct identification of regions whereas relation-aware attention finds the relationships ‘looking’ and ‘at’ between ‘cat’ and ‘mirror’. It results in the caption ‘A cat is looking at its reflection in a mirror’ using $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ encoder-decoder. In Figure 5 (first row), ‘players’, ‘long hair’ and ‘softball’ objects are identified by region-aware attention whereas relation-aware attention is accountable to find the relationships ‘two’, and ‘playing’ which generates the caption ‘Two players with long hair are playing softball in a field’ using attention selector. The qualitative results illustrated the effectiveness of our co-attention mechanism using the $\mathrm{G}\mathrm{R}\mathrm{U}$-$\mathrm{G}\mathrm{E}\mathrm{L}\mathrm{U}$ encoder-decoder.

In Figure 6, we have shown the visualization results of relation-aware attention, region-aware attention, and co-attention mechanism during the caption generation phase. For every word generated during the decoding process, we focus on the salient objects/regions, and capture the relationship between these objects/regions, highlighting the regions in red. Our model can capture the intra-modal and inter-modal interactions between these objects/regions using a transformer-based co-attention mechanism. It validates the capacity of the proposed model for generating quality captions from globally to locally.

Figure 6. Visualization results of attended regions/objects of our context-aware co-attention module during the decoding phase. Higher attention weights are shown in the form of brighter regions. (Best viewed in colour and $200\text{\%}$).

Limitations and future scope

We analyse the errors made by our model to study the limitations of our method. In some cases, our model fails to give accurate results. One of the limitations identified for our model is that it generates a similar caption for two different images which are visually similar. The reason behind this may be the lack of effective feature extraction of objects in the image. It may be due to not considering the mapping of object relationships and the traits of objects. For example, in Figure 7 (leftmost and middle), our model has generated the same caption ‘A man is riding a bicycle’, for both images. The other limitation of our model is that it sometimes fails to differentiate between similar-looking objects with respect to shape. For example, in Figure 7 (rightmost), our model fails to identify ‘onion’ and ‘orange’, which are similar, and thus generates the wrong caption. The reason behind this may be the lack of detailed descriptions of image objects which results in the captions of less relevance. The future scope of our work may be to use a graph network-based encoder-decoder model to represent the association between properties and relationships of image objects. In this way, we can resolve the first limitation of this work. The other above-mentioned limitation of current work can be handled by using a co-attention mechanism such that it will be able to provide detailed image descriptions for improved caption generation.

Figure 7. Same caption is generated by the proposed model for the left and middle images, as these two images are visually similar. In the rightmost image, our model fails to identify “onion” and “orange” and thus generates the wrong caption.

Conclusion

In this paper, we propose a transformer-based co-attention network that uses linguistic information to capture the pairwise visual relationships among objects and significant visual features to generate entity and interactive words. The proposed model uses both region-aware attention and relation-aware attention to discover and focus on the related pairwise relationship between objects selectively to generate interactive words. Moreover, we can integrate this technique into other vision language tasks such as visual question answering and Image QA. The proposed model obtains the improved results as: $44.1/30.8/61.9/132.1/24.1$ scores on the MSCOCO dataset and $33.6/25.1/55.1/69.8/17.8$ scores on the Flickr30k dataset corresponding to $\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}@4/\mathrm{M}\mathrm{E}\mathrm{T}\mathrm{E}\mathrm{O}\mathrm{R}/\mathrm{R}\mathrm{O}\mathrm{U}\mathrm{G}\mathrm{E}/\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}\mathrm{r}/\mathrm{S}\mathrm{P}\mathrm{I}\mathrm{C}\mathrm{E}$, respectively. The effectiveness of our model is further validated by the ablation tests and visualization.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Himanshu Sharma

Himanshu Sharma is an Associate Professor in the Department of Computer Engineering and Applications, GLA University, Mathura, India. He has done his Ph.D. from GLA University Mathura and M.Tech. from NSIT, New Delhi, India. His area of research is Computer Vision, Image Processing, and Natural Language Processing. He has published many research papers in reputed journals and conferences.

Swati Srivastava

Swati Srivastava is currently working as an Assistant Professor in the Department of Computer Engineering and Applications, GLA University, Mathura, India. She has completed her Ph.D. in Computational Intelligence from HBTU Kanpur, India, and M.Tech. from NIT Allahabad, India. Her area of research includes high-dimensional neurocomputing, computational intelligence, machine learning, and computer vision focused on biometrics. She has published many research papers in reputed journals and conferences.