MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning

Abstract

This paper presents MoCoUTRL: a Momentum Contrastive Framework for Unsupervised Text Representation Learning. This model improves two aspects of recently popular contrastive learning algorithms in natural language processing (NLP). Firstly, MoCoUTRL employs multi-granularity semantic contrastive learning objectives, enabling a more comprehensive understanding of the semantic features of samples. Secondly, MoCoUTRL uses a dynamic dictionary to act as the approximately ground-truth representation for each token, providing the pseudo labels for token-level contrastive learning. The MoCoUTRL can extend the use of pre-trained language models (PLM) and even large-scale language models (LLM) into a plug-and-play semantic feature extractor that can fuel multiple downstream tasks. Experimental results on several publicly available datasets and further theoretical analysis validate the effectiveness and interpretability of the proposed method in this paper.

KEYWORDS:

• Natural language processing
• text representation learning
• momentum contrast
• alignment
• uniformity

1. Introduction

Discovering universally applicable representations is one core objective of deep learning (DL) (Hjelm et al., 2019), and one intuitive way is to train a representation learning function. Representation learning is beneficial for a wide range of machine learning tasks (Girshick et al., 2014; Hill et al., 2016; Oquab et al., 2014) based on it (Bengio et al., 2013). Deep neural networks for natural language processing (NLP) have advanced rapidly in past decades. As Bengio et al. presented the so-called neural net language models (NNLMs) (Bengio et al., 2003), distributed representations for symbolic data (Hinton, 1986) were introduced to the NLP field, leading to continuous exploration and application of representation learning in the NLP field.

Research on text representation learning can be roughly divided into two phases: methods before and after neural networks are utilised. The most primitive and naive text representation method uses statistical concepts such as word frequency; commonly used methods are one-hot encoding and TF-IDF (term frequency-inverse document frequency). Besides its intuitiveness and ease of operation, this method has obvious shortcomings. On the one hand, representation methods based on statistical quantities cannot sufficiently capture the semantic similarity relationship of fine-grained text sequences. On the other hand, the text representations generated by these methods have high dimensions, which are prone to the problem of the curse of dimensionality.

Since the emergence of NNLMs, researchers have focused on research that utilises pre-trained language models (PLMs) to obtain text representations. These methods are called word embedding technology or distributed representation methods of words. The resulting embedding of each token is wholly derived from the context of surrounding words, thus providing a more refined reflection of semantic similarity and correlation between words. Due to limited data and computing capacity, early word embedding models were of smaller scale and could only produce static word vectors. However, with the advancement of computing power, context word embeddings generated by large-scale PLMs have become the essential underlying method for various tasks in the NLP field.

Despite the excellent performance in many NLP applications, exponentially increasing computing power threshold and negligible marginal benefits push research on PTMs into a tricky spot. Therefore, this work focuses on text representation learning from another perspective – contrastive learning. Contrastive learning aims to generate a universal representation for each sample by pulling semantically close neighbours together and pushing apart non-neighbours (Hinton & Salakhutdinov, 2006). This training framework was first adopted in CV to train unsupervised pre-trained models since the previous mainstream pre-trained models in CV were obtained by supervised training on ImageNet (Deng et al., 2009). The success of contrastive learning in CV led to studies on contrastive learning for text representation learning. However, in recent research on text contrastive learning, there remain two problems: Firstly, The smallest unit of text data is a word, but existing works on text contrastive learning always use a sequence of text as the smallest unit of representation. We deem that introducing words as the smallest semantic unit in contrastive learning is necessary. From conventional static word embedding models like word2vec(Mikolov, Chen, et al., 2013; Mikolov, Sutskever, et al., 2013), to pre-trained language models like BERT (Devlin et al., 2019), and even to currently state-of-the-art ultra-large language models such as GPT-4(OpenAI, 2023), PaLM (Chowdhery et al., 2022) and LLaMA (Touvron et al., 2023), their training processes rely on the inspiration of the distributional hypothesis(Harris, 1954), where the meaning of a word is jointly determined by its context. In the ideal case of a long text sequence, rich contextual information can provide sufficient semantic details; but the distributed hypothesis may face significant challenges in short text sequences. For example, “Jason ran away happily without any words” and “Jason ran away angrily without any words” can be considered two text sequences with totally different semantics. However, there is no difference between the two sentences except for “happily” and “angrily”. Even the most influential models cannot deduce opposed semantic expressions from the same context. Existing researchers (Gao et al., 2021; Qian et al., 2022) lack direct learning of the smallest semantic unit of terms, which may not perform well on short sequences.

Secondly, unlike image data, where high-dimensional representations can be obtained directly from the original documents (He et al., 2022), text data lacks a reliable representation of text labels (Lee et al., 2022) if contrastive learning at the representational level is performed on text. Recent advances still follow the in-batch contrastive learning approach based on NTXent-like loss. The data within a batch comprises multiple sample pairs, and a sample pair contains a pair of semantically similar samples, for example, a pair of text sequences containing an entailment relation is used as a sample pair in the SimCSE (Gao et al., 2021). During training, the two samples within each sample pair are mutually positive, while all other data within the batch are mutually negative sample pairs. However, this within-batch contrastive learning approach depends on the batch size and usually works well for large-size batches (usually containing thousands of sample pairs). Also, the two samples in a positive pair are not semantically identical to each other. For example, the “Two dogs are running.” and “There are animals outdoors.” are treated as a pair of positive samples. Although the two sentences are semantically related, it does not mean that the two sentences have exactly the same meaning, and the training goal of NTXent will contribute to the convergence of the representations of the two utterances, which is inconsistent with the actual semantic situation.

To solve the above problems, we propose MoCoUTRL (a Momentum Contrastive Framework for Unsupervised Text Representation Learning), mainly making the following contributions. The code of this work is open-sourced on https://github.com/Uchiha-Monroe/MoCoUTRL_pub_ver

1. MoCoUTRL introduces a contrastive learning loss at the token level in addition to the conventional sentence level so that the semantic features of the samples can be learned more fully, especially the model’s performance on shorter text sequences.

2. Unlike the positive and negative sample generation methods used in previous text contrastive learning, this work constructs a dynamic representation lexicon as an enhanced positive sample pair for token-level contrastive learning, and this approach avoids the semantic bias problem associated with traditional text enhancement.

3. In addition to the conventional contrastive experiments, this work further validates the rationality and interpretability of this work by visualising and quantitatively analysing the representations of the trained MoCoUTRL model using two attributes related to the quality of characterisation learning, alignment and uniformity (Wang & Isola, 2020).

2. Related works

2.1 Contrastive learning

Contrastive learning is an essential unsupervised learning method because it eschews manual labelling. Contrastive learning originated in CV (Dosovitskiy et al., 2014; Hadsell et al., 2006) and has also made progress in other fields of DL (Bachman et al., 2019; Kong et al., 2019; Oord et al., 2019). The inspiration for contrastive learning was inherited from the InfoMax criterion (Linsker, 1988) and bred a bunch of innovative research over the years. SimCLR (Chen et al., 2020) proposes a straightforward, high-performance image representation learning method using contrastive learning. The nonlinear projector specially introduced at the end of the model significantly improves the performance. MoCo (He et al., 2020) uses a momentum encoder to represent the original memory bank, a data structure that stores fixed label representations, dramatically improving the algorithm’s efficiency. BYOL (Grill et al., 2020) innovatively proposes a generative contrastive learning method, which abandons using negative samples in traditional contrastive learning methods. SwAV (Caron et al., 2020) creates a prototype label similar to the cluster centre for samples in contrastive learning, improving the training process’s stability. There is also much recent research on contrastive learning to various subfields in NLP. Numerous empirical successes reveal the great potential of contrastive learning for representation learning.

2.2 Text representation learning

A good representation contains rich semantic information and can reveal data distribution. Various features can better apply semantic information to various downstream tasks. Text data is probably the most abstract and complex data format currently faced by deep learning, and research on its representation learning has always been the focus of the field of natural language processing. The introduction of words’ distributed representation (Mikolov, Sutskever, et al., 2013) makes the text representation out of the statistical bag-of-words methods such as TF-IDF. Word embedding is assigning a distributed high-dimensional representation to each word. The proposal of Neural net language models (Bengio et al., 2003) introduces the subsequent exploration of text representation. Text representation learning is closely related to language models in most subsequent studies. In the era of contextual word embedding, the research on text representation learning has entered a rapid development stage from this point on. We will introduce the background of contextual word embedding and PLMs in the next section.

2.3 Pre-trained language models

Word embedding models, such as Word2vec (Mikolov, Sutskever, et al., 2013) and GloVe (Pennington et al., 2014), provide each token with a unique and invariable representation and suffer from problems with polysemous words and out-of-vocabulary (OOV) words. With the subsequent emergence of ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), large-scale PTMs have gradually become the mainstream method in NLP. The so-called contextual embedding generated by them is a dynamic representation method, which solves the problems quickly faced by their static counterparts. Roughly, PTMs can be divided into the following categories according to the model structure: autoregressive models represented by GPT (Radford et al., 2018), CTRL (Keskar et al., 2019), Transformer-XL (Dai et al., 2019), Reformer (Kitaev et al., 2020), autoencoding models represented by BERT (Devlin et al., 2019), ALIBERT (Lan et al., 2020), RoBERTa (Liu et al., 2021), and sequence-to-sequence models represented by BART (Lewis et al., 2020), T5 (Raffel et al., 2020). Overall, autoencoding language models like BERT are the first vital achievements to achieve universal breakthroughs in various NLP sub-tasks, and these models are better at handling multiple discriminative downstream tasks. The autoregressive language models represented by GPT follow the design pattern of traditional one-direction language models and are relatively good at solving text-generation tasks. seq2seq structured pre-trained language models are constructed by combining the advantages of the above two frameworks in order to have both discriminative and generative downstream tasks, and experiments have shown that models such as T5 have indeed achieved very good experimental results at that time. It has been shown that models such as T5 have indeed achieved very good experimental results at the time.

3. Methodology

The general architecture of our proposed MoCoUTRL is depicted in Figure 1. We aim to learn a text representation function mapping each input text to a feature in an unsupervised way. Generally, the framework learns by comparing the distance between inputs and their augmented samples in the feature space. There are totally five major components in this framework: (1) a random data augmentation module, (2) a PTM-based encoder, (3) a momentum-based dynamic dictionary, (4) a nonlinear feature projector, and (5) a contrastive loss function module. In the rest of this section, we will first go through the overall workflow of the framework and then describe each of these five structure components.

Figure 1. The overall structure of MoCoUTRL.

In this part, we first give the overall structure of MoCoUTRL, as shown in Figure 1, and briefly describe the proposed method. Later, this work elaborates on the data authentication module, momentum-based dynamic dictionary, PTM-based representation encoder, nonlinear Feature projector and contrastive loss function in five sections. Finally, an overall MoCoUTRL algorithm flow is given in detail.

Figure 1 depicts the overall structure of our proposed MoCoUTRL model. The meaning of SLD and TLD in the figure stands for Semantic Latent Decoder and Token Latent Decoder, respectively. For the input text, MoCoUTRL only adopts different components in transforming the text into a vector representation for the first time. It uses a regular Embedding Layer for the prototype sample and Dynamic Dictionary for positive and negative samples while using identical model components for the rest of the Encoder, SLD Projector, and TLD Projector.

The whole training corpus $\mathcal{C}=\{x_1,x_2,x_3,\dots ,x_N\}$ consists of cleaned text sentences from the unlabelled text. Before training, data augmentation converts each sample $x_i$ into a positive $x_i^{+}$ and a negative sample $x_i^{-}$. Then, all these samples are converted into ordered sequences of tokens by tokenisation. The tokenised $x_i=[w_{cls},w_1,w_2,\dots ,w_n]$ will be transformed into a tensor ${\mathbf{\text{x}}}_i\in {\mathbb{R}}^{n\times emb}$ through the word embedding layer $e_{\phi }(\cdot )$, where $n$ is the length of the token sequence and $emb$ is the dimension of the word embeddings. Similarly, augmented samples $x_i^{+}$ and $x_i^{-}$ are also transformered into tensors ${\mathbf{\text{x}}}_i^{+}$ and ${\mathbf{\text{x}}}_i^{-}$ by tokenisation and vectorisation, but not using $e_{\phi }(\cdot )$ but a dynamic dictionary $d_{\mathit{\xi }}(\cdot )$. Afterwards, these tensors will be respectively input into a PTM-based feature encoder $f_{\mathit{\theta }}(\cdot )$ to generate feature representations.

3.1 Data augmentation module

Data augmentation is of essential significance in discriminative contrast learning. The pattern of data augmentation and the number of augmented samples associated with each sample during the model training profoundly impact the overall contrastive learning framework. Typical contrastive learning involves bringing positive sample pairs as close as possible to each other and pushing negative sample pairs as far apart as possible in the feature space so that the final representations of input samples generated by the feature encoder $f_{\mathit{\theta }}(\cdot )$ in the feature space contain rich semantic information. Unlike many intuitive and effective data augmentation methods such as crop, rotation and colour distortion in CV (Chen et al., 2020), there is no consensus on text enhancement methods in the NLP field.

The MoCoUTRL conducts data augmentation using synonym and antonym substitution. During training, MoCoUTRL uses both sample-level and token-level contrastive learning tasks. Specifically, for a sample-level text denoted as $x^{ori}$, a randomly selected token is used for a synonym or antonym substitution, thus producing positive and negative samples labelled as $x^{syn}$ and $x^{ant}$ respectively.

3.2 Momentum update-based dynamic dictionary

MoCoUTRL trains the encoder $f_{\mathit{\theta }}(\cdot )$ by matching an encoded input $q$ to a dynamic dictionary of encoded keys $d_{\mathit{\xi }}(\cdot )$ by contrastive training. Unlike the traditional contrastive learning method based on the Siamese model, MoCoUTRL employs a conventional encoder to transform input samples into high-dimensional representations. In contrast, the positively and negatively augmented samples utilise an extra dynamically maintained feature dictionary to obtain real-time feature representations. Theoretically, we can ultimately use a shared encoder to embed the raw samples and the positive and negative samples, which is one of the fundamental paradigms of contrastive learning (or so-called metric learning) in recent years. However, the training framework that utilises dynamic feature dictionaries can converge more quickly and stably, which has been validated in the research of the CV field. The dynamic dictionary can be seen as composed of initial ground-truth representations of all tokens, whose parameters are initialised by a word embedding layer of PTMs, and then trained according to momentum updates in each training process. Note that in previous studies of the CV field, the size of the whole dynamic dictionary is often so large that it cannot be stored in memory. However, this problem does not exist in NLP, as the size of the entire dictionary obtained after tokenisation is acceptable for modern high-performance GPUs. Here, this work opts for the empirical approach (He et al., 2020), momentum contrast, to update the parameters of the dynamic dictionary. Specifically, the parameters of the word embedding matrix $e_{\phi }(\cdot )$ are updated by commonly used backpropagation, while the parameter matrix of the dynamic dictionary is updated by means of momentum contrast. Denote the representation of the kth word in the word list in the word embedding layer and the dynamic dictionary layer as ${\phi }_k$ and ${\xi }_k$, respectively. Thus, at each training step, the parameters involved in the dynamic dictionary will be iterated according to the following momentum update: (1) ${\xi }_k\leftarrow m{\xi }_k+(m-1){\phi }_k$(1) Here $m\in [0,1)$ is a hyperparameter to control the rate of momentum update of ${\xi }_k$. In our experiments, we take relatively large values of $m$(close to 1), thus allowing ${\xi }_k$ to evolve in a slow but smoother and more stable manner compared to ${\phi }_k$. In the CV field, this similar ground-truth table can be implemented intuitively, i.e. the original RGB value in the pixel matrix. However, there is no direct way to transform words into real-value vectors, so MoCoUTRL uses the word embedding layer of PTMs to initialise this ground-truth table.

3.3 PTM-based representation encoder

Deep neural networks built upon PTMs have become the most common paradigm in NLP today by fine-tuning or prompt learning on downstream applications, achieving state-of-the-art results on various sub-tasks. These previous studies demonstrate that Transformer-based PTMs are currently the best method for processing textual data. This work uses pre-trained DeBERTa (He et al., 2021) as the feature encoder, an improved version of the BERT model with significant improvements in training corpus size, parameter settings, and pre-training tasks.

3.4 Nonlinear feature projector

It has been demonstrated in multiple empirical studies that a nonlinear transformation layer for contrastive learning after the feature encoding layer helps the feature encoder to obtain better representation. Although contrastive learning is currently a powerful and prevalent tool for unsupervised representation learning, especially in the CV field. There is a gap between the target of contrastive learning and that of most downstream tasks, and the role of the nonlinear transformation layer is to fill this gap. Then, the model can generate features with universal semantics but less task-related knowledge.

3.5 Contrastive loss function

In this work, loss functions are designed for the semantic units of sentences and tokens’ granularity.

3.5.1 Long-span contrastive loss

For each $x_i(1\le i\le N)$, we can get its positive variant $x_i^{+}$ and negative variant $x_i^{-}$ by data augmentation. Then the encoder can achieve their instant representations $f_{\mathit{\theta }}(\cdot )$, and the nonlinear projector is utilised to obtain their feature representations for the contrastive learning training task. (2) $\begin{array}{rl}{\mathbf{\text{z}}}_i& =\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(2)}\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(1)}Pooling(f_{\mathit{\theta }}(x_i)+{\mathbf{\text{b}}}_{\text{SLD}}^{(1)}))+{\mathbf{\text{b}}}_{\text{SLD}}^{(2)})\end{array}$(2) (3) $\begin{array}{rl}{\mathbf{\text{z}}}_i^{+}& =\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(2)}\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(1)}Pooling(f_{\mathit{\theta }}(x_i^{+})+{\mathbf{\text{b}}}_{\text{SLD}}^{(1)}))+{\mathbf{\text{b}}}_{\text{SLD}}^{(2)})\end{array}$(3) (4) $\begin{array}{rl}{\mathbf{\text{z}}}_i^{-}& =\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(2)}\sigma ({\mathbf{\text{W}}}_{\text{SLD}}^{(1)}Pooling(f_{\mathit{\theta }}(x_i^{-})+{\mathbf{\text{b}}}_{\text{SLD}}^{(1)}))+{\mathbf{\text{b}}}_{\text{SLD}}^{(2)})\end{array}$(4)

In the above equations, the $\sigma (\cdot )$ means the sigmoid activation function, the $\mathbf{\text{W}}$ and $\mathbf{\text{b}}$ in the above equations are respectively the weight matrix and bias vector of the affine transformation. The $Pooling(\cdot )$ in Eq. (2–4) refers to a pooling operation that aggregates the high-dimensional features generated by each input token into a feature that represents the entire text sequence. In downstream tasks, the [CLS]-pooling method is often used to integrate all output representations. However, many studies have shown that this method is ineffective in representation learning. Sometimes, the performance is less than word embedding models like GloVe (Pennington et al., 2014). The commonly used pooling method for sentence representation uses the average of all token features output by the last layer of the model.

The $\sigma (\cdot )$ in the above equations is Gaussian Error Linear Units (GELU) (Wolf et al., 2020) activation function. (5) $\sigma (x)=GELU(x)=x\cdot \mathrm{\Phi }(x)=x\cdot \frac{2}{\sqrt{\pi }}{\int }_0^x{e}^{-t^2}dt$(5) We referenced the idea of InfoNCE (Oord et al., 2019) and designed a similar but not entirely identical loss function for training. In the original InfoNCE, a batch of data consists of pairs of positive samples, and any unmatched samples form negative sample pairs. The InfoNCE objective function, also known as the NTXent loss function, was proposed in the literature (Oord et al., 2019). This paper constructs a generative prediction task for contrastive learning and creates an objective function by maximising the mutual information between the current context and the data to be predicted. In this work, we construct a separate negative sample for each sample to increase the diversity between opposite semantic samples. Therefore, in actual training, negative samples only come from separately created negative samples, not from any unmatched sample. Precisely, assuming a training batch consists of $B$ (original sample, positive sample, negative sample) triplets, the loss for the i-th triplet is calculated as follows: (6) ${\mathcal{L}}_{longspan}(i)=-\log \frac{exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_i^{+}/\tau )}{{\sum }_{j=1}^{B}exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_j^{-}/\tau )}$(6)

The $\tau$ appears in equation (6) is the temperature factor which controls the “softness and hardness degree” of the softmax-like loss function.

3.5.2 Token-level contrastive loss

Because of the importance of tokens in the NLP field, the MoCoUTRL constructs a token-level contrastive loss function in this part. For each $x_i=[w_{cls},w_1,w_2,\dots ,w_n]$, the token chosen to be the anchor token is denoted as $w_i^{ori}$. The text obtained by replacing the anchor word $w_i^{ori}$ with a synonym $w_i^{syn}$ is a positive sample $x_i^{+}$, and the text obtained by replacing the anchor word with an antonym $w_i^{ant}$ is a negative sample $x_i^{-}$. For the replaced anchor words $w_i^{syn}$ and $w_i^{ant}$, the output of the last layer of the encoder is directly used as its feature representation: (7) $\begin{array}{rl}{\mathbf{\text{w}}}_i^{syn}& =\sigma ({\mathbf{\text{W}}}_{\text{TLD}}^{(2)}\sigma ({\mathbf{\text{W}}}_{\text{TLD}}^{(1)}{f}_{\mathit{\theta }}(w_i^{syn})+{\mathbf{\text{b}}}_{\text{TLD}}^{(1)})+{\mathbf{\text{b}}}_{\text{TLD}}^{(2)})\end{array}$(7) (8) $\begin{array}{rl}{\mathbf{\text{w}}}_i^{ant}& =\sigma ({\mathbf{\text{W}}}_{\text{TLD}}^{(2)}\sigma ({\mathbf{\text{W}}}_{\text{TLD}}^{(1)}{f}_{\mathit{\theta }}(w_i^{ant})+{\mathbf{\text{b}}}_{\text{TLD}}^{(1)})+{\mathbf{\text{b}}}_{\text{TLD}}^{(2)})\end{array}$(8)

For each anchor word, we will use the dynamic dictionary $d_{\mathit{\xi }}(\cdot )$ to obtain their feature representation, respectively, and use this as the anchor representation as the label of token-level comparative learning. (9) ${\mathbf{\text{e}}}_i^{ori}=d_{\mathit{\xi }}(w_i^{ori})$(9) In each training batch, the following methods can be used to calculate the token-level loss: (10) ${\mathcal{L}}_{token-level}=\frac{1}{N}{\sum }_{i=1}^N[MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{syn})-MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{ant})]$(10)

The $MSELoss(\cdot ,\cdot )$ in Equation (10) means the mean squared error (MSE) loss between a pair of embeddings. In this work, a hyperparameter $\lambda \in [0,1]$ is used to balance the significance of these two losses: (11) $\mathcal{L}=\lambda \cdot {\mathcal{L}}_{longspan}+(1-\lambda )\cdot {\mathcal{L}}_{token-level}$(11)

3.6 Algorithm framework of MoCoUTRL

After introducing all components of the proposed model, we depict the steps and details of MoCoUTRL as a whole in Algorithm 1.

Table

Algorithm 1: MoCoUTRL algorithm.
Input: training corpus $\mathcal{C}=\{x_i{\}}_1^N$, encoder $f_{\mathit{\theta }}(\cdot )$, dynamic dictionary $d_{\mathit{\xi }}(\cdot )$, nonlinear projector of latent semantic space ${p^{SLS}}_{\mathit{\eta }}(\cdot )$ and token latent space ${p^{TLS}}_{\nu }(\cdot )$, hyperparameter num_epochs, batch_size, initial learning rate ($lr$), $m\in [0,1)$, and $\lambda \in [0,1]$
Output: encoder $f_{\mathit{\theta }}(\cdot )$, discarding $d_{\mathit{\xi }}(\cdot )$, ${p^{SLS}}_{\mathit{\eta }}(\cdot )$, and ${p^{TLS}}_{\nu }(\cdot )$
Initialisation: $f_{\mathit{\theta }}(\cdot )$ is initialised by PTM (e.g. DeBERTa), $d_{\mathit{\xi }}(\cdot )$ is initialised by the embedding layer of PTM (e.g. DeBERTa), ${p^{SLS}}_{\mathit{\eta }}(\cdot )$, and ${p^{TLS}}_{\nu }(\cdot )$ are initialised by Glorot initialisation
for i_epoch = 1 to num_epochs do
for i_batch = 1 to num_batchs do
fetch data $x^{ori}$, $x^{syn}$, $x^{ant}$, $w_i^{ori}$, $w_i^{syn}$, and $w_i^{ant}$
${\mathbf{\text{x}}}^{ori}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{ori})),{\mathbf{\text{x}}}^{syn}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{syn})),{\mathbf{\text{x}}}^{ant}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{ant}))$
${\mathcal{L}}_{longspan}=-\log \frac{exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_i^{+}/\tau )}{{\sum }_{i=1}^{N}exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_i^{-}/\tau )}$
${\mathbf{\text{e}}}_i^{ori}=d_{\mathit{\xi }}(w_i^{ori}),{\mathbf{\text{w}}}_i^{syn}={p^{TLS}}_{\nu }(f_{\mathit{\theta }}(w_i^{syn})),{\mathbf{\text{w}}}_i^{ant}={p^{TLS}}_{\nu }(f_{\mathit{\theta }}(w_i^{ant}))$
${\mathcal{L}}_{token-level}=\frac{1}{N}{\sum }_{i=1}^N[MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{syn})-MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{ant})]$
$\mathcal{L}=\lambda \cdot {\mathcal{L}}_{longspan}+(1-\lambda )\cdot {\mathcal{L}}_{token-level}$
${\theta }_{t+1}\leftarrow {\theta }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\theta }_t},{\eta }_{t+1}\leftarrow {\eta }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\eta }_t},{\nu }_{t+1}\leftarrow {\nu }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\nu }_t}$
${\xi }_{t+1}\leftarrow m{\xi }_t+(m-1){\phi }_t$ ($\phi$ are parameters of the embedding layer of $f_{\mathit{\theta }}(\cdot )$)
update $lr$ by schedule
End for
End for

4. Experiments

4.1 Datasets

The experiments of this work are all carried out on Ubuntu 18.0 with four GPUs (NVIDIA Tesla V100 PCIe-32GB). This work is completed upon open-source frameworks such as Pytorch (Paszke et al., 2019) and Transformers (Wolf et al., 2020).

The BookCorpus (Zhu et al., 2015) is used to construct training data during training. The BookCorpus dataset consists of free novels created by unpublished authors. It contains 11,038 novels, about 74M sentences and about 1G words, and can be divided into 16 themes, such as history and adventure. The semantic Textual Similarity (STS) dataset (Cer et al., 2017) and the Web of Science (WOS) dataset (Kowsari et al., 2017) are used to conduct comparative experiments. The STS Benchmark comprises a selection of the English datasets used in the STS tasks organised in SemEval between 2012 and 2017. Its corpus is derived from image captions, news headlines, and forum texts. The WOS dataset is a text classification dataset collected by the literature (Kowsari et al., 2017) from 46,985 published Web of Science papers containing seven categories and 134 subcategories.

4.2 Experimental settings

First and foremost, in selecting the encoder model, we have tried Bert, Roberta, and Deberta. These three have been strong benchmark models for an extended period since the large-scale PTM was proposed at the end of 2018. After evaluation, this work selects the Deberta model as the backbone of the encoder. In preprocessing the BookCorpus, this paper adopts the method of literature (Kowsari et al., 2017) to convert it into one single sentence per sample and then perform data enhancement to construct the training format (anchor samples, positive samples, and negative samples).

In the experiment, the temperature constant is set to 0.07 by default, the momentum coefficient is set to 0.999, the primary learning rate is set to 1e-6, and the cosine annealing schedule dynamically adjusts the learning rate during the training process. The default batch size of the model in the comparative learning phase is set to 32 so that the total number of samples contained in each training batch is 96, and the number of positive samples at the long span and token-level levels corresponding to each anchor sample is 1. The corresponding explicit negative samples are 3 × 32 − 2 = 94. The parameter momentum update of the dictionary relies on the entire word vector matrix of the encoder, so other tokens in the entire vocabulary are implicitly used as weak negative samples for training in the token-level loss function.

4.3 Comparative experiments on STS Benchmark and WOS

We use the Spearman correlation coefficient as the evaluation criterion on the STS Benchmark and SICK datasets. Note that the ground truth of the whole dataset is $\{y_i{\}}_{i=1}^N$, and the model’s output is $\{{\hat{y}}_i{\}}_{i=1}^N$. The Spearman correlation coefficient is computed as follows: (12) $\rho =\frac{{\sum }_{i=1}^N(y_i-{\overline{y}}_i)({\hat{y}}_i-{\overline{\hat{y}}}_i)}{\sqrt{{\sum }_{i=1}^N{(y_i-{\overline{y}}_i)}^2{\sum }_{i=1}^N{({\hat{y}}_i-{\overline{\hat{y}}}_i)}^2}}$(12) This work adopts unsupervised and supervised models to carry out contrast experiments. The main unsupervised benchmark models used include GloVe, BERT, BERT-flow (Li et al., 2020), BERT-whitening (Su et al., 2021), RoBERTa, CT-BERT (Carlsson et al., 2021), SimCSE-BERT (Gao et al., 2021), SimCSE-RoBERTa (Gao et al., 2021) and DiffCSE (Chuang et al., 2022). The adopted supervised benchmark models include InferSent-GloVe, SentenceBert (Reimers & Gurevych, 2019), SentenceBert-flow (Carlsson et al., 2021), SentenceBert-whitening (Gao et al., 2021), SentenceRoBERTa (Reimers & Gurevych, 2019), and SimCSE-RoBERTa (Gao et al., 2021).

The WOS dataset is a text multi-classification task, and the proportions of different categories differ. This work uses Accuracy, Recall-Macro, Precision-Macro and F1score-Macro to compare MoCoUTRL with other models.

The mathematical expressions of evaluation metrics used in the WOS dataset are listed in Eq. (13)–(16). Note that this work multiplies the original value of each metric by 100 for straightforward representation. (13) $\begin{array}{rl}Accuracy& =\frac{TP+TN}{TP+TN+FP+FN}\times 100\end{array}$(13) (14) $\begin{array}{rl}Precision& =\frac{TP}{TP+FP}\times 100\end{array}$(14) (15) $\begin{array}{rl}Recall& =\frac{TP}{TP+FN}\times 100\end{array}$(15) (16) $\begin{array}{rl}F1-Score& =\frac{2\times (Precision\times Recall)}{Precision+Recall}\end{array}$(16)

In the above formulations, True Positive (TP) represents the number of literature of a specific category is classified correctly, False Positive (FP) means the number of literature of a specific category is classified incorrectly, True Negative (TN) denotes the number of literature that does not belong to a specific category is classified correctly. False Negative (FN) indicates the number of literature that does not belong to a specific category is classified incorrectly. The final evaluation metrics of all six categories in the WOS dataset are obtained by “macro” (calculating the average).

The performance of MoCoUTRL and other methods is presented in Table 1, and experimental results on the WOS dataset are depicted in Table 2. In Table 1, the unsupervised and supervised algorithms are presented separately in the upper and lower parts of the table. The table displays the experimental results of these models on the STS series tasks and the SICK task, with the highest score in each task for unsupervised and supervised models bolded for differentiation. In the models in Table 1, the content in parentheses represents the pooling method used for feature extraction with PTM. “last2 avg” represents the average output of the last two layers of the model. In contrast, “first-last avg” represents the average output of the first and last layers of the model. The benchmark model names followed by “flow” and “whitening” represent the BERT-flow and BERT-whitening models, respectively. The “MoCoUTRL + MLP” means we use a Multi-Layer Perceptron behind MoCoUTRL to do supervised training on downstream datasets.

Table 1. Performance on STS tasks.

Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK
Unsupervised Methods
GloVe(avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76
BERT-base(last2 avg.)	41.72	60.43	50.12	65.27	67.21	51.46	62.13
BERT-base(first-last avg.)	39.70	59.38	49.67	66.03	66.19	53.87	62.06
BERT-base-flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47
BERT-whitening	57.83	66.90	60.90	75.08	71.31	68.24	63.73
CT-BERT-base	61.63	76.80	68.47	77.50	76.48	74.31	69.19
SimCSE-BERT-base	68.40	82.41	74.38	80.91	78.56	76.85	72.23
RoBERTa-base(last2 avg.)	41.65	62.31	52.43	66.75	67.51	58.31	60.43
RoBERTa-base(first-last avg.)	40.88	58.74	49.07	65.63	61.48	58.55	61.63
SimCSE-RoBERTa-base	70.16	81.77	73.24	81.36	80.65	80.22	68.56
DiffCSE-BERT-base	72.28	84.43	76.47	83.90	80.54	80.59	71.23
DiffCSE-RoBERTa-base	70.05	83.43	75.49	82.81	82.12	82.38	71.19
MoCoUTRL	73.21	78.65	75.76	83.40	82.17	82.12	71.17
Supervised Methods
InferSent-GloVe	52.86	66.75	62.15	72.77	66.87	68.03	65.65
SentenceBERT-base	70.97	76.53	73.19	79.09	74.30	77.03	72.91
SentenceBERT-base-flow	69.78	77.27	74.35	82.01	77.46	79.12	76.21
SentenceBERT-base-whitening	69.65	77.57	74.66	82.27	78.39	79.52	76.91
CT-SentenceBERT-base	74.84	83.20	78.07	83.84	77.93	81.46	76.42
SimCSE-BERT-base	75.30	84.67	80.19	85.40	80.82	84.25	80.39
SentenceRoBERTa-base	71.54	72.49	70.80	78.74	73.69	77.77	74.46
SentenceRoBERTa-base-whitening	70.46	77.07	74.46	81.64	76.43	79.49	76.65
SimCSE-RoBERTa-base	76.53	85.21	80.95	86.03	82.57	86.70	81.95
MoCoUTRL + MLP	76.91	85.87	80.31	85.75	83.58	88.31	81.43

Table 2. Performance on the WOS dataset.

Model	Acc	R	P	F1
BERT-base	81.27	80.42	81.94	82.17
RoBERTa-base	85.54	85.36	86.15	85.75
DeBERTa-base	85.71	85.64	85.89	85.61
MoCoUTRL + MLP	88.13	87.79	85.83	86.80

4.4 Analysis of alignment and uniformity

There has always been discussion and research on the problem of anisotropy in text representation learning, which is more evident in the PTMs. Because the output text representation shows strong anisotropy, most features are generally distributed in a relatively narrow high-dimensional cone space, severely limiting its expressive ability. In the former literature (Wang & Isola, 2020), the author proposes alignment and uniformity, two properties that a good feature distribution needs to satisfy. Alignment means that similar samples need to have similar feature representations, and uniformity means that the feature representations generated by the model should retain as much information as possible.

Previous research experiments have proved that text representations obtained by many PTMs have obvious anisotropy problems, contrary to the ideal theoretical properties expected. To further explore the characteristics generated by MoCoUTRL qualitatively and quantitatively, this work analyses the attributes of alignment and uniformity of the output features.

Regarding the nature of alignment, two methods are used to visualise the performance: the first is to count the L2 distance of the output features of the same category of samples, and the second is to select the representations of 5 tokens for each sample randomly. These features are reduced to ${\mathbb{R}}^2$ space using linear kernel PCA, then plot the feature distributions with Gaussian kernel density estimation (KDE), and calculate its von Mises-Fisher (vMF) KDE on angles ($\arctan (x_1,x_2)$ is used in this work) for each sample $(x_1,x_2)$ in ${\mathbb{R}}^2$. In terms of uniformity, we use a similar operation to the second method of exploring alignment properties, except that the input samples are replaced by those belonging to the unified category. The features of four subcategories are selected in the figure for display.

It can be seen from Figure 2 that the pre-trained DeBERTa model has strong anisotropy, the distance between any two samples is close, and the entire representation space is in the shape of a super cone. The supervised DeBERTa has better performance in distinguishing samples of different categories. Nevertheless, it performs poorly in property alignment: Most sample representations are limited to several isolated spaces, contrary to the alignment property. MoCoUTRL has both suitable alignment property and uniformity property. Although there is no special supervised training, MoCoUTRL can still clearly distinguish different types of samples, which can be seen from the representation distribution of samples. The representations generated by MoCoUTRL can be more evenly distributed in the entire representation space, thus preserving maximum information.

Figure 2. Analyses on alignment and uniformity.

4.5 L2 normalisation with Gaussian perturbation

When computing ${\mathcal{L}}_{token-level}$, a dynamic vocabulary updated with parameter momentum is used as the label representation for each token. Current contrastive methods use L2 regularisation to ensure all representations are released on a hypersphere. Thus, if cosine similarity is the similarity of any two representations, we can use the dot product to simplify the calculation. However, this approach also has specific problems. One of the most prominent problems is the sparsity of data distribution in high-dimensional space, illustrated by Figure 3.

Figure 3. The sparsity of high-dimensional space.

Compared with all representations distributed on the hypersphere, this work distributes the features on the outermost thin shell of the hypersphere with a certain thickness. Similarly, taking the 2D situation as an example, it is assumed that the entire circle with radius 1 is the range of all representations, and the blue circle part is the actual sample representation distribution range output by the model. In this case, the proportion of the ring part to the entire space is as follows: (17) $\eta =\frac{V_{ring}}{V_C}=\frac{\pi \cdot r^2-\pi {(r(1-\epsilon ))}^2}{\pi \cdot r^2}=1-(1-\epsilon {)}^2$(17) Extend it to a high-dimensional space, then: (18) $\eta =\frac{V_{ring}}{V_C}=\frac{k\cdot \pi \cdot r^n-k\cdot \pi {(r(1-\epsilon ))}^n}{k\cdot \pi \cdot r^n}=1-(1-\epsilon {)}^n$(18)

Since $\epsilon \in (0,1)$, when the dimension n of the high-dimensional space approaches positive infinity, then there is: (19) $\eta =\underset{n\to \mathrm{\infty }}{lim}1-(1-\epsilon {)}^n=1$(19) That is, as long as the thickness of the outer shell of the hypersphere is any constant greater than 0, the proportion of the volume occupied by the shell in the high-dimensional space is infinitely close to the volume of the entire representation space. In practical experiments, since the generated feature representations are finite-dimensional, it is necessary to consider more realistic situations. The benchmark encoder model used in the experiment is Deberta-base, and the output feature dimension is 768 dimensions. Assuming that the thickness of the shell is 1% of the radius of the hypersphere, then: (20) $\eta =1-(1-0.01{)}^{768}=0.9996$(20)

It can be seen from this that in the higher-dimensional space, a large part of the volume of the hypersphere is concentrated on the outermost shell. Therefore, it is only necessary to limit the thickness of the shell to one-hundredth of the radius of the hypersphere, which can significantly improve the utilisation of the feature space for the sample representation. To achieve this, we subtract a random variable $\mathbf{\text{d}}\sim (\mathit{\mu },\mathbf{\Sigma })$ obeying a Gaussian distribution from the “label” representation (that is, the representation of the dynamic vocabulary) after L2 regularisation, where $\mathbf{\text{d}}\sim {\mathbb{R}}^D$, $\mathit{\mu }=[0.005,0.005,\cdots ,0.005{]}_D^T$, and $\mathbf{\Sigma }$ is a diagonal matrix $\mathbf{\Sigma }=diag(0.005/3)$. This paper adopts the $3\sigma$ rule commonly used in the industry, which can ensure that most of the variables of this distribution (about 99.74%) fall within the $3\sigma$ interval near the mean. In summary, for the representation ${\mathbf{\text{e}}}_i=d_{\mathit{\xi }}(w_i)$ of any token in the dynamic vocabulary after L2 regularisation, the final representation with Gaussian perturbation is: (21) ${\widetilde{\mathbf{\text{e}}}}_i={\mathbf{\text{e}}}_i-\mathbf{\text{d}},\mathbf{\text{d}}\sim (\mathit{\mu },\mathbf{\Sigma })$(21)

4.6 Ablation study

In order to comprehensively test the superiority of the method proposed, we will conduct a series of ablation experiments in this section to verify the effectiveness of each component of MoCoUTRL. Specifically, we first conducted experiments on the backend of MoCoUTRL’s Encoder, comparing BERT, RoBERTa, and DeBERTa. Finally, we selected DeBERTa as the ultimate backend model. Then, we analysed the effects of two types of contrastive losses at different granularities. In addition, L2 regularisation with Gaussian perturbation is also a component that is believed to have a significant effect in this work, and we also analysed its effectiveness. We conduct all our ablation studies on the BookCorpus dataset.

The experimental data in Table 3 is composed of the best results obtained from selecting five independent repeated experiments. From the experimental results, it is not difficult to see that DeBERTa and RoBERTa can achieve significant performance advantages as backends compared to BERT. This is also why this article chooses the DeBERTa model as the backend model for MoCoUTRL. Regarding the two semantic granularity losses, repeated experiments have shown that the role of the token-level contrastive loss is significantly smaller than that of the long-span contrastive loss. That is to say, the performance improvement of the model mainly comes from the overall understanding level of a text sequence rather than the semantic difference of a single token. Intuitively, language diversity means that two text segments with similar semantics may not be the same length. Two text segments with opposite semantics may not need to be implemented using antonyms and negating predicate or auxiliary verbs. This diversity may be why the token-level contrastive loss cannot dominate and can only be used as an auxiliary training loss. We conducted multiple experiments on L2 Norm with Gaussian Perturbation and found that this operation can bring MoCoUTRL a relatively small but stable performance improvement. We guess that through sufficient training, MoCoUTRL has been able to output representations that can meet the two properties of alignment and uniformity and can basically meet the distribution on the hypersphere with a radius of 1. The role of Formula (21) is only to make it more in line with the ideal distribution in theory.

Table 3. Ablation study of the proposed MoCoUTRL.

Model		Accuracy	F1
MoCoUTRL (DeBERTa)		85.71	85.89
MoCoUTRL (RoBERTa)		85.54	86.15
MoCoUTRL (BERT)		81.27	81.94
	–long span contrastive loss	79.14	80.15
	–token-level contrastive loss	83.36	83.41
	–L2 Norm with Gaussian Perturbation	84.54	84.87

5 Conclusion

Although previous studies have fully proved large-scale PTMs built upon Transformer to have strong semantic capture ability and generalisation ability in multiple downstream applications, the growing computing requirements and high training costs make it impossible for any research institution or individual to train PTMs with thousands of billions of parameters. Moreover, if PTMs are regarded as feature encoders, the representations generated by a single PTM cannot achieve performance matching its parameter size and training costs. Therefore, this paper introduces contrastive learning based on PTM. Through the construction of target losses at the sample level and token level and a dynamic vocabulary updated by momentum and used as the token’s ground-truth representation, MoCoUTRL’s comparative experiments on STS and WOS datasets have achieved achievements that match or even exceed the current cutting-edge methods. The quantitative analysis of the two attributes of alignment and uniformity of the representation output by MoCoUTRL also proves the validity of MoCoUTRL theoretically.

Several recent studies on large language models (LLMs) have been published and attracted widespread attention, and they have been the de facto new research paradigm in NLP with their impressive performance on many complex downstream tasks. Although these LLMs are capable of many tasks, they still have areas where they are not good. For example, in information retrieval, even the GPT-4 model, which is currently in the absolute top tier of performance, still cannot achieve more accurate and effective performance than traditional search engines. the contrastive learning framework proposed by MoCoUTRL can better exploit the potential of LLMs in retrieval. By training the text representations output by LLM for comparative learning, a semantic index of the documents to be retrieved can be constructed containing deep semantic information, and then document retrieval based on deep semantic information can be achieved with only the most naive similarity calculation. At present, Microsoft Bing’s ChatGPT interface still requires the use of traditional search engines, which also confirms that LLM is still difficult to perform in the face of large-scale information retrieval tasks, and if the MoCoUTRL framework can achieve high-quality deep-level semantic retrieval, it may bring a new breakthrough to the current industrial information retrieval industry.

Whether in academia or industry, training large-scale pre-trained language models from scratch is costly. However, directly applying pre-trained models open-sourced by other institutions or companies to specific business scenarios may encounter problems such as complex convergence and uneven data distribution. The training framework of MoCoUTRL can alleviate this problem to some extent by constructing MoCoUTRL’s contrastive learning task on data within the field, enabling the model to quickly adapt to the data distribution within the application field and generate higher-quality text representations, thus assisting in the specific deployment of downstream applications.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Defense Industrial Technology Development Program: [Grant Number JCKY2020601B018].