MoCoUTRL: a momentum contrastive framework for unsupervised text representation learning

Figures & data

Figure 1. The overall structure of MoCoUTRL.

Table

Algorithm 1: MoCoUTRL algorithm.
Input: training corpus $\mathcal{C}=\{x_i{\}}_1^N$, encoder $f_{\mathit{\theta }}(\cdot )$, dynamic dictionary $d_{\mathit{\xi }}(\cdot )$, nonlinear projector of latent semantic space ${p^{SLS}}_{\mathit{\eta }}(\cdot )$ and token latent space ${p^{TLS}}_{\nu }(\cdot )$, hyperparameter num_epochs, batch_size, initial learning rate ($lr$), $m\in [0,1)$, and $\lambda \in [0,1]$
Output: encoder $f_{\mathit{\theta }}(\cdot )$, discarding $d_{\mathit{\xi }}(\cdot )$, ${p^{SLS}}_{\mathit{\eta }}(\cdot )$, and ${p^{TLS}}_{\nu }(\cdot )$
Initialisation: $f_{\mathit{\theta }}(\cdot )$ is initialised by PTM (e.g. DeBERTa), $d_{\mathit{\xi }}(\cdot )$ is initialised by the embedding layer of PTM (e.g. DeBERTa), ${p^{SLS}}_{\mathit{\eta }}(\cdot )$, and ${p^{TLS}}_{\nu }(\cdot )$ are initialised by Glorot initialisation
for i_epoch = 1 to num_epochs do
for i_batch = 1 to num_batchs do
fetch data $x^{ori}$, $x^{syn}$, $x^{ant}$, $w_i^{ori}$, $w_i^{syn}$, and $w_i^{ant}$
${\mathbf{\text{x}}}^{ori}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{ori})),{\mathbf{\text{x}}}^{syn}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{syn})),{\mathbf{\text{x}}}^{ant}={p^{SLS}}_{\mathit{\eta }}(f_{\mathit{\theta }}(x^{ant}))$
${\mathcal{L}}_{longspan}=-\log \frac{exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_i^{+}/\tau )}{{\sum }_{i=1}^{N}exp({\mathbf{\text{z}}}_i\cdot {\mathbf{\text{z}}}_i^{-}/\tau )}$
${\mathbf{\text{e}}}_i^{ori}=d_{\mathit{\xi }}(w_i^{ori}),{\mathbf{\text{w}}}_i^{syn}={p^{TLS}}_{\nu }(f_{\mathit{\theta }}(w_i^{syn})),{\mathbf{\text{w}}}_i^{ant}={p^{TLS}}_{\nu }(f_{\mathit{\theta }}(w_i^{ant}))$
${\mathcal{L}}_{token-level}=\frac{1}{N}{\sum }_{i=1}^N[MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{syn})-MSELoss({\mathbf{\text{e}}}_i^{ori},{\mathbf{\text{w}}}_i^{ant})]$
$\mathcal{L}=\lambda \cdot {\mathcal{L}}_{longspan}+(1-\lambda )\cdot {\mathcal{L}}_{token-level}$
${\theta }_{t+1}\leftarrow {\theta }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\theta }_t},{\eta }_{t+1}\leftarrow {\eta }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\eta }_t},{\nu }_{t+1}\leftarrow {\nu }_t-lr\cdot \frac{\delta \mathcal{L}}{\delta {\nu }_t}$
${\xi }_{t+1}\leftarrow m{\xi }_t+(m-1){\phi }_t$ ($\phi$ are parameters of the embedding layer of $f_{\mathit{\theta }}(\cdot )$)
update $lr$ by schedule
End for
End for

Table 1. Performance on STS tasks.

Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK
Unsupervised Methods
GloVe(avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76
BERT-base(last2 avg.)	41.72	60.43	50.12	65.27	67.21	51.46	62.13
BERT-base(first-last avg.)	39.70	59.38	49.67	66.03	66.19	53.87	62.06
BERT-base-flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47
BERT-whitening	57.83	66.90	60.90	75.08	71.31	68.24	63.73
CT-BERT-base	61.63	76.80	68.47	77.50	76.48	74.31	69.19
SimCSE-BERT-base	68.40	82.41	74.38	80.91	78.56	76.85	72.23
RoBERTa-base(last2 avg.)	41.65	62.31	52.43	66.75	67.51	58.31	60.43
RoBERTa-base(first-last avg.)	40.88	58.74	49.07	65.63	61.48	58.55	61.63
SimCSE-RoBERTa-base	70.16	81.77	73.24	81.36	80.65	80.22	68.56
DiffCSE-BERT-base	72.28	84.43	76.47	83.90	80.54	80.59	71.23
DiffCSE-RoBERTa-base	70.05	83.43	75.49	82.81	82.12	82.38	71.19
MoCoUTRL	73.21	78.65	75.76	83.40	82.17	82.12	71.17
Supervised Methods
InferSent-GloVe	52.86	66.75	62.15	72.77	66.87	68.03	65.65
SentenceBERT-base	70.97	76.53	73.19	79.09	74.30	77.03	72.91
SentenceBERT-base-flow	69.78	77.27	74.35	82.01	77.46	79.12	76.21
SentenceBERT-base-whitening	69.65	77.57	74.66	82.27	78.39	79.52	76.91
CT-SentenceBERT-base	74.84	83.20	78.07	83.84	77.93	81.46	76.42
SimCSE-BERT-base	75.30	84.67	80.19	85.40	80.82	84.25	80.39
SentenceRoBERTa-base	71.54	72.49	70.80	78.74	73.69	77.77	74.46
SentenceRoBERTa-base-whitening	70.46	77.07	74.46	81.64	76.43	79.49	76.65
SimCSE-RoBERTa-base	76.53	85.21	80.95	86.03	82.57	86.70	81.95
MoCoUTRL + MLP	76.91	85.87	80.31	85.75	83.58	88.31	81.43

Table 2. Performance on the WOS dataset.

Model	Acc	R	P	F1
BERT-base	81.27	80.42	81.94	82.17
RoBERTa-base	85.54	85.36	86.15	85.75
DeBERTa-base	85.71	85.64	85.89	85.61
MoCoUTRL + MLP	88.13	87.79	85.83	86.80

Figure 2. Analyses on alignment and uniformity.

Figure 3. The sparsity of high-dimensional space.

Table 3. Ablation study of the proposed MoCoUTRL.

Model		Accuracy	F1
MoCoUTRL (DeBERTa)		85.71	85.89
MoCoUTRL (RoBERTa)		85.54	86.15
MoCoUTRL (BERT)		81.27	81.94
	–long span contrastive loss	79.14	80.15
	–token-level contrastive loss	83.36	83.41
	–L2 Norm with Gaussian Perturbation	84.54	84.87