The detection of distributional discrepancy for language GANs

ABSTRACT

A pre-trained neural language model (LM) is usually used to generate texts. Due to exposure bias, the generated text is not as good as real text. Many researchers claimed they employed the Generative Adversarial Nets (GAN) to alleviate this issue by feeding reward signals from a discriminator to update the LM (generator). However, some researchers argued that GAN did not work by evaluating the generated texts with a quality-diversity metric such as Bleu versus self-Bleu, and language model score versus reverse language model score. Unfortunately, these two-dimension metrics are not reliable. Furthermore, the existing methods only assessed the final generated texts, thus neglecting the dynamic evaluating the adversarial learning process. Different from the above-mentioned methods, we adopted the most recent metric functions, which measure the distributional discrepancy between real and generated text. Besides that, we design a comprehensive experiment to investigate the performance during the learning process. First, we evaluate a language model with two functions and identify a large discrepancy. Then, several methods with the detected discrepancy signal to improve the generator were tried. Experimenting with two language GANs on two benchmark datasets, we found that the distributional discrepancy increases with more adversarial learning rounds. Our research provides convicted evidence that the language GANs fail.

KEYWORDS:

• Text generation
• generative adversarial nets
• distributional discrepancy

1. Introduction

Text generation based on neural language models (LM) (e.g. LSTM Hochreiter & Schmidhuber, 1997) has received much attention and has been used for news generation (Zellers et al., 2019), text summarisation (Lin et al., 2022) and image captioning (Xu et al., 2015). However, the generated sentences are still of low quality with regards to semantics and global coherence and are often imperfect grammatically speaking (Caccia et al., 2020).

These issues give rise to a large discrepancy between generated text and real text. Two underlying reasons are the architecture and the number of parameters of the LM itself (Radford et al., 2019; Santoro et al., 2018). Many researchers attribute this to exposure bias (Bengio et al., 2015) because an LM is trained with a maximum likelihood estimate (MLE) and predicts the next word conditioned on words from the ground truth during training. However, an LM only uses words that it has generated during the reference.

Statistically, this discrepancy means the two distributional functions of real texts and generated texts are different. Reducing this distributional difference may be a practicable way to improve text generation.

Some researchers try to reduce this difference with GAN (Goodfellow et al., 2014) as the success in image generation (Wu et al., 2021), image classification (Cao et al., 2021) and stock prediction (Li et al., 2022; Wu et al., 2022). They used a discriminator to detect the discrepancy between real samples and generated samples, and fed the signal back to upgrade the generator (an LM). To solve the non-differential issue that arises by the need to handle discrete tokens, reinforcement learning (RL) (Williams, 1992) was adopted by SeqGAN (Yu et al., 2017), RankGAN (Lin et al., 2017), and LeakGAN (Guo et al., 2018). The Gumbel-Softmax is also introduced by GSGAN (Jang et al., 2017) and RelGAN (Nie et al., 2019) to solve this issue. These language GANs pre-train both the generator (G) and the discriminator (D) before adversarial learning.¹ During adversarial learning, for each round, the G is trained several epochs and the D is trained tens of epochs. The learning process will not stop until the model converges. Furthermore, considering the generated texts' quality and diversity simultaneously (Shi et al., 2018), MaskGAN (Fedus et al., 2018), DpGAN (Xu et al., 2018) and FMGAN (Chen et al., 2018) are proposed. They evaluate the generated text with Bleu (Papineni et al., 2002) versus self-Bleu (Zhu et al., 2018) or LM score versus reverse LM score (Cífka et al., 2018), and claim these GANs improve the performance of the generator.

However, some questions have been recently raised over these claims. Semeniuta et al. (2019) and Caccia et al. (2020) showed that by more precise experiments and evaluation, these considered GAN variants are outperformed by a well-adjusted language model. They draw a performance line in a quality-diversity space by adjusting the softmax temperature. Bleu and language model scores are usually used for measuring local and global quality, respectively; self-Bleu and reverse language model scores are used for measuring local and global diversity, respectively. To overcome these two-dimension metrics, de Masson et al. (2019) proposed a single metric Fréchet embedding distance (FED). It computes the Fréchet distance between two Gaussian distributions. However, Cai et al. (2021) showed that all metrics are not appropriate to evaluate an un-conditional text generator and proposed a novel metric. In short, whether these language GANs fail or not is still an open problem.

We investigate this issue in depth. For language GANs, several critical issues are still not clear, such as whether D detects the discrepancy, whether the detected discrepancy is severe, and whether the signals from D can improve the generator. In this paper, we try to solve these problems by investigating GAN in both pre-training and the adversarial learning processes. Theoretically analysing the signal from D, we employ approximate discrepancy and absolute discrepancy (Cai et al., 2021) to measure the distributional discrepancy. With these two functions, we first measure the discrepancy between the real text and the faked text, which is generated by an MLE-trained language model (pre-train). Second, we attempt some methods to update the generator with a feedback signal from D. Then, we use these metric functions to evaluate the updated generator. Finally, we analyse the performance of two typical language GANs during adversarial learning with these two functions across two benchmark datasets.

Our contributions are as follows:

• We are the first to measure the variation of the distributional discrepancy between real text and generated text during the training of language GANs by using a discriminator to design and implement two equations.

• Although this discrepancy could be detected by a discriminator (D), the feedback signal from D cannot improve G using existing methods. This manifests as an increase in the discrepancy with adversarial learning.

• Experimenting on two existing language GANs, SeqGAN and RelGAN, the distributional discrepancy between real text and generated text increases with more adversarial learning rounds. This demonstrates that existing adversarial learning does not work. Thus, the industrial systems need not try in this way.

The rest of the paper is organised as follows. Section 2 describes the related work. Section 3 introduces the proposed method to measure the distributional discrepancy. The next section presents the experimental procedure in detail. The experiments and analysis are shown in Section 5. Finally, we give a short summary in Section 6.

2. Related work

Many GAN-based models were proposed to improve neural language models. SeqGAN (Yu et al., 2017) attacked the non-differential issue by resorting to RL. By applying a policy gradient method (Sutton et al., 2000), they optimised the LSTM generator with rewards received through Monte Carlo (MC) sampling. Many researchers such as RankGAN and MailGAN (Che et al., 2017) also used this technique, although it is ineffective in MC search. The RL-free model, e.g. GSGAN, contained continuous application of the approximating softmax function and working on latent continuous space directly. TextGAN (Salimans et al., 2016) added Maximum Mean Discrepancy to the original objective of GAN based on feature matching. Considering the drawbacks of pre-training a neural language model, Nie et al. (2019) proposed a RelGAN that uses the relation memory (Santoro et al., 2018), which allows for interactions between memory slots by using the self-attention mechanism (Vaswani et al., 2017). Gu & Cheung (2018) optimised GAN by evolutionary algorithms. We selected SeqGAN and RelGAN as representatives for this study. The results show that the adversarial learning does not work for either of these models.

Caccia et al. (2020) argued the current evaluation measures correlate with human judgment (Cífka et al., 2018) was treacherous. They furthermore proposed a temperature sweep, which evaluates model at many temperature settings rather than only one. By drawing lines in a quality-diversity space such as Bleu versus self-Bleu, or language model score versus reverse language model score, they show that a well-adjusted language model can beat those considered language GANs. Unfortunately, the limitations of Bleu vs self-Bleu were shown by training a 5-gram language model and its scores were even better than the training data (de Masson et al., 2019). Cai et al. (2021) also revealed that they were unreliable and proposed a novel metric to evaluate unconditional text generation by calculating the distributional discrepancy between two text sets. This single metric could simultaneously measure both the diversity and quality. We adopt it and propose a simpler version. Semeniuta et al. (2019) and He et al. (2021) also argued GAN-based models were weaker than LM, because they observed a less severe impact of exposure bias. The latter furtherly quantified the exposure bias by using conditional distribution. Obviously, the existing methods only assessed the final generated texts, thus neglecting the dynamic evaluating the adversarial learning processes. Different from the above-mentioned methods, we investigate the mechanism of language GANs and quantify the discrepancy between real texts and generated texts both after the pre-training and the whole adversarial learning process.

3. Method

In GAN, the generator $G_{\theta }$ implicitly defines a probability distribution $p_{\theta }\left(x\right)$ to mimic the real data distribution $p_d\left(x\right)$. θ is the parameters of the language model $G_{\theta }$ and is the parameter of the value function V, which is listed as follow. $p_{\theta }$ is the distributional function of $G_{\theta }$. (1) $\underset{G_{\theta }}{min}\underset{D_{\varphi }}{max}V(D_{\varphi },G_{\theta })={\mathbb{E}}_{p_d\left(x\right)}\left[\log D_{\varphi }\left(x\right)\right]+{\mathbb{E}}_{p_{\theta }\left(x\right)}\left[\log (1-D_{\varphi }(x\left)\right)\right]$(1) Alternating optimisation of $G_{\theta }$ and $D_{\varphi }$ is used to resolve the above equation. Given θ, to detect the discrepancy between $p_{\theta }\left(x\right)$ and $p_d\left(x\right)$, we optimise $D_{\varphi }$ as follows: (2) $\underset{D_{\varphi }}{max}V(D_{\varphi },G_{\theta })=\underset{D_{\varphi }}{max}{\mathbb{E}}_{p_d\left(x\right)}\left[\log D_{\varphi }\left(x\right)\right]+{\mathbb{E}}_{p_{\theta }\left(x\right)}\left[\log (1-D_{\varphi }(x\left)\right)\right]$(2) Assuming $D_{\varphi }^{\ast }\left(x\right)$ is the optimal solution for a given θ, according to Goodfellow et al. (2014), it will be, (3) $D_{\varphi }^{\ast }\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(3) and it is obvious to get the follow formula: (4) $\{\begin{array}{l}{D}_{\varphi }^{\ast }\left(x\right)\ge 0.5,\mathrm{i}\mathrm{i}\mathrm{f}{p}_d\left(x\right)\ge p_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)<0.5,\mathrm{i}\mathrm{i}\mathrm{f}{p}_d\left(x\right)<p_{\theta }\left(x\right)\end{array}$(4) Because the real distribution $p_d$ cannot be obtained in practice, it is impossible to directly measure the discrepancy according to Equation (3(3) $D_{\varphi }^{\ast }\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(3) ). Fortunately, we have massive real sentences and each x could be a sample from $p_d$. Based on real samples and the above equation, we obtain a way to estimate the distributional discrepancy.

3.1. Approximate discrepancy

Let, (5) $q_d\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)},q_{\theta }\left(x\right)=\frac{p_{\theta }\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(5) Therefore, $q_d\left(x\right)=p\left(x\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{s}\mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}|x\right)$, $q_{\theta }\left(x\right)=p\left(x\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{s}\mathrm{f}\mathrm{r}\mathrm{o}\mathrm{m}\mathrm{g}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}|x\right)$, $q_d\left(x\right)+q_{\theta }\left(x\right)=1$. With Equation (5(5) $q_d\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)},q_{\theta }\left(x\right)=\frac{p_{\theta }\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(5) ), we can get a constraint and an approximate measure of distributional function. Figure 1(a) illustrates the relationship between $q_{\theta }\left(x\right)$ and $q_d\left(x\right)$.

Figure 1. Illustration of two measures. (a) The yellow area denotes the negative, and the green one denotes the positive. (b) Half of the shaded area equals the result of Equation (10(10) $d_s=\frac{1}{2}\left[{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}\right(1)-{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)+{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)-{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}(1\left)\right]$(10) ). (a) Approximate discrepancy. (b) Absolute discrepancy.

Let, (6) $u_d={\mathbb{E}}_{p_d\left(x\right)}\left(D_{\varphi }^{\ast }\right(x\left)\right),u_{\theta }={\mathbb{E}}_{p_{\theta }\left(x\right)}\left(D_{\varphi }^{\ast }\right(x\left)\right)$(6) These are two equations for these two statistics, which are the expectation of the $D_{\varphi }^{\ast }$'s predictions on real text and on generated text. From the above equation, it is easy to obtain the following equation: (7) $\frac{1}{2}[u_d+u_{\theta }]=0.5$(7) The results give a constraint for $D_{\varphi }$ converging to $D_{\varphi }^{\ast }$. We should take this constraint into account when estimating the ideal function $D_{\varphi }^{\ast }$. From Equation (3(3) $D_{\varphi }^{\ast }\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(3) ), the optimisation process for the discriminator increases $u_d$ and decreases $u_{\theta }$ to as small a value as possible. So, we can estimate the distributional discrepancy according to the following function.

Intuitively, using $u_d$ and $u_{\theta }$, we get a metric function to measure the discrepancy between $p_{\theta }\left(x\right)$ and $p_d\left(x\right)$, (8) $d_a=|u_d-u_{\theta }|$(8) We call this approximate discrepancy. It is the difference in the average score that a well-trained discriminator (denoted as ${\hat{D}}_{\varphi }$) makes in the predictions on real samples compared to generated samples. It reflects the discrepancy between these two sets to some degree. From Equations (5(5) $q_d\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)},q_{\theta }\left(x\right)=\frac{p_{\theta }\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(5) ), (6(6) $u_d={\mathbb{E}}_{p_d\left(x\right)}\left(D_{\varphi }^{\ast }\right(x\left)\right),u_{\theta }={\mathbb{E}}_{p_{\theta }\left(x\right)}\left(D_{\varphi }^{\ast }\right(x\left)\right)$(6) ) and (8(8) $d_a=|u_d-u_{\theta }|$(8) ), we get Equation (9(9) $d_a=\int \left|q_d\right(x)-q_{\theta }(x\left)|p_d\right(x)\mathrm{d}x={\mathbb{E}}_{p_d\left(x\right)}|q_d\left(x\right)-q_{\theta }\left(x\right)|$(9) ), (9) $d_a=\int \left|q_d\right(x)-q_{\theta }(x\left)|p_d\right(x)\mathrm{d}x={\mathbb{E}}_{p_d\left(x\right)}|q_d\left(x\right)-q_{\theta }\left(x\right)|$(9) The range of $d_a$ is $0\sim 1$. The bigger its value is, the larger the discrepancy. When $p_d\left(x\right)=p_{\theta }\left(x\right)$, namely there is no discrepancy, there will be $d_a=0$. On the contrary, $d_a=1$ if $\mathrm{\forall }x,p_d\left(x\right)\ast p_{\theta }\left(x\right)\equiv 0$. Figure 1(a) illustrates the discrepancy between two distributional functions $q_{\theta }\left(x\right)$ and $q_d\left(x\right)$. Both of them are systematic to the line of q = 0.5.

Cai et al. (2021) proposed a novel metric that is more complete than ours because there are a positive part and also a negative part, as presented in Figure 1(b), and it is defined as absolute discrepancy $d_s$. This is represented by the following equation: (10) $d_s=\frac{1}{2}\left[{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}\right(1)-{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)+{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)-{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}(1\left)\right]$(10) The range of $d_s$ is also $0\sim 1$. The drawback of this metric is that it needs more computation than ours. Both metrics are used in this paper.

3.2. Using $D_{\varphi }^{\ast }\left(x\right)$ to improve $G_{\theta }$

Given an instance x generated by $G_{\theta }$, if $D_{\varphi }^{\ast }\left(x\right)$ is larger, it means the possibility of x in real data is larger. For an instance x, if $D_{\varphi }^{\ast }\left(x\right)=0.8$, there will be $p_{\theta }\left(x\right)<p_d\left(x\right)$ according to Equation (3(3) $D_{\varphi }^{\ast }\left(x\right)=\frac{p_d\left(x\right)}{p_d\left(x\right)+p_{\theta }\left(x\right)}$(3) ). So, we should update $G_{\theta }$ to increase the probability density $p_{\theta }\left(x\right)$. It may improve the performance of $G_{\theta }$. Based on this, we can select some generated instances by the value of $D_{\varphi }^{\ast }\left(x\right)$ to update the generator. In fact, we find it improves the performance a little when compared with the random selection. However, this method is still worse than without it. Experiment 5.3 shows the results.

4. Implementation procedure

The optimal function $D_{\varphi }^{\ast }$ is an ideal function that can only be statistically estimated by an approximated function. We can design a function $D_{\varphi }$ and sample from real data and generated data; then, we train $D_{\varphi }$ according to Equation (2(2) $\underset{D_{\varphi }}{max}V(D_{\varphi },G_{\theta })=\underset{D_{\varphi }}{max}{\mathbb{E}}_{p_d\left(x\right)}\left[\log D_{\varphi }\left(x\right)\right]+{\mathbb{E}}_{p_{\theta }\left(x\right)}\left[\log (1-D_{\varphi }(x\left)\right)\right]$(2) ). When the results convergence, we get ${\hat{D}}_{\varphi }$, which is the approximated function of $D_{\varphi }^{\ast }$. The degree of approximation is mainly determined by three factors: the structure and the number of the parameters number of $D_{\varphi }$, the volume of training data, and the settings of hyper-parameters.

Based on the above analysis, we obtain two metric functions to measure the distributional discrepancy between dataset A and B (for example, A is composed of real sentences while B consists of machine-generated sentences). The implementation procedure is described as follows:

Step 1: Design a discriminator $D_{\varphi }$.

Step 2: Sets A and B are, respectively, divided into a training set $D_{trainA}$ and $D_{trainB}$, a validation set $D_{devA}$ and $D_{devB}$, and a test set $D_{testA}$ and $D_{testB}$. The partition should be as equal an amount of instances as possible for classification training.

Step 3: $D_{\varphi }$ is optimised with $D_{trainA}$ and $D_{trainB}$ according to the Equation (2(2) $\underset{D_{\varphi }}{max}V(D_{\varphi },G_{\theta })=\underset{D_{\varphi }}{max}{\mathbb{E}}_{p_d\left(x\right)}\left[\log D_{\varphi }\left(x\right)\right]+{\mathbb{E}}_{p_{\theta }\left(x\right)}\left[\log (1-D_{\varphi }(x\left)\right)\right]$(2) ). Validating with $D_{devA}$ and $D_{devB}$, we can judge whether $D_{\varphi }$ convergences or not and then get ${\hat{D}}_{\varphi }$.

Step 4: According to Equation (8(8) $d_a=|u_d-u_{\theta }|$(8) ) and (10(10) $d_s=\frac{1}{2}\left[{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}\right(1)-{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)+{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)-{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}(1\left)\right]$(10) ), with two test datasets, we can estimate the discrepancy of two distributional functions between dataset A and B. ${\hat{d}}_s$ denotes the absolute discrepancy, and ${\hat{d}}_a$ denotes the approximate discrepancy, respectively.

Algorithm 1 illustrates the procedure. Generally speaking, there should be $d_s\le {\hat{d}}_s$. Because $D_{\varphi }^{\ast }$ cannot be obtained, it is hard to get the degree of the approximation of $d_s$ to ${\hat{d}}_s$. Many research results have shown that discriminators with deep neural networks are very powerful, some of which can even exceed human performance on tasks such as image classification (He et al., 2016) and text classification (Kim, 2014). So, if $D_{\varphi }$ with CNN, and an attention mechanism is well trained, ${\hat{D}}_{\varphi }$ will be a meaningful approximation of $D_{\varphi }^{\ast }$. Therefore, we can obtain the meaningful approximation of $d_s$ and $d_a$ via ${\hat{D}}_{\varphi }$.

5. Experiment

We select SeqGAN and RelGAN as representative models for our experiment, and the benchmark datasets are also the same as used by these models previously. Then, we show that the well-trained discriminator $D_{\varphi }$ can measure the discrepancy between the real and generated texts, and then point out that the existing GAN-based methods does not work. Finally, a third-party discriminator is used to evaluate the performance of adversarial learning with incremental training iterations.

5.1. Datasets and model settings

Both SeqGAN and RelGAN used a relatively short sentences dataset (COCO image caption)² and a long sentences dataset (EMNLP2017 WMT news).³ For the former dataset, the sentences' average length is about 11 words. There are, in total 4682-word types, and the longest sentence consists of 37 words. Both the training and test data contain 10,000 sentences. For the latter dataset, the average length of sentences is about 20 words. There are in total 5255-word types and the longest sentence consists of 51 words. All training data, about 280 thousand sentences, is used and there are 10,000 sentences in the test data. According to Section 3, each test data is divided into two parts. Half is the validation set and the remaining half is the test set. We always generate the same number of sentences to compare with the two test datasets, respectively.

For these two models, all hyper-parameters, including word embedding size, learning rate and dropout, are set the same as in their original papers. For RelGAN, the standard GAN loss function (the non-saturating version) is adopted because the relative standard loss which is used in (Nie et al., 2019) does not meet the constraints of Equation (7(7) $\frac{1}{2}[u_d+u_{\theta }]=0.5$(7) ). But, when measuring RelGAN's discrepancy during the adversarial stage, its own loss function is still a relatively standard loss. A critical hyper-parameter, temperature, is set to 100, which is the best result in their paper. During the process of training $D_{\varphi }$, we always train 10,000 epochs and observe performance on the validation dataset.

5.2. Distributional differences in pre-training

We estimate the distributional differences caused by the MLE-based generators. We first train the generator for N epochs and then $D_{\varphi }$ until it converges (this needs 10,000 epochs). For example, following Nie et al. (2019), we train $G_{\theta }$ for 150 epochs, and select the one whose perplexity (PPL) is the smallest by measuring on the validation set. Then, $D_{\varphi }$ is trained following the procedure in Section 4.

Figure 2 shows the discrepancy between real and generated texts. The discrepancy increases with more training for discriminator until the training is stable.

Figure 2. Discrepancy between real and generated texts with two pre-training generators after 80 epochs across two datasets. Red lines denote absolute discrepancy, and blue ones mean approximate discrepancy. Pale lines denote batch instances' discrepancy, and the curve is the exponential moving average on this sampled batch for each epoch. (a) Discrepancy of SeqGAN on EMNLP. (b) Discrepancy of SeqGAN on COCO. (c) Discrepancy of RelGAN on EMNLP. (d) Discrepancy of RelGAN on COCO.

Figure 3 shows the discriminator's prediction on real and machine-generated texts, respectively. The more difference between the scores on these two datasets, the distributional discrepancy is larger. From this figure, we can see $D_{\varphi }$ convergences after about 3000 epochs for RelGAN, but SeqGAN needs more epochs to train the discriminator because the latter used an LSTM as the generator.

Figure 3. Discriminator scores on real and generated texts with pre-training two generators after 80 epochs across two datasets. Red lines denote the score on the real validation set, and blue ones mean the score on the machine-generated validation set. Pale lines denote batch instances' discrepancy, and the curve is the exponential moving average on this sampled batch for each epoch. (a) Accuracy with SeqGAN on EMNLP. (b) Accuracy with SeqGAN on COCO. (c) Accuracy with RelGAN on EMNLP. (d) accuracy with RelGAN on COCO.

Considering the smoothed value on one batch rather than the prediction on the whole data, we use the convergence discriminator to predict on the all validation data and generated data.⁴ Table 1 summarises the discrepancy across two models and two datasets. It shows that the difference between real text and generated text does exist and it is huge.

Table 1. Discrepancy across two models and two datasets after pre-training.

Model	Dataset	${\hat{d}}_s$	${\hat{d}}_a$	Accuracy
SeqGAN	COCO	0.42	0.44	0.71
	EMNLP	0.57	0.47	0.78
RelGAN	COCO	0.64	0.45	0.82
	EMNLP	0.52	0.31	0.76

For both $d_a$ and $d_s$, lower is better.

5.3. Detected discrepancy by ${\hat{D}}_{\varphi }$ cannot improve the generator

We explore the improvement of $G_{\theta }$ with the discrepancy detected by ${\hat{D}}_{\varphi }$ at the end of pre-training. We select the best pre-train epochs for $G_{\theta }$. It should be noted that ${\hat{D}}_{\varphi }$ is well-trained with sufficient real sentences and generated ones by $G_{\theta }$. Then, $G_{\theta }$ is updated according to the signals from the ${\hat{D}}_{\varphi }$. To verify the effect of the feedback signals, we generate many rather than only several batch-size instances to adjust θ.

Then, fixing $G_{\theta }$, we re-train $D_{\varphi }$ with 10,000 epochs to get a new convergence discriminator to compute two distributional functions according to Equations (9(9) $d_a=\int \left|q_d\right(x)-q_{\theta }(x\left)|p_d\right(x)\mathrm{d}x={\mathbb{E}}_{p_d\left(x\right)}|q_d\left(x\right)-q_{\theta }\left(x\right)|$(9) ) and (10(10) $d_s=\frac{1}{2}\left[{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}\right(1)-{\mathbb{E}}_{\begin{array}{c}{p}_d\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)+{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)\le 0.5\end{array}}(1)-{\mathbb{E}}_{\begin{array}{c}{p}_{\theta }\left(x\right)\\ D_{\varphi }^{\ast }\left(x\right)>0.5\end{array}}(1\left)\right]$(10) ). Unfortunately, in the view of both absolute and approximated discrepancy, the discrepancy always exceeds the original value computed in pre-training. It demonstrates that the generator is not improved further. Figure 4 illustrates the comparison.

Figure 4. The comparison of the discrepancy between pre-train and the generator is updated with the feedback signals from ${\hat{D}}_{\varphi }$, which is obtained from pre-train. The vertical dash line represents the end of pre-training. (a) Approximate discrepancy on EMNLP. (b) Absolute discrepancy on EMNLP. (c) Approximate discrepancy on COCO. (d) Absolute discrepancy on COCO.

Besides following Zhu et al. (2018), we also propose a new method to update $G_{\theta }$ in the adversarial way. Rather than using all the generated instances to update $G_{\theta }$, only the ones, which are assigned relatively high scores by $D_{\varphi }$, are used. We denote it as HW. The reason is that we assume that the higher score instances may be more informative than the lower ones. The method that only the relatively low scores samples are used to adjust the generator is also experimented with. Regretfully, all of them fail. Table 2 lists the discrepancy across two datasets with different settings. The discrepancy is always larger than that of the pre-training.

Table 2. Comparison between the absolute discrepancy in pre-training and $G_{\theta }$ updated by ${\hat{D}}_{\varphi }$'s feedback signal.

Dataset	COCO					EMNLP
Pre-train	0.42					0.57
#Samples	0.1S	0.5S	1S	2S	5S	0.1S	0.5S	1S	2S	5S
Random	0.52	0.45	0.54	0.53	0.55	0.68	0.68	0.67	0.67	0.67
<0.3	0.60	0.58	0.54	0.54	0.72	0.77	0.76	0.73	0.76	0.73
0.5–0.9	0.52	0.51	0.47	0.54	0.58	0.66	0.57	0.60	0.64	0.62
≥0.9	0.68	0.46	0.50	0.44	0.58	0.66	0.60	0.59	0.60	0.62

Notes: It should be noted that the lower discrepancy is better. #samples denotes the amount of the generated data is used for updating the $G_{\theta }$. For example, 2S means the generated instances are two times the amount of training data. Random denotes the existing way, but the other rows are the results according to HW. 0.3–0.5 means the generated instances whose scores are between 0.3 and 0.5 assigned by $D_{\varphi }$ are selected out.

5.4. A third-party discriminator evaluates these language GANs

In order to evaluate different adversarial learning's GANs, we use a third-party discriminator $D_{\varphi }^3$, which is a clone of the discriminator in its counterpart language GAN except for the parameters' values. For each adversarial round, we train $D_{\varphi }^3$ from scratch many epochs (verifying its convergence) with real text and generated text. Then, two distributional functions are computed according to its prediction. Figure 5 shows the dynamic evaluation result. In view of both approximate discrepancy and absolute discrepancy, the distribution difference on the real text and generated text does not decrease when more adversarial learning rounds are adapted. Once again, the results show that the approach of the existing language GANs cannot improve text generation.

Figure 5. A third party discriminator evaluates two GANs' performance on COCO. (a) SeqGAN. (b) RelGAN.

6. Conclusion and future work

Unconditional text generation is the step-stone of conditional text generation such as news generation and text summarisation. It is not clear that GAN can improve the unconditional text generation. We present two metric functions to measure the discrepancy between real text and generated text. Numerous experiments show that this discrepancy does exist. We use various methods to update the generator parameters according to the detected discrepancy signals. Unfortunately, the distributional difference between real data and generated data does not decrease, indicating the difficulty of generator improvement with these signals. Finally, we use a third-part discriminator to evaluate the effectiveness of GAN and find that with more adversarial learning epochs, the discrepancy increases rather than decreases. Our study provided valuable information for the industry by analysing the existing language GANs do not work in-depth.

Many studies could be done in the future. First, the novel method used to facilitate the reward signals to improve the generator is worth further study. Besides the constraints from intrinsic language characteristics, common sense and logic should be introduced to improve text generation. Finally, diversity, such as conversation generation in chat platforms, should be further investigated.

Acknowledgments

We thank the anonymous reviewers for their valuable comments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [grant numbers 61936012, 61976114, 81373056] and the National Key Research and Development Program of China [grant number 2018YFB1005102].

Notes

1 An exception is RelGAN which does not need to pre-train D.

2 http://cocodataset.org/.

3 http://www.statmt.org/wmt17/.

4 According to Section 3, we sample generated instances as much as test instances.