Reward estimation with scheduled knowledge distillation for dialogue policy learning

Figures & data

Table 1. An dialogue example with different user goals and the evaluation results.

User goal 1: Find a restaurant, ask its phone number.

User goal 2: Find a Chinese restaurant.

$ag{t}_1$: What can I help you?

$us{r}_2$: I want to find a place to eat.

$ag{t}_3$: What about La Pergola, it's a nice Italian restaurant.

$us{r}_4$: OK, what's the phone-number?

$ag{t}_5$: 1456-2311.

$us{r}_6$: Sorry, I change my mind, is there a restaurant that serves Thai food?

$ag{t}_7$: Sure, Above eleven, its number is 1334-2587.

$us{r}_8$: Thank you.

$ag{t}_9$: Thank you.

For user goal 1: Succeed, all requirements satisfied.

For user goal 2: Failed, not a Chinese restaurant.

Note: The user goal changes from Italian restaurant to Thai one at turn 6. agt stands for agent while usr stands for user.

Table 2. Summarisation of related work.

RL-based dialogue policy
Categories	Drawbacks		References
DDQ-based framwork	Reward sparsity	Ignore user goals	(Creswell et al., 2018; Peng et al., 2018b; Wu et al., 2021)
Reward shaping	Requiring extra resources Unstable	Uncapable of dealing with shifty user goals	(Khandelwal, 2021; Peng et al., 2018a; Su, Li et al., 2018; Zhang et al., 2021; Zhao et al., 2020)
Curriculum learning	Reward sparsity		(Liu et al., 2021; Zhao et al., 2022; Zhao, Huang et al., 2021; Zhu et al., 2021)
Knowledge distillation
Applications	References
Natural Language processing	(Greco et al., 2021; Ren et al., 2021; Wang et al., 2020)
Computer vision	(Haidar & Rezagholizadeh, 2019; Li et al., 2021; Liu et al., 2019)
Other applications	(Ren et al., 2021; Shen et al., 2020)
Curriculum learning
Application scopes	References
Supervised learning	(El-Bouri et al., 2020; Guo et al., 2018; Platanios et al., 2019)
Reinforcement learning	(Narvekar & Stone, 2019; Qu et al., 2018)
Graph learning	(Chu et al., 2021; Dong et al., 2021; Wang, Wang et al., 2021)

Figure 1. Different strategies of learning dialogue policy. (a) DQN (b) DDQ.

Figure 2. Architecture of the proposed scheduled knowledge distillation for dialogue policy learning.

Figure 3. Architecture of teacher and student reward estimators. (a) Teacher reward estimator (b) Student reward estimator.

Figure 4. Dialogue state and user goal representations. (a) Multi-hot state representations (b) Embedded state representations (c) Embedded user goal representations.

Table 3. The statistics of datasets.

Dataset		#Intents	#Slots	#Dialogues
MDC	Movie-Ticket Booking	11	29	2890
	Restaurant Reservation	11	30	4103
	Taxi Ordering	11	29	3094
MultiWOZ		13	30	10,438

Table 4. Experimental results on Microsoft Dialogue Challenge dataset.

		Epoch=100			Epoch=200			Epoch=300
Agent		Success	Reward	Turns	Success	Reward	Turns	Success	Reward	Turns
Movie-Ticket Booking
Rule-based		0.2135	−10.42	27.14	0.2367	−11.45	29.01	0.2256	−11.01	28.91
DQN-based	DQN	0.4676	−4.54	32.91	0.5586	10.98	25.91	0.6978	28.91	24.24
	VACL	0.4079	−4.79	29.49	0.4331	−1.42	28.81	0.4339	−1.33	28.79
	ACL-DQN	0.5717	15.92	27.36	0.7523	37.39	21.30	0.7573	45.28	18.57
	DQN-SKD	0.6418	18.56	20.42	0.7716	39.81	20.14	0.8128	48.10	15.30
DDQ-based	DDQ(10)	0.5918	23.14	27.12	0.6321	36.19	19.17	0.7316	40.19	18.45
	DR-D3Q	0.6183	15.94	21.42	0.6812	22.67	19.27	0.7899	33.75	16.68
	ES-DDQ	0.6243	22.74	22.78	0.7232	35.76	20.78	0.7581	40.10	18.55
	D3Q(10)	0.6333	28.99	16.01	0.7000	37.24	15.52	0.6667	33.09	15.83
	GP-DDQ	0.7069	35.09	21.48	0.7706	43.65	19.60	0.7874	45.72	19.54
	DDQ-SDPL	0.7600	44.75	14.90	0.8000	51.15	13.29	0.8300	54.63	12.73
	DDQ-SKD	0.7645	43.31	14.78	0.8144	53.12	13.91	0.8347	57.01	12.56
Restaurant Reservation
Rule-based		0.1784	−19.56	30.09	0.1891	−20.11	31.52	0.1894	−20.20	30.91
DQN-based	DQN	0.3567	2.63	26.24	0.4098	8.13	22.19	0.4132	6.78	23.15
	VACL	0.3664	−8.49	24.93	0.4352	−1.81	23.96	0.4321	−2.17	24.05
	DQN-SKD	0.4134	10.44	23.14	0.5143	15.14	18.09	0.5222	17.78	17.01
DDQ-based	DDQ(10)	0.4356	12.56	21.45	0.5723	22.07	15.43	0.5678	19.01	14.23
	DR-D3Q	0.4233	14.43	20.23	0.5924	25.13	14.67	0.5762	24.13	13.91
	D3Q(10)	0.4566	15.33	18.34	0.5531	19.01	13.91	0.5626	23.15	17.35
	DDQ-SKD	0.4716	16.01	20.24	0.6145	28.77	14.89	0.6241	25.16	12.33
Taxi Ordering
Rule-based		0.2791	-8.40	27.81	0.2809	−9.12	28.09	0.2894	−9.01	26.56
DQN-based	DQN	0.5133	15.32	20.13	0.6784	24.12	17.49	0.7136	27.43	16.31
	VACL	0.4814	3.75	21.96	0.6460	19.20	19.89	0.6556	20.23	19.56
	DQN-SKD	0.5732	18.43	18.90	0.7123	27.31	16.76	0.7321	28.94	16.01
DDQ-based	DDQ(10)	0.5567	16.71	19.12	0.6533	25.88	18.32	0.7451	30.13	15.67
	DR-D3Q	0.5331	16.43	18.23	0.7198	28.19	16.71	0.7261	25.78	15.94
	D3Q(10)	0.5812	20.44	16.12	0.7316	31.04	15.57	0.7819	34.41	14.13
	DDQ-SKD	0.6256	23.19	15.79	0.7644	35.28	16.47	0.8201	36.37	15.01

Note: Best results are marked in bold.

Table 5. Experimental results on MultiWOZ dataset.

		Epoch=100			Epoch=200			Epoch=300
Agent		Success	Reward	Turns	Success	Reward	Turns	Success	Reward	Turns
DQN-based	DQN	0.4321	13.57	15.61	0.4789	19.15	16.14	0.5577	26.42	14.16
	DQN-SDPL	0.4500	2.51	13.49	0.5200	11.04	13.36	0.5700	12.22	13.81
	DQN-SKD	0.4912	21.66	13.51	0.5581	26.17	12.42	0.5914	28.19	12.45
DDQ-based	DDQ(10)	0.5132	23.19	16.12	0.5714	27.35	15.33	0.5963	28.27	16.24
	DR-D3Q	0.6177	25.66	15.67	0.6231	31.64	16.79	0.6201	30.56	15.66
	D3Q(10)	0.5735	23.91	14.85	0.6047	29.98	14.16	0.6316	32.61	13.91
	DDQ-SKD	0.6317	27.15	14.19	0.6434	34.01	13.67	0.6536	31.98	13.41

Note: Best results are marked in bold.

Figure 5. Learning curves of agents discarding different components on Movie Ticket Booking dataset. (a) DQN-based agents (b) DDQ-based agents.

Table 6. Success rate of removing the curriculum learning (CL), knowledge distillation (KD), weak supervision (WS) and reinforcement learning (RL) on Movie-Ticket Booking dataset.

Model	Epoch=100	Epoch=200	Epoch=300	Epoch=400
DQN-SKD	0.6418	0.7716	0.8128	0.8145
wo.KD-1	0.5218	0.6168	0.6841	0.7361
wo.KD-2	0.5584	0.6916	0.7616	0.7841
wo.CL	0.5091	0.7329	0.7960	0.7946
wo.WS	0.6010	0.7563	0.7746	0.8012
wo.RL	0.5557	0.7081	0.7314	0.7535
DDQ-SKD	0.7645	0.8144	0.8447	0.8367
wo.KD-1	0.6045	0.6995	0.7631	0.7732
wo.KD-2	0.6433	0.7513	0.7654	0.8110
wo.CL	0.6896	0.7266	0.7789	0.7981
wo.WS	0.7292	0.8012	0.8141	0.8257
wo.RL	0.6320	0.7288	0.7548	0.8032

Figure 6. Box plots of DQN and DDQ based agents on Movie-Ticket Booking dataset. The center, bottom and top line of the box show the median, ${25}^{th}$ and ${75}^{th}$ percentiles, respectively. The lines that extend from the box represent the expected variation of the data. The shorter the box, the more stable it is. (a) DQN-based agents (b) DDQ-based agents.

Figure 7. Learning curves of DQN-SKD and DDQ-SKD with different anchor reward $R_a$. (a) DQN-based (b) DDQ-based.

Figure 8. Reward visualisation of a dialogue simulation between a rule based agent and a user simulator. The reward is provided by a trained student reward estimator.

Figure 9. Success rate of different agents dealing with shifty user goals on MultiWOZ. The number in each bar represents success rate degradation compared to that in Table 6 at Epoch=300.

Table 7. Experimental results of DDQ-SKD with different complexity measurer on MultiWOZ dataset.

	Epoch=100			Epoch=200			Epoch=300
Complexity measurer	Success	Reward	Turns	Success	Reward	Turns	Success	Reward	Turns
Stu-Tea	0.6317	27.51	14.19	0.6434	34.01	13.67	0.6536	31.98	13.41
Gold-Tea	0.6178	26.71	16.43	0.6281	29.76	14.57	0.6458	32.51	13.62
Gold-Stu	0.5315	22.47	15.44	0.5754	26.52	16.78	0.5999	28.14	15.79
Rule	0.5589	21.01	17.43	0.5671	25.51	17.10	0.6031	27.42	15.22
wo. CL	0.4918	20.43	18.91	0.5549	23.91	17.49	0.5838	24.76	16.44

Note: wo. CL means without curriculum learning.

Table 8. Dialogue examples of simulated user interacting with different agents trained at 100 epochs.

$\mathcal{G}=\left(\begin{array}{rl}\mathcal{C}& =\left[\begin{array}{rl}& Date:tomorrow\\ & City:detroit\\ & MoiveName:deadpool\\ & StartTime:7pm\end{array}\right]\\ \mathcal{R}& =[Theater]\end{array}\right)$	$\mathcal{G}=\left(\begin{array}{rl}\mathcal{C}& =\left[\begin{array}{rl}& City:duQuoin\\ & NumberofPeople:2\\ & MovieName:StarWars\end{array}\right]\\ \mathcal{R}& =\left[\begin{array}{rl}& Date\\ & Thether\\ & StartTime\end{array}\right]\end{array}\right)$.
DQN	DDQ
$us{r}_1$: What theater will play the deadpool at 7pm tomorrow?	$us{r}_1$: What start time can I book 2 tickets for star wars?
$ag{t}_2$: What movie are you interested in?	$ag{t}_2$: What movie are you interested in?
$us{r}_3$: I want to watch deadpool.	$us{r}_3$: I want to watch star wars.
$ag{t}_4$: Which city would you like?	$ag{t}_4$: What time would you like to see it?
$us{r}_5$: I want to watch at detroit.	$us{r}_5$: Any time is fine.
$ag{t}_6$: What date would you like to watch it?	$ag{t}_6$: Which city would you like?
$us{r}_7$: I want to set it up tomorrow.	$us{r}_7$: I want to watch at du Quoin
$ag{t}_8$: Which theater would you like?	$ag{t}_8$: How many tickets do you need?
$us{r}_9$: Which theater is available?	$us{r}_9$: I want 2 tickets please.
$ag{t}_{10}$: Which theater would you like?	$ag{t}_{10}$: Ok, your purchase of 2 tickets for star wars at du Quoin is confirmed.
···	$us{r}_{11}$: Oh, sorry this is wrong ticket.
	$ag{t}_{12}$: Thank you.
Failure: the conversation is repeated and the agent is not able to book the ticket within the maximum turns.	Failure: the agent is not able to book the tickets with all requests identified.
DQN-SKD	DDQ-SKD
$us{r}_1$: What theater will play the dead pool at 7pm tomorrow?	$us{r}_1$: What start time can I book 2 tickets for star wars?
$ag{t}_2$: Which city would you like?	$ag{t}_2$: Which city would you like?
$us{r}_3$: I want to watch at detroit.	$us{r}_3$: I want to watch at du Quoin.
$ag{t}_4$: How many tickets do you want?	$ag{t}_4$: What date do you want?
$us{r}_5$: I want 3 tickets please!	$us{r}_5$: Any day will do.
$ag{t}_6$: Great, I was able to book 3 tickets for you to watch dead pool at theater in detroit at 7pm tomorrow!	$ag{t}_6$: How about 2 tickets for star wars at du Quoin for tomorrow at theater?
$us{r}_7$: Thank you.	$us{r}_7$: What time will it start?
$ag{t}_8$: Thank you.	$ag{t}_8$: It starts at 7pm.
	$us{r}_9$: Ok, thank you.
	$us{r}_{10}$: Thank you.
Success	Success

Note: usr denotes user, while agt denotes agent.

Figure 10. Human evaluation results of DQN, DQN-SKD, DDQ and DDQ-SKD agents. The number of test dialogues are presented in on each bar.

Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018). Generative adversarial networks: An overview. IEEE Signal Processing Magazine, 35(1), 53–65. https://doi.org/10.1109/MSP.2017.2765202

 Web of Science ®Google Scholar

Peng, B., Li, X., Gao, J., Liu, J., & Wong, K. F. (2018). Deep Dyna-Q: Integrating Planning for Task-Completion Dialogue Policy Learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 1., Long papers) (pp. 2182–2192).

 Google Scholar

Wu, G., Fang, W., Wang, J., Cao, J., Bao, W., Ping, Y., & Wang, Z. (2021). Gaussian Process based Deep Dyna-Q approach for Dialogue Policy Learning. In Findings of the Association for Computational Linguistics: Acl-ijcnlp (pp. 1786–1795).

 Google Scholar

Khandelwal, A. (2021). WeaSuL: Weakly Supervised Dialogue Policy Learning: Reward Estimation for Multi-turn Dialogue. In Proceedings of the 14th International Conference on Natural Language Generation (pp. 64–75).

 Google Scholar

Peng, B., Li, X., Gao, J., Liu, J., Chen, Y. N., & Wong, K. F. (2018). Adversarial advantage actor-critic model for task-completion dialogue policy learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6149–6153).

 Google Scholar

Su, S. Y., Li, X. , Gao, J. , Liu, J., & Chen, Y. N. (2018). Discriminative Deep Dyna-Q: Robust Planning for Dialogue Policy Learning. In EMNLP.

 Google Scholar

Zhang, R., Wang, Z., Zheng, M., Zhao, Y., & Huang, Z. (2021). Emotion-sensitive deep dyna-Q learning for task-completion dialogue policy learning. Neurocomputing, 459, 122–130. https://doi.org/10.1016/j.neucom.2021.06.075

 Web of Science ®Google Scholar

Zhao, Y., Wang, Z., Yin, K., Zhang, R., Huang, Z., & Wang, P. (2020). Dynamic Reward-Based Dueling Deep Dyna-Q: Robust Policy Learning in Noisy Environments. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, pp. 9676–9684).

 Google Scholar

Liu, S., Zhang, J., He, K., Xu, W., & Zhou, J. (2021). Scheduled Dialog Policy Learning: An Automatic Curriculum Learning Framework for Task-oriented Dialog System. In Findings of the Association for Computational Linguistics: Acl-ijcnlp (pp. 1091–1102).

 Google Scholar

Zhao, Y., Qin, H., Zhenyu, W., Zhu, C., & Wang, S. (2022). A Versatile Adaptive Curriculum Learning Framework for Task-oriented Dialogue Policy Learning. In Findings of the Association for Computational Linguistics: Naacl 2022 (pp. 711–723).

 Google Scholar

Zhao, Y., Wang, Z., & Huang, Z. (2021). Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy Learning.

 Google Scholar

Zhu, H., Zhao, Y., & Qin, H. (2021). Cold-started Curriculum Learning for Task-oriented Dialogue Policy. In IEEE International Conference on E-Business Engineering (icebe) (pp. 100–105).

 Google Scholar

Greco, A., Saggese, A., Vento, M., & Vigilante, V. (2021). Effective training of convolutional neural networks for age estimation based on knowledge distillation. Neural Computing and Applications, 34(24), 1–16.

 PubMed Web of Science ®Google Scholar

Ren, S., Guo, K., Ma, J., Zhu, F., Hu, B., & Zhou, H. (2021). Realistic medical image super-resolution with pyramidal feature multi-distillation networks for intelligent healthcare systems. Neural Computing and Applications, 1–16. https://doi.org/10.1007/s00521-021-06287-x

 PubMed Web of Science ®Google Scholar

Wang, X., Fu, T., Liao, S., Wang, S., Lei, Z., & Mei, T. (2020). Exclusivity-consistency regularized knowledge distillation for face recognition. In European Conference on Computer Vision (pp. 325–342).

 Google Scholar

Haidar, M., & Rezagholizadeh, M. (2019). Textkd-gan: Text generation using knowledge distillation and generative adversarial networks. In Canadian Conference on Artificial Intelligence (pp. 107–118).

 Google Scholar

Li, B., Wang, Z., Liu, H., Du, Q., Xiao, T., Zhang, C., & Zhu, J. (2021). Learning Light-Weight Translation Models from Deep Transformer. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, pp. 13217–13225).

 Google Scholar

Liu, J., Chen, Y., & Liu, K. (2019). Exploiting the ground-truth: An adversarial imitation based knowledge distillation approach for event detection. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 33, pp. 6754–6761).

 Google Scholar

Shen, Y., Xu, X., & Cao, J. (2020). Reconciling predictive and interpretable performance in repeat buyer prediction via model distillation and heterogeneous classifiers fusion. Neural Computing and Applications, 32(13), 9495–9508. https://doi.org/10.1007/s00521-019-04462-9

 Web of Science ®Google Scholar

El-Bouri, R., Eyre, D., Watkinson, P., Zhu, T., & Clifton, D. (2020). Student-teacher curriculum learning via reinforcement learning: predicting hospital inpatient admission location. In International Conference on Machine Learning (pp. 2848–2857).

 Google Scholar

Guo, S., Huang, W., Zhang, H., Zhuang, C., Dong, D., Scott, M. R., & Huang, D. (2018). Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 135–150).

 Google Scholar

Platanios, E. A., Stretcu, O., Neubig, G., Póczos, B., & Mitchell, T. (2019). Competence-based Curriculum Learning for Neural Machine Translation. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human language Technologies (Vol. 1., long and short papers) (pp. 1162–1172).

 Google Scholar

Narvekar, S., & Stone, P. (2019). Learning Curriculum Policies for Reinforcement Learning. In Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (pp. 25–33).

 Google Scholar

Qu, M., Tang, J., & Han, J. (2018). Curriculum learning for heterogeneous star network embedding via deep reinforcement learning. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (pp. 468–476).

 Google Scholar

Chu, G., Wang, X., Shi, C., & Jiang, X. (2021). CuCo: Graph Representation with Curriculum Contrastive Learning. In IJCAI (pp. 2300–2306).

 Google Scholar

Dong, X., Long, C., Xu, W., & Xiao, C. (2021). Dual graph convolutional networks with transformer and curriculum learning for image captioning. In Proceedings of the 29th ACM International Conference on Multimedia (pp. 2615–2624).

 Google Scholar

Wang, Y., Wang, W., Liang, Y., Cai, Y., & Hooi, B. (2021). Curgraph: Curriculum learning for graph classification. In Proceedings of the Web Conference (pp. 1238–1248).

 Google Scholar

Data availability

The Microsoft Dialogue Challenge and MultiWOZ dialogue datasets analysed during the current study are available in E2EDialog and ConvLab-2 repositories, respectively.