Towards Autoencoding Variational Inference for Aspect-Based Opinion Summary

Figures & data

Figure 1. Aspect-based opinion summary result of one product.

Figure 2. Probabilistic graphical model of LDA and JST.

Figure 3. A general autoencoder system.

Figure 4. Variational auto encoder model where observable variable $x$ and its correspondent latent $z$ are distributed on Gaussian distribution $\mathcal{N}({\mu }_{x|z},{\mathrm{\Sigma }}_{x|z})$ and $\mathcal{N}({\mu }_{z|x},{\mathrm{\Sigma }}_{z|x})$, respectively.

Figure 6. AutoEncoding variational inference for aspect discovery. As illustrated, the yellow block $\theta$ and $\beta$ is corresponded to the document-topic and topic-word distributions which is described in Figure 2a, respectively. Meanwhile, $\gamma$ and ${\gamma }^{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{r}}$ are additional blocks which play an important role in the aspect discovery task.

Figure 5. Prior distribution matrix.

Table 1. Discovered aspects (bold text indicates seed words).

Aspect	Discovered Terms
Food	sauce, salad, cheese, onion, crab.
Staff	service, staff, rude, hostess, waiter.
Ambience	scene, place, wall, decorate, romantic.

Figure 7. Example of constructing the square loss term. Firstly, every rows in the $\gamma$ matrix is normalized via softmax function. After that, a submatrix is constructed by choosing only set of words in the normalized matrix $\gamma$ such that this word must also exists in a given ${\gamma }^{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{r}}$ matrix. Finally, the prior loss is computed via the Euclidean distance between this submatrix and ${\gamma }^{\mathrm{p}\mathrm{r}\mathrm{i}\mathrm{o}\mathrm{r}}$.

Figure 8. Autoencoding variational inference for joint sentiment/topic modeling where $\theta$, $\beta$ and $\pi$ are the corresponding latent variables in JST graphical model (Figure 2b). Moreover, the yellow block which is used to mapping document $x$ to latent variable $\pi$ can be represented as any modern classification deep neural network.

Table 2. AVIAD and WLDA seedwords.

Aspect	Seed words
Food	food sauce shrimp cheese potato fry tomato
Staff	staff service friendly rude hostess waiter bartender waitress
Ambience	atmosphere scene place table outside outdoor ambiance

Table 3. Statistics for IMDB and Yelp datasets.

Dataset	#labeled	#unlabeled	#testset	#tokens	average length
IMDB	25,000	50,000	25,000	11,737,946	235
Yelp	200,000	257,156	51,432	14,799,168	58

Table 4. Topics extracted by AVIAD and WLDA.

Model	Score	Top words
AVIAD	0.16	scallop saute shrimp broccoli spinach tuna bean
	0.17	apology bill refill behavior phone busboy manage
	0.28	lit chandelier banquet dimly paint wooden mirror
WLDA	0.09	dish like pork good fresh taste menu
	0.06	us food wait order time place even
	0.10	great restaurant room nice like bar music

Table 5. IMDB Topics extracted by AVIJST and JST.

Positive			Negative
AVIJST
Wrestle (0.31)	Starwar (0.30)	Kungfu (0.29)	Horror (0.20)	War (0.22)	Novel (0.27)
wwe	leia	hong	lugosi	germans	austen
michaels	ewoks	cheung	bela	war	zelah
undertaker	jedi	kung	karloff	soldiers	eyre
hogan	darth	sammo	dracula	german	rochester
hulk	vader	tsui	serum	hitler	novel
cena	soldiers	shaolin	undead	soldier	novels
JST
Music (0.14)	Shows (0.11)	Kungfu (0.21)	Horror (0.13)	War (0.17)	Novel (0.15)
music	show	action	zombie	war	book
musical	series	fight	horror	soldiers	the
dance	the	japanese	zombies	military	read
songs	episode	martial	gore	army	novel
song	tv	fu	blood	soldier	books
singing	episodes	kong	dead	battle	adaptation

Table 6. Yelp Topics extracted by AVIJST and JST.

Positive			Negative
AVIJST
Coffee (0.20)	Beer (0.26)	Thai food (0.36)	Staff (0.16)	Ambience (0.15)	Japanese food (0.34)
coffee	beers	pad	rude	reservation	sushi
latte	beer	thai	disrespectful	reservations	sashimi
espresso	brews	panang	attitude	waited	nigiri
baristas	craft	curry	unprofessional	closed	rolls
wifi	tap	kha	filthy	pm	ayce
study	ipa	thailand	yelled	phone	tempura
JST
Coffee (0.07)	Beer (0.13)	Japanese food (0.20)	Staff (0.08)	Ambience (0.13)	Thai food (0.19)
coffee	beer	sushi	staff	wait	thai
shop	bar	roll	rude	table	curry
cafe	selection	rolls	customers	seated	pad
local	beers	fish	people	pm	chicken
store	tap	fresh	customer	night	spicy
latte	pub	tuna	manager	reservation	rice

Figure 9. Aspect discovery performance on restaurant dataset.

Figure 10. Joint sentiment/topic modeling performance based on average topic coherent.

Table 7. Sentiment words discovered.

Datasets	Polarity	Sentiment words
IMDB	Positive	excellent, wonderful, underrated, superb, flawless
	Negative	poorly, waste, worst, pointless, awful
Yelp	Positive	recommended, delicious, amazingly, underestimate, amicable
	Negative	tasteless, flavorless, unimpressed, inedible, rude

Table 8. Aspect identification results.

	Metrics
Aspect	Model	Precision	Recall	F1
Food	AVIAD	0.948	0.806	0.870
	WLDA	0.940	0.700	0.803
Staff	AVIAD	0.805	0.887	0.842
	WLDA	0.662	0.895	0.761
Ambience	AVIAD	0.843	0.903	0.871
	WLDA	0.844	0.793	0.817

Table 9. Accuracy on test set for IMDB.

	Labels
Model	0	100	250	500	1k	5k	12k5	Full
MLP	–	68.5	77.1	80.5	82.5	84.8	86.1	87.0
CNN	–	58.3	73.8	79.2	81.9	84.5	87.3	88.8
SSVAEII-MLP	–	75.6	77.0	80.8	81.1	84.1	86.2	87.0
SSVAEII-CNN	–	72.1	77.2	79.4	81.1	85.1	87.1	89.0
JST	57.1	–	–	–	–	–	–	–
AVIJST-MLP	–	68.2	72.7	79.9	81.2	85.9	88.2	89.7
AVIJST-CNN	–	76.0	80.4	81.7	83.1	87.2	88.4	90.6

Table 10. Accuracy on test set for Yelp.

	Labels
Model	0	500	1000	Full
MLP	–	86.2	88.3	94.1
CNN	–	85.1	87.0	94.5
SSVAEII-MLP	–	87.2	87.8	94.5
SSVAEII-CNN	–	84.5	86.0	94.5
JST	78.0	–	–	–
AVIJST-MLP	–	87.5	90.4	94.5
AVIJST-CNN	–	88.4	89.8	94.9