Learning Bilingual Word Embedding Mappings with Similar Words in Related Languages Using GAN

Figures & data

Figure 1. Overall organization of the paper.

Figure 2. The architecture of CBOW and Skip-gram as described in (Mikolov et al. 2013b).

Table 1. Word embedding mapping methods

Regression Methods	Orthogonal methods	Canonical methods	Margin methods
Mikolov et al. 2013a	Xing et al. 2015	Faruqui and Dyer 2014	Lazaridou, Dinu, and Baroni 2015
Hauer et al. 2017	Zhang et al. 2017	Lu et al. 2015	Dinu et al. 2015
Mogadala et al. 2016	Artetxe, Labaka, and Agirre 2018	Ammar et al. 2016	Mrkšić et al. 2017
Conneau et al. 2018	Smith et al.	Rajendran et al2016	Calixto et al. 2017

Table 2. An approximate count of articles and tokens in Wikipedia dumps for each language (K = 1000)

Language	Number of articles	Number of tokens
Turkish	448k	492 K
Azerbaijani	180 K	282 K
Uzbek	140 K	152 K
Turkmen	23 K	35 K
Swedish	2887 K	917 K
Danish	270k	342 K
Norwegian (Bokmål)	569 K	563k
Serbian	651 K	643 K
Bosnian	88 K	207 K
Croatian	209 K	388 K

Figure 3. The process of learning word vectors in each language.

Table 3. The number of words in seed dictionaries and size of the training, validation, and test sets (K = 1000)

Language Pairs	Seed dictionary Size	Training set size	Validation set size	Test set size
Turkish-Azerbaijani	82 K	52 K	15 K	15k
Turkish-Uzbek	24 K	16 K	4 K	4 K
Turkish-Turkmen	5.5 K	3.5 K	1 K	1 K
Swedish-Danish	167 K	117 K	25 K	25 K
Swedish-Norwegian	254 K	174 K	40 K	40 K
Serbian-Bosnian	111 K	71 K	20 K	20 K
Serbian-Croatian	135 K	91 K	22 K	22 K

Figure 4. Overview of the proposed model.

Figure 5. Encoder-Decoder architecture with an attention mechanism (Bahdanau, Cho, and Bengio 2016).

Table 4. Implemented model’s performance in different networks

Language pairs	Vanilla LSTM	Encoder-decoder	Encoder-decoder with attention	Our proposed model
Turkish-Azerbaijani	53.2%	54.6%	55.3%	78.32%
Turkish-Uzbek	54.1%	55.2%	55.9%	68.79%
Turkish-Turkmen	47.3%	49.01%	53.65%	54.63%
Swedish-Danish	51.37%	53.6%	56.4%	77.04%
Swedish-Norwegian	50.14%	52.86%	56.2%	74.96%
Serbian-Bosnian	54.3%	55.8%	58.3%	76.48%
Serbian-Croatian	49.12%	51.16%	54.8%	69.14%
Average	51.36%	53.18%	55.79%	71.34%

Figure 6. Initial seed dictionary impact on the bilingual transform mapping.

Figure 7. Differences between real and generated vectors in 3 sample words.

Table 5. Accuracy of the proposed method compared with previous works

Work	Language pair	Accuracy
Mikolov, Le, and Sutskever (2013)	EN-DE	35.00%
Faruqui and Dyer (2014)	EN-IT	38.40%
Shigeto et al. (2015)	EN-DE	43.07%
Lazaridou, Dinu, and Baroni (2015)	EN-DE	38.93%
Lazaridou, Dinu, and Baroni (2015)	EN-IT	40.20%
Xing et al. (2015))	EN-DE	41.27%
Zhang et al. (2016)	EN-DE	40.80%
Artetxe, Labaka, and Agirre (2016)	EN-DE	41.87%
Smith et al. ()	EN-DE	43.33%
Artetxe, Labaka, and Agirre (2018)	EN-IT	45.27%
Our proposed method	TR-AZ	78.32%
Our proposed method	TR-UZ	71.79%
Our proposed method	TR-TK	54.63%
Our proposed method	SV-DA	77.04%
Our proposed method	SV-NO	74.96%
Our proposed method	SR-BS	76.48%
Our proposed method	SR-CR	69.14%
Our proposed method Average		71.77%

Mikolov, T., I. Sutskever, K. Chen, G. Corrado, and J. Dean. 2013b. Distributed representations of words and phrases and their compositionality. https://arxiv.org/abs/1310.4546

 Google Scholar

Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013a. Efficient estimation of word representations in vector space. arXiv:1301.3781 2:3111–19.

 Google Scholar

Xing, C., D. Wang, C. Liu, and Y. Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. Proceedings of NAACL-HLT Denver, USA (Association for Computational Linguistics), 1005–10.

 Google Scholar

Faruqui, M., and C. Dyer. 2014. Improving vector space word representations using multilingual correlation. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden, 462–71.

 Google Scholar

Lazaridou, A., G. Dinu, and M. Baroni. 2015. Hubness and pollution: Delving into cross space mapping for zero-shot learning. Proceedings of ACL, Beijing, China.

 Google Scholar

Hauer, Bradley, Garrett, Nicolai, and Grzeg, Kondrak. 2017. Bootstrapping unsupervised bilingual lexicon induction. In Proceedings of EACL, 619–624

 Google Scholar

Zhang, M., Y. Liu, H. Luan, and M. Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics Vancouver, Canada, Vol. 1 (Association for Computational Linguistics), 1959–1970 https://aclanthology.org/P17-1 doi:10.18653/v1/P17-1.

 Google Scholar

Lu, Ang, Wang, Weiran, Bansal, Mohit, Gimple, Kevin, and Livescu, Karen. 2015. Deep multilingual correlation for improved word embeddings Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 250–256.

 Google Scholar

Dinu, Georgiana, Lazaridou, Angeliki, and Baroni, Marco. 2015. Improving zero-shot learning by mitigating the hubness problem, In Proceedings of ICLR (Workshop Track).

 Google Scholar

Mogadala, Aditya, and Rettinger, Achim. 2016. Bilingual word embeddings from parallel and nonparallel corpora for cross-language text classification Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 692–702.

 Google Scholar

Artetxe, M., G. Labaka, and E. Agirre. 2018. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations, Proceedings of the AAAI Conference on Artificial Intelligence, 32 1 https://ojs.aaai.org/index.php/AAAI/article/view/11992

 Google Scholar

Ammar, W., G. Mulcaire, Y. Tsvetkov, G. Lample, C. Dyer, and N. A. Smith. 2016. Massively multilingual word embeddings. CoRR abs/1602.01925. http://arxiv.org/abs/1602.01925

 Google Scholar

Mrkšić, N., I. Vulić, D. Ó. Séaghdha, İ. Leviant, R. Reichart, M. Gašić, and A. Korh. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics 5:309–24. doi:10.1162/tacl_a_00063.

 Google Scholar

Conneau, A., G. Lample, M. Ranzato, L. Denoyer, and H. Jégo. 2018. Word translation without parallel data, 6th International Conference on Learning Representations Vancouver, BC, Canada, OpenReview.net.

 Google Scholar

Rajendran, Janarthanan, Khapra, Mitesh M, Chandar, Sarath, and Ravindran, Balaraman. 2016. Bridge correlational neural networks for multilingual multimodal representation learning, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 171–181

 Google Scholar

Iacer, Calixto, Qun, Liu, and Nick, Campbell. 2017. Multilingual Multi-modal Embeddings for Natural Language Processing, CoRR, abs/1702.01101

 Google Scholar

Bahdanau, D., K. Cho, and Y. Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. https://arxiv.org/abs/1409.0473

 Google Scholar

Mikolov, T., Q. V. Le, and I. Sutskever. 2013. Exploiting similarities among languages for machine translation. https://arxiv.org/abs/1309.4168

 Google Scholar

Shigeto, Y., I. Suzuki, K. Hara, M. Shimbo, and Y. Matsumoto. 2015. Ridge Regression, Hubness, and Zero-Shot Learning. https://arxiv.org/abs/1507.00825

 Google Scholar

Zhang, Y., D. Gaddy, R. Barzilay, and T. Jaakkola. 2016. Ten pairs to tag – multilingual POS tagging via coarse mapping between embeddings. Proceedings of NAACL-HLT San Diego, USA (Association for Computational Linguistics), 1307–1317.

 Google Scholar

Artetxe, M., G. Labaka, and E. Agirre. 2016. Learning principled bilingual mappings of word embedding while preserving monolingual invariance. Conference on empirical methods in natural language processing, 2289–1561 .

 Google Scholar