308
Views
5
CrossRef citations to date
0
Altmetric
Computers and Computing

AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages

ORCID Icon, , &

REFERENCES

  • I. Pierre, C. Cherry, and G. Foster, “A challenge set approach to evaluating machine translation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, Copenhagen, 2017, pp. 2486–2496.
  • H. Lifeng. “Machine translation evaluation resources and methods: A survey,” arXiv preprint arXiv:1605.04515, Cornell University Library, 2018.
  • C. Elisabet, and J. Atserias, “VERTa: a linguistic approach to automatic machine translation evaluation,” Lang. Resour. Eval., Vol. 53, no. 1, pp. 57–86, 2019. doi:10.1007/s10579-018-9430-2
  • C. Eirini, “How to evaluate machine translation: A review of automated and human metrics,” Nat. Lang. Eng., Vol. 26, no. 2, pp. 137–161, 2020. doi:10.1017/S1351324919000469
  • P. Kishore, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318.
  • Q. Ying, and L. Specia, “Truly exploring multiple references for machine translation evaluation,” in Proceedings of the 18th Annual Conference of the European Association for Machine Translation, Antalya, Turkey, 2015, pp. 113–120.
  • Ananthakrishnan, P. Bhattacharyya, M. Sasikumar, and R. M. Shah, “Some issues in automatic evaluation of English-hindi mt: more blues for bleu,” in Proceeding of 5th International Conference on Natural Language Processing, Hyderabad.
  • K. Nadeem Jadoon, W. Anwar, and N. Durrani. “Machine translation approaches and survey for Indian languages,” arXiv preprint arXiv:1701.04290, 2017.
  • R. N. Patel, P. B. Pimple, and M. Sasikumar, “Machine translation in Indian languages: challenges and resolution,” J. Intell. Syst., Vol. 28, no. 3, pp. 437–445, 2019. doi:10.1515/jisys-2018-0014
  • S. Elior, O. Abend, and A. Rappoport. “Bleu is not suitable for the evaluation of text simplification,” arXiv preprint arXiv:1810.05995, 2018.
  • G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics,” in Proceeding of the 2nd International Conference on Human Language Technology Resources, San Diego, CA, 2002, pp. 138–145.
  • S. Matthew, N. Madnani, B. Dorr, and R. Schwartz, “Fluency, adequacy, or HTER? Exploring different human judgments with a tunable MT metric,” in Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March 2009, pp. 259–268.
  • C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, Proceedings of the ACL-04 Workshop, Barcelona, Spain, 2004, pp. 74–81.
  • W. Billy, and C. Kit, “ATEC: automatic evaluation of machine translation via word choice and word order,” Mach. Transl., Vol. 23, no. 2–3, pp. 141–155, 2009. doi:10.1007/s10590-009-9061-x
  • C. Boxing, R. Kuhn, and S. Larkin, “Port: A precision-order-recall MT evaluation metric for tuning,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 2012, pp. 930–939.
  • GTM Link. https://nlp.cs.nyu.edu/GTM/.
  • B. Satanjeev, and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, 2005, pp. 65–72.
  • G. Ankush, S. Venkatapathy, and R. Sangal, “METEOR-Hindi: Automatic MT evaluation metric for Hindi as a target,” in Proceedings of ICON-2010: 8th International Conference on Natural Language Processing. India: Macmillan Publishers, 2010.
  • N. Tomer, D. Sinha, and K. Rai, “Evaluating BLEU metric for English to Hindi language using ManTra machine translation engine,” Int. J. Adv. Res. Comput. Sci., Vol. 3, no. 6, 2012.
  • N. Tomer, and D. Sinha, “Evaluating NIST metric for English to Hindi language using ManTra machine translation engine,” Int J Adv. Res. Comput. Eng. & Technol. (IJARCET), Vol. 1, no. 8, 2012.
  • P. Maja, “Chrf: Character n-gram F-score for automatic MT evaluation,” in Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015, pp. 392–395.
  • S. Miloš, and K. Sima’an, “Evaluating MT systems with BEER,” The Prague Bull. Math. Ling., Vol. 104, no. 1, pp. 17–26, 2015. doi:10.1515/pralin-2015-0010
  • L. Pairui, C. Chen, W. Zheng, Y. Deng, F. Ye, and Z. Zheng, “STD: An automatic evaluation metric for machine translation based on word embeddings,” IEEE/ACM Trans. Audio, Speech, Lang. Process, Vol. 27, no. 10, pp. 1497–1506, 2019. doi:10.1109/TASLP.2019.2922845
  • W. John, T. B. Kirkpatrick, K. Gimpel, and G. Neubig. “Beyond bleu: Training neural machine translation with semantic similarity,” arXiv preprint arXiv:1909.06694, 2019.
  • Universal dependencies,2020. Available: https://universaldependencies.org/.
  • H. Matthew, and I. Montani, “Spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing,” To Appear, Vol. 7, no. 1, 2017.
  • Department of Information Technology. Unified Parts of Speech (POS) standard in Indian languages-Draft standard Version 1.0. Ministry of Communications and Information Technology, Govt. of India, 2018, pp. 1–76.
  • P. Martha, R. Bhatt, B. Narasimhan, O. Rambow, D. M. Sharma, and F. Xia, “Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure,” in Proceedings of the 7th International Conference on Natural Language Processing, 2009, pp. 14–17.
  • B. R. Ahmad, R. Bhatt, A. Farudi, P. Klassen, B. Narasimhan, M. Palmer, and O. Rambow. “The Hindi/Urdu treebank project,” in Handbook of linguistic annotation. Dordrecht: Springer, 2017, pp. 659-697.
  • Universal Dependencies /UD_Hindi-HDTB, Nov. 2020. Available: https://github.com/UniversalDependencies/UD_Hindi-HDTB.
  • Universal Dependencies/UD_Urdu-UDTB, Nov. 2020. Available: https://github.com/UniversalDependencies/UD_Urdu-UDTB.
  • Universal Dependencies/UD_Telugu-MTG, , Nov. 2020. Available. https://github.com/UniversalDependencies/UD_Telugu-MTG.
  • V. Ashish, et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
  • K. Yunsu, J. Geng, and H. Ney, “Improving unsupervised word-by-word translation with language model and denoising autoencoder,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, 2018, pp. 862–868.
  • A. Mikel, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in Proceedings of the 6th International Conference on Learning Representations, 2018.
  • Mantra, Sept. 2017, Available: https://www.cdac.in/index.aspx?id=mc_mat_mantra_technology.
  • A. Kunchukuttan, P. Mehta, and P. Bhattacharyya, “The IIT Bombay English-hindi parallel corpus,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation, 2018.
  • P. Matt, C. Callison-Burch, and M. Osborne, “Constructing parallel corpora for six Indian languages via crowdsourcing,” in Proceedings of the Seventh Workshop on Statistical Machine Translation, 2012, pp. 401–409.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.