AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages

REFERENCES

• I. Pierre, C. Cherry, and G. Foster, “A challenge set approach to evaluating machine translation,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing, Copenhagen, 2017, pp. 2486–2496.
   Google Scholar
• H. Lifeng. “Machine translation evaluation resources and methods: A survey,” arXiv preprint arXiv:1605.04515, Cornell University Library, 2018.
   Google Scholar
• C. Elisabet, and J. Atserias, “VERTa: a linguistic approach to automatic machine translation evaluation,” Lang. Resour. Eval., Vol. 53, no. 1, pp. 57–86, 2019. doi:10.1007/s10579-018-9430-2
   Web of Science ®Google Scholar
• C. Eirini, “How to evaluate machine translation: A review of automated and human metrics,” Nat. Lang. Eng., Vol. 26, no. 2, pp. 137–161, 2020. doi:10.1017/S1351324919000469
   Web of Science ®Google Scholar
• P. Kishore, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318.
   Google Scholar
• Q. Ying, and L. Specia, “Truly exploring multiple references for machine translation evaluation,” in Proceedings of the 18th Annual Conference of the European Association for Machine Translation, Antalya, Turkey, 2015, pp. 113–120.
   Google Scholar
• Ananthakrishnan, P. Bhattacharyya, M. Sasikumar, and R. M. Shah, “Some issues in automatic evaluation of English-hindi mt: more blues for bleu,” in Proceeding of 5th International Conference on Natural Language Processing, Hyderabad.
   Google Scholar
• K. Nadeem Jadoon, W. Anwar, and N. Durrani. “Machine translation approaches and survey for Indian languages,” arXiv preprint arXiv:1701.04290, 2017.
   Google Scholar
• R. N. Patel, P. B. Pimple, and M. Sasikumar, “Machine translation in Indian languages: challenges and resolution,” J. Intell. Syst., Vol. 28, no. 3, pp. 437–445, 2019. doi:10.1515/jisys-2018-0014
   Web of Science ®Google Scholar
• S. Elior, O. Abend, and A. Rappoport. “Bleu is not suitable for the evaluation of text simplification,” arXiv preprint arXiv:1810.05995, 2018.
   Google Scholar
• G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics,” in Proceeding of the 2nd International Conference on Human Language Technology Resources, San Diego, CA, 2002, pp. 138–145.
   Google Scholar
• S. Matthew, N. Madnani, B. Dorr, and R. Schwartz, “Fluency, adequacy, or HTER? Exploring different human judgments with a tunable MT metric,” in Proceedings of the Fourth Workshop on Statistical Machine Translation, Athens, Greece, March 2009, pp. 259–268.
   Google Scholar
• C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, Proceedings of the ACL-04 Workshop, Barcelona, Spain, 2004, pp. 74–81.
   Google Scholar
• W. Billy, and C. Kit, “ATEC: automatic evaluation of machine translation via word choice and word order,” Mach. Transl., Vol. 23, no. 2–3, pp. 141–155, 2009. doi:10.1007/s10590-009-9061-x
   Google Scholar
• C. Boxing, R. Kuhn, and S. Larkin, “Port: A precision-order-recall MT evaluation metric for tuning,” in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 2012, pp. 930–939.
   Google Scholar
• GTM Link. https://nlp.cs.nyu.edu/GTM/.
   Google Scholar
• B. Satanjeev, and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, 2005, pp. 65–72.
   Google Scholar
• G. Ankush, S. Venkatapathy, and R. Sangal, “METEOR-Hindi: Automatic MT evaluation metric for Hindi as a target,” in Proceedings of ICON-2010: 8th International Conference on Natural Language Processing. India: Macmillan Publishers, 2010.
   Google Scholar
• N. Tomer, D. Sinha, and K. Rai, “Evaluating BLEU metric for English to Hindi language using ManTra machine translation engine,” Int. J. Adv. Res. Comput. Sci., Vol. 3, no. 6, 2012.
   Google Scholar
• N. Tomer, and D. Sinha, “Evaluating NIST metric for English to Hindi language using ManTra machine translation engine,” Int J Adv. Res. Comput. Eng. & Technol. (IJARCET), Vol. 1, no. 8, 2012.
   Google Scholar
• P. Maja, “Chrf: Character n-gram F-score for automatic MT evaluation,” in Proceedings of the Tenth Workshop on Statistical Machine Translation, 2015, pp. 392–395.
   Google Scholar
• S. Miloš, and K. Sima’an, “Evaluating MT systems with BEER,” The Prague Bull. Math. Ling., Vol. 104, no. 1, pp. 17–26, 2015. doi:10.1515/pralin-2015-0010
   Google Scholar
• L. Pairui, C. Chen, W. Zheng, Y. Deng, F. Ye, and Z. Zheng, “STD: An automatic evaluation metric for machine translation based on word embeddings,” IEEE/ACM Trans. Audio, Speech, Lang. Process, Vol. 27, no. 10, pp. 1497–1506, 2019. doi:10.1109/TASLP.2019.2922845
   Web of Science ®Google Scholar
• W. John, T. B. Kirkpatrick, K. Gimpel, and G. Neubig. “Beyond bleu: Training neural machine translation with semantic similarity,” arXiv preprint arXiv:1909.06694, 2019.
   Google Scholar
• Universal dependencies,2020. Available: https://universaldependencies.org/.
   Google Scholar
• H. Matthew, and I. Montani, “Spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing,” To Appear, Vol. 7, no. 1, 2017.
   Google Scholar
• Department of Information Technology. Unified Parts of Speech (POS) standard in Indian languages-Draft standard Version 1.0. Ministry of Communications and Information Technology, Govt. of India, 2018, pp. 1–76.
   Google Scholar
• P. Martha, R. Bhatt, B. Narasimhan, O. Rambow, D. M. Sharma, and F. Xia, “Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure,” in Proceedings of the 7th International Conference on Natural Language Processing, 2009, pp. 14–17.
   Google Scholar
• B. R. Ahmad, R. Bhatt, A. Farudi, P. Klassen, B. Narasimhan, M. Palmer, and O. Rambow. “The Hindi/Urdu treebank project,” in Handbook of linguistic annotation. Dordrecht: Springer, 2017, pp. 659-697.
   Google Scholar
• Universal Dependencies /UD_Hindi-HDTB, Nov. 2020. Available: https://github.com/UniversalDependencies/UD_Hindi-HDTB.
   Google Scholar
• Universal Dependencies/UD_Urdu-UDTB, Nov. 2020. Available: https://github.com/UniversalDependencies/UD_Urdu-UDTB.
   Google Scholar
• Universal Dependencies/UD_Telugu-MTG, , Nov. 2020. Available. https://github.com/UniversalDependencies/UD_Telugu-MTG.
   Google Scholar
• V. Ashish, et al., “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
   Google Scholar
• K. Yunsu, J. Geng, and H. Ney, “Improving unsupervised word-by-word translation with language model and denoising autoencoder,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, 2018, pp. 862–868.
   Google Scholar
• A. Mikel, G. Labaka, E. Agirre, and K. Cho, “Unsupervised neural machine translation,” in Proceedings of the 6th International Conference on Learning Representations, 2018.
   Google Scholar
• Mantra, Sept. 2017, Available: https://www.cdac.in/index.aspx?id=mc_mat_mantra_technology.
   Google Scholar
• A. Kunchukuttan, P. Mehta, and P. Bhattacharyya, “The IIT Bombay English-hindi parallel corpus,” in Proceedings of the Eleventh International Conference on Language Resources and Evaluation, 2018.
   Google Scholar
• P. Matt, C. Callison-Burch, and M. Osborne, “Constructing parallel corpora for six Indian languages via crowdsourcing,” in Proceedings of the Seventh Workshop on Statistical Machine Translation, 2012, pp. 401–409.
   Google Scholar