AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages

Abstract

Machine Translation (MT) depends upon the MT evaluation for comparing several MT systems and gives a measure of their efficiency in terms of translation sentences. Consequently, MT evaluation has become an integral part of the MT procedure and it has led to the development of several MT evaluation metrics which can automatically assess the MT systems. Lexical metrics like BLEU have been mostly used in MT evaluation. However, these metrics poorly represent lexical relationships and impose strict identity matching, leading to less correlation with human evaluation for morphologically rich languages.

To overcome the limitations posed by the BLEU evaluation metric for morphologically rich languages, we propose a MT evaluation score called AdaBLEU that is a modified BLEU evaluation score. The proposed score considers the lexical and syntactical properties for any language including the morphologically rich languages. It considers the Parts-of-Speech tags and Dependency Parsing tags along with the BLEU score of sentences. Our modification to the BLEU score does not require multiple reference sentences for evaluation. The evaluation of the performance of AdaBLEU has been conducted by comparing our proposed metric’s performance with several other evaluation metrices on different test datasets for different morphological languages. Experimental results show an improved performance in the case of the proposed score.

KEYWORDS:

• BLEU score
• Evaluation metrics
• Machine translation
• Machine translation evaluation
• Morphological languages
• Natural language processing

Additional information

Notes on contributors

Shweta Chauhan

Shweta Chauhan obtained her BTech (Hons) in ECE from Shimla University HP and Master of Technology in VLSI design automation and techniques from the National Institute of Technology, Hamirpur in ECE Department. She is pursuing her PhD in neural machine translation and evaluation for morphologically rich and low resource languages from the National Institute of Technology, Hamirpur (Himachal Pradesh). She has two years’ experience in the software industry and seven years of teaching experience.

Philemon Daniel

Philemon Daniel received PhD in electronics and communication engineering (NIT, Hamirpur), MTech in VLSI design (VIT, Vellore), BE in electronics and communication engineering (Bharathidasan University). Daniel has over 13 years of teaching experience at NIT, Hamirpur. Prior to joining NIT, Hamirpur, he worked as design engineer at Sasken Communication Technologies Limited, Bangalore. His research interests include VLSI testing, embedded systems, IoT, image and speech processing, natural language processing, self-driving cars and deep learning architectures. He has mentored many students to win several national competitions. To name some of the few: third in Mentor Graphics Design Contest (2011), first and second in ARM Design Contest (2017), third in Anveshan by Analog Devices (2016). He gives regular talks on hardware and embedded system design, deep learning architectures, ARM processors and applications and similar areas. He was awarded ARM Accredited Microcontroller Engineer (AAME) in 2015. He is the recipient of NVIDIA GPU Grant in 2018. He is responsible for many Industry Institute partnership with Mentor Graphics, Analog Devices, Seimens, GE, Xilinx to name a few. Email id: [email protected]

Archita Mishra

Archita Mishra is pursuing integrated MTech in electronics and communication engineering from National Institute of Technology, Hamirpur. She has been working in the field of NLP and social network analysis using Deep Learning for the last two years. Email id: [email protected]

Abhay Kumar

Abhay Kumar is currently in the final year majoring in electronics and communication engineering from National Institute of Technology, Hamirpur. He has been working for the past one year in the field of natural language processing. Email id: [email protected]