Deep Insights of Erroneous Bengali–English Code-Mixed Bilingual Language

ABSTRACT

Code-switching, the combination of more than one language within a single utterance is popular in social media sites for informal communication, generating a huge amount of data, though not fully analysed for knowledge extraction due to lack of documentation. Moreover, the code-mixed data is erroneous, and to detect and correct different types of errors, the model requires large erroneous code-mixed language data, not publicly available. This paper first defines generic rules to write Bengali–English code-mixed language in English script, considering the inherent complexity of the Bengali language. Different types of typographical and cognitive errors are induced to obtain the huge erroneous data, based on human behaviour and perception as consulted with the language experts. The errors considered here are applicable to other code-mixed Indic languages and would be beneficial for the researchers. To demonstrate the applicability of the model, we have included the errors in the Hindi–English code-mixed Indic language. An attention-based two-level deep network architecture (uses LSTM as a basic unit) is used for error detection, correction, and translation of code-mixed sentences into a monolingual sentence. Results are reported in terms of accuracy, ROUGE score, and BLEU scores at word level and sentence level for both Bengali–English and Hindi–English code-mixed languages.

Keywords:

• Attention-based Encoder–Decoder model
• Bengali–English code mixed language
• Code-switching language
• Hindi–English code mixed language
• LSTM
• Mixed language
• Word-level and sentence-level error

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Isha Ganguli

Isha Ganguli is presently working as PhD research scholar in IIEST, Shibpur, India. Her areas of interest are text mining, deep learning etc.

Rajat Subhra Bhowmick

Rajat Subhra Bhowmik is presently working as PhD scholar in IIEST, Shibpur, India. His areas of interest are data mining, deep learning, etc. Email: [email protected]

Jaya Sil

Jaya Sil is presently working as professor in IIEST, Shibpur, India. Her domain of interest includes artificial intelligence, image processing and computer vision, bio-informatics, etc. Email: [email protected]