Towards Malay named entity recognition: an open-source dataset and a multi-task framework

Abstract

Named entity recognition (NER) is a key component of many natural language processing (NLP) applications. The majority of advanced research, however, has not been widely applied to low-resource languages represented by Malay due to the data-hungry problem. In this paper, we present a system for building a Malay NER dataset (MS-NER) of 20,146 sentences through labelled datasets of homologous languages and iterative optimisation. Additionally, we propose a Multi-Task framework, namely MTBR, to integrate boundary information more effectively for NER. Specifically, boundary detection is treated as an auxiliary task and an enhanced Bidirectional Revision module with a gated ignoring mechanism is proposed to undertake conditional label transfer. This can reduce error propagation by the auxiliary task. We conduct extensive experiments on Malay, Indonesian, and English. Experimental results show that MTBR could achieve competitive performance and tends to outperform multiple baselines. The constructed dataset and model would be made available to the public as a new, reliable benchmark for Malay NER.

Keywords:

• Malay
• named entity recognition
• dataset
• multi-task learning
• Bi-revision

1. Introduction

Named Entity Recognition (NER) plays an essential role in multiple downstream NLP applications such as information retrieval (Guo et al., 2009), question answering (Aliod et al., 2006), and knowledge graph construction (Etzioni et al., 2005). It is intended to identify the entity boundaries and classify them into predefined categories (such as person, location, organisation, etc.). With the widespread application of neural networks recently, various neural approaches such as bidirectional long short-term memory (Bi-LSTM) (Ma & Hovy, 2016), convolutional neural network (CNN) (Chiu & Nichols, 2016), or recently pre-trained language models (PLMs) (Akbik et al., 2019, 2018) for NER have been proposed. Looking at the current development of NER, there are two main challenges:

• Dependence on massive labelled data: neural NER models are highly successful for languages/domains with a large amount of labelled data. However, most low-resource languages (such as Malay, Tamil, etc.) do not have enough labelled data to train fully supervised models (Lin et al., 2020).

• Boundary recognition error (BRE): One of the significant elements influencing NER performance is BRE (Li et al., 2021). In our preliminary experiments, we found that the current NER models are insufficient in recognising the entity boundaries (especially for long entities). Therefore, a solution to the BRE problem is urgently needed.

As the official language of Malaysia, Brunei, and Singapore, Malay belongs to Malayo-Polynesian branch in the Austronesian language family. According to Word tips¹, the total number of Malay speakers is over 19 million, of which 16 million are native speakers. Faced with massive raw data in Malay, it is particularly significant to utilise NLP technology to extract pattern information. Unfortunately, Malay is a low-resource language with scarce labelled resource and under-developed NLP technology. Malay NER applications are insufficient for the following reasons:

• Lack of large NER datasets: The Malay NER dataset construction was the subject of some earlier studies (Asmai et al., 2018; Ulanganathan et al., 2017). However, to the best of our knowledge, none of them is publicly available, and the data sizes are relatively small. In addition, fully manually labelled datasets are time and expert resource intensive and difficult to be widely used in low-resource languages.

• Poor development of NER techniques: Since neural techniques rely heavily on abundant labelled data, as discussed above. Malay possesses quite limited labelled resources, therefore thwarts the neural methods applied to Malay (Morsidi et al., 2015). Due to the data-hungry restriction, the majority of current studies are still based on rules and machine learning(ML).

Facing the challenges above, it is necessary to explore an effective dataset construction method with automated strategies to alleviate the cost of manual annotation and to build neural NER models based on the constructed datasets for Malay language. In terms of dataset construction, given some specific cases that some low-resource languages have little labelled resources, while their homologous languages have certain labelled resources, and there are significant similarities between homologous languages such as Spanish-Portuguese and Indonesian-Malay (Ranaivo-Malançon, 2006), a potential dataset construction method is borrowing from the datasets of homologous languages. To that end, we propose a dataset construction framework for Malay NER. 20,146 sentences make up our Malay NER dataset (MS-NER). The framework is divided into two components specifically: (1) preliminary dataset generation using homologous-language labelled datasets and rules, and (2) iterative optimisation, a semi-manual technique to iteratively optimise the dataset using NER models and manual audition. This is a trade-off method that could reduce the overhead of manual annotation compared with the fully manual construction method and ensure the dataset quality to a certain degree by means of manual audition compared with fully automated methods. It could potentially work for the languages whose homologous languages have labelled datasets.

As for Malay NER models, based on the constructed dataset, we first apply multiple advanced neural methods for Malay NER and systematically compare and analyse their performance. In addition, to alleviate the negative impact brought by the BRE, we propose a neural multi-task (MT) framework with a bidirectional revision (Bi-revision) mechanism (MTBR). Instead of auxiliary features, we treat boundary detection (BD) as an auxiliary task and take advantage of the relationships between two tasks in a more sophisticated and clever manner: (1) multi-task learning, which has a regularisation impact that can effectively reduce overfitting to NER, provides general representations of both tasks. (2) we leverage explicit boundary information to guide the model to locate and classify entities precisely. Specifically, the label probabilities of the BD task would explicitly revise that of the NER task (main task). However, one main concern is that BD prediction errors may negatively affect the NER outputs. To tackle the error propagation issue, we introduce a gated ignoring mechanism by using the hidden features of the NER module to determine the degree of the BD revision. Gated mechanism is widely used in multiple applications (Zhao et al., 2022). Furthermore, a random probability is utilised to control whether the gate mechanism would take effect.

The following are this paper's main contributions:

1. A dataset construction framework using homologous-language labelled datasets and iterative optimisation is proposed for Malay NER. The framework could be potentially applied to some specific homologous language pairs.

2. A large Malay NER dataset (MS-NER), which is composed of 20,146 sentences, is constructed and would be publicly available as a benchmark dataset.

3. A neural multi-task (MT) framework with a bidirectional revision (Bi-revision) mechanism is presented to tackle the Malay NER task (MTBR). MTBR achieves competitive performance compared with multiple single-task and multi-task baselines on MS-NER. Besides, we also evaluate MTBR in more languages, including a representative high-resource language English and a Malay-related language Indonesian, to show its effectiveness.

The structure of this paper is as follows: The related works of the NER dataset and technique for high-resource and low-resource languages are highlighted in Section 2 correspondingly. The procedure for constructing the MS-NER dataset is thoroughly described in Section 3. The specifics of the MTBR framework are discussed in Section 4. Section 5 elaborates on the experimental setup and analyses the results. Finally, Section 6 draws conclusions.

2. Related work

We briefly cover the work associated with this research in this section. We first review the well-known datasets of resource-rich languages and the construction methods for low-resource language datasets. Then we introduce some currently widely used and effective NER models and review some approaches for low-resource NER. Finally, we review some related works on Malay NER.

2.1. Dataset

2.1.1. Well-known NER datasets

Over recent years, quite a few NER datasets have been proposed. Here are some widely used datasets:

• CoNLL-2003 (Sang & Meulder, 2003) is considered to be one of the most widely used NER datasets for English and German. The dataset comes from news sentences on Reuters RCV1 corpus and includes four coarse-grained entity types: person, location, organisation, and miscellaneous;

• OntoNotes (Hovy et al., 2006) is a fairly large dataset for multiple languages. It consists of many tagsets including parse trees, relation, word sense disambiguation, and entity types. As for the NER task, the OntoNotes dataset is composed of 18 fine-grained entity types such as organisation, location, percentage, date, etc. It is one of the largest and most difficult benchmark datasets for NER due to the diverse range of data sources, which consists of over 2,945,000 tokens in total.

• WNUT2017 shared task (Derczynski et al., 2017) focuses on identifying peculiar, never-before-seen entities in the context of developing debates. It focuses on six entity types: location, person, group, corporation, product, and creative work.

These datasets are completely manually annotated, mainly for high-resource languages. However, building an entire-manually labelled dataset takes time and a lot of specialised resources. It is difficult to widely apply to resource-poor languages. To reduce the expense of manual annotation, it is required to investigate an efficient dataset construction method with automated techniques.

2.1.2. Low-resource NER datasets

To reduce the workload of manual annotation, one ideal solution is to automatically annotate the dataset. For example, a method is proposed by Menezes et al. (2019) to automatically annotate a labelled dataset for NER by DBpedia and Wikipedia links and structured data. Due to their enormous size, the resulting dataset has a total of 87,769,158 tokens. In addition, distant supervision (Mintz et al., 2009) is also an effective way to automatically construct labelled datasets by making use of domain gazetteers. Alfina et al. (2016) proposed a dataset construction framework based on distant supervision for Indonesian NER. They first annotate a seed NER dataset with the gazetteer in Indonesian DBpedia and then propose some rules to modify the noisy labels in the seed dataset. The ANEA (Hedderich et al., 2021) is another tool that provides the functionality to use distant supervision approaches in practice for many languages and named entity types. It also proposes a framework to iteratively improve the dataset and model during model training. Though automatically annotating datasets can sufficiently reduce the annotation load, some noises may exist in these fully automatic datasets without manual verification.

Tamil and Indonesian are representative Malay-related languages. Specifically, Tamil is a low-resource language which is also spoken in Singapore and Malaysia. In order to develop benchmark datasets for Indian Languages, AU-KBC conducted NER-shared work in Forum for Information Retrieval for Evaluation (FIRE) 2013 and 2014². The dataset is publicly available in English as well as in 4 Indian languages, including Bengali, Hindi, Malayalam, and Tamil. The NER benchmark for Tamil consists of 6096 and 1318 sentences, respectively, for training and testing. As for Indonesian, some studies present their efforts on Indonesian NER dataset construction with a series of automatic strategies (Alfina et al., 2016, 2017; Luthfi et al., 2014). The final modified dataset is comprised of (training, development, test) sets with (19000, 1240, 737) sentences. In addition, Fu et al. (2021) propose a method, which is based on distant supervision and manual audit, to build a large Indonesian NER dataset with 50098 sentences. Since Malay and Indonesian come from the same language family and share a high language similarity, we can potentially construct a Malay NER dataset based on current large Indonesian NER datasets.

2.1.3. Malay NER datasets

Recently, some works have presented their effort to Malay NER dataset construction. Nevertheless, to the best of our knowledge, their quality and size are not satisfactory enough to support the development of neural technology. Additionally, none of them is released in the community. Table 1 shows the detailed dataset statistics. “/” indicates that the information is not available.

Table 1. Malay NER datasets.

Datasets	Sen Num	Tok Num	Entity type	Data source
Asmai et al. (2018)	17,527	/	5	Crime
Salleh et al. (2017)	/	/	4	News
Ulanganathan et al. (2017)	/	275,322	4	/
Zamin and Bakar (2015)	/	16,061	3	News
Zamin et al. (2013)	263	5413	3	News

Taking the above issues into account, this work seeks to construct a labelled dataset for Malay by automated means, while introducing a small amount of expert effort to ensure the quality of the dataset. When investigating related Malay language datasets, we found an interesting phenomenon that the Malay language have little labelled resource, while the Indonesian language (a homologous language of Malay) have certain labelled resources (Alfina et al., 2016, 2017; Fu et al., 2021), and there are significant similarities between them. Therefore, a potential dataset construction method is borrowing from the datasets of Indonesian. This method can also be extended to other homologous languages.

2.2. Method

We first discuss some existing NER models in this subsection, respectively, for high-resource and low-resource languages. Our next step is to do a thorough literature review on Malay NER.

2.2.1. NER for high-resource language

There has been an increase in NER research in recent years. Conditional Random Fields (CRF) (McCallum, 2003) and Hidden Markov Models (Zhou & Su, 2002) within handcrafted features are examples of traditional ML approaches to NER. Modern neural NER methods combine character embeddings produced from CNN layer or Bi-LSTM layers with pre-trained word embeddings (Mikolov et al., 2013; Pennington et al., 2014). According to Lample et al. (2016) and Ma and Hovy (2016), a Bi-LSTM layer receives these features before perhaps being followed by a CRF layer.

Recently, fine-tuned pre-trained language models (PLMs) such as ELMO (Akbik et al., 2018) and BERT (Devlin et al., 2019) are also popular for NER. These models may encode semantic and syntactic information of tokens in the context. Later, NER is carried out using these context-aware embeddings. In particular, these PLMs are connected to a softmax classifier to perform entity label classification.

Most of these methods are evaluated in large-size CoNLL-2003 and OntoNotes datasets. However, the current NER models treat NER as a sequence labelling task without explicit use of entity boundary information, and hence they are insufficient in recognising the entity boundaries, especially for long entities to a certain degree.

2.2.2. NER for low-resource language

As mentioned above that most languages, especially low-resource languages, do not have enough labelled data to train such fully supervised models. Most of the NER models for low-resource are based on data augmentation, distant supervision, or cross-lingual transfer (Hedderich et al., 2021).

Data augmentation can modify features on the existing samples without changing the label, thereby obtaining new samples. There are multiple pieces of research being proposed for low-resource NER with data augmentation techniques (Dai & Adel, 2020), such as word replacement (Wei & Zou, 2019), mention replacement (Raiman & Miller, 2017), swap words (Wei & Zou, 2019), generative models (Xia et al., 2019), etc.

Cross-lingual NER has recently been put out as a viable option to effectively address the data-hungry issue for NER with limited resources. This method attempts to transfer knowledge from a source language, which is typically a high-resource language with a wealth of labelled data, to a target language, which is typically a low-resource language with little to no labelled data. Cross-lingual NER techniques can roughly be divided into three categories: knowledge distillation (KD)-based, data transfer-based, and model transfer-based.

• Data transfer-based approach aims to build a pseudo labelled dataset for a target language by projecting alignment information such as parallel resources and machine translation from a source language (Jain et al., 2019; Mayhew et al., 2017; Ni et al., 2017). Then, the target NER model is trained on this pseudo labelled dataset.

• Model transfer-based approach aims to directly apply the source-language trained model with language-independent features to target language (Keung et al., 2019; Wu & Dredze, 2019; Zirikly, 2015).

• Recently, some works extend knowledge distillation (KD) in cross-lingual NER based on a teacher-student training framework (Wu et al., 2020a, 2020b). The idea is to produce pseudo labels for target language data using a source-language NER model without external resources.

Due to huge language differences (such as word order, grammar, etc.), cross-language NER seems difficult to achieve satisfactory results in target languages.

As for the NER task of Tamil and Indonesian, open-source benchmark datasets facilitate the use of supervised methods. For Tamil NER, the FIRE 2014 shared task attracts multiple systems such as CRF (Prabhakar et al., 2014), SVM, and their combination(Abinaya et al., 2014). Naive Bayes is also used for Tamil NER (Srinivasan & Subalalitha, 2019). Recently, Anbukkarasi et al. (2022) employ deep learning technology and construct a GRU model based on the Multilingual Universal Sentence Encoder for Tamil NER. The abundance of NER resources in Indonesian has brought about the widespread use of neural NER techniques. Specifically, Gunawan et al. (2018) construct a hybrid Bi-LSTM and CNN model for Indonesian NER. The performance of neural models incorporating word- and character-level features in Indonesian conversational texts is examined by Kurniawan and Louvan (2018). Besides, transfer learning is recently introduced to tackle the task. To enhance the performance of Indonesian NER, Ikhwantri (2019) suggests fine-tuning PLMs from high-resource to low-resource languages. Kosasih and Khodra (2018) explore a transfer learning framework to transfer knowledge from a part-of-speech tagging system to a NER system.

2.2.3. Malay NER

Most current research on Malay NER is based on rules and ML. Alfred et al. (2014) utilise a rule-based methodology to recognise named entities in Malay texts. This work centre on designing rules based on the POS tagging features and contextual features. Ulanganathan et al. (2017) use CRF to construct a Malay language NER engine (Mi-NER) based on a POS tagger. Furthermore, Sulaiman et al. (2017) explore using Stanford NER and Illinois NER technologies to find Malay-named entities. By combining fuzzy c-means and the K-nearest neighbours algorithm, Asmai et al. (2018) propose an improved Malay NER model for crime texts. Salleh et al. (2017) propose a conceptual model of Automatic Malay NER that uses the CRF method to identify entities from unstructured Malay text data. By using a projection technique to project entity labels from an English corpus to a Malay corpus, Zamin and Bakar (2015) conduct cross-lingual transfer from English to Malay. The Dice Coefficient function and the bigram scoring technique with domain-specific rules are used for projection. Sharum et al. (2011) propose to recognise names in unstructured Malay text documents based on POS features and semantic rules.

The models above are all based on different non-public datasets, and there is no authoritative benchmark for the Malay NER task. This paper aims to develop an effective dataset construction method to relieve the annotation workload for Malay language. In addition, based on the constructed dataset, an effective neural model based on multi-task learning for Malay NER is explored. The dataset and model have the potential to set up a new baseline for follow-up research.

3. Dataset construction

The foundation for the advancement of NLP technology is high-quality datasets. However, as far as we are aware, there are no NER datasets in Malay that are open-source. Additionally, fully human-annotated NER datasets are costly and time-consuming, and are therefore only in a small amount. In order to lessen the load of manual annotation, a semi-manual strategy is introduced to build a Malay NER dataset.

3.1. Dataset design

Indonesian and Malay are close languages that have high lexical similarity (Ranaivo-Malançon, 2006). Based on this language feature, we propose to construct a Malay preliminary dataset from the existing Indonesian datasets (Alfina et al., 2016, 2017; Fu et al., 2021). In addition, some rules Alfred et al. (2014) are leveraged to detect entities in Malay texts to expand the preliminary dataset. After that, a semi-manual method is proposed to iteratively update the dataset and the model (introduced in Section 4).

3.2. Data sources

We build the dataset using Malay new articles. We crawl articles from Malay news websites that include articles on a variety of subjects, such as politics, economics, society, military, etc. The websites are displayed in Table 2. We separate paragraphs into individual sentences and then choose 30,000 sentences at random for annotation. Besides, we construct a large Malay vocabulary with 81,691 tokens from the news articles.

Table 2. Data source.

Website	URL
Bharian	http://www.bharian.com.my/
Utusan	http://www.utusan.com.my/
Malaysiakini	http://www.malaysiakini.com/bm

3.3. Dataset construction

As shown in Figure 1, the construction process consists of two parts: preliminary construction and iterative optimisation.

1. Collect the existing Indonesian NER datasets and separate them into individual sentences. Search whether all tokens of each sentence exist in the Malay dictionary. If so, add the sentence to the preliminary dataset; otherwise, discard it.

2. Expand the preliminary dataset within the unlabelled Malay sentences according to the rules. Only the sentences having entities are considered.

Figure 1. Dataset construction process. It consists of two parts: preliminary construction and iterative optimisation.

Through the above two steps, a seed dataset is constructed consisting of 20,146 sentences. However, the dataset constructed in these ways often suffers from mislabelling errors. Therefore, we propose the following steps to optimise the dataset.

1. Use all labelled sentences for both model training and testing.

2. Manually audit the sentences with different labels in two stages. Two auditors are hired to audit each sentence in order to guarantee the audited sentence quality. Ask a different auditor to review the sentences if two audit results dispute. The annotators are native speakers of the Malay language or have a post-graduate degree in Malay. The audit guideline is shown in Table 3.

3. Re-train the model with the audited dataset.

4. Repeat steps (1)–(3) until the dataset and the model converge.

Table 3. Audit guideline.

Entity	Guideline	Example
Person	Include punctuation in the middle of names.	Edry D'putra Hayadi
	Fictional characters referred to by name are included.	Latif Purnama
	Pronouns, reflexive pronouns, name prefixes are not included.	Mr., You, Me
Location	Include punctuation in the middle of names.	Madinat al-Zahra
	Fictional locations referred to by name are included.	Negara Terbang
	Geopolitical entities (such as city, state, country, continent, etc.) are included.	Kuala Lumpur
	Generic references are not included.	lapangan terbang
Organization	Include punctuation in the middle of names.	Al-Qaeda
	Fictional organisations referred to by name are included.	Tentera Udara Diraja Terbang
	Products or branches are not included.	Ghee Hiang
O	Tokens that do not belong to person, location, and organisation are included.	ketika

3.4. Analysis of MS-NER

The dataset contains 20,146 sentences with three entity categories: person (e.g. Sabrina Shawali), location (e.g. Brunei Darussalam), and organisation (e.g. UMNO Bahagian Besut). In the actual setup, the iteration number is set as 3 because there is no difference between the results of the dataset in the training and testing phases after 3 iterations, which in turn indicates that the dataset construction process has converged. We randomly choose 2,000 sentences from the corpus for manual verification to assess the corpus's quality, and the labelling error rate was 0.1 %.

Building seed datasets from existing datasets in the homogeneous language and semi-manual iterative optimisation strategy can greatly increase the speed of dataset annotation and reduce the cost required for manual annotation. Thus, it has the potential to migrate to specific language pairs and domain pairs in the industry. One prerequisite, which tends to be a concern, is the existence of a homogenous language for the target language and the availability of relevant labelled data for that homogenous language. Also, the similarity between the target language and the homogenous language needs to be high enough to ensure the quality of the seed dataset to a certain extent.

As for experiments, the dataset is (80%, 10%, 10%) divided into training, validation, and testing for subsequent experiments in BIO format. A labelled example is presented in Table 4.

Table 4. An example of MS-NER.

Token	Label
Namun	O
,	O
satu	O
insiden	O
berlaku	O
di	O
Lapangan	B-LOC
Terbang	I-LOC
Incheon	I-LOC
menemukan	O
Bella	B-PER
dengan	O
seorang	O
lelaki	O
kacukan	O
Korea	B-LOC
iaitu	O
Danial	B-PER
Lee	I-PER
Jae	I-PER
Hwan	I-PER
.	O

4. Model

4.1. Overview

Consider a sentence S with L tokens ${w_i}_{i=1}^L$, we, respectively, assign each token $w_i$ with start label $y_s^i$ and end label $y_e^i$ for the BD task and entity label $y_{ner}^i$ for the NER task. In addition, we introduce a span classification for the BD module to assign one of four entity labels (PER, LOC, ORG, and O) $y_{au\mathrm{\_}ner}$ for span representations.

Following the overview in Figure 2, MTBR in this paper is a multi-task model consisting of two modules, respectively, for BD and NER tasks. The BD module is meant to recognise the entity spans with two classifiers (respectively, for start and end predictions), along with another classifier that classifies the spans as corresponding entity labels. The NER module is to predict the NER labels for each token. The token representation from the encoder is universally shared in the downstream two tasks. In addition to implicitly improving the contextual representation of NER through the vanilla multi-task framework, the span classifier is designed to explicitly revise the NER outputs through a gated ignoring mechanism. Two tasks are trained simultaneously.

Figure 2. The MTBR framework structure. Due to space limitation, [B-P, I-P, B-L, I-L, B-O, I-O, O] in the figure represents [B-PER, I-PER, B-LOC, I-LOC, B-ORG, I-ORG, OTHER].

4.2. Encoder

The pre-trained BERT (Devlin et al., 2019) is utilised as our encoder for contextual representations $h_i$. It is noted that other network structures such as LSTM are also suitable for the encoder here. It is formulated as: (1) $h_i=BERT(w_i)$(1) After obtaining the token representations from the encoder, we apply two separate MLPs to create different representations $(h_i^{bd}/h_i^{ner})$ for the (BD/NER) modules. (2a) $\begin{array}{rl}& h_i^{bd}=ML{P}_{bd}(h_i)\end{array}$(2a) (2b) $\begin{array}{rl}& h_i^{ner}=ML{P}_{ner}(h_i)\end{array}$(2b)

4.3. BD module

4.3.1. Standard BD

We use two token-wise classifiers to predict the start and end positions of entities. The contextual representation $h_i^{bd}$ is sent to an MLP classifier for boundary label prediction of $w_i$. (3a) $\begin{array}{rl}& p_s^i=softmax(W_s\cdot h_i^{bd}+b_s)\end{array}$(3a) (3b) $\begin{array}{rl}& p_e^i=softmax(W_e\cdot h_i^{bd}+b_e)\end{array}$(3b) where $W_s$ and $W_e$ are fully connected matrices, $b_s$ and $b_e$ are bias vectors.

4.3.2. Span classification

After obtaining the entity boundaries, to better prompt the training of the NER task, we further introduce a task that classifies the spans as corresponding entity labels. We define consecutive tokens between the nearest pair of the start and end boundaries as a to-be-label entity and the tokens outside the boundaries as non-entities. Assuming that there are K spans in sentence S, we calculate the k-th summarised span representation and the label by averaging the task-specific representations $v_{sp}^k$ in their corresponding boundaries $(i,j)$. (4) $v_{sp}^k=\frac{1}{j-i+1}\sum _{t=i}^j{h}_t^{bd}$(4) Later, the entity representation $v_{sp}^k$ is fed into an MLP classifier to predict its entity tag. (5) $p_{sp}^k=softmax(W_{sp}\cdot v_{sp}^k+b_{sp})$(5) where $W_{sp}$ and $b_{sp}$ are connected matric and bias vectors, respectively.

4.3.3. Loss

For the BD task, we minimise two cross-entropy (CE) losses of the start and end boundaries ${\mathcal{L}}_{bd}={\mathcal{L}}_{bd}^s+{\mathcal{L}}_{bd}^e$ and a CE loss ${\mathcal{L}}_{sp}$ for span classification. The weighted sum of two losses is the total loss, and $w_i$ in this paper is 0.5. (6a) $\begin{array}{rl}& {\mathcal{L}}_{bd}^s=\frac{1}{L}\sum _{i=1}^{L}CE(p_s^i,y_s^i)\end{array}$(6a) (6b) $\begin{array}{rl}& {\mathcal{L}}_{bd}^e=\frac{1}{L}\sum _{i=1}^{L}CE(p_e^i,y_e^i)\end{array}$(6b) (7) $\begin{array}{rl}& L_{sp}=\frac{1}{K}\sum _{k=1}^{K}CE(p_{sp}^k,y_{au\mathrm{\_}ner}^k)\end{array}$(7) (8) $\begin{array}{rl}& {\mathcal{L}}_1=w_1{\mathcal{L}}_{bd}+(1-w_1){\mathcal{L}}_{sp}\end{array}$(8)

4.4. NER module

4.4.1. Standard NER

The contextual representation $h_i^{ner}$ for NER module is sent to an MLP classifier for entity label prediction of $w_i$. (9) $p_{ner}^i=softmax(W_{ner}\cdot h_i^{ner}+b_{ner})$(9) where $W_{ner}$ is a fully connected matrix, $b_{ner}$ is a bias vector.

4.4.2. Bi-revision mechanism

In addition to implicitly improving the NER performance within the vanilla multi-task framework, we assume that the output probabilities of span classification can further revise the result of the NER module. Conversely, to reduce the error propagation caused by the BD task, the NER module is conducive to verifying the revision of the BD task. Thus, we utilise the label probabilities of the span classification through a gated ignoring mechanism to obtain an adjusted probability for the NER module. Specifically, we first need to align the label probabilities of the span classification and NER tasks, as in Figure 3. For each token, the span classification module outputs the probability value $p_{sp}=[p_{per},p_{loc},p_{org},p_o]$ of $y_{au\mathrm{\_}ner}^i$. Each type of $y_{au\mathrm{\_}ner}^i$ except for Other label corresponds to two $y_{ner}^i$ labels (for instance, PER label corresponds to B-PER and I-PER labels). So, we transform $p_{sp}$ to $p_{sp\mathrm{\_}new}=[p_{b-per},p_{i-per},p_{b-loc},p_{i-loc},p_{b-org},p_{i-org},p_o]$. If $w_i$ is the first token of the detected entity, the values of $(p_{per},p_{loc},p_{org},p_o)$ are assigned to $(p_{b-per},p_{b-loc},p_{b-org},p_o)$, or they are assigned to $(p_{i-per},p_{i-loc},p_{i-org},p_o)$. Only the label probabilities of the predicted spans in the BD module would affect that in the NER module. After obtaining the aligned probability $p_{sp\mathrm{\_}new}^i$, we calculate the revised NER probability as: (10) $p_{ne{r}^{\prime }}^i=p_{ner}^i+p_{sp\mathrm{\_}new}^i$(10) As stated above, we introduce the gated ignoring mechanism to alleviate the error propagation of the BD outputs. It consists of two components: a gate mechanism that determines the degree of probability revision: (11) $\begin{array}{rl}& gat{e}_i=sigmoid(W_g\cdot h_i^{ner}+b_g)\end{array}$(11) (12) $\begin{array}{rl}& p_{ne{r}^{\prime }}^i=p_{ner}^i+gat{e}_i\cdot p_{sp\mathrm{\_}new}^i\end{array}$(12) where $W_g$ is a fully connected matrix, $b_g$ is a bias vector. And a random probability p with a threshold α to control the gate mechanism: (13) $p_{final\mathrm{\_}ner}=\{\begin{array}{ll}{p}_{ner}^i& \mathrm{p}>\alpha \\ p_{ne{r}^{\prime }}^i& \mathrm{otherwise}\end{array}$(13)

Figure 3. Probability alignment. If $w_i$ is the first token of the detected entity, the probabilities would flow towards the black arrow; otherwise they would flow towards the red arrow.

4.4.3. Loss

We minimise a cross-entropy loss ${\mathcal{L}}_2$ on $p_{final\mathrm{\_}ner}$ for NER task. (14) ${\mathcal{L}}_2=\frac{1}{L}\sum _{i=1}^{L}CE(p_{final\mathrm{\_}ner}^i,y_{ner}^i)$(14)

4.5. Training and inference

In each iteration, the proposed model simultaneously trains two tasks (BD and NER) end-to-end. The overall training loss is as follows: (15) $\mathcal{L}={\mathcal{L}}_1+{\mathcal{L}}_2$(15) During inference, we utilise the output in the NER module as the final output.

5. Experiment

5.1. Baseline models

Most current Malay NER models are based on rules and ML. Because of their high cost and poor portability, they are not used as baselines in this paper. We compare MTBR with some advanced NER single-task and multi-task models:

• In NeuralNER (Lample et al., 2016), word features are extracted using a Bi-LSTM layer, and the features are subsequently fed to a CRF layer for entity label prediction.

• A CNN layer is used by CNNs-BiLSTM (Chiu & Nichols, 2016) to extract character features. These features are sent to a Bi-LSTM layer, and then a softmax layer is used to make the final prediction.

• For the CNNs-Bi-LSTM-CRF (Ma & Hovy, 2016) model's final prediction, a CRF layer is used rather than a softmax layer.

• ELMO-softmax (Akbik et al., 2018) uses a pre-trained ELMO model as the encoder to extract context-aware features. These features are passed to a softmax classifier for final prediction.

• BERT-softmax (Devlin et al., 2019)is similar to ELMO-softmax and uses the pre-trained ELMO as the encoder.

• BERT-CRF is similar to BERT-softmax, which uses a CRF layer instead of a softmax classifier for final prediction.

• BERT-MRC (Li et al., 2020) formulate the NER task as a machine reading comprehension (MRC) task and propose a unified MRC framework for different kinds of NER tasks.

• For the purpose of detecting entity boundaries, Bi-LSTM-PN (Li et al., 2021) employs BiLSTM as the encoder and another LSTM with pointer networks as the decoder. The predicted entity chunks are then categorised using a softmax classifier. We re-implement this baseline in accordance with Li et al. (2021) by training two tasks in a multi-task framework and adding one softmax layer for the NER task.

• Another multi-task system called MT-BERT (Zhao et al., 2019) uses pre-trained BERT as the encoder to jointly train a number of tasks in an end-to-end manner. In this study, we use this system to jointly train NER and BD problems.

• MT-MED (Zhao et al., 2019) is another multi-task framework with explicit interaction between multiple tasks. In this paper, we re-implement this framework to jointly train BD and NER tasks.

We also train a GloVe (Pennington et al., 2014) embedding for NeuralNER, CNNs-BiLSTM, and CNNs-Bi-LSTM-CRF as well as an ELMO model for ELMO-softmax with massive Malay news articles. Additionally, Bahasa-BERT³ is utilised as the encoder for all BERT-based models.

5.2. Model setups

Our model is based on PyTorch⁴ with HuggingFace's Transformers⁵ package in one single NVIDIA Quadro RTX 8000. The model setups are shown in Table 5. As for the evaluation metrics, we report the precision, recall, and F1 score of multiple NER models.

Table 5. Model settings.

Model	Component	Parameter	Value
Neural components	CNN	Filter size	3
		Number of filters	30
	Bi-LSTM	Number of units	200
	Optimizer	Optimizer function	SGD
		Learning rate	0.01
	Word embedding	Word embedding	GloVe
		Dimension	300
ELMO	Optimizer	Optimizer function	SGD
		Learning Rate	0.015
		Learning Rate Decay	0.05
BERT	Optimizer	Optimizer function	Adam
		Learning rate	5e−5
		Weight decay	0.001
–	Common Max length	128
		Data source	News

5.3. Main performance

The first section of Table 6 presents the results of multiple previous top-performing single-task NER systems. We can see that some improvements to NER performance are made at the CNN layer for char-level representation. And it seems that adding a CRF layer on Bi-LSTM-CNNs cannot bring significant improvements. Among these previous studies, fine-tuned language models (ELMO, mBERT, and Bahasa-BERT) significantly outperform classic neural models represented by Bi-LSTM-CNNs-CRF. In addition, it is worthwhile to note that ELMO works slightly better than mBERT, we hold the idea that ELMO is Malay-oriented that is pre-trained in Malay news articles whereas mBERT is pre-trained in corpora of multiple languages and focuses on learning language-independent knowledge, leading to insufficient language-specific knowledge for Malay NER. Besides, Bahasa-BERT achieves the best performance in all language models, so we leverage it as our encoder for the follow-up experiments. Among the Bahasa-BERT-based models, Bahasa-BERT-CRF tends to work slightly better than Bahasa-BERT-softmax in this case, and Bahasa-BERT-MRC achieves the best performance in all single-task frameworks.

Table 6. Main performance.

Framework	Method	P (%)	R (%)	F1 (%)
Single-task Model	NeuralNER	88.66	87.31	87.98
	Bi-LSTM-CNNs	89.01	87.72	88.36
	Bi-LSTM-CNNs-CRF	88.73	88.12	88.42
	ELMO-softmax	91.12	89.67	90.39
	mBERT-softmax	91.02	89.55	90.28
	Bahasa-BERT-softmax	91.01	90.11	90.56
	Bahasa-BERT-CRF	90.89	91.22	91.05
	Bahasa-BERT-MRC	91.33	91.02	91.17
Multi-task model	Bi-LSTM-PN	88.25	88.46	88.36
	MT-MED	89.57	88.85	89.20
	MT-BERT	91.80	90.24	91.01
	MTBR (CNNs-Bi-LSTM)	89.91	89.69	89.80
	MTBR (ELMO)	92.97	91.44	92.20
	MTBR (Bahasa-BERT)	92.70	92.01	92.36

Note: • mBERT(https://huggingface.co/bert-base-multilingual-cased)

The second part of Table 6 reports the results of some multi-task models. In most cases, the performances of multi-task models outperform those of single-task models with the same encoder (Bi-LSTM-PN vs Bi-LSTM-CNNs, Bi-LSTM-MT-MED vs Bi-LSTM-CNNs-CRF, MT-BERT vs Bahasa-BERT-softmax). We also experiment with different encoders for the proposed MTBR framework, and the experimental results show that MTBR consistently achieves some performance gains over each single-task model, ranging from 1.44% to 1.81% in F1 score. This illustrates the effectiveness of the BD auxiliary task, which provides knowledge that can significantly improve the NER performance.

Taking a closer look at the multi-task models, we can see that MT-BERT outperforms Bi-LSTM-PN and MT-MED, which should be credited to pre-trained language models represented by BERT. In addition, MTBR(Bi-LSTM-CNNs) has a certain performance improvement of 1.44% and 0.60% in F1 score compared to both Bi-LSTM-PN and MT-MED, indicating that the proposed framework stimulates the potential of the BD task to the NER task in a more advanced interaction way. MTBR(Bahasa-BERT) obtains the best performance by improving more than 1.3% in F1 score over the best baseline model MT-BERT. Because of the novel design of the Bi-revision mechanism, MTBR can utilise not only the general representations of different tasks but also label probabilities of different tasks. Hence, MTBR obtains significant improvements compared with multi-task models and single-task models. Meanwhile, we could see that the MTBR(Bahasa-BERT) tends to outperform multiple baselines, which could be regarded as the new baseline method.

5.4. Ablation study

To evaluate the effectiveness of different components in MTBR, we remove a certain part of the model for ablation study and the results are presented in Table 7. Results show that almost every component can boost the whole MTBR. It seems that MTBR can make full use of the advantages of multi-task learning from different perspectives. Removing each of them would generally lead to a performance drop. Our analysis of the observations is drawn as follows.

1. w/o Boundary Detection: in this experiment, we remove the BD task and MTBR degenerates into a vanilla single-task model. Within the multi-task framework, the BD task contributes to boosting the NER performance with an improvement of 1.81% in F1 score, which should be credited to its BRE correction capability.

2. w/o Bi-revision: in this experiment, we remove the Bi-revision mechanism and the model degenerates into a vanilla multi-task model. This setting causes a performance drop of 1.35 on the F1 score. However, the model could achieve some performance gains compared to the vanilla single-task model. This illustrates that fusing the BD auxiliary task can improve the NER performance, but training both tasks simultaneously without external strategies may prevent the two tasks from interacting sufficiently and cannot fully exploit the potential of the BD task. In contrast, Bi-revision can explicitly fuse the outputs of the two tasks, allowing the BD output to effectively revise the NER output, thus further boosting the NER task.

3. w/o Gated Ignoring Mechanism: in this experiment, we remove the $gat{e}_i$ and let the BD output probability all add up to the NER output probability as Equation (10(10) $p_{ne{r}^{\prime }}^i=p_{ner}^i+p_{sp\mathrm{\_}new}^i$(10) ). This leads to a performance drop of 0.57% in F1 score. The gated ignoring mechanism can verify the correctness of the revision and control the degree of revision of the auxiliary task. It can well alleviate error propagation caused the prediction error in the BD module.

4. w/o random probability: in this experiment, we remove the random probability and use the revised probability all the time as Equation (12(12) $\begin{array}{rl}& p_{ne{r}^{\prime }}^i=p_{ner}^i+gat{e}_i\cdot p_{sp\mathrm{\_}new}^i\end{array}$(12) ). Using the gated ignoring mechanism seems to enhance the model to some extend. Using a random probability to control whether BD revision is applied could further reduce error propagation. However, how BD revision could be more effectively controlled needs further exploration, which is left for future work.

Table 7. Performance of different modules.

Model	P (%)	R (%)	F1 (%)
w/o Boundary detection	91.01	90.11	90.56
w/o Bi-Revision	91.80	90.24	91.01
w/o Gated ignoring mechanism	92.70	90.89	91.79
w/o Random probability	92.66	91.34	92.00
MTBR	92.70	92.01	92.36

5.5. BRE analysis

Since we explicitly introduce an auxiliary task of boundary recognition, BRE tends to decline. Table 8 compares the ratios of BRE in different models. Our proposed MTBR framework with the BD auxiliary task can significantly reduce BRE, thus improving the NER performance.

Table 8. BRE analysis.

Model	BRE (%)
Sigle-task Model	3.53
Standard Multi-task model	2.79
MTBR	1.89

5.6. Case study

Table 9 shows some representative cases in different modules of the proposed MTBR in this paper. From the first and second examples, we can see that the BD module can effectively revise the boundary and label of the entity in the NER module. In the first example, “Tan Sri Dr Mohd Irwan Seregar Abdullah” is an extra-long person entity and feeding it directly in the NER model for prediction would cause boundary detection error. In contrast, the BD module can correctly identify the boundary of the entity because it explicitly learns the boundary information of the entity and then pass this information through the Bi-revision mechanism to the NER model and effectively revise the entity boundaries. And in the second example, the polysemous word “Thailand” is likely to lead to ambiguity i.e. “Thailand” itself refers to a “LOC”, but in “Kelab Wartawan Asing Thailand”, it is part of an “ORG”, but the NER module incorrectly identifies “Thailand” as a “LOC” and identifies “Kelab Wartawan Asing” as an “ORG”, resulting in incorrect identification of the entity boundary, which can be effectively revised by the BD module. Moreover, in the third example, the BD module incorrectly identifies “Brahim's Dewina” as a “PER” and the NER module correctly identifies “Brahim's Dewina” as an “ORG”, if the output of the BD module is simply revised for the output of the NER module, it would cause error transmission. While in MTBR, the “gate” mechanism of the NER module can lower the threshold of transmitting information in the BD module to reject its revision. This demonstrates that our model can revise the output of the NER module through the BD module, while effectively preventing the error transmission of the BD module.

Table 9. Case study. MTBR gives all correct predictions in these cases.

Example1	Glyn Davies kerana kritikan yang dibuatkannya pada satu majlis di Kelab Wartawan Asing Thailand. (Glyn Davies for the criticism he made at an event at the Thai Foreign Journalists Club.)
BD module	Glyn Davies: [PER], Kelab Wartawan Asing Thailand: [ORG]
NER module	Glyn Davies: [PER], Kelab Wartawan Asing: [ORG], Thailand: [LOC]
MTBR	Glyn Davies: [PER], Kelab Wartawan Asing Thailand: [ORG]
Example2	Ketua Setiausaha Perbendaharaan, Tan Sri Dr Mohd Irwan Seregar Abdullah, (Chief Secretary of the Treasury, Tan Sri Dr Mohd Irwan Seregar Abdullah,)
BD module	Tan Sri Dr Mohd Irwan Seregar Abdullah: [PER]
NER module	Tan Sri: [PER], Dr Mohd Irwan Seregar: [PER]
MTBR	Tan Sri Dr Mohd Irwan Seregar Abdullah: [PER]
Example3	Brahim's Dewina dengan kerjasama Baitul Hayati pula akan mengumpul dan mengagihkan makanan berkenaan setiap hari termasuk memasak serta membungkuskannya. (Brahim's Dewina in collaboration with Baitul Hayati will collect and distribute the food every day including cooking and packing it.)
BD module	Brahim's Dewina: [PER], Baitul Hayati: [PER]
NER module	Brahim's Dewina: [ORG], Baitul Hayati: [ORG]
MTBR	Brahim's Dewina: [ORG], Baitul Hayati: [ORG]

5.7. Evaluation on more languages

To verify the effectiveness of the proposed model MTBR, we compare MTBR and the baseline methods on more languages, including a representative high-resource language English (en, CoNLL-2003 dataset) and a Malay-related language Indonesian (id, IDNER dataset). Notably, we use BERT-base-cased⁶ and mBERT, respectively, for English and Indonesian. The F1 scores are shown in Table 10, and we can see that in addition to MS-NER, the proposed MTBR model also works well in two evaluated datasets. Besides, MTBR with BERT as encoder achieves the best performance with an F1 score of 93.36% and 91.13%, respectively, in English and Indonesian. This demonstrates the effectiveness of MTBR and that it can be widely extended to languages other than Malay.

Table 10. MTBR performance on more languages.

Framework	Method	en	id
Single-task Model	NeuralNER	90.94	86.94
	Bi-LSTM-CNNs	91.62	87.44
	Bi-LSTM-CNNs-CRF	91.21	87.84
	ELMO-softmax	92.20	88.71
	BERT-softmax	92.40	88.61
	BERT-CRF	92.60	88.50
	BERT-MRC	93.03	90.02
Multi-task Model	Bi-LSTM-PN	90.32	85.43
	MT-MED	92.01	86.34
	MT-BERT	92.89	90.38
	MTBR (CNNs-Bi-LSTM)	92.09	89.02
	MTBR (ELMO)	92.99	90.87
	MTBR (BERT)	93.36	91.13

6. Conclusion

For Malay named entity recognition (NER), we present a dataset construction framework based on homologous-language labelled datasets and iterative optimisation to construct a Malay NER dataset (MS-NER). The proposed framework tends to work for the languages whose homologous languages have labelled datasets. In addition, previous studies demonstrate the potential benefits of the boundary detection (BD) task for NER. Based on MS-NER, we further explore how the BD task can improve NER performance and propose a neural multi-task framework with a bidirectional revision (Bi-revision) mechanism (MTBR), taking both model transfer and label transfer of BD task into account to improve the NER performance. We also systematically evaluate and analyse some advanced neural NER models widely used for English on MS-NER. Experimental results show that MTBR can obtain competitive performance over multiple baselines in MS-NER. The dataset and model would be made available to the general public as a new benchmark.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Science and Technology Program of Guangzhou [grant number 202002030227] and GDUFS Yunshan Office on Public Opinion, one of the Internet culture demonstration projects of higher education institutions in Guangdong.

Notes

1 https://word.tips/100-most-spoken-languages/

2 https://dl.acm.org/doi/proceedings/10.1145/2824864

3 https://huggingface.co/malay-huggingface/bert-base-bahasa-cased

4 https://pytorch.org/

5 https://github.com/huggingface/transformers

6 https://huggingface.co/bert-base-cased