Multi-Paragraph Machine Reading Comprehension with Hybrid Reader over Tables and Text

ABSTRACT

In machine reading comprehension, answers to questions may be found in either text or tables. Previous studies have shown that table-specific pre-trained language models perform well when applied to tables; however, applying such models to input data that consists of both tables and text can be challenging. To address this issue, we introduce the hybrid reader model that can manage both tables and text using a modified K-Adapter architecture for effectively encoding the structured information of tables. The training process infuses knowledge for tabular data into a pre-trained model while retaining its original weights from pre-training. Hence, the pre-trained model is able to learn and utilize table information without sacrificing its previous training. Our proposed hybrid reader model achieved comparable or superior performance to that of a specialized model on the Korean MRC dataset, KorQuAD 2.0, using the provided adapters. Furthermore, we conducted experiments on an additional English MRC dataset and confirmed that our proposed model achieves performance comparable to that of the existing model. Our study indicates that employing a single hybrid model instead of two separate models can require fewer computing resources and less time while achieving comparable or superior performance, especially for techniques that apply projection and adapter.

Introduction

Machine Reading Comprehension (MRC) refers to the ability of a machine to understand and interpret documents and locate and provide specific answers to the given question within the document. When a model finds the answer to a given question in a multi-paragraph input, it goes through the process of selecting the relevant paragraph, processing the selected paragraph, finding an answer span, and choosing the final answer from answer candidates. There are large-scale MRC datasets (Campos et al. 2016; Rajpurkar et al. 2016b) for this purpose. The development and advancement of pre-trained language models (Devlin et al. 2018; Liu et al. 2019) has led to considerable performance improvement in MRC tasks.

MRC can be categorized into various types based on the form of the answer, type, or number of input paragraphs. In some MRC datasets, answers are tagged not only in unstructured text but also in structured text, such as tables and lists. In another case, structured texts such as tables and lists could be used as input paragraphs. However, the performance of the model of structured text is worse than that of unstructured text.

A simple way to improve the performance of semi-structured data is by using the extension structure of the model or pre-training on structured data. Here, if the input paragraph has both the table and text, using separate reader models, which are two models trained for each structured and unstructured data, is required to improve performance. However, these separated reader models increase GPU memory usage and are computationally intensive because they use an additional BERT (Bi-directional Transformers) model for tables. Furthermore, all of the MRC processes are in the model pipeline and are independently preceded. However, using separate readers creates extra processes of verifying the final answer because of the inconsistent outputs from the separate readers. The performance of the MRC model may be compromised when faced with inconsistent outputs from each individual model, resulting in a decrease in the accuracy of the final answer.

Figure 1 shows an example of a KorQuAD 2.0 dataset (Kim et al. 2019). Compared to the KorQuAD 1.0 (Lim, Kim, and Lee 2019), which used only short paragraphs as an answer, the KorQuAD 2.0 used a full Wikipedia document that was too long to process at once. The answers are tagged in tables and lists just as in the text. Hence, to find the correct answer in KorQuAD 2.0, the MRC model must read the semi-structured data.

Figure 1. Example of a paragraph from a full Wikipedia document with a title, text paragraph, and a table.

In this paper, we propose a multi-paragraph MRC model, a hybrid reader, with an adapter layer. The adapter layer minimizes the catastrophe of forgetting knowledge from a natural text in pre-training tasks over tables. The hybrid reader uses a unified reader model for both text and tabular data. We will present the extension of the BERT architecture to encode semi-structured data without particular embedding of tabular data in this research.

Related Work

Machine Reading Comprehension

MRC is advanced with large-scale datasets, such as SQuAD (Rajpurkar, Jia, and Liang 2018), MS Marco, and Trivia QA (Joshi et al. 2017).

In addition to QA datasets focused on machine reading comprehension from text, several QA datasets have been introduced that diverge from traditional QA datasets by necessitating the simultaneous reasoning and extraction of answers from heterogeneous sources, including tables and unstructured texts. HybridQA (Section 2) is a multi-hop question-answering dataset that requires reasoning on tables and texts. This dataset was constructed by utilizing Wikipedia tables and multiple free-form corpora linked with the entities within these tables. TAT-QA (Zhu et al. 2021) and FinQA (Z. Chen et al. 2021) proposed QA datasets over tables and texts from financial reports, which require numerical reasoning to infer the answer. In TAT-QA, the questions are crafted to aggregate both tabular information and text information. FinQA includes questions that require domain-specific knowledge to infer answers. For the Korean language, there are MRC datasets, such as KorQuAD 1.0, KorQuAD 2.0, NIA MRC dataset,¹ KLUE (Park et al. 2021), and (Zhong, Xiong, and Socher 2017). KorQuAD 1.0, NIA MRC, and KLUE-MRC utilized text paragraphs as input context. KorWikiTableQuestions used tabular data as input context.

KorQuAD 1.0 is a dataset developed by benchmarking SQuAD 1.1, utilizing Korean Wikipedia documents. KorQuAD 2.0 represents an expansion of the original dataset KorQuAD 1.0 dataset. While KorQuAD 1.0 utilizes short paragraphs as context, KorQuAD 2.0 provides full Wikipedia documents as context. Full Wikipedia documents typically exceed the token limit of 512, commonly used in transformer-based encoder models. Consequently, a process of selecting or ranking specific paragraphs that are likely to contain answers becomes necessary. The answer spans in KorQuAD 2.0 are tagged not only in unstructured text but also in structured text (such as tables and lists) as well.

In early research stages, MRC models with an attention mechanism that allows the model to focus on a specific area within a query-relevant paragraph of text achieved impressive results. The BiDAF (Seo et al. 2016) achieved promising results with the MRC model using bi-directional attention between query and context. QA-Net (Yu et al. 2018) proposed an MRC model architecture using a convolution neural network and global self-attention without a recurrent neural network to reduce training and inference time. Recently, MRC fine-tuning language models (Zhang, Yang, and Zhao 2021) pre-trained with large-scale raw corpus based on transformers achieved state-of-the-art performance. Dong et al. (2023) introduces unified multi-module end-to-end model for multi-passage MRC, which encompasses passage retrieval, answer extraction, and ranking of multiple answer verifications. This model employs an answer verification method that utilizes the attention mechanism to fuse the features of answer candidates, demonstrating improvement in multi-passage MRC tasks. FinBERT-MRC (Zhang and Zhang 2022) proposed a novel approach to solving MRC problems by formulating financial named entity recognition. This study employs the FinBERT (Yang, Uy, and Huang 2020), a domain-specific pre-trained model, to enhance the capability to identify target entities. Fin-EMRC (Chai et al. 2023) proposed a novel MRC model augmented with structured financial domain knowledge. They employed a new query template scheme to improve the MRC framework’s inference efficiency.

TableQA and Semantic Parsing

A semantic parsing task can be viewed as question answering over structured text. In semantic parsing tasks, the language models that were pre-trained with tables achieved state-of-the-art results. TABERT (Yin et al. 2020) was proposed as a pre-trained language model that learns joint representations for natural language text and structured text. TAPAS (Herzig et al. 2020) extends BERT architecture with additional embeddings for rows, columns, and rankings to capture the tabular structure of tables. TAPAS and TABERT were pre-trained on millions of tables, and their outputs were learned with a masked language model (MLM) objective.

Automatically generated synthetic data is used to inject reasoning skills in pre-training language models. Pruksachatkun et al. (Eisenschlos, Krichene, and Müller 2020) proposed an intermediate pre-training task to predict whether statements generated automatically for a table with context-free grammar are positive or negative. GRAPPA (Yu et al. 2020) constructed synthetic question-SQL pairs over tables with synchronous context-free grammar (SCFG) and pre-trained language models with MLM and SQL semantic prediction (SSP) objectives.

Previous studies on Korean table Question Answering (QA) have been mainly conducted in the form of span prediction for tables. TabQA (Park et al. 2018) proposed an end-to-end neural model to encode table cell features. They constructed table QA datasets consisting of 20,000 tables and 100,000 natural questions related to telephone services. Park et al. (2020) proposed both novice and intermediate Korean TableQA tasks that infer the answer to a question from structured tables to build a question-answering pair. Novice tasks constructed questions with simple rules and templates. In intermediate tasks, they imposed lexical variations and added more complex levels of queries.

HybridQA

In HybridQA, both tabular and text data are used as input context, necessitating aggregating information from both sources to infer answers. W. Chen et al. (2020) a proposed cross-block reader based on sparse attention to aggregate information across top-K text and tabular blocks. TAT-QA (Zhu et al. 2021) introduced a novel QA model, TAGOP, for practical reasoning across tabular and textual data. TAGOP employs a sequence tagging approach to extract relevant cells and text spans, then applies symbolic reasoning combined with aggregation operators to process them. UniRPG (Zhou et al. 2022) used a semantic parsing-based approach, which consists of a neural programmer and a symbolic program executer. The neural programmer generates programs, which the program executor then directly employs for conducting discrete reasoning over tabular and textual data. Nararatwong, Kertkeidkachorn, and Ichise (2022) proposed a QA model that builds upon TAGOP, integrating explicit tabular structures with a graph neural network. The model consists of an evidence extraction module with GNN and an improved numerical reasoning module. FinMath (Li, Ye, and Zhao 2022) proposed the FinMath framework, which enhances the model’s performance on numerical reasoning through multi-step reasoning using a tree-structured neural network.

Adapter

When fine-tuning is performed to inject additional information into a pre-trained model, a catastrophic forgetting problem can occur where knowledge trained during pre-training disappears. K-Adapter (Wang et al. 2020) has been developed to mitigate the issue of catastrophic forgetting during fine-tuning. K-Adapter is a neural adapter that can be connected to pre-trained language models like BERT, enabling the utilization of additional knowledge without the risk of losing pre-trained information.

Proposed Methodology

We propose a multi-paragraph MRC system consisting of two main components: a paragraph ranking model and a paragraph reader model. Figure 2 shows the architecture of the proposed MRC system. The paragraph ranking model selects top-k paragraphs from a full Wikipedia document. The paragraph reader model predicts two things: answer spans and whether input paragraphs include answers or not. The Separated Reader model uses separate models for tables and texts, while the hybrid reader Model uses a unified reader model for both tables and texts.

Figure 2. Architecture of (a) pipeline models with a separated reader, (b) pipeline models with hybrid reader.

Paragraph Ranking

An input document is divided into several parts if the number of input tokens exceeds the maximum length that the reader model can process at once. It makes hard to find answers in the splited several paragraphs. Therefore, it is efficient to rank the paragraphs that are likely to have the correct answer. Thus, we trained the paragraph ranking model to predict the probability of whether the correct answer to the question is in the input paragraph. We use heuristic exact matching (HEM) (Eisenschlos, Krichene, and Müller 2020) as an auxiliary method for paragraph ranking. The HEM score between the query and paragraph is obtained as follows. First, words in the paragraph are broken down into tokens of morphs. To compare the similarity between query and paragraph, nouns are examined since they undergo fewer morphological changes than other types of part of speech, and importantly they convey crucial semantic information in the context. Using the bi-gram features, we computed the Jaccard coefficient, which serves as a HEM score between query and paragraph:

(1) $\mathit{M}=\mathit{jaccard}(T_q,T_p),$(1)

where $T_q,T_p$ indicate the tokens in the query and paragraph, respectively. Answer paragraphs and top-k negative paragraphs are sorted out based on the HEM score with in-batch negatives. Tables are flattened by adding commas to separate each cell. If there is no negative table paragraph corresponding to the positive one, we generate a negative table sample to prevent paragraphs with only text from being selected as a negative sample. Figure 3 shows a method for generating a negative table sample by processing the original tables into positive and negative tables. We use this method if the document has no other tables.

Figure 3. Example of generating negative and positive paragraph table from the original paragraph.

In KorQuAD 2.0, there are short answers with answer spans tagged in a part of a sentence and long answers as well, where the answer spans are tagged on an entire paragraph or several paragraphs. Therefore, in this section, our key challenge is to output the probability of short answers, long answers, and no answers in a number of paragraphs for a given question and rank the paragraphs. As mentioned earlier, the model can process at a maximum length at one time. If the input document exceeds the maximum, it is broken down into smaller units of paragraphs.

When inputting paragraphs into the model, a batch is made up of a paragraph with answers and the rest containing no answers. After inputting the vector, we gain three probability values of short answers, long answers, and no answers for each paragraph in the last hidden vector of the [CLS] token. Then, the probability vector is computed using the SoftMax operation.

(2) $\mathit{p}=\mathit{SoftMax}(\mathit{FFN}(\mathit{h})),$(2)

(3) $\mathit{Los}{s}_p=\mathit{CE}(\mathit{p}),$(3)

where $\mathit{h}$ is the [CLS] token’s final hidden vector output of BERT. $\mathit{p}$ is the probability of whether the answer is a short answer, a long answer, or unanswerable. We calculate $\mathit{Los}{s}_p$ with cross-entropy between two vectors to see how much the resulting vector deviates from the actual correct answer. We also used titles of documents and paragraphs as input and cross-entropy loss to train the ranking model.

Next, we check whether the derived probability vector output can find the position of the paragraph with the answer. We extract the max values from the probabilities of short answers and long answers and reconstruct them into a single vector. Then,

(4) $m_i=\mathit{max}(h_{\mathit{short},\mathit{i}},h_{\mathit{long},\mathit{i}}),$(4)

$h_{\mathit{short},\mathit{i}},h_{\mathit{long},\mathit{i}}$ is the probability of short and long answers of the $\mathit{i}$-$\mathit{th}$ component of the vector of the [CLS] token mentioned above, $m^i\in \mathit{m}$ is a$\mathit{i}$-$\mathit{th}$ vector created by applying the max function of short and long answers in the vector output $\mathit{h}$. When we apply SoftMax operation to $\mathit{m}$, the following equation is derived:

(5) $\mathit{q}=\mathit{SoftMax}(\mathit{m}),$(5)

(6) $\mathit{Los}{s}_m=\mathit{CE}(\mathit{q}),$(6)

$\mathit{q}$ indicates a vector that applies SoftMax operation to vector $\mathit{m}$. Then, we reshaped its result to be the same side as the initial input vector. Cross-entropy is used to compute the $\mathit{Los}{s}_m$ between the correct answer vector and the reshaped vector. $\mathit{m}$ is reshaped to the max probability vector between short answers and long answers.

So far, we have obtained the HEM scores and computed the probabilities of short answers, long answers, and no answers for each paragraph. A score to select the top-k paragraph is computed as illustrated below:

(7) $\mathit{score}=(p_0+p_1)\ast \mathit{M},$(7)

where $\mathit{M}$ is the HEM score between a question and a paragraph morph token, $\mathit{score}$ is the final score for paragraph ranking, $p_0$ is the probability of a short answer, and $p_1$ is the probability of a long answer.

Finally, paragraphs are sorted based on the final score, and the top-5 paragraphs computed by the ranking model are delivered to the reader models to find the correct answer.

Separated Reader Model

In the Separated Reader Model, different reader models are used depending on whether the input paragraph, selected from the ranking model, is structured text or unstructured text. The BERT model is pre-trained with unstructured corpora and we used it as the backbone model. If a table is in the selected paragraph, the table reader model outputs the answer span; if not, the text reader model outputs the answer span.

For the language model to capture a tabular structure, additional pre-training on structured data and special embeddings for tabular data are required. We used an additional embedding of a column, row, and ranking (TAPAS) and an additional pre-training language model with the Masked Language Model (MLM) and SQL semantic prediction (SSP) objective (GraPPa). The MLM predicts the original token replaced with a special token mask. SSP predicts whether a column occurs in the SQL utterance and which operation is triggered.

In this paper, we extend the SSP task to predict cells, rows, and column heads that occur in the SQL utterance. We extract table-utterance pairs from Korean Wikipedia and Namuwiki, and then generate synthetic data using these tables. MLM loss is computed to optimize the model for natural language utterances and SSP is computed to optimize the model for synthetically generated utterances.

Each reader model is fine-tuned independently with KorQuAD 2.0, depending on whether the answer is tagged in the table or text. For unanswerable questions, we set the answer span to point [CLS] token for both start and end span prediction and used it as no-answer probabilities to verify the final answer. Predictions of the separated model are computed as follows:

(8) $h_{\mathit{text}}=\mathit{BER}{T}_{\mathit{TAPAS}}(\mathit{x}),$(8)

(9) $h_{\mathit{table}}=\mathit{BER}{T}_{\mathit{table}}(\mathit{x}),$(9)

(10) $s_{\mathit{table}},e_{\mathit{table}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{table}}(h_{\mathit{table}})),$(10)

(11) $s_{\mathit{text}},e_{\mathit{text}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{text}}(h_{\mathit{text}})),$(11)

(12) $v_i=\left\{\begin{array}{cc}\frac{2-(s_{t\mathrm{able}}^{\prime }+c_{t\mathrm{able}}^{\prime })}{2},& \mathit{if}\mathit{x}=\mathit{table}\\ \frac{2-(s_{t\mathrm{able}}^{\prime }+c_{t\mathrm{able}}^{\prime })}{2},& \mathit{else}\end{array}\right.$(12)

(13) $i^{\ast }=\underset{\mathit{i}\in 0,1,2,\dots ,\mathit{N}}{\mathrm{argmax}}{v}_i\ast M_i$(13)

$h_{\mathit{table}},h_{\mathit{text}}$ are the final hidden states of the language model pre-trained with text data and table data, $s_{\mathit{table}},e_{\mathit{table}}$ and $s_{\mathit{text}},e_{\mathit{text}}$ are the probability of the answer span for start and end. The former probability is selected if a table is in the paragraph otherwise the latter is selected. The term $v_i$ represents the score of the i-th input to verify the answer among answer candidates. The final answer is selected where the candidate answers are extracted from the input with the highest score. $M_i$ represents the HEM score, as described in Section 3.1, between the query and i-th input paragraph. It is particularly useful when the verification of answers becomes challenging due to the similarity of input formats, such as in situations where the inputs include monthly match result tables that have the same head data.

Hybrid Reader Model

While the Separated Reader Model uses different reader models for tables and texts, the hybrid reader model uses a unified reader model for data with an adapter structure. We used two approaches: one without additional pre-training over tables, and the other one with additional pre-training of an adapter structure in the hybrid reader Model over tables. We will refer to the two methods that we have employed as the projection method and the adapter method, respectively (Figure 4).

Figure 4. Architecture of a hybrid reader with projection.

Using the BERT model with additional embedding for tables makes it difficult to use a unified model. Instead, we use a unified pre-trained language model and a separate classification for tables and texts. While we present vector representations of BERT directly to the FeedForward Network (FFN) for the case of text paragraph, we use additional embeddings, such as columns, rows, and rank embedding, to encode semi-structured data and present concatenation of all vectors to FFN.

Additional Embeddings

To acquire column and row representations, we applied a summation operation on the output vectors that corresponded to the same column or row, respectively. Afterward, BERT’s encoder was utilized without any additional alterations. The equation for obtaining column and row embeddings is expressed as follows:

(14) $\mathit{h}=\mathit{BERT}(\mathit{x}),$(14)

(15) $h^r=\mathit{LSTM}(\mathit{h}\cdot r_{\mathit{ids}}),$(15)

(16) $h^c=\mathit{LSTM}(\mathit{h}\cdot c_{\mathit{ids}}),$(16)

(17) ${\hat{h}}^{\mathit{r}}=h^r\cdot {{\mathit{r}}_{\mathit{ids}}}^{\mathit{T}},$(17)

(18) ${\hat{h}}^{\mathit{c}}=h^c\cdot {{\mathit{c}}_{\mathit{ids}}}^{\mathit{T}},$(18)

(19) $\mathit{h}{ }^{\mathrm{\prime }}=\mathit{relu}(W_h[{\hat{h}}^{\mathit{r}};{\hat{h}}^{\mathit{c}};\mathit{h}]),$(19)

where $\mathit{h}\in {\mathbb{R}}^{\mathit{d}\times \mathit{S}}$ is the final hidden state of BERT, ${r_i}^{\mathit{ids}}\in {\mathbb{R}}^{\mathit{S}\times \mathit{R}}$ and ${c_i}^{\mathit{ids}}\in {\mathbb{R}}^{\mathit{S}\times \mathit{C}}$ are row and column vectors that one-hot encoded row and column indexes which indicate the row and column index of input tokens, $\mathit{R}$ is the max row length, $\mathit{C}$ is the max column length. To model the temporal interactions between table cells, we used the bidirectional LSTM (Hochreiter and Schmidhuber 1997) on the column and row representations.

As illustrated in the equation above, we obtain the final hidden states in the output layer, where $h^r$/$h^c$ is a row/column representation vector from LSTM, ${\hat{h}}^{\mathit{r}}$/${\hat{h}}^{\mathit{c}}$ is a row/column representation vector at the indexes of each token input, [;] is the vector concatenation, $W_h$ is the trainable weight matrix, and $\mathit{h}{ }^{\mathrm{\prime }}\in {\mathbb{R}}^{3\mathit{d}\times \mathit{S}}$ is a modified hidden state for structured text.

Attention Restriction

Motivated by (Eisenschlos et al. 2021), we restricted the BERT’s attention heads to consider cell representations in the same rows and columns by masking. By attention mask, $t_i$, a token at the position $\mathit{i}$ can be obtained:

(20) $t_i=\left\{\begin{array}{cc}0,& \mathit{if}\mathit{i}\mathrm{in}\mathbf{Q}\mathrm{or}\mathit{i}\mathrm{in}\mathbf{H}\mathrm{or}{c}_i=c_j\mathrm{or}{r}_i=r_j\\ -\mathrm{\infty },& \mathit{else}\end{array}\right.$(20)

where $\mathit{Q}$ is a query set, $\mathit{H}$ is the head of columns and rows, and $c_i,r_i$ are the columns and rows at the $\mathit{i}$-$\mathit{th}$ position.

If the input is a paragraph with a table, only the loss in the table classifier is minimized. Otherwise, only the loss in the text classifier is minimized. Final predictions and losses are computed as:

(21) $s_{\mathit{table}},e_{\mathit{table}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{table}}([\mathit{h}{ }^{\mathrm{\prime }};\mathit{h}])),$(21)

(22) $s_{\mathit{text}},e_{\mathit{text}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{text}}(\mathit{h})),$(22)

(23) $\mathit{Loss}=\mathit{a}(s_{\mathit{table}},e_{\mathit{table}})+(1-\mathit{a})(s_{\mathit{text}},e_{\mathit{text}}),$(23)

$s_{\mathit{table}}$, $e_{\mathit{table}}$ and $s_{\mathit{text}}$, $e_{\mathit{text}}$ are the probability of the answer span for start and end, and $\mathit{a}$ is a constant value which indicates whether input paragraph is table or not.

The answer verification is applied using the same approach described in Section 3.2.

Hybrid Reader with Adapter

For the Unified Reader Model to handle both table and text, we used additional pre-training over tables with an adapter structure. The adapter structure consists of seven adapter layers to capture the tabular structure. Of those adapter layers, four adapter layers obtain input directly from the pre-trained transformer layer and the other three obtain input from previous adapter layers. Motivated by previous work (K-adapter), we added additional transformer layers for pre-training tasks on the table data. As shown in Figure 5, adapter layers consist of two parts and the pre-training over a table is made up of two phases. In the first phase, only the weights of the four adapter layers, which obtain input directly from transformers, are updated. In the second phase, the weights of all adapter layers including the other three adapter layers are updated. The transformers’ weights are permanently fixed after pre-training from unstructured corpora and each of {0,4,8,12}$-\mathit{th}$ transformer layers are plugged into each of the four adapter layers.

Figure 5. Architecture of a hybrid reader with adapter.

Figure 6 shows the architecture of the adapter layer. We extend the transformer layer with the projection layer to capture the tabular structure. We compute column and row embeddings by summing the output representations of the transformer layer without modifying the embedding layer of BERT, as shown in the following equations.

Figure 6. Architecture of adapter layer; row-wise, direct, column-wise projection is concatenated and passed to adapter transformer layer.

(24) $h_i=\mathit{gelu}(W_d[h_{L_i}^{\prime };{\bar{h}}_{\mathit{i}-1}]),$(24)

(25) $h_i^r=\mathit{LSTM}(h_i\cdot r_{\mathit{ids}}),$(25)

(26) $h_i^c=\mathit{LSTM}(h_i\cdot c_{\mathit{ids}}),$(26)

where $h_i$ is the input of $\mathit{i}$-$\mathit{th}$ Adapter Layer, $h_{L_i}^{\prime }$ is the hidden states from the, $L_i$, $\mathit{i}$-$\mathit{th}$ layer of pre-trained transformer and $\bar{h}$ is the output of former adapter layer. In the first adapter layer, special embeddings used in (Herzig et al. 2020) were used as input instead of the former layer’s output.

We obtained the row and column representation, then concatenate them with the transformer layer’s outputs.

(27) $\hat{h}\mathit{ir}=h_i^r\cdot {{\mathit{r}}_{\mathit{ids}}}^{\mathit{T}},$(27)

(28) $\hat{h}\mathit{ic}=h_i^c\cdot {{\mathit{c}}_{\mathit{ids}}}^{\mathit{T}},$(28)

(29) ${\hat{h}}_{\mathit{i}}=\mathit{gelu}(W_o[\hat{h}\mathit{ir};\hat{h}\mathit{ic};\mathit{hi}]),$(29)

(30) ${\bar{h}}_{\mathit{i}}=\mathit{transformer}{s}_i^a(h_{\mathit{i}{\mathit{ }}^{\mathrm{\prime }}}),$(30)

$h_i^r$/$h_i^c$ is the row/column representation vector of the adapter layer, $\hat{h}\mathit{ic}$/$\hat{h}\mathit{ic}$ is the row/column representation vector of index position from the adapter layer, [;] is the vector concatenation, $W_d\in {\mathbb{R}}^{\mathit{h}\times 3\mathit{h}}$ and $W_o\in {\mathbb{R}}^{\mathit{h}\times 3\mathit{h}}$ are the trainable weight of down projection layer, and output of down projection layer $h_{\mathit{i}{\mathit{ }}^{\mathrm{\prime }}}$ is passed to $\mathit{transforme}{r}_{\mathit{adapter}}$, transformer layer at $\mathit{i}$-$\mathit{th}$ adapter layer; then we acquire final hidden states ${\bar{h}}_{\mathit{i}}$.

Details of Pre-training Tasks for Adapter

We use the BERT-base (L = 2, H = 768, A = 12) model as the transformer layer and initialize the adapter layer (N = 4, L = 1, H = 1024, A = 8) randomly. There are two phases in pre-training procedures. In the first pre-training phase, we only train four adapter layers that are plugged from transformer layers. In the second pre-training procedure, weights of all the adapter layers are learned, while the original parameters of transformer layers pre-trained on unstructured corpora are fixed in all phases. Following Yu et al. (2018), we generated queries using synchronous context-free grammar with Korean Wikipedia tables and trained the model with masked language model loss and SQL semantic loss using augmented examples.

Using a unified pre-trained model, we cannot add special embedding or additional pre-training for tables in the hybrid reader model. However, with an adapter, we feed input special embeddings of columns, rows, and rankings to the first adapter layer in the adapter layers.

In the first pre-training phase, we train four adapter layers, as shown in the middle of Figure 5, with MLM and SSP objectives. In the second pre-training phase, we use intermediate (Eisenschlos, Krichene, and Müller 2020) pre-training tasks, table entailment with synthetic data generation, and relevant statement prediction to train all the adapter layers. With these pre-training phases, the weight of the transformer is fixed, which is pre-trained from the text at the very begining.

We generate table-dependent synthetic statements using context-free grammars introduced in Pruksachatkun et al. (Eisenschlos, Krichene, and Müller 2020) to train entailment recognition tasks. We excluded counterfactual statements because of an insufficient number of tables with statements in the Korean Wikipedia. A relevant statement prediction task is added to improve answer verification in the table reader. Statement($\mathit{s}$) and table($\mathit{T}$) pairs present in the same paragraph are collected as positive samples. Statement($\mathit{s}$) and table($\mathit{T}$) pairs in different paragraphs or documents are collected as negative samples.

(31) $\mathit{p}(\mathit{s}|\mathit{T})=\mathit{W}([h_a;h_b;h_c])+\mathit{b},$(31)

(32) $\mathit{Loss}=\mathit{CE}(\mathit{p}(\mathit{s}|\mathit{T})),$(32)

We use BERT and adapter’s [CLS] token representations to model the joint probability of the relevance statement prediction and table entailment tasks. We used binary cross entropy loss for the table entailment and relevant table prediction (RTP) tasks described in Figure 6.

Transformers for Long Sequences with Structured Data

While the previous method uses special embedding for the table on the embedding layer, our model architecture captures the tabular structure with a representation vector from the existing language model and could be applied to long sequence transformers such as Big Bird (Zaheer et al. 2020).

Figure 7 shows the hybrid reader model architecture with long sequence transformer. This model takes the top-K paragraph from the Ranking Model and outputs the hidden vectors for each paragraph. From those hidden vectors, we pick the hidden vectors from the table paragraph, and table architectures in the hybrid reader model were applied to the picked hidden vectors. Encoding structural information with projection is applied as an equations 10(10) $s_{\mathit{table}},e_{\mathit{table}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{table}}(h_{\mathit{table}})),$(10) –15(15) $h^r=\mathit{LSTM}(\mathit{h}\cdot r_{\mathit{ids}}),$(15) .

Figure 7. The architecture of hybrid reader model with projection using long sequence transformer.

Figure 8 illustrates the architecture of the adapter layer, in which the output vector of the long sequence transformer model and the output of the tokens of the question are concatenated and passed to the adapter layer as input. Encoding structural information with an adapter is applied as an equations 21(21) $s_{\mathit{table}},e_{\mathit{table}}\propto \mathit{SoftMax}(\mathit{FF}{N}_{\mathit{table}}([\mathit{h}{ }^{\mathrm{\prime }};\mathit{h}])),$(21) –26(26) $h_i^c=\mathit{LSTM}(h_i\cdot c_{\mathit{ids}}),$(26) . The vectors from the adapter are projected to sequence-wise projection and concatenated to Big Bird representation, and we acquire the final H.’

Figure 8. The architecture of adapter layer for tables.

Experiments

We used a pre-trained $\mathit{bigbir}{d}_{\mathit{base}}$ (Zaheer et al. 2020) transformer model from Huggingface. The $\mathit{bigbir}{d}_{\mathit{base}}$ model is specialized in long sequence lengths and full attention; it is more efficient than the BERT model and can be used while limiting the length of the sequence to 512. To compare our results with those of other models, we implemented the model pre-trained with a masked language model over tables, with additional embedding columns, rows, and ranking following Herzig et al. (2020). Instead of millions of tables used in Herzig et al., we collected 70 thousand tables from Korean Wikipedia.

Dataset

Our model was trained and evaluated using KorQuAD 2.0, a Korean machine reading comprehension dataset that includes both textual and tabular data. The dataset consists of 83,686, 10165, and 9,309 question-answering data for training, development, and testing, respectively. The data is collected from 47,957 documents, including Korean Wikipedia and other web pages. The development dataset comprises 1663, 6877, and 1625 instances of long answers, answers from a text paragraph, and answers from a table, respectively. Since the number of answers from the table is much smaller compared to that of answers from the text, the performance enhancement in a table would not be coupled with the overall performance improvement.

There are two types of answers in KorQuAD 2.0; one is a short answer that is a word or phrase, and the other one is a long answer that tags the whole paragraph as an answer. Answers are tagged not only from the text but also from the table and list. The number of syllables in the long answer is larger than or equal to 240, and 17.8% of KorQuAD 2.0 data are long answers.

In order to conduct experimentation on English QA, four datasets were utilized: SQuAD (Rajpurkar et al. 2016a), WikiSQL (Zhong, Xiong, and Socher 2017), NQ-Tables (Kwiatkowski et al. 2019), and TAT-QA (Zhu et al. 2021). SQuAD v1.1 is a reading comprehension dataset comprising 100,000 questions that were generated by crowd workers using information from Wikipedia. WikiSQL is a dataset for text-to-SQL generation. This dataset focuses on translating text to SQL. The dataset was created by converting template-based questions into natural language and was created and vetted by crowd workers. The dataset consists of 80,654 questions and SQL queries from Wikipedia. NQ-tables is a tabular question-answer dataset created from the NATURAL QUESTIONS dataset. It was created by extracting 12,000 data from the NQ dataset, which are cases where the correct answer also exists in the table. Additionally, we conducted experiments on the TAT-QA dataset to explore the task of extracting answers from both tables and text. TAT-QA dataset was constructed based on real financial reports’ text paragraphs and tables, requiring numerical reasoning to extract answers. TAT-QA is composed of three types: table, table-text, and text, depending on the input source type from which the answers are tagged. The model needs to aggregate information from both tables and text to infer the answers from hybrid sources.

Experimental Method

As shown in Figure 2, we compared the Separated Reader Model and our proposed hybrid reader model. To compare the differences between application methods within these models, we conduct four types of experiments: Vanilla which is the base model (BERT pre-trained on unstructured text), TAPAS, with column-row-wise projection, and with Adapter structure. All experiments are conducted in three types of data: only table, only text, and both table and text.

As there was no publicly available pre-trained TAPAS model for Korean language, we implemented a pre-trained TAPAS model for Korean language by utilizing approximately 100,000 tables from Korean Wikipedia and 500,000 tables from Namu Wiki.² This pre-trained model was employed as the table-specific reader model in separated reader model. Since the full Wikipedia document, which serves as the context input for KorQuAD 2.0, exceeds the maximum input token length of 512, it is divided into sub-paragraphs. In the separated model, each sub-paragraph is inputted into different reader models depending on whether it contains tables or not.

In this study, we conducted additional experiments on English QA datasets to prove the universality of our hybrid reader model. To replicate the experimental environment of KorQuAD 2.0 dataset which involves processing both text and table data inputs, we used a mixture of SQuAD v1.1, NQ-tables and WikiSQL datasets. SQuAD v1.1 dataset was used to evaluate the MRC performance on text inputs. NQ-tables and WikiSQL datasets were used to evaluate the MRC performance on tabular inputs. To train the MRC model with WikiSQL, we set the positions of tagged answer cell’s start token and end token to the answer span. We used pre-trained RoBERTa base model³ from huggingface as backbone model. We used pre-trained TAPAS base model⁴ for comparison with hybrid model. In the experiment with TAT-QA, we retained the overall architecture of TAGOP model (Zhu et al. 2021) and replaced the encoder model of TAGOP with our proposed hybrid reader model.

Training Details

The concatenated vector of the Row-wise Projection, Column-wise Projection, and Transformer Layer in the adapter layer has a size of 768 × 3, and it is projected to the Adapter Transformer layer using a Down projection Layer. The hidden size of the Adapter Transformer Layer is set to 768. The BERT model in Figure 4 and Language Model with a transformer in Figure 5 were optimized with Adam optimizer at a learning rate of 0.0002. The epochs and dropout rate were set to 5 and 0.2, respectively. We applied the same hyperparameters used in the experiments of TAGOP (Zhu et al. 2021). We ensured reproducibility by utilizing the code provied in TAGOP code release, with only the encoder part replaced by our proposed model with projection layers and adapter layers.

Result

Experiment Result

We evaluated two different types of readers; a Separated Reader which uses two independent readers for each text and table, and a hybrid reader which has one reader for table and text. Table 1 shows the main evaluation for KorQuAD 2.0, which is a multi-paragraph machine reading comprehension on the entire HTML Wikipedia document. The evaluation is focused on data types (text, table, or both), additional structures, and the existence of a paragraph ranking model. We experimented on text, table, and both table and text data in KorQuAD 2.0 to see performance change by using table-specialized modifications. Vanilla is a basic BERT model without modifications for tables. TAPAS is a table-specific BERT that uses BERT input embedding with additional embeddings of a row, column, and raking for the table. We implemented additional pre-training with the table and raw text corpus for the TAPAS. Instead, Projection uses only BERT input embeddings by projecting the BERT representation vector to its corresponding row and column(row, column representation). The adapter uses adapter layers for the table with additional pre-training over tables. Using adapter layers with two phases of training, we achieved the best performance by capturing the tabular knowledge in the data without performance loss in text. For the paragraph ranking, we used the top five paragraph outputs from the paragraph ranking model as reader input, which means there could be no answer in the paragraph. For the “oracle” experiment, we gave the paragraph which contains the answer as input context for the reader model. Compared to paragraph ranking, the performance for the table data improved in all metrics by at most 0.11 points in Oracle, because there is always an answer in Oracle, and the model can find the answer span for it.

Table 1. Result comparison between separated model and hybrid model with and without paragraph ranking.

		With paragraph ranking			oracle
		All	Text	Table	All	Text	Table
Model		EM/F1	EM/F1	EM/F1	EM/F1	EM/F1	EM/F1
Separated Model	Vanilla	58.32/78.95		38.36/61.59	65.04/84.79		66.68/77.15
	TAPAS	60.03/80.25	63.60/83.52	46.29/67.59	66.80/86.19	78.25/90.00	77.50/85.70
	Projection	59.67/79.98		44.85/66.62	66.66/86.04		76.63/84.78
	Adapter	59.21/79.53		42.45/64.33	66.82/86.26		77.62/86.16
Hybrid Model	Vanilla	56.90/77.93	63.00/83.05	33.75/58.51	64.81/84.71	77.80/89.76	67.11/77.61
	TAPAS	57.18/77.98	63.64/83.34	32.69/57.68	67.32/83.90	78.87/86.41	78.12/86.41
	Projection	59.64/79.82	63.45/83.35	45.19/66.41	66.80/83.40	78.37/85.95	77.00/85.27
	Adapter	59.87/80.13	63.71/83.49	45.28/67.36	67.88/86.99	79.94/91.15	77.13/85.91

The values in bold indicate the highest performance achieved in each category.

Table 2 presents the experimental results of applying projection layers and adapter layers to the BirBird model, a long sequence transformer. The model with projection layers showed significant performance improvement on tabular data, yet a decline in performance was observed on text data. While the model with adapter layers improved performance on tabular data, it exhibited a lower increase compared to the model with projection layers and a marginal decline on text data.

Table 2. Result of the new architecture of BigBird model with our modifications.

Model	ALL	Table	Text
(BigBird)	EM/F1	EM/F1	EM/F1
Vanilla	58.15/80.60	51.20/64.01	74.31/86.87
Projection Layer	58.80/81.01	59.89/70.58	73.17/85.89
Adapter	58.55/80.99	54.83/67.33	74.02/86.65

The values in bold indicate the highest performance achieved in each category.

Table 3 shows the evaluation results on the three English MRC datasets. Among the three datasets, SQuAD v1.1 and WikiSQL are textual QA datasets, and NQ-Tables is a tabular QA dataset. Vanilla model performed well on the text dataset SQuAD v1.1 but showed low performance on the table dataset NQ-Tables. Conversely, Tapas model showed limited efficacy on the SQuAD v1.1, yet performed well on the NQ-Tables. The models we propose in this study, Projection, and Adapter, have similar performance on the datasets where Vanilla and Tapas perform well. This outcome demonstrates that the proposed hybrid reader model can be effectively applied to datasets in both English and Korean, indicating its potential to achieve universality across multiple languages.

Table 3. Results for experiments using the English MRC dataset, SQuAD 1.1, WikiSQL, and the NQ-Tables on hybrid reader model.

	Datasets
	SQuAD v1.1	WikiSQL	NQ-Tables
Model	EM/F1	Accuracy	EM/F1
Vanilla	80.84/93.05	56.78	69.59/81.12
Tapas	46.34/71.78	81.79	54.94/73.65
Projection	78.27/91.65	80.73	76.16/85.59
Adapter	80.19/92.77	80.65	74.45/84.49

The values in bold indicate the highest performance achieved in each category.

Table 4 shows the results of the experiment on the TAT-QA dataset. When the encoder model of TAGOP was replaced with the table-specific pre-trained model TAPAS, it showed improved performance on table data but showed lower performance than the baseline model on table-text data and text data. In contrast, when the encoder model of TAGOP was replaced with the hybrid reader model, it showed a slight decline in text data performance and improvements in both table and table-text data. Compared to the baseline model, the model with projection layers showed an improvement of EM 7.08 and F1 7.1, and the model with adapter layers showed an improvement of EM 7.87 and F1 7.75.

Table 4. Result for experiments using TAT-QA dataset.

	Table	Table-Text	Text	Total
Model	EM/F1	EM/F1	EM/F1	EM/F1
TAGOP (Zhu et al. 2021)	44.62/46.81	54.58/59.05	48.01/72.56	48.70/58.81
Tapas	56.47/57.83	59.15/63.58	20.85/44.43	49.18/56.66
Projection	55.37/57.17	61.90/66.51	47.75/73.05	55.78/63.91
Adapter	57.43/59.33	63.00/67.11	45.64/71.20	56.57/64.59

The values in bold indicate the highest performance achieved in each category.

Ablation Study

To evaluate the reasoning performance of the language models, we divided the MRC table data into reasoning data and cell-matching data. In reasoning data, there are questions to compare between cells or examine multiple cells to find an answer. In cell-matching data, there are questions paraphrased and require knowledge to answer. Figure 9 shows the comparison of the performance of language models for reasoning data and cell-matching data. Pre-training language models with adversarial training did not have much effect on reasoning data but showed improved performance in cell-matching data.

Figure 9. The performance comparison of each vanilla, projection, TAPAS, and adapter.

Conclusion

In this paper, we propose a hybrid reader model over both tables and text with the adapter structure. The projection layer gets column and row representations with column-wise and row-wise summation of encoder outputs and combines it with encoder outputs. The reader model with projection layers showed improved results on table data, even without additional pre-training on table data. In the experiments with adapter structure, we used weights of the BERT model pre-trained from unstructured texts, fixed in additional pre-training tasks over tables; instead, the weights of BERT in adapter layers are updated by two stages. Adapter layers infuse knowledge from unstructured texts to tasks over structured texts and capture tabular structure. This approach enhanced the performance of question answers over tables and texts. It showed that the hybrid reader model with adapter captures tabular knowledge better than BERT or TAPAS.

In the experiments on TAT-QA dataset, significant performance improvements were observed even when only replacing the encoder model of the baseline model with our proposed hybrid reader model. This demonstrates that our model can be utilized in various research, simply by replacing the encoder model. Thus, in future research, we plan to conduct further experiments by combining our model with other models that have demonstrated good performance on hybrid QA datasets, such as FinMath (Li, Ye, and Zhao 2022), UniRPG and TAT-QA (Nararatwong, Kertkeidkachorn, and Ichise 2022). Additionally, we will expand our experiments to various hybrid QA datasets.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Source codes and dataset used in this paper can be accessed in our github page.⁵

Additional information

Funding

This work was supported by the National Research Foundation of Korea, NRF under Grant 2022R1F1A1075637.

Notes

1. https://www.aihub.or.kr.

2. https://namu.wiki.

3. https://huggingface.co/roberta-base.

4. https://huggingface.co/google/tapas-base.

5. https://github.com/delosyCho/Multi-Paragraph-Machine-Reading-Comprehension-with-Hybrid-Reader.