A study on the evaluation of tokenizer performance in natural language processing

Figures & data

Figure 1. Illustrative of 10-fold cross-validation in this study.

Figure 2. Illustrative of overall research flow.

Table 1. The results of accuracy for SentencePiece and Mecab-Ko each classification algorithm.

Algorithm	Mecab-Ko	No. of Tokens in SentencePiece
		5000	10000	15000	20000	25000	30000
NB	0.880	0.903	0.900	0.907	0.906	0.921	0.914
kNN	0.884	0.875	0.904	0.912	0.922	0.933	0.924
SVM	0.948	0.950	0.951	0.948	0.948	0.948	0.949
ANN	0.948	0.947	0.960	0.956	0.948	0.948	0.943
LSTM-RNN	0.954	0.953	0.964	0.962	0.970	0.966	0.964

Table 2. The results of precision for SentencePiece and Mecab-Ko each classification algorithm.

Algorithm	Mecab-Ko	No. of Tokens in SentencePiece
		5000	10000	15000	20000	25000	30000
NB	0.953	0.953	0.953	0.952	0.953	0.953	0.953
kNN	0.966	0.964	0.953	0.973	0.970	0.971	0.974
SVM	0.970	0.970	0.977	0.976	0.970	0.970	0.966
ANN	0.957	0.958	0.960	0.958	0.957	0.958	0.957
LSTM-RNN	0.968	0.955	0.967	0.966	0.973	0.971	0.967

Table 3. The results of recall for SentencePiece and Mecab-Ko each classification algorithm.

Algorithm	Mecab-Ko	No. of Tokens in SentencePiece
		5000	10000	15000	20000	25000	30000
NB	0.919	0.945	0.942	0.949	0.949	0.965	0.973
kNN	0.912	0.903	0.946	0.933	0.948	0.949	0.956
SVM	0.976	0.974	0.981	0.979	0.976	0.975	0.975
ANN	0.990	0.991	0.990	0.989	0.990	0.989	0.991
LSTM-RNN	0.984	0.996	0.996	0.996	0.996	0.994	0.996

Table 4. The results of F1-score for SentencePiece and Mecab-Ko each classification algorithm.

Algorithm	Mecab-Ko	No. of Tokens in SentencePiece
		5000	10000	15000	20000	25000	30000
NB	0.936	0.949	0.947	0.951	0.950	0.959	0.963
kNN	0.938	0.933	0.949	0.953	0.959	0.960	0.965
SVM	0.973	0.972	0.979	0.977	0.973	0.972	0.970
ANN	0.973	0.974	0.975	0.973	0.973	0.973	0.974
LSTM-RNN	0.976	0.975	0.981	0.981	0.984	0.982	0.981

Figure 3. Result of the accuracy for token number of SentencePiece of each learning algorithm.

Figure 4. Result of the F1-score for token number of SentencePiece of each learning algorithm.