Open access
4,616
Views
6
CrossRef citations to date
0
Altmetric
Research Article
A study on the evaluation of tokenizer performance in natural language processing
Sanghyun Chooa Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh, NC, USA
https://orcid.org/0000-0002-8884-3437View further author information
Wonjoon Kimb Division of Future Convergence (HCI Science Major), Dongduk Women’s University, Seoul, South KoreaCorrespondence[email protected]
https://orcid.org/0000-0001-5177-8072View further author information
Article: 2175112
|
Received 17 Jun 2022, Accepted 27 Jan 2023, Published online: 09 Feb 2023
Reprints and Permissions
This is an open access article distributed under the terms of the Creative Commons CC BY license, which permits unrestricted use, distribution, reproduction in any medium, provided the original work is properly cited.
You are not required to obtain permission to reuse this article in part or whole.
Related research
People also read lists articles that other readers of this article have read.
Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.
Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.