Evaluation of Placename Geoparsers

Abstract

This study analyzes the results of six natural-language-processing geoparsing tools, using a set of 120 articles, to determine which would be most useful to incorporate into Journalmap.org, a scientific literature discovery tool. The evaluation of accurate locations was performed against a set of manually-verified geographic locations in the selected set of articles. Geoparser accuracy was determined by comparing the planar distance between the geoparser result and verified geographic locations. Parser-determined locations less than or equal to 161 km (100 mi) from the known geographic location were considered accurate. The tools tested in this study produced varying degrees of accuracy ranging from 20% to 65%. Libraries may use these tools for geographic indexing of literature or for similar discovery use cases, but emerging large language models (LLMs) appear ready to offer potentially improved results in the future.

Keywords:

• geoparsing
• geocoding
• natural language processing (NLP)
• named entity extraction
• large language model (LLM)

Acknowledgments

The authors would like to acknowledge the assistance that they received from Luke Sheneman and Tanner Varrelman of Research Computing and Data Services at the University of Idaho. This work was supported, in part, by a National Leadership Grant from the Institute for Museum and Library Services #LG-252316-OLS-2.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://github.com/jkenyon/evaluation-geoparsing

2 https://arcgis.com/home/item.html?id=97369a6f1200428ba060410d13dbb078

3 https://openai.com/chatgpt

4 https://www.anthropic.com/index/claude-2

5 https://unpaywall.org