Linguistic Challenges in Generative Artificial Intelligence: Implications for Low-Resource Languages in the Developing World

ABSTRACT

Proficiency in English is pivotal for leveraging information and communication technologies, but it holds even greater significance in the realm of generative artificial intelligence (GAI), which is poised as the next digital frontier. However, the dominance of English in benchmarks and training data for large language models (LLMs) exacerbates challenges for individuals and organizations in the developing world, predominantly non-English speakers. Despite the commendable performance of GAI in select developed languages like French, Spanish, and Japanese, it struggles to deliver comparable results in low-resource languages (LRLs) such as Bengali, Hindi, and Swahili. These languages, deprived of adequate online content, face obstacles in training specialized models due to script complexities and limited lexical resources. While countries like Japan and Iceland offer promising models for addressing linguistic challenges, the road ahead necessitates collaborative efforts to develop LLMs tailored for LRLs and rectify linguistic inaccuracies, ensuring inclusive and equitable AI development.

KEYWORDS:

• Developing world
• generative artificial intelligence
• large language models
• low-resource languages
• natural language processing

Notes

1. https://www.businesstoday.in/opinion/columns/the-unique-challenge-india-presents-to-natural-language-processing/story/283650.html

2. https://www.livemint.com/AI/OHvuLFS2UCnItjk7CAcdpK/How-AI-is-helping-firms-tap-users-of-Indian-languages.html

3. https://www.forbes.com/sites/cognitiveworld/2019/04/22/debunking-the-myths-and-reality-of-artificial-intelligence/#108bfa1543b5

4. https://www.axios.com/2023/09/08/ai-language-gap-chatgpt

5. https://www.businesstoday.in/opinion/columns/the-unique-challenge-india-presents-to-natural-language-processing/story/283650.html

6. https://medium.com/neuralspace/low-resource-language-what-does-it-mean-d067ec85dea5

7. https://slator.com/how-effective-are-large-language-models-in-low-resource-language-translation/

8. https://www.bbc.com/news/business-48139212

9. https://qz.com/africa/1475763/african-languages-are-lagging-behind-when-it-comes-to-voice-recognition-innovations/

10. https://arxiv.org/pdf/2007.11865.pdf

11. https://www.economist.com/business/2023/05/09/just-how-good-can-china-get-at-generative-ai

12. https://asia.nikkei.com/Business/Technology/ChatGPT-rush-kicks-U.S.-China-AI-race-into-higher-gear

13. https://blog.yenniejun.com/p/all-languages-are-not-created-tokenized

14. https://gptforwork.com/guides/openai-gpt3-tokens

15. https://www.nature.com/articles/d41586-023-02868-z

16. https://grapevine.is/news/2023/03/15/icelandic-developed-as-a-language-model-within-chatgpt/

Additional information

Notes on contributors

Nir Kshetri

Nir Kshetri is a professor at University of North Carolina-Greensboro and research fellow at Kobe University, Japan. He has authored thirteen books and about 250 academic articles. Nir’s work has been featured by hundreds of media outlets, such as Al Jazeera, BBC, Bloomberg TV, Economist, Foreign Policy, Forbes, Fortune, Newsweek, Public Radio International, Scientific American, US News and World Report and Wall Street Journal. Nir is a two-time TEDx speaker about the roles of emerging technologies such as artificial intelligence (https://www.youtube.com/watch?v=W6da0kEfBsY) and blockchain (https://www.youtube.com/watch?v=Wdo_Jlov9R4) in fighting poverty.