‘No, Alexa, no!’: designing child-safe AI and protecting children from the risks of the ‘empathy gap’ in large language models

References

• Abbasi, N. I., M. Spitale, J. Anderson, T. Ford, P. B. Jones, and H. Gunes. 2022, August. “Can Robots Help in the Evaluation of Mental Wellbeing in Children? An Empirical Study.” In 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 1459–1466. IEEE.
   Google Scholar
• Andries, V., and J. Robertson. 2023. “‘Alexa doesn't Have that Many Feelings’: Children's Understanding of AI through Interactions with Smart Speakers in their Homes.” Computers and Education: Artificial Intelligence 5:100176. https://doi.org/10.1016/j.caeai.2023.100176.
   Google Scholar
• Atlas, S. 2023. “ChatGPT for Higher Education and Professional Development: A Guide to Conversational AI.” https://digitalcommons.uri.edu/cba_facpubs/548.
   Google Scholar
• Ball, S. J., and E. Grimaldi. 2022. “Neoliberal Education and the Neoliberal Digital Classroom.” Learning, Media and Technology 47 (2): 288–302. https://doi.org/10.1080/17439884.2021.1963980.
   Web of Science ®Google Scholar
• Bayne, S. 2023. “Digital Education Utopia.” Learning, Media and Technology, 1–16. https://doi.org/10.1080/17439884.2023.2262382
   Web of Science ®Google Scholar
• BBC News. 2021. “Alexa Tells 10-year-old Girl to Touch Live Plug with Penny.” BBC News, December 28. https://www.bbc.co.uk/news/technology-59810383.
   Google Scholar
• Bedingfield, W. 2023. “A Chatbot Encouraged Him to Kill the Queen. It’s Just the Beginning.” Wired UK, October 10. https://www.wired.co.uk/article/chatbot-kill-the-queen-eliza-effect.
   Google Scholar
• Bradbury, A. 2019. “Datafied at Four: The Role of Data in the ‘Schoolification’ of Early Childhood Education in England.” Learning, Media and Technology 44 (1): 7–21. doi: 10.1080/17439884.2018.1511577
   Web of Science ®Google Scholar
• Cambria, E., and B. White. 2014. “Jumping NLP Curves: A Review of Natural Language Processing Research.” IEEE Computational Intelligence Magazine 9 (2): 48–57. https://doi.org/10.1109/MCI.2014.2307227.
   Web of Science ®Google Scholar
• Chen, J., and J. C. Lin. 2023. “Artificial Intelligence as a Double-edged Sword: Wielding the POWER Principles to Maximize its Positive Effects and Minimize its Negative Effects.” Contemporary Issues in Early Childhood. https://doi.org/10.1177/14639491231169813.
   PubMed Web of Science ®Google Scholar
• Chowdhury, G. 2003. “Natural Language Processing.” Annual Review of Information Science and Technology 37 (1): 51–89. https://doi.org/10.1002/aris.1440370103.
   Google Scholar
• Common Sense Media. 2023. Impact Research Report. https://www.commonsensemedia.org/sites/default/files/featured-content/files/common-sense-ai-polling-memo-may-10-2023-final.pdf.
   Google Scholar
• Darling, K. 2017. “‘Who is Johnny?’ Anthropomorphic Framing in Human-robot Interaction, Integration, and Policy.” In Robot Ethics 2.0: From Autonomous Cars to Artificial Intelligence, edited by P. Lin, K. Abney, and R. Jenkins, 1–10. New York, NY: Oxford Scholarship Online.
   Google Scholar
• DeFranco, J. F., and J. Voas. 2022. “‘Smarter’ Seeking ‘Wiser’.” Computer 55 (07): 13–14. https://doi.org/10.1109/MC.2022.3146647.
   Google Scholar
• Druga, S., R. Williams, H. W. Park, and C. Breazeal. 2018. “How Smart are the Smart Toys? Children and Parents’ Agent Interaction and Intelligence Attribution.” In Proceedings of the 17th ACM Conference on Interaction Design and Children, 231–240. https://doi.org/10.1145/3202185.3202741.
   Google Scholar
• Dwivedi, Y. K., N. Kshetri, L. Hughes, E. L. Slade, A. Jeyaraj, A. K. Kar, A. M. Baabdullah, et al. 2023. “So What if ChatGPT Wrote it?” Multidisciplinary Perspectives on Opportunities, Challenges and Implications of Generative Conversational AI for Research, Practice and Policy.” International Journal of Information Management 71:102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642.
   Web of Science ®Google Scholar
• El Atillah, I. 2023. “Man Ends his Life after an AI Chatbot ‘Encouraged’ him to Sacrifice Himself to Stop Climate Change.” Euro News, March 31. https://www.euronews.com/next/2023/03/31/man-ends-his-life-after-an-ai-chatbot-encouraged-him-to-sacrifice-himself-to-stop-climate-.
   Google Scholar
• Fowler, G. 2023. “Snapchat Tried to Make a Safe AI. It Chats with Me about Booze and Sex.” Washington Post. https://www.washingtonpost.com/technology/2023/03/14/snapchat-myai/.
   Google Scholar
• Gabriel, I., A. Manzini, G. Keeling, L. A. Hendricks, V. Rieser, H. Iqbal, N. Tomašev, et al. 2024. “The Ethics of Advanced AI Assistants.” arXiv preprint arXiv 2404: 16244.
   Google Scholar
• Garg, R., and S. Sengupta. 2020. “Conversational Technologies for In-Home Learning: Using Co-design to Understand Children's and Parents’ Perspectives.” In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–13.
   Google Scholar
• Google PAIR. 2019. People + AI Guidebook. https://design.google/ai-guidebook.
   Google Scholar
• Holmes, W., D. Bektik, B. Woolf, and R. Luckin. 2019. “Ethics in AIED: Who Cares?” In 20th International Conference on Artificial Intelligence in Education (AIED’19), 25–29 Jun 2019.
   Google Scholar
• Holmes, W., and K. Porayska-Pomsta, eds. 2022. The Ethics of Artificial Intelligence in Education: Practices, Challenges, and Debates. London: Taylor & Francis.
   Google Scholar
• InternetMatters. 2017. “The Secret Life of Six Year Olds.” https://www.internetmatters.org/hub/press_release/revealed-the-secret-life-of-six-year-olds-online/.
   Google Scholar
• Ischen, C., T. Araujo, H. Voorveld, G. van Noort, and E. Smit. 2019. “Privacy Concerns in Chatbot Interactions.” In Chatbot Research and Design: Third International Workshop, CONVERSATIONS 2019, Amsterdam, The Netherlands, November 19–20, 2019, Revised Selected Papers 3, 34–48. Springer International Publishing.
   Google Scholar
• Jarke, J., and A. Breiter. 2019. “The Datafication of Education.” Learning, Media and Technology 44 (1): 1–6. https://doi.org/10.1080/17439884.2019.1573833.
   Web of Science ®Google Scholar
• Ji, H., I. Han, and Y. Ko. 2023. “A Systematic Review of Conversational AI in Language Education: Focusing on the Collaboration with Human Teachers.” Journal of Research on Technology in Education 55 (1): 48–63. https://doi.org/10.1080/15391523.2022.2142873.
   Web of Science ®Google Scholar
• Jobin, A., M. Ienca, and E. Vayena. 2019. “The Global Landscape of AI Ethics Guidelines.” Nature Machine Intelligence 1 (9): 389–399. https://doi.org/10.1038/s42256-019-0088-2.
   Google Scholar
• Knox, J., B. Williamson, and S. Bayne. 2020. “Machine Behaviourism: Future Visions of ‘Learnification’ and ‘Datafication’ across Humans and Digital Technologies.” Learning, Media and Technology 45 (1): 31–45. https://doi.org/10.1080/17439884.2019.1623251.
   Web of Science ®Google Scholar
• Kulkarni, P., A. Mahabaleshwarkar, M. Kulkarni, N. Sirsikar, and K. Gadgil. 2019. “Conversational AI: An Overview of Methodologies, Applications & Future Scope.” In 2019 5th International Conference On Computing, Communication, Control And Automation (ICCUBEA), September, 1–7. IEEE.
   Google Scholar
• Kuntsman, A. 2012. “Introduction: Affective Fabrics of Digital Cultures.” In Digital Cultures and the Politics of Emotion: Feelings, Affect and Technological Change, edited by A. Karatzogianni and A. Kuntsman, 1–17. London: Palgrave Macmillan UK.
   Google Scholar
• Kurian, N. 2023a. “AI's Empathy Gap: The Risks of Conversational Artificial Intelligence for Young Children's Well-being and Key Ethical Considerations for Early Childhood Education and Care.” Contemporary Issues in Early Childhood, 14639491231206004. https://doi.org/10.1177/14639491231206004
   Web of Science ®Google Scholar
• Kurian, N. 2023b. “Toddlers and Robots? The Ethics of Supporting Young Children with Disabilities with AI Companions and the Implications for Children’s Rights.” International Journal of Human Rights Education 7 (1): 9.
   Google Scholar
• Lai, S. S., V. Andelsman, and S. Flensburg. 2023. “Datafied School Life: The Hidden Commodification of Digital Learning.” Learning, Media and Technology, 1–17. https://doi.org/10.1080/17439884.2023.2219063
   Web of Science ®Google Scholar
• Macgilchrist, F. 2019. “Cruel Optimism in Edtech: When the Digital Data Practices of Educational Technology Providers Inadvertently Hinder Educational Equity.” Learning, Media and Technology 44 (1): 77–86. https://doi.org/10.1080/17439884.2018.1556217.
   Web of Science ®Google Scholar
• Manolev, J., A. Sullivan, and R. Slee. 2020. “The Datafication of Discipline: ClassDojo, Surveillance and a Performative Classroom Culture.” In The Datafication of Education, 37–52. Routledge.
   Google Scholar
• Marcin, T. 2023. Microsoft's Bing AI chatbot has said a lot of weird things. Here's a list. Mashable. https://mashable.com/article/microsoft-bing-ai-chatbot-weird-scary-responses.
   Google Scholar
• McTear, M. 2022. Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots. London: Springer Nature.
   Google Scholar
• Murtarelli, G., A. Gregory, and S. Romenti. 2021. “A Conversation-based Perspective for Shaping Ethical Human–Machine Interactions: The Particular Challenge of Chatbots.” Journal of Business Research 129:927–935. https://doi.org/10.1016/j.jbusres.2020.09.018.
   Web of Science ®Google Scholar
• Naveed, H., A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Barnes, and A. Mian. 2023. “A Comprehensive Overview of Large Language Models.” arXiv preprint arXiv:2307.06435.
   Google Scholar
• OECD. 2021. OECD AI’s Live Repository of Over 260 AI Strategies & Policies. OECD.AI, powered by EC/OECD. Accessed 17 February 2024. www.oecd.ai/dashboards.
   Google Scholar
• Olsson, F. 2009. “A Literature Survey of Active Machine Learning in the Context of Natural Language Processing.”
   Google Scholar
• Pesapane, F., C. Volonté, M. Codari, and F. Sardanelli. 2018. “Artificial Intelligence as a Medical Device in Radiology: Ethical and Regulatory Issues in Europe and the United States.” Insights Into Imaging 9 (5): 745–753. https://doi.org/10.1007/s13244-018-0645-y.
   PubMed Web of Science ®Google Scholar
• Rudolph, J., S. Tan, and S. Tan. 2023. “War of the Chatbots: Bard, Bing Chat, ChatGPT, Ernie and Beyond. The New AI Gold Rush and its Impact on Higher Education.” Journal of Applied Learning and Teaching 6 (1): 364–389.
   Google Scholar
• Selwyn, N. 2015. “Data Entry: Towards the Critical Study of Digital Data and Education.” Learning, Media and Technology 40 (1): 64–82. https://doi.org/10.1080/17439884.2014.921628.
   Web of Science ®Google Scholar
• Shead, S. 2021. Amazon’s Alexa assistant told a child to do a potentially lethal challenge. https://www.cnbc.com/2021/12/29/amazons-alexa-told-a-child-to-do-a-potentially-lethal-challenge.html.
   Google Scholar
• Shum, H. Y., X. D. He, and D. Li. 2018. “From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots.” Frontiers of Information Technology & Electronic Engineering 19 (1): 10–26. https://doi.org/10.1631/FITEE.1700826.
   Web of Science ®Google Scholar
• Su, J., D. T. Ng, and S. K. Chu. 2023. “Artificial Intelligence (AI) Literacy in Early Childhood Education: The Challenges and Opportunities.” Computers and Education: Artificial Intelligence 4:100124. https://doi.org/10.1016/j.caeai.2023.100124.
   Google Scholar
• Sundar, S. S., and J. Kim. 2019. “Machine Heuristic: When We Trust Computers More than Humans with Our Personal Information.” In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–9. New York, NY: Association for Computing Machinery.
   Google Scholar
• Thaler, R. H., and C. R. Sunstein. 2009. Nudge: Improving Decisions about Health, Wealth, and Happiness. Penguin.
   Google Scholar
• Turkle, S. 2011. Alone Together: Why We Expect More from Technology and Less from Each Other. New York: Basic Books.
   Google Scholar
• UNESCO. 2019. Beijing Consensus on Artificial Intelligence (AI) and Education. Accessed 17 February 2024. https://unesdoc.unesco.org/ark:/48223/pf0000368303.
   Google Scholar
• UNICEF. 2020. Safeguarding Girls and Boys: When Chatbots Answer their Private Questions. Accessed 6 August 2022. https://www.unicef.org/eap/media/5376/file.
   Google Scholar
• Van Brummelen, J., T. Heng, and V. Tabunshchyk. 2021, May. “Teaching Tech to Talk: K-12 Conversational Artificial Intelligence Literacy Curriculum and Development Tools.” In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, No. 17, 15655–15663. https://doi.org/10.1609/aaai.v35i17.17844.
   Google Scholar
• Weidinger, L., J. Mellor, M. Rauh, C. Griffin, J. Uesato, P. S. Huang, M. Cheng, et al. 2021. “Ethical and Social Risks of Harm from Language Models.” arXiv preprint arXiv:2112.04359.
   Google Scholar
• White, G. 2018. “Child Advice Chatbots Fail to Spot Sexual Abuse.” https://www.bbc.co.uk/news/technology-46507900.
   Google Scholar
• Williamson, B. 2017. “Decoding ClassDojo: Psycho-policy, Social-emotional Learning and Persuasive Educational Technologies.” Learning, Media and Technology 42 (4): 440–453. https://doi.org/10.1080/17439884.2017.1278020.
   Web of Science ®Google Scholar
• Williamson, B., F. Macgilchrist, and J. Potter. 2023. “Re-examining AI, Automation and Datafication in Education.” Learning, Media and Technology 48 (1): 1–5. https://doi.org/10.1080/17439884.2023.2167830.
   Web of Science ®Google Scholar