294
Views
0
CrossRef citations to date
0
Altmetric
Research Article

Can AI Models Summarize Your Diary Entries? Investigating Utility of Abstractive Summarization for Autobiographical Text

ORCID Icon, ORCID Icon, ORCID Icon, ORCID Icon, ORCID Icon & ORCID Icon
Received 05 Aug 2023, Accepted 13 Nov 2023, Published online: 30 Nov 2023
 

Abstract

Journaling is a widely adopted technique, known to improve mental health and well-being by enabling reflection on past events. Large amounts of text in digital journaling applications could hinder the reflection process due to information overload. Abstractive summarization can solve this problem by generating short summaries to quickly glance at and reminisce. In this paper, we present an investigation of the utility of large language models in the context of autobiographical text summarization. We study two approaches to adapt a self-supervised learning (SSL) model to the domain of autobiographical text. One model employs transfer learning using our new autobiographical text summary dataset to fine-tune the SSL model. The second model leverages existing news datasets for high-quality text summarization mixed with our autobiographical summary dataset. We conducted mixed methods research to analyze the performance of these two models. Through objective evaluation using ROUGE and BART scores, we find that both these approaches perform significantly better than the SSL model fine-tuned with only high-quality news datasets, showing the importance of domain adaptation and autobiographical text summary dataset for this task. Secondly, through a subjective evaluation on a crowd-sourcing platform, we evaluated the summaries generated from these models on various quality criteria such as grammar, non-redundancy, structure, and coherence. We found that on all criteria, these summaries score >4 out of 5, and the two models show comparable results. We deployed a proof-of-concept web-based journaling application to assess the practical real-world implications of incorporating abstractive summarization in a digital journaling context. We found that the participants showed a high consensus that the summaries generated by the system captured the main idea of their journal entry (80% of the 75 participants gave a Likert scale rating of 5.0 out of 7.0, with the overall mean rating of 5.56 ± 1.32) while being factually correct, and they found it to be a useful feature of a journaling application. Finally, we conducted human evaluation studies to compare the quality of the summaries generated from a commercial tool ChatGPT and mixed distribution fine-tuned SSL model, and present insights into these systems in the context of autobiographical abstractive text summarization. We have made our model, dataset, and subjective evaluation questionnaire openly available to the research community.

Acknowledgements

We would like to acknowledge the participants who took part in our user studies.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

Additional information

Funding

This work was supported by the Assistive Augmentation research grant under the Entrepreneurial Universities (EU) initiative of New Zealand.

Notes on contributors

Shamane Siriwardhana

Shamane Siriwardhana earned his PhD from the Department of Bioengineering at The University of Auckland, with a research focus on adapting foundation models for domain-specific natural language processing models.

Chitralekha Gupta

Chitralekha Gupta is a Senior Research Fellow at the School of Computing at National University of Singapore. Her research interests lie at the intersection of computing, speech, and music, particularly in singing voice analysis, audio synthesis, applications of automatic speech recognition (ASR) in music, and audio-based assistive technologies.

Tharindu Kaluarachchi

Tharindu Kaluarachchi completed his PhD at The University of Auckland. His research interests lie in developing AI applications for non-AI-experts using the Human-Centered Machine Learning approach. Currently Tharindu works as a Chief Technology Officer (CTO) in Singapore, developing AI solutions for the Tea Industry.

Vipula Dissanayake

Vipula Dissanayake earned his PhD from Auckland Bioengineering institute at the University of Auckland. His research interests spread across machine learning and human computer interaction.

Suveen Ellawela

Suveen Ellawela is an undergraduate student at School of Computing at National University of Singapore. His research interests include autonomous agents, user experience design and human-computer interaction.

Suranga Nanayakkara

Suranga Nanayakkara is an Associate Professor at Department of Information Systems & Analytics, School of Computing at National University of Singapore. He founded the “Augmented Human Lab” to explore ways of designing intelligent human-computer interfaces that extend the limits of our perceptual and cognitive capabilities.

Log in via your institution

Log in to Taylor & Francis Online

PDF download + Online access

  • 48 hours access to article PDF & online version
  • Article PDF can be downloaded
  • Article PDF can be printed
USD 61.00 Add to cart

Issue Purchase

  • 30 days online access to complete issue
  • Article PDFs can be downloaded
  • Article PDFs can be printed
USD 306.00 Add to cart

* Local tax will be added as applicable

Related Research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.