Can AI Models Summarize Your Diary Entries? Investigating Utility of Abstractive Summarization for Autobiographical Text

Abstract

Journaling is a widely adopted technique, known to improve mental health and well-being by enabling reflection on past events. Large amounts of text in digital journaling applications could hinder the reflection process due to information overload. Abstractive summarization can solve this problem by generating short summaries to quickly glance at and reminisce. In this paper, we present an investigation of the utility of large language models in the context of autobiographical text summarization. We study two approaches to adapt a self-supervised learning (SSL) model to the domain of autobiographical text. One model employs transfer learning using our new autobiographical text summary dataset to fine-tune the SSL model. The second model leverages existing news datasets for high-quality text summarization mixed with our autobiographical summary dataset. We conducted mixed methods research to analyze the performance of these two models. Through objective evaluation using ROUGE and BART scores, we find that both these approaches perform significantly better than the SSL model fine-tuned with only high-quality news datasets, showing the importance of domain adaptation and autobiographical text summary dataset for this task. Secondly, through a subjective evaluation on a crowd-sourcing platform, we evaluated the summaries generated from these models on various quality criteria such as grammar, non-redundancy, structure, and coherence. We found that on all criteria, these summaries score >4 out of 5, and the two models show comparable results. We deployed a proof-of-concept web-based journaling application to assess the practical real-world implications of incorporating abstractive summarization in a digital journaling context. We found that the participants showed a high consensus that the summaries generated by the system captured the main idea of their journal entry (80% of the 75 participants gave a Likert scale rating of $\ge 5.0$ out of 7.0, with the overall mean rating of 5.56 ± 1.32) while being factually correct, and they found it to be a useful feature of a journaling application. Finally, we conducted human evaluation studies to compare the quality of the summaries generated from a commercial tool ChatGPT and mixed distribution fine-tuned SSL model, and present insights into these systems in the context of autobiographical abstractive text summarization. We have made our model, dataset, and subjective evaluation questionnaire openly available to the research community.

Keywords:

• Self supervised learning
• autobiographical text summarization
• digital journaling

Acknowledgements

We would like to acknowledge the participants who took part in our user studies.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 https://openai.com/blog/chatgpt

2 https://drive.google.com/drive/folders/1T4dk3FZxlsArICLL8dRB5vcycL5KkhWF?usp=sharing Google Drive link for review

3 https://valohai.com/subreddit-finder/

4 reddit_train_links.

5 https://github.com/pytorch/fairseq/blob/master/examples/bart/README.summarization.md#fine-tuning-bart-on-cnn-dailymail-summarization-task Link to fine tuning BART on CNN-Dailymail dataset.

6 https://drive.google.com/drive/folders/1T4dk3FZxlsArICLL8dRB5vcycL5KkhWF Google Drive link for review.

7 https://github.com/neulab/BARTScore

8 https://www.mturk.com/

9 https://flask.palletsprojects.com/en/2.1.x/

10 https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API

11 https://developer.chrome.com/blog/voice-driven-web-apps-introduction-to-the-web-speech-api/

12 https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API

13 https://platform.openai.com/docs/models/gpt-3-5

Additional information

Funding

This work was supported by the Assistive Augmentation research grant under the Entrepreneurial Universities (EU) initiative of New Zealand.

Notes on contributors

Shamane Siriwardhana

Shamane Siriwardhana earned his PhD from the Department of Bioengineering at The University of Auckland, with a research focus on adapting foundation models for domain-specific natural language processing models.

Chitralekha Gupta

Chitralekha Gupta is a Senior Research Fellow at the School of Computing at National University of Singapore. Her research interests lie at the intersection of computing, speech, and music, particularly in singing voice analysis, audio synthesis, applications of automatic speech recognition (ASR) in music, and audio-based assistive technologies.

Tharindu Kaluarachchi

Tharindu Kaluarachchi completed his PhD at The University of Auckland. His research interests lie in developing AI applications for non-AI-experts using the Human-Centered Machine Learning approach. Currently Tharindu works as a Chief Technology Officer (CTO) in Singapore, developing AI solutions for the Tea Industry.

Vipula Dissanayake

Vipula Dissanayake earned his PhD from Auckland Bioengineering institute at the University of Auckland. His research interests spread across machine learning and human computer interaction.

Suveen Ellawela

Suveen Ellawela is an undergraduate student at School of Computing at National University of Singapore. His research interests include autonomous agents, user experience design and human-computer interaction.

Suranga Nanayakkara

Suranga Nanayakkara is an Associate Professor at Department of Information Systems & Analytics, School of Computing at National University of Singapore. He founded the “Augmented Human Lab” to explore ways of designing intelligent human-computer interfaces that extend the limits of our perceptual and cognitive capabilities.