Extracting Latent Moral Information from Text Narratives: Relevance, Challenges, and Solutions

ABSTRACT

Moral Foundations Theory (MFT) and the Model of Intuitive Morality and Exemplars (MIME) contend that moral judgments are built on a universal set of basic moral intuitions. A large body of research has supported many of MFT’s and the MIME’s central hypotheses. Yet, an important prerequisite of this research—the ability to extract latent moral content represented in media stimuli with a reliable procedure—has not been systematically studied. In this article, we subject different extraction procedures to rigorous tests, underscore challenges by identifying a range of reliabilities, develop new reliability test and coding procedures employing computational methods, and provide solutions that maximize the reliability and validity of moral intuition extraction. In six content analytical studies, including a large crowd-based study, we demonstrate that: (1) traditional content analytical approaches lead to rather low reliabilities; (2) variation in coding reliabilities can be predicted by both text features and characteristics of the human coders; and (3) reliability is largely unaffected by the detail of coder training. We show that a coding task with simplified training and a coding technique that treats moral foundations as fast, spontaneous intuitions leads to acceptable inter-rater agreement, and potentially to more valid moral intuition extractions. While this study was motivated by issues related to MFT and MIME research, the methods and findings in this study have implications for extracting latent content from text narratives that go beyond moral information. Accordingly, we provide a tool for researchers interested in applying this new approach in their own work.

Acknowledgments

This research would not have been possible with the help of our research assistants at the University of California Santa Barbara (Mitch Grimes, Brandon Mims, Rachel Glikes, Douglas Keith, Sierra Scott, Cathy Chen, and Dane Asto) and at Michigan State University (Brandon Walling, Kathryn Hollemans, Erica Lydey, Maryssa Mitchell, Anna Young, Erika Lentz, Ellen Grimes, Kristin Barndt, Tyler Lawrence, Allison Aigner, Riley Hoffman, Elizabeth Paulson, Sierra Richards, Savannah Jenuwine, Pooja Dandamundi, Will Marchetti, and Abagail Johnson).

Notes

^1. The terms “moral foundations” and “moral intuitions” are sometimes used interchangeably in the literature. We use the term “foundations” to refer to the conceptual dimensions of MFT, i.e., the universal dimensions that categorize moral judgments. We use the term “intuitions” to refer to the experiential, subjective processes of moral judgment.

^2. We only used multi-foundation coders in coder group six.

Additional information

Funding

Contract grant sponsors: US Army Research Laboratory (to R.W.), contract grant number: W911NF-15-2-0115.