Teaching the Computer to Code Frames in News: Comparing Two Supervised Machine Learning Approaches to Frame Analysis

Abstract

We explore the application of supervised machine learning (SML) to frame coding. By automating the coding of frames in news, SML facilitates the incorporation of large-scale content analysis into framing research, even if financial resources are scarce. This furthers a more integrated investigation of framing processes conceptually as well as methodologically. We conduct several experiments in which we automate the coding of four generic frames that are operationalised as a set of indicator questions. In doing so, we compare two approaches to modelling the coherence between indicator questions and frames as an SML task. The results of our experiments show that SML is well suited to automate frame coding but that coding performance is dependent on the way SML is implemented.

Notes

¹ All four frames are introduced in detail in the next section.

² Instead of a single split into held-in and held-out, the vectors of predictions are obtained through 10-fold cross-validation.

³ We also tried alternative bag-of-words transformations, for example, binary-word presence, word counts, and parsimonious language models (Hiemstra et al., 2004). Additionally, we tried representing all articles in terms of n-grams and latent topics as derived from a LDA-model (Radim & Sojka, 2010). These variations in feature representation, as well as combinations of them, did not improve on TF.IDF weighting. We suggest applying syntactic (e.g., part of speech tags) or semantic features in future research.

⁴ The Python code used can be provided upon request.

⁵ Coders were required to answer ’yes’ or ’no’ to the following question:”Is the story political in nature?”

⁶ In previous research, these questions have been shown to be reliable indicators of the four frames (e.g., Semetko & Valkenburg, 2000; Vreese et al., 2001).

⁷ We performed a principal component analysis with non-orthogonal rotation to establish the coherence of the indicator questions and their relationships to the frames. As expected, we found a four-factor solution in which all indicators show significant positive loadings (>.5) on the expected frame.

⁸ Nearly all coders were involved, because multiple pairs of coders were used for reliability testing.

⁹ It is a well-known issue that Krippendorff’s alpha measures tend to be relatively low when assessing inter-coder agreement of binary classification tasks with unbalanced class distributions. This especially is the case with the morality frame, where we observe a substantial difference between the pairwise agreement measure and Krippendorff’s alpha measure.

¹⁰ The research is supported through a VENI grant from the Dutch Science Foundation, and the Dutch national program COMMIT.

¹¹ Please note that the cross-validation sample that was used to estimate weights for the ensemble of classifiers is nested in the cross-validation sample, which we used to assess coding performance.

¹² We always used a random sample of 2,000 articles as a training set and a random sample of 1,000 articles as test set.

¹³ We found the same pattern when applying the indicator-based approach.