![Sample our Politics & International Relations journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5947/TOC-Subject-Sample-2021-Politics-International-Relations.jpg"})
Abstract
In the debate on minimal media effects and their causes, methodological concerns about measurement are rarely discussed. We argue that even in state-of-the-art media-effects studies that combine measures of media messages and media use (i.e., linkage analyses), measurement error in both the media content analysis and the media use self-reports will typically lead to severely downward-biased effect estimates. We demonstrate this phenomenon using a large Monte Carlo simulation with varying parameters of the content analysis and the survey study. Results show that measurement error in the content analysis and media use variables does indeed lead to smaller effect estimates, especially when the media messages of interest are relatively rare. We discuss these findings as well as possible remedies and implications for future research.
Supplemental Material
Supplemental data for this article can be accessed on the publisher’s website at http://dx.doi.org/10.1080/10584609.2016.1235640.
Notes
1. We chose the term linkage analysis to honor the contribution of Miller and colleagues (1979), who were—to our knowledge—the first to describe and implement this research design most similarly to the design of modern linkage analyses in the field of political communication. We acknowledge, however, that the idea to combine content analytical measures of media messages with media use reports from surveys has been implemented before. Notably, the seminal “People’s Choice” study included “the index of political exposure bias,” which weighted self-reports of exposure to several media items and campaign events with a measure of their leaning toward Republicans or Democrats (Lazarsfeld et al., Citation1968, pp. 177–178).
2. The examples intentionally exclude the line of research on the effects of political advertisements (mostly in the context of U.S. elections), which has much in common with media content linkage analysis (e.g., Franz & Ridout, Citation2007; Freedman & Goldstein, Citation1999) and shares some of its problems with regard to measuring exposure using self-reports (Vavreck, Citation2007). However, the measurement of the relevant media messages (i.e., the content of the ads) is comparatively straightforward and rich data sources exist to assist message measurement and geographic allocation of the ads (Goldstein & Freedman, Citation2002). In consequence, our main argument with regard to the combination of measurement error from both media content analysis and media use self-reports is less relevant. The results of the Monte Carlo simulation under the condition of (almost) perfect coding (i.e., ≥ .9) might give some indication for the ability of these studies to uncover true advertising effects. It should be noted, however, that the construction of individual measures of ad exposure also often differs from the procedures applied in mass media linkage analysis. Most importantly, information on ad buy has to be incorporated. Thus, the reliability of the ad exposure measure might differ from simple media use self-reports.
3. For the sake of simplicity, we assume only one media message variable and no temporal weighting for the following steps. Consequently, the subsequent description is limited to only one message exposure measure. If multiple relevant media messages are measured in Step 1, several message exposure measures can by constructed in Step 4 and used as predictors in Step 5. We also neglect the possibility of multistep aggregation, where, for example, media message variables are first aggregated from the coding units of single statement to the unit of news stories.
4. It should be noted that underreporting is also present in LaCour and Vavreck’s (2014) data, mainly among those who reported never to watch TV news. Similarly, the amount of time spent watching TV is often underreported (Wonneberger & Irazoqui, Citation2016). However, most published linkage analyses use frequency rather than duration measures, and are therefore more likely to encounter overreporting.
5. See the Technical Appendix in the online supplemental material for the formula to obtain the expected proportions under these assumptions.
6. The reported reliability of the media message variables depends on the choice of the reliability coefficient. Chance-corrected coefficients such as Krippendorff’s α and Scott’s π are currently the consensus measures for reliability in content analysis. Estimates based on these measures are (sometimes considerably) lower compared to simple measures of percent agreement and they depend on the marginal distributions in the test material. The message variable of a study that reports a percent agreement of .9 may well have a worse true-score reliability. In contrast, a message variable with an estimated reliability of Krippendorff’s α = .6 from a test with a highly skewed distribution of the message variable and a small sample may actually underestimate the true-score reliability. It is beyond the scope of the present article to join the debate about which coefficient is best under which circumstances. We merely note that media-effects researchers who want to evaluate the quality of their linkage analysis should follow the ongoing discussions about reliability measurement in content analysis and apply state-of-the-art measures that most closely approximate true-score reliability.
7. Details on the computation and estimation can be found in the Technical Appendix in the online supplemental material.
Additional information
Notes on contributors
Michael Scharkow
Michael Scharkow and Marko Bachl are Postdoctoral Researchers, Department of Communication, University of Hohenheim
Marko Bachl
Michael Scharkow and Marko Bachl are Postdoctoral Researchers, Department of Communication, University of Hohenheim