Manipulated vs. Measured: Using an Experimental Benchmark to Investigate the Performance of Self-Reported Media Exposure

ABSTRACT

Media exposure is one of the most important concepts in the social sciences, and yet scholars have struggled with how to operationalize it for decades. Some researchers have focused on the effects of variously worded self-report measures. Others advocate the use of aggregate and/or behavioral data that does not rely on a person’s ability to accurately recall exposure. Our study illustrates how an experimental design can be used to improve measures of exposure. In particular, we show how an experimental benchmark can be employed to (1) compare actual (i.e., manipulated) and self-reported values of news exposure; (2) assess how closely the self-reported items approximate the performance of “true” exposure in an empirical application; and (3) investigate whether a variation in question wording improves the accuracy of self-reported exposure measures.

Acknowledgments

We thank Yanna Krupnikov and Scott Clifford for helpful comments on previous versions of this article.

Notes

¹ Convergent validity refers to the fit between independent measures of the same construct, while predictive validity refers to the ability of a construct to predict a criterion variable (e.g., political knowledge, in the case of media exposure; see Dilliplane et al., 2013).

² Nielsen ratings represent another alternative to self-reported exposure measures (Prior, 2009b), but they are an aggregate-level measure (i.e., the ratings represent the total audience for a particular program). Additionally, the ratings may contain error because (historically) they have been based on “people-meters,” which require viewers to push a button to indicate the beginning and ending of their viewing.

³ We view the general vs. specific distinction as one of many potential variations in question wording that may be used to facilitate recall (see Van Elsas, Lubbe, Van Der Meer, & Van Der Brug, 2014; or Timotijevic, Barnett, Shepherd, & Senior, 2009 for other applications).

⁴ Subjects were recruited to participate in exchange for extra credit and instructed to sign up for the study through an online appointment system. The studies were approved by the Institutional Review Board at Stony Brook University (Application #: 580472). See Druckman and Kam (2011) on the use of student samples in experimental research.

⁵ There were slight imbalances across conditions on partisanship, political knowledge, risk aversion, and presidential approval (see Table A1), however, a joint test reveals no significant difference in the overall composition of our experimental groups (p = .48).

⁶ In the original study, 66% of the respondents from Time 1 completed the follow-up. The re-contact rates for the first and second replications were 80% and 55%, respectively. Attrition was not significantly related to treatment assignment (see Table A1).

⁷ The treatment was an edited version (338 words) of an actual USA Today story that appeared on the newspaper’s website approximately 1 year before our study. We selected this article because it was geared toward young job seekers, and thus the topic should have had some appeal for our subjects.

⁸ To the extent that there are “errors” in self-reported measures of news exposure, these items will allow us to examine the characteristics of people who over or under report their exposure.

⁹ When combined with the initial (dichotomous) recall item, this resulted in a six-point measure ranging from “Yes, absolutely certain” to “No, absolutely certain.”

¹⁰ Respondents received a general or specific version of this question, depending on treatment assignment, along with the certainty follow-up. We confirmed via media content analysis that there had been no media coverage of human cloning around the time of the study.

¹¹ All statistical tests are two-tailed unless otherwise noted.

¹² For the treatment indicator, the marginal effect is .11. This model includes indicators for the first and second replications, along with the pretreatment measure of respondent attention. Excluding the latter (only including controls for the replications) results in a p-value of .14 for the treatment indicator (marginal effect = .11).

¹³ The sign and significance of the coefficient on the treatment indicator remains unchanged in a model that excludes the pretreatment measure of attention.

¹⁴ 67% of treated respondents accurately report exposure, which is slightly higher than the analogous figure reported in Ansolabehere and Iyengar (1995). Notably, there is no difference in the amount of time spent viewing the story according to self-reported recall status (i.e., answering “yes” or “no” to the recall question; p = .95).

¹⁵ We ignore the variation in question wording and collapse across the general and specific wording conditions (the effect of question wording is examined in the next subsection).

¹⁶ The results remain unchanged when we include a measure of self-reported attention to politics, which itself is insignificant (coeff = .06; s.e. = .12).

¹⁷ Table A3 provides a tabular presentation of the data from Figure 2. We estimated the model from Table 2 separately for respondents who accurately report their exposure and those who were inaccurate (over or under reporting). For accurate respondents, there is a positive but non-significant relationship between self-reported exposure and knowledge. For inaccurate respondents, that relationship is negative and significant (p < .001). This pattern underscores the importance of devising self-reported exposure questions that elicit accurate responses.

¹⁸ In explaining why more politically interested people over report media exposure, Prior states, they “rely too heavily on their generally high political involvement when estimating network news exposure without help. Even when they recall only a few episodes of news exposure, they may infer frequent exposure from their considerable interest in (and knowledge of) politics” (2009a, p. 901).

¹⁹ As noted earlier, those who said they recalled seeing something about job growth were asked whether the information pertained to “rates of job creation in different regions of the United States,” “a decline in the number of jobs in the restaurant industry,” or “a new federal program to train people receiving unemployment benefits.” This question appeared in Wave 2 and it represents a more challenging recall task than the previous two items (overall recall and recall certainty). That said, we are not comparing groups formed by random assignment since we analyze a subset of people from the treatment group (i.e., those who recalled exposure).

²⁰ Another challenge facing researchers who use survey-based measures of exposure is the steep decline in response rates (Kohut, Keeter, Doherty, Dimock, & Christian, 2012), and the potential for unobserved factors to be correlated with survey participation and self-reported media use (irrespective of how the exposure question is worded).

²¹ Implicit techniques such as the Implicit Association Test (IAT) or lexical decision tasks are another class of tools for measuring media effects, however, many challenges remain in adapting these measures for communications studies (see Hefner, Rothmund, Klimmt, & Gollwitzer, 2011 for discussion).

²² Experimental subjects who have seen a stimulus are asked about the likelihood they would have ordinarily encountered that material in the real world.