Interpreting Perceived Effectiveness: Understanding and Addressing the Problem of Mean Validity

Abstract

Research has shown that perceived message effectiveness (PE) correlates reasonably well with indices of actual effectiveness, but little attention has been given to how to interpret mean PE. This article describes the problem of mean validity and presents a research design that can be used to address it. Participants (N = 195) viewed messages that advocated being screened for colorectal cancer. The results showed downward bias in PE among members of the non–target audience (persons younger than 50) and upward bias as the referent for the judgment became more abstract/distant (self vs. persons older than 50 vs. general). The need for more research on mean validity is discussed. For applied researchers, recommendations for preferred indices of PE are offered.

Notes

¹ Although there was some variation in both correlations and means across the messages, it was not of sufficient magnitude to alter any substantive conclusions.

² It was possible to test this guideline by breaking our sample into three age groups: 50 and older, 35–49, and 18–34. For each of the three PE indices, the data showed a pattern of means in which the two oldest groups were similar and the youngest group was noticeably lower. For example, self-PE means were 1.89, 1.93, and 1.36 for the oldest to youngest groups, respectively. The youngest group was different from each of the other groups at p < .08. The more similar the group, the more accurate the PE estimate.

³ In this vein, we note that the younger than 50 PE—general mean (1.74) is closer to the gold-standard group (1.89) than is the mean for older than 50 PE—self (2.23, which was different from the gold standard of 1.89 at p < .05). For distant-target data, the general estimate was more accurate than the self estimate.