Framing COVID-19 Preprint Research as Uncertain: A Mixed-Method Study of Public Reactions

ABSTRACT

During the COVID-19 pandemic, journalists were encouraged to convey uncertainty surrounding preliminary scientific evidence, including mentioning when research is unpublished or unverified by peer review. To understand how public audiences interpret this information, we conducted a mixed method study with U.S. adults. Participants read a news article about preprint COVID-19 vaccine research in early April 2021, just as the vaccine was becoming widely available to the U.S. public. We modified the article to test two ways of conveying uncertainty (hedging of scientific claims and mention of preprint status) in a 2 × 2 between-participants factorial design. To complement this, we collected open-ended data to assess participants’ understanding of the concept of a scientific preprint. In all, participants who read hedged (vs. unhedged) versions of the article reported less favorable vaccine attitudes and intentions and found the scientists and news reporting less trustworthy. These effects were moderated by participants’ epistemic beliefs and their preference for information about scientific uncertainty. However, there was no impact of describing the study as a preprint, and participants’ qualitative responses indicated a limited understanding of the concept. We discuss implications of these findings for communicating initial scientific evidence to the public and we outline important next steps for research and theory-building.

Author contributions

CR, AK, and JJ conceptualized the study and developed the questionnaire. CR and RW developed the experimental stimulus. JJ supervised data collection. CR performed quantitative analyses. CR and AF developed the codebook and analyzed the qualitative data. CR, AF, RW, and BH wrote the manuscript and AK and JJ provided substantive comments on the manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability

Data and supplementary materials are available at https://osf.io/w4djv/.

Notes

1. https://connect.biorxiv.org/relate/content/181

2. We included these statements of assurance, which were present in the original news article, in all conditions to avoid raising doubt about vaccine efficacy, which we believe could have ethical consequences that outweigh the benefits of a cleaner manipulation. Additionally, participants were debriefed about the manipulations after the study and given the chance to read the original news article.

3. Because we administered the PIUS scale posttest, we examined whether participants’ condition may have influenced their response to the scale. Participants who received hedged messages reported slightly lower preference for uncertain information (M = 3.82, SD = .88) than those who received unhedged messages (M = 3.96, SD = .85), but the difference was not statistically significant (M_diff = .14, p = .091). There was no difference in preference between those who received preprint disclosure (M = 3.93, SD = .83) versus non-disclosure (M = 3.87, SD = .90; M_diff = .06, p = .480). Therefore, we assume PIUS was not influenced by the experimental messages.

4. This made it possible for the stimulus to influence participants’ open-ended responses, so we used a chi-square test to assess whether people in the preprint disclosure conditions gave more accurate definitions of preprint. As described in the Supplemental Files, there was no evidence of a contaminating effect.

5. The WHO reported in March 2021 that “mild side effects after getting vaccinated … are signs that your body is building protection” (WHO, 2021). This claim received widespread news coverage. The information presented in our study pertained to uncertainty about whether a person was still protected in the absence of side effects. It is possible that some people misinterpreted this to indicate uncertainty about the link between side effects and efficacy in either direction, which could have created a perception of conflicting information.

6. It will be useful to gain a sense of how common low (or high) PIUS is in the general population. We can offer two data points. When examining the distribution of a continuous variable, the 16^th, 50^th, and 84^th percentiles are useful benchmarks for low, moderate, and high levels (Hayes, 2018). In the current study, mean scale values representing the 16^th, 50^th, and 84^th percentiles of PIUSwere 3.00, 4.00, and 4.86, respectively (range: 1–5). Therefore, negative effects of hedging occurred for participants with low to moderate PIUS (M ≤ 3.76–4.11, depending on the outcome), and these participants comprised roughly half the sample. We observed a similar distribution in another study (Ratcliff & Wicke, 2022), where mean values representing the 16^th, 50^th, and 84^th percentiles were 3.43, 4.00, and 4.71. In that study, which investigated similar research questions in a different health context, negative effects of hedging on scientist trustworthiness and news credibility occurred only for those with low PIUS (~20%), while positive effects occurred for those with high PIUS (~20%), and the rest of the sample responded neutrally to hedging. The topic of that study was a relatively low-stakes issue, which could explain why responses to hedging were generally more favorable in that study compared to the current study.

Additional information

Funding

The work was supported by the University of Utah Immunology, Inflammation, and Infectious Disease Initiative [PIs: J. D. Jensen & A. J. King].