Message-level Claims Require Message-level Data Analyses: Aligning Claims and Evidence in Communication Research

References

• Anderson, S., Frerichs, L., Kaysin, A., Wheeler, S. B., Halpern, C. T., & Lich, K. H. (2019). Effects of two educational posters on contraceptive knowledge and intentions: A randomized controlled trial. Obstetrics and Gynecology, 133(1), 53–62. https://doi.org/10.1097/AOG.0000000000003012
   PubMed Web of Science ®Google Scholar
• Atomistic Fallacy. (2008). In M. Porta (Ed.), A dictionary of epidemiology. Oxford University Press. https://doi.org/10.1093/acref/9780195314496.001.0001
   Google Scholar
• Bozinoff, L., & Ghingold, M. (1983). Evaluating guilt arousing marketing communications. Journal of Business Research, 11(2), 243–255. https://doi.org/10.1016/0148-2963(83)90031-0
   Web of Science ®Google Scholar
• Brashers, D. E., & Jackson, S. (1999). Changing conceptions of “message effects”: A 24-year overview. Human Communication Research, 25(4), 457–477. https://doi.org/10.1111/j.1468-2958.1999.tb00456.x
   Web of Science ®Google Scholar
• Bullock, J. G., & Green, D. P. (2021). The failings of conventional mediation analysis and a design-based alternative. Advances in Methods and Practices in Psychological Science, 4(4), 1–18. https://doi.org/10.1177/25152459211047227
   Google Scholar
• Burns, S. M., Barnes, L. N., McCulloh, I. A., Dagher, M. M., Falk, E. B., Storey, J. D., & Lieberman, M. D. (2019). Making social neuroscience less WEIRD: Using fNIRS to measure neural signatures of persuasive influence in a Middle East participant sample. Journal of Personality and Social Psychology, 116(3), E1–E11. https://doi.org/10.1037/pspa0000144
   PubMed Web of Science ®Google Scholar
• Chan, M., Hu, P., & Mak, M. K. F. (2022). Mediation analysis and warranted inferences in media and communication research: Examining research design in communication journals from 1996 to 2017. Journalism & Mass Communication Quarterly, 99(2), 463–486. https://doi.org/10.1177/1077699020961519
   Web of Science ®Google Scholar
• Choi, W., Kim, S.-Y., & Luo, M. (2022). Design matters in web credibility assessment: Interactive design as a social validation tool for online health information seekers. Asian Communication Research, 19(3), 119–138. https://doi.org/10.20879/acr.2022.19.3.119
   Google Scholar
• Clifford, S., Leeper, T. J., & Rainey, C. (in press). Generalizing survey experiments using topic sampling: An application to party cues. Political Behavior. https://doi.org/10.1007/s11109-023-09870-1
   Web of Science ®Google Scholar
• Clifford, S., & Rainey, C. (2024, February 19). The limits (and strengths) of single-topic experiments. SocArxiv Manuscript. https://doi.org/10.31235/osf.io/zaykd
   Google Scholar
• Diez-Roux, A. V. (1998). Bringing context back into epidemiology: Variables and fallacies in multilevel analysis. American Journal of Public Health, 88(2), 216–222. https://doi.org/10.2105/ajph.88.2.216
   PubMed Web of Science ®Google Scholar
• Dillard, A. J., Ferrer, R. A., & Welch, J. D. (2018). Associations between narrative transportation, risk perception and behaviour intentions following narrative messages about skin cancer. Psychology & Health, 33(5), 573–593. https://doi.org/10.1080/08870446.2017.1380811
   PubMed Web of Science ®Google Scholar
• Dillard, J. P., Weber, K. M., & Vail, R. G. (2007). The relationship between the perceived and actual effectiveness of persuasive messages: A meta-analysis with implications for formative campaign research. Journal of Communication, 57(4), 613–631. https://doi.org/10.1111/j.1460-2466.2007.00360.x
   Web of Science ®Google Scholar
• Falk, E. B., Cascio, C. N., & Coronel, J. C. (2015). Neural prediction of communication-relevant outcomes. Communication Methods and Measures, 9(1–2), 30–54. https://doi.org/10.1080/19312458.2014.999750
   Web of Science ®Google Scholar
• Fiedler, K., Harris, C., & Schott, M. (2018). Unwarranted inferences from statistical mediation tests: An analysis of articles published in 2015. Journal of Experimental Social Psychology, 75, 95–102. https://doi.org/10.1016/j.jesp.2017.11.008
   Web of Science ®Google Scholar
• Firebaugh, G. (2015). Ecological fallacy, statistics of. International Encyclopedia of the Social & Behavioral Sciences (2nd ed. Vol. 6, pp. 865–867). https://doi.org/10.1016/B978-0-08-097086-8.44017-1
   Google Scholar
• Gong, H., Huang, M., & Liu, X. (2022). Message persuasion in the pandemic: U.S. and Chinese respondents’ reactions to mediating mechanisms of efficacy. International Journal of Communication, 16, 840–863. https://ijoc.org/index.php/ijoc/article/view/17839
   Web of Science ®Google Scholar
• Green, M. C., & Brock, T. C. (2000). The role of transportation in the persuasiveness of public narratives. Journal of Personality and Social Psychology, 79(5), 701–721. https://doi.org/10.1037/0022-3514.79.5.701
   PubMed Web of Science ®Google Scholar
• Green, M. C., & Clark, J. L. (2013). Transportation into narrative worlds: Implications for entertainment media influences on tobacco use. Addiction, 108(3), 477–484. https://doi.org/10.1111/j.1360-0443.2012.04088.x
   PubMed Web of Science ®Google Scholar
• Hackworth, E. E., Budiongan, J. R., Lambert, V. C., Kim, M., Ferguson, S. G., Niederdeppe, J., Hardin, J., & Thrasher, J. F. (2023). A mixed-method study of perceptions of cigarette pack inserts among adult smokers from New York and South Carolina exposed as part of a randomized controlled trial. Health Education Research, 38(6), 548–562. https://doi.org/10.1093/her/cyad030
   PubMed Web of Science ®Google Scholar
• Imhof, M. A., Schmalzle, R., Renner, B., & Schupp, H. T. (2017). How real-life health messages engage our brains: Shared processing of effective anti-alcohol videos. Social Cognitive and Affective Neuroscience, 12(7), 1188–1196. https://doi.org/10.1093/scan/nsx044
   PubMed Web of Science ®Google Scholar
• Jackson, S., & Jacobs, S. (1983). Generalizing about messages: Suggestions for design and analysis of experiments. Human Communication Research, 9(2), 169181. https://doi.org/10.1111/j.1468-2958.1983.tb00691.x
   Web of Science ®Google Scholar
• Jackson, S., O’Keefe, D. J., & Jacobs, S. (1988). The search for reliable generalizations about messages: A comparison of research strategies. Human Communication Research, 15(1), 127–142. https://doi.org/10.1111/j.1468-2958.1988.tb00174.x
   Web of Science ®Google Scholar
• Jackson, S., O’Keefe, D. J., Jacobs, S., & Brashers, D. E. (1989). Messages as replications: Toward a message-centered design strategy. Communication Monographs, 56(4), 364–384. https://doi.org/10.1080/03637758909390270
   Web of Science ®Google Scholar
• Jongenelis, M. I., Drane, C., Hasking, P., Chikritzhs, T., Miller, P., Hastings, G., & Pettigrew, S. (2023). Development and validation of the alcohol message perceived effectiveness scale. Scientific Reports, 13(1), 997. https://doi.org/10.1038/s41598-023-28141-x
   PubMedGoogle Scholar
• Kievit, R. A., Frankenhuis, W. E., Waldorp, L. J., & Borsboom, D. (2013). Simpson’s paradox in psychological science: A practical guide. Frontiers in Psychology, 4, 513. https://doi.org/10.3389/fpsyg.2013.00513
   PubMed Web of Science ®Google Scholar
• Kikut, A. I., & Trzebiński, W. (2023). The doctor knows or the evidence shows: An online survey experiment testing the effects of source trust, pro-vaccine evidence, and dual-processing in expert messages recommending child COVID-19 vaccination to parents. Public Library of Science ONE, 18(7), e0288272. https://doi.org/10.1371/journal.pone.0288272
   Google Scholar
• Koranteng, F. N., Ham, J., Wiafe, I., & Matzat, U. (2022). The role of usability, aesthetics, usefulness and primary task support in predicting the perceived credibility of academic social networking sites. Behaviour & Information Technology, 41(16), 3617–3632. https://doi.org/10.1080/0144929X.2021.2009570
   Web of Science ®Google Scholar
• Krisher, H. P., Darley, S. A., & Darley, J. M. (1973). Fear-provoking recommendations, intentions to take preventive actions, and actual preventive actions. Journal of Personality and Social Psychology, 26(2), 301–308. https://doi.org/10.1037/h0034465
   PubMed Web of Science ®Google Scholar
• Kwon, E. S., King, K. W., Nyilasy, G., & Reid, L. N. (2019). Impact of media context on advertising memory: A meta-analysis of advertising effectiveness. Journal of Advertising Research, 59(1), 99–128. https://doi.org/10.2501/JAR-2018-016
   Web of Science ®Google Scholar
• Lee, S. J., Brennan, E., Gibson, L. A., Tan, A. S. L., Kybert-Momjian, A., Liu, J., & Hornik, R. (2016). Predictive validity of an empirical approach for selecting promising message topics: A randomized-controlled study. Journal of Communication, 66(3), 433–453. https://doi.org/10.1111/jcom.12227
   PubMed Web of Science ®Google Scholar
• Loh, W. W., & Ren, D. (2022). Improving causal inference of mediation analysis with multiple mediators using interventional indirect effects. Social and Personality Psychology Compass, 16(10), e12708. https://doi.org/10.1111/spc3.12708
   Web of Science ®Google Scholar
• Mackenbach, J. P. (2000). Dwalingen in de methodologie. XXVI. De ecologische valkuil en zijn minder bekende tegenhanger, de atomistische valkuil [Roaming through methodology. XXVI. The ecological fallacy and its less well-known counterpart, the atomistic fallacy]. Nederlands tijdschrift voor geneeskunde, 144(44), 2097–2100. https://www.ntvg.nl/artikelen/dwalingen-de-methodologie-xxvi-de-ecologische-valkuil-en-zijn-minder-bekende-tegenhanger
   PubMedGoogle Scholar
• Malinas, G., & Bigelow, J. (2016). Simpson’s paradox. The Stanford encyclopedia of philosophy. https://plato.stanford.edu/archives/fall2016/entries/paradox-simpson
   Google Scholar
• Malthouse, E. C., & Calder, B. J. (2010). Media placement versus advertising execution. International Journal of Market Research, 52(2), 217–230. https://doi.org/10.2501/S1470785309201181
   Web of Science ®Google Scholar
• Malthouse, E. C., Calder, B. J., & Tamhane, A. (2007). The effects of media context experiences on advertising effectiveness. Journal of Advertising, 36(3), 7–18. https://doi.org/10.2753/JOA0091-3367360301
   Web of Science ®Google Scholar
• Ma, Z., & Ma, R. (2022). Predicting intentions to vaccinate against COVID-19 and seasonal flu: The role of consideration of future and immediate consequences. Health Communication, 37(8), 952–961. https://doi.org/10.1080/10410236.2021.1877913
   PubMed Web of Science ®Google Scholar
• Meule, A. (2019). Contemporary understanding of mediation testing. Meta-Psychology, 3, MP.2018.870. https://doi.org/10.15626/MP.2018.870
   Google Scholar
• Nan, X., Dahlstrom, M. F., Richards, A., & Rangarajan, S. (2015). Influence of evidence type and narrative type on HPV risk perception and intention to obtain the HPV vaccine. Health Communication, 30(3), 301–308. https://doi.org/10.1080/10410236.2014.888629
   PubMed Web of Science ®Google Scholar
• Ntoumanis, I., Davydova, A., Sheronova, J., Panidi, K., Kosonogov, V., Shestakova, A. N., Jääskeläinen, I. P., & Klucharev, V. (2023). Neural mechanisms of expert persuasion on willingness to pay for sugar. Frontiers in Behavioral Neuroscience, 17, 1147140. https://doi.org/10.3389/fnbeh.2023.1147140
   PubMed Web of Science ®Google Scholar
• O’Keefe, D. J. (2018). Message pretesting using assessments of expected or perceived persuasiveness: Evidence about diagnosticity of relative actual persuasiveness. Journal of Communication, 68(1), 120–142. https://doi.org/10.1093/joc/jqx009
   Web of Science ®Google Scholar
• O’Keefe, D. J. (2020). Message pretesting using perceived persuasiveness measures: Reconsidering the correlational evidence. Communication Methods and Measures, 14(1), 25–37. https://doi.org/10.1080/19312458.2019.1620711
   Web of Science ®Google Scholar
• O’Keefe, D. J. (2021). Persuasive message pretesting using non-behavioral outcomes: Differences in attitudinal and intention effects as diagnostic of differences in behavioral effects. Journal of Communication, 71(4), 623–645. https://doi.org/10.1093/joc/jqab017
   Web of Science ®Google Scholar
• Panagopoulos, C. (2014). Raising hope: Hope inducement and voter turnout. Basic and Applied Social Psychology, 36(6), 494–501. https://doi.org/10.1080/01973533.2014.958228
   Web of Science ®Google Scholar
• Park, H. S., Eveland, W. P., Jr., & Cudeck, R. (2008). Multilevel modeling: Studying people in contexts. In A. F. Hayes, M. D. Slater, & L. B. Snyder (Eds.), The SAGE sourcebook of advanced data analysis methods for communication research (pp. 219–245). Sage.
   Google Scholar
• Ratcliff, C. L., Jensen, J. D., Scherr, C. L., Krakow, M., & Crossley, K. (2019). Loss/Gain framing, dose, and reactance: A message experiment. Risk Analysis, 39(12), 2640–2652. https://doi.org/10.1111/risa.13379
   PubMed Web of Science ®Google Scholar
• Reeves, B., Yeykelis, L., & Cummings, J. J. (2016). The use of media in media psychology. Media Psychology, 19(1), 49–71. https://doi.org/10.1080/15213269.2015.1030083
   Web of Science ®Google Scholar
• Roberto, A., Eden, J., Deiss, D., Savage, M., & Ramos-Salazar, L. (2017). The short-term effects of a cyberbullying prevention intervention for parents of middle school students. International Journal of Environmental Research and Public Health, 14(9), 1038. https://doi.org/10.3390/ijerph14091038
   PubMed Web of Science ®Google Scholar
• Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15(3), 351–357. https://doi.org/10.2307/2087176
   Web of Science ®Google Scholar
• Robson, K., & Pevalin, D. (2016). Multilevel modeling in plain language. Sage.
   Google Scholar
• Rohrer, J. M., Hünermund, P., Arslan, R. C., & Elson, M. (2022). That’s a lot to process! Pitfalls of popular path models. Advances in Methods and Practices in Psychological Science, 5(2). https://doi.org/10.1177/25152459221095827
   PubMedGoogle Scholar
• Simchon, A., Edwards, M., & Lewandowsky, S. (2024). The persuasive effects of political microtargeting in the age of generative artificial intelligence. Proceedings of the National Academy of Sciences Nexus, 3(2), 035. https://doi.org/10.1093/pnasnexus/pgae035
   Google Scholar
• Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13(2), 238–241. https://doi.org/10.1111/j.2517-6161.1951.tb00088.x
   Web of Science ®Google Scholar
• Tal-Or, N., & Cohen, J. (2016). Unpacking engagement: Convergence and divergence in transportation and identification. Annals of the International Communication Association, 40(1), 33–66. https://doi.org/10.1080/23808985.2015.11735255
   Google Scholar
• Wang, A. L., Ruparel, K., Loughead, J. W., Strasser, A. A., Blady, S. J., Lynch, K. G., Romer, D., Cappella, J. N., Lerman, C., & Langleben, D. D. (2013). Content matters: Neuroimaging investigation of brain and behavioral impact of televised anti-tobacco public service announcements. Journal of Neuroscience, 33(17), 7420–7427. https://doi.org/10.1523/JNEUROSCI.3840-12.2013
   PubMed Web of Science ®Google Scholar
• Xu, J. (2022). The impact of guilt and shame in charity advertising: The role of self‐construal. Journal of Philanthropy and Marketing, 27(1), e1709. https://doi.org/10.1002/nvsm.1709
   Web of Science ®Google Scholar
• Xu, S., Coffman, D. L., Luta, G., & Niaura, R. S. (2023). Tutorial on causal mediation analysis with binary variables: An application to health psychology research. Health Psychology, 42(11), 778–787. https://doi.org/10.1037/hea0001299
   PubMed Web of Science ®Google Scholar
• Xu, Z., & Guo, H. (2018). A meta-analysis of the effectiveness of guilt on health-related attitudes and intentions. Health Communication, 33(5), 519–525. https://doi.org/10.1080/10410236.2017.1278633
   PubMed Web of Science ®Google Scholar