Improving the replicability and generalizability of inferences in quantitative communication research

ABSTRACT

This paper discusses the assessment of quality of quantitative communication research in light of the so-called ‘replicability crisis’ that has affected neighboring disciplines. For social scientific research, it is useful to think of research results as estimates which include error. I propose a framework suited to a variable field like communication, factoring in all sources of error, for assessing the quality of research. In communication research, greater consideration of generalizability is essential, which is at once both a higher standard than replicability but also a goal that should increase it. Furthermore, more explicit discussion of generalizability may help to further internationalize the discipline by clarifying the limitations of the large portion of research conducted within a narrow subset of world cultures.

KEYWORDS:

• Inference
• replication
• generalizability

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1 To be clear, the available evidence suggests the largest causes of the replication crisis in psychology were a combination of publication bias, questionable research practices, and research designs with poor statistical power (Vermeulen et al., 2015). But when better statistical and disclosure practices are established, issues of generalizability will become one of the most important remaining obstacles for valid inferences.

2 Communication Research, Journal of Broadcasting & Electronic Media, Journal of Communication, Journal of Computer-Mediated Communication, Journalism and Mass Communication Quarterly, and Mass Communication and Society.

3 This comparison is explained in more technical detail by Mercer et al. (2017).

4 I omit some references to the differences between White and Black soldiers.

5 This is not always an example of model-based inference. Sometimes, probability of selection is known due to the sampling design and in this case the weighting is still part of design-based statistical inference. Other cases, like the use of population parameters to generate weights, are model-based inferences targeted to reduce sampling error.

Additional information

Funding

The author reported there is no funding associated with the work featured in this article.