Three Reasons Samples Become Separated From Their Populations in Communication Research

Abstract

There are three reasons why probability samples often become separated from their putative populations in mass communication research. First, researchers sometimes believe populations must surely exist when in fact they do not exist. Yet they relentlessly pursue these phantom populations rather than heed classic warning signs that something is wrong. Second, researchers often struggle with underpowered samples. Rather than render those samples using more appropriate qualitative methods, they plow ahead using misapplied statistical methods and thus never inferentially connect to a population. Third, researchers frequently and incorrectly use random assignment rather than random selection as a sample method. This, in turn, can leave a sample unconnected to a population if the population can be drawn only through random selection. As obvious as these three errors are, researchers nonetheless stumble into them regularly. We examine why that is and what researchers can do to avoid these errors.

ACKNOWLEDGMENTS

We acknowledge the assistance of Professor Michael Shapiro, Department of Communication, Cornell University; Professor Jerome Frieman, Department of Psychology, Kansas State University; and Professor Dhavan Shah, School of Journalism and Mass Communication, University of Wisconsin–Madison. In addition, we thank the three anonymous reviewers and the journal editor who improved the finished product immeasurably.

Notes

¹To be clear, a population may exist even when a hypothesis is correctly disconfirmed. That may be a case where a probability sample does not respond to a particular independent variable cluster because those variables were not properly presented to participants (most recently addressed by Thorson, Wicks, & Leshner, 2012). There are also cases where samples do not record a response to independent variables because traditional analytical techniques do not detect those responses. We have characterized Type II errors as mass communication researchers' “elephant in the room,” given how many data sets are likely discarded because researchers are using off-the-shelf statistical packages, which use calculational formulas that are not suited to detect some mass communication effects (Grimes, Frieman, & Bergen, 2008). Namely, the way those products calculate error terms often washes away some communication effects. The upshot is that, with some adjustment in the stimulus set, the analysis, or an adjustment in an experimental design, the living, breathing sample may eventually connect to its living, breathing population.

Additional information

Notes on contributors

Tom Grimes

Tom Grimes (Ph.D., Indiana University, 1986) is a Professor in the School of Journalism and Mass Communication at Texas State University. His two principal research interests include issues in human information processing and the relationship between various psychopathologies and their interaction with media violence.

Kate Peirce

Kate Peirce (Ph.D., University of Texas–Austin, 1982) is a Professor in the School of Journalism and Mass Communication at Texas State University. Her research interests include the study of gender and media and the scholarship of pedagogy.