Sampling and validity

ABSTRACT

Sampling strategies are directly related to external validity. The choices researchers make in selecting sampling frames and sampling participants need to be clearly articulated. Sampling choices can introduce a variety of biases into research findings that reduce the external validity of samples. This essay discusses the relationship between sampling and external validity and provides a brief overview of important sampling concepts including power, the central limit theorem, nonprobability sampling and probability sampling. Sampling related biases are overviewed in the context of communication research. Finally, strategies that researchers and the discipline as a whole might adopt to ameliorate validity concerns related to sampling are proposed.

KEYWORDS:

• Sampling
• external validity
• replication
• probability sampling
• convenience sampling

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 The example here uses a mean as the statistic of interest for simplicity’s sake. Standard errors can be calculated for many different statistics.

2 Although Cohen (1992) provides cut-offs for small, medium, and large effects, it is still an open question of how large an effect size must be in communication studies to provide any useful application in the real world.

3 For specific convenience sampling techniques, the reader is referred to Croucher and Cronn-Mills (2015).

4 Certainly, one hopes judicious reviewers would not critique a sample simply for being a conveniently collected. The underlying concern is representativeness.

5 There are other valid reasons besides a negative correlation between GRE and success to question the use of the GRE in graduate admissions (see Posselt, 2016 for an overview).

6 Chmielwski and Kucker set their Wave 4 HIT requirements at ≥ 90% approval rate, and ≥ 100 approved HITs suggesting setting high approval requirements will not fully ameliorate MTurk data problems.

7 Sheehan (2018) presents multiple best practices for researchers using MTurk including those related to internal validity (such as including attention checks) as well as external validity (paying for specific characteristics needed in the sample, or using screening surveys rather than overt requirements to select on required variables).