![Sample our Humanities journals, sign in here to start your FREE access for 14 days]({"type":"image" , "src" : "/sda/5939/TOC-Subject-Sample-2021-Humanities.jpg"})
ABSTRACT
A content analysis of 900 political- and health-related media articles published in 11 outlets from 2010 through 2015 reveals a complete disconnect between discussions of statistical power and alpha levels. This study proposes the use of discontinuous criterion power analyses to address this power-alpha blind spot. Additional analyses indicate a sizeable percentage (41.1%) of published works in these areas retaining sufficient statistical power (i.e. .95 or greater) to warrant the use of a stricter alpha level than the traditional α = .05 to detect small effects. The surgical use of more stringent alpha levels to guard against Type I error will instill greater confidence in the empirical findings generated by media research conducted in political and health contexts.
Disclosure statement
No potential conflict of interest was reported by the authors.
Notes on contributors
R. Lance Holbert, Ph.D. is Professor and Chair of Department of Communication and Social Influence.
Bruce W. Hardy is an Assistant Professor in the Department of Communication and Social Influence.
Esul Park is a doctoral student in the Media and Communication program.
Nicholas W. Robinson is a doctoral student in the Media and Communication program.
Heeyoung Jung is a doctoral student in the Media and Communication program.
Chen Zeng is a doctoral student in the Media and Communication program.
Erin Drouin is a graduate student in the Department of Communication.
Kelly Sweeney is a graduate student in the Department of Communication.
Notes
1 The definition of political communication used for this study comes from Perloff (Citation2014):
Political communication is the process by which language and symbols, employed by leaders, media, and citizens, exert intended or unintended effects on the political cognitions, attitudes, or behaviors of individuals or on outcomes that bear on public policy of a nation, state, or community (p. 30).
2 A dictionary of terms was also created that included words or symbols often associated with sample size, effect size, and alpha level to look for any implicit discussions of statistical power. Researchers could be implying the concept of statistical power through their commentary on a combination of sample size, effect size, and alpha level in close proximity of one another and this study wanted to account for this possibility. The following terms were included in the dictionary: ‘effect’, ‘coefficient’, ‘alpha’, ‘α’, ‘p-’, ‘p <’, ‘p >’, ‘subject’, ‘sample’, ‘observation’, and ‘participant’. Each term was inserted in the AdobeAcrobat search function and the same process used for the term ‘power’ in the earlier explicit phase was followed for each term in a random selection of 200 of the political communication works. This laborious process produced only one article for inclusion (see Kaufhold, Valenzuela, & de Zuniga, Citation2010, Note 47, p. 529). This one piece was included for subsequent analysis, but we did not perform this analysis activity for the remaining political communication pieces or any of the health communication pieces given the low yield rate. In short, we feel confident in stating that if communication researchers are discussing statistical power they are doing so explicitly (i.e. using the term power).
3 The definition of health communication used for this study comes from Kreps and Thornton (Citation1992): health communication is ‘how we seek, process, and share meaning regarding health and health information’ (p. 2).
4 Power of .95 is utilized in this example. However, a researcher could just as easily employ a weaker guard against Type II error (e.g. power = .80, power = .90) that is still deemed acceptable by Cohen (Citation1988) and utilize this power level throughout the same iterative process. It all depends on what degree of Type II error a researcher wishes to work with while seeking to address Type I error.
5 ‘pwr’ utilizes the same Cohen-based power analysis procedures as GPower, but does not include as wide a range of statistical procedures. R contains several packages of potential interest to political media researchers who wish to be mindful of statistical power (e.g. Qiu [Citation2013] and ‘powerMediation’).
6 GPower does retain a criterion power analysis option as well, but this function is most clearly associated with the type of unbridled analysis (i.e. allowing for the generation of alpha levels greater than .05) that we have chosen to restrain. In addition, the criterion power procedures will produce exact alpha levels (e.g. .034, .017) that do not conform to the traditional cut-offs established within the field (e.g. .05, .01, .001). It is for these reasons we feel the use of the post-hoc function works best for the proposed discontinuous criterion power analysis.
7 The focus of this work is on the detection of small effects given that if we find a substantial number of works can utilize stricter alpha levels to detect this size of an effect, then a stronger case is made for the need to engage in the proposed discontinuous criterion power analysis procedure. However, a researcher could also make an argument that the focus of a given research effort is on phenomenon that should reflect moderate or large effect sizes and the discontinuous criterion power analysis could be employed just as easily while working under this assumption.
8 Studies utilizing analytical procedures in the binomial family (e.g. logistic regression) do not report the details needed to conduct the type of analyses undertaken for this study. For example, one necessary component to input within GPower for logistic regression is ‘R2 other X’ which is the variance accounted for in the predictor variable of interest by all other covariates included in an equation. This is not a detail typically supplied in tabular or textual form for logistic regression analyses, but would be a statistic that could be calculated easily by the researchers for the purposes of a discontinuous criterion power analysis. Any randomly-selected article utilizing analyses in the binominal family was placed to the side and a new article was selected that allowed for a proper discontinuous criterion power analysis to be undertaken given the information reported in the work. The same procedure was performed for the random selection of health communication articles.