The Statistical and Methodological Acuity of Scholarship Appearing in the International Journal of Listening (1987–2011)

Abstract

This article investigates the quality of social scientific listening research that reports numerical data to substantiate claims appearing in the International Journal of Listening between 1987 and 2011. Of the 225 published articles, 100 included one or more studies reporting numerical data. We frame our results in terms of eight recommendations to improve future listening scholarship. In particular, the results suggest needed variation in demographics and added attention to psychometric properties of scores. Standards for reporting and inspecting data should also be followed with more regularity, and tests of statistical assumptions along with information about missing data are urged. Effect sizes are rarely included in results, and no studies reported confidence intervals, suggesting overreliance on null hypothesis statistical testing when drawing implications for practice. Lastly, there were some noteworthy misappropriations of statistical techniques that are discussed.

Notes

¹We wish to point out here the focus of this article is on one branch of research on listening, and although large, it in no way constitutes the entirety of the tree. As such, this article essentially “ignores” other, equally valid conceptualizations (Gehrke, 2009). The interested readers are directed to the following sources for alternative conceptualizations of listening: Beard (2009), Ihde (2007), Lipari (2009, 2010), and Purdy (2000).

²Our focus on research reporting numbers in no way asserts a preference for this type of research; it merely reflects the expertise of the authors. We encourage others to engage in similar projects focusing on separate classes of research.

³We cautiously utilize the term “quantitative” here and urge readers not to interpret us as supporting the quantitative-qualitative dichotomy (Bodie, 2011).

⁴Reporting conventions for meta-analytic reviews are remarkably different from those for individual studies.

⁵Because the sample size of those studies that reported age and standard deviation did not meet the requirements for detecting small effects, a Kruskal-Wallis test was performed in each instance indicating that the mean ranks of each category did not vary significantly across five-year intervals, supporting the non-significant ANOVA results: average age means, H(4) = 2.35, p < .67, and mean standard deviations, H(2) = 0.614, p < .73. The listing of age descriptives varied with 25 (64.1%) reporting means, 26 (39.4%) age ranges, and 11 (16.7%) including standard deviations.

FIGURE 2 Mean age and standard deviations per five-year interval. The mean age for 1987–1991 is skewed because of a single study that utilized third grade students and reported a mean age of 8.8 years. With that study removed, the average age is 22.4 years (color figure available online).

⁶Calculating the median (ME = 160) when leaving the outliers in the distribution shows that the two measures of central tendency are approximately equal.

⁷The authors are fully aware that newer versions, Forms D and E, are now commercially available. We warn against their use in studies, however, without proper inspection of psychometric properties of the data prior to reporting results. The results presented with respect to Form C are not uncommon, as similar results were reported for earlier versions (e.g., Villaume & Weaver, 1997; Fitch-Hauser & Hughes, 1987).