Improving IS Practical Significance through Effect Size Measures

ABSTRACT

Evidence-based practice in management assigns a high value to research results as a guide to practices that have been rigorously shown to be effective. To emphasize the practical relevance and outcomes for information systems research, statistical research should generally report its effect sizes. Specifying effect sizes not only reveals the utility of our results, but it also enables evidence-based practitioners to easily compare the known effects of different interventions applied in different studies. Effect size reporting has become a standard practice in many fields, however, though information systems researchers have adopted many other elements of statistical rigor, effect sizes are often overlooked. This paper surveys the current use of effect size calculations in information systems research, explains how such effects sizes are calculated, offers recommendations on when each of the different formulae is appropriate, and provides foundational work toward an index of expected effect sizes in information systems research.

KEYWORDS:

• Effect size
• p-value
• Cohen’s d
• statistical significance
• practical significance

Acknowledgments

The authors gratefully acknowledge the very helpful comments of Detmar Straub and Kaveh Mohajeri.

Notes

¹ This has not always been the case. As recently as 2012 medical journal editors were calling for greater reporting of effect size in their discipline.⁴⁹ The successes and improvements in other disciplines should serve as encouragement that information systems can follow suit.

² These measures are used when dependent variables are continuous variables. On the other hand, when dependent variables are categorical variables, measures such as the relative risk, the odds ratio, and rate ratio are used to determine whether the probability of a certain event differs across groups.⁵

³ There are more than 70 effect size measures in the literature. For a more complete list, please refer to Table 5.1 from Kirk.¹⁸

⁴ Interested readers can refer to Cohen⁶ for a detailed explanation regarding why p (H₀|D) ≠ p (D|H₀).

⁵ According to Kelley and Preacher,⁵⁰ Effect size is “a quantitative reflection of the magnitude of some phenomenon that is used for the purpose of addressing a question of interest” (p. 140). Therefore, R² is less helpful because it describes the effect of all independent variables in the regression rather than the effect of a particular independent variable. Although the increment in R² can be calculated to assess the effect of certain variables, it does not represent the rate of change.⁵¹ Therefore, R² is less helpful to understand the practical significance of certain variables.

⁶ https://aisnet.org/?SeniorScholarBasket

⁷ Please note that practical significance may not always lead to relevance, or vice versa (Mohajeri et al. Forthcoming).