Statistical considerations (or recommendations) for publishing in Science and Medicine in Football

During the last months, the world of sport science has witnessed an important but also hard-nosed debate on a statistical approach commonly used in our area: the Magnitude Based Inferences (MBI; Batterham and Hopkins 2006). While concerns have already been raised about this method a few years ago (Welsh and Knight 2015), the publication by Sainani in 2018 (Sainani 2018) has attracted widespread attention of other statisticians and researchers (Borg et al. 2018). In addition to the recent scientific publications, several discussions have also been posted in social media and blogs, which exemplify the modern way to debate, even in science. The replies from the proponents of MBI (Alan Batterham and Will Hopkins) in response to the points of criticism being made can be found here: www.sportsci.org. Based on the criticism of the method, some journals such as Medicine and Science in Sports and Exercise (see journal’s Instructions for Authors), which incidentally is also the journal which contributed to the popularisation of this method (Hopkins et al. 2009), has decided to ban MBI analyses completely. Irrespective of the decision from other journals, the question inevitably also affects Science and Medicine in Football. As editors we try to offer a reasonable and balanced solution, while following the debate closely and possibly awaiting a conclusion. This editorial summarizes our recommendations on the preferred statistical approaches when submitting a paper to our journal.

MBIs were developed to address the drawbacks related to the Null Hypothesis Significance Testing (NHST). A shared view in this debate, however, is that effectively NHST has several shortcomings as already underlined in the literature (Szucs and Ioannidis 2017). Where the positions diverge is in the ‘solution’. In light of this, we believe that the main analysis reported in the studies should provide information about the magnitude of an effect and the precision of the estimate. In other words, we recommend that papers submitted to our journal report the confidence intervals and effect sizes according to other guidelines such as those of the American Psychological Association (2010) or author guidelines like the instructions of the International Journal of Epidemiology (IJE 2018). As commonly used, the confidence level should be 95%. However, in particular cases, the author may give reasons to apply a different level of confidence. Other approaches such as MBI or NHST can be used but only as a complementary analysis. Relying on confidence intervals and effect sizes facilitates researchers to evaluate and interpret the effects also in terms of relevance (practical, physiological or clinical). This was one of the goals of MBI but even without using this approach, we strongly encourage the authors to discuss and comment the relevance of the magnitude of the effects even if in a qualitative way. As an example, in studies on football match analysis: Is a difference of 50 m in high intensity running practically important? Is this difference large enough to justify different training strategies? And what about the uncertainty as reflected by the width of the confidence interval? It is also important to remember that the confidence intervals can be interpreted in the sense of the NHST by observing whether the confidence intervals overlap the value of no effect, i.e., 0 for group differences, or 1 in the case of relative risks, corresponding to the null hypothesis. So, reporting the p values may even be redundant (but we leave the authors freedom to decide whether to report or not). Furthermore, as suggested by Lakens (2017), there are established alternative test procedures that can also be used such as the equivalence tests if we want to stay in the frequentist realm or the region of practical equivalence if we move to a fully Bayesian approach. The latter has been suggested to provide similar results compared to the MBI and uses the smallest effect of interest instead of the smallest worthwhile change of the MBI (Hopkins and Batterham 2018). Recommending the confidence intervals and the effect sizes as primary analysis, we clearly push the authors to go beyond the p values and for this reason, the main interpretation of the study results should be based on confidence intervals, effect sizes, practical significance and uncertainty (width of the confidence interval).

Whatever you use, interpret it correctly

While the aim of this editorial is not to be a tutorial, we would, however, like to remind authors that whatever the statistical approach they decide to use, the interpretation of the results should be appropriate, commensurate to the method used, and interpreted in light of practical, as well as clinical relevance. We, therefore, strongly encourage avoiding overinterpretation or biased conclusions. For interpreting in the correct way, confidence intervals and – in the framework of complementary analyses – p values we suggest reading the several methodological papers available in the literature (e.g., Greenland et al. 2016). As far as MBIs are concerned, as stated above, we do permit its use in Science and Medicine in Football as a potential additional tool, but we also strongly advise the authors to interpret it in the correct way. For example, a ‘likely’ or ‘possible’ effect cannot be considered ‘certainly’ substantial (beneficial or harmful) as it sometimes appears in several paper discussions and/or conclusions (in good or bad faith). A likely or possible beneficial/substantial effect implies that the confidence interval overlaps the trivial limit, which means the effect could still be trivial. In MBIs as long as the intervention is not harmful, it is considered beneficial, but again this does not mean it is certainly beneficial. Furthermore, we invite the authors to reflect about the underlying reasons for selecting a specific smallest worthwhile change (minimal clinically or otherwise important change/difference) and to highlight the rationale for their choice.

Finally, we would like to point out that any statistical approach, even the most elaborate one, will never compensate for a bad or biased research design. Therefore, the identification of the statistical approach is ‘secondary’ to the design quality and methodological appropriateness of the study. Furthermore, we suggest the use of an adequate and large enough sample to improve the precision of the estimates. Although we are aware of the difficulties of conducting and recruiting participants for intervention studies, the frequent use of the magic sample number of 10–12 per group is highly questionable at least in terms of adequate precision. By encouraging, supporting and advocating strong study rationales, well-reflected research designs, with appropriate (and importantly open and honest) statistical approaches which are interpreted and reported in an adequate manner, we can significantly improve the field of football science and the practical recommendations that reach the field.