Methodological issues in a meta-analysis

Original manuscript: Roshni Patel and Gaurang Shah. Effect of metformin on clinical, metabolic and endocrine outcomes in women with polycystic ovary syndrome: a meta-analysis of randomized controlled trials. Current Medical Research and Opinion 2017 doi: 10.1080/03007995.2017.1279597

Dear Editor,

Recently, Patel and Shah¹ published a meta-analysis about the effect of metformin on clinical, metabolic, and endocrine outcomes in women with polycystic ovary syndrome (PCOS). We noticed some methodological points in the paper that may benefit from comments:

First, publication bias and outlier detection must be considered². No indices were reported for publication bias.

Second, the authors pointed out, “if the heterogeneity test was not significant (p > .1) the fixed inverse variance method was used in the meta-analysis”. Selection type of a model for pooling means the difference must not be merely based on statistical significance (i.e. Q test and I² index), because these tests lack enough power to detect heterogeneity in the case of the small number of studies and small sample sizes in each study^3–6. Non-significant results for heterogeneity cannot be taken as evidence of homogeneity. For example, regarding the free androgen index (FAI) variable, there are only five studies, and, regardless of the low power of analysis for detecting heterogeneity, fixed inverse variance was used. Moreover, choosing the random or fixed model should be considered based on the sampling frame, and not on statistical tests². In addition, values of the I² index indicate the percentage of heterogeneity and can be accompanied by a 95% uncertainty interval, as proposed by Higgins et al.⁷ and Higgins and Thompson⁸.

Third, the purpose of a meta-analysis is not merely reporting a pooled estimation and, regardless of the investigating sources of heterogeneity, the results could be invalid². For example, in Figure 5f, the heterogeneity index for fasting insulin level is too high (I² = 100%), consequently the reported pooled estimation is not valid. This mistake occurred in most figures. This inconsistency within studies could be attributed to methodological and clinical issues^2,7. Furthermore, different duration and dose of treatment are potential sources of heterogeneity and substantially affect the pooled estimation^2,7. As it is clear in Table 1, a duration of more than 2–24 months was included, which should be stratified. It should be noted that, in the case of low power, reported pooled estimation is invalid. Additionally, the dose of metformin treatment may be a leading source of heterogeneity. Surprisingly, in this study all doses are combined. The dose of metformin in the studies ranged between 805–1700 mg/day, creating both methodological and clinical heterogeneity factors.

Finally, for accurate interpretation of the effect size in each study, statistical significance should be considered along with clinical significance. For example, in Table 3, it is indicated that metformin reduces the waist-to-hip ratio (WHR) (–0.02; 95% CI = –0.03–0.00). Do the authors maintain that this effect is of clinical relevance? Is it within the range of clinical measurement?