Inappropriate Analysis of a Cluster Randomized Controlled Trial Due to Not Accounting for Nesting and Clustering: Comment on “A Home-Based Dyadic Music-with-Movement Intervention for People with Dementia and Caregivers: A Hybrid Type 2 Cluster-Randomized Effectiveness-Implementation Design” [Letter]

View responses to this article:

Inappropriate Analysis of a Cluster Randomized Controlled Trial Due to Not Accounting for Nesting and Clustering [Response to Letter]

Dear editor

Cheung et al accepted an important challenge by conducting a cluster randomized controlled trial (cRCT) primarily designed to assess the effect of a music-with-movement (MM) intervention on psychosocial wellbeing of patients with dementia and their caregivers.¹ Due to identified barriers in applying MM, the secondary aim of their experiment was to determine implementation strategy effectiveness of the home-based, dyadic MM intervention.¹

To appropriately assess outcomes using a cluster randomized design requires that intracluster dependency, typically as quantified by the intracluster correlation coefficient (ICC), be accounted for in the statistical analysis.² Accounting for non-independence allows for comparison between individuals in the clusters assigned to intervention or control conditions because it accounts for potential non-independence of model residuals within clusters.² Given this, there are concerns within the publication by Cheung et al.

Most importantly, the authors did not explain how clustering was accommodated within the statistical analysis used, suggesting that it was not. Additionally, accounting for nesting in cRCTs is critical, especially with a small number of clusters (K), because if K is small, the available degrees of freedom for estimating effect sizes must also be small.² Ignoring either non-independence or limited degrees of freedom often inflates type I error rate.²

We also highlight a concern with the article’s reported sample size calculations. The cRCT CONSORT guidelines require that authors report the methods used for sample size calculation, number of clusters (K), size of clusters (m), whether equal or unequal cluster sizes were assumed, assumed ICC, and an indication of the ICC uncertainty. The authors did include the assumed ICC (0.01) for sample size calculation of their trial registration’s statistical analysis plan (NCT03575026). However, they reported a sample size calculation only with respect to the number of individual participants, not accounting for the number of clusters required to reach statistical power. The number of clusters has greater impact on statistical power in cRCTs than does the number of individual participants.²

We note that the authors used the generalized estimating equations (GEE) approach in their article.¹ While GEE can be used to account for clustering, as we previously describe Cheung et al appear not to have done so. Further, they randomized a small number of clusters (7 in total)¹ for which GEE can produce inflated type I error rates.³ Thus, switching to an appropriate statistical model for reanalysis using a valid approach, such as linear mixed models accounting for the small number of clusters, is warranted. The authors could also report the results of a reanalysis using GEE with a small-sample correction.⁴

If we are correct in our understanding that clustering and nesting were not accounted for in the analysis, per the Committee on Publication Ethics (COPE) guidelines for handling post-publication critiques,⁵ a reanalysis of the data using valid statistical procedures for the study design in a published correction, or retraction and republication, seems essential.

Disclosure

Dr. Allison and his institution (Indiana University) have received payments for consultation, grants, contracts, in-kind donations, and contributions from multiple for-profit and not-for-profit entities (Alkermes, Inc., Amin, Talati, Wasserman for KSF Acquisition Corp (Glanbia), Big Sky Health/Zero, Clark Hill, PLC, Eli Lilly and Company, Kaleido Biosciences, Mars, Incorporated, Medpace/Gelesis, Novo Nordisk Foundation, Ronald Marron, Law Offices, Soleno Therapeutics, Sports Research Corp., and WW [formerly Weight Watchers]) interested in statistical design and analysis of experiments, but not directly related to the research questions addressed in this letter. The other authors report no conflicts of interest in this communication.

Additional information

Funding

Supported in part by NIH grants U24AG056053 and P30AG050886. The assertions expressed are those of the authors and not necessarily those of the NIH or any other organization.