“Ensemble Subsampling for Imbalanced Multivariate Two-Sample Tests,”

This article refers to:

Ensemble Subsampling for Imbalanced Multivariate Two-Sample Tests

In our recent article (Chen, Dou, and Qiao 2013), we did a simulation that included the crossmatch statistic (Rosenbaum 2005) for comparison. Dr. Rosenbaum raised a concern about the Type I error rate we reported for the crossmatch test at the sample size ratio q = 64 in Table 3. Rosenbaum (2005, Sec. 3.1) proved that the crossmatch statistic has an exact distribution-free null distribution; therefore, the test procedure using the exact distribution always controls the Type I error rate. Though we used a permutation test based on the crossmatch statistic rather than its exact null distribution, the Type I error rate should still be controlled approximately at the significance level.

In response to this concern, we have checked our computation and found that the power and the Type I error rate at q = 64 were inflated. The over-rejection in simulation referred in our remark was due to an underestimation of the p-values in the permutation test by not counting the probability that the test statistic based on permutated data equals the test statistic based on the original data. If this probability were counted, the p-values would be corrected and the results would confirm that the crossmatch test controls the Type I error rate and has lower power at q = 64 compared to the other smaller q’s.

To eliminate the approximation error in the power and the Type 1 error rate induced by the permutation test, we further studied the exact null distribution of the crossmatch statistic at q = 64 and derived a rejection region at a significance level of 0.05. In the following, we report the power and the Type I error rate (in parentheses) computed based on 1000 simulations: 0.05 (0.03) and 0.12 (0.05) for Models 1.1 and 1.2; 0.05 (0.04) and 0.07 (0.03) for Models 2.1 and 2.2; 0.10 (0.04) and 0.04 (0.03) for Model 3.1 and 3.2. We thank Dr. Rosenbaum for raising the concern and for helpful discussions.