A note on the limitations of the CAT procedure with application to mixed-effects models

Abstract

In the recent past, the CAT procedure introduced by Pal, Lim, and Ling has been successfully applied to obtain improved testing procedures in numerous applications. Having seen such results, practitioners may resort to the CAT procedure in all testing problems, assuming that improvement is assured in all applications. To avoid such myths, in this article, we present an important class of applications, where the CAT test performs poorly, and then discuss the type of applications where CAT procedure could be accepted or avoided. However, this does not mean that it is not possible to develop improved tests by taking the CAT approach, as we show in this article by employing the LRT statistic instead of ML/REML-based CAT tests, as authors of the original CAT article also now advocate. In fact, in terms of the Type I error, LRT-based CAT test performed well among tests we studied, except when k, the number of groups in one-way layout is small, in which case the generalized p-value-based tests can be employed. We believe this note will encourage further research to take full benefits of the CAT approach, in such problems as higher way ANOVA and mixed-effects regression models, for which generalized tests are currently available.

Keywords:

• Generalized inference
• Generalized p-values
• LRT
• ML
• REML
• Variance components

Acknowledgment

We are grateful to the two reviewers, whose constructive comments and suggestions led to a substantial improvement of this article.