A Revisit to the Behrens–Fisher Problem: Comparison of Five Test Methods

Abstract

We revisit the well-known Behrens–Fisher problem and apply a newly developed ‘Computational Approach Test’ (CAT) to test the equality of two population means where the populations are assumed to be normal with unknown and possibly unequal variances. An advantage of the CAT is that it does not require the explicit knowledge of the sampling distribution of the test statistic. The CAT is then compared with three widely accepted tests—Welch–Satterthwaite test (WST), Cochran–Cox test (CCT), ‘Generalized p-value’ test (GPT)—and a recently suggested test based on the jackknife procedure, called Singh–Saxena–Srivastava test (SSST). Further, model robustness of these five tests are studied when the data actually came from t-distributions, but wrongly perceived as normal ones. Our detailed study based on a comprehensive simulation indicate some interesting results including the facts that the GPT is quite conservative, and the SSST is not as good as it has been claimed in the literature. To the best of our knowledge, the trends observed in our study have not been reported earlier in the existing literature.

Keywords:

• Generalized p-value
• Hypothesis testing
• Parametric bootstrap
• Power function
• Size

Mathematics Subject Classification:

• Primary 62F03
• Secondary 62E17