Logical Extremes, Beta, and the Power of the Test

Abstract

There is a potential misuse of the power function under the logical extreme when the null hypothesis is true. The power function is defined to measure the probability of rejecting the null given any value of the parameter being tested. It can be used to obtain the power and the β values only under the alternative hypothesis. When the null is true, the power function can be used to obtain the size of the test. The power and the probability of committing a Type II error are, however, undefined and, hence, the power function should not be used to obtain these values.

• Power function
• Simulations
• Type II error

1. Introduction

It is not an easy or smooth task to calculate the power of a test for beginning statistics students. Until computer algorithms and simulations were designed to help in this process (see, for example, Doane, Mathieson, and Tracy 2001), the best one could do was to calculate a single point on the power function representing 1 – β (as in Kvanli, Ravur, and Guynes 2000). Yu and Behrens (1994) illustrate the usefulness of simulations in determining the power of a test. It is possible to calculate every point on the power curve, but in-class demonstrations are usually limited to single points due to computational constraints and the repetitive nature of the process. Logical extremes, as detailed below, are usually ignored. The purpose of this note is to demonstrate that the logical extremes are nontrivial and that one of the extremes results in calculations of β and the power of the test that are inconsistent with the common pedagogy. A detailed history of the nature of hypothesis testing, including a section of the power of the test in light of the Fisher approach versus the Neyman-Pearson approach, can be found in Lehmann (1993). He concluded that combining the best features of both could unify the two approaches. This historical debate will not be addressed in this paper.

2. Calculation Process

We will use the familiar test on the population mean of a normal distribution with a known variance to set up the stage for logical extremes. We assume that the data generating process is where φ is the probability density function of a normal distribution with an unknown mean μ and a known variance σ².

We first consider testing the simple null hypothesis against a two-sided composite alternative assuming σ = 0.1 and n = 36. In our example, under the null hypothesis, the sampling distribution of the sample mean is normal, centered at with a standard error of . This is illustrated by the normal distribution on the left in Figure 1, which was generated using Visual Statistics (Doane, Mathieson, and Tracy 2001). The size of the z-test (the probability of committing a Type I error), α, is chosen to be 10%. Figure 1 also presents a second normal distribution on the right, that is the sampling distribution of the sample mean at the specific value of μ = 1.51 when the null hypothesis is false. The shaded area under this sampling distribution in the middle acceptance region measures the probability of committing a Type II error while the shaded areas in the rejection regions provide the power of the test, 1 – β.

Figure 1 Beta and the Power of the Test for Two-Tailed Test.

The power function, defined as the probability of rejecting the null given a specific value of μ, for the z test is given by , where Φ is the cumulative distribution function of the standard normal distribution. This yields 1 – β = 0.16 on the power function at this single point of μ = 1.51. The entire power function is shown in Figure 2 and resembles an upside down normal distribution. It should be noted that the power function in this example approaches one asymptotically as we move away from 1.5 in either direction.

Figure 2 The Power Function for a Two-Tailed Test.

For the one-tailed test, we consider the composite null hypothesis versus the composite alternative . The sampling distributions of the sample mean under the null of and the specific alternative of are illustrated, respectively, by the left and right distributions in Figure 3. The power function is given by . The power of the test at . is 0.25 and the whole power function is illustrated in Figure 4. The power function in this example approaches one and zero asymptotically as we move away from 1.5 in the positive and negative directions, respectively.

Figure 3 Beta and the Power of the Test for One-Tail Test.

Figure 4 The Power Function for a One-Tail Test.

3. The Logical Extremes

The first logical extreme occurs when the two sampling distributions in both Figure 1 and 3 pull apart to a point where there is very little overlap between them. In this case the power of the test approaches 1 (as the two distributions in Figure 1 and Figure 3 separate), and the β approaches zero. This can easily be seen in Figure 2 when μ moves away from in both directions and also in Figure 4 when μ moves away from to the right.

The second logical extreme occurs when μ approaches . One subtle point that has not usually been made explicitly in the definition of the power function is the fact that the power function yields the power of a test only when the null hypothesis is false. Under the null hypothesis, the power function provides the probability of committing a Type I error instead of the power of the test. As presented in Figure 2, the power function yields the 10% size at according to the accepted pedagogy (Freund and Walpole 1987).

This is where this logical extreme breaks down. In this case, it is incorrect to say that “the power of the test is 10% or the probability of committing a Type II error is 90%” since the power is defined as “the probability of rejecting a false null hypothesis” and a Type II error is defined to be “the failure to reject the false null hypothesis.” But there is no false null hypothesis when the null hypothesis is true at . β should be undefined rather than having a maximum value of 90%. Thus, the contention that the power of the test is at a minimum of 10% under the null cannot be supported because there is no false null hypothesis when the null hypothesis is true at . Both the power and the probability of committing a Type II error are undefined under this logical extreme. Unfortunately, this is usually overlooked. For example Figure 5 is generated for using Visual Statistics which labels the area under the “accept” region as “Type II” error and the area under the “Reject” regions as “Power.” This misconception is also apparent in the few current textbooks that attempt to address the power function. In Berenson, Levine and Krehbiel (2002, pp. CD9-6) they state “Interestingly, if the population mean is actually 368 grams, the power of the test is equal to α, the level of significance (which is 0.05 in this example), because the null hypothesis is actually true.” In Newbold, Carlson and Thorne (2003, p. 357), the authors make a similar statement that includes “The power of the test at μ₀ equals α, …”

Figure 5 Logical Extreme: Beta and the Power of the Test for Two-Tailed Test.

Similarly, the sampling distribution and the power function for the one-tailed test are presented in Figures 6 and 7.

Figure 6 Logical Extreme: Beta and the Power of the Test for One-Tail Test (Right).

Figure 7 Logical Extreme: The Power Function for a One-Tail Test (Right).

The power function in Figure 7 provides the probability of rejecting the null hypothesis over the whole range of values of μ. It can be used to obtain the power of the test only when . The size of the test is defined as the supremum of the power function over . This yields a size of 10% according to accepted pedagogy (Mood, Graybill and Boes 1974; Tamhane and Dunlop 2000). The power and β remain undefined under the null when .

4. Conclusions

There is a potential misuse of the power function under the logical extreme when the null hypothesis is true. The power function is defined to measure the probability of rejecting the null given any value of the parameter being tested. It can be used to obtain the power and the β values only under the alternative hypothesis. When the null is true, the power function can be used to obtain the size of the test. The power and the probability of committing a Type II error are, however, undefined and, hence, the power function should not be used to obtain these values.

Acknowledgments

The authors wish to thank two anonymous referees for their detailed and useful comments. All remaining errors are the responsibility of the authors.