When Should One Stop Testing Software?

Abstract

We derive an optimal rule for stopping the testing of a module of software prior to release, based on the trade-off between the cost of continued testing and the expected losses due to any bugs that remain in the released code. Our economic assumptions are that testing costs are perfectly predictable, but of arbitrary form f(t), and that the costs of fixing a bug, both during testing and subsequent to release, are known constants. Our stochastic assumptions are as follows: (a) The total number of bugs in the module (N) is unknown; (b) N has a Poisson distribution with a mean λ that is distributed randomly (across modules) according to a known gamma distribution; (c) given a particular bug, the time it takes to find it during testing is random, with a known distribution G of arbitrary form; and (d) the lifetimes of different bugs are independent. With these assumptions, we find conditions on f and G that ensure that the optimal stopping rule has a simple form, which can be found by a dynamic programming argument. The central condition is that f(G ^-1(1 - e^-y )) is convex while finite. When N is Poisson with λ fixed, the optimal strategy is a fixed time stopping rule. When λ is variable and large with high probability, the optimum rule approaches a simple asymptotic form, namely, stop as soon as k, the observed number of bugs, satisfies k ≤ f′(t)G(t)/cG′(t), where c is the difference between the cost of fixing a bug during testing and the cost incurred by finding it after release. Further, the number of bugs remaining has a Poisson limiting distribution with mean f′(t)(1 - G(t))/cG′(t). We extend the theory to allow for several classes of bugs with different characteristics. We suggest that graphical display of the asymptotic stopping rule, for a selection of values of the cost parameters involved, may prove helpful to persons faced with the problem of managing software development. An example of such a display is given.

Key Words:

• Decision theory
• Sequential Bayes testing
• Software reliability