Application of Sequential Probability Ratio Test to Computerized Criterion-Referenced Testing

Abstract

The adaptive testing is an important testing method in the modern educational/psychological testings. In adaptive mastery testing, the items are selected adaptively according to the estimated information of the unknown latent ability levels, and given then to the test-takers, sequentially. Hence, the decision (master or non-master; pass or fail) for each test-taker is made sequentially based on each test-taker's responses to a particular sequence of items administered to him/her. Thus, statistically speaking, by the natural character of the adaptive mastery testing, it is a sequential problem with dependent observations. The Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] SPRT (sequential probability ratio test) has been applied to this kind of mental testing problem by many researchers in the field of educational/psychological measurement theory; for example, Reckase [Reckase, M. A procedure for decision making using tailored testing. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 238–257], Kingsbury and Weiss [Kingsbury, G.; Weiss, D.J. A comparison of IRT-based adaptive mastery testing and a sequential mastery testing procedure. In New Horizons in Testing – Latent Trait Test Theory and Computerized Adaptive Testing; Weiss, D.J., Ed.; Academic Press: New York, 1983; 258–288] and Spray [Spray, J. Multiple-Category Classification Using a Sequential Probability Ratio Test, ACT Research Report Series; 1993; 93–7]. Most of their results are empirical studies of the performance of SPRT with different item selection schemes. In statistical literature, the SPRT with i.i.d. observations has been intensively studied by many statisticians since Wald [Wald, A. Sequential Analysis; Wiley: New York, 1947]. In this paper, we concentrate on the properties of the stopping time of the SPRT under adaptive mastery testing; i.e., the test items (observations) are adaptively selected. Not only are there a few people discussing the properties of SPRT under non-i.i.d. setup, but also most of them study just large sample properties, which provide very little information for test-makers to design good mastery tests. Without the independence property of the observations, it will require different approaches to analyze its performance. Here we apply some results of linear growth processes and a martingale extension of the Wald's [Wald, A. Sequential Analysis; Wiley: New York, 1947] equation to obtain a bound for the expectation of the stopping rules (i.e., the test length) used in adaptive mastery tests.

Key Words:

• Adaptive testing
• Computerized mastery testing
• Martingale
• Sequential probability ratio test
• Wald's identity

AMS 1991 Classification:

• Primary 62F12
• Secondary 62J05

Acknowledgments

I sincerely thank two referees, an associate editor and the Editor for their constructive suggestions and comments that led to significant improvements in the paper.

Notes

Recommended by M. Baron.