Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37

Abstract

The process of setting a standard when pass/fail decisions have to be made inevitably involves judgment about the point on the test score scale where performance is deemed to be adequate for the purpose for which the examination is set. As with any process which involves human judgment, setting this standard is likely to include a certain degree of error, which may result in some false positive and false negative decisions. The customary practice of maintaining a constant point on the test score scale at which pass/fail separations are made cannot be justified, as examinations vary in difficulty. The aim of standard setting procedures is to minimize such errors while accounting for the varying difficulty of examinations.

A standard may be norm-referenced, where it is dependent on the performance of the particular group of examinees, or criterion-referenced, where it is based on predetermined criteria, irrespective of examinee performance. Where certification of competence is the primary purpose of an examination, the latter is preferred as the decision to be made is whether an individual is competent to practise rather than competent compared to peers. Several methods of standard setting have been used, some of which are based solely on predetermined criteria, while others compromise between norm- and criterion-referenced standards.

This guide examines the more commonly used methods of standard setting, illustrates the procedure used in each with the help of an example, and discusses the advantages and disadvantages associated with the use of each. The common errors made by judges in the standard setting process are pointed out and the manner in which judges should be selected, trained and instructed emphasized. A method used for equating similar tests set at different times with the intention of maintaining standards from one examination to the next is illustrated with an example. Finally, the guide proposes a practical method for arriving at a pre-determined standard by the proportionate selection of test-items of known relative difficulties in relation to minimally competent examinees.

Additional information

Notes on contributors

Raja C. Bandaranayake

RAJA C. BANDARANAYAKE, MBBS, PhD, MSEd, FRACS is a retired professor of Anatomy and an international consultant in medical education. His most recent appointments were as Director and Associate Professor in the School of Medical Education, University of New South Wales, Sydney, Australia, and as Professor & Chairman, Department of Anatomy, College of Medicine & Medical Sciences, Arabian Gulf University, Bahrain. He was Deputy Chairman of the Board of Examiners, Chairman of the Examinations Committee, and Chairman of the Anatomy Sub-committee, of the Royal Australasian College of Surgeons, as well as Technical Adviser and Member of the Board of Examiners of the Australian Medical Council. As a member of a working party of the World Federation for Medical Education, and of a core committee of the Institute for International Medical Education in New York, he has contributed to the development and testing of international standards in medical education. As a consultant of international repute he has facilitated the development of curricula, including student assessment and programme evaluation procedures, in many health professional schools around the world.