General overview of the theories used in assessment: AMEE Guide No. 57

Abstract

There are no scientific theories that are uniquely related to assessment in medical education. There are many theories in adjacent fields, however, that can be informative for assessment in medical education, and in the recent decades they have proven their value. In this AMEE Guide we discuss theories on expertise development and psychometric theories, and the relatively young and emerging framework of assessment for learning. Expertise theories highlight the multistage processes involved. The transition from novice to expert is characterised by an increase in the aggregation of concepts from isolated facts, through semantic networks to illness scripts and instance scripts. The latter two stages enable the expert to recognise the problem quickly and form a quick and accurate representation of the problem in his/her working memory. Striking differences between experts and novices is not per se the possession of more explicit knowledge but the superior organisation of knowledge in his/her brain and pairing it with multiple real experiences, enabling not only better problem solving but also more efficient problem solving. Psychometric theories focus on the validity of the assessment – does it measure what it purports to measure and reliability – are the outcomes of the assessment reproducible. Validity is currently seen as building a train of arguments of how best observations of behaviour (answering a multiple-choice question is also a behaviour) can be translated into scores and how these can be used at the end to make inferences about the construct of interest. Reliability theories can be categorised into classical test theory, generalisability theory and item response theory. All three approaches have specific advantages and disadvantages and different areas of application. Finally in the Guide, we discuss the phenomenon of assessment for learning as opposed to assessment of learning and its implications for current and future development and research.

Notes

1. From: George Santayana (1905) Reason in Common Sense, volume 1 of The Life of Reason.

2. Which he explained first in Logik der Forschung. Julius Springer Verlag, Vienna, 1935 and later in The Logic of Scientific Discovery. Hutchinson, London, 1959.

3. Of course, this is not the only assumption that is needed for the application of CTT, another important assumption is that of local independence of individual observations, i.e. that all data points are independent of each other except for the construct the test aims to measure. An extensive discussion of the theoretical assumptions for each of these theories falls outside the scope of this AMEE Guide. Also understanding the assumptions mentioned in this AMEE Guide suffices for almost all normal everyday test situations.

4. It may seem a bit enigmatic how these conclusions are drawn but one has to bear in mind that the SDs are 5%. In a normal distribution, roughly 68% of the observations is located between the mean minus 1 SD and the mean plus 1 SD. From this, it is logical to infer that in test 1 more observations will fall into the 95-CI area than in test 2. This is an example based on a somewhat normally shaped symmetrical distribution, needless to say that if the distribution is more extremely skewed towards more high scores, the influence of the reliability on the reproducibility of pass–fails decisions is even less.

5. In fact, this formula does not describe a generalisability coefficient but a dependability coefficient. We have used this formula because it is more intuitive and therefore more helpful in understanding G-theory. We will explain the difference between a generalisability and dependability coefficient later on in this section.