International comparisons and sensitivity to instruction

Abstract

While international comparisons such as those provided by PISA may be meaningful in terms of overall judgements about the performance of educational systems, caution is needed in terms of more fine‐grained judgements. In particular it is argued that the results of PISA to draw conclusions about the quality of instruction in different systems is affected by the fact that assessments used in PISA are, like all standardised assessments, relatively insensitive to the effects of high‐quality instruction, for three reasons. First, the rate of progress of individuals is small when compared with the range of achievement within an age cohort. Second, the standard methods of test construction tend to remove items that show high sensitivity to instruction. Third, the use within PISA of differential item functioning to identify and exclude items that are not comparable across languages results in further reduction of the sensitivity of the assessments to instruction.