The problem of semantic openness and constructed response

Abstract

During the last three decades the constructed response format has gradually gained entry in large‐scale assessments of reading‐comprehension. In their 1991 Reading Literacy Study The International Association for the Evaluation of Educational Achievement (IEA) included constructed response items on an exploratory basis. Ten years later, in Progress in International Reading Literacy Study (PIRLS) 2001, the constructed response format is ascribed special significance as bearer of central insights to the definition of reading literacy. This article focuses on the significance of the scoring guides and the relation between these guides on the one hand, and the text and the items on the other hand. A discussion of this relation as it is to be found in PIRLS 2001 is performed, showing both examples of success and more problematic aspects in the operationalisation of the intentions expressed in the theoretical framework for the test. Handling the problem of semantic openness is essential in representing depth of understanding and represents a field of possibilities for further research and development.

Notes

1. Complete scoring guides for the texts and items that are discussed in the article can be found among the released assessment questions at http://nces.ed.gov/Surveys/PIRLS/released.asp or in Mullis et al. (2004). PIRLS 2006 Assessment Frameworks and Specifications. Amsterdam: International Association for the Evaluation of Educational Achievement.

2. It might be a coincidence, but the first time the connection between CR and depth is made, it is without reference to textual support (Campbell et al. 2001, 37). This contributes to the general impression of the importance of depth in the PIRLS framework.

3. Clay 11, Flowers, Pufflings 7 and 8. These items all yield two points.

4. Norwegian: ‘Ingen brukte ham, ingen lagde noe av ham. En gutt bare lot ham t⊘rke, En jente var snill og lagde ham, En jente brukte ham. Han var lykkelig.’

5. Norwegian: ‘F⊘rst ville han bli til noe. Så ble han til noe.’

6. Norwegian: ‘F⊘rst var han trist og lei seg. Så til slutt var han stolt og glad. Han ble forandret fordi f⊘rst ble han ikke brukt, så kom han fram og på en måte på utstilling.’

7. This frequency distribution is representative of items relating to other fictional texts with the same question structure. On items of this type, between 40% and 53% of pupils in the Norwegian material score no points.

8. Norwegian: ‘F⊘rst var leirklumpen lei og tr⊘tt så ble den glad og så ble den fortvilt så ble den lettet så ble den spent så ble den glad og så ble den lykkelig og forn⊘yd med livet sitt.’