Getting it ‘better’: the importance of improving background questionnaires in international large‐scale assessment

Abstract

In addition to collecting achievement data, international large‐scale assessment programmes gather auxiliary information from students and schools regarding the context of teaching and learning. In an effort to clarify some of the opacity surrounding international large‐scale assessment programmes and the potential problems associated with less than optimal background questionnaires, this paper outlines how auxiliary student background data influence the black box of achievement score construction. This discussion is supplemented with a number of empirical examples that point to possible threats to accurate achievement estimation, including missing data treatment, poor scale reliability, and questionnaire respondents’ misunderstanding or inaccurate answers to the questions.

Keywords:

• large‐scale assessment
• international organizations
• PISA
• TIMSS
• background questionnaires
• PIRLS

Acknowledgements

We would like to thank the JCS editors and reviewers for their feedback during the revision of this paper. We would also like to thank our colleagues at the IEA Data Processing and Research Centre for their support while writing this manuscript. Our views and findings are our own and we accept all responsibility for the final manuscript.

Notes

1. This is not often discussed outside of highly specialized journals and technical reports.

2. See Mullis et al. (2005) for complete details on the TIMSS 2007 assessment design.

3. While the technical details associated with plausible value methods are omitted here, the approach uses a modification of a well‐known missing data method (Rubin 1987) to treat student achievement as if it were a missing value to be ‘filled in’.

4. The reason that more than one value is assigned to each student is because there is a meaningful and non‐ignorable level of uncertainty associated with a single achievement value because so few responses are gathered from any individual student. Sophisticated methods are used to impute achievement.

5. We reason that this might be in part because most of the background questionnaire committee comes from economically developed countries The TIMSS 2007 questionnaire item review committee was comprised of seven members, six of whom were from economically developed countries. The PIRLS questionnaire development group was comprised of eight members, six of whom were from economically developed countries. Of the two remaining committee members, one was from China, a country that did not take part in PIRLS.

6. Researchers and LSA developers have at their disposal a number of tools for improving these measures—including focus groups, regional options based on sound theory and empirical evidence, and the purposeful inclusion of a variety of committee members in item‐development groups and decision‐making bodies.