The potential of anchoring vignettes to increase intercultural comparability of non-cognitive factors

ABSTRACT

We investigate whether Anchoring Vignettes (AV) improve intercultural comparability of non-cognitive student-directed factors (e.g., procrastination). So far, correlation analyses for anchored and non-anchored scores with a criterion have been used to demonstrate the effectiveness of AV in improving data quality. However, correlation analyses are often used to investigate external validity of a scale. Nonetheless, before testing for validity, the reliability of the measurement of a construct should be examined. In the present study, we tested for measurement invariance across countries and languages and compared anchored and non-anchored student-directed self-reports that are highly relevant for the students’ self and their behaviour and performance. In addition, we apply further criteria for testing reliability. The results indicate that the data quality for some of the constructs can – in fact – be improved slightly by anchoring; whereas, for other self-reports, anchoring is less successful than was hoped. We discuss with regard to possible consequences for research methodology.

KEYWORDS:

• Anchoring vignettes
• measurement invariance
• culture-specific responses
• multi-group confirmatory factor analysis

Acknowledgements

During the preparation of this manuscript Susanne Kuger was partially funded by CIDER (College for Interdisciplinary Educational Research) and the ZIB (Center for International Student Assessment)

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Country-specific names were inserted instead of <NAME> for each vignette.

2. Not tabled; the EFA results can be provided upon request (for Procrastination, Industriousness, Planning and Organisation, all anchored using the vignette on Organisation).

Additional information

Funding

This work was supported by the CIDER (College for Interdisciplinary Educational Research); ZIB (Center for International Student Assessment).

Notes on contributors

Tamara Marksteiner

Tamara Marksteiner studied Sociology (university diploma in 2009) at the University of Mannheim. There she received her doctorate in Psychology in 2013 and taught from 2009 to 2013 as a lecturer at the Psychological Institute.

Susanne Kuger

Dr. Susanne Kuger is head of the Department of Social Monitoring and Methodology at the German Youth Institute in Munich (Germany). Her research interests and teaching topics are research on the effects of family, early childhood education and care, school and out-of-school environments on child and youth development, survey methodology, as well as refining modelling techniques for complex quantitative data analyses in education research.

Eckhard Klieme

Eckhard Klieme studied Mathematics (university diploma in 1978), Psychology (university diploma in 1981), Communication Research and Pedagogy at the University of Bonn. There he received his doctorate in Psychology in 1988 and taught from 1986 to 1989 as a lecturer at the Psychological Institute of the University of Bonn. From 1983 to 1997 he was a research assistant at the Institute for Test and Gifted Research of the Studienstiftung des Deutschen Volkes in Bonn. From 1998 to 2001 he was a research assistant at the Max Planck Institute for Human Development in Berlin. In 2000, he completed his habilitation in Educational Science at the Free University of Berlin and in 2001 was appointed professor at the University of Frankfurt am Main. Since 2001, Eckhard Klieme has also headed the department "Educational Quality and Evaluation" at the German Institute for International Educational Research and was its director from 2004 to 2008.