Testing across languages in international comparisons: cultural adaptation of consensus-based test translation review procedures

ABSTRACT

We investigated whether consensus-based test translation review procedures can be used effectively in cultural contexts with high social stratification. We staged two test translation review panels in China – where social stratification may potentially inhibit individuals’ ability to express opinion and disagreement. We adapted a consensus-based test translation review procedure in ways intended to ensure that all panellists could express their thinking. With the adapted procedure, we facilitated two panels charged with identifying errors in Chinese translations of items from PISA (OECD’s international test on skills for life). While social interactions during the panels reflected social status differences, the adaptations made it possible to capture the thinking of all panellists. The panels identified similar numbers of translation errors, but different sets of translation errors. We conclude that cultural adaptation addressing social stratification is critical to successfully implementing consensus-based procedures for test translation review in contexts with high social stratification. Ultimately, it contributes to ensuring fair, valid testing for culturally and linguistically diverse populations.

KEYWORDS:

• Test translation
• test translation review
• fidelity of implementation
• international comparisons
• cultural contexts

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 While other changes were made (compare Steps 1–4 and in Procedures A and B/C), these changes addressed practical circumstances (e.g. materials, resources, timeline) under which the test translation review sessions took place. Those changes were not considered as relevant to the cultural aspects that were the focus of this investigation.

2 Ibid.

Additional information

Notes on contributors

Xueyu Zhao

Dr Xueyu Zhao is a faculty in the School of Foreign Languages at Southeast University, China. Her research focuses on international test comparisons, test adaptations and translations, and cultural and linguistic issues in international large-scale assessments. Her research addresses the testing of linguistically diverse populations beyond the United States and Latin America. She conducts research to provide translation reviewers the analytical tools they can use in the process of examining translation errors, and to create translation review procedures that are sensitive to local cultures.

Guillermo Solano-Flores

Dr Guillermo Solano-Flores is Professor of Education at the Stanford University Graduate School of Education. He specializes in educational assessment and the linguistic and cultural issues that are relevant to both international test comparisons and the testing of cultural and linguistic minorities. His research is based on the use of multidisciplinary approaches that use psychometrics, sociolinguistics, semiotics, and cognitive science in combination. He has conducted research on the development, translation, localization, and review of science and mathematics tests. He has been principal investigator in several National Science Foundation-funded projects that have examined the intersection of psychometrics, semiotics, and linguistics in testing.