The Effects of a Translation Bias on the Scores for the Basic Economics Test

Abstract

International comparisons of economic understanding generally require a translation of a standardized test written in English into another language. Test results can differ based on how researchers translate the English written exam into one in their own language. To confirm this hypothesis, two differently translated versions of the Basic Economics Test (BET) (Walstad, Rebeck, and Butters 2010a) were given to elementary school students in Korea. We found the possibility of overestimating or underestimating the levels of economic understanding by various sources of translation bias. Therefore, it is important to carefully interpret the assessment results of international comparisons. Our study is applicable not only to international comparisons of economic literacy but also to any comparison of knowledge across cultures or languages within a country.

Keywords:

• BET
• economic understanding
• translation bias

JEL codes:

• A21
• A22
• I21

Acknowledgments

The authors thank Kristin Chase and seminar participants at the 2011 National Conference on Teaching Economics at Palo Alto, California, for their helpful comments. This article has benefitted from valuable comments by two anonymous referees. We are responsible for any remaining errors and deficiencies in this article.

This article is based on an paper that was presented at the National Conference on Teaching Economics held at Stanford University on June 1–3, 2011.

Notes

1. The PISA is an international assessment of scholastic performance of 15-year-old school pupils in OECD member countries. It was first administered in 2000 and is repeated every three years.

2. The TIMSS is an international assessment of the mathematics and science knowledge of fourth- and eighth-grade students around the world. It was first performed in 1995 and is repeated every five years.

3. We thank Kristin Chase for pointing out a broad range of studies to which our results are applicable.

4. We considered a total of 52 questions because there are 8 common questions in forms A and B. However, we decided not to include all these questions for our study because of time constraints for elementary school students. We selected 34 questions that might generate translation bias and chose 3 common questions to test the homogeneity of the two test groups.

5. We first translated the test by ourselves and sent the benchmark versions of the test to professional linguists for review. After reflecting on several comments, we finalized the translated versions of the test.

6. Even though it is a word-for-word translation, we modified the grammatical structure of the English version according to Korean grammar.

7. A translated copy of the BET is available from the authors upon request.

8. We selected schools by considering both administrative area and region in which they are located (city versus rural). We tried to select them randomly, although this is not purely random in a statistical sense. There might be a potential self-selection bias if higher-income schools are more likely to administer a test.

9. For this study, 243 students and 252 students were asked to take version 1 and version 2, respectively. However, 8 students in version 1 and 1 student in version 2 scored less than 20 out of 100. We felt that these students did not show adequate effort on the test and dropped them from our sample in order to preserve the accuracy of our measurements. However, this change does not qualitatively affect our main results.

10. It seems that the question in version 1 is crystal clear in English, as an anonymous referee pointed out. “The amount sellers produce” is often translated into one word meaning “the amount to sell” in Korean. Likewise, “the amount buyers want to buy” is translated into one word meaning “the amount to purchase” in Korean.

11. The percentage of correct answers for the BET was 36 percent in the United States.

12. We translated “profit” as “gain” in version 1 and “business profit” in version 2 for question 18, in which “profit” is not the correct answer. As a result, both the distributions of the answers and the percentages of the correct answer did not show a significant difference.