Loanwords and Vocabulary Size Test Scores: A Case of Different Estimates for Different L1 Learners

ABSTRACT

The article investigated how the inclusion of loanwords in vocabulary size tests affected the test scores of two L1 groups of EFL learners: Hebrew and Japanese. New BNC- and COCA-based vocabulary size tests were constructed in three modalities: word form recall, word form recognition, and word meaning recall. Depending on the test modality, the tests measured the knowledge of 8,000 lemmas or word families through 80 randomly sampled items, 6 of which were loanwords in Hebrew and 13 in Japanese. Therefore, we added the same number of non-loanwords from corresponding frequencies and performed within-subject comparisons between the scores of the original tests with loanwords and their non-loanword versions in which non-loanwords replaced loanwords. The comparisons were done for each L1 group, at each test modality, and at three L2 proficiency levels, as defined by the total non-loanword test score. We also compared the two L1 groups on the degree of loanword effect. In both L1 groups, tests with loanwords yielded significantly higher scores in all test modalities and among most proficiency groups. Less able participants gained more from the presence of loanwords. However, loanwords differently influenced the size estimates of the two L1 groups. Implications are suggested for creating vocabulary size tests and making inferences from vocabulary test data.

Notes

¹ The original CATSS includes a fourth modality, “passive recognition,” in which the target word is provided and the examinee selects a meaning explanation from four options. We did not construct this modality because Laufer et al. (2004) showed that in monolingual tests, there is no significant difference between active and passive recognition scores. The terminology of test modalities is adapted from this study as well.

² Because BNC is arranged by families and COCA by lemmas, selecting words of the same frequency meant that one of the derivatives of COCA was of the same frequency as in the BNC. For example, “private” is in the first 1,000 most frequent items in both corpora. However, “privatize” is a separate entry in COCA, but not in the BNC, because it is a member of the same family as “private.” The assumption of BNC-based tests is that if learners know “private,” they will also know “privatize.” Whether learners know derivatives is controversial. We believe that even if derivational knowledge can be assumed in comprehension tests, it cannot be taken for granted in production tests. In this article, the focus is on the difference between test versions with and without loanwords, rather than on the word knowledge (with or without derivatives). Hence, the issue of lemmas and word families is not critical to our findings.

³ As mentioned earlier, in this study, loanwords are considered to be words that can be recognized without prior learning by learners as familiar based on their form and meaning similarity to words in L1. For Hebrew, the status of loanwords in the test was decided on by one of the researchers, based on extensive teaching experience. For Japanese, a group of learners similar to those in the study was interviewed on a list of loanwords and whenever they recognized a word as similar to a Japanese meaning equivalent, the loan word was included in the test. The resulting lists of loanwords and the test items were shown to Japanese SLA lecturers, who agreed with the students’ opinions.

⁴ Plonsky and Oswald (2014) recommend the following scales for roughly gauging small, medium, and large effects relative to others in the SLA field, based on the 25th, 50th, and 75th percentiles for observed effects in their L2 research sample: for mean differences (d) resulting from between-groups contrasts, .4, .7, 1.0; for mean differences (d) resulting from within-groups/prepost contrasts, .6, 1.0, 1.4.

⁵ The cutoff points of the divisions were decided on the basis of two criteria. One was the vocabulary sizes necessary to function in the different modalities at the various levels. Thus, 2,000 word families represented by a score of 20 are necessary to conduct a simple conversation (Nation, 2006), but 3,000 word families reflected in a score of 30 are necessary for basic listening and reading. The second criterion for the division was the distribution of scores. It would not make much sense to divide the 40–80 range into two groups, 40–60 and 61–80, because, as can be seen from Table 4, the number of learners who received more than 40 on the recall tests is rather low.