The Role of Lexical Properties and Cohesive Devices in Text Integration and Their Effect on Human Ratings of Speaking Proficiency

Abstract

There has been a growing interest in the use of integrated tasks in the field of second language testing to enhance the authenticity of language tests. However, the role of text integration in test takers’ performance has not been widely investigated. The purpose of the current study is to examine the effects of text-based relational (i.e., cohesion) and propositional-specific (i.e., lexical) features on subsequent source text integration in spoken responses. In addition, the current study investigates the effects of word integration on human ratings of speaking performance. Sixty test-takers’ speaking samples from the listen/speak section of the TOEFL-iBT were analyzed in terms of textual integration at the lexical and cohesion level and how this integration predicted human scores of speaking proficiency. The results indicate that the properties of the source text are strongly predicative of which words test takers will integrate into their response. Moreover, it was found that text integration is an important factor that affects human ratings of speaking proficiency. The findings of the study are discussed in terms of item difficulty and construct representation.

ACKNOWLEDGMENTS

TOEFL® test material are reprinted by permission of Educational Testing Service, the copyright owner.

Notes

¹ We did not use the raw scores because these scores correlated highly with the number of words integrated or not integrated. Because there were fewer words integrated, the raw scores were biased.

² Word hypernymy is related to word concreteness, because the lower a word is located on a hypernymic hierarchy the more likely the word will be concrete (Rosch, 1973; Tversky & Hemenway, 1984).