Text Integration and Speaking Proficiency: Linguistic, Individual Differences, and Strategy Use Considerations

ABSTRACT

The current study examined the effects of text-based relational (i.e., cohesion), propositional-specific (i.e., lexical), and syntactic features in a source text on subsequent integration of the source text in spoken responses. It further investigated the effects of word integration on human ratings of speaking performance while taking into consideration individual characteristics in test-takers (e.g., listening proficiency, age, grade point average, working memory capacity) and test-taker strategy use (e.g., note-taking strategies). A total of 263 test-takers’ speaking samples were collected using TOEFL-iBT research forms of integrated listen/speak items. This data and, individual characteristics measures and note-taking data were collected over two days. These spoken samples were transcribed and analyzed in terms of textual integration at lexical, cohesion, and syntactic levels. The linguistic features along with the individual characteristics and note-taking data were used to predict human scores of speaking proficiency. The results indicate that the linguistic properties of the source text are almost perfect predicators of which words test-takers will integrate into their response. Moreover, it was found that text integration is an important factor that affects human ratings of speaking proficiency that goes beyond individual test takers’ characteristics and note-taking strategies.

Acknowledgments

This research was funded by the Educational Testing Service (ETS) under a Committee of Examiners and the Test of English as a Foreign Language research grant. ETS does not discount or endorse the methodology, results, implications, or opinions presented by the researchers. TOEFL® test material is reprinted by permission of Educational Testing Service, the copyright owner. We are also indebted to our support team who helped collect, transcribe, and code the data. Our team included Melinda Childs, Kristopher Kyle, Stephen Skalicky, Dani Francuz Rose, and Paul Joseph Hobbs.

Notes

¹ We used R²_GLMM to present the variance explained in our model. Historically, using R² in mixed-effects models has been problematic because R² algorithms may report decreased or increased R² in larger models. R²_GLMM calculates marginal and conditional R² that are less susceptible to these problems. Marginal effects are concerned with the variance explained by fixed factors while conditional effects concern the variance explained by both fixed and random factors (Nakagawa & Schielzeth, 2012).

Additional information

Funding

This work was supported by the Education Testing Services.