Exploring the Use of Video-Conferencing Technology in the Assessment of Spoken Language: A Mixed-Methods Study

ABSTRACT

This research explores how Internet-based video-conferencing technology can be used to deliver and conduct a speaking test, and what similarities and differences can be discerned between the standard and computer-mediated face-to-face modes. The context of the study is a high-stakes speaking test, and the motivation for the research is the need for test providers to keep under constant review the extent to which their tests are accessible and fair to a wide constituency of test takers. The study examines test-takers’ scores and linguistic output, and examiners’ test administration and rating behaviors across the two modes. A convergent parallel mixed-methods research design was used, analyzing test-takers’ scores and language functions elicited, examiners’ written comments, feedback questionnaires and verbal reports, as well as observation notes taken by researchers. While the two delivery modes generated similar test score outcomes, some differences were observed in test-takers’ functional output and the behavior of examiners who served as both raters and interlocutors.

Acknowledgments

We acknowledge the participation of Lynda Taylor in the design of both Examiner and Test-taker Questionnaires, and Jamie Dunlea for the FACETS analysis of the score data. Our special thanks go to Jermaine Prince for his technical support, careful observations and professional feedback. This research was funded by the IELTS Partners: British Council, Cambridge English Language Assessment and IDP (IELTS) Australia.

Notes

¹ This article presents selected parts of the larger research reported in Nakatsuhara et al. (2016). In addition to the data sources listed in Figure 1, the full report includes test-takers’ responses to feedback questionnaires and semistructured interviews, and examiners’ questionnaire responses to 27 closed-ended questions regarding their perceptions of the two test delivery modes. However, a discussion of these components is beyond the scope of this article.

² The score distribution was slightly positively skewed and slightly more peaked than a Gaussian distribution (Shapiro-Wilk test: W=0.919, df = 32, p = .02). This influences the choice of statistical tests in SPSS, but FACETS makes no assumptions about data distributions (M. Linacre, personal communication, November 16, 2016).

³ The comparability of the two versions was supported by Mann Whitney’s U tests, which showed no significant differences in any analytic or overall scores, and by the analysis of observers’ notes that did not identify any differences in examiner behavior. The use of a few language functions was found to be significantly different between the versions (e.g., Part 3 Asking for clarification in the face-to-face test (Z = −2.118, p = .034); Part 1 agreeing in the video-conferencing test (Z = −2.932, p = .003)). However, the counterbalancing of the two versions between the two delivery modes is considered to have minimized such version effects.

⁴ More information on the IELTS Speaking test tasks is available at http://takeielts.britishcouncil.org/prepare-test/understand-test-format/speaking-test.

⁵ The public version of the IELTS Speaking band descriptors is available at https://www.ielts.org/about-the-test/how-ielts-is-scored

⁶ Although the data distributions indicated slight non-normality, parametric tests were thought to be more appropriate to avoid potential Type 2 errors, given the purpose of this research (N. Verhelst, personal communication, May 6, 2016).

⁷ Dimensionality in the two delivery modes was also examined by correlating scores given to the two modes of the test. Correlation coefficients obtained by Spearman’s rho tests (N = 32) were 0.829 for fluency, 0.827 for lexis, 0.739 for grammar, 0.848 for pronunciation, and 0.914 for the overall scores, which were all significant at a 0.001 level.

⁸ The 14 functions are providing personal information, expressing a preference, providing non-personal information, comparing, expressing opinions, summarizing, explaining, conversation repair, suggesting, contrasting, justifying opinions, narrating and paraphrasing, speculating, and analyzing.

⁹ It should be noted that, in this exploratory study, transmission demands were low because examiner rooms and test-taker rooms were located in the same building. Further technical trials with a bespoke platform for the video-conferencing mode are in progress with geographically distant examiners and test takers.