Investigating Score Dependability in English/Chinese Interpreter Certification Performance Testing: A Generalizability Theory Approach

ABSTRACT

As a property of test scores, reliability/dependability constitutes an important psychometric consideration, and it underpins the validity of measurement results. A review of interpreter certification performance tests (ICPTs) reveals that (a) although reliability/dependability checking has been recognized as an important concern, its theoretical importance overshadows the operational efforts to measure score reliability/dependability; and (b) although multiple sources of measurement error can contribute variability to total score variance, rater effects have been regarded as the only source of error and modeled through classical test theory in the form of interrater reliability coefficients. Therefore, the study was initiated to investigate score dependability for a rater-mediated assessment of English-to-Chinese simultaneous interpreting, using generalizability theory. The results show the following: (a) the information completeness (InfoCom) ratings were more dependable than the ratings of fluency of delivery (FluDel) and target-language quality (TLQual); (b) the addition of tasks was more effective in raising dependability for InfoCom than the use of extra raters, but the effect was reversed for FluDel and TLQual; and (c) three different weighting schemes produced small variations in the composite-score dependability, but the InfoCom rating scale accounted for the largest proportions of the composite universe-score variances. These results are discussed for English/Chinese ICPTs.

Notes

¹ There are usually four different forms of interpreting: simultaneous interpreting (SI), consecutive interpreting (CI), sight translation (SiT), and dialogue interpreting (DI). SI is performed when an interpreter listens to a source-language speech while interpreting simultaneously in the target language. During CI, an interpreter usually listens to a speaker’s speech for a few minutes while taking notes and then interprets what the speaker has said when s/he stops. SiT involves an interpreter’s reading of a text from a source language into a target language simultaneously. During DI, an interpreter usually interprets dialogue-like interactions rather than speeches.