Convergence and discriminant: assessing multiple traits using multiple methods

Abstract

Multiple traits of language proficiency as well as test method effects were concurrently analyzed to investigate interrelations of construct validity, convergent validity, and discriminant validity using multitrait-multimethod (MTMM) matrices. A total of 585 test takers' scores were derived from the field test of the Pearson Test of English Academic. An MTMM confirmatory factor analysis model was parameterized using 4 traits and 3 assessment methods. The 4 traits included listening, reading, speaking, and integrated skills, while the 3 methods included prescribed multiple-choice responses, constructed responses, and summarized responses. The trait factor loadings were systematically greater than those of methods, providing evidence that the indicators were strongly related to their latent constructs, after adjusting for the method effects. The results showed robust convergent validity, moderate discriminant validity, and insignificant method effects. Implications are discussed.

Keywords:

• multitrait-multimethod matrix
• adult English language learners
• confirmatory factor analysis
• convergent validity
• discriminant validity
• method effects

Notes

1. Although the author speaks of the validity and reliability of a test for convenience's sake, it actually refers to the validity and reliability of test scores.

2. Since it is unknown as to whether the participants learned English as a second language (L2) or foreign language (FL), L2 and FL are used interchangeably in this paper for the sake of convenience and consistency in the literature.