Protocol Analysis in the Validation of Language Tests: Potential of the Method, State of the Evidence

References

• Alavi, S. M. (2005). On the adequacy of verbal protocols in examining an underlying construct of a test. Studies in Educational Evaluation, 31, 1–26.
   Google Scholar
• Anastasi, A. (1986). Evolving concepts of test validation. Annual Review of Psychology, 37, 1–16.
   Google Scholar
• Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8(1), 41–66.
   Google Scholar
• Angoff, W. (1988). Validity: An evolving concept. In H. Wainer & H. Braun (Eds.), Test validity, pp.19–32. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Banerjee, J., & Luoma, S. (1997). Qualitative approaches to test validation. In C. Clapham & D. Corson (Eds.), Encyclopedia of language and education: Language testing and assessment, 7, 275–287. Dordrecht: Kluwer Academic.
   Google Scholar
• Clapham, C. (1998). The effect of language proficiency and background knowledge on EAP students’ reading comprehension. In A. J. Kunnan (Ed.), Validation in language assessment: Selected papers from the 17th language testing and research colloquium, pp.141–168. Mahwah, NJ: Erlbaum.
   Google Scholar
• Cohen, A. D. (1984). On taking language tests: What the students report. Language Testing, 1, 70–81.
   Google Scholar
• Cohen, A. D. (1987). Using verbal reports in research on language learning. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.82–95. Clevedon, UK: Multiligual.
   Google Scholar
• Cohen, A. D. (1997). Toward enhancing verbal reports as a source of insights on test-taking strategies. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment, pp.339–366. Kopijyva, Jyva¨skyla: University of Jyva¨skyla.
   Google Scholar
• Crain-Thoreson, C., Lippman, M. Z., & McClendon-Magnuson, D. (1997). Windows on comprehension: Reading comprehension processes as revealed by two think-aloud procedures. Journal of Educational Psychology, 89, 579–591.
   Google Scholar
• Cumming, A. (1996). Introduction: The concept of validation in language testing. In A. Cumming & R. Berwick (Eds.), Validation in language testing, pp.1–13. Clevedon, UK: Multilingual.
   Google Scholar
• Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge: University of Cambridge Local Examinations Syndicate and Cambridge University Press.
   Google Scholar
• Duran, R. P. (1989). Testing of linguistic minorities. In R. L. Linn (Ed.), Educational measurement (3rd ed.), pp.573–587. New York: Macmillan.
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1987). Verbal reports on thinking. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.24–53. Clevedon, UK: Multilingual.
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
   Google Scholar
• Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27, 209–226.
   Google Scholar
• Garner, R. (1988). Verbal report data on cognitive and metacognitive strategies. In C. E. Weinstein & E. T. Goetz (Eds.), Learning and study strategies: Issues in Assessment, instruction, and evaluation, pp.63–76. London: Academic.
   Google Scholar
• Ginther, A., & Stevens, J. (1998). Language background, ethnicity, and the internal construct validity of the advanced placement Spanish language examination. In A. J. Kunnan (Ed.), Validation in language assessment: Selected papers from the 17th language testing and research colloquium, pp.169–194. Mahwah, NJ: Erlbaum.
   Google Scholar
• Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: University of Cambridge Local Examinations Syndicate and Cambridge University Press.
   Google Scholar
• Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing, 3, 159–185.
   Google Scholar
• Grotjahn, R. (1987). On the methodological basis of introspective methods. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.54–81. Clevedon, UK: Multilingual.
   Google Scholar
• Hamiliton, L. S., Nussbaum, E. M., & Snow, R. E. (1997). Interviewing procedures for validating science assessments. Applied Measurement in Education, 10(2), 181–200.
   Google Scholar
• Hamilton, L. S. (1994). Validating hands-on science assessments through an investigation of response processes. ERIC document reproduction service, ED 376 202.
   Google Scholar
• Kunnan, A. J. (1997). Connecting fairness with validation in language assessment. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment, pp.85–106. Jyva¨skyla: University of Jyva¨skyla.
   Google Scholar
• Kunnan, A. J. (1998). Approaches to validation in language assessment. In A. J. Kunnan (Ed.), Validation in language assessment, pp.1–15. Mahwah, NJ: Erlbaum.
   Google Scholar
• Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. The American Journal of Psychology, 113, 387–404.
   Google Scholar
• McNamara, T. (2000). Language testing. Oxford, UK: Oxford University Press.
   Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.), pp.13–103. New York: Macmillan.
   Google Scholar
• Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6(2), 199–255.
   Google Scholar
• Norris, S. P. (1991). Informal reasoning assessment: Using verbal reports of thinking to improve multiple-choice test validity. In J. F. Voss, D. N. Perkins & J. W. Segal (Eds.), Informal reasoning and education, pp.451–471. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Ortiz, S. O. (2005). Acculturational factors in psychological assessment. In R. L. Rhodes, S. H. Ochoa & S.
   Google Scholar
• O. Ortiz (Eds.), Assessing culturally and linguistically diverse students: A practical guide, pp.124–136. New York, NY: Guilford.
   Google Scholar
• Parry, K. J. (1987). Reading in a second culture. In J. Devine, P. L. Carrell & D. E. Eskey (Eds.), Research in reading in English as a second language, pp.59–70. Washington DC: Teachers of English to speakers of other language.
   Google Scholar
• Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing, 20(1), 26–56.
   Google Scholar
• Pritchard, R. (1990). The effects of cultural schemata on reading processing strategies. Reading Research Quarterly, 25, 273–295.
   Google Scholar
• Sasaki, M. (2000). Effects of cultural schemata on students’ test-taking processes for cloze tests: A multiple data source approach. Language Testing, 17(1), 85–114.
   Google Scholar
• Sepassi, F. (2003). How do learners of different language ability perform on the cloze? A verbal protocol analysis of EFL test takers performance on cloze tests. Indian Journal of Applied Linguistics, 29(2), 5–33.
   Google Scholar
• Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38(5), 553–573.
   Google Scholar
• Steffensen, M. S., & Joeg-Dev, C. (1984). Cultural knowledge and reading. In J. C. Alderson & A. H. Urquhart (Eds.), Reading in a foreign language. New York: Longman Group Ltd.
   Google Scholar
• Storey, P. (1997). Examining the test-taking process: A cognitive perspective on the discourse cloze test. Language Testing, 14(2), 214–231.
   Google Scholar
• Weir. C. J. (2005). Language testing and validation: An evidence-based approach. New York, NY: MacMillan.
   Google Scholar
• Wijgh, I. F. (1996). A communicative test in analysis: Strategies in reading authentic texts. In A. Cumming & R. Berwick (Eds.), Validation in language testing, pp.154–165. Clevedon, UK: Multilingual.
   Google Scholar
• Wu, Y. A. (1998). What do tests of listening comprehension test? - A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing, 15(1), 21–44.
   Google Scholar