Using Evidence-Centered Design to Support the Development of Culturally and Linguistically Sensitive Collaborative Problem-Solving Assessments

References

• Almond, R. G., Mislevy, R. J., Steinberg, L. S., Yan, D., & Williamson, D. M. (2015). Bayesian networks in educational assessment. New York, NY: Springer-Verlag.
   Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Baepler, P., Walker, J. D., & Driessen, M. (2014). It’s not about seat time: Blending, flipping, and efficiency in active learning classrooms. Computers & Education, 78, 227–236. doi: 10.1016/j.compedu.2014.06.006
   Web of Science ®Google Scholar
• Beaver, D. D. (2013). The many faces of collaboration and teamwork in scientific research: Updated reflections on scientific collaboration. COLLNET Journal of Scientometrics and Information Management, 7(1), 45–54. doi: 10.1080/09737766.2013.802629
   Google Scholar
• Bremner, S., Peirson-Smith, A., Jones, R., & Bhatia, V. (2014). Task design and interaction in collaborative writing the students’ story. Business and Professional Communication Quarterly, 77(2), 150–168. doi: 10.1177/2329490613514598
   Google Scholar
• Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.). (2011). Building a validity argument for the Test of English as a Foreign Language. London: Routledge.
   Google Scholar
• Chen, G., Donahue, L. M., & Klimoski, R. J. (2004). Training undergraduates to work in organizational teams. Academy of Management Learning and Education, 3, 27–40.
   Web of Science ®Google Scholar
• Conley, D. T. (2007). Redefining college readiness. Eugene, OR: Educational Policy Improvement Center.
   Google Scholar
• Cox, T. H., Lobel, S. A., & McLeod, P. L. (1991). Effects of ethnic group cultural differences on cooperative and competitive behavior on a group task. Academy of Management Journal, 34(4), 827–847. doi: 10.2307/256391
   Web of Science ®Google Scholar
• Daimler-Und, G., & Benz-Stiftung, K. (2001). Enhancing performance in high risk environments: Recommendations for the use of behavioural markers. Workshop, SwissAir training centre, Zurich, Switzerland, July 5-6, 2001. Kolleg Group interaction in High Risk Environments. Retrieved from http://www.abdn.ac.uk/iprc/documents/ants/GIHRE21_rec_for_use_of_beh_markers.pdf
   Google Scholar
• Daouk-Öyry, L., Zeinoun, P., Choueiri, L., & van de Vijver, F. J. (2016). Integrating global and local perspectives in psycholexical studies: A GloCal approach. Journal of Research in Personality, 62, 19–28. doi: 10.1016/j.jrp.2016.02.008
   Web of Science ®Google Scholar
• DeBarger, A. H., & Riconscente, M. M. (2005). An example-based exploration of design patterns in measurement. PADI Technical Reports (Report No. 8). Stanford, CA: SRI. Retrieved from http://padi.sri.com/downloads/TR8_VersionForDesign.pdf
   Google Scholar
• Ercikan, K., & Oliveri, M. E. (2016). Assessing 21st century skills: In search of validity evidence in support of the interpretation and use of assessments of complex constructs (Manuscript submitted to special issue on Contemporary Assessment Challenges: The Measurement of 21st Century Skills). Applied Measurement in Education, 29, 310–318
   Web of Science ®Google Scholar
• Ercikan, K., & Pellegrino, J.W. (Eds.). (2017). Validation of score meaning in the next generation of assessments. London: Routledge.
   Google Scholar
• Gorin, J. S., & Mislevy, R. J. (2013, September). Inherent measurement challenges in the next generation science standards for both formative and summative assessment. Paper presented at the Invitational Research Symposium on Science Assessment, Washington DC.
   Google Scholar
• Gupta, V. (2015). 6 secrets to navigating cross-cultural differences. Entrepreneur. Retrieved from https://www.entrepreneur.com/article/241372.
   Google Scholar
• Hambleton, R. K., Merenda, P., & Spielberger, C. (Eds.). (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. Mahwah, NJ: Erlbaum.
   Google Scholar
• Hart Research Associates. (2015). Falling short? College learning and career success: Selected findings from online surveys of employers and college students conducted on behalf of the Association of American Colleges & Universities. Washington, DC: Author. Retrieved from https://www.aacu.org/sites/default/files/files/LEAP/2015employerstudentsurvey.pdf
   Google Scholar
• Heckman, J. J., Stixrud, J., & Urzua, S. (2006). The effects of cognitive and noncognitive abilities on labor market outcomes and social behavior (Research Report No. w12006). Washington, DC: National Bureau of Economic Research.
   Google Scholar
• Hesse, F., Care, E., Buder, J., Sassenberg, K., & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach. Dordrecht, The Netherlands: Springer.
   Google Scholar
• International Test Commission. (2018). ITC guidelines for the large-scale assessment of linguistically and culturally diverse populations. Retrieved from www.InTestCom.org
   Google Scholar
• Johnston, C. G., James, R. H., Lye, J. N., & McDonald, I. M. (2000). An evaluation of collaborative problem solving for learning economics. Journal of Economic Education, 31(1), 13–29. doi: 10.1080/00220480009596758
   Web of Science ®Google Scholar
• Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. doi: 10.1037/0033-2909.112.3.527
   Web of Science ®Google Scholar
• Kim, K. J., & Bonk, C. J. (2002). Cross‐cultural comparisons of online collaboration. Journal of Computer-Mediated Communication, 8(1), 0. doi: 10.1111/j.1083-6101.2002.tb00163.x.
   Google Scholar
• Kyllonen, P. C. (2012). Measurement of 21st century skills within the common core state standards. Paper presented at the K-12 Center at ETS invitational research symposium on technology enhanced assessments. Retrieved from http://www.k12center.org/rsc/pdf/session5-kyllonen-paper-tea2012.pdf
   Google Scholar
• LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88.
   Google Scholar
• Lewis, R. D. (2006). When cultures collide: Leading across cultures (pp. 125–139). Boston, MA: Nicholas Brealey Publishing.
   Google Scholar
• Liu, M., & Haertel, G. (2011). Design patterns: A tool to support assessment task authoring (Large-Scale Assessment Technical Report 11). Menlo Park, CA: SRI International.
   Google Scholar
• Lowry, P. B., Curtis, A., & Lowry, M. R. (2004). Building a taxonomy and nomenclature of collaborative writing to improve interdisciplinary research and practice. Journal of Business Communication, 41(1), 66–99. doi: 10.1177/0021943603259363
   Google Scholar
• Maddox, B. (2015). The neglected situation: Assessment performance and interaction in context. Assessment in Education, 22(4), 427–443. doi: 10.1080/0969594X.2015.1026246
   Web of Science ®Google Scholar
• Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13–23. Retrieved from http://www.jstor.org/stable/1176219 doi: 10.3102/0013189X023002013
   Google Scholar
• Mislevy, R. J. (2017). Resolving the paradox of rich performance tasks. In H. Jiao & Lissitz, R. W. (Eds.), Test fairness in the new generation of large-scale assessment (pp. 1–46). Charlotte, NC: Information Age Publishing.
   Google Scholar
• Mislevy, R. J. (2018). Sociocognitive foundations of educational measurement. London: Routledge.
   Google Scholar
• Mislevy, R. J., & Durán, R. P. (2014). A sociocognitive perspective on assessing EL students in the age of Common Core and Next Generation Science Standards. TESOL Quarterly, 48(3), 560–585. doi: 10.1002/tesq.177
   Web of Science ®Google Scholar
• Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence‐centered design for educational testing. Educational Measurement: Issues and Practice, 25(4), 6–20. doi: 10.1111/j.1745-3992.2006.00075.x
   Google Scholar
• Mislevy, R. J., Haertel, G. D., Cheng, B., Rutstein, D., Vendlinski, T., Murray, E., … Colker, A. M. (2013). Conditional inferences related to focal and additional knowledge, skills, and abilities. Assessment for Students with Disabilities Technical Report 5. Menlo Park, CA: SRI International. Retrieved from http://cresst.org/wp-content/uploads/TR587.pdf
   Google Scholar
• Mislevy, R. J., Haertel, G. D., Riconscente, M. M., Rutstein, D. W., & Ziker, C. (2017). Assessing model-based reasoning using evidence-centered design. Cham, Switzerland: Springer Nature.
   Google Scholar
• Mislevy, R. J., Hamel, L., Fried, R., Gaffney, T., Haertel, G., … Wenk, A. (2003). Design patterns for assessing science inquiry. Principled Assessment Designs for Inquiry (PADI) Technical Report 1. Menlo Park, CA: SRI International. Retrieved from http://ecd.sri.com/downloads/ECD_TR6_Model-Based_Reasoning.pdf
   Google Scholar
• Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2003). On the structure of educational assessment (with discussion). Measurement: Interdisciplinary Research and Perspective, 1(1), 3–62. doi: 10.1207/S15366359MEA0101_02
   Google Scholar
• Mislevy, R. J., & Yin, C. (2009). If language is a complex adaptive system, what is language assessment? Language Learning, 59, 249–267. doi: 10.1111/j.1467-9922.2009.00543.x
   Web of Science ®Google Scholar
• National Research Council. (2010). Assessing 21st century skills: Summary of a workshop. J. A. Koenig, Rapporteur, Committee on the Assessment of 21st Century Skills, Board on Testing and Assessment, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
   Google Scholar
• National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21^st century. Committee on Defining Deeper Learning and 21^st Century Skills. In J. W. Pellegrino and M. L. Hilton (Eds.), Board on Testing and Assessment and Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
   Google Scholar
• Oliveri, M. E., & Ercikan, K. (2011). Do different approaches to examining construct comparability in multilanguage assessments lead to similar conclusions? Applied Measurement in Education, 24(4), 349–366. doi: 10.1080/08957347.2011.607063
   Web of Science ®Google Scholar
• Oliveri, M. E., & Lawless, R. R. (2018). Analyzing the validity of large-scale assessments administered globally. ETS Research Report. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Oliveri, M. E., Lawless, R. R., & Molloy, H. (2017). A review of collaborative problem solving. ETS Research Report. Princeton, NJ: Educational Testing Service. doi: 10.1002/ets2.12133
   Google Scholar
• Oliveri, M. E., & Markle, R. (2017). Expanding skills in higher education. ETS Research Report, 17-09. Princeton, NJ: Educational Testing Service. doi: 10.1002/ets2.12137
   Google Scholar
• Oliveri, M. E., & Tannenbaum, R. J. (in press). Are we teaching & assessing the relevant English skills for success in the international workplace? A case to expand the skillset to mitigate negative consequences of low English language proficiency. In V. H. Kenon & S. V. Palsole (Eds.), Wiley Handbooks in Education: The Wiley Handbook of Global Workplace Learning. Hoboken, NJ: Wiley-Blackwell.
   Google Scholar
• Oliveri, M. E., & von Davier, A. A. (2016). Psychometrics in support of a valid assessment of linguistic minorities: Implications for the test and sampling designs. International Journal of Testing, 16(3), 200–239. doi: 10.1080/15305058.2015.1069743
   Web of Science ®Google Scholar
• Oliveri, M. E., & von Davier, M. (2017). Examining trends in item misfit in international large-scale assessments. In H. Jiao & R. W. Lissitz (Eds.), Test fairness in the new generation of large-scale assessment (pp. 121–146). Charlotte, NC: Information Age.
   Google Scholar
• Partnership for 21st Century Skills. (2012). Framework for 21st century learning. Retrieved from http://www.p21.org/our-work/p21-framework.
   Google Scholar
• Peterson, S. E., & Miller, J. A. (2004). Comparing the quality of students’ experiences during cooperative learning and large-group instruction. The Journal of Educational Research, 97(3), 123–134. doi: 10.3200/JOER.97.3.123-134
   Web of Science ®Google Scholar
• Riconscente, M. M., Mislevy, R. J., & Hamel, L. (2005). An introduction to PADI task templates. PADI Technical Report (Report No. 3). Stanford, CA: SRI.
   Google Scholar
• Sinharay, S., Puhan, G., & Haberman, S. J. (2010). Reporting diagnostic scores in educational testing: Temptations, pitfalls, and some solutions. Multivariate Behavioral Research, 45(3), 553–573. doi: 10.1080/00273171.2010.483382
   PubMed Web of Science ®Google Scholar
• Taras, V., Kirkman, B. L., & Steel, P. (2010). Examining the impact of culture’s consequences: A three-decade, multilevel, meta-analytic review of Hofstede’s cultural value dimensions. The Journal of Applied Psychology, 95, 405–439.
   PubMed Web of Science ®Google Scholar
• Toulmin, S. (1958). The uses of argument. Cambridge, UK: Cambridge University Press.
   Google Scholar
• von Davier, A. A., & Halpin, P. F. (2013). Collaborative problem solving and the assessment of cognitive skills: Psychometric considerations (Research Report No. RR-2013-2). Princeton, NJ: Educational Testing Service.
   Google Scholar