Using Evidence-Centered Design to Support the Development of Culturally and Linguistically Sensitive Collaborative Problem-Solving Assessments

Abstract

Collaborative problem solving (CPS) ranks among the top five most critical skills necessary for college graduates to meet workforce demands (Hart Research Associates, 2015). It is also deemed a critical skill for educational success (Beaver, 2013). It thus deserves more prominence in the suite of courses and subjects assessed in K-16. Such inclusion, however, presents the need for improvements in the conceptualization, design, and analysis of CPS, which challenges us to think differently about assessing the skills than the current focus given to assessing individuals’ substantive knowledge. In this article, we discuss an Evidence-Centered Design approach to assess CPS in a culturally and linguistically diverse educational environment. We demonstrate ways to consider a sociocognitive perspective to conceptualize and model possible linguistic and/or cultural differences between populations along key stages of assessment development including assessment conceptualization and design to help reduce possible construct-irrelevant differences when assessing complex constructs with diverse populations.

Keywords:

• Evidence-centered design
• collaborative problem solving
• assessment design

Notes

1 While it is true that differences may exist in individuals’ behaviors in real-world contexts, it is also true that commonalities in skill-related behaviors exist across cultures and populations. Note that by culture, we do not solely mean culture as determined by nationality (e.g., US citizens, Canadians, or Japanese), but we also suggest culture in terms of the places where learning may take place, instructional practices, and content domains.

2 As an example, the suite of DPs to support designing tasks to assess model-based reasoning found in Mislevy, Haertel, Riconscente, Rutstein, and Ziker (2017) includes seven DPs for identifiable components of the construct, such as Model Formation and Model Revision, but also includes a DP for coordinating these components in an investigation. The DPs support constructing tasks that address single aspects of model-based reasoning or the joint application or interaction of the aspects in combinations that are needed for the assessment application.