Measuring collaborative problem solving: research agenda and assessment instrument

References

• Adams, R., Vista, A., Scoular, C., Awwal, N., Griffin, P., & Care, E. (2015). Automatic coding procedures for collaborative problem solving. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 115–132). Springer.
   Google Scholar
• Andrews-Todd, J., & Forsyth, C. M. (2020). Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Computers in Human Behavior, 104, 105759. https://doi.org/10.1016/j.chb.2018.10.025
   Web of Science ®Google Scholar
• Binkley, M., Erstad, O., Herman, J., Raizen, S., Martin, R., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 17–66). Springer.
   Google Scholar
• Care, E., & Griffin, P. (2017). Assessment of collaborative problem solving processes. In B. Csapó & J. Funke (Eds.), The nature of problem solving: Using research to inspire 21st century learning (pp. 227–243). OECD Publishing.
   Google Scholar
• Care, E., Griffin, P., Scoular, C., Awwal, N., & Zoanetti, N. (2015). Collaborative problem solving tasks. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 85–104). Springer.
   Google Scholar
• Chung, G. K. W. K., O’Neil, H. F., & Herl, H. E. (1999). The use of computer-based collaborative knowledge mapping to measure team processes and team outcomes. Computers in Human Behavior, 15(3), 463–493. https://doi.org/10.1016/S0747-5632(99)00032-1
   Web of Science ®Google Scholar
• Csapó, B., Ainley, J., Bennett, R., Latour, T., & Law, N. (2012). Technological issues of computer-based assessment of 21st century skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 143–230). Springer.
   Google Scholar
• Csapó, B., & Funke, J. (Eds.). (2017). The nature of problem solving: Using research to inspire 21st century learning. OECD Publishing.
   Google Scholar
• Csapó, B., & Molnár, G. (2019). Online diagnostic assessment in support of personalized teaching and learning: The eDia system. Frontiers in Psychology, 10, 1522. https://doi.org/10.3389/fpsyg.2019.01522
   PubMed Web of Science ®Google Scholar
• Dingler, C., von Davier, A. A., & Hao, J. (2017). Methodological challenges in measuring collaborative problem-solving skills over time. In Team dynamics over time (Research on managing groups and teams). (Vol. 18, pp. 51–70). Emerald Publishing Limited. https://doi.org/10.1108/S1534-085620160000018003
   Google Scholar
• Dowell, N. M., Nixon, T. M., & Graesser, A. C. (2019). Group communication analysis: A computational linguistics approach for detecting sociocognitive roles in multiparty interactions. Behavior Research Methods, 51(3), 1007–1041. https://doi.org/10.3758/s13428-018-1102-z
   PubMed Web of Science ®Google Scholar
• Fiore, S. M., Graesser, A., & Greiff, S. (2018). Collaborative problem-solving education for the twenty-first-century workforce. Nature Human Behaviour, 2(6), 367–369. https://doi.org/10.1038/s41562-018-0363-y
   PubMed Web of Science ®Google Scholar
• Fiore, S. M., Graesser, A., Greiff, S., Griffin, P., Gong, B., Kyllonen, P., Massey, C., O’Neil, H., Pellegrino, J., Rothman, R., Soulé, H., & von Davier, A. (2017). Collaborative problem solving: Considerations for the National Assessment of Educational Progress.
   Google Scholar
• Fiore, S. M., & Kapalo, K. A. (2017). Innovation in team interaction: New methods for assessing collaboration between brains and bodies using a multi-level framework. In A. A. von Davier, M. Zhu, & P. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 51–64). Springer.
   Google Scholar
• Fiore, S. M., & Wiltshire, T. J. (2016). Technology as teammate: Examining the role of external cognition in support of team cognitive processes. Frontiers in Psychology, 7, 1531. https://doi.org/10.3389/fpsyg.2016.01531
   PubMed Web of Science ®Google Scholar
• Fischer, A., Greiff, S., & Funke, J. (2012). The process of solving complex problems. The Journal of Problem Solving, 4(1), 19–42. https://doi.org/10.7771/1932-6246.1118
   Google Scholar
• Fischer, A., Greiff, S., & Funke, J. (2017). The history of complex problem solving. In B. Csapó & J. Funke (Eds.), The nature of problem solving: Using research to inspire 21st century learning (pp. 107–121). OECD Publishing.
   Google Scholar
• Fischer, A., Greiff, S., Wüstenberg, S., Fleischer, J., Buchwald, F., & Funke, J. (2015). Assessing analytic and interactive aspects of problem solving competency. Learning and Individual Differences, 39, 172–179. https://doi.org/10.1016/j.lindif.2015.02.008
   Web of Science ®Google Scholar
• Fuks, H., Pimentel, M., & de Lucena, C. J. P. (2006). RU-Typing-2-Me? Evolving a chat tool to increase understanding in learning activities. International Journal of Computer-Supported Collaborative Learning, 1(1), 117–142. https://doi.org/10.1007/s11412-006-6845-3
   Web of Science ®Google Scholar
• Graesser, A. C., Dowell, N., & Clewley, D. (2017). Assessing collaborative problem solving through conversational agents. In A. von Davier, M. Zhu, & P. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 65–80). Springer.
   Google Scholar
• Graesser, A. C., Fiore, S. M., Greiff, S., Andrews-Todd, J., Foltz, P. W., & Hesse, F. W. (2018a). Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19(2), 59–92. https://doi.org/10.1177/1529100618808244
   PubMed Web of Science ®Google Scholar
• Graesser, A. C., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. L. (2018b). Challenges of assessing collaborative problem solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and teaching of 21st century skills (pp. 55–73). Springer.
   Google Scholar
• Graesser, A. C., Forsyth, C. M., & Foltz, P. (2017). Assessing conversation quality, reasoning, and problem solving with computer agents. In B. Csapó & J. Funke (Eds.), The nature of problem solving: Using research to inspire 21st century learning (pp. 245–261). OECD Publishing.
   Google Scholar
• Greiff, S., & Funke, J. (2017). Interactive problem solving: Exploring the potential of minimal complex systems. In B. Csapó & J. Funke (Eds.), The nature of problem solving: Using research to inspire 21st century learning (pp. 93–105). OECD Publishing.
   Google Scholar
• Greiff, S., Wüstenberg, S., & Funke, J. (2012). Dynamic problem solving: A new assessment perspective. Applied Psychological Measurement, 36(3), 189–213. https://doi.org/10.1177/0146621612439620
   Web of Science ®Google Scholar
• Griffin, P., & Care, E. (Eds.). (2015). Assessment & teaching of 21st century skills. Methods and approach. Springer.
   Google Scholar
• Hao, J., Liu, L., von Davier, A. A., & Kyllonen, P. C. (2017). Initial steps towards a standardized assessment for collaborative problem solving (CPS): Practical challenges and strategies. In A. von Davier, M. Zhu, & P. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 135–156). Springer.
   Google Scholar
• He, Q., von Davier, M., Greiff, S., Steinhauer, E. W., & Borysewicz, P. B. (2017). Collaborative problem solving measures in the programme for international student assessment (PISA). In A. A. von Davier, M. Zhu, & P. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 95–111). Springer.
   Google Scholar
• Herborn, K., Stadler, M., Mustafić, M., & Greiff, S. (2020). The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans? Computers in Human Behavior, 104, 105624. https://doi.org/10.1016/j.chb.2018.07.035
   Web of Science ®Google Scholar
• Hesse, F., Care, E., Buder, J., Sassenberg, K, & Griffin, P. (2015). A framework for teachable collaborative problem solving skills. In P. Griffin & E. Care (Eds.), Assessment and teaching of 21st century skills: Methods and approach (pp. 37–56). Springer.
   Google Scholar
• Hsieh, I. L., & O’Neil, H. F. (2002). Types of feedback in a computer-based collaborative problem-solving group task. Computers in Human Behavior, 18(1), 699–715. https://doi.org/10.1016/S0747-5632(02)00025-0
   Google Scholar
• Krkovic, K., Pásztor-Kovács, A., Molnár, G., & Greiff, S. (2014). New technologies in psychological assessment: The example of computer-based collaborative problem solving assessment. International Journal of e-Assessment, 1(1).
   Google Scholar
• Krkovic, K., Wüstenberg, S., & Greiff, S. (2016). Assessing collaborative behavior in students. European Journal of Psychological Assessment, 32(1), 52–60. https://doi.org/10.1027/1015-5759/a000329
   Web of Science ®Google Scholar
• Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (Eds.). (2013). Handbook of latent semantic analysis. Psychology Press.
   Google Scholar
• Liu, L., Hao, J., von Davier, A., Kyllonen, P., & Zapata-Rivera, D. (2016). A tough nut to crack: Measuring collaborative problem solving. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-life skill development (pp. 344–359). IGI Global.
   Google Scholar
• Molnár, G., & Csapó, B. (2019). Making the psychological dimension of learning visible: Using technology-based assessment to monitor students’ cognitive development. Frontiers in Psychology, 10, 1368. https://doi.org/10.3389/fpsyg.2019.01368
   PubMed Web of Science ®Google Scholar
• OECD. (2017). PISA 2015 results (volume V): Collaborative problem solving.
   Google Scholar
• O’Neil, H. F., Chuang, S., & Chung, G. K. W. K. (2003). Issues in the computer-based assessment of collaborative problem solving. Assessment in Education, 10(3), 361–373. https://doi.org/10.1080/0969594032000148190
   Google Scholar
• O’Neil, H. F., Chung, G. K. W. K., & Brown, R. S. (1997). Use of networked simulations as a context to measure team competencies. In H. F. O’Neil (Ed.), Workforce readiness: Competencies and assessment (pp. 411–452). Lawrence Erlbaum.
   Google Scholar
• Pásztor-Kovács, A. (2018). A kollaboratív problémamegoldó képesség mérése [The assessment of collaborative problem solving skills]. [Doctoral dissertation]. Doctoral School of Education, University of Szeged.
   Google Scholar
• Pásztor-Kovács, A., Pásztor, A., & Molnár, G. (2018). Kollaboratív problémamegoldó képességet vizsgáló dinamikus teszt fejlesztése [Development of an online interactive instrument for assessing collaborative problem solving comptence]. Magyar Pedagógia, 118(1), 73–102. https://doi.org/10.17670/MPed.2018.1.73
   Google Scholar
• Reilly, J. M., & Schneider, B. (2019). Predicting the quality of collaborative problem solving through linguistic analysis of discourse. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of The 12th International Conference on Educational Data Mining (EDM 2019) (pp. 149–157).
   Google Scholar
• Rosé, C. P., Howley, I., Wen, M., Yang, D., & Ferschke, O. (2017). Assessment of discussion in learning contexts. In A. von Davier, M. Zhu, & P. Kyllonen (Eds.), Innovative assessment of collaboration (pp. 81–94). Springer.
   Google Scholar
• Rosen, Y. (2017). Assessing students in human-to-agent settings to inform collaborative problem-solving learning. Journal of Educational Measurement, 54(1), 36–53. https://doi.org/10.1111/jedm.12131
   Web of Science ®Google Scholar
• Rosen, Y., & Foltz, P. (2014). Assessing collaborative problem solving through automated technologies. Research and Practice in Technology Enhanced Learning, 9(3), 389–410.
   Google Scholar
• Rosen, Y., & Mosharraf, M. (2016). Computer agent technologies in collaborative assessments. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-life skill development (pp. 319–343). IGI Global.
   Google Scholar
• Rosen, Y., Wolf, I., & Stoeffler, K. (2020). Fostering collaborative problem solving skills in science: The Animalia project. Computers in Human Behavior, 104, 105922. https://doi.org/10.1016/j.chb.2019.02.018
   Web of Science ®Google Scholar
• Scoular, C., & Care, E. (2020). Monitoring patterns of social and cognitive student behaviors in online collaborative problem solving assessments. Computers in Human Behavior, 104, 105874. https://doi.org/10.1016/j.chb.2019.01.007
   Web of Science ®Google Scholar
• Scoular, C., Care, E., & Hesse, F. W. (2017). Designs for operationalizing collaborative problem solving for automated assessment. Journal of Educational Measurement, 54(1), 12–35. https://doi.org/10.1111/jedm.12130
   Web of Science ®Google Scholar
• Stadler, M., Herborn, K., Mustafić, M., & Greiff, S. (2020). The assessment of collaborative problem solving in PISA 2015: An investigation of the validity of the PISA 2015 CPS tasks. Computers & Education, 157, 103964. https://doi.org/10.1016/j.compedu.2020.103964
   Web of Science ®Google Scholar
• Stoeffler, K., Rosen, Y., Bolsinova, M., & von Davier, A. A. (2020). Gamified performance assessment of collaborative problem solving skills. Computers in Human Behavior, 104, 106036. https://doi.org/10.1016/j.chb.2019.05.033
   Web of Science ®Google Scholar
• Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D'Mello, S. (2020). Towards a generalized competency model of collaborative problem solving. Computers & Education, 143, 103672. https://doi.org/10.1016/j.compedu.2019.103672
   Web of Science ®Google Scholar
• Wüstenberg, S., Greiff, S., & Funke, J. (2012). Complex problem solving — More than reasoning? Intelligence, 40(1), 1–14. https://doi.org/10.1016/j.intell.2011.11.003
   Web of Science ®Google Scholar
• Yuan, J., Liu, H., & Liu, Y. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10, 369. https://doi.org/10.3389/fpsyg.2019.00369
   PubMed Web of Science ®Google Scholar
• Zhang, J. (1998). A distributed representation approach to group problem solving. Journal of the American Society for Information Science, 49(9), 801–809. https://doi.org/10.1002/(SICI)1097-4571(199807)49:9<801::AID-ASI5>3.0.CO;2-Q
   Web of Science ®Google Scholar