The development and validation of an instrument to measure evaluative judgement: a focus on engineering students’ judgement of intercultural competence

References

• Ajjawi, R., J. Tai, P. Dawson, and D. Boud. 2018. “Conceptualising Evaluative Judgement for Sustainable Assessment in Higher Education.” In Developing Evaluative Judgement in Higher Education: Assessment for Knowing and Producing Quality Work, edited by D. Boud, R. Ajjawi, P. Dawson, and J. Tai, 7–18. Abingdon: Routledge.
   Google Scholar
• Bentler, P. M., and D. G. Bonett. 1980. “Significance Tests and Goodness of Fit in the Analysis of Covariance Structures.” Psychological Bulletin 88 (3): 588–606. doi:10.1037/0033-2909.88.3.588.
   Web of Science ®Google Scholar
• Boud, D. 2000. “Sustainable Assessment: Rethinking Assessment for the Learning Society.” Studies in Continuing Education 22 (2): 151–167. doi:10.1080/713695728.
   Google Scholar
• Boud, D., P. Dawson, J. Tai, and R. Ajjawi. 2018. “Creating an Agenda for Developing Students’ Evaluative Judgement.” In Developing Evaluative Judgement in Higher Education: Assessment for Knowing and Producing Quality Work, edited by D. Boud, R. Ajjawi, P. Dawson, and J. Tai, 186–197. Abingdon: Routledge.
   Google Scholar
• Bourke, R. 2014. “Self-Assessment in Professional Programmes within Tertiary Institutions.” Teaching in Higher Education 19 (8): 908–918. doi:10.1080/13562517.2014.934353.
   Web of Science ®Google Scholar
• Brown, T. A. 2006. Confirmatory Factor Analysis for Applied Research. New York: Guilford Press.
   Google Scholar
• Brown, G. T. L., and L. R. Harris. 2013. “Student Self-Assessment.” In The SAGE Handbook of Research on Classroom Assessment, edited by J. H. McMillan, 367–393. Thousand Oaks, CA: Sage.
   Google Scholar
• Chan, C. K. Y., E. T. Y. Fong, L. Y. Y. Luk, and R. Ho. 2017. “A Review of Literature on Challenges in the Development and Implementation of Generic Competencies in Higher Education Curriculum.” International Journal of Educational Development 57: 1–10. doi:10.1016/j.ijedudev.2017.08.010.
   Web of Science ®Google Scholar
• Chan, C. K. Y., and L. Y. Y. Luk. 2022. “Going ‘Grade-Free’? – Teachers’ and Students’ Perceived Value and Grading Preferences for Holistic Competency Assessment.” Higher Education Research & Development 41 (3): 647–664. doi:10.1080/07294360.2021.1877628.
   Web of Science ®Google Scholar
• Chen, L., S. Howitt, D. Higgins, and S. Murray. 2022. “Students’ Use of Evaluative Judgement in an Online Peer Learning Community.” Assessment & Evaluation in Higher Education 47 (4): 493–506. doi:10.1080/02602938.2021.1933378.
   Web of Science ®Google Scholar
• Cortina, J. M. 1993. “What is Coefficient Alpha? An Examination of Theory and Applications.” Journal of Applied Psychology 78 (1): 98–104. doi:10.1037/0021-9010.78.1.98.
   Web of Science ®Google Scholar
• Cowan, J. 2010. “Developing the Ability for Making Evaluative Judgements.” Teaching in Higher Education 15 (3): 323–334. doi:10.1080/13562510903560036.
   Web of Science ®Google Scholar
• Dawson, P., R. Ajjawi, D. Boud, and J. Tai. 2018. Introduction: What is Evaluative Judgement? In Developing Evaluative Judgement in Higher Education: Assessment for Knowing and Producing Quality Work, edited by D. Boud, R. Ajjawi, P. Dawson, & J. Tai, 1–4. Abingdon: Routledge.
   Google Scholar
• Deardorff, D. K. 2006. “Identification and Assessment of Intercultural Competence as a Student Outcome of Internationalization.” Journal of Studies in International Education 10 (3): 241–266. doi:10.1177/1028315306287002.
   Google Scholar
• Falchikov, N., and D. Boud. 1989. “Student Self-Assessment in Higher Education: A Meta-Analysis.” Review of Educational Research 59 (4): 395–430. doi:10.3102/00346543059004395.
   Web of Science ®Google Scholar
• Field, A. P. 2009. Discovering Statistics Using SPSS. 3rd ed. London: Sage.
   Google Scholar
• Fornell, C., and D. F. Larcker. 1981. “Evaluating Structural Equation Models with Unobservable Variables and Measurement Error.” Journal of Marketing Research 18 (1): 39–50. doi:10.1177/002224378101800313.
   Web of Science ®Google Scholar
• Gladovic, C., J. H. M. Tai, and P. Dawson. 2022. “Qualitative Approaches to Researching Evaluative Judgement in Pedagogical Activities: A Case Study.” Assessment & Evaluation in Higher Education 47 (2): 231–244. doi:10.1080/02602938.2021.1901854.
   Web of Science ®Google Scholar
• Gyamfi, G., B. E. Hanna, and H. Khosravi. 2022. “The Effects of Rubrics on Evaluative Judgement: A Randomised Controlled Experiment.” Assessment & Evaluation in Higher Education 47 (1): 126–143. doi:10.1080/02602938.2021.1887081.
   Web of Science ®Google Scholar
• Handford, M., J. Van Maele, P. Matous, and Y. Maemura. 2019. “Which “Culture”? a Critical Analysis of Intercultural Communication in Engineering Education.” Journal of Engineering Education 108 (2): 161–177. doi:10.1002/jee.20254.
   Web of Science ®Google Scholar
• Hoyle, R. H., and M. R. Sherrill. 2006. “Future Orientation in the Self-System: Possible Selves, Self-Regulation, and Behavior.” Journal of Personality 74 (6): 1673–1696. doi:10.1111/j.1467-6494.2006.00424.x.
   PubMed Web of Science ®Google Scholar
• Hu, L. T., and P. M. Bentler. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives.” Structural Equation Modeling: A Multidisciplinary Journal 6 (1): 1–55. doi:10.1080/10705519909540118.
   Web of Science ®Google Scholar
• Kaiser, H. F. 1960. “The Application of Electronic Computers to Factor Analysis.” Educational and Psychological Measurement 20 (1): 141–151. doi:10.1177/001316446002000116.
   Web of Science ®Google Scholar
• Kensington-Miller, B., B. Knewstubb, A. Longley, and A. Gilbert. 2018. “From Invisible to SEEN: A Conceptual Framework for Identifying, Developing and Evidencing Unassessed Graduate Attributes.” Higher Education Research & Development 37 (7): 1439–1453. doi:10.1080/07294360.2018.1483903.
   Web of Science ®Google Scholar
• Kline, R. B. 2011. Principles and Practice of Structural Equation Modeling. 3rd ed. Guilford.
   Google Scholar
• Kramsch, C. 1999. “Thirdness: The Intercultural Stance.” In Language, Culture and Identity, edited by T. Vestergaard, 53–89. Aalborg: Aalborg University Press.
   Google Scholar
• Li, H., Y. Xiong, X. Zang, M. L. Kornhaber, Y. Lyu, K. S. Chung, and H. K. Suen. 2016. “Peer Assessment in the Digital Age: A Meta-Analysis Comparing Peer and Teacher Ratings.” Assessment & Evaluation in Higher Education 41 (2): 245–264. doi:10.1080/02602938.2014.999746.
   Web of Science ®Google Scholar
• Luo, J., and C. K. Y. Chan. 2022a. “Qualitative Methods to Assess Intercultural Competence in Higher Education Research: A Systematic Review with Practical Implications.” Educational Research Review 37: 100476. doi:10.1016/j.edurev.2022.100476.
   Web of Science ®Google Scholar
• Luo, J., and C. K. Y. Chan. 2022b. “Exploring the Process of Evaluative Judgement: The Case of Engineering Students Judging Intercultural Competence.” Assessment & Evaluation in Higher Education.
   Web of Science ®Google Scholar
• Luo, J., and C. K. Y. Chan. 2023. “Conceptualising Evaluative Judgement in the Context of Holistic Competency Development: Results of a Delphi Study.” Assessment & Evaluation in Higher Education 48 (4): 513–528. doi:10.1080/02602938.2022.2088690.
   Web of Science ®Google Scholar
• Mahadevan, J. 2014. “Intercultural Engineering beyond Stereotypes.” European Journal of Training and Development 38 (7): 658–672. doi:10.1108/EJTD-10-2013-0107.
   Google Scholar
• Miller, R. B., and S. J. Brickman. 2004. “A Model of Future-Oriented Motivation and Self-Regulation.” Educational Psychology Review 16 (1): 9–33. doi:10.1023/B:EDPR.0000012343.96370.39.
   Web of Science ®Google Scholar
• Ng, J. R., and J. K. Earl. 2008. “Accuracy in Self-Assessment: The Role of Ability, Feedback, Self-Efficacy and Goal Orientation.” Australian Journal of Career Development 17 (3): 39–50. doi:10.1177/103841620801700307.
   Google Scholar
• Osborne, J. W., and A. B. Costello. 2005. “Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most from Your Analysis.” Pan-Pacific Management Review 12 (2): 131–146.
   Google Scholar
• Panadero, E., J. Broadbent, D. Boud, and J. M. Lodge. 2019. “Using Formative Assessment to Influence Self- and co-Regulated Learning: The Role of Evaluative Judgement.” European Journal of Psychology of Education 34 (3): 535–557. doi:10.1007/s10212-018-0407-8.
   Web of Science ®Google Scholar
• Panadero, E., A. Jonsson, and J. Botella. 2017. “Effects of Self-Assessment on Self-Regulated Learning and Self-Efficacy: Four Meta-Analyses.” Educational Research Review 22: 74–98. doi:10.1016/j.edurev.2017.08.004.
   Web of Science ®Google Scholar
• Rees, C. E., A. Bullock, K. L. Mattick, and L. V. Monrouxe. 2018. “Using Workplace-Learning Narratives to Explore Evaluative Judgement in Action.” In Developing Evaluative Judgement in Higher Education: Assessment for Knowing and Producing Quality Work, edited by D. Boud, R. Ajjawi, P. Dawson, and J. Tai, 176–186. Abingdon: Routledge.
   Google Scholar
• Rico-Garcia, M., and L. V. Fielden Burns. 2020. “Intercultural Communication in Engineering Studies: A Key Competence in Global Labour Markets.” European Journal of Engineering Education 45 (6): 833–853. doi:10.1080/03043797.2019.1654980.
   Web of Science ®Google Scholar
• Roth, A., S. Ogrin, and B. Schmitz. 2016. “Assessing Self-Regulated Learning in Higher Education: A Systematic Literature Review of Self-Report Instruments.” Educational Assessment, Evaluation and Accountability 28 (3): 225–250. doi:10.1007/s11092-015-9229-2.
   Web of Science ®Google Scholar
• Sadler, D. R. 1989. “Formative Assessment and the Design of Instructional Systems.” Instructional Science 18 (2): 119–144. doi:10.1007/BF00117714.
   Web of Science ®Google Scholar
• Shmueli, G. 2010. “To Explain or to Predict?” Statistical Science 25 (3): 289–310. doi:10.1214/10-STS330.
   Web of Science ®Google Scholar
• Sung, Y. T., K. E. Chang, T. H. Chang, and W. C. Yu. 2010. “How Many Heads Are Better than One? The Reliability and Validity of Teenagers’ Self- and Peer Assessments.” Journal of Adolescence 33 (1): 135–145. doi:10.1016/j.adolescence.2009.04.004.
   PubMed Web of Science ®Google Scholar
• Tai, J., R. Ajjawi, D. Boud, P. Dawson, and E. Panadero. 2018. “Developing Evaluative Judgement: Enabling Students to Make Decisions about the Quality of Work.” Higher Education 76 (3): 467–481. doi:10.1007/s10734-017-0220-3.
   Web of Science ®Google Scholar
• Tan, K. H. K. 2009. “Meanings and Practices of Power in Academics’ Conceptions of Student Self-Assessment.” Teaching in Higher Education 14 (4): 361–373. doi:10.1080/13562510903050111.
   Web of Science ®Google Scholar
• Thomas, G., D. Anderson, and S. Nashon. 2008. “Development of an Instrument Designed to Investigate Elements of Science Students’ Metacognition, Self-Efficacy and Learning Processes: The SEMLI-S.” International Journal of Science Education 30 (13): 1701–1724. doi:10.1080/09500690701482493.
   Web of Science ®Google Scholar
• Thomas, D. C., Y. Liao, Z. Aycan, J.-L. Cerdin, A. A. Pekerti, E. C. Ravlin, G. K. Stahl, et al. 2015. “Cultural Intelligence: A Theory-Based, Short Form Measure.” Journal of International Business Studies 46 (9): 1099–1118. doi:10.1057/jibs.2014.67.
   Web of Science ®Google Scholar
• Wilbur, G. 2016. “The Staying Power of Intercultural Learning through Reflective Inquiry.” Reflective Practice 17 (1): 59–71. doi:10.1080/14623943.2015.1123687.
   Web of Science ®Google Scholar
• Winstone, N., and D. Carless. 2019. “Designing Effective Feedback Processes in Higher Education: A Learning-Focused Approach.” London: Routledge.
   Google Scholar
• Yan, Z. 2020. “Developing a Short Form of the Self-Assessment Practices Scale: Psychometric Evidence.” Frontiers in Education 4: 1–10. doi:10.3389/feduc.2019.00153.
   Google Scholar
• Yan, Z., and G. T. L. Brown. 2017. “A Cyclical Self-Assessment Process: Towards a Model of How Students Engage in Self-Assessment.” Assessment & Evaluation in Higher Education 42 (8): 1247–1262. doi:10.1080/02602938.2016.1260091.
   Web of Science ®Google Scholar
• Zhan, Y. 2022. “Developing and Validating a Student Feedback Literacy Scale.” Assessment & Evaluation in Higher Education 47 (7): 1087–1100. doi:10.1080/02602938.2021.2001430.
   Web of Science ®Google Scholar