An explanatory mixed methods study on the validity and validation of students’ assessment results in the undergraduate surgery course

Abstract

Background/purpose: There is inadequate evidence of reported validity of the results of assessment instruments used to assess clinical competence. This study aimed at combining multiple lines of quantitative and qualitative evidence to support interpretation and use of assessment results.

Method: This study is a mixed methods explanatory research set in two stages of data collection and analysis (QUAN : qual). Guided by Messick’s conceptual model, quantitative evidences as reliability and correlation coefficients of various validity components were calculated using students’ scores, grades and success rates of the whole population of students in 2012/2013 and 2013/2014 (n= 383; 326). The underlying values that scaffold validity evidences were identified via Focus Group Discussions (FGD) with faculty and students; sampling technique was purposive; and results were analyzed by content analysis.

Results: (1) Themes that resulted from content analysis aligned with quantitative evidences. (2) Assessment results showed: (a) content validity (table of specifications and blueprinting in another study); (b) consequential validity (positive unintended consequences resulted from new assessment approach); (c) relationships to other variables [a statistically significant correlation among various assessment methods; with combined score (0.64–0.86) and between mid and final exam results (r = 0.672)]; (d) internal consistency (high reliability of MCQ and OSCE: 0.81, 0.80); (3) success rates and grades distribution alone could not provide evidence to advocate an argument on validity of results.

Conclusion: The unified approach pursued in this study created a strong evidential basis for meaningful interpretation of assessment scores that could be applied in clinical assessments.

Acknowledgments

The authors very much appreciate the Head of Surgery Department, Doctor Hussein Jabbad, for his commitment to provide us with the required data. The Saudi Supplement on Medical Education is sponsored and supported by Al Imam Mohammed Ibn Saud Islamic University.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Omayma Hamed

Omayma Hamed, MBBCh, MSc, MD, MHPE, FAIMER: Professor in Medical Education Department; Senior Consultant for Assessment Unit; Chairman, Quality & Academic Accreditation Unit; Member in Faculty Main Assessment Committee; Member in Committee for Development of Academic Curricula, Faculty of Medicine, King Abdulaziz University (KAU).

Husain Hamza Jabbad

Husain Hamza Jabbad, FRCSc, Cardiovascular and Thoracic Surgery. Former Head of Surgery Department, Faculty of Medicine, KAU.

Omar I. Saadah

Omar I. Saadah, MBBS, CABP, MRCP, DCH: Vice Dean for Development; Head of Faculty Main Assessment Committee; Member of Committee for Development of Academic Curricula, Faculty of Medicine, KAU.

Mahmoud S. Al Ahwal

Mahmoud S. Al Ahwal, MBChB, ABIM, FRCPC: Diplomate American Board Internat Medicine Alberta, Canada, 1991. Fellow: Royal College Physicians & Surgeons, Canada: Dean Faculty of Medicine, KAU.

Fatin M. Al-Sayes

Fatin M. Al-Sayes, MSc, FRCPI, FRCpath: Vice dean for Faculty of Medicine Female Campus; Member in Faculty Main Assessment Committee; Coordinator of Committee for Development of Academic Curricula, Faculty of Medicine, KAU.