Twelve tips to aid interpretation of post-assessment psychometric reports

Abstract

Post-assessments psychometric reports are a vital component of the assessment cycle to ensure that assessments are reliable, valid and fair to make appropriate pass-fail decisions. Students’ scores can be summarised by examination of frequency distributions, central tendency measures and dispersion measures. Item discrimination indicies to assess the quality of items, and distractors that differentiate between students achieving or not achieving the learning outcomes are key. Estimating individual item reliability and item validity indices can maximise test-score reliability and validity. Test accuracy can be evaluated by assessing test reliability, consistency and validity and standard error of measurement can be used to measure the variation. Standard setting, even by experts, may be unreliable and reality checks such as the Hofstee method, P values and correlation analysis can improve validity. The Rasch model of student ability and item difficulty assists in modifying assessment questions, pinpointing areas for additional instruction. We propose 12 tips to support test developers in interpreting structured psychometric reports, including analysis and refinement of flawed items and ensuring fair assessments with accurate and defensible marks.

Keywords:

• Educational measurement; post-examination analysis; psychometric report

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Funding

The authors reported there is no funding associated with the work featured in this article.

Notes on contributors

Mohsen Tavakol

Mohsen Tavakol, MSc PhD, MClinEd, is an associated professor in Psychometrics. His main interests are in medical education assessment, assessment feedback, psychometric analysis (Classical Test Theory, Generalisability theory, Item Response Theory Models), robust statistical methods, multivariate statistics, Structural Equation Modelling, Meta-analysis, quantitative and qualitative research methods and communication skills.

David G. O’Brien

David G. O’Brien, MBChB MMedSci MD FHEA FRCP, is Professor of Medical Education, School of Medicine, University of Nottingham, Clinical Vice Dean and Director of Admissions for the Lincoln Medical School and an Honorary Consultant Interventional Cardiologist. His research interests include psychometrics of assessment, medical school selection and also widening access and participation to Medicine. Twitter@ViceDeanLMS

Claire C. Sharpe

Claire C. Sharpe, MBBS, BSC, PhD, PGCert MedEd, is the Dean of Education and Professor of Renal Medicine and Medical Education in the University of Nottingham School of Medicine. Her interests are in innovations in medical curricula and progressive ways of designing and aligning high quality assessments to teaching.

Claire Stewart

Claire Stewart, BMedSci (Hons), FRCGP MSc, MedEd, FAcadMEd, SFHEA, is the Dean and Head of the School of Medicine at the University of Nottingham and a Professor in Medical Education and Assessment. Claire is a keen medical educationalist, specialising in assessment. Her main interests are improving the reliability and fairness of assessment and digital assessments to ensure that assessment can aid the educational process for all stakeholders.