A novel approach to assess the impact of changes in admission criteria

Dear Sir

The very competitive admission to medical schools depends on the criteria adopted by the schools. Evaluation of the impact of a change in the admission criterion is limited by the “impossible problems of predictive validity studies” described in the “Consensus statement and recommendations from the Ottawa 2010 Conference” (Prideaux et al. 2011). In view of this limitation, commonly attributed to the homogeneity of the applicants, the latter report recommends developing novel tools for “predictive validity studies”. In this letter, I propose a novel approach to assess the implications of a change in the admission criterion due to including non-cognitive traits.

The direct consequence of a change in the admission criteria is that several applicants are admitted thanks to the change, instead of applicants with slightly higher grades that have been rejected because of the change. These two groups can be easily identified by comparing two lists of the candidates, both arranged in decreasing order of grades, one according to the old criterion and the other according to the new criterion. The information on the applicants is sufficient to reflect the demographic outcome of the change. The effect of the change on the academic and professional outcome cannot be evaluated on the basis of the same approach because we have no information on the achievements of those rejected candidates that would have been admitted according to the old criterion.

Yet, an approximation can be achieved by a “semi-direct” approach based on analysis of a 20% reduced class. Comparison of two lists of the class, both arranged in decreasing order of grades, one according to the old criterion and the other according to the new criterion, enables identification of both those students who would have been admitted to “the smaller class” according to the old criterion but rejected because of the change, and of those who would have been accepted thanks to the change. Comparing the records of the latter two groups can be used to evaluate the outcome of interest (grades, evaluations and alike) of changing the admission process.

We continue following the performance of those students that would have been affected by the change of the admission criteria and will report on the results in due time.