Predictive validity of the personal qualities assessment for selection of medical students in Scotland

Abstract

Background: The Personal Qualities Assessment (PQA) was developed to enhance medical student selection by measuring a range of non-cognitive attributes in the applicants to medical school. Applicants to the five Scottish medical schools were invited to pilot the test in 2001 and 2002.

Aims: To evaluate the predictive validity of PQA for selecting medical students.

Methods: A longitudinal cohort study was conducted in which PQA scores were compared with senior year medical school performance.

Results: Consent to access performance markers was obtained from 626 students (61.6% of 1017 entrants in 2002–2003). Linkable Foundation Year (4th) rankings were available for 411 (66%) students and objective structured clinical examination (OSCE) rankings for 335 (54%) of those consenting.

Both samples were representative of the original cohort. No significant correlations were detected between separate elements of the PQA assessment and student performance. However, using the algorithm advocated by Powis et al. those defined as ‘non-extreme’ (<±1.5 SD from the cohort mean scores; SD, standard deviation) character types on the involved-detached and on the libertarian–communitarian moral orientation scales were ranked higher in OSCEs (average of 7.5% or 25 out of 335, p = 0.049).

Conclusions: This study was limited by high attrition and basic outcome markers which are insensitive to relevant non-cognitive characteristics. However, it is the largest currently available study of predictive validity for the PQA assessment. There was one finding of significance: that those students who were identified by PQA as ‘not extreme’ on the two personal characteristics scales performed better in an OSCE measure of professionalism. Futures studies are required since psychometric testing for both cognitive and non-cognitive attributes are increasingly used in admission process and these should include more and better measures of professionalism against which to correlate non-cognitive traits.

Background

Medical schools are challenged with the task of selecting applicants who will not only excel academically but also possess personal qualities befitting a career in medicine. In ‘Tomorrow's Doctors,’ the UK General Medical Council (2009) stipulates that doctors are required as professionals, to be competent, empathic and ethical practitioners. Accordingly, many medical schools are seeking new ways of taking into account a range of personal characteristics in addition to academic achievement and potential in their selection procedure. The UK Clinical Aptitude Test (UKCAT) is currently piloting a ‘non-cognitive’ section which seeks to measure a range of non-academic personal characteristics (including a modified version of the Personal Qualities Assessment, PQA). While applicants’ cognitive abilities can reliably be measured using academic grades at the point of application (in the UK context, both GCSE and A level results have established predictive validity), selection on the basis of personal characteristics remains unproven (McManus et al. 2005). It would be very helpful if it could be shown that non-cognitive attributes assessed through self-report questionnaires have predictive validity for clinical professions such as medicine.

Table

Practice points

Medical school admissions remain an imprecise process with considerable variation and limited evidence. Non-cognitive measures are widely acknowledged to be important but complex to measure, evaluate and use. The Personal Qualities Assessment (PQA) tests were developed specifically for health care profession and are now being piloted widely through the UKCAT assessment and used in some centres in Australia and several other countries. It is important to assess the potential of such measures but this requires detailed longitudinal studies which are rarely funded. This study reports equivocal findings regarding the PQA tests. ‘Predictive validity of the Personal Qualities Assessment for selection of medical students in Scotland’.

In the 2001/2002 and 2002/2003 admissions cycles, two cohorts of school applicants to the four Scottish Medical Schools were invited to sit the PQA tests (Munro et al. 2005). This is a battery of psychometric tests developed at Newcastle University in Australia (for further information, see www.pqa.net.au). This article reports on two instruments: one tailored to measure applicants’ moral orientation in terms of being Libertarian versus Communitarian (LibCom), and the other assessed the four traits Narcissism, Aloofness, Empathy and Confidence (in dealing with people), hence ‘NACE’. It also provides a combined score known as ‘ECAN’ in which scores for empathy and confidence are scored positively and offset by scores for narcissism and aloofness which are scored negatively (Lumsden et al. 2005).

A higher LibCom score indicates that the respondent gives greater value to the needs, rights, laws and norms of society rather than the individual (Communitarian). In contrast, a low LibCom score indicates that the respondent gives greater value to the needs, rights and well-being of individuals (Libertarian; Bore et al. 2005a, b). A high ECAN score indicates a greater tendency to be involved with others (high Empathy and Confidence) and a low score greater detachment from others (high Narcissism and Aloofness; Munro et al. 2005). While some evidence exists for the validity of the PQA tests in relation to other personality markers and medical student behaviours (Bore et al. 2005a, b; Munro et al. 2005; Adam et al. submitted), this article seeks to establish evidence of their predictive validity in terms of medical school performance, including simulated clinical exams such as Objective Structured Clinical Examinations (OSCE; Table 1).

Table 1.  Showing the Pearson correlation with LibCom, ECAN and foundation year and OSCE rankings

	N	No. of items	Mean	SD	Alpha
LibCom	12,245	49	116	15.4	0.90
Narcissism	9301	24	53	8.6	0.84
Aloofness	9301	24	46	8.0	0.86
Confidence	9301	24	70	8.6	0.85
Empathy	9301	24	74	7.0	0.82
ECAN	9301	96	285	22.5	0.92

Methods

We sought comparable performance data from the four Scottish clinical medical schools to match with PQA scores gathered prior to admission. Consent from students was sought both soon after entry to medical school and through an online survey conducted in final year prior to graduation in 2007 and 2008. Students providing consent, whose data could be matched with certainty (school, name and one other ID such as DoB), and who entered in 2002 or 2003 and graduated in 2007 or 2008, formed the study cohort. After an analysis revealed significant differences in the assessment systems, each school was asked to provide a ‘rank-within-year’ for these graduates based upon written (knowledge-based) and OSCE (performance-based) achievements in years 4 or 5, as appropriate for the school's clinical assessment system. In addition, ranks were available for all students after 4th year as part of the standard UK allocation system for junior (Foundation) hospital posts, based upon each university's weighted system that includes some recognition of work done in the earlier years; this provided a single comparable estimate of rank within year group. To cater for differing year group sizes within respective medical schools, an individual's rank within his/her year group was converted into a percentile ranking so that direct comparisons were possible between those at different institutions. This approach effectively assumes that the cohorts at different schools were similar at entry and that students ranked top or bottom at each school were comparable.

Unfortunately, because our follow-up was retrospective, we could not determine reasons for study drop-out, which could include both good reasons (e.g. intercalated degrees) as well as bad (academic failure or health/professional issues). Thus, we are unable to comment on progression rates.

Comparisons were made between the NACE subscale scores, ECAN and LibCom measures and the overall Foundation and OSCE rankings (though data provided by some schools did not allow us to rank their students within each exam, numbers are reduced for the comparison based on OSCE performance). A further comparison was based upon whether students’ PQA scores put them into one of the ‘moral types’ or groups of outliers (more than 1.5 SDs from the cohort norm on either the Libertarian–Communitarian or the Involved-Detached dimension) proposed by Lumsden et al. in 2005 to see if these performed differently from those in the central or ‘normal’ ranges on both these variables (Lumsden et al. 2005; Powis et al. 2005).

Results

Participants

Of the original pool of 1017 (63% female) medical school entrants from whom PQA data were gathered in 2001 and 2002, we secured consent from 626 to seek their subsequent performance data. Data from one school were not released, meaning foundation year ranking data, ECAN and LibCom scores, were available on only 411 of these 626 students. Of these, 284 (69%) were female; 224 (54.5 %) attended Glasgow, 113 (27.5%) Dundee and 74 (18%) Aberdeen. OSCE performance data (that aims to test practical and professional skills rather than knowledge) were available for 335 of these students.

Analysis

Data were analysed with SAS version 9.1. Kolmogorov–Smirnov tests were used to compare the two study samples (411 and 335) with the original cohort of 1017. No significant differences were found in terms of gender, LibCom or ECAN scores; so it appeared that our sample was representative of the original study group.

Pearson Coefficients were used to assess the correlation between performance within the student's study group and LibCom or ECAN scores or any of the NACE subscale scores. No correlations were found (Table 2).

Table 2.  Descriptive statistics (norms) and reliability of LibCom and NACE and its subsets (unpublished data from PQA accumulated database)

Foundation year rankings and PQA variables (n = 411)
	Combined year 4 rankings	LibCom	Narcissism	Aloofness	Confidence	Empathy	ECAN
Combined year 4 rankings	1.000	0.009	0.080	0.009	−0.043	0.040	−0.036
p-Value		0.851	0.104	0.858	0.381	0.421	0.467
OSCE rankings and PQA variables (n = 335)
	OSCE percentile	LibCom	Narcissism	Aloofness	Confidence	Empathy	ECAN
OSCE percentile	1.000	−0.065	−0.085	−0.011	−0.046	−0.042	0.004
p-Value		0.232	0.119	0.842	0.400	0.444	0.948

Using the schema described above, we compared students by ‘moral type’. A total of 313 (76%) of the applicants fell within the central square, whose limits are set at ±1.5 SD of the sample mean score standardized ‘Z’ scores; 26% were defined as ‘extreme’ (Figure 1).

Figure 1. Scatter plot showing moral types (n = 411). Axes show ‘Z’ scores: standardized mean = 0, SD = 1.

The Mann–Whitney test comparing Foundation Year ranking between extreme and non-extreme students demonstrated no difference (p = 0.829). However, observing the performance in the final OSCE Examination, it was found that non-extreme applicants performed slightly better, with an average rank 7.5% higher (or 25 out of 335, p = 0.049).

Discussion

This study represents the largest cohort and most prolonged follow-up available for the Personal Qualities Assessment to date. However, it must be recognized that it has a number of weaknesses. First, there was a disappointing follow-up rate due to a combination of factors. Consent to follow-up was not obtained at the time of sitting the assessments but had to be sought retrospectively after the Data Protection Act came into force. There were also identification discrepancies, students leaving their original year groups for a number of reasons (unknown) and the reluctance of one school to release performance data. In addition, it is not possible to explore reasons for attrition as these data could not be obtained retrospectively without consent. These issues are not unique but highlight the challenges of multicentre long-term follow-up studies without core funding.

It was clear from the outset that the outcome measures available would probably be insensitive to the non-cognitive attributes which are the focus of the PQA. However, the assessments used were reliable, discriminated effectively between students and are known, in general terms, to measure qualities relevant to postgraduate performance. Although knowledge-based achievements might not be expected to correlate with non-cognitive tests, an association with OSCE performance would be of great interest.

Despite these reservations, it is important to determine if performance in these broad areas is associated with global PQA scores or individual elements thereof. The positive relationship found in one area thus bears closer scrutiny because the UKCAT is now also testing a very similar set of scales on a far wider basis. It is difficult to know what weight can be put on the link between better OSCE performance and those classified as ‘non-extreme’ according to the combined LibCom and ECAN scales but this does relate to a novel ‘comprehensive approach’ proposed by Bore et al. (2009). Clearly, further work is required in this area with close attention to course attrition rates and reasons, and establishing summative measures of professionalism within medical school assessment procedures.

In summary, this study aimed to establish the predictive validity of the PQA assessments using 5-year follow-up in Scottish medical schools and, while identifying one relationship with outcome, it has not otherwise found a clear link. However, there are a number of reasons why this might be the case and this study has provided some insight. It is important that further predictive validity studies, perhaps focused on the UKCAT test instruments, are performed before such tests become used for selection. This will require more sophisticated follow-up techniques and appropriate outcome markers where these are available (Cook et al. 2010).

Acknowledgements

The authors are grateful to the Scottish Board for Academic Medicine for funding this study and the Scottish Medical Admissions Working Group for assisting with data collection.

Declaration of interest: David Powis, Miles Bore and Don Munro have developed the instrument being evaluated.