The UK Clinical Aptitude Test: Is it a fair test for selecting medical students?

Abstract

Background: The United Kingdom Clinical Aptitude Test (UKCAT) is designed to increase diversity and fairness in selection to study medicine.

Aim: The aim of this study is to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and thereby unfairly advantage some candidates over others.

Methods: Confidential, self-completed, on-line questionnaire of applicants to study on an undergraduate medical degree course who had taken the UKCAT in 2010.

Results: Differentials in access to support and advice, in modes of preparation, type of school/college attended, in level of achievement in mathematics, gender and age were found to be associated with candidate performance in the UKCAT.

Conclusion: The findings imply that the UKCAT may disadvantage some candidate groups. This inequity would likely be improved if tutors and career advisors in schools and colleges were more informed about the UKCAT and able to offer appropriate advice on preparation for the test.

Background

Introduced in 2006, the United Kingdom Clinical Aptitude Test (UKCAT) is designed to complement and improve upon existing medical school selection tools (Turner & Nicholson 2011). The test assesses cognitive ability in four domains: verbal reasoning (VR), quantitative reasoning (QR), abstract reasoning (AR) and decision analysis (DA; UK Consortium 2008a). It assesses cognitive ability independent of academic knowledge in order to broaden the criteria for selection beyond the abilities already assessed by A-level examinations (James et al. 2010). A-levels are the final school leaving examinations usually sat at 18 years of age in England, the results of which normally govern the allocation of university places.

Internationally, standardised admission tests, such as the Medical College Admissions Test (Association of American Medical Colleges 2011), the Graduate Australian Medical School Admissions Test (Australian Council for Educational Research) and the Undergraduate Medicine and Health Sciences Admissions Test (Australian Council for Educational Research 2011), are widely used. However, their added value to the selection process is yet to be clearly established (Callahan et al. 2010; Prideaux & Roberts 2011).

Table

Practice points

Selection to study medicine should be fair. Aptitude tests aim to improve on existing selection tools and widen access to the study of medicine. This study reports that differentials in access to information about the UKCAT and advice on preparation influence candidate performance in the test and undermine its aim to widen participation.

Studies of the UKCAT to date have mainly focussed on its predictive validity, which remains controversial (Lynch et al. 2009; Yates & James 2010), and evidence of its ability to widen access to medicine is contradictory. One study has found that candidate gender and type of school attended are not predictive of UKCAT score (Wright & Bradley 2010). In contrast, another has found that the test's intention to increase diversity and fairness by enabling selection on ‘aptitude rather than solely on academic achievement’ is undermined by bias in favour of males, candidates from higher socio-economic classes and those from independent or grammar schools (James et al. 2010). Thus the UKCAT may be exacerbating, rather than ameliorating, the school background and social class selection bias of A-levels (McManus et al. 2005). Furthermore, in the UK, concern over differentials in access to familiarisation and practise sessions has raised doubts about the ability of aptitude tests in general to widen participation in Higher Education (Supporting Professionalism in Admissions 2010).

To our knowledge, nothing is known about whether candidates’ preparation for the UKCAT influences their performance. The aim of this study is to determine if differences in: access to support and advice, in modes of preparation, type of school/college attended, level of achievement in mathematics, gender and age influence candidate performance in the UKCAT and thereby unfairly advantage some candidates over others.

Methods

The questionnaire

The questionnaire comprised four sections with 23 questions in total: (1) finding out about the UKCAT, (2) preparation for the UKCAT, (3) opinions about the UKCAT and (4) demographics. Respondents self-declared their UKCAT scores. The sample frame comprised one applicant cohort to a Bachelor of Medicine, Bachelor of Surgery undergraduate course at the Peninsula College of Medicine and Dentistry in the UK. The aims of the on-line survey study and assurance of confidentiality was explained in both an initial and non-response follow-up email. Refinement of the questionnaire was informed by a pilot study with first-year students at this medical school. The questionnaire was completed between November and December 2010 and can be accessed at http://www.pcmd.ac.uk/downloads/ukcat_survey.pdf.

Statistical analysis

Multivariate regression analysis. The software Stata10 (Stata Corp, Statistical Software Release 10.0, College Station, TX, USA) was used for all analyses. Candidates can score a maximum of 3600; potentially 900 in each of the UKCAT's four subsections. Analyses were conducted using the following models:

Multivariate linear regression, outcome scores in: (1) total UKCAT, (2) VR, (3) QR, (4) AR and (5) DA. In each individual regression model of a subsection score, the remaining three subsection scores were included as covariates.

Where appropriate, the tables reporting the results of the regression analyses include the effect size of significant predictors. Effect size quantifies the difference between two groups, in this case the baseline group's score and a comparison group. For instance, an effect size of 0.6 means that compared to the average score of those in the baseline group, the average score of the comparator group is 0.6 standard deviations greater (Coe 2002).

The categorical predictors in the five models were:

1. preparation time (0–5 hours, 5–20 hours and 20+ hours),

2. preparation resource used (only commercial practise resources, only the UKCAT on-line practise test and both types of practise resource),

3. mathematics studied beyond the General Certificate of Secondary Education (GCSE) level (yes versus no),

4. type of school attended (Comprehensive, Grammar no fees, Grammar fee-paying, Public/Independent, Sixth Form College and Further Education College),

5. gender,

6. age (age ≤ 19 years versus age ≥ 20 years).

Ethics

The ethical aspects of this study were approved by the Peninsula College of Medicine and Dentistry Research Ethics Committee.

Results

The response rate was good (66%), with 787 (49% male:51% female) of the 1188 applicants to the course returning completed on-line questionnaires. Sample mean scores are: total UKCAT 2582 (SD 238), VR 598 (SD 78), QR 692 (SD 84), AR 645 (SD 81) and DA 648 (SD 5).

Advice and support

The survey results revealed that only 30% of respondents first heard about the UKCAT from a tutor, teacher or career advisor at the school or college they attended. The remainder first heard from a friend, a family member, medical prospectus or an on-line source.

Around 69% of respondents reported that they had received advice to prepare for the test, and had done so from two distinct sources – those advised to prepare for the test by a tutor, teacher or career advisor at the school or college they attended, and those advised by a friend, family, medical school prospectus or on-line forum, but not by a tutor/teacher or career advisor. The proportion advised to prepare by a tutor, teacher or career advisor varied between types of schools and colleges in the educational sector (Figure 1). Furthermore, less than half of the respondents, 49%, considered that their school or college was fully informed about the UKCAT; however, the proportion who did so also varied considerably between types of school and college (Figure 1). Around 51% of respondents reported that they were not aware of what constituted a threshold UKCAT score for entry to medicine. However, 63% considered that preparation enabled them to score more highly in the UKCAT.

Figure 1. (a) Bar graph of the percentage of respondents who were advised to prepare for the test by a tutor, teacher or career advisor contrasted by the type of school/college a respondent attended. (b) Bar graph of the percentage of respondents who considered their school or college was fully informed about the UKCAT contrasted by the type of school/college a candidate attended. Note: School/college type attended: SFC = Sixth Form College and FEC Further Education College.

Multivariate regression

Approximately 14% of the variance in total UKCAT score, 25% in VR score, 33% in QR score, 22% in AR score and 22% in DA scores was accounted for by the variables in the respective ordinary least-squares regression models.

There was wide variation in the time respondents spent preparing for the UKCAT. 29% prepared for 0–5 hours, 50% for 5–20 hours and 21% prepared for over 20 hours. Compared to the baseline of 0–5 hours, preparation times of 5–20 hours (p < 0.05) and 20+ hours (p < 0.05) were associated with higher mean scores, 43 and 68 respectively, for total UKCAT (Table 1). Preparation time was not found to be an independent predictor of performance in the subsections of the test (Tables 2–5).

Table 1.  Independent predictors of respondent performance on total UKCAT score, model fit statistics total UKCAT score (F (10, 556)) = 8.18 p = 0.0000), R² = 0.14

	Mean score difference^#	SE	t	p > t	95% Confidence interval		Effect size	95% Confidence interval
Preparation for the UKCAT
Prepared for 5–20 hrs	43	23.3	1.9	*	2	89	0.2	0.05–0.39
Prepared for 20+hrs	68	30.2	2.3	*	9	127	0.2	0.02–0.45
Used UKCAT + commercial resources	56	20.8	2.7	**	15	96	0.4	0.22–0.64
Mathematics studied beyond GCSE	60	22.1	2.7	**	16	103	0.6	0.38–0.82
Educational institution
Last attended comprehensive school	97	27.5	3.5	***	43	151	0.5	0.23–0.77
Last attended Grammar (no-fees) school	116	25.4	4.7	***	67	165	0.6	0.36–0.84
Last attended Grammar (fees) school	131	53.0	2.5	*	27	214	0.7	0.21–1.20
Last attended Public/Independent school	65	24.7	2.6	**	17	113	0.4	0.16–0.64
Last attended Further Education College	–68	43.1	–1.6	0.11	–153	15
Last attended other type of school	65	67.5	0.9	0.33	–66	198
Gender and age
Male	34	17.8	2.0	*	0.2	70	0.2	0.03–0.37
Age-group ≥ 20 years of age	–134	38.9	–3.4	**	–210	–58	0.8	0.61–1.19
Constant	2445	33.5	72.8	***	2379	2511	–	–

Notes: ^#The coefficient is the difference in the mean total UKCAT score compared to the reference group. Reference groups: prepared for 0–5 hours; used commercial resource or UKCAT on-line practise test only; no mathematics studied beyond GCSE level; educational background reference group was Sixth Form College; female was gender reference group; aged 19 years and under was the age-group reference group. Effect size calculated as mean total UKCAT score difference between predictor and its baseline category divided by pooled estimate of SDs of both groups.

Statistically significant differences are indicated by * = p < 0.05, ** = p < 0.01 and *** = p < 0.001, in column p > t.

Table 2.  Independent predictors of respondent performance in the VR subsection of the UKCAT, model fit statistics (F (13, 552)) = 14.24 p = 0.0000), R² = 0.25]

Subsection scores	Mean score difference^#	SE	t	p > t	95% Confidence interval
QR	0.32	0.05	6.76	***	0.22	0.41
AR	0.11	0.04	2.70	**	0.03	0.20
DA	0.12	0.03	3.23	**	0.04	0.19
Constant	227	36.52	6.21	***	155.11	298.58

Note: Refer notes of Table 1.

Table 3.  Independent predictors of respondent performance in the QR subsection of the UKCAT, model fit statistics (F (13, 552)) = 21.51 p = 0.0000), R² = 0.33

Subsection scores	Mean score difference^#	SE	t	p > t	95% Confidence interval		Effect size	95% Confidence interval
VR	0.33	0.05	6.41	***	0.23	0.44
AR	0.11	0.03	2.88	**	0.03	0.19
DA	0.13	0.04	3.86	***	0.07	0.21
Last attended Grammar (no-fees) school	38	7.64	4.97	***	23.01	53.04	0.5	0.08–1.08
Mathematics studied beyond GCSE	32	7.52	4.31	***	17.63	47.20	0.6	0.20–1.00
Male	19	5.82	3.22	**	7.32	30.19	0.3	0.13–0.47
Constant	271	34.54	7.85	***	203.48	339.19

Notes: Refer notes of Table 1.

Table 4.  Independent predictors of respondent performance in the AR subsection of the UKCAT, model fit statistics (F (13, 552)) = 12.06 p = 0.0000), R² = 0.22

Subsection scores	Mean score difference^#	SE	t	p > t	95% Confidence interval		Effect size	95% Confidence interval
VR	0.13	0.04	2.70	**	0.03	0.22
QR	0.12	0.04	2.81	**	0.04	0.21
DA	0.15	0.04	4.19	***	0.08	0.23
Used UKCAT + commercial resources	24	6.23	3.87	***	11.87	36.38	0.4	0.21–0.6
Age-group ≥ 20 years	–23	11.9	–1.9	*	–46.89	–0.32	0.6	0.32–0.87
Last attended Grammar (fee-paying) school	48	17.73	2.73	**	13.54	83.21	0.8	0.66–0.90
Constant	355	35.67	9.95	***	284.98	425.12

Notes: Refer notes of Table 1.

Table 5.  Independent predictors of respondent performance in the DA subsection of the UKCAT, model fit statistics (F (13, 552)) = 11.84 p = 0.0000), R² = 0.22

Subsection scores	Mean score difference^#	SE	t	p > t	95% Confidence Interval		Effect size	95% Confidence interval
VR	0.17	0.05	3.32	**	0.07	0.28
QR	0.20	0.05	3.94	***	0.10	0.30
AR	0.20	0.05	4.02	***	0.10	0.30
Age-group ≥ 20 years	–26	12.03	–2.1	*	–49.36	–2.09	1.6	0.66–2.54
Constant	254	41.79	6.09	***	172.50	336.69

Notes: Refer notes of Table 1.

Only 7% of respondents reported that they had attended preparation courses at their school or college and 9% commercially available preparation courses. We did not find any significant differences in UKCAT scores between those who had been ‘coached’ and those who had not and thus excluded this predictor from the regression analyses.

A large proportion of respondents, 76%, had used the UKCAT provided on-line practise test. Respondents were asked if they had used any commercially available resources to prepare for the test. The major resources identified were – specialist books, on-line practise tests and UKCAT preparation courses. Three groups of resource usage were identified; those who had used the UKCAT on-line practise test as well as commercial practise resources (68%), those who had only used commercial practise resources (23%) and those who had only used the UKCAT on-line practise test (9%). The group of respondents who prepared for the test using two resource types was associated with higher mean scores; 56 points for total UKCAT (p < 0.01; Table 1) and 24 points for AR (p < 0.001; Table 4) than the group which used only commercial resources or used only the UKCAT on-line practise test.

The majority of respondents, 79%, had taken mathematics beyond GCSE level (UK examinations usually taken at 16 years of age). Those who had studied mathematics beyond GCSE level scored on average 60 points higher for total UKCAT (p < 0.01; Table 1) and 32 points higher for QR (p < 0.001) than those who had not continued studying mathematics. An effect size statistics of 0.6 indicates the impact of the study of mathematics beyond GCSE level on these scores (Tables 1 and 3).

There were significant differences in the mean total UKCAT scores of respondents depending on the school/college attended. Those who attended Grammar (non-fee paying and fee-paying), Comprehensive and Public/Independent schools outperformed those who attended Sixth Form Colleges in mean total UKCAT scores (Table 1). The respective effect size statistics illustrated the impact of type of school/college attended on these scores. For instance, an effect size of 0.6 and 0.7, respectively, for fee-paying and non-fee paying Grammar schools in respect of total UKCAT score with a predicted difference of 116 and 131 points, respectively, above the average score of those who attended a Sixth Form College (Table 1). Respondents who had attended a Grammar school (no-fees) scored on average 38 points higher for QR (p < 0.001; Table 3), and those who had attended a Grammar school (fee-paying) on average 48 points higher for AR (p < 0.01) than those who attended Sixth Form Colleges (Table 4).

Male respondents scored significantly higher in mean total UKCAT (p < 0.05; Table 1) and the QR (p < 0.01; Table 3) subsection of the UKCAT, 34 and 19 points, respectively. However, gender had only an effect size of 0.2 on these scores.

Age was also a significant predictor of performance in the UKCAT. Respondents aged 20 years or above scored significantly lower for the mean total UKCAT (p < 0.05; Table 1) and for scores in AR (p < 0.05; Table 4) and DA (p < 0.05; Table 5) than those aged 19 years or younger. Indeed, the effect size statistics were 0.8, 0.6 and 1.6, respectively.

We included, individually, all interaction terms between each of the predictors in the above-mentioned multivariate regression models. However, we only found a statistically significant interaction between the time a respondent had spent in preparation and the type of preparation resources a respondent had used. The interaction had an effect on total UKCAT score (F (12, 550) = 13.68, p < 0.0001) but not on performance in the subsections of the test. Total UKCAT score was shown to increase with increased practise time. However, at all levels of practise time, those respondents who had used both the UKCAT on-line practise resource and commercially available practise resources outperformed those who solely used either of them (Figure 2).

Figure 2. The interaction effect between preparation time and type of resource used on total UKCAT score. Notes: UKCAT or commercial defines respondents who used either solely the UKCAT on-line practise test or solely commercial practise resources in their preparation for the test. Both define respondents who use both the UKCAT on-line practise test and commercial practise resources in preparation for the test. Fitted values are the values predicted by the regression model including the interaction term.

Opinions about the UKCAT

The majority of respondents, 86%, reported that they agreed with the statement that you can prepare for the UKCAT, 44% agreed that advice on the UKCAT was confusing, 55% agreed that the test was fair and 44% agreed with the statement that the UKCAT measures attributes relevant for the study of medicine.

Discussion

We believe this is the first survey to determine if advice, support, an awareness about the requirements of the test, differentials in preparation time and the use of preparation resources, together with the type of school/college attended, age and gender, have effects on candidate performance in a medical school standardised admissions test.

Finding out about the UKCAT

The results show that respondents who first heard about the UKCAT from a tutor, teacher or career advisor and were advised by them to prepare for the test, and who considered that their school or college was fully informed about the UKCAT, outperformed corresponding respondent groups. However, there was considerable variation between types of school and college in the proportion of respondents who felt that their school/college was fully informed about the UKCAT.

This is consistent with the conclusion reached by the British Medical Association, that a ‘lack of guidance in applying to medicine is not … a problem found exclusively among lower socio-economic groups’ (British Medical Association 2009). These findings also agree with those of the UKCAT Consortium that 40% of candidates felt that their school or college was not well informed about the test and that UK medical schools should provide more information about how the test is used (UK Consortium 2010). These findings align with those of the recent Independent Review of Higher Education Funding and Student Finance, which recommended that all schools be required to make individualised careers advice available to all pupils provided by well-informed career professionals (Browne 2010).

Preparation for the test

We found that only a small number of respondents had attended specific preparation courses and thus could be described as having been coached. This agrees with the findings of the 2009 candidate survey conducted by the UK Consortium (UK Consortium 2010). We found no association between attendance of a preparation course and performance in the UKCAT, in agreement with the findings of Sackett that coaching ‘is not a major determinant of test performance’ (Sackett et al. 2008).

However, the results of this survey show that those respondents who put in more preparation time and used both commercial resources and the official UKCAT on-line practise test performed much better than those who did not. This finding is consistent with research from elsewhere, which has showed that practise improves scores on aptitude and achievement tests (Kulik et al. 1984).

Educational background

We found the type of school attended to be a significant predictor of performance in the UKCAT. Although this is consistent with the work of James et al. (2010) who have shown private sector and grammar schooling to be predictive of scores in the test, this study's more nuanced analysis specifically revealed that Sixth Form or Further Education College candidates were predicted to perform markedly less well in the test. Given that 19% of UK domiciled applicants to study medicine in 2010 were from a Sixth Form or Further Education College further analysis of this phenomenon is warranted. A potential consequence is the observation that Sixth Form and Further Education Colleges have the lowest percentage of applicants and rate of acceptance to study medicine among categories of the UK educational sector (Universities and Colleges Admission Service 2010).

Whilst we acknowledge that there are marked differences between educational sectors in A-level performance, there is little variance in the Universities and Colleges Admissions Service (UCAS) tariff scores of those subset of UCAS applicants who apply to study medicine (Universities and Colleges Admission Service 2010). Therefore, it would seem reasonable to suggest that differences in UKCAT performance are not entirely due to educational sector differences in candidates’ academic attributes, but also likely related to other factors. Indeed, in respect of applications to Higher Education generally, research is being conducted to ‘develop understanding of the positive interactions and practice that make up a good applicant experience’ because it is believed that a poor experience ‘perpetuates barriers to entry [and] disengages potential applicants’ (Supporting Professionalism in Admissions 2010).

Level of achievement in mathematics

Another differential finding here was that performance in the UKCAT was improved if mathematics had been studied beyond the GCSE level (post 16 years of age). This finding was unsurprising given that those respondents who had chosen to study mathematics beyond GCSE level as one of their A-level subjects were likely to have a better aptitude for QR, and were possibly advantaged because they continued to practise their mathematical skills more regularly, compared to those who did not study mathematics beyond the GCSE level.

It may be prudent for the UKCAT Board to advise candidates of the potential benefits of both practise and familiarisation with the specific mathematical skills and concepts required for the test. This is especially recommended for those candidates who only study mathematics to the GCSE level.

An important implication of the effect of the level of mathematics studied by a candidate on UKCAT performance is that the UKCAT may not be a pure aptitude test but instead act as an indirect indicator of A-level achievement. This finding undermines the test's intention to assess cognitive ability independent of academic knowledge in order to broaden the criteria for selection beyond the abilities already assessed by A-level examinations. The finding is in agreement with that of a recent study, which found that the UKCAT provides ‘a reasonable proxy for A levels in the selection process’ (James et al. 2010).

Gender

We found gender to be a statistically significant predictor of overall candidate performance in the UKCAT, and the QR subsection score, with males outperforming females. This agrees with the work of James et al. who found males performed better than females for the whole test, with the largest gender differential in performance in the QR subsection (James et al. 2010). However, the existence of a gender differential in mathematics ability in general is much contested (Ding 2007; Else-Quest et al. 2010).

Age

Age had a negative association with overall UKCAT score and performance in the AR and DA subsections. However, in Higher Education generally, mature-age students tend to outperform younger students because of higher levels of motivation and conscientiousness (Sheard 2009). These are characteristics that should serve mature candidates well in preparation for the UKCAT, which made their relative underperformance perplexing. The underperformance of older candidates in the UKCAT is increasingly salient given that the mean age of applicants to study medicine has been increasing in recent years and thus warrants further scrutiny (Mathers 2011).

Opinions about the UKCAT

We found that respondents in the main were unconvinced that the test measured attributes relevant for the study of medicine. In addition, this study found that almost half of the respondents reported that they found advice on the test confusing and that the test was unfair. A significant proportion considered that their school or college was not well informed about the test and only one in three respondents had found out about the test at the school or college they attended. These findings agree with those of the satisfaction survey conducted by the UK Consortium (UK Consortium 2009).

Test bias

Research, using differential item functioning, has provided evidence that the measurement properties of individual UKCAT questions are invariant across groups (UK Consortium 2008a; UK Consortium 2008b; UK Consortium 2010). However, it is increasingly recognised that ‘apart from item level bias, most of the bias in assessment is extrinsic to the test … and differences between groups come about as a result of the impact of real differences in society’ (Rust 2007). Indeed, all tests are essentially a trade-off between the conflicting objectives of test validity and the politics of diversity in selection (Sackett et al. 2008).

Importantly, in respect of extrinsic test bias, one of the definitions of fairness noted by the Standards for Educational and Psychological Testing is the ‘equitable treatment of all examinees’ in terms of access to practise materials (American Educational Research Association et al. 1999). Furthermore, the UK medical profession advocates that admission systems to Higher Education should minimise barriers ‘arising from the … varying resources and support available to applicants’ (British Medical Association 2009).

We acknowledge that to this end, the UKCAT Board provide easily accessible advice and encourage candidates to use the UKCAT provided on-line practise test to familiarise themselves with the format and varying styles of the subsections of the test. Nevertheless, the UKCAT Board do not endorse commercially available training materials and contend that as the test does not examine acquired knowledge coaching is not necessary, desirable or advantageous (UK Consortium 2011).

We defined coaching as an activity that prepares a student for a specific test, and includes practise on problems that resemble those on the test, and involve instruction and practise with test taking strategies (Briggs 2002). It may be provided by the school or college a candidate attends or, by a commercial training centre. However, as we have shown, there is also a considerable amount of advice and practise resource available through books, on-line and via Schools/Colleges, which cannot be classified as coaching, and all of which may enhance the performance of those that have access to this material. Furthermore, UKCAT candidates are assured that only knowledge of mathematics to GCSE standard (taken at 16 years of age) is required to undertake the test. Our findings have suggested that the on-line advice provided by the UKCAT Board needs to be reconsidered.

Recommendations

The findings of this survey could inform the advice given to prospective applicants to medical/dental programmes (by the UKCAT Board, medical schools and schools/colleges) on how to best prepare for the test. The findings highlight the need for all medical (dental) schools using the UKCAT to monitor the impact of its use. This is especially salient given the emphasis of the Browne Report on the importance of an institution providing evidence of widening access to medicine (Browne 2010).

Limitations of the study

The use of non-probability sampling prevents evaluation of the reliability of the resulting estimates and thus limits knowledge of how much confidence can be placed in the interpretation of our survey findings (Lemeshow & Levy 1999). Furthermore, the response rate of 66% raised the possibility of non-response bias (i.e. non-responders differ from responders in their socio-demography, behaviour or attitudes). Given the anonymity of the survey, we cannot quantify the effect of non-response on the generalisability of our findings to the wider population that took the UKCAT in 2010.

In addition, we assumed that respondents had answered honestly. However, we appreciate that numerous factors can contribute to a response bias, which can result in a questionnaire survey over- or under-estimating the prevalence of an attitude or behaviour. Sources of response bias include misunderstanding of the question, recall of past behaviours, attitudes and beliefs (Schwarz & Oyserman 2001) and the tendency to give socially desirable answers (Paulhus 2002). However, to ameliorate the impact of response bias, we have avoided ambiguous terms, used simple language and pre-tested the questionnaire in order to determine what the respondents think is being asked. Bias may also have been introduced because the results are based on a cross-sectional survey of a single cohort to a single institution. Clearly, a survey of all who had taken the UKCAT in 2010, which included data on all applications to medical school and outcomes, would naturally have been our favoured method.

We acknowledge that individually most of the statistically significant predictors identified by the multivariate regression in this study had a small effect size statistic. However, to put these findings into context, a change in practise whose effect size is 0.5–0.7 would result in an improvement of about one grade at the GCSE level. The effect size of the study of mathematics beyond GCSE was shown to be 0.6 and that for age from 0.6 to 0.8 (Coe 2002).

In summary, these findings imply that the UKCAT as it is currently used may disadvantage some candidates applying to study on an undergraduate medical (or dental) course. This inequity would likely be improved if tutors and career advisors in UK schools and colleges were more informed about the UKCAT and able to offer timely and appropriate advice on preparation for the test.

Acknowledgements

We thank Juliette Bristow for helping to identify the need to conduct this study. We also thank Terry Vallance, James Mitchell, Linda Lobb and Rupert Frankum for their invaluable help in the logistics of this survey and a group of students at Peninsula Medical School for helping to refine the survey questionnaire. This research was financially supported by Peninsula College of Medicine and Dentistry.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.