Relationship between medical programme progress test performance and surgical clinical attachment timing and performance

Abstract

Purpose

Progress tests (PTs) assess applied knowledge, promote knowledge integration, and facilitate retention. Clinical attachments catalyse learning through an appropriate learning context. The relationship between PT results and clinical attachment sequence and performance are under-explored. Aims: (1) Determine the effect of Year 4 general surgical attachment (GSA) completion and sequence on overall PT performance, and for surgically coded items; (2) Determine the association between PT results in the first 2 years and GSA assessment outcomes.

Materials and methods

All students enrolled in the medical programme, who started Year 2 between January 2013 and January 2016, were included; with follow up until December 2018. A linear mixed model was applied to study the effect of undertaking a GSA on subsequent PT results. Logistic regressions were used to explore the effect of past PT performance on the likelihood of a student receiving a distinction grade in the GSA.

Results

965 students were included, representing 2191 PT items (363 surgical items). Sequenced exposure to the GSA in Year 4 was associated with increased performance on surgically coded PT items, but not overall performance on the PT, with the difference decreasing over the year. PT performance in Years 2–3 was associated with an increased likelihood of being awarded a GSA distinction grade (OR 1.62, p < 0.001), with overall PT performance a better predictor than performance on surgically coded items.

Conclusions

Exposure to a surgical attachment improves PT results in surgically coded PT items, although with a diminishing effect over time, implying clinical exposure may accelerate subject specific learning. Timing of the GSA did not influence end of year performance in the PT. There is some evidence that students who perform well on PTs in preclinical years are more likely to receive a distinction grade in a surgical attachment than those with lower PT scores.

Keywords:

• Medical students
• progress testing
• assessment
• clinical attachment

Introduction

The literature is sparse on the relationship between learning context and progress testing. It is presumed that clinical attachments contribute to learning for the PT. The limited data on relationships between PT performance and learning activities or other assessments may be because a common tenet of PT is that students cannot specifically study for a test, as it covers the whole breadth of learning in the programme (Wrigley et al. 2012; Dion et al. 2022). Students should study continually and are likely to focus on current attachments and areas of weakness identified in previous PTs (Schuwirth and van der Vleuten 2010; Neeley et al. 2016). This study was undertaken for two reasons. First, anecdote and feedback indicated that some students were concerned with the ‘fairness’ of their clinical attachment rotation cycles in relation to the timing of our three annual PTs. Students felt that starting with certain attachments gave some advantage in the PTs. Second, our department of surgery was interested in knowing how their clinical attachments contributed to PT knowledge development, particularly in surgical items. Although comparison of PT performance with other undergraduate assessments may be part of institutional practice, it is not described in the literature. This study sought to investigate whether the sequence of a clinical attachment (general surgery) interferes with the performance on the PT and whether PT scores correlate with surgical attachment assessments.

Practice points

• Students on surgical placement showed a transient performance improvement in surgical knowledge on progress tests.

• This effect disappears once all students have completed the attachment.

• Timing (sequence) of a surgical attachment does not ultimately advantage or disadvantage a student.

• Weaker evidence that high-performing students on progress tests in early years were more likely to be high performers in their later surgical attachment.

Background

Progress Tests (PTs) have been used in medical education since the 1970s, with wider adoption since the 1990s, in both undergraduate and postgraduate medical training (Freeman et al. 2010; Wrigley et al. 2012; Albanese and Case 2016). PTs are designed to provide a longitudinal test of students’ applied knowledge and are an assessment ‘of’ and ‘for’ learning (Schuwirth and van der Vleuten 2012; Norman et al. 2010). Key principles of progress testing are that all students in a programme of study sit the same test from a large question bank, drawn against a blueprint, several times a year; questions are set at graduating or completing level (Wrigley et al. 2012). Compared with one-off assessments, PTs improve acquisition of integrated, higher-order knowledge and facilitate long-term knowledge retention (Cecilio-Fernandes et al. 2016; van der Vleuten et al. 2018; Görlich and Friederichs 2021). This form of testing allows demonstration of knowledge growth as students progress, helps to identify knowledge gaps, and allows staff to apply targeted and timely remediation (Muijtjens et al. 2008; Norman et al. 2010; Neeley et al. 2016; Tio et al. 2016; Heeneman et al. 2017).

The PT literature has mainly focused on operational aspects (planning, implementation, collaboration, evaluation) and the metrics of testing (Wrigley et al. 2012; Neeley et al. 2016; Dion et al. 2022); description and justification studies (Cook et al. 2008). In terms of experimental clarification studies, there are a few correlational studies that have explored the predictive value of PTs on future performance in postgraduate assessment (Willoughby et al. 1977; Pugh et al. 2016; Karay and Schauber 2018; Hamamoto Filho et al. 2019; Wang et al. 2021), and some exploring the relationship between PT and other assessments or learning activities, with students (Van Diest et al. 2004; De Champlain et al. 2010; Schaap et al. 2012; Ali et al. 2018; Laird-Fick et al. 2018; Cecilio-Fernandes, Aalders, et al. 2018; Cecilio-Fernandes, Kerdijk, et al. 2018; Couto et al. 2019). Only two of these studies have explored the effect of clinical attachment sequencing (Van Diest et al. 2004; De Champlain et al. 2010). The longitudinal nature of PTs fits within the paradigm of programmatic assessment (Pugh and Regehr 2016; Heeneman et al. 2017). Some triangulation between assessments is desirable to confirm consistency of performance across a range of assessments. Ideally, each learning experience should contribute to overall learning in several domain.

Impact of timing/sequence of clinical attachments on PT performance

Two studies were found that explore this (Van Diest et al. 2004; De Champlain et al. 2010), with one other survey study indicating that medical and dental students found early patient contact more useful than textbook learning for the PT (Ali et al. 2018). De Champlain’s et al. study (2010) considered the effect of rotation order of two clinical blocks on PT performance in one academic year of their programme. Block 1 was a generalist block (family medicine, general medicine and surgery) and Block 2 was specialities (neurology, paediatrics, psychiatry and obstetrics & gynaecology). Knowledge of all disciplines grew over four PTs. Clinical attachments in a particular discipline were associated with knowledge growth in that area. However, students who took Block 1 first, performed better in all tests and for all discipline areas in the final test. The authors hypothesised that medicine and surgery may provide the foundation for study in more specialised subjects. Van Diest et al. (2004) found a knowledge growth effect for psychiatry and behavioural sciences items in their PT across a 6-year programme, reaching a plateau in the last 2 years. Learning opportunities in these disciplinary areas were provided in problem-based learning sessions and a later psychiatric attachment, but the study did not look specifically at timing of experiences and performance. Cecilio-Fernandes et al. compared the impact of curriculum design on oncology learning for their own institution and for a group of four in the Netherlands (Cecilio-Fernandes, Kerdijk, et al. 2018; Cecilio-Fernandes, Aalders, et al. 2018); the national PT was used as the measure of learning. In the single site study, a block of teaching was more effective in the long term than spaced out learning across 3 years (Cecilio-Fernandes, Kerdijk, et al. 2018). In the second, contact hours, a focussed semester of learning, and pre-internship training was associated with better knowledge gain (Cecilio-Fernandes, Aalders, et al. 2018). Neither study was strictly about attachment sequencing.

Correlation between PT and performance in other assessments

Five studies assessed the correlation between PT scores during the medical programme and other performance in programme or subsequent performance in the postgraduate arena (Willoughby et al. 1977; Karay and Schauber 2018; Couto et al. 2019; Hamamoto Filho et al. 2019; Wang et al. 2021). Couto et al. correlated PT scores, formative assessment in small group learning (SGL) and OSCE scores (2019). PT correlated positively at all stages with SGL and OSCE, separately, and all three correlated most strongly at the 8th semester timepoint (latest point tested). For the other four studies, PT performance as an undergraduate predicted postgraduate performance in national licensing examinations or selection for training. In a sixth study, Pugh et al. (2016) developed an OSCE PT (a repeated OSCE based on PT philosophy) for postgraduate internal medicine trainees and compared performance in this test with performance in the disciplinary college OSCE. They found a correlation between OSCE PT performance and scores in both the written and practical college assessments; importantly they predicted failure of candidates positioned in the lower end of PT performance.

In our programme, each student has a pre-determined clinical rotation, shared with a small group of students. This allowed for the study of rotation sequence and knowledge growth in one discipline area without an intervention; the intention being to expand on this gap in the literature. General surgery provided a good discipline marker as it was the only dedicated surgical attachment in our Year 4. Unlike De Champlain et al. (2010), we rotated all disciplines and could remove the potential effect of combined discipline blocks. However, we do agree with their assertion that there is generic core learning in all clinical attachments.

Research positioning

In this study we have adopted a realist ontological stance. Therefore, a quantitative approach was taken, with a correlational focus, based on existing data and the rationales for the study. The stance is post-positivist, seeking some objective answers, but acknowledging the fact that there may be several explanations for the findings, and that biases and confounders exist in a complex educational system. Alongside this, the authors hold a cognitive and social constructionist worldview in relation to learning. While this analysis seeks correlations within empirical data, we recognise that students take responsibility for their learning and approach this in different ways. Teaching and assessment play important roles in that process, but students also engage in their own study practices around their experiential learning.

Therefore, using the surgical attachment as our marker, the aims of our study were summarised as two research questions: to identify whether (1) there was any relationship between the sequence of undertaking this clinical attachment and PT performance in related disciplinary questions and/or performance overall (timing), (2) determine any association between PT results (overall and in items coded to surgical topics) and assessment outcomes in the Year 4 general surgery attachment (correlation). Our hypothesis was that there would be an effect in both cases, but no specific effect on non-surgical PT items.

Materials and methods

This was a longitudinal observational study using a retrospective analysis of prospectively collected student assessment data. Ethics approval was granted by the University of Auckland Human Participants Ethics Committee (# 022281).

Setting

Students enter the MBChB programme at Year 2; selected from a common health science first year, or as graduates. Year 2 and 3 (Y2–3) are predominantly campus based and cover early learning for clinical practice (including simulation and some clinical workplace learning). In Year 4–5 (Y4–5) students rotate through a range of clinical attachments. Year 6 (Y6) is a preparation for practice year. The number of students per year group has increased purposively from 2013, with intakes ranging from 242 to 272 students in the current study.

Progress Testing was introduced to the medical programme in 2013 (Lillis et al. 2014) and is a summative assessment for the ‘Applied Science for Medicine’ domain (The other four domains are; ‘Clinical and Communication Skills’, ‘Personal and Professional Skills’, ‘Hauora Māori’ (Māori health), and ‘Population Health’).

Participants

The study included all students enrolled in the medical programme who started between January 2013 and January 2016. PT performance was tracked to the end of 2018. Students who did not complete Y4 during the study period, had not completed the GSA, or did not complete all three PTs during Y4 were excluded. For students who repeated a year of the programme, only grades from their first full attempt at Y4 were included.

Context

All students take the PT three times a year, simultaneously, in every year. For Y2–5, the PT is norm-referenced and for Y6 standards-based. Each test is unique and contains 125 single-best-answer questions (one from five) set at graduate level. It is constructed against a blueprint and there is a 3-year stand-down for questions used. The blueprint was adopted from our PT partner in the UK. The details of our PT have been published previously (Lillis et al. 2014). There is a ‘don’t know’ option and formula scoring is used (1 mark for correct, −0.25 marks for incorrect, 0 for ‘don’t know’). For each PT, a small number (2–5) of questions are excluded through a moderation process, based on item metrics, resulting in the number of contributing questions being a little less than 125 for most tests. The final item sets had solid internal consistency for each PT, with test-and-year-group-specific Cronbach’s alphas and McDonald’s Omegas both ranging from 0.74-0.92.

Year 4 of the programme is 44 weeks, with 3 weeks of vacation and 4 weeks of non-attachment learning. The remaining time is divided between eight clinical discipline contexts (general medicine, geriatrics, specialty medicine, general practice/primary care, general surgery, emergency medicine, anaesthetics, and musculoskeletal) ranging from 1 to 6 weeks in duration. The GSA is 6 weeks long and students may be assigned to general, paediatric or vascular surgery teams. There are three assessment components: two clinical supervisor’s reports (CSR), a logbook (documents history taking, clinical examination, and self-directed learning from cases), and critical appraisal of a published study to answer a surgery-related question (CAT). Students receive a categorical grade for each component, based on standards, and a rubric is then used to combine them into four outcomes (distinction, pass, borderline performance, fail).

Data extraction

Data on student age, gender, entry pathway (first year or graduate), PT results for Y2–6 and GSA results were extracted from our databases. PTs were numbered sequentially for the purposes of the study as PT1–3 (Y2), PT4–6 (Y3) and PT7–9 (Y4). PT10–12 (Y5) and PT13–15 (Y6) are included for illustrative purposes in Figure 1 to demonstrate knowledge growth over the medical programme but are not otherwise part of the analysis.

Figure 1. Progress Test score growth over time for all and surgical coded items, where Z = 0 at Y6 (graduating year) mean. This figure contains data for students as they move through the programme. Abbreviations: PT, Progress Test; Y, year.

Data processing

The PT blueprint does not include surgery as a named topic, as this was considered to be too broad a category when the PT was being designed. Therefore, for this study, all PT items from the study period were reviewed and coded as either surgical or non-surgical. The coding was initially completed by investigator BL, and items were independently reviewed by two other team members (LP, AW). Surgically coded items were defined with reference to the general surgery curriculum and disagreement resolved by consensus. Items related to other surgical specialty topics were coded as non-surgical for the purposes of this analysis, as they have their own clinical attachments and some are covered by specific blueprint topics (ENT, eye, musculoskeletal, neurological, women’s health). Thus, two question sets were used in the analysis: the full set (all items), and the surgical set (those coded as general surgery). For the surgical attachment assessment outcomes, to account for very low numbers who received a borderline or fail result, these were dichotomised into distinction and non-distinction outcomes for analysis.

Statistical analysis

Data were analysed using R 3.5.2 64 bit for Windows (R Foundation for Statistical Computing, Vienna, Austria) in RStudio Version 1.1.383 for Windows (RStudio Inc., Boston, MA). Theglm and nlme4 packages were used for logistic regression and linear mixed model analyses respectively.

Logistic regression analyses were used to test whether past PT performance (full set and surgical set, PTs 1–6) was related to the likelihood of achieving a distinction grade in the GSA assessments. The PT score for each student in each test was converted to a Z score by using the test and year group specific mean and standard deviation. The mean of the student’s Z scores from Y2–3 in the programme (PTs 1–6) was used as a measure of the student’s average PT performance level within their cohort during the preclinical years and is referred to as ‘PT performance’. Gender and entry pathway were included as main effects in all models by forced entry. The mean of each student’s standardised past PT scores were made available for selection as part of a stepwise model selection procedure, as were the interactions between all variables (gender, entry pathway, PT performance).

Linear mixed models were used to test whether there was a relationship between having undertaken a Y4 surgical attachment and PT 6–9 scores (full set and surgical set). To give a meaningful measure of student performance across tests and within question sets, each student’s PT score on each test was standardised by comparing the score the student obtained to the mean and standard deviation of the Y6 students sitting the same question set in the same test. Y6 students were used as a reference point as their scores are typically stable and represent the endpoint of the assessment. Main effects for PT timepoint, student gender (male vs. female) and graduate entry status (first year vs. graduate entry) were included by forced entry. Random intercepts were used for each student to account for underlying differences in student performance. The timepoint was entered as test sequential order, which was considered a surrogate for calendar time (i.e. PT7, PT8 and PT9 occur in order and are separated by roughly equal time between tests). Stepwise model selection procedures were used to assess whether adding additional fixed variables was warranted. The main variable of interest was whether a student had undertaken their GSA (started or completed) by the time of the PT, but interactions between any combination of the fixed variables were also made available for selection. Stepwise procedures were chosen for all analyses as there were no specific predictions made about the likely contribution of any interactions, and if the variables of interest did have an impact they should be selected by the procedure without manual interference. The Akaike Information Criterion (AIC) was used for selection in all cases (Stoica and Selén 2004). Optima; mode; AIC used and quoted in Table 2.

Results

There were 1035 eligible students. Of these, 70 were excluded by the criteria outlined in the methods section. A total of 965 students were included. Their demographic details were: 519 females (53.8%) and 446 males (46.2%), 853 (88.4%) aged 25 or less, 724 (75.0%) via first year entry and 241 (25.0%) via graduate pathway.

Overall knowledge growth

A total of 2191 PT items were included in the analysis (surgical n = 363; 16.6%). As shown in Figure 1, PT scores increased from Y2 to Y6. In the Y2 to Y4 range focussed on in this study, statistically significant positive correlations were present for the full set of items (r = .78, p < 0.001) and the surgical set (r = .63, p < 0.001), showing knowledge growth over time.

Association between prior PT performance and clinical attachment results

GSA assessments are shown in Table 1. In the tables and text, ‘Performance’ relates to the average standardised (Z score) performance for students in their cohort. The results of the logistic regression analyses are summarised in Table 2. The stepwise procedure selected Performance into the model in all cases. Performance was associated with an increased probability of receiving a distinction grade in all GSA assessments, and an overall distinction, regardless of which item set was used; the exception being the logbook model and surgical question set. Excluding coefficients that did not reach statistical significance, a one unit increase in the Performance was associated with an increase in the odds of receiving a distinction grade of between 1.24 and 1.63, dependent on assessment and question set (Table 2). Based on the AIC, performance data from the full PT set was a better predictor of GSA results than the surgical set. Female gender and graduate entry were also associated with an overall distinction, and a distinction in some GSA components. There was no significant interaction between Performance and either gender or graduate entry. However, as the largest McFadden’s Pseudo R² observed was .032 each relationship appears weak and caution in the interpretation of these results is warranted.

Table 1. General surgical attachment (GSA) assessment results.

Assessment	Fail N (%)	Borderline N (%)	Pass N (%)	Distinction N (%)
Critically appraised topic (CAT)	1 (0.1)	4 (0.4)	705 (73.1)	255 (26.4)
Clinical supervisor’s report (CSR)	2 (0.2)	9 (0.9)	531 (55.0)	423 (43.8)
Logbook	0 (0.0)	7 (0.7)	616 (63.8)	342 (35.4)
Overall	0 (0.0)	0 (0.0)	676 (70.1)	289 (29.9)

Table 2. Predictors of a distinction grade in each component of the surgical attachment assessment.

Assessment	CSR		CAT		Logbook		Overall
Question Set	All	Surgical	All	Surgical	All	Surgical	All	Surgical
(Intercept)	0.52 (0.41,0.65) p < 0.001	0.51 (0.41,0.64) p < 0.001	0.29 (0.22,0.37) p < 0.001	0.28 (0.22,0.36) p < 0.001	0.35 (0.28,0.44) p < 0.001	0.35 (0.28,0.44) p < 0.001	0.22 (0.16,0.28) p < 0.001	0.22 (0.16,0.28) p < 0.001
PT Performance	1.34 (1.14,1.57) p < 0.001	1.37 (1.11,1.68) p = 0.003	1.53 (1.23,1.91) p < 0.001	1.63 (1.24,2.15) p = 0.001	1.24 (1.05,1.46) p = 0.012	1.20 (0.97,1.48) p = 0.091	1.62 (1.3,2.03) p < 0.001	1.45 (1.16,1.81) p = 0.001
Graduate	1.76 (1.1,2.81) p = 0.018	1.87 (1.18,2.98) p = 0.008	1.69 (1.02,2.82) p = 0.044	1.76 (1.06,2.91) p = 0.028	0.95 (0.69,1.31) p = 0.767	1.00 (0.73,1.37) p = 0.996	2.11 (1.25,3.57) p = 0.005	2.07 (1.23,3.46) p = 0.006
Gender F	1.86 (1.38,2.52) p < 0.001	1.86 (1.37,2.51) p < 0.001	1.41 (1,1.99) p = 0.049	1.41 (1,1.99) p = 0.048	2.22 (1.68,2.92) p < 0.001	2.20 (1.67,2.9) p < 0.001	2.90 (2.04,4.12) p < 0.001	2.84 (2.01,4.03) p < 0.001
PT Performance: Graduate	–	–	0.74 (0.52,1.06) p = 0.096	0.7 (0.44,1.11) p = 0.131	–	–	0.71 (0.5,1.02) p = 0.061	–
PT Performance: Gender F	0.7 (0.49,1.02) p = 0.061	–	–	–	–	0.69 (0.42,1.11) p = 0.121	–	–
Graduate:Gender F	0.61 (0.34,1.12) p = 0.111	0.6 (0.33,1.09) p = 0.095	0.55 (0.29,1.07) p = 0.077	0.55 (0.28,1.05) p = 0.072	–	–	0.45 (0.24,0.87) p = 0.017	0.46 (0.2,40.89) p = 0.020
PT Performance : Graduate:Gender F	–	–	–	–	–	–	–	–
AIC	1296.01	1299.51	1104.36	1106.98	1223.6	1227.99	1127.52	1134.57
McFadden’s pseudo R^2	.032	.029	.032	.029	.031	.028	.032	.029

Data are presented as Odds Ratios (CI 95%) and p values.

Abbreviations/key: AIC: akaike information criterion; Gender F: female gender; Graduate: graduate entry; PT performance: average PT Z score in Years 2–3 relative to cohort. Bold text for significant results. A dash indicates a variable that was available for selection but not included in the model.

Effect of surgical attachment sequence on subsequent PT performance

The results of the linear mixed models are presented in Table 3. The stepwise procedure selected terms related to students undertaking a GSA when describing the relationship with standardised performance on the surgical PT set, but not when describing the relationship with standardised performance on the full PT set. For the surgical set, the selected model had statistically significant coefficients for test sequence, GSA, and the interaction between test sequence and GSA. Taken together (shown in Figure 2), performance on the surgical set improves over time regardless of exposure to a GSA. Performance initially increased more with exposure to a GSA, although the effect became reduced at the second time point in Y4 (PT 8). The benefit at PT 9 could not be measured, as all students had been exposed to a GSA by the final test of the year.

Figure 2. Effect of undertaking Surgical Attachment on subsequent Progress Test (PT) performance. A: All PT items. B: Surgical PT items. PT scores calculated as Z distance from the Year 6 mean by PT number and whether students have undertaken a surgical attachment. Year 4 PTs are numbed PT 7-9.

Table 3. Effect on PT scores of completing a surgical attachment.

	All items	Surgical items
Fixed effects
(Intercept)	−6.16 (−6.33,−5.99) p < 0.001	−6.1 (−6.49,−5.7) p < 0.001
GSA	–	4.45 (3.85,5.06) p < 0.001
Test	0.51 (0.49,0.53) p < 0.001	0.6 (0.54,0.66) p < 0.001
Graduate	0.17 (0.04,0.31) p = 0.013	0.1 (−0.07,0.28) p = 0.256
Gender F	−0.08 (−0.2,0.04) p = 0.185	−0.02 (−0.13,0.1) p = 0.77
GSA:Test	–	−0.52 (−0.6,−0.44) p < 0.001
GSA:Graduate	–	–
GSA:Gender F	–	–
Test:Grad	–	–
Test:Gender F	–	–
Graduate:Gender F	–	−0.17 (−0.4,0.06) p = 0.143
GSA:Graduate:Gender F	–	–
GSA:Test:Graduate	–	–
GSA:Test:Gender F	–	–
Test:Graduate:Gender F	–	–
SA:Test:Graduate:Gender F	–	–
Random effects
Random intercept SD	0.88	0.62
Random residual SD	0.96	0.93

Abbreviations/key: Gender F, female gender; Grad, graduate entry; Test, Number of test in a standard test sequence. Bold text for significant results. A dash indicates a variable that was available for selection by the stepwise procedure but was not included in the model by the stepwise procedure.

Data presented as Estimated Coefficient (CI 95%) and p values.

Discussion

This longitudinal observational study assessed the relationship between PT performance and both the timing sequence of a General Surgical Attachment (GSA) and performance in its assessments. Previous high performance in the PT is predictive of a high performance in the GSA assessments. Timing was explored by looking for a sequencing effect (when students undertook the GSA), and their performance in the three PTs across the year. Undertaking the GSA was associated with an increase in performance on the surgical question set but had no effect in performance on the complete question set (Figure 2). Scores in the surgical PT set increased across Y4 regardless of GSA exposure, such that the positive effect diminished over time. Thus, the beneficial effect is removed once all students have completed the attachment. The gain, irrespective of timing, suggests that students gain some surgical knowledge during the rest of the year from non-surgical attachments, other learning activities (e.g. online-learning modules, lectures) and from self-study.

All GSA clinical assessments showed a significant association with past PT performance, when the scores were based on the full PT set, and all except the logbook showed a significant association with past PT performance for the surgical PT set. This association is seen as an increased likelihood of students receiving a distinction grade in GSA assessments. However, performance on the surgical PT set did not provide a better predictor of GSA results than performance on the entire PT set. In fact, model selection techniques indicated that the results from the full set were preferable to those from the surgical set. This could be due to several factors; issues of coding of the surgical set, that the full set is a larger and more stable data set, and that general knowledge may be more helpful when entering an attachment than attachment-specific knowledge. While additional research would be required to test these, the fact that exposure to a GSA was associated with an accelerated increase in performance on subsequent surgical PT items, but not the entire PT set, provides initial evidence in favour of the benefit of general PT knowledge/performance over potential issues with question coding. Taken overall, the findings for PT performance and GSA assessment performance, indicate that higher-performing students in the knowledge domain in Y2–3 are also more likely to be higher performers in the GSA assessments. This is interesting because the PT assesses applied knowledge and the GSA assessments assess clinical skills, performance in the clinical environment and professionalism. The findings suggest that students are constructing learning and using knowledge across domains. This aligns with a constructivist view and sits well with the need for triangulation and multiple overlapping sampling in programmatic learning (Pugh and Regehr 2016; Heeneman et al. 2017). Although Pugh et al. (2016) found a relationship between an OSCE PT and performance in postgraduate examinations, these are within the same domain.

Having undertaken a GSA is associated with a higher level of performance on the surgical PT set, although this difference decreases in the second test and is absent in the final test. This relationship appears to be specific to the surgical PT set, and not overall PT performance which grows steadily. This finding aligns with both De Champlain et al. (2010) and Van Diest et al. (2004) findings. However, De Champlain et al. (2010) grouped clinical attachments and considered only two variations (block 1 then block 2 or vice versa), whereas our study explored timing of a single discipline attachment across the year in multiple combinations. The loss of the benefit of exposure to a GSA by the end of the year was not initially predicted, as previous findings in psychiatry were that plateauing occurred over a much longer period (Van Diest et al. 2004). However, that study did not attempt to formally relate timing of learning to performance, it simply found that discipline-specific knowledge grew across the programme and noted that students had relevant learning experiences across the same period. The pattern observed here suggests that as the year continues, knowledge related to general surgery may be developed within the wider learning environment and not just the attachment, and/or students who completed a GSA earlier in the year lose the advantage they did have, essentially diluting the effect. The first scenario (gaining surgical knowledge elsewhere) is understandable when considering the student experience of clinical attachments, where surgeons may be asked to consult on a case, patients may be transferred to surgery, students may share experiences, other learning opportunities are available, etc. This hypothesis aligns with that of De Champlain et al. (2010) who posited the value of core and general knowledge learning taking place in any attachment. The second scenario is more in line with a proximity/current attachment argument and is an area worth considering in future research.

While not the primary focus of this research, associations with gender and/or graduate entry status were present but less consistent. Significant interactions between these variables were found for the overall GSA assessment and CAT assessments, with significant relationships with gender present for logbooks, and CSR assessments in the absence of significant interactions. Further investigation into these areas may be of interest. Gender difference in performance has been identified previously in assessments (Kacprzyk et al. 2019), but in favour of male students. In this study, where there was an effect, it was in favour female students in CSAs.

Limitations

The study relied on coding of the PT to identity surgical items. Whilst coding was checked independently by three authors, it was undertaken for the study (similar to Van Diest’s et al.’s study (2004); coding appears to have been directly from the blueprint in Cecilio-Fernandes et al. (2018)). Alternative approaches to explore the relationship between PT performance and clinical attachment results could include using an attachment for which PT items were specifically designed, such as paediatrics (although again there would be overlap with learning in general practice and women’s health). This study assessed the effect of having either started or completed a GSA, which means there were some students who were classified as having GSA exposure without gaining the full benefits of their experiential learning; having the potential to weaken the effects observed. The study also used mean performance from the first phase as a predictor of categorical GSA performance. While this returned significant results, the amount of variance explained was low. Future research could benefit from exploring other ways of measuring student ability from PT data, and/or using non-categorical measures of GSA performance. Unfortunately, in our case, meaningful GSA performance measures are only available in a categorical form.

Conclusions

This study demonstrated a relationship between PT performance and GSA results. Specifically, students who score highly in Year 2–3 PTs, prior to clinical attachments, are more likely to receive a distinction grade during their GSA. Sequencing of the GSA has a positive effect on performance in surgical PT items in Year 4, but the effect becomes diminished over the year and disappears by the final test. Further research is required to determine whether these results are generalisable to other clinical specialties, but it is likely that experiential learning during clinical attachments accelerates acquisition of discipline specific knowledge and may also improve generalist knowledge. The findings are important for several reasons: First, students who perform well in knowledge tests at the start of the medical programme are likely to achieve at a high level in subsequent experiential learning. Second, experiential disciplinary learning increases discipline specific knowledge. Finally, timing of clinical attachments has a temporary advantageous effect on discipline-based knowledge performance, but this advantage disappears once all students in the cohort have undergone the attachment.

Acknowledgements

We thank Dr Leah Pointon (LP) who assisted with validation of coding of the surgical items.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Andy Wearn

Andy Wearn, MBChB, MMedSc, MRCGP, FRNZCGP, is an academic general practitioner and became Head of the Medical Programme during the study period.

Vanshay Bindra

Vanshay Bindra, MBChB, transitioned from medical student to junior doctor during the data analysis.

Bradley Patten

Bradley Patten, BSc, PhD, is a psychometrician with experience in educational assessment.

Benjamin P. T. Loveday

Benjamin P. T. Loveday, MBChB, PhD, FRACS, is an academic surgeon with an interest in surgical education.