Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment

Figures & data

Table

Practice points

Using the progress test (PT) in a comprehensive program of assessment (PoA) is beneficial in terms of results and educational impact. Stimulating the use of PT feedback by students through analysis of patterns, formulation, and follow-up of learning objectives is helpful for further learning and progress in the knowledge domain. This study calls on assessment and curricula designers to consider formative use of PTs in a PoA to maximize the benefits of the longitudinal and repetitive features of the PT and use of its feedback.

Table 1. Main features of the curriculum and assessment program of both medical programs.

6yrBM program	4yrM program
Majority of students enter the medical course directly after secondary education/high school, average age at entry is 18 years, at graduation 24 years	Graduate-entry program, student have completed a biomedical bachelor program, average age at entry is 21 years, at graduation 25 years
Three year (preclinical) Bachelor program, three year (clinical) Master program.	1.5 year preclinical program, 2.5 year clinical program
300 students/year	30 students/year in 2010 and 2011, 50 students/year in 2012
Students graduate as Physicians	Students graduate as Physician-Clinical Investigators, emphasis in the curriculum on scholarly topics
Problem/Patient-based learning (PBL) in the pre-clinical years
Clinical rotations two years: 11 disciplines, two electives; Senior Clerkship Clinical Care in discipline of choice (18 weeks) and Senior Clerkship Clinical/scientific research in discipline of choice (18 weeks)	Clinical rotations 1.8 years (11 disciplines), Senior Clerkship combined clinical care and research in discipline of choice (30 weeks)
Separate educational units such as the PBL units, clinical clerkships, the progress test (see below) are appointed credits/ECTS, adding up to 60 ECTS/year	All assessment information and feedback, including the progress test, is used by the student for reflective activities and setting learning objectives. All information and reflective activities is aggregated in a portfolio. The portfolio is assessed at the end of the year, and appointed the majority of ECTS/year in 2010 and all ECTS in 2011 and 2012
Student has a mentor and portfolio in Year 1 and 5	Students has a portfolio and the same mentor in all years of the program
Four progress tests per year
Feedback on progress test score/performance – total and by subject area
The individual total score on each progress test is calculated (as described in the method section). A pass/fail cutoff of the test scores is set at the mean minus the standard deviation (for the cohort of students in the same year-group), and student receives an indication of fail–pass–good for each individual progress test
Summative assessment: the four scores are combined to determine whether the student has made adequate progress and receives a (summative) overall pass for the progress tests of that year, yielding nine ECTS (European Credit Transfer System)/year (of 60 ECTS/year) in the Bachelor and 12 ECTS/year (of 60 ECTS/year) in the Master	The progress test results and feedback are (1) (in)formative and not connected to a pass–fail decision for the progress test only, and (2) are embedded in the portfolio assessment. The portfolio assessment yields all 60 ECTS/year)

Table 2. Set of pre-defined codes, used for the initial template.

1. FEEDBACK

a. Detect deficiencies

b. Detect patterns in scoring

2. FOLLOW-UP FEEDBACK

a. Discuss with mentor

b. Formulate learning objectives

c. Discuss with peers

d. Reflection/analysis

e. Preparation for the PT

f. Strategy

3. SCORE

a. Cut off/standard setting

b. Pass–fail decision

4. SIGNIFICANCE PROGRESS TEST

a. Useful

b. Aggregate with other results

c. Personal assessment of knowledge level

d. Induces stress

5. POSITION PROGRESS TEST

a. In program of assessment

Figure 1. Outline of the study design.

Table 3. Effect size (Cohen's d) and statistical significance for between-group score differences of students in the 4yrM and the 6yrBM programs at Maastricht University, for the total progress test and for sub-domains, in academic years 2010–2011, 2011–2012, and 2012–2013.

	Total test (188 items^b)				Basic sciences (55 items)				Clinical sciences (100 items)				Scholarly topics (16 items)
MM^a	2010^c	2011	2012	Ave^d	2010	2011	2012	Ave	2010	2011	2012	Ave	2010	2011	2012	Ave
9	−0.97***	−1.27***	−1.09***	−1.10	−0.05	−0.40	−0.19	−0.21	−1.56***	−1.63***	−1.62***	−1.60	0.27	0.23	0.25	0.25
10	−0.16	−0.46	−0.60***	−0.45	1.06***	−0.10	−0.01	0.22	−0.91***	−0.77***	−0.92***	−0.87	0.57	0.76***	0.46**	0.57
11	−0.06	−0.49	−0.32	−0.31	0.85***	0.07	0.40*	0.40	−0.46*	−0.68***	−0.38	−0.49	0.91***	0.51**	0.90***	0.78
12	−0.35	−0.14	−0.06	−0.16	0.31	0.46	0.31	0.35	−0.79***	−0.66**	−0.42*	−0.58	0.77***	0.25	0.47**	0.48
13	0.20	−0.24	−0.08	−0.03	0.51	0.43	0.41	0.45	−0.29	−0.74***	−0.48*	−0.48	0.56	0.45	0.81***	0.61
14	0.71**	0.43	0.41	0.51	1.16***	0.49	0.49	0.70	0.04	0.14	−0.10	0.02	1.28***	0.48	1.35***	1.03
15	0.82**	0.59	0.48	0.62	1.28***	0.78**	0.80***	0.93	0.27	0.20	0.21	0.22	0.36	0.48	0.66*	0.50
16	0.10	0.42	0.43	0.31	0.44	0.95***	0.61*	0.65	−0.26	−0.17	0.08	−0.11	0.64*	0.73*	0.77**	0.71
17	0.49	−0.15	0.15	0.15	0.68*	0.43	0.84***	0.64	−0.22	−0.85***	−0.46	−0.51	1.39***	0.86***	0.64*	0.94
18	0.68**	0.55	0.12	0.45	1.22***	0.65*	0.73**	0.85	0.10	0.12	−0.37	−0.04	0.81**	0.58	0.92***	0.76
19	0.34	0.67*	0.50	0.50	0.78**	0.99***	0.67*	0.81	−0.04	−0.10	0.27	0.04	0.67*	1.08***	0.67*	0.80
20	0.30	0.62*	0.37	0.43	0.84***	1.41***	0.86***	1.04	−0.11	−0.08	−0.10	−0.09	0.60*	0.41	0.36	0.45
21	0.32	0.60*	0.94***	0.62	0.38	0.63*	1.01***	0.67	0.08	0.48	0.60*	0.39	0.74***	0.38	0.94***	0.69
22	0.39	0.55	0.58	0.50	0.70**	0.70**	0.56	0.65	0.14	0.30	0.31	0.25	0.36	0.24	0.62*	0.40
23	0.62*	1.00***	0.74*	0.77	0.89***	1.08***	0.76**	0.90	0.36	0.52	0.54	0.46	0.19	0.71**	0.66*	0.50
24	0.48	1.49***	1.33***	1.02	0.42	1.63***	1.21***	0.99	0.35	1.04***	0.86**	0.71	1.25***	0.91**	1.33***	1.14

*p < .05, **p < .01, ***p < .001 (Bonferroni corrected p-value of between-group difference, two-sample t-test, two-sided).

^a MM: measurement moment; with four progress tests per year and six-year groups a student throughout the program is assessed at 24 measurement moments.

^b Average number of items in total test and subtests.

^c Academic year 2010–2011 (September 2010–June 2011).

^d Ave: average effect size over academic years (weighted average using weights proportional to the inverse of the variance of Cohen's d); no statistical significance testing at this level.

Figure 2. Progress test scores (formula score: percentage correct minus penalty for incorrect answers) for students of the four-year graduate-entry Master program (4yrM) and students of the (traditional) six-year bachelor–master program (6yrBM) at Maastricht University, the Netherlands. Shown are average scores per measurement moment for each program, and for the between program difference (see legend), for the total test, and for sub-domains basic sciences, clinical sciences, and scholarly topics. Scores of three academic years (2010–2011, 2011–2012, and 2012–2013), and the average over academic years are presented. Per academic year four progress tests were taken by students of all year groups resulting in scores at 6 × 4 = 24 measurement moments for the 6yrBM program. When comparing the scores of the two programs, the 16 measurement moments of 4yrM students were considered to correspond with measurement moments 9 up to 24 of the 6yrBM students.

Figure 3. Use of the online Progress test feedback system (ProF) by 4yrM and 6yrBM student. Figure shows the results for measurement moments 9–24 in 2012–2013 for both programs, each point on a curve represents the average number of ProF sessions per student in the period of the progress test corresponding to the measurement moment, and the subsequent test.