Exploring students’ feedback seeking behavior in the context of programmatic assessment

Abstract

In response to dissatisfaction with testing cultures in higher education, programmatic assessment has been introduced as an alternative approach. Programmatic assessment involves the longitudinal collection of data points about student learning, aimed at continuous monitoring and feedback. High-stakes decisions are based on a multitude of data points, involving aggregation, saturation and group-decision making. Evidence about the value of programmatic assessment is emerging in health sciences education. However, research also shows that students find it difficult to take an active role in the assessment process and seek feedback. Lower performing students are underrepresented in research on programmatic assessment, which until now mainly focuses on health sciences education. This study therefore explored low and high performing students’ experiences with learning and decision-making in programmatic assessment in relation to their feedback-seeking behaviour in a Communication Sciences program. In total, 55 students filled out a questionnaire about their perceptions of programmatic assessment, their feedback-seeking behaviour and learning performance. Low-performing and high-performing students were selected and interviewed. Several designable elements of programmatic assessment were distinguished that promote or hinder students’ feedback-seeking behaviour, learning and uptake of feedback.

Keywords:

• Programmatic assessment
• feedback seeking
• formative assessment

In higher education, students are traditionally assessed after each course, leading to a pass/fail-decision about attainment on that course. If the student passes all courses, the educational program is completed and a diploma is awarded. Although many higher education institutes are experimenting with new approaches to assessment, many students still experience a testing culture, involving a high number of (summative) assessments and a culture of teaching and learning to the test (Jessop et al. 2014; Jessop and Tomas 2017). Programmatic assessment has been introduced as an alternative to this system and has been implemented in various health sciences education institutes (Schut, et al., 2021).

Programmatic assessment aims to integrate and simultaneously optimise decision-making about students (summative purpose; assessment of learning) and stimulate student (self-regulated) learning by the longitudinal and frequent use of feedback (formative purpose; assessment for/as learning). In short, programmatic assessment means that information about learner development is purposefully and regularly collected, creating a longitudinal flow of information (called data points) about development. Pass/fail decisions are only made after a longer period of time, based on aggregation and saturation of this multitude of assessment information (van der Vleuten et al. 2010, 2012). A recent review on programmatic assessment in health sciences education (Schut et al. 2021) shows promising results regarding both decision-making and student learning. However, gaps in our understanding relate to programmatic assessment outside the context of health sciences education.

When it comes to assessment for learning, programmatic assessment aims to stimulate students to monitor and self-regulate their learning (Altahawi et al. 2012; Li, Sherbino, and Chan 2017; Schut et al. 2018) and actively seek feedback (Bok, Teunissen, Spruijt et al. 2013; de Jong et al. 2017). This is often experienced by students as a culture shift from getting grades as a proof of progress (Pitt, Bearman, and Esterhazy 2020). Students are not used to having an active role in the assessment process, in seeking feedback, reflecting on feedback and self-regulating their learning (Altahawi et al. 2012; de Jong et al. 2017). This might hold even more true for low-performing students. Students in health sciences education might not be comparable to students in other domains. One study on programmatic assessment reported an influence of learners’ performance level on their feedback-seeking behaviour (de Jong et al. 2017). Previous research on feedback shows that high performers generally hold more positive perceptions of feedback (van der Kleij 2019), and are more intrinsically motivated to seek feedback (de Jong et al. 2017). When it comes to feedback seeking behaviour, high performers use different feedback seeking strategies (Leenknecht, Hompus, and van der Schaaf 2019).

Altogether, there is a need to study students’ experiences with programmatic assessment outside the context of health sciences education, including bachelor students and low performers. This need focuses specifically on the learning function of programmatic assessment and on its aim to stimulate students’ active use of feedback and self-regulation. The current study therefore focuses on students’ perceptions of programmatic assessment in a bachelor context, in the domain of Communication Sciences, a non-medical domain in which students are not typically well-performing. The program in which this study has been carried out implemented programmatic assessment five years ago. The design aims specifically at stimulating students to actively seek and give feedback. We therefore studied: (1) students’ perceptions of the learning and decision-making functions of programmatic assessment in their bachelor program, (2) students’ perceptions of their own feedback seeking behaviour and how programmatic assessment influences this behaviour, and (3) whether differences can be found between low-performing and high-performing students regarding their perceptions and behaviour.

Principles of programmatic assessment

Programmatic assessment aims to optimise both student learning and robust or reliable decision-making (van der Vleuten et al. 2012). Recently, a number of theoretical principles of programmatic assessment have been agreed upon in a consensus meeting (Heeneman et al. 2021). A fundamental principle is that any individual assessment – regardless of the assessment method – provides insufficient evidence for robust decision-making about learner progression on complex competences (van der Vleuten et al. 2010). Therefore, decisions are based on a multitude of assessments (called data points) which are aggregated and combined into a holistic judgement (Wilkinson et al. 2011). Data points are collected throughout the learning process and can comprise a multitude of forms of evidence about student learning, ranging from knowledge tests to practical assignments, observations in practice and feedback gathered from teachers, peers and clients or customers. Programmatic assessment fits curricula focussing on students as active seekers of feedback rather than passive recipients (Bok, Teunisssen, Spruijt et al. 2013; de Jong et al. 2017).

Programmatic assessment attempts to overcome the dichotomy of assessment purposes as either formative (assessment for/as learning) or summative (assessment of learning). The distinction between assessment for learning and assessment of learning was introduced by the Assessment Reform Group (1999) to emphasise the relationship between assessment and learning. A third notion has been added by Earl (2013), denoted as “assessment as learning”, focussing on the active involvement of students as the critical connectors between assessment information and their learning process, which is sometimes seen as an inherent feature of assessment for learning as well (Clark 2012; Gulikers, Veugen, and Baartman 2021). Schellekens et al. (2021) synthesised the literature on assessment of/for/as learning and argue that these approaches should be seen together to establish an assessment culture that facilitates student learning.

Programmatic assessment tries to integrate assessment of/for/as learning, by the regular collection of information (data points) and the active involvement of learners to seek feedback, make sense of assessment information and use this information to guide further learning. However, it uses a continuum of assessment stakes instead of a distinction between assessment of learning and assessment for/as learning. This continuum ranges from low-stakes decisions which have little consequence for students (e.g. feedback about an assignment) to high-stakes decisions with potentially great consequences (e.g. many credit points). Individual data points never involve pass/fail decisions and are aimed at providing useful feedback for further learning. High-stakes decisions are based on expert holistic judgement, through the combination, aggregation and saturation of information from a variety of data points, providing both quantitative and qualitative information about student progress. Decisions are made by multiple assessors, who mostly have no relationship with the student. Intermediate-stakes decisions are based on all data points gathered so far and focus on remediation and adaptation of learning (e.g. the learning goals for the next learning period).

The recent review by Schut et al. (2021) suggests that programmatic assessment generates sufficient information to enable robust high-stakes decision making. Results show high levels of assessor agreement, perceived fairness and acceptability, validity evidence showing that all competencies were considered in the decision, and early detection of struggling learners (Wilkinson et al. 2011; Driessen et al. 2012; Li, Sherbino, and Chan 2017; de Jong et al. 2019). Schut et al. (2021) report that fifteen studies concluded that programmatic assessment could be used as a catalyst for learning. Programmatic assessment supports ownership and shifts learners’ attention to perceive assessment as information that could be used for further learning rather than as evidence about learning (Zijlstra-Shaw, Roberts, and Robinson 2017; Castanelli et al. 2020). The continuous flow of information provided by the data points stimulates students to monitor progress and improves identification of strengths and weaknesses (Wilkinson et al. 2011; Bok, Teunisssen, Favier et al. 2013).

Students’ perceptions of programmatic assessment

The impact of any assessment system on learning is mediated by the learners’ perceptions (Nijhuis, Segers, and Gijselaers 2008; Cilliers et al. 2010; Altahawi et al. 2012). Research has suggested that students’ perceptions considerably influence whether feedback facilitates learning. Assessment takes place within a context, and students’ responses are shaped by judgments they make about the perceived value placed on assessment (Watling and Lingard 2012). Although many principles of programmatic assessment may sound logical, the continuum of stakes and the dual purpose of data points, which are used both for feedback purposes and later for decision making, fundamentally differs from what students are used to. Therefore, students’ perceptions of programmatic assessment need to be studied carefully, as their perceptions and experiences might differ from what was intended by curriculum designers: the difference between the intended curriculum as specified in materials, the implemented curriculum as enacted and the attained curriculum as experienced by students (van den Akker 2004).

Several studies so far provide insight into students’ perceptions of (programmatic) assessment. When students perceive the assessment process to be unfair, they may develop negative emotions which negatively influence the acceptance of feedback (Watling and Lingard 2012). Response to feedback also seems to depend on the perceived consequences of the assessment, particularly in relation to summative assessment (Cilliers et al. 2010). In programmatic assessment, feedback is generally separated from decision-making, as this might compromise the student-teacher relationship. However, research on students’ perceptions shows that data points which are meant to be low-stakes are often perceived as high-stakes (Bindal, Wall, and Goodyear 2011; Castanelli et al. 2020). Some students are reported to select easy cases or lenient assessors, thereby anticipating decision-making instead of using data points for learning (Bok et al. 2016; Acai et al. 2019). Low-stakes perceptions seem to be enhanced by opportunities for agency, a supportive structure and the role of the teacher (Heeneman and de Grave 2017; Schut, et al. 2021). Whereas we have some indications as to students’ perceptions of the learning and decision-making functions of programmatic assessment, we know little about potential differences between high-performing and low-performing students, and in general about students’ perceptions outside health sciences education.

Feedback-Seeking behavior and programmatic assessment

The way assessment is designed and implemented, and, in turn, perceived by students, may impact on the way students process feedback and regulate their learning. Altahawi et al. (2012) focussed on students’ perceptions of a competency-based portfolio assessment system, which was designed to promote self-regulation and reflective practice. Rather than giving grades, this program provides continuous formative feedback, which allows students to monitor performance. The results of this study showed that students experienced a culture shift, from reliance on a structured assessment system to self-regulation and seeking feedback. Students had to let go of the external validation anchor in the form of grades and accept multi-source feedback. They went from “slavishly studying material” to “actively seeking feedback and acting on it without prompt from the system” (Altahawi et al. 2012, 223). However, when introduced to this system, students experienced uncertainty and were reluctant.

Programmatic assessment views students as active feedback seekers, but do students also have this view? Feedback-seeking behaviour can be defined as purposely seeking information about one’s own level of performance, interpreting it and applying it to reach one’s goals (Leenknecht, Hompus, and van der Schaaf 2019). Two strategies can be distinguished: inquiry (directly asking for feedback) and monitoring (observations and deduction of information from the context). Research on feedback-seeking behaviour has focussed on several key aspects: strategies, frequency, timing, characteristics and topics on which feedback is sought (Crommelinck and Anseel 2013). Regarding frequency, individuals vary and frequency measures only provide an overall picture of the feedback-seeking process. When it comes to timing, most individuals act strategically when asking for feedback, for example only asking for feedback at fixed moments or only asking a specific teacher. Leenknecht, Hompus, and van der Schaaf (2019) investigated the two strategies of feedback-seeking behaviour. They found that students applied both strategies, but monitoring was used significantly more often. Leenknecht, Hompus, and van der Schaaf (2019) studied feedback seeking behaviour in an educational program they labelled as a ‘feedback-friendly culture’, in which there is a balance between formal and informal feedback, a positive orientation towards feedback, and a focus on continuous learning and development. These characteristics resemble programmatic assessment. Therefore, we expect that feedback-seeking behaviour would also play an important role in the learning function of programmatic assessment.

Another issue we will address in this study is whether there are differences in feedback-seeking behaviour between high-performing and low-performing students. Orsmond and Merry (2013) stated that students who underperform are generally poorly represented in empirical studies and therefore less well understood. However, as Orsmond and Merry argue, if lecturers attempt to help low-achievers by providing more feedback, this might limit students’ capacity to develop self-regulation. These students may not only struggle to use feedback, but also struggle to regulate their learning and identify strategies to better use feedback. There is some empirical evidence concerning differences in feedback-seeking between high-performing and low-performing students. For instance, de Jong et al. (2017) showed that high-performing students were more intrinsically motivated to seek feedback. Their main reason for seeking feedback was to learn from feedback. Low-performing students experienced higher external motivation to seek feedback. Their main reason was to collect information to meet the requirements of the assessment procedure, which resulted in seeking feedback only when required. Leenknecht, Hompus, and van der Schaaf (2019) found that high-performing students scored highest on inquiry, whereas low-performing students scored highest on monitoring.

Methods

Context: Assessment system of the bachelor communication sciences

The research setting of this study is the second year of a BA program in Communication Sciences. This program prepares students to become communication professionals, ranging from organising communication about large events to giving communication advice in large governmental organisations. Programmatic assessment has been implemented from year 2 onwards and was chosen because in their professional work many communication professionals are self-employed. The program therefore aims to prepare students to self-regulate their own professional development. Assessment has been designed to optimally stimulate students’ capability to give and ask for feedback, and to self-regulate their learning based on feedback. In short, the design characteristics of the program are:

• Six competencies, worked out in three levels, provide benchmarks for decision-making on student performance at the end of the study year and monitor development throughout the year.

• Students construct a portfolio composed of data points collected throughout the study year, including professional products (i.e. authentic tasks, resembling the future work field), self-reflections, peer feedback, feedback given by experts and – if applicable – evidence of learning collected outside the educational program (e.g. voluntary work or job). No credit points are awarded for individual data points.

• All professional products (i.e. data points) are assessed (formatively) by teachers and/or experts using a holistic scale with 3 levels: “below expectations”, “as expected” and “above expectations”. An assessment form is used, including this scale and narrative feedback.

• Intermediate-stakes progress interviews are held mid-way through the study year (January). Evidence of learning so far is discussed and new learning goals are set together with the student. Annual progress interviews are held at the end of the study year (June/July). Students present their portfolio, and two assessors interview the student about the portfolio.

• Credit points are awarded at the end of the study year (60 credit points), when all data points are aggregated and judged holistically to decide whether the student has attained the required level of development.

• Students and teachers are trained to give and receive feedback. The program design stimulates students to seek feedback, for example by setting a minimum of eight feedback items per competency in the portfolio.

Participants

A mixed methods design was used. A quantitative approach was chosen to measure students’ perceptions of programmatic assessment, their feedback seeking behaviour and learning performance using a questionnaire, to reach as many students as possible. This was followed by a focus on underlying attitudes, beliefs and reasons, as reported by students in individual interviews. This approach allowed us to make a purposeful selection of students for the interviews, including both high-performing and low-performing students, and students scoring high and low on feedback seeking behaviour.

A total of 55  students filled out the questionnaire (µ = 20,69 years, SD = 1,275, range 19-24). The sample consisted of 8 males (14,5%) and 47 females (85,5%). A total of 16 students participated in online individual in-depth interviews. A representative sample of students for the interviews was based on the questionnaire scores. All students participating in this study were divided into 4 groups (see Table 1). A score above the mean was considered as high, a score below the mean as low feedback seeking behaviour or learning performance. Students were invited for an in-depth interview on their student mail. Twelve students agreed to participate, divided over the four groups (µ =20,50, SD = 0,798, range 19-21). Of those 12 students, two were males (16,7%) and 10 were females (83,3%). Participants were compensated with a certificate of participation they could use as a data point in their portfolio.

Table 1. Students participating in the interviews*.

	high-performing	low-performing
high FSB	Participant no. 44 (LP = 2; FSB = 4,18) Participant no. 16 (LP = 2; FSB = 3,45 Participant no. 68 (LP = 2; FSB = 3,64)	Participant no. 15 (LP = 1; FSB = 3,45) Participant no. 7 (LP = 1; FSB = 4,55) Participant no. 25 (LP = 1; FSB = 3,91)
low FSB	Participant no. 4 (LP = 2; FSB = 2,91) Participant no. 42 (LP = 2; FSB = 2,91)	Participant no. 11 (LP = 1; FSB = 2,64) Participant no. 19 (LP = 1; FSB = 2,73) Participant no. 67 (LP = 1; FSB = 2,71) Participant no. 10 (LP = 1; FSB = 2,64)

* Table contains student respondent numbers; learning performance (LP) is divided in categories low (1) and high (2)

Data collection

Questionnaire

Both a paper and online version of the questionnaire were used, to increase the number of participants (Covid-19 lock-down measures started during data collection). Participants who filled out the paper version of the questionnaire during one of their lessons were compensated with a chocolate bar. The questionnaire consisted of three parts (see Table 2 for reliabilities and example items):

Table 2. Reliability analyses of student questionnaire.

	Cronbach’s alpha	Number of items	Example item
Perceptions PA: fitness for purpose	.639	6	In the assessment program I recognise my future profession/professional field
Perceptions PA: validity	.802	8	The competencies clearly describe what you (as a professional) must know and can do
Perceptions PA: learning function	.745	10	The assessment program encourages me to request feedback in a timely manner
Perceptions PA: decision function	.633	6	The decisions about my competence level were in line with my expectations.
Perceptions PA: conditions	.750	7	My teachers provide good feedback.
FSB – inquiry	.781	6	How often do you ask your supervisor how well you are performing on the job?
FSB – monitoring	.735	5	How often do you pay attention to informal, unsolicited feedback from others?

1. Students’ perceptions of programmatic assessment: five scales were constructed, based on existing practical instruments for teacher teams to evaluate their assessment program (Baartman et al. 2007, Baartman, Kloppenburg, and Prins 2017), but now focussing on students’ perceptions instead of teachers’. The scales focussed on students’ perceptions of: fitness for purpose, validity, learning function, decision-making function and conditions. All scales consisted of six to ten items (total of 38 items), all measured with a five-point Likert scale (1 = not at all; 5 = completely).

2. Feedback-seeking behaviour: measured with the feedback-seeking scale of Williams and Johnson (2000), consisting of two categories: inquiry and monitoring. Both scales used a six-point Likert scale (1 = never; 6 = always).

3. Learning performance: students were asked to truthfully fill out their learning performance over the past 6 months (self-report). This was done by asking students about their results on 13 professional products (below expectation, as expected, above expectation).

Interviews

A semi-structured interview guideline was developed and tested in a pilot interview, after which minor adjustments were made. Interview questions focussed on students’ perceptions of the learning and decision-making in the context of programmatic assessment and their feedback seeking. For example, students were asked whether they perceived the decisions to be fair, and how they seek and use feedback.

An information letter and informed consent were sent together with the questionnaire. Students who did not sign the informed consent were excluded from the study. The online interviews varied between 25 and 40 min. At the start of the interview, the participants were made aware of the anonymity and confidentiality of the study. The interviews were video-recorded, with permission of the students.

Analysis

Questionnaire data were analyzed descriptively and correlations between students’ perceptions of programmatic assessment and their feedback seeking behaviour were calculated. An exploratory factor analysis with a large sample and well-conditioned data would have been valuable to understand the structure of the different constructs. Unfortunately, the data of this study did not meet the conditions (e.g. high levels of loadings, low number of factors and high number of variables) in which this could yield reliable results ( De Winter* et al., 2009). For this reason, the theoretical subdivision of scales has been used in this study and questionnaire data have only been analyzed descriptively to provide insight in the characteristics of the students’ perceptions. Questionnaire data were used as input for the follow-up in-depth interviews, in which the results were discussed with the students.

Thematic template analysis was used to explore interview data, which allows analysis of the data in depth through an iterative process (Brooks et al. 2015). Central to template analysis is the development of a coding template based on a subset of data, which is then applied to further data, revised and refined. Template analysis does not prescribe a preset number of coding levels, but rather themes are developed more extensively where the richest data are found. In Table 3, the subsequent steps of the template analyses are summarised.

Table 3. Steps in the thematic template analysis.

step 1	All three authors read 2 transcripts to get acquainted with the data and to suggest input for an initial template. This template was adjusted and refined in a meeting with the three authors.
step 2	All three authors coded 3 new transcripts using the template. Again, the findings were discussed, and the template was adjusted and refined. The codes related to students’ perceptions of decision-making and learning were worked out in more detail. For example, separate codes were developed related to design elements of PA that seemed to cause feelings of (un)fairness.
step 3	This procedure was repeated for the next 4 interviews. In all iterations, the first author (LB) created a new version of the template based on the discussion between the three authors.
step 4	The final template was used by the first author (LB) to recode all transcripts. 20% of the fragments were randomly selected and coded by one of the other authors. Minor discrepancies were discussed.
step 5	The first author (LB) made a summary per theme, checked by the other authors. These summaries formed the basis for the description of the results.

Results

Questionnaire results

The descriptive statistics of students’ perceptions of programmatic assessment, their feedback seeking behaviour and their learning performance are reported in Table 4. Correlation coefficients are presented in Table 5. Some significant correlations are noteworthy. For feedback seeking behaviour, a significant correlation was found between inquiry and monitoring, indicating that students who more often make use of monitoring also more often use inquiry and vice versa. Significant correlations were found between inquiry, students’ perceptions of the learning function and of the conditions of programmatic assessment. This means that if students perceive their assessment program to focus on learning, they more often directly ask for feedback (i.e. inquiry) and vice versa. When students perceive the necessary conditions for learning are met (i.e. teachers and students are capable of giving feedback and have enough time to give feedback), they more often use inquiry as a feedback-seeking strategy and vice versa. Finally, a negative correlation was found between students’ perceptions of the necessary conditions and their learning performance, indicating that if learning performance increases, students’ perceptions of these necessary conditions decrease and vice versa.

Table 4. Descriptive statistics of students’ perceptions of PA, FSB and learning performance.

		N	M	SD	Scale range
Student perceptions of PA	Fitness for purpose	55	3.53	.51	1-5
	Validity	55	3.58	.56	1-5
	Learning function	55	3.41	.54	1-5
	Decision-making function	55	3.56	.52	1-5
	Conditions	55	3.49	.61	1-5
FSB	Inquiry	51	2.83	.73	1-6
	Monitoring	51	3.49	.82	1-6
Learning performance		50	2.26	.32	1-3

Table 5. Correlations between students’ perceptions of PA, FSB and learning performance.

	Fitness for purpose	validity	learning	Descision-making	conditions	FSB inquiry	FSB monitoring	LP
Fitness for purpose	–
Validity	.699**	–
Learning	.631**	.668**	–
Decision-making	.497**	.576**	.476**	–
Conditions	.576**	.541**	.591**	.717**	–
FSB Inquiry	.221	.253	.525**	.200	.352*	–
FSB Monitoring	.039	-.036	.170	.228	.250	.427**	–
LP	-.168	-.208	-.071	-.255	-.408**	-.074	0.045	–

Interview results

Three main themes were identified that describe students’ perceptions of: (1) competencies as the backbone of programmatic assessment, (2) assessment for learning and the role of feedback seeking, and (3) decision-making.

Theme 1. Students’ perceptions of competencies as the backbone of programmatic assessment

The Communication Sciences program uses six competencies worked out in three levels: below expectation, as expected and above expectation. Students collect feedback on their professional products by means of a feedback/evaluation form, which specifies the six competencies and three levels. In the interviews, students reported they generally have clear insight in this backbone. They experience the competencies as the central point of their learning process. Most students perceive the professional products (data points) as a means to the goal of developing their competencies, and thus, not as a goal or assessment in itself. The feedback/evaluation forms seem to be important in guiding students’ perceptions towards the competencies – instead of focussing on their performance on just this single professional product.

The feedback/evaluation forms clearly describe to which competencies it [i.e. the professional product] is linked. So, for me it is clear what is expected for each competency … you are assessed per competency, and you also receive feedback per competency. That helps me in my development. (student 42, high-performing)

Students had to get used to programmatic assessment. This study focussed on second-year students, and the first year of their course was organised around modules, each assessed separately. The students report confusion at the start of their second year. They mention the mid-way progress interviews as very important for understanding the six competencies as the backbone of the program. This mid-way progress interview is an intermediate-stakes decision organised by students themselves. They read each other’s portfolios and ask questions, resembling the appraisal interview at the end of the year.

That backbone became clear to me at the end of the first semester, not from the start of the year. Later everything fell into place and then I discovered why we had to do it this way. The mid-way interview was very clarifying in that respect, like ah, this is what we have to do. (student 11, low-performing)

Students also expressed the need to get a clear grip on their development. They can use the backbone as a kind of ruler and judge their own development against it. This is not what the students are used to, however. Here, a difference can be observed between high-performing and low-performing students. High-performing students are able to judge their own development from the feedback they get, their portfolio and the progress interview. They reflect on their performance and formulate new learning goals for their next assignments. Low-performing students tend to compare their own performance to their fellow-students. They express a preference for marks (in the Netherlands, the use of marks from 1 to 10 is common practice), but also acknowledge the importance of feedback from different perspectives.

I find the below expectations, as expected, and above expectations a bit unpleasant. I prefer marks. I think this ruler is quite vague because you have only three criteria instead of 1 to 10. I think that is unpractical. But on the other hand, you can’t do anything with marks, so I understand why they chose this. You can use the feedback a little bit to judge the mark you would get. (student 25, low-performing)

High-performing students tell how they feel confident about their progress because of the feedback they receive. They ask for and use feedback to judge their own performance. This gives them a sense of being on the right track.

I am working on my portfolio right now, and then I can see I am growing a little bit all the time, towards being a communication professional. (student 42, high-performing)

Theme 2. Students’ perceptions of learning and feedback seeking

In general, the programmatic design stimulates feedback seeking, reflection and the use of feedback. Students are encouraged to use feedback to continuously improve:

I find the way of learning and assessing superb… I like it and I find it better because you can continuously ask for feedback. You get feedback and you can improve. You get the feeling you are working in a professional atmosphere. It is much more than learning for a test. (student 44, high-performing)

Several elements in the design stimulate feedback seeking: a minimum of eight feedback items per competency in the portfolio, training in giving and asking for feedback, and the learning teams. Students describe their learning team as a safe haven in which they give feedback, ask questions and discuss their development. Students know what good feedback entails and expect their teachers to give it. Asking and giving feedback are part of the learning culture.

You hear it all around you. You learn how to give feedback and we are constantly busy giving feedback. It is the normal thing to do, actually. (student 16, high-performing)

I think it works positively for me. When I ask feedback a lot of times, you also learn how to cope with setbacks. In the assessment program you get feedback all year long. The program thus makes sure you are well prepared. (student 67, low-performing)

Students actively seek feedback by asking teachers or fellow students. The only time when students are not inclined to use feedback is when their professional product is judged as ‘as expected’ or ‘above expectations’.

When I get a ‘below expectations’ I always ask for more feedback from my fellow-students. And if I find an assignment difficult, then I ask for feedback. If I get ‘above expectations’ then I just let go of the feedback. I look at it, but I do little with the feedback. (student 68, high-performing)

Theme 3. Students’ perceptions of decision-making

In general, students seem to experience more difficulties accepting decision-making, probably because it is very different from what they are used to. Some elements of the programmatic design seem to cause students to experience decision-making as fair, while other elements cause uncertainty and doubt.

The interview at the end of the year positively influences students’ perceptions of fairness because possible inequalities like free-riding are being corrected. On the other hand, free-riding and the possibility of faking during the interview also cause feelings of doubt. In the individual end-of-year interview, students have to explain and substantiate how they developed their professional products, which cannot be easily faked. The students acknowledge that many pieces of evidence about their performance are being collected throughout the study year, which increases fairness.

I like that fact that it [the decision] is about the entire year, and that they do not decide every moment we do a test. And that they base a decision about whether you are competent just on one test. Now we get many chances to show we can do it after all. (student 19, low-performing)

Regarding fairness, students expressed concern that they get different feedback from different teachers, or some teachers are stricter than others. Students think their teachers should be on the same wavelength when it comes to giving feedback. In programmatic assessment, high-stakes decisions are based on many data points to ensure that many subjective individual judgments are combined, aggregated and triangulated. This is an underlying principle of programmatic assessment students apparently do not easily perceive or accept.

Mixed feelings about the fairness of decision making also seem to be caused by a design element that is not entirely aligned with programmatic assessment principles, namely the principle that no decisions should be based on single data points. In this program, students have to get an ‘as expected’ for each professional product, before they are allowed to enter the end-of-year interview. This may have caused summative feelings about ‘passing’ the professional products and the feeling that the end-of-year interview is the final assessment or the moment at which they have to perform well to be able to pass.

I really did not have any idea about what the 2^nd year would be like. But it gradually became clear. I just thought it was possible to get a ‘below expectations’ and then compensate for it later. But actually, that was not the intention. You have to get everything ‘as expected’ by means of resits. That causes some stress. (student 19, low-performing)

Conclusion and discussion

This study aimed to explore students’ perceptions of the learning and decision-making function of programmatic assessment, in relation to their feedback seeking behaviour. We explored whether differences could be found between low-performing and high-performing students. We did so in the context of a BA program in Communication Sciences, adding to the currently limited body of knowledge about programmatic assessment outside health sciences education. Students’ perceptions can be translated to a number of designable elements of programmatic assessment that seem to positively or negatively impact their feedback seeking, learning and uptake of feedback. Designable elements are aspects of a curriculum that can be deliberately designed to reach the intended goals of the curriculum (van den Akker 2004; Bouw, Zitter, and de Bruijn 2021).

Elements that seem to positively impact students’ feedback seeking, learning and uptake of feedback were found. Regarding assessment as/for learning (Wiliam 2011; Schellekens et al. 2021) students experience a feedback culture in which giving and asking for feedback is daily practice. Designable elements that impact this are the deliberate promotion of feedback seeking, training students and teachers in giving and asking for feedback, and positive relationships between students and teachers. Comparable to the educational programs in health sciences education described by Altahawi et al. (2012) and Li, Sherbino, and Chan (2017), the assessment system in this study deliberately promotes feedback seeking. The program incorporates large numbers of feedback givers for each competency, stimulating a culture of frequent and daily feedback. As in the study of Bok et al. (2016), students and teachers were trained in giving and asking for feedback. This might increase students’ appreciation and uptake of feedback, as this depends on the experienced quality of feedback and credibility of the feedback giver (Heeneman and de Grave 2017). Students reported that the learning teams were important for feedback seeking, as they were experienced as a ‘safe haven’. Positive relationships between teachers and students (Bok et al. 2016) and students’ feelings of autonomy and safeness (Schut et al. 2018) have been reported as important factors in other studies. Like Leenknecht, Hompus, and Van der Schaaf (2019) we found students made most use of monitoring strategies. Other studies reported that students were reluctant in asking for feedback, or acted strategically in asking more lenient feedback givers, because they feared the possible consequences of negative feedback in later decision-making (Bok et al. 2016; Acai et al. 2019; Castanelli et al. 2020). This was not reported by the students in the current study.

Some designable elements were found that seem to negatively impact students’ feedback seeking, learning and uptake of feedback. Students expressed the feeling they have to ‘pass’ data points and do not use feedback when their assignments are judged as ‘as expected’ or ‘above expectations’. Low-performing students find it difficult to use data points as information sources to judge their progress. Students expected their feedback givers to be on the same wavelength and provide similar feedback, and expressed feelings of confusion about how to act when this was not the case. Two designable elements of the program in the current study need further attention as they shed light on students’ perceptions of ‘passing’ data points. First, one of the principles of programmatic assessment is that no (high-stakes) decisions are based on single data points (van der Vleuten et al. 2012). In the current program, students had to get a ‘as expected’ for all data points before they could enter the end-of-year interview. Consequently, students felt they had to ‘resit’ or ‘redo’ an assignment and making mistakes was not allowed.

Second, the end-of-year interview was viewed by students as the crucial decision-making moment, whereas actually – in the intended curriculum (van den Akker 2004) – it was designed as one of the data points: students collect data points in their portfolio, are interviewed about these data points in the end-of-year interview, but decision-making is based on all data points. These perceptions might have been caused by the interview being obligatory and immediately followed by decision-making by the assessors. Altogether, this study again shows how some designable elements can give ‘summative signals’ to students and hamper learning opportunities, as was reported for other designable elements like grades, numerical scales or the obligatory nature of assessment information in the portfolio (Schut et al., 2021).

Regarding students’ perceptions of feedback, this study again shows the intricate balance between assessment for/as learning and assessment of learning. Comparable to Altahawi et al. (2012), the students in our study experienced uncertainty in this new assessment system. The language used by the students shows they are in-between two paradigms of assessment of learning and assessment as/for learning, using both expressions like ‘resit’ and ‘pass a professional product’ while on the other hand appreciating a feedback culture and viewing data points as learning opportunities. As a consequence of prior assessment experiences, students expect teachers and other feedback givers to be on the same wavelength and give comparable feedback. The participating students did not understand the possible value of different perspectives on their work. More research is needed on how to change these perceptions, which may be found in studies on feedback literacy, which also involves the appreciation of differences between feedback sources (Molloy, Boud, and Henderson 2020), which can be used to develop evaluative judgment (Tai et al. 2018).

Another worthwhile direction for further research is the idea of students as internal feedback generators (Nicol, 2021), as this perspective and the active role of the student in the feedback process fits programmatic approaches to assessment. Nicol views students as feedback generators when they compare their performance against a diverse variety of ‘comparators’. From this perspective, data points and feedback by diverse stakeholders can be viewed as comparators, and it is the students who compares their performance against these – possibly diverse – sources of information and opinions of others. For low performers this could be a worthwhile strategy, as they need to experience that learning is not linear, but always involved bumps and disappointments (Pitt, Bearman, and Esterhazy 2020). This can be achieved, Pitt and colleagues suggest, by exposing learners to many repeated low-stakes feedback encounters that make it possible to practice and make mistakes, and continuously improve. These perspectives on feedback nicely fit programmatic assessment, but more research is needed on whether all students (also low performers) are able to generate this kind of internal feedback, and how students can be supported to do so.

Finally, some limitations of the current study need to be noted. Programmatic assessment is based on a number of agreed-upon theoretical principles (Heeneman et al. 2021), but different design choices are made by educational programs, fitting their own context, students, views on learning and professional domain. The findings of this study thus might not be generalisable to other programs implementing programmatic assessment. Regarding differences between low-performing and high-performing students, it needs to be noted that it was difficult – methodologically – to differentiate between them. A binary distinction between low and high performing students does not do justice to actual variations in performance. Students might be high performing on one competence and low performing on another. Methodologically, due to the system of programmatic assessment itself, in which no marks are available, it was difficult to make these distinctions. As researchers, we thus experienced the same problem students reported: how to judge progress and tell who are doing well and who are falling behind. In this study, we asked students to indicate whether their professional products were judged as ‘below expectations’, ‘as expected’ or ‘above expectations’ by their feedback givers. In future studies, it might be advisable to ask students to self-evaluate their progress on the different competences, or to discuss the data points collected so far with students and teachers to get an indication of progress. This better fits the principles of programmatic assessment and might give researchers a better impression of student progress.

This study adds to the existing body of knowledge about students’ perceptions and their learning behaviour in the context of programmatic assessment, outside the domain of health sciences education and with regard to both high-performing and low-performing students. Some design elements were distinguished that stimulate and support assessment for learning, such as incorporating different feedback sources in the design, training for students and teachers, and relationships between students and teachers. Other design elements were found to have more negative effects, such as summative signals like ‘passing’ data points and students’ perceptions that feedback givers need to be on the same wavelength. These design elements can help institutions to make design choices that fit the principles of programmatic assessment, but also do justice to local contextual factors. Further research needs to focus on students’ and teachers’ feedback literacy in the context of programmatic assessment, with a focus on how students – both low-performing and high-performing – can be supported in making sense of diverse feedback sources and using this to improve further work.

Disclosure statement

No potential conflict of interest was reported by the authors.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Additional information

Notes on contributors

Liesbeth Baartman

Liesbeth Baartman is a professor working at the research group Vocational Education at Utrecht University of Applied Sciences. Her area of specialization is assessment in vocational and professional education, with a focus on programmatic and formative assessment.

Hanneke Baukema

Hanneke Baukema is a Master student at Utrecht University. This research was part of her Master’s thesis research.

Frans Prins

Frans Prins is an associate professor at Utrecht University, who specializes on assessment, (peer) feedback, motivation and metacognition.