Searching students’ reflective writing for linguistic correlates of their tendency to ignore instructors’ feedback

ABSTRACT

Students who ignore feedback are poorly positioned to reap its intended benefits. In this study we examined three reflective assignments written by undergraduate Psychology students about their experiences of receiving feedback. We also recorded what proportion of their instructors' feedback each student had accessed during the first two years of their degree, plus their average grades. Using linguistic text analysis software we searched for linguistic features of students’ reflective writing that were statistically associated with their tendency to ignore instructors’ feedback. We found no meaningful associations between feedback-accessing and students’ language use. Exploratory analyses, however, indicated that a greater tendency to ignore feedback was associated with lower grades, and that students with lower grades tended to focus relatively more on the past or present in their reflections than on the future. We discuss the possible merits of using language as an indirect measure in studies of feedback literacy.

KEYWORDS:

• Feedback
• engagement
• language
• assessment
• text analysis
• grades

The willingness of learners to receive and engage with feedback information is a critical aspect of feedback literacy: a topic that enjoys a booming research literature (e.g. Chong, 2021; Nieminen & Carless, 2023; Sutton, 2012). Insofar that feedback is a key means by which learners improve their skills and performance, it stands to reason that learners who tend to ignore instructors’ advice are in a poor position to reap its intended benefits. But whereas we are beginning to understand some of the contextual factors that drive students’ tendency to either engage with or to ignore feedback comments, we still know little about individual difference factors to this end (Brown & Zhao, 2023). In this paper we capitalise on a unique opportunity to analyse samples of written work produced by undergraduate students about their experiences of receiving and using feedback. Specifically, we ask whether quantifiable aspects of the language used by these students are associated with individual variability in their tendency to access – or to ignore – their instructors’ feedback.

Measuring engagement with feedback

Students’ engagement with feedback information can involve numerous types and levels of behaviour, some of which complex and some simple (Winstone & Nash, 2023). The beneficial effects of feedback are perhaps most likely to emerge when students act as ‘proactive recipients’ of the advice they receive, engaging in a breadth of complex behaviours such as reflecting constructively and repeatedly on the advice, and planning future goals and actions thereupon (Handley et al., 2011; Jönsson, 2013; Winstone et al., 2017). Yet at the simpler end of the spectrum, we might identify engagement behaviour based solely on whether or not students even access (e.g. view) the feedback information at all. The mere accessing of feedback is probably too simplistic a behaviour in itself to afford improvements in skills or performance, but it is an essential first step nonetheless: one without which feedback can never be impactful and the feedback-giver’s time is wasted.

Feedback accessing has therefore been of particular interest in academic research because of concerns – among educators and researchers alike – that many students in Higher Education do not meet this lowest threshold of engagement. In one small survey of UK Psychology academics and their students, for example, whereas 96% of the students claimed they ‘always’ or ‘often’ read their written feedback, only 45% of academics believed the same of their students (Hulme & Forshaw, 2009). And even students themselves often admit to lapses in accessing feedback on their assessments. Sendziuk (2010), for example, describes an informal survey of second- and third-year Australian History students, of whom almost 60% admitted to having failed to collect their marked assignments at least once during their time at university, and almost one-third reported having done so three times or more.

Learning Management Systems (LMS) are an increasingly popular source of data and insight on students’ tendency to access their feedback. In one LMS study, researchers evaluated biomedical science students’ engagement with the audio- and written-feedback they received on a series of laboratory reports (Zimbardi et al., 2017). The data showed that after students received feedback on their first report, those who accessed this feedback for at least 1 hour typically saw significant grade improvements on their next report. In contrast, those who accessed their initial feedback for less than 1 hour, or not at all, typically saw no significant improvements until their third or fourth report. Whereas engagement with feedback in Zimbardi et al.’s (2017) study was strong—92% of first and 85% of second year students accessed their feedback, and 58% spent at least an hour with it – other LMS studies show that feedback-accessing can be more modest and dependent on contextual factors. Kuepper-Tetzel and Gardner (2021), for instance, found that only 78% of Psychology students accessed their electronic written feedback on a practical report, but that this figure rose to 95% the following academic year when the written feedback was released 3 days ahead of students’ grades. These findings mirror those of Mensink and King (2020), whose LMS data from nearly 500 students across various degree pathways showed that students were more likely to access their written feedback if their grades were embedded in that same feedback (83% of feedback files were ever accessed), rather than available separately (58% of feedback files were ever accessed). Such findings may reflect a primary interest among students in learning their grades and a lesser interest in hearing feedback. However, undergraduates in another study noted that a common part of the problem is the difficulty of locating instructors’ feedback within many electronic Learning Management Systems (Winstone, Bourne, et al., 2021).

Individual variability in feedback-accessing

Studies such as those described above help us to understand the contextual and systemic factors that contribute to students’ decisions to access vs. ignore feedback comments. But it is equally important to consider individual difference factors; that is to say, how variability between students can likewise contribute to those decisions. Few studies speak to this question. In one study of 360 final-year medical students in Scotland, only 46% collected written formative feedback on their essays, but female students and those who earned higher grades were significantly more likely to do so (Sinclair & Cleland, 2007). In another study, over 700 students aged 16–18 completed measures that probed aspects of their personality and achievement goal orientation, and they also self-reported their prior grades and their tendency to use the feedback they receive (Winstone, Hepper et al., 2021). Replicating Sinclair and Cleland (2007), students who reported greater use of feedback – compared with those who reported lesser use of feedback – had higher prior grades. They were also more conscientious, and more driven both by mastery goals and by performance goals, and these relationships were mediated by students’ self-efficacy in using feedback. That is to say, conscientiousness, mastery- and performance-orientations were all associated with greater self-efficacy, which in turn predicted greater feedback use (for related findings, see Adams et al., 2020).

From Winstone, Hepper et al.’s (2021) findings we might anticipate that students’ willingness to access or engage with feedback is related to individual differences in what they believe about and how they construe the value and importance of receiving feedback information. Yet those researchers used simplistic self-report measures of students’ beliefs about feedback, which lend themselves easily to demand and desirability effects. One possible way to avoid these kinds of biases is to use indirect measurements of students’ beliefs, and one such approach is to examine the specific words students use when reflecting on their experiences of receiving and using feedback. Put differently, rather than directly asking students about their beliefs on feedback, we might instead look at the language they use when they speak or write more freely about their feedback experiences. Linguistic text analyses methods represent a potentially valuable tool to this end.

Language as an indirect measure of feedback beliefs

The use of quantitative text analysis has a long history in social science research (e.g. Gottschalk & Gleser, 1979; Stone et al., 1962), and there is diverse evidence that people’s choice of words can reveal insights into their personalities, their beliefs and understanding, and their feelings and preoccupations (e.g. Chung & Pennebaker, 2018; Tausczik & Pennebaker, 2010). Whereas linguistic text analyses have been rare in the assessment and feedback literatures, there are a growing number of examples, with researchers recently having analysed the words used in instructors’ feedback to learners (e.g. Derham et al., 2022; Nemec & Dintzner, 2016); in institutional policies on assessment and feedback (e.g. Davies, 2023; Winstone, 2022); in university educators’ descriptions of feedback processes (Winstone, Pitt, et al., 2021); and in the feedback research literature itself (Winstone et al., 2022). However, to our knowledge no studies have used linguistic text analysis to examine how students reflect on their feedback experiences. Doing so stands to be informative, because the way students describe their experiences of receiving feedback could provide insight into aspects of their beliefs that motivate their behavioural responses to and engagement with feedback.

For instance, one feature of language that we chose to examine in the present study – which is both easily quantified and potentially associated with feedback accessing – is its focus on the future, as compared with the past or the present. A wealth of research stemming from Future Time Perspective Theory (Nuttin & Lens, 1985), shows that students who tend to focus more on anticipating the distant future and visualising their future goals, typically achieve better grades and show greater persistence in and satisfaction with studying (Zaleski, 1987; Zimbardo & Boyd, 1999), whereas those who are strongly oriented to the past are less likely to perceive their present actions as benefiting future goals (Simons et al., 2004). In general, having a strong mental representation of one’s own future, and future goals, can support learners in seeing the long-term value of their current work and study (de Bilde et al., 2011), and we might therefore predict that students who typically access their feedback more reliably would demonstrate more of a future-oriented focus when they describe their experiences of receiving feedback.

Similarly, we know that students’ negative emotional reactions to feedback can often be a barrier to them being willing to receive and engage with it (e.g. Pitt & Norton, 2017). By examining the extent of positive and negative affective language within a text, we might predict that students who typically access their feedback more reliably would use more positive words and fewer negative words when describing their feedback experiences. As a third example, language research tells us that people – at least in many Western cultures – tend to use first-person pronouns less frequently when they wish to distance themselves from the subject matter or from perceived stressors or threats (e.g. Kross et al., 2014; Newman et al., 2003). We might then predict that students who typically access their feedback more reliably would use more first-person singular pronouns when reflecting on their feedback experiences. Here, both students’ emotional language and their use of first-person pronouns were among the linguistic measures we examined with relation to feedback accessing.

The present study

In sum, our study served to examine whether specific linguistic features of students’ reflections about receiving and using feedback would be associated with their track record of accessing vs. ignoring their instructors’ feedback. To this end we analysed three writing samples produced by undergraduate students, each of which in different ways required students to reflect on feedback they had received, and on how they had engaged or intended to engage with it (see Hoo et al., 2022; Coppens et al., 2023 for very different analytic approaches to similar writing samples). We explored several linguistic features without making directional predictions, as described below; however, for some linguistic features we made (and pre-registered) predictions. Specifically, we predicted that greater levels of feedback-accessing would be associated with greater use of the personal pronoun ‘I’, with a more positive overall tone, greater positive emotional language and less negative emotional language, greater future focus, and greater references to achievement motives. Alongside examining these questions, we also conducted exploratory analyses of how our key variables were associated with students’ grades. Whereas we did not pre-register hypotheses relating to students’ grades, prior work might lead us to predict positive associations between grades and individual differences in feedback-accessing, as described above (e.g. Sinclair & Cleland, 2007; Winstone, Hepper et al., 2021).

Method

Ethical approval for this study was granted by Aston University’s Research Ethics Committee. We preregistered our hypotheses and analytic plan for the linguistic analyses at https://aspredicted.org/XBM_Z97, and our coded, anonymised datasets can be accessed at https://osf.io/maf8g/?view_only=7ec9747bc6844325a9834a0949aebaf6. To preserve students’ anonymity and confidentiality, we cannot share the original writing samples analysed in this study.

Description of the dataset

This study was conducted using data gathered entirely from Turnitin via the University’s Blackboard LMS. We focused on the records pertaining to 760 undergraduate single- or joint-honours Psychology students at Aston University who commenced their first year of study between 2017–2019. Whereas we did not obtain demographic data for these students, the demographic of the student cohorts as a whole comprised approximately 80% females, most were aged 18–19 at the time of starting university, and approximately 35% were of White/White British ethnicities, 35% of South Asian ethnicities, 15% Black African/Black Caribbean ethnicities, and 15% mixed or other ethnicities. For each of these students we sought to record, where available, each of the following forms of data.

Accessing feedback

Each student submitted a maximum of 11 items of coursework during their first and second years of study, for each of which they received written feedback comments. For a minority of students who were enrolled on a joint-honours programme, the maximum was 5 items of coursework, and not all students in the dataset as a whole completed every assignment (total individual items of coursework with feedback = 6491, or M = 8.54 items per student; note that we only extracted data for those occasions where a student had submitted and received feedback on their coursework during the main assessment period, thus ignored all repeated and deferred assessments). For each of these coursework assignment submissions, we recorded whether or not the student had ever accessed the individualised written feedback that their instructor provided in Turnitin. These LMS data were available via the Turnitin platform in binary form (yes/no). To be identified as having accessed an item of feedback, students would have needed merely to have opened it for a minimum of 30 sec; it is therefore impossible to determine the extent of genuine ‘engagement’ that occurred during this $\ge$30-sec period. It is important to note that in all cases, it was possible for students to learn their grade for each assignment without accessing their feedback.

Writing samples

All three reflective writing samples analysed in this study involved students – in different ways – reflecting on their experiences of receiving and of using different types of feedback. During their first year of study, each student completed a coursework assignment on the topic of engaging with feedback, as part of a compulsory introductory Social Psychology module (for details of the assignment, see Winstone and Carless (2020, pp. 34–38). This assignment contained two elements of interest to this study, as described below. For each of these two elements, we extracted students’ written words verbatim and in full for the purpose of linguistic text analysis. We also extracted a third writing sample that was completed by a smaller sub-group of these students who later completed a professional placement during their third year of studies. We identified this third writing sample after pre-registration of the study; our pre-registration therefore only describes the first two writing samples but we adopt the same analytic approach for the third.

Writing sample 1: reflection on group feedback

Prior to completing their main assignment for the Year 1 Social Psychology module, students completed a series of short formative tasks, and subsequently received group-level feedback on these tasks in the form of a ~30-minute screencast recording. This feedback described common areas of good practice and common errors that had been made, as well as some more general advice on academic and reflective writing. Next students were required to write a short reflection (at least 200–250 words, but no maximum), and were instructed as follows ‘After reviewing the group-level feedback, summarise three key bits of advice from the feedback that you think you personally would most benefit from taking on board in your essay. For each of these bits of advice, describe what specific steps you could/will take, to enable you to put the advice into practice’. Students’ short written reflections, on how they intended to use and apply the group feedback they had received, represented our first writing sample of interest.

Writing sample 2: essay on the social psychology of receiving feedback

As their final assignment for this same module, students were required to write a reflective essay (maximum 1650 words) in which they considered one or two past occasions when they had received feedback, and were asked to discuss social psychological research and theory that might shed light on how they had reacted to this feedback. In choosing which feedback experiences to focus on, students were advised: ‘You might choose examples from an educational context (e.g. at school, or university), a different context (e.g. a workplace; a social interaction), or both. Likewise, you might choose examples of formal feedback (e.g. written comments on your work), informal feedback (e.g. a conversation), or even implicit kinds of feedback (e.g. someone’s facial expression when you did something)’. These essays in their entireties represented our second writing sample of interest.

Writing sample 3: reflection on feedback received whilst on professional placement

A subset of these students completed a professional placement year during the third year of their degree, and they completed a written reflective assignment during this year. As one element of this assignment, students were asked to ‘Describe a time when you changed your behaviour based on feedback you had been given [during your placement], and evaluate how effective the change was’ (approximately 300 words). Students’ short reflections on this question represented our third writing sample of interest.

Grades

All students’ assignments were assessed by instructors on a percentage scale from 0–100%, where the passing grade was 40%. For each of the coursework assignments for which we gathered feedback-accessing data, we also recorded the grade that had been awarded.

Linguistic Inquiry & Word Count (LIWC)

For our linguistic analyses we used LIWC (Linguistic Inquiry and Word Count; Pennebaker et al., 2015), a software tool used for quantitatively assessing the types of words used in excerpts of written language. Based on pre-existing, empirically derived linguistic ‘dictionaries’, LIWC searches each text/writing sample for exemplars of defined word-categories (for example, the first-person singular pronoun dictionary ‘I’ identifies instances of the word ‘I’ itself, as well as ‘me’, ‘my’, ‘myself’, etc.); it then exports a numerical quantifier of the extent of that linguistic feature’s occurrence in the text as a whole. In this way, LIWC codes these various linguistic features automatically, without manual human coding or interpretation (for more details about the formulation and psychometric properties of LIWC’s dictionaries, see https://www.liwc.app/help/psychometrics-manuals). In this study we chose 17 linguistic features to examine using LIWC, as outlined in our pre-registration, which we selected based on theoretical and intuitive appraisals of which dictionaries might be relevant to students’ engagement with feedback. These are listed and explained in Table 1, where we also note all directional predictions that we pre-registered for these variables.

Table 1. Mean value of each linguistic feature for each writing sample (ranges of scores within this sample are shown in parentheses).

Linguistic feature	Explanation or examples (from Pennebaker et al., 2015)	Predicted relationship with feedback accessing†	Writing Sample 1 (n = 618)	Writing Sample 2 (n = 639)	Writing Sample 3 (n = 321)
Clout	A score between 0–100. ‘A high number suggests that the author is speaking from the perspective of high expertise and is confident; low Clout numbers suggest a more tentative, humble, even anxious style’	?	24.0 (1.0–82.1)	51.3 (15.8–88.9)	42.1 (7.1–97.8)
Authentic	A score between 0–100. ‘Higher numbers are associated with a more honest, personal, and disclosing text; lower numbers suggest a more guarded, distanced form of discourse’	?	64.4 (3.1–99.0)	56.2 (5.14–93.6)	71.4 (5.7–99.0)
Emotional tone	A score between 0–100. ‘A high number is associated with a more positive, upbeat style; a low number reveals greater anxiety, sadness, or hostility. A number around 50 suggests either a lack of emotionality or different levels of ambivalence’	+	56.2 (3.3–99.0)	51.8 (2.6–98.0)	64.3 (4.1–99.0)
‘I’*	I, me, mine*	+	7.7 (0.0–15.9)	5.9 (0.0–12.1)	8.2 (0.6–15.1)
Positive emotion*	love, nice, sweet*	+	2.6 (0.0–9.1)	3.1 (1.0–6.1)	3.3 (0.4–7.6)
Negative emotion*	hurt, ugly, nasty*	-	0.9 (0.0–3.8)	1.7 (0.3–5.0)	1.0 (0.0–3.9)
Anxiety*	worried, fearful*	?	0.3 (0.0–2.4)	0.4 (0.0–1.7)	0.4 (0.0–2.4)
Tentative*	maybe, perhaps*	?	2.9 (0.0–10.8)	2.6 (0.9–5.7)	1.8 (0.0–8.3)
Certainty*	always, never*	?	2.0 (0.0–6.2)	1.3 (0.3–3.0)	1.5 (0.0–4.5)
Affiliation*	ally, friend, social*	?	0.7 (0.0–4.0)	2.7 (0.2–7.9)	2.2 (0.0–10.4)
Achievement*	win, success, better*	+	2.9 (0.0–8.7)	3.1 (0.4–7.7)	4.0 (0.7–9.3)
Power*	superior, bully*	?	1.7 (0.0–6.5)	3.0 (0.8–9.3)	3.6 (0.3–9.1)
Reward*	take, prize, benefit*	?	1.2 (0.0–5.2)	1.8 (0.3–5.1)	1.7 (0.0–4.9)
Risk*	danger, doubt*	?	0.5 (0.0–3.5)	0.6 (0.0–2.0)	0.5 (0.0–2.3)
Past focus*	ago, did, talked*	?	2.9 (0.0–11.4)	5.2 (1.1–9.6)	7.0 (1.5–13.7)
Present focus*	today, is, now*	?	10.4 (3.3–19.0)	6.5 (2.8–10.6)	5.2 (1.2–16.1)
Future focus*	may, will, soon*	+	2.0 (0.0–9.2)	0.9 (0.0–2.9)	0.8 (0.0–3.3)

*Output variable is the percentage of the total words in each text that feature in the respective dictionary, e.g. a value of 2.6 means that 2.6% of the total words used were from that particular linguistic category.

†The column ‘Predicted relationship with feedback accessing’ notes whether or not we pre-registered a directional prediction. ‘+’ = Positive relationship predicted; ‘-‘ = Negative relationship predicted; ‘?’ = No directional prediction pre-registered.

Results

Only 66.3% of feedback items were ever accessed by students, and the distribution of feedback-accessing among this student sample is illustrated in Figure 1. Of note, just 16.9% of students accessed every piece of coursework feedback they received throughout their first two years of study, and 3.3% of students accessed none of their feedback throughout this entire period.

Figure 1. Distribution of feedback-accessing.

Pre-registered linguistic text analysis

We conducted all inferential analyses using jamovi v1.6.23.0. We begin by addressing our main research question, namely, whether the linguistic features of students’ written reflections on receiving feedback would be associated with their tendency to access their summative feedback. To this end we first calculated – for each student – the proportion of those assignments for which the student had subsequently accessed their feedback. Then, using LIWC we calculated descriptive statistics for each of our 17 pre-selected linguistic features, as shown in Table 1. Per our pre-registration, we analysed the correlations between each of these linguistic features and the proportion of feedback accessed by students, excluding any students who had submitted fewer than five assignments in total. For these analyses we used Spearman co-efficients, and α = .001 to account for the large number of inferential tests and the relatively large number of observations; note that the numbers of students differ between the three writing samples because not all students completed every assessment.

As Table 2 shows, none of the linguistic features of Writing Samples 1 or 3 were significantly associated with students’ tendency to access their feedback. For Writing Sample 2, only one linguistic feature was statistically significant at the $\mathit{\alpha }$ = .001 level; namely, students who accessed more of their feedback tended to use more negative emotional language – not less, as we had predicted – in their essays about receiving feedback. This correlation remained significant even after controlling for students’ average grades, r_{Spearman-partial}(637) = .09, p = .02.

Table 2. Associations (Spearman coefficients) between the linguistic features of each writing sample, and students’ feedback accessing and average grade.

	Writing Sample 1 (n = 618)		Writing Sample 2 (n = 639)		Writing Sample 3 (n = 321)
Linguistic feature	Correlation with % of feedback accessed	Correlation with average grade	Correlation with % of feedback accessed	Correlation with average grade	Correlation with % of feedback accessed	Correlation with average grade
Clout	−.04	−.01	.00	.03	−.07	−.07
Authentic	.05	.00	−.07	−.06	.10	.03
Emotional tone	−.02	−.02	−.04	−.09*	.06	.00
‘I’	.04	.00	−.02	−.02	.12*	.07
Positive emotion	−.01	−.01	.05	.00	.08	.04
Negative emotion	.05	.01	.15***	.17***	−.03	.07
Anxiety	−.03	−.01	.12**	.07	−.05	.00
Tentative	−.01	.13**	−.01	−.03	−.09	−.03
Certainty	−.06	−.04	−.02	−.09 *	.06	−.01
Affiliation	.03	−.03	−.06	−.02	−.07	−.07
Achievement	.02	.01	.02	−.06	.08	.06
Power	.04	−.02	−.04	.02	−.08	−.03
Reward	−.02	−.05	.02	−.08*	.01	−.00
Risk	.06	.01	.13**	.14***	.04	.09
Past focus	−.02	−.07	−.07	-.14***	.07	−.01
Present focus	.01	−.07	−.03	-.22***	−.10	−.11
Future focus	.08	.14***	.02	−.06	.14*	.07

*p < .05; ** p < .01; *** p < .001 (and treated as statistically significant).

Exploratory analyses

We did not pre-register any predictions relating to students’ grades, as noted above, but for exploratory purposes we also examined to what extent variability in these grades was associated with students’ feedback-accessing, or with any of the 17 linguistic features of their written reflections.

Associations between feedback-accessing and grades

Across all assignments, students’ grades on each individual assignment ranged from 0% to 91% (M = 61%, Mdn = 62%, SD = 9.3%). Treating all observations as independent (i.e. ignoring the fact that most students submitted more than one assignment; N_assignments = 6491), students received significantly higher grades for assignments whose feedback they subsequently accessed (M_grade = 62.1%) than for assignments whose feedback they never accessed (M_grade = 58.3%), t(6489) = 15.6, p < .001, d = .41. To replicate this analysis whilst accounting for the repeated observations within students, we conducted a linear mixed models analysis predicting individual grades from whether or not the feedback was accessed (as a fixed effect), and including random intercepts for students. The same result held, t(6434) = 6.21, p < .001.

Associations between language features and grades

Table 2 contains the results of our analyses of the correlations between each of the 17 linguistic features and students’ average grades. For clarity, we emphasise that these analyses are based on students’ average grades across all coursework they submitted in Years 1 and 2. Our analyses showed that students who earned higher average grades typically used significantly more future-focused language in Writing Sample 1, and significantly less past- and present-focused language in Writing Sample 2. Students who earned higher average grades also used significantly more negative emotional language, and more references to ‘risk’, in Writing Sample 2. No other correlations were statistically significant at our pre-determined threshold.

Because we were particularly interested in the data relating to students’ time-perspective, we decided to conduct an additional exploratory analysis by combining the three time-perspective linguistic measures into one. Specifically, for each writing sample we calculated each student’s relative orientation towards the future by subtracting their past focus and present focus scores from their future focus score [i.e. Relative future perspective = Future focus – (Past focus + Present focus)]. Students’ tendency to access feedback was correlated significantly with their relative future perspective scores for Writing Sample 2, r (637) = .09, p = .02, but not for Writing Sample 1, r (616) = .05, p = .20, or Writing Sample 3, r (319) = .07, p = .20. But students’ average grades were correlated significantly with their relative future perspective scores for all three writing samples [Writing Sample 1, r (616) = .15, p < .001; Writing Sample 2, r (637) = .26, p < .001; Writing Sample 3, r (319) = .11, p = .046].

Discussion

Students do not always reliably access or engage with the summative feedback they receive from their instructors. Our principal question in this study was whether students’ reflections on their experiences of receiving feedback might betray subtle linguistic cues that, in turn, are associated with those students’ tendencies to access their instructors’ feedback. Our data, on the whole, reveal no compelling evidence to this effect, at least with regard to the specific linguistic features we chose to examine. Indeed, for the 17 linguistic measures we assessed across three different writing samples, only one measure for one writing sample correlated with students’ feedback accessing to a magnitude that met our threshold for statistical significance. And because that one significant correlation was not replicated across the other writing samples, we should most likely attribute it either to some nuance of the specific assignment, or to chance.

Why might students’ reflections on how they receive feedback have so little measurable connection with their observable feedback-accessing behavior? One possibility is that feedback attitudes and beliefs – which we attempted to indirectly index here through their language-use – are not as stable within individuals as some researchers have often assumed. Indeed, Brown and Zhao (2023) note that almost no psychometric studies on students’ feedback beliefs assess or report test-retest reliability measures, and Nieminen and Carless (2023) recommend concerted efforts to address this critical gap in our understanding of feedback literacy. An alternative explanation is that students’ attitudes towards and beliefs about feedback are stronger drivers of higher-level engagement behaviours – such as discussion-seeking, or the elaboration of specific and achievable goals – than of lower-level behaviours such as the mere accessing of feedback. This possibility, if correct, would have important implications for the design of feedback interventions, as it would imply that targeting students’ feedback-literacy beliefs is unlikely to be an effective way of enhancing their engagement with feedback, unless systemic barriers to their preliminary feedback-accessing are overcome first. As a means to explore this practical question further, it would be interesting to consider how we might measure higher-level feedback engagement behaviours systematically and objectively (e.g. Panadero, 2023; Winstone & Nash, 2023).

It is nevertheless important to consider some limitations of our dataset that could have prevented us from detecting reliable associations between students’ reflective language and their feedback-accessing behaviour. One such limitation is that the specific writing samples we examined may have been inadequate for measuring stable individual differences in students’ language use. In particular, both Writing Samples 1 and 3 were rather short (Writing Sample 1, M = 261 words; Writing Sample 3, M = 317 words), which means that the individual LIWC metrics would have contained greater noise than they would with longer assessments. Whereas Writing Sample 2 was considerably longer (M = 1500 words), it combined individual reflective writing with more traditional academic content describing psychological studies and theories; the latter content would likely have also added noise to the Writing Sample 2 data. Our relatively large dataset partially compensates for noisy data, but researchers who follow up on the present work should ideally seek lengthier writing samples that assess students' beliefs and reflections more ‘purely’. A second limitation is that because all three writing samples were written for the purpose of assessment, students may have felt unable to reflect in a wholly truthful or open way about their responses to feedback (see e.g. Maloney et al., 2013; Truykov, 2023, for detailed considerations of the role of honesty in students’ summative reflective writing). For example, they may have felt obliged to present a receptive and feedback-literate identity through their writing that poorly mirrors their true attitudes and behaviours. These forms of ‘desirable responding’ – akin to what Thomas and Liu (2012) call ‘sunshining’ – would in principle have weakened the possibility of detecting any meaningful linguistic correlates of behaviour. We therefore propose that students’ more spontaneous, non-assessed reflective language could be an interesting source for future analyses of this type.

Looking to our exploratory analyses, our data replicate the prior finding that students who receive higher grades are, on average, more likely to access the feedback they receive (Sinclair & Cleland, 2007; Winstone, Hepper et al., 2021). Indeed, with well over 6000 individual assignments and feedback items in our dataset, we know of no larger published analysis of the relationship between grades and feedback-accessing (the prior largest to our knowledge is Mensink & King, 2020, who examined this relationship with 1462 assignments). The characteristics of our dataset mean that we cannot infer the relationship’s direction of causality, as students could view their grades independently of their feedback if they wished. It may therefore be that students often used their grades to inform whether or not to access their feedback, preferring to ignore it whenever their grades were disappointing. Alternatively, it may be that students’ tendency to access (and presumably – at least sometimes – engage further with) their feedback plays a direct role in their ability and tendency to receive higher grades. We suspect that both explanations play a role.

For the 17 linguistic measures we assessed across three different writing samples, only five of the 51 correlations with students’ average grades met our threshold for statistical significance, and none of these five correlations were for the same linguistic measure across multiple writing samples. Nevertheless, we did find that students’ average grades were positively associated with their use of language that was relatively more oriented towards the future than towards the past and present. This finding must be interpreted with caution given its basis in exploratory analysis, and given the small effect sizes. Nevertheless it fits with empirically supported theory on the links between learners’ achievement and their tendency to be relatively oriented towards the future in their thinking (e.g. Zimbardo & Boyd, 1999), and it was consistent across all three of our writing samples, which were themselves quite different (for example, Writing Samples 2 and 3 were largely retrospective in focus whereas Writing Sample 1 was largely prospective; Writing Sample 1 focused on academic feedback, Writing Sample 3 on professional feedback, and Writing Sample 2 on any kind of interpersonal feedback). Given that the three writing samples we analysed all dealt with students’ reflections on the feedback process, it would be useful to establish whether these associations – if they are replicable more widely – are specific to how students think about feedback, or more generally characteristic of academically capable students. That is to say, students who earn higher grades might be more likely to take a future-oriented perspective in how they think about and receive feedback; alternatively, they might be more future-oriented in their academic approach in general, including in but not limited to the domain of feedback. Both interpretations of our findings would be of theoretical and practical importance, but further work is needed to tease them apart.

There is increasing interest in studying students’ personal reflections as a source of insight into their feedback literacy and beliefs (Coppens et al., 2023; Hoo et al., 2022). This study, to our knowledge, represents the first empirical use of linguistic text analysis to this end, and in this respect it addresses the narrow repertoire of quantitative methodologies that have thus far informed the feedback literacy literature (see Nieminen & Carless, 2023). Our findings provide no compelling evidence that these linguistic metrics were associated with students’ feedback engagement behaviour, and only limited evidence that they were associated with academic performance. Nevertheless, in order to establish valid and robust theoretical accounts of feedback literacy, it is important that nonsignificant empirical findings feature transparently in the literature alongside significant findings (see e.g. Patall, 2021). In the context of current drives towards using stronger measures of behaviour in feedback research, we believe that this linguistic approach offers an insightful means of testing emerging theory on individual differences in students’ feedback literacy.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Robert A. Nash

Dr Robert A. Nash is a Reader in Psychology at Aston University in Birmingham, UK, and a Senior Fellow of the Higher Education Academy. His main areas of expertise are applied cognitive psychology, and the pedagogy and psychology of learners’ (and people’s) engagement with feedback.

Jason M. Thomas

Dr Jason M. Thomas is a Senior Lecturer in Psychology at Aston University in Birmingham, UK, and a Senior Fellow of the Higher Education Academy. His main area of expertise are the biopsychology of eating behaviour, and social influence.