Towards a measure of learning gain. A journey. With obstacles

Abstract

From a stance of enthusiasm for learning gain, this opinion piece suggests difficulties with the concept, and thereby explores some deficiencies in the current practice of higher education.

Keywords:

• Learning gain
• assessment
• value added

Introduction

It’s a very attractive idea. Measure the learning gain; that is, the academic, professional and/or personal value added, that higher education provides. And then maybe the irritating folk who keep asking us to justify our expenditure of public and private money will go away, satisfied.

What is stopping us?

Obstacle 1 – the appeal of complexity

Academics glory in complexity. We are, after all, paid to think difficult thoughts. Our ability to think difficult thoughts is a huge strength. Yet, when it comes to gathering the evidence and making the case for our own and our sector’s value, we complain about the administrative burden involved, rather than embracing the complexity, the challenge, the opportunity.

Obstacle 2 – a reluctance to justify

Romantics as we sometimes are, we seem to resent the fact that higher education is (also) a business, indeed an industry, contributing in total some 2.8% of UK GDP in 2012 (UUK, 2014, p. 2), substantially publicly funded, some £3.8bn in 2015–6. And therefore we are sometimes called upon to justify our existence and our functioning. Every night, some half a million children go to bed hungry in the UK (Horsey, 2014, p. 17). There are other demands on public funds.

We should, again, embrace being called to account. We should welcome the opportunity to show how well, how productively, and in what complex and appropriate ways we use the resources we consume, using robust and defensible metrics. We could address the research challenge implicit in the comment attributed to Derek Bok at Harvard – ‘If you think education is expensive, try ignorance’. The research challenge of course is ‘prove it’.

Because we take pride in our value, in our quantifiable virtue.

Don’t we?

Obstacle 3 – whose learning gain?

A thought experiment:

Two students enter higher education with the same number of UCAS tariff points. One graduates with a first class honours degree. Her fellow student leaves after one year with a Certificate of Higher Education. Who has the larger learning gain?

From the institutional perspective, the answer is obvious. From the individual perspective, the answer may be different. The first class graduate may have left without an idea of what they want to do next, exhausted and baffled, with a qualification that means little to them, in a subject with which they wish to have nothing more to do. And heavily in debt! The CertHE holder meanwhile may have used their first and only year at university to identify where their passion lies, to make contacts, to define, locate and get just the job they discovered during the year that they wanted, or indeed to start their own business, with a clear view of what they want to achieve over the next few years. Given this further information, which student has achieved the greater learning gain through their time at university?

I use two extreme but not unrealistic examples to suggest that we have to look at the individual learner, and ask them what they gained and how they value it. If we judge it is worth the effort to do so. The student self-report surveys advocated by HEFCE (2017) may get at this individual dimension, depending, as always with surveys, on what questions are asked, and then on how the responses are analysed and weighted.

Obstacle 4 – the rubber ruler

The awards universities give, and the awards we accept for entry to university, are matched to closely defined levels and quantities of learning, and subject to extensive QA. We act as if, for the most part, we believe in the entry qualifications we specify and the exit qualifications we award. This in turn must mean that we also believe in, have faith in, the methods and processes and judgements and evidence base which inform these qualifications. Surely this apparatus, run by able and honourable people, should give us the courage to take seriously the qualifications we accept and award, and to say that a British degree is a British degree, wherever and in whatever discipline it is earned?

Here, of course, things start to crumble. Boud (2018) reminds us of, to put it mildly, the fallibility of our current approaches to assessment. Considering learning gain, Yorke (2007) notes:

Indicators based on entry qualifications and exit performances can be affected by institutions, since it is possible for them to lower the former … and raise the latter … concern [has been] noted that in both the US and the UK the grades gained by students in higher education have increased over time.

Newstead (2002) confirms the unreliability of assessment.

So, some of the time we act as if assessment is largely unproblematic, and, even if all is not well with the world of assessment, all is good enough. Yet all is not well, not even good enough, with assessment. The technical problem for qualifications-based learning gain of the different scales used for entry and exit qualifications can readily be solved. The much bigger problem, of what assessment scores mean, remains.

Would using our current assessments to measure learning gain be optimism, hypocrisy, pragmatism or wilful incompetence? Discuss.

Obstacle 5 – politics

Try another thought experiment. Assume we adopt qualifications-based learning gain. Which would show the greater learning gain – our ancient universities or our newer ones? Imagine the headlines.

Conclusion

The higher education sector will probably do with learning gain what it usually does with changes it doesn’t like – that is, with almost any externally imposed or suggested change – learning outcomes, quality assurance, lecturer accreditation, records of achievement, the Research and now the Teaching Excellence Frameworks…

The sector will probably first (continue to) argue with it, giving copious erudite reasons why it won’t work. (For learning gain, the sector will obviously have to be very cautious about using the argument above concerning the inadequacy of our assessment practices, to avoid any major self-inflicted injury.) A common next step is some version of ‘We are doing that already’. If the outside pressures continue, some at least of us will then embrace learning gain, with more or less good grace, exploiting any advantage we can find in it. As for the rest of it, the sector will quietly seek to absorb it and then dilute it, render it ineffective. Universities have been expert at such dynamic conservatism, bureaucratizing hard to stay as close as possible to the same place for many decades.

But – wouldn’t it be great if we could show the learning gain that results from all our work?

Disclosure statement

No potential conflict of interest was reported by the author.