Attitudes to and Usage of CAT in Assessment in Higher Education

Figures & data

Table 1 Distribution of participants according to subject area (N=69)

Subject Centre	N
Art Design Media	2
Bioscience	2
Business Management Accountancy and Finance	4
Education	5
Engineering	3
English	1
Geography, Earth and Environmental Sciences	1
Information and Computer Sciences	46
Maths, Stats and OR	3
Medicine, Dentistry and Veterinary Medicine	1
Social Policy and Social Work	1

Table 2 Formative assessment methods currently in use. Participants were asked to indicate all methods employed.

Assessment method	Response Count
Objective tests (e.g. multiple-choice, multiple-response, fill-in-the blanks)	44
Practical projects	37
Short answer questions	32
Essays	30
Group projects	22
Portfolios	17
Peer assessment	2
Presentations and/or seminars	2
Problem-Based Learning	2
In-class Socratic dialogue	1
Online discussion forums	1
Reflective journals	1
Self-assessment	1

Table 3 Summative assessment methods currently in use. Participants were asked to select all methods employed.

Assessment method	Response Count
Practical projects	50
Exams	47
Essays	41
Objective tests (e.g. multiple-choice, multiple-response, fill-in-the blanks)	29
Group projects	27
Vivas	26
Short answer questions	23
Portfolios	18
Practical tests	2
Presentations	2
Problem-Based Learning	2
Reports	2
Video reports and/or reflections	2

Table 4 Staff attitude towards objective testing (N=69)

Statement	Strongly disagree	Disagree	Neither agree, nor disagree	Agree	Strongly gree	Median
	(1)	(2)	(3)	(5)
I am happy to use objective testing in formative assessment.	6	6	8	26	23	4
I am happy to use objective testing in low-stakes summative assessment (i.e. less than 10% of the overall mark).	5	7	9	24	24	4
I am happy to use objective testing in high-stakes summative assessment (i.e. greater than 10% of the overall mark).	12	13	15	18	11	3

Table 5 Knowledge of CAT techniques (N=52)

Level of knowledge	N
Limited knowledge	14
Some knowledge	25
Good knowledge	9
Expert	4

Table 6 Reasons why respondents were not employing CAT in their current assessment practice. Participants were asked to select all reasons that applied.

Reason	Response count
Adaptive testing algorithms are more difficult to implement than conventional testing.	19
The items in the pool must be calibrated.	16
Students may perceive the test as being unfair, given that the set of items administered for each student is different.	14
The item pool is larger than that required by conventional testing.	13
Students may not fully understand how their final scores are calculated.	12
The technique is not suitable for the subject that I teach.	8
I think that the advantages of computer-adaptive testing are outweighed by its disadvantages.	7
The software required is too expensive. Substantial start-up costs.	7
I see little or no merit in the idea of computer-adaptive testing in an educational setting.	3
To date, I have not had an opportunity to make use of computer-adaptive testing.	3
I would not know how to set up software for this.	2
It is not possible to integrate technique into existing VLE.	2
Institutional practices make this prohibitive in terms of the effort required to implement it.	2
I am a computer-phobic.	1

Table 7 Perceived benefits of CAT approach (N=7)

Statement	Strongly disagree	Disagree	Neither agree, nor disagree	Agree	Strongly agree	Median
	(1)	(2)	(3)	(4)	(5)
The ability to tailor tests to the proficiency level of individual learners.	0	0	1	2	4	5
The ability to administer tests that take less time.	0	0	5	2	0	3
The ability to administer tests with fewer items.	0	1	5	1	0	3
The ability to provide learners with tailored feedback.	0	0	1	3	3	4
The potential to reduce cheating, given that learners sitting the same test will be presented with a different set of items.	0	0	2	5	0	4

Table 8 Perceived limitations of CAT approach (N=7)

Statement	Strongly disagree	Disagree	Neither agree, nor disagree	Agree	Strongly agree	Median
	(1)	(2)	(3)	(4)	(5)
Students may not fully understand how their final scores are calculated.	0	1	1	4	1	4
Students may perceive the test as being unfair, given that the set of items administered for each student is different.	0	1	1	4	1	4
The item pool is larger than that required by conventional testing.	0	2	3	1	1	3
The items in the pool must be calibrated.	0	0	2	4	1	4
Adaptive testing algorithms are more difficult to implement than conventional testing.	1	2	0	4	0	4