Predicting first-year university progression using early warning signals from accounting education: A machine learning approach

Figures & data

Figure 1. Flowchart of the machine learning approach.

Figure 2. Timeline of the research design.

Figure 3. Flowchart of the sample.

Table 1. Frequency table: outcome split by gender, probability of success and pre-sessional attendance.

Panel A: gender (n = 609)
	Male	Female	Total
A: Dropout	135	57	192 (32%)
B: Repeat Year	112	54	166 (27%)
C: Pass	157	94	251 (41%)
Total	404	205	609 (100%)

Panel B: Probability of Success (n = 388)^a
	0–30%	31–50%	51–70%	71–85%	86–100%	Total
A: Dropout	2	11	82	29	2	126 (32%)
B: Repeat Year	3	8	57	24	4	96 (25%)
C: Pass	0	6	90	57	13	166 (42%)
Total	5	25	229	110	19	388 (100%)

Panel C: Pre-sessional attendance (n = 609)
	Did not attend pre-sessional	Attended pre-sessional	Total
A: Dropout	145 (33%)	47 (27%)	192 (32%)
B: Repeat Year	123 (28%)	43 (25%)	166 (27%)
C: Pass	171 (39%)	80 (47%)	251 (41%)
Total	439 (100%)	170 (100%)	609 (100%)

^a The number of students providing their probability of success drops substantially due to missing values. This question is not straightforward and many students ticked the ‘don’t know’ option.

Table 2. Descriptives and group means depending on outcome (A = Drop-out, B = Repeat Year, C = Pass).

	Mean				Standard deviation				N
	A	B	C	Total	A	B	C	Total	A	B	C	Total
Backpack data
High school Score^a	69.74	69.93	74.51	72.26	5.90	5.04	5.59	5.98	78	98	189	365
Hours of Maths	5.7	6.0	6.4	6.0	1.4	1.1	1.1	1.3	144	108	188	440
Early engagement
Test 1 Score^b	0.71	0.75	0.79	0.75	0.17	0.14	0.11	0.14	184	160	249	593
Test 2 Score^b	0.62	0.62	0.66	0.64	0.16	0.14	0.14	0.15	176	163	245	584
Exam scores
Exam A score^c	8.72	10.81	14.19	11.61	2.82	2.74	2.49	3.54	178	165	250	593
Exam B score^c	6.76	9.15	14.10	11.09	3.10	3.61	2.98	4.45	106	149	248	503

^a Percentage, range from 1 to 100.

^b Range from 0 to 1.

^c Range from 0 to 20.

Table 3. Testing with χ² and ANOVA if the three outcome groups have statistically different proportions (χ²) or levels (ANOVA) in the respective variables.

	Variable	Test	Df(s)	Statistic	sig
Backpack	Gender	χ^2	2	3.07	0.216
	High school Performance	ANOVA	2, 362	32.64	0.000
	Hours of Maths	ANOVA	2, 437	12.94	0.000
	Perceived Probability of Success	χ^2	8	19.38	0.013
Early engagement	Pre-sessional attendance	χ^2	2	3.41	0.181
	Test 1 Score	ANOVA	2, 590	17.17	0.000
	Test 2 Score	ANOVA	2, 581	5.16	0.006
Exam scores	Exam A Score	ANOVA	2, 590	229.56	0.000
	Exam B Score	ANOVA	2, 500	233.6	0.000

Table 4. Machine learning using the classification method random forest.

Model	Total sample	Training sample (60%)	Test sample (40%)	F1 Naïve baseline (predicting all students in test sample pass)	F1 Test sample using the random forest classifier	Predictive feature importance (provided by random forest)
0. Backpack only = Week 0	254	152	102	35.53%	52.28%	Gender: 7% Maths: 19% High school: 58% Probability of Success: 14%
1. Early Engagement = week 1–6	593	355	238	24.86%	32.59%	Pre-sessional: 1% Test 1: 99%
2. Early Engagement = week 1–9	578	346	232	24.65%	38.11%	Pre-sessional: 1% Test 1: 50% Test 2: 48%
3. Early Engagement + Early Exam scores = Week 1–20	568	340	228	25.84%	55.42%	Pre-sessional: 3% Test 1: 31% Test 2: 31% Accounting A: 34%
4. Early Engagement + Exam scores = Week 1–40	484	290	194	33.33%	61.29%	Pre-sessional: 3% Test 1: 20% Test 2: 22% Accounting A: 19% Accounting B: 33%
5. Backpack + Early Engagement + Exam scores = Week 0–40	230	138	92	39.54%	61.23%	Gender: 2% Maths: 3% High school: 14% Probability of Success: 4% Pre-sessional: 2% Test 1: 11% Test 2: 13% Accounting A: 17% Accounting B: 30%

Note: F₁ is a performance metric for classification accuracy giving equal weight to Type I and II errors (false positives and false negatives).

Table 5. Classification Table with actual and predicted outcome.

Panel A: Model 4 (test set is 194 students): Early engagement + Exam Scores
Actual outcome	Predicted outcome
	A: Dropout	B: Repeat Year	C: Pass
A: Dropout	19	15	6
B: Repeat Year	22	22	13
C: Pass	3	16	78
Panel B: Model 5 (test set is 92 students): Backpack + Early engagement + Exam Scores
Actual outcome	Predicted outcome
	A: Dropout	B: Repeat Year	C: Pass
A: Dropout	8	5	3
B: Repeat Year	6	7	12
C: Pass	3	5	43

Table 6. Ordered Logistic regression on total sample.

	Model 0		Model 1		Model 2		Model 3		Model 4		Model 5
	Coef.	Sig	Coef.	Sig	Coef.	Sig	Coef.	Sig	Coef.	Sig	Coef.	Sig
Gender	0.028	.925									.096	.789
Maths	.395	.000									.398	.004
High School	.130	.000									.075	.016
Probability of Success	.142	.497									−.219	.397
Pre-sessional			.145	.401	.138	.426	−.012	.953	.316	.169	.208	.523
Test 1			3.116	.000	2.786	.000	1.032	.173	1.400	.110	1.307	.365
Test 2					.604	.281	−.299	.646	−.774	.310	−1.639	.149
Accounting A							.510	.000	.194	.000	.169	.024
Accounting B									.336	.000	.368	.000
Pseudo R²	0.201		.063		.056		.478		.545		.566
N (valid)	254		593		578		568		484		230