Is England’s Office for Students likely to falsely identify courses as below threshold on the B3 progression metric?

Figures & data

Figure 1. This graph depicts 100 estimates with 90% CI from a population with 50% of positive outcomes and a sample size of 80 students from a course of 160 students. The figure shows how many samples would not be properly identified as below the 60% threshold.

Figure 2. This graph depicts 100 estimates with 90% CI from a population with 65% of positive outcomes and a sample size of 108 students from a course of 360 students. Five samples would be identified as being below the 60% limit with an upper confidence interval below the 60% threshold.

Table 1. Demonstrates the number and percentage of false positives by confidence intervals.

Confidence Interval	N° total estimations	Mean CI range	N° false positives	% false positives
90	998,400	9.76	82,115	8.22
95	998,400	11.61	105,777	10.59
99	998,400	15.18	156,804	15.71

Table 2. Results of a logistic regression model predicting the likelihood of misclassification by 95% and 99% CIs compared to 90%.

	Odds ratio to 90% CI
95% CI	1.32
	SE = 0.006 (p = <.001)
99% CI	2.08
	SE = 0.009 (p = <.001)
Num. Obs.	2,995,200

Table 3. Illustrates how the percentages in graduate jobs impact the number of false positives for 90%, 95% and 99% CIs.

% Population in graduate job	N° total estimations	N° false positives 99CI	% false positives 99CI	N° false positives 95CI	% false positives 95CI	N° false positives 90CI	% false positives 90CI
20	124,800	30	0.02	4	0.00	1	0.00
25	124,800	69	0.06	15	0.01	4	0.00
30	124,800	765	0.61	224	0.18	109	0.09
35	124,800	1840	1.47	564	0.45	282	0.23
40	124,800	8028	6.43	3972	3.18	2456	1.97
45	124,800	15,788	12.65	9712	7.78	7073	5.67
50	124,800	34,920	27.98	21060	16.88	15,864	12.71
55	124,800	95,364	76.41	70226	56.27	56,326	45.13

Figure 3. These graphs depict the percentage of false positive cases by confidence interval and sample size that is sampled from the population. Each graph represents the percentage with a positive outcome in the population starting from 40% top left to 55% bottom right. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI.

Figure 4. These graphs depict the percentage of false positive cases by confidence interval and percentage of the population that is sampled. Each graph represents the percentage with a positive outcome/graduate job in the population starting from 40% top left to 55% bottom right. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI.

Table 4. Results of two logistic regression models on positive outcomes at 50% and 55% in the population with 90%, 95% and 99% CIs.

	Odds ratio to 90% CI: 50% of a positive outcome	Odds ratio to 90% CI: 55% of a positive outcome
95% CI	1.39	1.56
	SE = 0.016 (p = <.001)	SE = 0.013 (p = <.001)
99% CI	2.67	3.94
	SE = 0.028 (p = <.001)	SE = 0.035 (p = <.001)
Num. Obs.	374,400	374,400

Table 5. Demonstrates the number and percentage of false negatives by confidence intervals on samples with 60% or above in positive outcomes.

Confidence Interval	N° total estimations	Mean CI range	N° false negatives	% false negatives
90	998,400	9.76	5656	0.57
95	998,400	11.61	2372	0.24
99	998,400	15.18	258	0.03

Table 6. Illustrates how the number of false negatives fluctuates for 90%, 95% and 99% CIs depending on the percentages in graduate jobs.

% Population in graduate job	N° total estimations	N° false negatives 99CI	% false negatives 99CI	N° false negatives 95CI	% false negatives 95CI	N° false negatives 90CI	% false negatives 90CI
60	124,800	255	0.2	2327	1.86	5502	4.41
65	124,800	3	0.0	43	0.03	148	0.12
70	124,800	0	0.0	2	0.00	6	0.00
75	124,800	0	0.0	0	0.00	0	0.00
80	124,800	0	0.0	0	0.00	0	0.00
85	124,800	0	0.0	0	0.00	0	0.00
90	124,800	0	0.0	0	0.00	0	0.00
95	124,800	0	0.0	0	0.00	0	0.00

Figure 5. These graphs illustrate the percentage of false negative cases by confidence interval and sample size that is sampled from the population. The two graphs represent the percentage with a positive outcome/graduate job in the population: 60% top and 65% bottom. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI; the straight line (━) – 1% error for 99% CI.

Figure 6. These graphs illustrate the percentage of false negative cases by confidence interval and the percentage of the population that is sampled. The two graphs represent the percentage with a positive outcome/graduate job in the population: 60% top and 65% bottom. The horizontal lines represent levels of error expected for each confidence interval: the dotted line (•••) – 10% error for 90% CI; the dashed line (- - -) – 5% error for 95% CI; the straight line (━) – 1% error for 99% CI.

Table 7. Results of a logistic regression model predicting the likelihood of false negatives by 95% and 99% CIs as compared to 90%.

	Odds ratio to 90% CI
95% CI	0.418
	SE = 0.010 (p = <.001)
99% CI	0.045
	SE = 0.003 (p = <.001)
Num. Obs.	2,995,200

Data availability statement

The dataset and code to reproduce the results are available from the Open Science Framework at: https://osf.io/pd43c/?view_only = bb46a701d69846d99c0518616737df4f.