Using Conformal Win Probability to Predict the Winners of the Canceled 2020 NCAA Basketball Tournaments

Figures & data

Fig. 1 Bracket for eight-team single-elimination tournament.

Fig. 2 MOV conformal predictive distribution for South Carolina versus Oregon State with $\tau =1/2$ using regular season data from 2019 to 2020 NCAA women’s basketball season. The blue dotted line identifies a MOV for South Carolina of 5, that is, South Carolina (home) beating Oregon State (away) by five points, with $\pi (5,1/2)=0.160$ identified by the red dashed line.

Table 1 Top 10 NCAA women’s teams for 2019–2020 season.

Team	Estimated strength	Rank	AP	RPI	CSM
Oregon	42.42	1	2	2	2
Baylor	41.92	2	3	4	4
South Carolina	40.18	3	1	1	1
Maryland	39.29	4	4	3	6
Connecticut	36.79	5	5	4	3
Stanford	29.54	8	6	6	7
Mississippi St.	28.93	7	9	10	12
Louisville	28.11	8	6	7	6
Indiana	27.48	9	20	14	19
Oregon State	26.24	10	14	20	17

Table 2 Top 10 NCAA men’s teams for 2019–2020 season.

Team	Estimated strength	Rank	AP	NET	KP
Kansas	24.96	1	1	2	1
Gonzaga	22.87	2	2	1	2
Duke	22.22	3	11	6	5
Michigan State	21.09	4	9	7	7
Baylor	20.23	5	5	5	3
Arizona	19.27	6	–	14	19
San Diego State	18.87	7	6	4	6
Ohio State	18.81	8	19	16	8
Dayton	18.37	9	3	3	4
West Virginia	17.92	10	24	17	10

Table 3 Training and validation set sample sizes spanning 2011–2012 and 2022–2023 seasons.

		Season
		2011–2012	2012–2013	2013–2014	2014–2015	2015–2016	2016–2017	2017–2018	2018–2019	2019–2020	2020–2021	2021–2022	2022–2023
Women	Train	4247	4336	4289	4745	4761	4785	4758	4800	4803	2957	4612	4927
	Validation	358	319	369	465	494	475	491	486	352	425	504	575
Men	Train	4750	4842	4802	4831	4836	4861	4895	4967	4972	3272	4824	5079
	Validation	501	502	479	541	549	546	545	552	300	455	568	583

Table 4 Situations for women’s bubble teams.

Situation	Teams
1	Texas, Alabama, Arizona St., Missouri St., TCU,
	Drake, James Madison, Oklahoma St.
2	Kansas St.
3	LSU, Marquette, North Carolina
4	Texas Tech, West Virginia, Oklahoma
5	all other bubble teams

Table 5 Probabilities of making NCAA tournament field for women’s bubble teams for 2019–2020 season.

Team	Situation	Overall rank	Probability
LSU	3	41	>0.999
Marquette	3	42	0.989
Kansas St.	2	43	0.873
North Carolina	3	44	0.456
Texas Tech	4	50	0.003
West Virginia	4	52	0.004
Oklahoma	4	62	<0.001

Fig. 3 Expert bracket win probabilities (left) and cumulative win probabilities (right) for top 10 women’s teams during the 2019–2020 season.

Table 6 March Madness win probabilities given exemplar brackets for top ranked women’s teams.

Team	Bracket 1	Bracket 2	Bracket 3	Smith	CSM	RTRPI
Oregon	0.308	0.308	0.308	0.307	0.284	0.296
Baylor	0.295	0.295	0.295	0.294	0.286	0.216
South Carolina	0.197	0.197	0.197	0.199	0.215	0.239
Maryland	0.123	0.124	0.124	0.130	0.130	0.175
Connecticut	0.057	0.057	0.057	0.055	0.065	0.056
Stanford	0.007	0.007	0.007	0.008	0.007	0.007
Mississippi St.	0.004	0.004	0.004	0.003	0.003	0.005
Louisville	0.002	0.002	0.002	0.001	0.003	0.002
Indiana	0.002	0.002	0.002	0.001	0.002	0.001
Oregon St.	0.001	0.001	0.001	0.000	0.001	0.001

Fig. 4 Round-by-round win probabilities for Baylor and South Carolina constructed with expert brackets from Michelle Smith (top left), College Sports Madness (top right) and RTRPI (bottom middle). The values shown indicate the probabilities of a team moving on from a particular round.

Fig. 5 Empirical calibration comparison for NCAA women’s and men’s basketball for 2011–2012 to 2022–2023 post-seasons for methods outlined in Section 3.

Fig. 6 Relative log-loss comparison for NCAA women’s and men’s basketball for win probability estimates associated with 2011–2012 to 2022–2023 post-seasons for methods outlined in Section 3.

Table 7 Relative log-loss for NCAA men’s and women’s basketball win probability estimates by league.

League	Conformal	Linear	Logistic
Women	1.000	1.016	1.023
Men	1.000	1.026	1.033

Table 8 Proportion of games corrected predicted for 2011–2023 seasons.

League	Conformal	Linear	Logistic
Women	0.751	0.743	0.743
Men	0.713	0.699	0.699
Overall	0.731	0.719	0.719

Table 9 log-loss performance of conformal win probabilities based on model (16) to Kaggle March Madness leaderboards for men’s March Madness, with higher percentiles indicating better performance.

Season	2015	2016	2017	2018	2019	2020	2021	2022
log-loss	0.516	0.570	0.497	0.716	0.469	–	0.631	0.670
Percentile	0.458	0.697	0.739	0.193	0.816	–	0.492	0.405
# of Submissions	345	598	441	934	863	–	707	930

Table 10 Kaggle methods used for comparison to conformal win probability.

Kaggle User	League	Method	Repository
raddar (raddar 2018)	NCAAW/NCAAM	XGBoost	https://github.com/fakyras/ncaa_women_2018
Gdub (Wierzbicki 2019)	NCAAM	Logistic Regression	https://github.com/gjwierz/NCAA_Kaggle_2019
Sapper (Turner 2021)	NCAAM	Random Forest	https://github.com/dusty-turner/ncaa_tournament_2021_beat_navy

Table 11 log-loss comparison of conformal win probabilities to well performing KMMLM competition models for 2015–2023 NCAAW March Madness tournaments.

Kaggle User	Model	2015	2016	2017	2018	2019	2021	2022	2023
raddar	Base	0.3712	0.5143	0.4387	0.4342	0.3658	0.4764	0.4652	0.4976
	Conformal	0.3873	0.5076	0.4379	0.4314	0.3763	0.4619	0.4655	0.4893

Table 12 log-loss comparison of conformal win probabilities to well performing KMMLM competition models for 2015–2023 NCAAM March Madness tournaments.

Kaggle User	Model	2015	2016	2017	2018	2019	2021	2022	2023
Gdub	Base	0.4970	0.5931	0.5244	0.6049	0.5033	0.6637	0.5780	0.6509
	Conformal	0.4983	0.5890	0.5078	0.6018	0.5103	0.6386	0.5847	0.6370
raddar	Base	0.5108	0.5871	0.4996	0.6741	0.4884	0.6160	0.6481	0.7031
	Conformal	0.5212	0.5809	0.5092	0.6052	0.5020	0.6024	0.6304	0.6416
Sapper	Base	0.5808	0.5611	0.5882	0.6022	0.4605	0.6001	0.7182	–
	Conformal	0.5318	0.5644	0.5147	0.6001	0.4951	0.6188	0.6274	–

raddar. (2018), “ncaa_women_2018,” available at https://github.com/fakyras/ncaa/_women/_2018

 Google Scholar

Wierzbicki, G. (2019), “NCAA_Kaggle_2019,” available at https://github.com/gjwierz/NCAA/_Kaggle/_2019

 Google Scholar

Turner, D. (2021), “ncaa_tournament_2021_beat_navy,” available at https://github.com/dusty-turner/ncaa/_tournament/_2021/_beat/_navy

 Google Scholar