A machine learning study to improve the reliability of project cost estimates

Figures & data

Figure 1. The research methodology.

This ML-based approach, presented in five stages and an alternative to the traditional EVM approach, is used to improve the final cost estimates in projects.

Figure 2. A typical cost S-curve of a project.

A line graph, plotting the three curves of the planned, earned, and actual cost values in EVM, respectively, resembles a typical S-shape pattern.

Table 1. The EVM data of a hypothetical project (the tracking period – month 3).

Time, months	PV, dollars	EV, dollars	AC, dollars	CPI	EAC($), dollars
1	50	40	80	0.50	1400
2	150	80	210	0.38	1838
3	400	270	500	0.54	1296
4	540	–	–	–	–
5	600	–	–	–	–
6	650	–	–	–	–
7	670	–	–	–	–
8	690	–	–	–	–
9	700	–	–	–	–

Table 2. The EVM data and inputs to the XGBoost model (the early-stage estimation).

Project ID	BAC, dollars	EV, dollars	AC, dollars	BAC norm	EV norm	AC norm	CPI	CAC norm
1	180485	13527	13527	1.00	0.07	0.07	1.00	1.00
2	180759	22455	23427	1.00	0.12	0.13	0.96	1.06
3	484398	48729	49235	1.00	0.10	0.10	0.99	1.02
4	3027133	253805	283419	1.00	0.08	0.09	0.90	1.02
5	21369836	263753	269856	1.00	0.01	0.01	0.98	1.22
…	…	…	…	…	…	…	…	…
106	4318950	470150	386654	1.00	0.11	0.09	1.22	0.98
107	1456000	418322	424560	1.00	0.29	0.29	0.98	1.01
108	1512000	118037	119087	1.00	0.08	0.08	0.99	1.01
109	107500	1400	4000	1.00	0.01	0.04	0.35	1.09
110	114700	5700	6050	1.00	0.05	0.05	0.94	1.12

Note: Project ID refers to the projects’ order in the dataset (from 1 to 110) and not to the number of projects. The number of projects used in the early-stage estimation is 99.

Figure 3. The training-testing protocol of the ML algorithm.

A flowchart, representing training and testing processes separately, depicts the process of implementing the ML algorithm to the EVM project data to estimate the final project cost.

Table 3. The ML models, their parameter settings and explanation.

Model	Parameter settings	Parameter explanation
XGBoost	reg_alpha = 7, 6, and 5 (for the early, middle, and late forecasting stages, respectively)	Penalises the complexity of the function leading to finding features that do not improve accuracy.
Random forest	max_depth = 3; n_estimators = 90	Max_depth determines the maximum depth of each tree, allowing each tree in the data to increase until every leaf is considered. The n_estimators defines the number of trees in the forest.
Support vector regression	kernel = rbf by GridSearchCV, C = 10, gamma = auto	The kernel parameter manages complex (non-linear) relationships between the input and output cost variables. The C parameter manages the trade-off between the regularisation term and the training accuracy. The gamma parameter defines the curvature of the kernel function with larger values capturing better the training data, and the lower values resulting in smoother reflecting to testing data.
LightGBM	max_depth = 3, num_leaves = 10	Max_depth determines the maximum depth of each tree, allowing each tree in the data to increase until every leaf is considered. The num_leaves manages the maximum number of leaves (complexity) in one tree.
CatBoost	depth = 4, iterations = 100, learning_rate = 0,07	Max_depth determines the maximum depth of each tree, allowing each tree in the data to increase until every leaf is considered. The learning-rate parameter helps to optimise the training process from the input data given the number of iterations decrease.

Table 4. The accuracy results of the forecasting models, MAPE%.

	Project stages
Model	Early	Middle	Late
MCD	3.92	3.70	1.87
Index-based model	17.43	15.35	14.35
XGBoost	Range: 6.53-9.70	Range: 6.42-8.57	Range: 6.22-8.28
Run 1	7.85	8.57	7.40
2	9.70	6.81	7.97
3	6.53	7.16	8.28
4	8.37	6.42	6.22
5	9.54	8.43	7.74
Random forest	Range: 13.44-14.51	Range: 12.51-13.89	Range: 11.84-12.92
Run 1	14.11	13.10	12.08
2	13.73	12.99	12.28
3	13.44	12.51	12.92
4	13.93	13.89	11.84
5	14.51	13.28	12.54
Support vector regression	Range: 17.92-18.82	Range: 14.14-16.86	Range: 11.34-15.52
Run 1	18.74	16.86	14.42
2	17.92	15.34	14.79
3	17.50	14.68	11.34
4	18.43	16.75	15.52
5	18.82	14.14	14.42
LightGBM	Range: 15.01-18.74	Range: 12.59-16.21	Range: 11.14-14.94
Run 1	15.93	15.81	11.14
2	16.18	12.59	12.98
3	15.01	14.59	13.93
4	17.42	13.19	14.94
5	18.74	16.21	14.19
CatBoost	Range: 13.97-14.92	Range: 11.56-12.54	Range: 11.48-12.52
Run 1	14.49	11.56	11.45
2	14.87	12.54	12.52
3	15.77	12.15	12.25
4	13.97	12.21	11.48
5	14.92	12.04	12.16
Non-linear GGM	25.41	21.14	6.64
Non-linear LGM	21.19	11.81	5.61

Note: for the early-stage estimation, of 99 projects in the dataset, GGM failed to obtain the convergence in 11 and LGM in 21 projects. For middle-stage estimation, of 107 projects, GGM failed to obtain the convergence in 12 and LGM in 28 projects. For the late-stage estimation, of 110 cases, GGM failed to obtain the convergence in 12 and LGM in 25 cases.

Table 5. The accuracy results of the forecasting models, NRMSE.

	Project stages
Model	Early	Middle	Late
Index-based	0.54	0.44	0.30
XGBoost	Range: 0.08-0.19	Range: 0.09-0.21	Range: 0.11-0.23
Run 1	0.15	0.09	0.19
2	0.08	0.16	0.11
3	0.19	0.12	0.16
4	0.13	0.14	0.23
5	0.18	0.21	0.20
Random forest	Range: 0.12-0.39	Range: 0.11-0.29	Range: 0.15-0.28
Run 1	0.19	0.24	0.15
2	0.23	0.17	0.28
3	0.12	0.11	0.26
4	0.39	0.12	0.18
5	0.27	0.29	0.23
Support vector regression	Range: 0.17-0.36	Range: 0.17-0.43	Range: 0.16-0.37
Run 1	0.28	0.17	0.23
2	0.36	0.13	0.16
3	0.31	0.43	0.29
4	0.35	0.28	0.22
5	0.17	0.22	0.37
LightGBM	Range: 0.18-0.29	Range: 0.13-0.23	Range: 0.14-0.21
Run 1	0.18	0.23	0.14
2	0.19	0.22	0.17
3	0.22	0.13	0.21
4	0.29	0.21	0.18
5	0.23	0.17	0.20
CatBoost	Range: 0.17-0.41	Range: 0.16-0.39	Range: 0.10-0.35
Run 1	0.37	0.16	0.30
2	0.20	0.23	0.10
3	0.24	0.27	0.21
4	0.17	0.39	0.35
5	0.41	0.18	0.17
Non-linear GGM	0.27	0.18	0.11
Non-linear LGM	0.18	0.24	0.13

Note: for the early-stage estimation, of 99 projects in the dataset, GGM failed to obtain the convergence in 11 and LGM in 21 projects. For middle-stage estimation, of 107 projects, GGM failed to obtain the convergence in 12 and LGM in 28 projects. For the late-stage estimation, of 110 cases, GGM failed to obtain the convergence in 12 and LGM in 25 cases.

Figure 4. The number of more accurate cases (frequency of projects) by the XGBoost and index-based benchmark models.

A chart, contrasting the estimates’ frequency by the index-based model and the XGBoost model, shows that the performance of XGBoost is significantly better in forecasting the final project cost.

Data availability statement

The authors confirm that the data supporting the findings of this study are available within the article and its supplementary materials.