Multi-Step Dynamic Ensemble Selection to Estimate Software Effort

ABSTRACT

Software Effort Estimation (SEE) is a foremost concern of software companies in order to successfully develop and deliver software products within a defined budget and time. Many software companies fail to deliver the product on time, either due to problems of over-estimation or under-estimation. In order to aid the decision-making process of the analyst and experts, the paper proposed a multi-step dynamic ensemble selection (MS-DES) approach. The DES works in two steps; I) in order to select the suitable models from the pool of models, which are anticipated to perform best when generating a prediction. II) to predict the labeled discretized effort more accurately. The paper utilized four software effort datasets discretized into labeled effort ranges. The performance of the proposed model is evaluated based on the K nearest neighbor oracle (KNORA) canonical approach to DES and in order to reduce the complexity, filter feature selection techniques are applied to extract the relevant feature set. The proposed feature selection-based MS-DES model outplayed the individual models in predicting labeled effort in terms of confusion metrics parameters with an accuracy of more than 90% with all datasets and the results are validated using the ROC curve.

Introduction

Software Effort Estimation is a critical task in software project development. The efforts required to complete the development process is been predicted in person-hours or man-months. There are multiple software development failures that are caused due to unattainable or poorly defined project goals, incorrect resource estimates; overestimation or underestimation, and a failure to control project complexities. Effort estimation is always crucial before the development of the software project which is also an ongoing activity to prevent software failures. In recent times, researchers proposed methods have been put forth, for achieving high predictability of effort. More lately, the community has become interested in the application of machine learning techniques, for software effort estimation. However, no single ML model is considered effective for all the software effort datasets. Therefore, identifying a model, that will be efficient for all the software datasets and will give the best performance in terms of accurate estimation is always a tough assignment.

Furthermore, Shepperd and Kadoda (2001) argue that choosing the best model for a given situation is more beneficial than choosing the best individual model because models have unpredictable behavior across different data sets. This is due to the fact that no model can be said to be the best for all software effort data sets, in accordance with the no-free lunch theorem as discussed by Bardsiri et al. (2013). One way to swamp this problem is to utilize diverse regression or classifier algorithms (ensemble learning) is a different way to get over these problematic situations as in Kocaguneli, Menzies, and Keung (2011), Idri, Hosni, and Abran (2016). Since ensembles are a combination of multiple ML algorithms, are more dependable as in Kocaguneli, Menzies, and Keung (2011). Increased accuracy and robustness have been noted as advantages of ensembles over individual models as in Mendes-Moreira et al. (2012).

The selection of models may be either static or dynamic. In a static selection, the same model subset would be used to make predictions for each test instance. In addition, these systematic reviews by Wen et al. (2012), Idri, Hosni, and Abran (2016), Jorgensen and Shepperd (2006) involve ML, ensembles, and effort; they also present a thorough analysis of dynamic selection by Britto, Sabourin, and Oliveira (2014); and, agile-based software development effort estimation by Usman et al. (2014).

Therefore, our research proposes a multi-step dynamic-ensemble selection (MSDES) of heterogeneous models to handle SEE challenges. Since SEE is a regression problem, the models utilized in the preliminary phase for effort prediction are regression models. Using the concept of discretization, the original data set is transformed into a new data set for classifiers. The data set’s target element in the original dataset (effort, worksup), is detached based on the phenomenon of discretization, and classes are formed for each.

This article evaluates the use of DES models with machine learning algorithms. The dynamic-selection of heterogeneous models, made up of a collection of classification models, is suggested as a means of solving the SEE problem. Performance evaluation and comparisons between the suggested method and previously published individual models and ensemble methods have both been done using the confusion metrics parameters. The present paper compares the proposed methods with individual classifiers, and dynamic ensemble selection methods, which shows the evolution of the proposed model from the individual models. The paper also utilizes feature selection techniques for extracting the best relevant features of the dataset, in order to increase the accuracy and minimize the complexity, with a limited number of features.

Our Contribution

The article aspires to facilitate the transition of both academic institutions and corporate entities from traditional modes of effort estimation to AI-driven transformation. It assesses the efficacy of employing DES models in conjunction with machine learning algorithms for this purpose. The proposed approach advocates the utilization of dynamically selected heterogeneous models, comprising a variety of classification models, as a strategy to address the Software Effort Estimation (SEE) challenge. Furthermore, it underscores the necessity of a comprehensive understanding of the impact of enabling technologies on digitization and adherence to design principles when formulating implementation plans. The main contributions of article are as follows:

• Employing machine learning and ensemble learning models on the datasets is aimed at achieving enhanced accuracy in effort estimation.

• The paper utilizes the phenomenon of discretization, the output variable is classified into ordered label, to transform the task from regression to classification.

• The paper evaluates the effectiveness of DES models with machine learning algorithms, particularly focusing on the dynamic selection of heterogeneous models comprising various classification models.

• The utilization of feature selection techniques to enhance accuracy and reduce complexity.

• The paper compares the performance of multiple canonical approach to dynamic ensemble selection based on evaluation parameters.

Article Organisation

The rest of the paper is organized as follows. Background briefly reviews the work in the domain and definition of SEE. Methodology discusses the methodology and formal definition of techniques utilized. Proposed work presents the proposed method, with algorithm and flow. Results and discussion shows the experiments’ statistical results based on evaluation metrics. Discussion and threats to validity, discusses results and their comparison with existing work and also discusses threats to validity. Lastly, Conclusion and future work summarizes and provides a future recommendation for the study.

Background

The section introduces the importance of software effort estimation, and work performed by various authors and researchers in the field of effort estimation, along with the motivation for the research.

Software Effort Estimation

An important factor in the effective development of a software project is the effort estimation of the software product prior to production. The decision-makers may manage and reuse the resources, budget, and product development time with the aid of accurate effort estimation. The scale of the project, the technology being utilized, the expertise of the company staff, the environment in which the project is being developed, and business standards are just a few of the variables that affect the effort estimation of software projects. These elements, often known as cost drivers, provide an estimation of the real effort needed to construct a software project.

The goal of our research is to increase the precision of estimates for developing software projects. Most software projects fail during development as a result of faulty budgets, defects, and effort estimates or go past the development deadline. The software product effort can be estimated using a variety of ways. Traditional methods include expert judgment, the top-down approach, and statistical and parametric approaches, which estimate the effort based on prior information and client data collected.

Related Work

In order to estimate the effort, Rao and Appa Rao (2021) suggested a novel model based on ensemble learning and recursive feature elimination. The suggested approach is possible to estimate the efforts with the characteristics like size and cost using feature ranking and selection procedure. With the COCOMO II dataset, simulation results for the suggested technique seem to be positive. In comparison to the individual techniques, the performance of the suggested solution is quite promising in respect of both actual cost and LOC.

Cabral and Oliveira (2021) has proposed DES an approach, a collection of regressors chosen in real time by classifiers, to estimate effort. The classifiers learn to choose the most effective regressor from a group of candidates across a number of different data sets. Classifiers choose regressors whose predictions are combined to form the final prediction.

Mousavi, Eftekhari, and Rahdari (2018) recommended an unconventional approach by integrating static and dynamic ensemble selection processes with Over-Bagging. The author uses the Over-Bagging method for learning about class imbalances. In order to determine the optimal classifiers and their combiner across all test samples, the proposed technique employs the Genetic Algorithm as the static ensemble selection method. The dynamic ensemble selection is then performed on a subset of the chosen classifiers for each test sample. The author claims that the suggested Omni-Ensemble Learning (OEL) has superior performance using G-mean, balance, and AUC metrics across seven NASA datasets.

Suresh Kumar et al. (2022) proposed a gradient boosting regressor model. The author uses COCOMO81 and CHINA datasets for training. The evaluation metrics for regression models, MAE, MSE, RMSE, and R2 were used to measure performance. The author states that the gradient boosting regressor model is effective, as evidenced by its accuracy of 98% with the COCOMO81 dataset and 93% with the CHINA dataset.

Alhazmi and Zubair Khan (2020) employed ensemble learning bagging using linear regression as the base learner, as well as MLP, REPTree, random forest, SMOReg, and M5Rule. Also constructed the feature selection techniques; BestFit and Genetic Algorithm. The model is tested with a China dataset. According to the author, the relative error of the Bagging M5 rule with Genetic Algorithm as Feature Selection is 10%.

Hidmi and Erdogdu Sakar (2017) proposed ensemble learning by merging the Support Vector Machine (SVM) and K-Nearest Neighbor (k-NN) machine learning algorithms. The author utilized labeled Desharnais and Maxwell, two public datasets. The author suggested that ensemble learning enhances the outcomes and that svm technique outperforms k-nn technique.

Shukla and Kumar (2019) suggested feature reduction, to determine the most important attributes in the Desharnais data set. For software effort estimation, they then applied MLPNN to the reduced data set and obtained an R2 value of 79%.

Marco, Sharifah Syed Ahmad, and Ahmad (2022) proposed AdaBoost ensemble learning and random forest (RF), as well as the Bayesian optimization method, to obtain the model’s hyperparameters. The SEE model was trained and tested using the PROMISE repository and the ISBSG dataset. The AdaBoost ensemble learning and bayesian optimization-based RF approach, according to the author, outperforms. The AdaBoost-based model also rates the relevance of each feature, making it a viable tool for estimating software work.

Three filter feature selection methods are used by Hosni, Idri, and Abran (2021); correlation-based feature selection (CFS), RReliefF, and linear correlation. In this study, the authors used ensemble modeling with the four machine learning models kNN, SVR, MLP, and decision tree. They divided the ensemble modeling into three categories: ensemble with no feature selection, ensemble with one feature selection, and ensemble with a different feature selection. According to the author, ensemble members can produce estimates that are more accurate by using various feature subsets as opposed to using the same feature subset or all of the available characteristics.

Hosni, Idri, and Abran (2017) used filter feature selection methods namely; correlation and rrelieff feature selection techniques. On six separate datasets; Albrecht, COCOMO81, China, Desharnais, Kemerer, and Miyazaki. The applied estimation model is an ensemble learning strategy with kNN, SVR, MLP, and decision trees. According to the authors, the ensemble learning model without the use of feature selection out-performed when applied with feature selection techniques in terms of accuracy.

Pospieszny, Czarnacka-Chrobot, and Kobylinski (2018), proposed three machine learning models; SVM, MLP and GLM, and ensembled them to obtain an average of the three models. The ensemble approach outperforms the effort prediction when each algorithm is applied individually.

To conclude, the authors in the previous works used ensembled learning, which combines the predictions from multiple models and extracts the best out of it. These studies collectively contribute to advancing SEE methodologies, offering improved accuracy and robustness in effort estimation for software projects. This paper also aims at the same but with the objective to extract the best models from the pool of ML models and then apply them to the datasets. The ensemble models are less interpretable as the output is hard to predict and understand. As discussed in a survey by Mahmood et al. (2022) the ensemble approaches outperform individual models. But there are a few advantages as well; ensemble modeling minimizes modeling method bias, and reduces variance and thus decreasing the likelihood of over-fitting.

Motivation

In order to keep up with the current demand for software development processes, the project manager must anticipate in beforehand how long the project development and how much effort are required in order to successfully develop the projects. The effort (cost) to complete software development is the key risk element. When considered in this light, the peculiarities of developing software projects can make the cost estimation process too challenging as discusses by Arau´jo, Soares, and Oliveira (2012). Due to this, software engineering still considers effort estimation as a challenge. Additionally, the emergence of diverse datasets and the need for more robust and adaptable models further drive the motivation for developing and evaluating new methodologies in SEE. Overall, the goal is to incorporate the phenomenon of discretization and multistep DES to enhance the effectiveness and reliability of effort estimation in software projects, ultimately contributing to improved project planning, resource allocation, and project success.

Methodology

The Background discussed the literature by the researchers, in addition, the section discusses our motivation for proposing a novel approach to the problem. This section discusses the description of datasets utilized in our research. The definitions of the proposed approaches along with their advantages and disadvantages are mentioned. Lastly, the evaluation parameters for validating the performance of the proposed model are also specified for clear understanding.

Taking this challenge and the work done by academicians, and researchers into consideration, we aim to create a model to minimize the selection process of classification techniques and speed up the decision-making of the analyst and experts in the field. The paper proposes an approach for selecting the most appropriate machine learning model from a given set of candidates and obtain the highest accuracy in terms of performance metrics.

Dataset Description

The four publicly available datasets China, Kitchenham, Maxwell, and Desharnais are used in this research to create a model for determining the effort needed for novel software development initiatives. In the area of estimating software effort, these datasets are the most often used.

J.M. Desharnais acquired 81 project data from a Canadian software business in the late 1980s, forming the Desharnais dataset by Desharnais (1989) and Mair et al. (2000). The original dataset has 12 attributes; however, the ProjectID attribute was left out of this study’s analyzation because it is irrelevant to the study’s goals. All of the variables in this dataset are numeric, and the left-hand side lists 10 independent qualities and one dependent attribute (effort), with the exception of the nominal Language attribute.

The 62 projects in the Maxwell dataset, that were only formed by Maxwell (2002), was accomplished between 1985 and 1993. Each project has 27 attributes that are all numerical and are used to summarize the project. A total of 27 independent features and one dependent feature (effort) as in Li (2009).

The 499 projects in the China dataset by Yun (2010) have a total of 19 attributes. In the China dataset, the Function Point serves as the measuring unit. In 2010, Tim Menzies released the dataset for the first time on the PROMISE repository. There isn’t any ambient information provided with the data.

The Kitchenham dataset was obtained from an American-based MNC Computer Science Corporation (CSC). It was released for the first time in 2002 by Kitchenham et al. (2002). The dataset was gathered from software projects that CSC worked on between 1994 and 1999. Total of 145 software and maintenance projects in the dataset, each with 10 attributes as in Tsunoda (2017). Start and expected completion dates are also included in the attributes. Function point is used to measure project size.

Data Preprocessing

Data Pre-processing of the input data in this research is the first and most important step in preventing undesired noise and artifacts. This sub-section gives a description of the employed methodology for pre-processing the data.

Discretization

The problem“Software Effort Estimation” is a regression problem, with continuous output variables. A regression problem can be changed into a classification. For instance, the attribute that needs to be forecasted might be divided into distinct classes or buckets. The process is frequently referred to as discretization as in Brownlee (2017), and the output variable is classified with ordered labels called ordinal. In order to transform the task from regression to classification, we discretized the numerical target variables into a set of classes or intervals. This is important since the machine learning methods we adopted only work with minimal modeling and do not make predictions about grayscale (continuous) values but about discrete (labeled) ones, in other words, paper works with the machine learning algorithms for the classification of labeled data. We balanced the data and labeled it into classes of five in each China, Desharnais, and Kitchenham datasets and four classes in Maxwell dataset as in Radlinski and Hoffmann (2010), the labeled explanation is depicted in Table 1. The datasets used in our work are discretized on the basis of the target field i.e Effort.

Table 1. Labeled dataset description with effort ranges.

		Classes (Effort Ranges)
Dataset	Class 0	Class 1	Class 2	Class 3	Class 4
China	0–1000	1001–2500	2501–5000	5001–10000	10000 & above
Desharnais	0–1500	1501–3000	3001–4500	4501–8000	8000 & above
Kitchenham	0–750	751–1400	1401–2100	2101–4000	4000 & above
Maxwell	0–2000	2001–5000	5001–10000	10000 & above	–

Data Normalisation

The datasets selected for the experiment are normalized using MinMax Pedregosa et al. (2011) scaling transformation in the range of 0 and 1. The minmax scaling technique preserves the shape of the original data distribution. Due to the wide scale of the feature range, the minmax scaling helps out the performance of the models over the datasets.

Feature Selection

As discussed earlier in data preprocessing, the paper targets software effort estimation of a regression problem, which is been converted into classification using the phenomenon of discretization. The relevant optimal feature set is selected using the SelectKBest as in Brownlee (2019) classification filter feature selection technique. The feature selection technique selects the best relevant features that have the greatest impact on the target variable, which proportionally increases the performance of the machine learning models. The selectKBest filter feature selection technique is a white box technique that extracts the relevant features according to k’s highest score. In this research, the value of k is 7 for the Kitchenham dataset, 10 for Desharnais and China datasets, and 12 for the Maxwell dataset assigned using the hit-and-trial approach.

Machine Learning Models

In the discipline of machine learning, often referred to as a branch of artificial intelligence, the primary emphasis is on the creation of techniques and systems that will enable computers to learn and carry out responsibilities on their own. Machine learning techniques, which have some similarities to the human mind, enable us to tackle complexities quickly. Machine learning involves training algorithms on data and then making predictions or taking actions based on the learned patterns. Machine learning algorithms have been recommended as an alternate method to estimate software effort in the last 20 years or so.

Dynamic Ensemble Selection (DES)

A subset of the ensemble members is automatically chosen using the ensemble learning technique known as ”dynamic ensemble selection” as discussed in Cruz et al. (2020) when formulating a prediction. This method requires fitting a large number of machine learning models to the training dataset, and then selecting the models that are expected to perform best when making a prediction for a given test case, all while taking into account the peculiarities of the target to be predicted. The selection mechanism considers the accuracy, diversity, and/or stability of the models in the pool and selects a set of models that complement each other and improve overall performance.

For the dynamic selection in the current approach, several distinct classifiers are used. Though not all of these classifiers adopt this same technique, some of them are based on local accuracy. The paper goes through multiple machine learning strategies that are applied to extract the best suitable model from the pool of models and then label discretized effort using DES again i.e. Multi-step DES (MS-DES) approach is been proposed. These techniques are both parametric and non-parametric which have been selected such as kNN, Random Forest, and Extra Tree are some non-parametric while Naive Bayes, MLP, and Logistic Regression are some parametric, this difference will enable us to reveal what kind of a model works better on software effort estimation datasets. The classifiers employed in the first dynamic selection method proposed to extract the best classification models in this work are shown in Table 2.

Table 2. Classification algorithms.

Algorithm	Parameter Description	Initials
AdaBoost	base classifier = RepTree	ADA
Bagging	base classifier = RepTree	BG
Decision Tree	criterion=“gini”, min samples leaf = 2	DT
Extra Tree	n estimators = 10, min samples leaf = 2	ET
Logistic Regression		LR
J48	criterion =”entropy”, random state = 100, max depth = 3	J48
k Nearest Neighbours	k = 3, 5, 7	kNN
Multi-layer Perceptron	hiddenlayer = 1, learningRate = 0.01	MLP
Naive Bayes		NB
Random Forest	n estimators = 100, min sample leaf = 1	RF
Support Vector Machine	kernel = rbf, degree = 3	SVM
XGBoost	base classifier = RepTree	XGB

Some standard and defined ensemble learning by Dong et al. (2020) methods such as AdaBoost, Bagging, Extra Tree, and Random Forest are incorporated into the pool. Decision trees with one level, or Decision trees with only one split, are the most popular algorithm used with AdaBoost. An ensemble meta-estimator known as a bagging classifier fits base classifiers one at a time to random subsets of the original dataset, and then combines each prediction (either by voting or by averaging) to get the final prediction. Extremely Randomized Trees Classifier, also known as Extra Trees Classifier, is a form of ensemble learning technique that combines the findings of various de-correlated decision trees gathered in a”forest” to produce its classification outcome. Random forest, which is again an ensemble meta-estimator, which fits multiple-decision trees and uses averaging for final prediction.

Extreme Gradient Boosting (XGBoost) is a distributed, scalable gradient-boosted decision tree (GBDT) machine learning framework, it offers parallel tree boosting. A feed-forward artificial neural network model called a multi-layer Perceptron classifier uses stochastic gradient descent to improve the log-loss function is utilized in Ray (2019). The pool includes some other machine learning models such as k Nearest Neighbors, support vector machines, logistic regression, decision trees, naive Bayes, and J48 as in Mahesh (2020).

Proposed Work

The primary goal of our paper is to extract the models from the pool of models, which work effectively in order to predict effort. The secondary goal is to achieve the highest precision of estimates for developing software projects. To do this, the paper proposed a multi-step dynamic ensemble selection (MS-DES), a system using a variety of machine-learning classification approaches that will be constructed and then applied to a pool of classification algorithms, in order to extract the best classification models. Then again propose dynamic ensemble selection (DES), which utilizes extracted classification models in order to improve effort estimation accuracy for software projects datasets as in Garc´ıa et al. (2018). The algorithmic description of the proposed Multi-step Dynamic Ensemble Selection (MS-DES) is depicted in 1.

Dynamic ensemble selection strategies are methods that pick a group of classifiers as opposed to only one. The ensemble of classifiers is made up of all base classifiers that meet a minimum degree of competency. Through the process of fitting many models, each with a low error rate, and then combining them into an ensemble that may perform better, DES may improve the accuracy of any learning system. The first step DES identifies the best relevant models for the classification of labeled datasets (China, Maxwell, Desharnais, and Kitchenham) depending on the training data. In the next step, DES is then applied with the extracted models which calculate the parameters based on the canonical approach to dynamic ensemble selection: k-Nearest Neighbor Oracle (KNORA); Eliminate, Union, and Meta-DES.

The overall procedure is divided into three phases; data preprocessing, proposed work (multi-step dynamic ensemble selection), and model evaluation, which are illustrated in Figure 1. In phase 1: Data Preprocessing; the datasets are refined using three methods; Discretization, Normalisation and Feature selection. The datasets are then split into training and testing data. The proportion of train to test data is 70:30. After refinement of the data, the preprocessed data is provided for training and testing purposes to the proposed MS-DES model. During the training phase, the model learns from the training data to capture patterns and relationships within the dataset. Following training, the model’s performance is evaluated using the testing data to assess its ability to generalize to unseen instances accurately.

Figure 1. Proposed model.

In phase 2: Multi-step Dynamic Ensemble Selection; initially, the pool of classifiers is trained using the training data, allowing each classifier to learn patterns and relationships within the dataset. Once trained, the first step of the DES process is applied. DES dynamically selects the most pertinent classifiers from the pool for making predictions. This selection is based on the classifiers’ performance and effectiveness in capturing the intricacies of the data. Following the selection of relevant classifiers, the next step in the MS-DES phase involves employing DES again.

Table

Agorithm 1 Proposed algorithm for classification of labelled effort ranges
1	Results: Multi-Step Dynamic Ensemble Model for Software Effort Estimation
2	Input: SEE Datasets; China; Desharnais, Maxwell, and Kitchenham
3	Output: Predict labelled Effort ranges
4	Procedure
5	Step 1: Utilizing data preprocessing techniques, to refine data for classification:
6	1.1 The regression data with the target variable is converted into
7	classification data, using discretization;
8	1.2 The data obtained is now normalized using minmax scaling
9	transformation technique within;
10	feature range = (0,1)
11	1.3 The feature selection technique is now applied to the labelled data
12	to extract the relevant features subset from the complete set of
13	features;
14	Step 2: Apply the first step dynamic ensemble selection technique (DES) is applied to the pool of classification models; ADA, BG, DT, ET, LR, J48, kNN, MLP, NB, RF, SVM, and XGB, to extract the best suitable models from the pool;
15	Step 3: The extracted classification models; classifier 1, classifier 2.. classifier n, are again ensembled into the second step dynamic ensemble selection (DES), in order to predict labelled effort ranges.;
16	Step 4: The accuracy of the proposed multi-step DES model is calculated using the canonical approach to dynamic ensemble selection that is k-Nearest Neighbor Oracle (KNORA);
17	Step 5: Generate and store obtained confusion matrix parameters values for;
18	1.1 KNORA-Eliminate (k = 3/5/7,voting =^′ hard^′);
19	1.2 KNORA-Union (k = 3/5/7,voting =^′ hard^′);
20	1.3 Meta-DES (k = 3/5/7,DSEL perc = 0.5);
21	Step 6: The final evaluated confusion matrix parameter values are validated using the ROC curve between the true positive and false positive rates;
22	END

more detailed understanding of the effort required for various aspects of the software development process, allowing for more precise estimation.

Finally phase 3: Model Evaluation; the trained MS-DES model is evaluated using testing data in terms of the canonical approach to DES and confusion metrics parameter values are generated and validated in form of results.

Results and Discussion

The section discusses the results obtained after implying the proposed approach in terms of evaluation metrics. The evolution of the proposed model is discussed. To validate the performance measure, the receiver operating curve is created and depicted in this section. In the proposed multi-step DES, the outcome of the first DES is the extracted classifiers relevant to the prediction. This set of classifiers is again employed with next-level DES. The classifier extracted from the first DES depends upon the input training data, by fitting a number of models, each with a low error rate. The extracted classifiers with four different datasets may vary, which are shown in Table 3.

Table 3. Outcome of first-step DES.

Datasets	Classifier 1	Classifier 2	Classifier 3	Classifier 4	Classifier 5
China	RF	ADA	MLP	ET	–
Maxwell	RF	XGB	ET	kNN	–
Desharnais	MLP	NB	kNN	ET	RF
Kitchenham	LR	RF	NB	DT	ET

Evaluation Metrics

The outcomes of the proposed model are analyzed using the confusion metrics parameters such as accuracy, sensitivity, F1-score, precision, or Positive Predicted Value (PPV), and r-squared value on the multiple datasets. The formulation of the abovementioned confusion matrix parameters is depicted in Table 4.

Table 4. Confusion matrix parameter calculation.

Parameter	Formula
Precision or Positive Predicated value(PPV)	$\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}{ }^{\ast }100$
Recall or Sensitivity	$\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}{ }^{\ast }100$
F1-Score	$\frac{2_{\ast }(\mathit{PPV}\ast \mathit{Sensitivity})}{\mathit{PPV}+\mathit{Sensitivity}}\ast 100$
Accuracy	$\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{FN}+\mathit{FN}+\mathit{TN}}\ast 100$
R-square Value (R^2)	$1-\frac{\mathit{RSS}}{\mathit{TSS}}$

Model Evolution

The proposed MS-DES model is evolved in an accumulative manner, the labeled datasets are earlier tested with the individual models; NB, RF, ET, MLP, LR and kNN. The performance of the models has been noted in terms of the confusion metric parameter; accuracy. Later, this is compared with the proposed multi-step DES applied with extracted classification models after the first step DES. The performance comparison of the proposed model with individual models is shown in Table 5.

Table 5. Accuracy of individual models vs dynamic ensemble selection.

Accuracy	Machine Learning Models						DES
Dataset	NB	RF	ET	MLP	LR	kNN	KNORA-E	KNORA-U	Meta-DES
Maxwell	0.6842	0.8947	0.4737	0.4211	0.4737	0.3158	0.8947	0.8947	0.8421
China	0.8467	0.9867	0.4733	0.7867	0.5600	0.7133	0.9867	0.9867	0.9800
Desharnais	0.8800	0.1600	0.6400	0.4800	0.5600	0.8800	0.8800	0.8000	0.8000
Kitchenham	0.7500	0.8863	0.4091	0.4091	0.5049	0.7950	0.8863	0.8863	0.8409

The selectKBest filter feature selection extracts the relevant subset of features set from all the features, which extracts the relevant features according to k’s highest score, in this research the value of k is 7 for the Kitchenham dataset, 10 for Desharnais and China datasets and 12 for the Maxwell dataset. The evolution shows Table 6 which compares the performance of the proposed MS-DES model in terms of accuracy, and Table 7 compares the performance of the proposed MS-DES model in terms of rsquared value with and without feature selection.

Table 6. Accuracy comparison of DES with and without feature selection.

Accuracy	DES without Feature Selection			DES with Feature Selection
Dataset	KNORA-E	KNORA-U	Meta-DES	KNORA-E	KNORA-U	Meta-DES
Maxwell	0.8947	0.8947	0.8421	0.9474	0.9474	0.9474
China	0.9867	0.9867	0.9800	0.9930	0.9930	0.9930
Desharnais	0.8800	0.8800	0.8000	0.9200	0.8800	0.8400
Kitchenham	0.8863	0.8863	0.8409	0.9545	0.9545	0.9318

Table 7. R-Square value comparison of DES with and without feature selection.

R Square	DES without Feature Selection			DES with Feature Selection
Dataset	KNORA-E	KNORA-U	Meta-DES	KNORA-E	KNORA-U	Meta-DES
Maxwell	0.7589	0.7595	0.4589	0.9518	0.8797	0.8553
China	0.9867	0.9914	0.9845	0.9960	0.9923	0.9884
Desharnais	0.9388	0.9286	0.9107	0.9592	0.9286	0.9286
Kitchenham	0.9358	0.9358	0.798	0.9816	0.9743	0.9615

The final results, feature selection based on the proposed Multi-step Dynamic Ensemble Selection gives the more accurate results, in terms of confusion metrics parameters; accuracy, precision, recall, and f1-score and r-squared value as shown in Table 8. The highest accuracy of 94.74%, 99.30%, 92%, and 95.45% are obtained with Maxwell, China, Desharnais, and Kitchenham datasets respectively. The highest r-squared value (close to 1) tends to be the effective and efficient performance model; the results appeared in terms of the r-squared value i.e. 95.18%, 99.60%, 95.92%, and 98.16% in Maxwell, China, Desharnais, and Kitchenham datasets respectively manifest that the proposed approach outperforms.

Table 8. Confusion metrics parameters values obtained based on proposed work.

Dataset	Accuracy	Precision	Recall	F1 Score	R Square Value
Maxwell	0.9474	0.9474	0.9474	0.9474	0.9518
China	0.9930	0.9958	0.9944	0.9950	0.9960
Desharnais	0.9200	0.9377	0.9377	0.9333	0.9592
Kitchenham	0.9545	0.9545	0.9545	0.9545	0.9816

ROC

The receiver operating characteristic (ROC) curve is obtained to forecast effort ranges and to validate the performance measure using the confusion metric. The ROC curve is used to assess the number of true positives and false positives in test results. The ROC curve has been shown with the true positive rate, which can be found along the Y-axis, and the false positive rate, which can be found along the X-axis. Figure 2 illustrates the ROC curve for the proposed feature selection-based multi-step DES model. The RoC area under the curve score obtained for employed datasets is depicted in Table 9.

Figure 2. RoC curves for proposed model.

Table 9. ROC AUC score of datasets.

Dataset	ROC AUC Score
Maxwell	0.9914
China	0.9998
Desharnais	0.9807
Kitchenham	0.9946

The proposed Multi-step Dynamic Ensemble Selection (MS-DES) model is developed incrementally, with initial testing conducted on individual models such as Naive Bayes, Random Forest, Extra Trees, Multilayer Perceptron, Logistic Regression, and k-Nearest Neighbors using labeled datasets. Performance is evaluated based on the accuracy metric. Subsequently, the MS-DES model is compared with these individual models after the initial DES step, showcasing their performance in terms of accuracy. Additionally, feature selection is performed using the selectKBest filter method to extract relevant subsets of features from the datasets. The evolution of the MSDES model’s performance with and without feature selection is documented in tables, highlighting improvements in accuracy and R-squared values. The results indicate that the MS-DES KNORA-E model with feature selection yields the most accurate predictions, achieving high accuracy and R-squared values across different datasets. Furthermore, the receiver operating characteristic (ROC) curve is utilized to assess the model’s performance in forecasting effort ranges, with the area under the curve

(AUC) score reported for each dataset. The proposed MS-DES model will contribute to the advancement of software engineering practices by providing more accurate effort estimation, enabling better decision-making, optimizing resource allocation, reducing project risks, and facilitating continuous improvement in project management processes.

Discussion and Threats to Validity

The results of a performance analysis of the three standard canonical methods of dynamic ensemble selection, k-Nearest Neighbor Oracle (KNORA); Eliminate, Union and Meta-DES suggest the following. The results suggest, KNORA-E outperforms other canonical approaches in terms of accuracy and r-squared value. It is clear from the suggested MS-DES approach that two levels of DES are used; first to separate the appropriate classifiers from a vast collection of classifiers. The first tier chooses a group of classifiers and the most suitable combiner for each test sample. In order to predict labeled effort ranges, the second-level DES is applied which employs chosen group of classifiers against each test sample. k- Nearest Neighbor Oracle, the standard dynamic ensemble selection method, is utilized to compute the performance of the model, which effectively and efficiently generates higher accuracy and quality performance of the model.

In order to validate the performance of the proposed MS-DES model, the paper compares the performance of the proposed model with existing models from the literature. The comparison is based on the r-squared value, the coefficient of determination between 0 and 1. R-squared is a statistical analytic method for evaluating the applicability and reliability of model performance. This statistic shows the proportion of the dependent variable’s variance that the independent variables account for collectively.

In comparison with existing work by authors in the same domain, an r2-squared value obtained by Malhotra and Jain (2011) is 93% with the China dataset, Ur Rehman, Ali, and Jan (2021) obtained r2-square value of 72%, 19%, and 88% with China, Desharnais, and Kitchenham datasets respectively. Burgess and Lefley (2001) acquired an r2-square value of 82% with the Desharnais dataset. Hidmi and Erdogdu Sakar (2017) obtained an accuracy of 91.35% with the Desharnais and 85.48% with Maxwell datasets. On comparing our proposed approach with the existing methods, the paper justifies the proposed MS-DES approach is outplaying other techniques. The proposed model performs efficiently with datasets and may yield generality for predicting effort in software companies on the basis of previously developed products.

We obtained significant results from this research for effort estimation to enhance the software development process. The reliability of our findings is, however, still threatened by a few factors (which may be related to the generalization of the data obtained). When testing our methodology, we chose four publicly available datasets that might pose the toughest threat. There are several additional publicly available datasets for effort estimation, such as ISBSG datasets, and the datasets selected only provide information for a small number of projects. However, we believe that our findings may be replicated using different datasets and that our suggested methodology would work effectively with any effort estimation dataset.

The software metrics were chosen to build predictors that pose another risk to validity. Even considering that we only used standard code measurements for labeled data; confusion metrics, we recognize the possibility of varying outcomes based on the software metric used. Instead of evaluating the effectiveness of predictors created using various kinds of software metrics, we want to investigate the role of a confined metric set in estimating effort from the perspective of the trade-off between generality, cost, and accuracy.

Conclusion and Future Work

This work proposes a heterogeneous Multi-step dynamic ensemble selection (MS-DES) approach, composed of a pool of classifiers dynamically chosen by classifiers, to address software development effort estimates. Additionally, using the suggested methodology, it conducts an experimental analysis using four pertinent datasets for software development effort estimation. The datasets were discretized into labeled data based on effort ranges. Since no single model is effective with all the datasets, as discussed earlier, the proposed feature selection-based MS-DES model performs effectively in predicting effort ranges, in terms of confusion metrics and resulted in the accuracy of 94.74%, 99.30%, 92%, and 95.45% with Maxwell. China, Desharnais, and Kitchenham datasets respectively. The accuracy of the proposed MS-DES model is calculated using the canonical approach to dynamic ensemble selection which is k-Nearest Neighbor Oracle (KNORA-E/U/Meta-DES); KNORA-E outperforms other canonical approaches in terms of accuracy and r-squared value.

In the future, we intend to implement the model on large datasets available in order to validate the performance of the proposed model. In the proposed model, which chooses the best suitable classifiers from the pool of classifiers, we intend to add more new classification techniques to improve the efficacy of the model and in order to develop an independent model for any classification problem.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

The datasets analyzed during the current study are available in different repositories: Desharnais dataset is available in Promise repository, http://promise.site.uottawa.ca/SERepository/datasets/desharnais.arff, Maxwell, China, and Kitchenham datasets are available in Zenodo repository their respective web links are; https://doi.org/10.5281/zenodo.268461, https://doi.org/10.5281/zenodo.268446, https://doi.org/10.5281/zenodo.268457.

Additional information

Funding

This research was funded by Horizon Europe (HORIZON) project number 101138678 — ZEBAI