A novel bias-alleviated hybrid ensemble model based on over-sampling and post-processing for fair classification

Abstract

With the rapid development of machine learning in the field of classification, the classification fairness has become the research emphasis second to prediction accuracy. However, the data bias and algorithmic discrimination that affect the fair classification of models have not been well resolved, which may damage or benefit the specific groups related to the sensitive attributes (e.g. age, race, and gender). To alleviate the unfairness of the classification model, this study proposes a novel bias-alleviated hybrid ensemble model (BAHEM) based on over-sampling and post-processing. First, a new clustering-based over-sampling method is proposed to reduce the data bias caused by the imbalance in label and sensitive attribute. Then, a stacking-based ensemble learning method is employed to obtain the higher performance and robustness of the BAHEM. Finally, a new classification with alternating normalisation (CAN)-based post-processing method is proposed to further improve the fairness and maintain the accuracy of the BAHEM. Three datasets with different sensitive attributes and four evaluation metrics were used to evaluate the prediction accuracy and fairness of the BAHEM. The experimental results verify the superior fairness of the BAHEM with little accuracy reduction.

KEYWORDS:

• Machine learning
• fair classification
• over-sampling
• ensemble learning
• post-processing

1. Introduction

With the extensive application of machine learning models for solving various classification problems, including credit scoring (Dastile et al., 2020), crime prediction (Kim et al., 2018), software performance prediction (Liu et al., 2021), and loan applications (Wang et al., 2020), social fairness has received increasing attention. Fair classification of models can be affected by data bias and algorithmic discrimination. The raw datasets that are used to train the machine learning models may contain human biases that are intended or unintended, such as the biases to gender, race, or age. The models trained using the biased datasets will learn and extract the data bias and discriminate against certain groups. Therefore, a fair machine learning model is urgently required.

The definition of the fair classification problem was firstly proposed by Kamiran and Calders (2009), and it is mainly resulted from the unbalanced training of some sensitive attributes in the machine learning models due to the imbalanced distribution of training data or the data bias. For example, if gender is taken as a sensitive attribute and the male-female ratio in a data set is 1:100, then the samples containing female will dominate the training stage of the model, thus making the model's prediction of the male samples inaccurate. In the field of credit scoring, traditional machine learning methods only consider the distribution of labels and tend to make full use of all available information to improve the prediction accuracy as much as possible. If the imbalanced distribution of sensitive attributes does exist, the classification result of the model may be biased towards the majority class, leading to the unfairness of the model. Therefore, a model that can deal with the trade-off between accuracy and fairness is needed, and its validity should be verified by experiments on the real datasets. The aim of research on the fair classification problem is to improve the fairness of the models with little reduction in prediction accuracy by alleviating data bias and algorithmic discrimination.

On the one hand, the datasets in the real world are commonly imbalanced, both in labels and sensitive attributes. Imbalanced labels may reduce the prediction accuracy of machine learning models, and imbalanced sensitive attributes may cause a discriminative impact on unprivileged groups, affecting the classification fairness of models. Researchers have proposed several methods to balance the datasets, including synthetic minority over-sampling technique (Chawla et al., 2002), adaptive synthetic sampling (ADASYN; He et al., 2008), and balance cascade (Liu et al., 2008). However, existing sampling methods simply consider the imbalance in labels while ignoring the imbalance in sensitive attributes. Therefore, a sampling method that considers both imbalance in the labels and the sensitive attributes is required.

On the other hand, to improve the performance and robustness of the machine learning models, ensemble learning methods are widely adopted, including extreme gradient boosting (XGBoost; Chen & Guestrin, 2016), light gradient boosting machine (LightGBM; Ke et al., 2017), and gradient boosting decision tree (GBDT; Friedman, 2001). Although deep learning methods have been widely used to solve a variety of classification problems with excellent performance, such as gesture recognition (Qi et al., 2021) and sarcasm identification (Onan & Toçoğlu, 2021), it is difficult to analyse the fairness of deep learning models due to their nature of black box. Therefore, this study analyses the fairness of machine learning models, and uses ensemble learning technology to enhance the fairness of models and maintain their accuracy. In addition, classification with alternating normalisation (CAN) was adopted to readjust the predicted results so as to further improve the classification performance of the machine learning models (Jia et al., 2021). However, these methods only focus on the performance of machine learning models while ignoring the classification fairness of models.

Therefore, the motivation of this study is to provide an ensemble model to alleviate the data bias and algorithmic discrimination for fair classification. The main contributions of this study are listed as follows:

1. A novel bias-alleviated hybrid ensemble model (BAHEM) based on over-sampling and post-processing is proposed in this study to enhance the classification fairness and maintain the accuracy of ensemble models.

2. A new clustering-based over-sampling method is proposed to balance the label and sensitive attribute automatically by generating new samples according to the data distributions. The clustering method can improve the sampling efficiency by separating the dataset into several subsets.

3. A stacking-based ensemble learning method is employed to adaptively select and integrate competent base classifiers with higher average rankings of accuracy and fairness, which are output from the first layer of stacking. Hence, the performance and robustness of the proposed BAHEM are improved.

4. A new CAN-based post-processing method is proposed to further improve the fairness and maintain the accuracy of BAHEM by modifying the prediction results with higher uncertainty and corresponding to the majority of the sensitive attribute.

5. Three datasets with different sensitive attributes and four evaluation metrics (two traditional performance metrics and two fairness metrics) are adopted to evaluate the classification performance and fairness of the BAHEM.

The remainder of this study is organised as follows. In Section 2, related work on data sampling methods, ensemble learning methods, and fair classification methods are reviewed. In Section 3, details of the proposed BAHEM are presented. The experimental settings, including the datasets, evaluation metrics, and parameter settings of the models, are presented in Section 4. In Section 5, the experimental investigation and comparison of the performances of the BAHEM and other benchmark models are described. The conclusions and suggestions for future work are presented in Section 6.

2. Related work

The proposed BAHEM in this study primarily involves three aspects: data sampling methods, ensemble learning methods, and fair classification methods. The literature on these three aspects is reviewed in this section.

2.1. Data sampling methods

The problem of imbalanced datasets is one of the greatest challenges in training machine learning models, because the models trained by the imbalanced datasets may be biased and favour the majority class (Thabtah et al., 2020). To reduce the negative influence of imbalanced datasets, two different data sampling methods are mainly used in the current research: one is under-sampling method and the other is over-sampling method. The under-sampling methods balance the datasets to reduce the size of the majority class by removing some of the majority samples. For example, Onan (2019) proposed a consensus clustering based under-sampling approach which combined five different clustering algorithms to balance the dataset. Devi et al. (2019) analysed the effects of data imbalance in machine learning models and proposed a Tomek-link under-sampling algorithm to solve the data imbalance. Guzmán-Ponce et al. (2021) proposed a two-stage under-sampling method that combines a clustering method for filtering the majority samples and a graph-based procedure for determining the appropriate imbalance ratio (IR) for each subset. Jiang et al. (2022) proposed a boosting random forest with static under-sampling and ensemble methods to reduce the overlap between classes. However, the under-sampling methods may lose some potential informative data while removing the majority samples.

In contrast to the under-sampling methods that balance the datasets by removing the majority samples, the over-sampling methods generate minority samples to balance the datasets. For instance, Tao et al. (2019) proposed a real-value negative selection over-sampling method that can generate minority samples without reusing minority samples from the original dataset and avoid generating noise samples. Puntumapon et al. (2016) proposed a clustering-based over-sampling method to reduce model overfitting and to improve the generalisation of minority samples that are generated. However, over-sampling methods in these researches mainly consider how to improve the prediction accuracy of the model, while ignoring the classification fairness of the model.

Therefore, in this study, a new clustering-based over-sampling method is proposed that can automatically balance the label and sensitive attribute in the datasets according to their IR, thereby improving the adaptability to imbalanced datasets and the classification fairness of the model.

2.2. Ensemble learning methods

Ensemble learning methods, one of the most effective ways to improve the performance of base classifiers, have been widely adopted and extended by researchers to solve various classification problems, such as text classification (Onan, 2018) and sentiment classification (Onan et al., 2017). Ensemble learning methods mainly include bootstrap aggregation (bagging; Breiman, 1996), boosting (Freund & Schapire, 1996), and stacking methods (Wolpert, 1992). Onan et al. (2016) integrated statistical keyword extraction methods by bagging and boosting and verified the effectiveness of the ensemble methods in the field of text classification. Among these methods, stacking methods have been proved to be an efficient and flexible ensemble learning method, which integrates the prediction results of base classifiers to obtain the final prediction results with increasing prediction accuracy. As an example, Xu et al. (2020) combined k-means clustering and ensemble learning to forecast the price of stock market. Potha et al. (2021) proposed a sophisticated extrinsic random-based ensemble method to detect the malware and demonstrated the effectiveness of ensemble learning methods. Xia et al. (2021) proposed a weighted stacking ensemble with sparsity regularisation, which adjusts the weights of the base classifiers according to the label correlations in multi-label classification problems. Li and Li (2022) improved adaptive boosting (AdaBoost) algorithm with weight adjustment factors to handle the imbalanced data classification with minority samples. In our previous study, Zhang et al. (2021) proposed a stacking ensemble method that combined the outlier detection and sampling methods to boost the prediction accuracy and generalisation ability of model.

However, existing stacking ensemble methods select and integrate the base classifiers with higher prediction accuracy while ignoring the classification fairness of the base classifiers, which may cause the obtained ensemble model to be biased toward certain groups. Therefore, in this study, a stacking-based ensemble learning method is employed to select competent base classifiers with higher accuracy and fairness to improve the classification fairness of the model.

2.3. Fair classification methods

The methods used to improve the classification fairness of the model can be separated into three categories: the pre-processing methods, the in-processing methods, and the post-processing methods.

Pre-processing methods aim to reduce data bias and ensure the classification fairness of the model by transforming the distribution of the datasets (d’Alessandro et al., 2017). Among the existing pre-processing methods, the most popular ones are instance sampling (Iosifidis et al., 2019), transformation (Calmon et al., 2017), and label swapping (Kamiran & Calders, 2012). For instance, Petrović et al. (2022) developed a sample re-weighting method to reduce data bias and improve classification fairness by learning sample weighting functions using adversarial training algorithms.

In-processing methods modify the state-of-the-art machine learning algorithms by changing the constraints to enhance the classification fairness of the model (Mehrabi et al., 2021). For instance, Iosifidis and Ntoutsi (2019) extended AdaBoost to a fairness-aware classifier that considers the classification fairness of each classifier while updating the sample weights. Zafar et al. (2017) proposed a notion of fairness as the constraint in the objective function of machine learning models.

Post-processing methods modify prediction results to enhance the classification fairness of the model (Iosifidis et al., 2019). As an example, Fish et al. (2016) designed a boosting classifier that improves classification fairness by modifying the decision boundary of the classifiers to protect unprivileged groups. Lohia et al. (2019) proposed a fairness post-processing method that ranks samples using a bias reduction algorithm to enhance both group fairness and individual fairness.

However, the fair classification methods in these three categories improve the classification fairness of the model by sacrificing much prediction accuracy, which may cause unexpected losses when using the model for decision making. Therefore, inspired by the pre-processing methods and post-processing methods, a novel BAHEM based on over-sampling and post-processing is proposed in this study, which includes a new clustering-based over-sampling method as the pre-processing method and a new CAN-based post-processing method to alleviate data bias and improve the classification fairness of the model with little reduction in prediction accuracy.

3. Model

In this study, a novel BAHEM based on over-sampling and post-processing is proposed to ensure the fairness and alleviate the data bias of the classification model. Figure 1 shows the framework of the proposed BAHEM. Three methods (i.e. clustering-based over-sampling method, stacking-based ensemble learning method and CAN-based post-processing method) are proposed as the three stages of the BAHEM. First, the original dataset is separated into two parts: training data and testing data. Then, the training set and validation set are obtained by separating the training data. Second, the training set is clustered into subsets and further over-sampled to obtain a balanced training set. Further, the base classifiers trained by the balanced training set are selected according to the accuracy and fairness of each classifier and integrated into the stacked ensemble model, which is used to obtain the prediction results by predicting the testing data. The prediction results are modified based on the uncertainty score of each result, and the modified prediction results are regarded as the final prediction results. The process details are presented in the following sub-sections.

Figure 1. Framework of the proposed BAHEM.

3.1. Clustering-based over-sampling method

Data imbalance is a common problem in real-world datasets, which exists not only in labels but also in sensitive attributes. Machine learning models trained using imbalanced data will be biased against some of the classes. Just like the imbalance of labels will affect the accuracy of the model, the imbalance of sensitive attributes will affect the fairness of the model. The clustering algorithm gathers the samples into different clusters, so that the samples in each cluster are as similar as possible, and the samples in different clusters are as different as possible. Therefore, the data imbalance within each class is reduced. Clustering algorithms have been widely used to solve the imbalance problem of datasets (Onan, 2019; Xu et al., 2020). In this study, a new clustering-based over-sampling method that considers the bias in both label and sensitive attribute, is proposed to balance them by comparing their respective IR.

As exhibited in Figure 2, the feature weight of the training set is firstly adjusted. For example, F₁ to F_n represent all features of the training set, and F_s represents the sensitive attribute in the training set. Then, F_s is duplicated as F_s′, which is appended to the training set, resulting in the adjusted training set. Apparently, the weight of the sensitive attribute F_s is increased, causing the subsequent clustering algorithm to pay more attention to the sensitive attribute in the adjusted training set. Subsequently, a clustering algorithm separates the adjusted training set into several subsets (e.g. subset 1 and subset 2) to improve the sampling efficiency. The IR of the label and the sensitive attribute in each subset are calculated and compared. The IR is the ratio of the number of the majority class to that of the minority class. For example, IR_L represents the IR of the label, and IR_SA represents the IR of the sensitive attribute. The proposed method can calculate both IR_L and IR_SA and compare them in each subset, and employ the popular over-sampling method (i.e. ADASYN) to balance them. If IR_SA is greater than or equal to IR_L (e.g. in subset 1), ADASYN is used to balance the sensitive attribute. If IR_SA is less than IR_L (e.g. in subset 2), ADASYN is used to balance the label. Finally, after all the subsets are over-sampled, they are integrated to produce a balanced training set.

Figure 2. Schematic diagram of clustering-based over-sampling method.

3.2. Stacking-based ensemble learning method

Base classifiers commonly have problems such as low accuracy and fairness, which can be alleviated through classifier integration. Therefore, a stacking-based ensemble learning method is employed in the proposed BAHEM to integrate multiple base classifiers with higher accuracy and fairness.

As shown in Figure 3, multiple base classifiers (Clf 1, Clf 2, … , Clf m) in the base classifier pool are trained using the balanced training set. Because accuracy and average odds difference (AOD; Bellamy et al., 2019) are the most commonly used indicators, they are evaluated in selecting the competent base classifiers. After the base classifiers are trained, the accuracy (ACC; Stehman, 1997) and AOD (Bellamy et al., 2019) of each base classifier in the validation set are calculated and sorted, respectively. The top k competent classifiers are then selected according to the average ranking of ACC and AOD. The k selected competent base classifiers (Sclf 1, Sclf 2, … , Sclf k) are further permuted and combined into several ensemble classifiers whose prediction results are used as new features to train the meta classifier. Because Xia et al. (2018) proved the superior performance of logistic regression (LR) as a meta classifier in stacking methods, LR is adopted as the meta classifier. Finally, the stacked ensemble model is obtained.

Figure 3. Schematic diagram of stacking-based ensemble learning method.

3.3. CAN-based post-processing method

Traditional machine learning models are commonly trained using training data and they predict testing data to evaluate the classification performance of the models. Jia et al. (2021) proved that classification performance can be improved by readjusting the prediction results of challenging samples, and they proposed the CAN method. In the post-processing method, how to select and modify the appropriate prediction results is the key to improve the accuracy and fairness of the model. Therefore, a new CAN-based post-processing method is proposed in this study. In contrast to the original CAN method, which only improves the accuracy of the model, the proposed method ensures the classification fairness of the model while maintaining accuracy.

To evaluate the fairness of the model intuitively, the data can be divided into unprivileged group and privileged group according to a sensitive attribute, and the difference between true positive rate and false negative rate in these two groups are calculated respectively, then their average value is taken as the fairness of the model. To improve the fairness of the model, it is necessary to make the difference between true positive rate and false negative rate in both unprivileged and privileged groups as small as possible. The CAN-based post-processing method proposed in this study can select and modify the prediction results with higher uncertainty to reduce the difference. Hence, the fairness of the model is improved.

As depicted in Figure 4, the stacked ensemble model is firstly used to obtain the original prediction results by predicting the testing data. Then, the CAN method, which defines entropy as the uncertainty score, is adopted to calculate the uncertainty score of each prediction result. After all the uncertainty scores of the prediction results are calculated, a threshold is given, and the prediction results with uncertainty scores higher than the threshold are selected. Because the prediction results obtained by the model with an imbalanced dataset will be biased towards the majority class, the datasets are usually balanced by reducing the majority class or increasing the minority class. Similarly, while analysing the fairness of the model, the prediction results will be biased towards the group that belongs to the majority of the sensitive attribute. To ensure that the modified prediction results improve the fairness of the proposed BAHEM, only the prediction results corresponding to the majority of the sensitive attribute are selected for modification. For example, it is assumed that the sensitive attribute and prediction results are both binary and “1” is the majority of the sensitive attribute, then, if the threshold is set as 0.5, all the prediction results with uncertainty scores higher than or equal to 0.5, are selected. Among these selected prediction results, if the corresponding sensitive attribute is “1”, then the prediction result is modified to the opposite results (i.e. “1” to “0”, and “0” to “1”). Finally, the modified prediction results are taken as the final prediction results.

Figure 4. Schematic diagram of CAN-based post-processing method.

4. Experimental settings

4.1. Datasets description

In this study, three standard datasets, namely, Adult (Kohavi, 1996), Bank (Moro et al., 2014), and German (Asuncion & Newman, 2007), from the UC Irvine (UCI) machine learning repository are used to estimate the classification performance and fairness of the proposed BAHEM. Table 1 lists the details of the datasets used in this study, including the sample size, numbers of positive and negative samples, number of total features, and sensitive attributes. In addition, the code of this study is available at Github.¹

Table 1. Detailed information of the datasets.

Dataset	Sample size	Negative samples	Positive samples	Dimension of features	Sensitive attribute
Adult	32561	7841	24720	20	Race
Bank	30488	3859	26629	20	Age
German	1000	300	700	14	Age, sex

4.2. Evaluation metrics

Four evaluation metrics, namely, ACC, AOD, balanced accuracy (BA; Brodersen et al., 2010), and equal opportunity difference (EOD; Hardt et al., 2016) were adopted in this study in order to evaluate the classification performance and fairness of the proposed BAHEM.

ACC is the basic evaluation metric used to indicate the overall performance of a model. The formula for ACC is given in Equation (1). Here, TP, TN, FP, and FN represent true positive, true negative, false positive and false negative, respectively. The higher ACC represents the higher prediction accuracy of the model. (1) $\text{ACC = }\frac{\text{TP+TN}}{\text{TP+TN+FP+FN}}$(1)

BA is an evaluation metric that is commonly adopted to evaluate the performance of a model trained by an imbalanced dataset. BA is calculated using the true positive rate (TPR) and true negative rate (TNR). The formula for BA, TPR, and TNR are given in Equation (2), Equation (3), and Equation (4), respectively. A higher BA represents the higher prediction accuracy of the model. (2) $\begin{array}{rl}\text{BA}& =\frac{\text{TPR + TNR}}{2}\end{array}$(2) (3) $\begin{array}{rl}\text{TPR}& =\frac{\text{TP}}{\text{TP + FN}}\end{array}$(3) (4) $\begin{array}{rl}\text{TNR}& =\frac{\text{TN}}{\text{FP + TN}}\end{array}$(4)

AOD and EOD are used to evaluate the classification fairness of the model, and AOD is calculated using Equation (5). Here, SA = 1 represents the sensitive attribute belonging to the unprivileged group, and SA = 0 represents the sensitive attribute belonging to the privileged group. The formula for FPR is given in Equation (6). TPR_SA = 1 and FPR_SA = 1 represent TPR and FPR in the unprivileged group, respectively, and TPR_SA = 0 and FPR_SA = 0 represent TPR and FPR in the privileged group respectively. The formula for EOD is presented in Equation (7). For a comprehensive comparison, the absolute value of EOD, i.e. |EOD|, is used in the following experimental comparison. A lower AOD or |EOD| represents the higher classification fairness of the model. (5) $\begin{array}{rl}\text{AOD}& =\frac{1}{2}[|\text{TP}{\text{R}}_{\mathrm{SA}=1}-\text{TP}{\text{R}}_{\mathrm{SA}=0}|+|\text{FP}{\text{R}}_{\mathrm{SA}=1}-\text{FP}{\text{R}}_{\mathrm{SA}=0}|]\end{array}$(5) (6) $\begin{array}{rl}\text{FPR}& =\frac{\text{FP}}{\text{FP}+\text{TN}}\end{array}$(6) (7) $\begin{array}{rl}\text{EOD}& =\mathrm{TP}{\text{R}}_{\mathrm{SA}=1}-\text{TP}{\text{R}}_{\mathrm{SA}=0}\end{array}$(7)

4.3. Parameter settings

The original dataset was randomly separated as follows: 20% of the original dataset was used as the testing data, and 80% of the original dataset was used as the training data. Among the training data, 80% was used as the training set and the rest was used as the validation set. In the clustering-based over-sampling method, k-means was used as the clustering method and the number of clustering centres was set as 2; ADASYN was used as the over-sampling method. The k-means and ADASYN were executed by the Python modules “sklearn” and “imblearn”, respectively. In the stacking-based ensemble learning method, XGBoost, GBDT, AdaBoost, random forest (RF), support vector machine (SVM), LR, and LightGBM were used as the base classifiers. XGBoost and LightGBM were executed by the Python module “xgboost” and “lightgbm,” respectively. GBDT, AdaBoost, RF, SVM, and LR were executed by the Python module “sklearn.”

In the CAN-based post-processing method, inspired by Agrawal et al. (2019), Kirar and Agrawal (2019) and Kirar et al. (2022), the ACC and AOD for each dataset using different thresholds of uncertainty scores are shown in Figure 5. Considering that the threshold is generally set as 0.5 by default in classification problems, this experiment compares the threshold values around the default value when selecting parameters, namely 0.4, 0.5 and 0.6. Although the ACC rises with the increase of the threshold for all datasets, the AOD for “Adult-race,” “Bank-age,” “German-age,” and “German-sex” performed well when thresholds were set as 0.6, 0.5, 0.5 and 0.6, respectively. Therefore, the threshold of uncertainty scores for “Adult-race,” “Bank-age,” “German-age,” and “German-sex” were set as 0.6, 0.5, 0.5 and 0.6, respectively, through trial-run experiments. For a fair comparison, all the parameters of the clustering methods, over-sampling methods, and base classifiers were set as default.

Figure 5. Performance comparison between different thresholds on each dataset.

5. Experimental analyzation

In this study, three datasets with different sensitive attributes and four evaluation metrics were adopted to evaluate the classification performance and fairness of the proposed BAHEM. Each evaluation metric was run and calculated 10 times to ensure the reliability of the experiment. The average results of each evaluation metric were used as the performance of the BAHEM. All the experiments were realised on Microsoft Windows 10 operating system, using Python Version 3.7.

5.1. Performance evaluation of baseline classifiers

To evaluate the classification performance and fairness of BAHEM, the baseline results of three datasets with different sensitive attributes were evaluated using four evaluation metrics. The performance of baseline classifiers is presented in Table 2. The German dataset with different sensitive attributes selected can be used as two datasets. To clarify the sensitive attributes of the datasets in the following experimental analysis, the datasets with different sensitive attributes are renamed as “Adult-race,” “Bank-age,” “German-age,” and “German-sex,” respectively.

Table 2. Performance evaluation of baseline classifiers.

Dataset	Base classifiers	ACC	AOD	BA	|EOD|
Adult-race	XGBoost	0.85992	0.02220	0.77105	0.02492
	GBDT	0.86138	0.02610	0.77614	0.03113
	AdaBoost	0.84984	0.01240	0.74887	0.00517
	RF	0.83924	0.04608	0.76614	0.04621
	SVM	0.83438	0.04422	0.74052	0.05594
	LR	0.84524	0.04916	0.76222	0.06149
	LightGBM	0.86796	0.03739	0.79715	0.04482
Bank-age	XGBoost	0.90812	0.04669	0.75311	0.05593
	GBDT	0.90894	0.05162	0.75918	0.06210
	AdaBoost	0.90454	0.04964	0.72872	0.06636
	RF	0.90251	0.05163	0.72488	0.05968
	SVM	0.88860	0.04505	0.60366	0.06627
	LR	0.89954	0.09592	0.68566	0.14589
	LightGBM	0.90764	0.06975	0.76657	0.08475
German-age	XGBoost	0.75450	0.09182	0.65327	0.11385
	GBDT	0.75500	0.08392	0.65821	0.10097
	AdaBoost	0.74350	0.09712	0.62760	0.14904
	RF	0.75250	0.07081	0.64329	0.07923
	SVM	0.74550	0.11809	0.60766	0.09794
	LR	0.74300	0.13572	0.65613	0.17532
	LightGBM	0.74900	0.08904	0.65979	0.09274
German-sex	XGBoost	0.75450	0.08883	0.65327	0.04598
	GBDT	0.75500	0.07061	0.65821	0.03699
	AdaBoost	0.74350	0.04448	0.62760	0.03332
	RF	0.75250	0.07586	0.64329	0.03519
	SVM	0.74550	0.09903	0.60766	0.05356
	LR	0.74300	0.11061	0.65613	0.06438
	LightGBM	0.74900	0.06816	0.65979	0.05487

5.2. Performance evaluation of clustering-based over-sampling method

To verify the effectiveness of the proposed clustering-based over-sampling method, the performances of base classifiers adopting the clustering-based over-sampling method are presented in Table 3. The results in bold mean that the corresponding base classifiers perform better after the clustering-based over-sampling method is employed. Although the ACC of most base classifiers decreased, the AOD, BA, and EOD improved, which demonstrates that the proposed clustering-based over-sampling method can effectively balance the label and sensitive attribute in the dataset to improve the prediction accuracy and fairness.

Table 3. Performance evaluation of clustering-based over-sampling method.

Dataset	Base classifiers	ACC	AOD	BA	|EOD|
Adult-race	XGBoost	0.85689	0.02737	0.77263	0.02882
	GBDT	0.85833	0.02501	0.77543	0.02512
	AdaBoost	0.84973	0.02079	0.75788	0.01359
	RF	0.83680	0.04020	0.76367	0.03908
	SVM	0.83666	0.04135	0.74562	0.05718
	LR	0.84472	0.03939	0.76615	0.04247
	LightGBM	0.86434	0.03059	0.79506	0.03375
Bank-age	XGBoost	0.90513	0.06504	0.73973	0.08506
	GBDT	0.90707	0.05619	0.75247	0.06405
	AdaBoost	0.90128	0.06101	0.70830	0.09065
	RF	0.90161	0.04041	0.71841	0.03809
	SVM	0.89113	0.12284	0.63405	0.16053
	LR	0.90062	0.05473	0.70736	0.06910
	LightGBM	0.90822	0.04035	0.76702	0.02266
German-age	XGBoost	0.75750	0.07790	0.67444	0.06567
	GBDT	0.75300	0.07672	0.67216	0.05546
	AdaBoost	0.75250	0.05571	0.66110	0.08334
	RF	0.75600	0.03919	0.64289	0.06184
	SVM	0.74950	0.10050	0.63509	0.11768
	LR	0.74850	0.06513	0.66718	0.07559
	LightGBM	0.74900	0.08418	0.67016	0.06124
German-sex	XGBoost	0.73950	0.03018	0.66637	0.00838
	GBDT	0.75050	0.02762	0.67786	0.00733
	AdaBoost	0.74450	0.03195	0.67495	0.01698
	RF	0.75250	0.00200	0.65893	0.02568
	SVM	0.72700	0.03828	0.67657	0.03939
	LR	0.72650	0.02457	0.69468	0.03262
	LightGBM	0.73950	0.00612	0.66254	0.00404

5.3. Performance evaluation of stacking-based ensemble learning method and CAN-based post-processing method

In the stacking-based ensemble learning method, the competent base classifiers (e.g. GBDT, AdaBoost, and LightGBM for the Adult dataset that uses race as the sensitive attribute) are selected from the base classifier pool for permutation and combination because of their outstanding ACC and AOD, as shown in Table 4.

Table 4. Competent base classifiers selected for different datasets.

	AdaBoost	GBDT	LightGBM	RF	XGBoost
Adult-race	√	√	√
Bank-age		√	√	√
German-age		√		√	√
German-sex		√	√	√

To demonstrate the outperformance of the proposed BAHEM, the performance of the stacked ensemble model without adopting the CAN-based post-processing method is compared with that of the BAHEM adopting the CAN-based post-processing method. Table 5 presents the performance comparison between the stacked ensemble model and the BAHEM. The results of BAHEM are highlighted in bold if the evaluation metric in the BAHEM is better than that of the stacked ensemble model. As shown in Table 5, although the partial ACC and BA of the BAHEM on some datasets (e.g. Bank-age and German-age) are slightly worse than those of the stacked ensemble model, BAHEM shows a significant improvement in the fairness metrics (i.e. AOD and EOD). The experimental results indicate the stacking-based ensemble learning method and CAN-based post-processing method can effectively improve the classification fairness of the model without sacrificing too much prediction accuracy.

Table 5. Performance comparison between the stacked ensemble model and BAHEM.

Dataset	Ensemble models	ACC	AOD	BA	|EOD|
Adult-race	Stacked ensemble model	0.76430	0.04932	0.67535	0.07702
	BAHEM	0.80858	0.00467	0.61851	0.01138
Bank-age	Stacked ensemble model	0.90107	0.01919	0.70492	0.00373
	BAHEM	0.88755	0.01673	0.69940	0.01324
German-age	Stacked ensemble model	0.70600	0.04805	0.63287	0.06485
	BAHEM	0.69400	0.00210	0.62397	0.04070
German-sex	Stacked ensemble model	0.69500	0.01145	0.62141	0.04039
	BAHEM	0.74350	0.00695	0.65863	0.00399

5.4. Performance comparison between base classifiers and the proposed BAHEM

To demonstrate that the proposed BAHEM can improve classification fairness with little reduction in prediction accuracy, the performance of the BAHEM is compared with seven base classifiers on four evaluation metrics. Histograms of the performance comparison between BAHEM and base classifiers are presented in Figure 6. As shown in Figure 6, although the ACC and BA of the BAHEM are slightly reduced compared with the base classifiers on each dataset, the fairness metrics (i.e., AOD and EOD) of BAHEM are improved, indicating that the proposed BAHEM can effectively improve fairness without sacrificing too much prediction accuracy.

Figure 6. Histograms of performance comparison between base classifiers and BAHEM.

5.5. Performance comparison between benchmark models and the proposed BAHEM

To further prove the higher classification performance and fairness of the proposed BAHEM, it is compared with the benchmark models proposed by Kearns et al. (2019), Celis et al. (2019), Pleiss et al. (2017), Hardt et al. (2016), Rezaei et al. (2020), and Wang et al. (2021). The source codes of related benchmark models are public. For a fair comparison, all experiments were conducted under the same experimental settings. The comparison results are shown in Table 6. Certain evaluation metrics were not adopted in the benchmark models; hence, the corresponding indicators are marked as “/” in this table. As the table shows, the ACC of the BAHEM are slightly reduced compared with most other benchmark models, but the BAHEM outperforms other models in AOD and EOD on most datasets, demonstrating that the proposed BAHEM can improve fairness without sacrificing too much prediction accuracy.

Table 6. Performance comparison between benchmark models and the proposed BAHEM.

Dataset	Benchmark models	ACC	AOD	BA	|EOD|
Adult-race	Kearns et al. (2019)	0.83734	0.05659	0.73100	0.08033
	Celis et al. (2019)	0.83703	0.05079	0.72923	0.07141
	Pleiss et al. (2017)	0.83082	0.01073	0.71987	0.04584
	Hardt et al. (2016)	0.83473	0.00304	0.73781	0.00967
	Rezaei et al. (2020)	0.75650	0.03120	/	0.05680
	Wang et al. (2021)	0.84290	0.06960	/	0.10250
	The proposed model	0.80858	0.00467	0.61851	0.01138
Bank-age	Kearns et al. (2019)	0.89677	0.05094	0.66105	0.08493
	Celis et al. (2019)	0.89682	0.04809	0.66114	0.08233
	Pleiss et al. (2017)	0.90110	0.01007	0.69846	0.00620
	Hardt et al. (2016)	0.87675	0.00843	0.69360	0.02007
	Rezaei et al. (2020)	0.87570	0.05550	/	0.07730
	Wang et al. (2021)	0.89610	0.05050	/	0.06300
	The proposed model	0.88755	0.01673	0.69940	0.01324
German-age	Kearns et al. (2019)	0.74250	0.13732	0.66238	0.18125
	Celis et al. (2019)	0.74700	0.07566	0.66011	0.12690
	Pleiss et al. (2017)	0.70800	0.39968	0.54321	0.28847
	Hardt et al. (2016)	0.62450	0.02662	0.58737	0.02085
	Rezaei et al. (2020)	0.71670	0.05300	/	0.00500
	Wang et al. (2021)	0.70670	0.04690	/	0.11440
	The proposed model	0.69400	0.00210	0.62397	0.04070
German-sex	Kearns et al. (2019)	0.72150	0.11400	0.65758	0.06399
	Celis et al. (2019)	0.73600	0.05728	0.65068	0.03453
	Pleiss et al. (2017)	0.73200	0.22222	0.60975	0.11367
	Hardt et al. (2016)	0.70500	0.02556	0.62343	0.00096
	The proposed model	0.74350	0.00695	0.65863	0.00399

6. Conclusion and future work

In this study, a novel BAHEM based on over-sampling and post-processing is proposed to alleviate data bias and improve classification fairness without sacrificing too much prediction accuracy. The proposed BAHEM mainly contains two main contributions. First, a new clustering-based over-sampling method is proposed, which generates the subsets with clustering methods and automatically balances the label and sensitive attribute to improve the adaptability to imbalanced datasets and the classification fairness of the model. Second, a new CAN-based post-processing method is proposed to select prediction results with higher uncertainty and modify them to further enhance the fairness of the BAHEM while maintaining the prediction accuracy. Three datasets (i.e. Adult, Bank, and German) with different sensitive attributes and four evaluation metrics (i.e. ACC, AOD, BA, and EOD) were used to evaluate the classification performance and fairness of the BAHEM. The experiment results show that classification performance and fairness of the BAHEM outperform those of other benchmark models.

However, the proposed BAHEM in this paper has some shortcomings in both method and practical application. In planned future work, the proposed BAHEM can be further improved by generating an ensemble model with base classifiers that are processed using in-processing methods. For sampling methods, the threshold can be adjusted adaptively according to different sample distributions. For post-processing methods, more comprehensive indicators in addition to uncertainty, can be considered in evaluating and modifying the prediction results to obtain higher classification performance and fairness of the model. For practical application, it is necessary to further enhance the interpretability of the model and the robustness of the results obtained from the model.

Compliance with ethical standards

Conflicts of interest: The authors declare that there is no conflict of interests regarding the publication of this article.

Ethical standard: The authors state that this research complies with ethical standards. This research does not involve either human participants or animals.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets analysed during the current study are available in the UCI repository. German dataset is from https://archive.ics.uci.edu/ml/machine-learning-databases/statlog/german, Adult dataset is from https://archive.ics.uci.edu/ml/machine-learning-databases/adult, and Bank dataset is from https://archive.ics.uci.edu/ml/machine-learning-databases/00222.

Additional information

Funding

This work has been supported by Fundamental Research Funds for the Provincial Universities of Zhejiang Institute of Economics and Trade (No. 19YQ19), National Natural Science Foundation of China (No. 51875503), Zhejiang Natural Science Foundation of China (No. LZ20E050001), and Zhejiang Key R & D Project of China (No. 2022C03166).

Notes

1 Available at https://github.com/yhefang/BAHEM.