Uncertainty-aware deep learning-based CAD system for breast cancer classification using ultrasound and mammography images

ABSTRACT

Breast cancer is one of the most common types of cancer in women. Early and accurate diagnosis of breast cancer can increase the treatment chances and decrease the mortality rate. Thus, the development of accurate and reliable Computer-Aided Diagnosis (CAD) systems using breast cancer images is an immediate priority for early diagnosis. Deep learning methods have had a widespread role in CAD systems. In spite of the advantages, most deep learning-based CADs cannot quantify the uncertainty of their predictions. Uncertainty in the predicted results from a CAD system might endanger people’s life in these crucial steps for breast cancer classification. Therefore, in this study, the Monte Carlo Dropout method is utilised as an uncertainty quantification method. Furthermore, differently from most existing studies, we present an uncertainty-aware deep learning model which can classify accurately from different types of images, namely, mammograms and ultrasound. The results show the proposed model achieved the accuracy of 99.95% in mammogram dataset, namely, DDSM and 98.40% and 91.34% in ultrasound datasets, namely, BUSM and BUSI, respectively. Finally, since we found the proposed model shows high uncertainty in misclassifications, we suggested to use the uncertainty values to find appropriate subsets for further inspection to increase the performance.

KEYWORDS:

• Breast cancer
• computer-aided diagnosis system
• deep learning
• uncertainty quantification

1. Introduction

Breast cancer is a type of cancer that develops from breast tissue. Worldwide, breast cancer is the most common cancer in women, other than nonmelanoma skin cancer (Waks and Winer 2019). Early and accurate diagnosis of breast cancer can increase the treatment chances and decrease the mortality rate (Yassin et al. 2018; Balkenende et al. 2022).

Using medical imaging is the most effective method for the diagnosis of breast cancer (Yassin et al. 2018; J Gaona et al. 2020; Murtaza et al. 2020; Mi et al. 2021; Balkenende et al. 2022). Mammography is the gold standard for breast cancer screening. Although, to overcome the limitations of mammography especially in dense breasts, supplemental screening for breast cancer with ultrasound imaging is suggested by clinicians (Geisel et al. 2018).

However, accurate diagnosis requires experienced medical specialists and experts. Thus, using a Computer-Aided Diagnosis (CAD) system is required to overcome the operator dependency. Also, CAD system can help to reduce the errors, effort, time, and cost in the diagnosis process (Zaalouk et al. 2022).

Over the years, ML (Machine learning) and DL (Deep Learning) methods focused on applications such as electronic health service (Gupta et al. 2019), the Internet of Things-based healthcare (Ben Othman et al. 2022), data extraction (Bhuyan et al. 2022; Li et al. 2023) and medical image processing (Ferdous Aurna et al. 2022; Ding et al. 2023; Wang et al. 2024). In recent years, several advances in deep learning (LeCun et al. 2015) have had a widespread role in CAD systems in breast cancer classification (Wu et al. 2023; Chougrad et al. 2018; Al-Antari et al. 2018; Muduli et al. 2021). Despite the advantages, Deep learning models tend to be overconfident about their predictions, and also classical deep learning methods do not capture uncertainty (Abdar et al. 2021; Abdullah et al. 2022; Gawlikowski et al. 2023).

Many situations can lead to uncertainty in deep learning models. Data uncertainty is introduced from noisy data such as measurement noise. This type of uncertainty is called Aleatoric uncertainty or irreducible uncertainty. Also, there is uncertainty about which values of model parameters will be good at predicting new data. Finally, there is structure uncertainty about the general structure of the model. The latter two uncertainties can be grouped under Epistemic or reducible uncertainty. Epistemic uncertainty can be reduced with more data while aleatoric uncertainty can not. Aleatoric uncertainty and epistemic uncertainty can then be utilised to induce predictive uncertainty (Ghahramani 2015; Abdar et al. 2021)

Due to use of point estimates of parameters and predictions in classical deep learning models, these models are too confident about their output and decision without taking into consideration the uncertainty associated with data and predictions of the model, so, there are some challenges with unreliability (Ghahramani 2015; Abdullah et al. 2022). In the other word, prediction uncertainty in CAD system results might endanger people’s life in these crucial steps for clinical diagnostics. Therefore, leveraging uncertainty information is imperative to ensure the safe use of these novel tools (Filos et al. 2019; Leibig et al. 2017).

In another word, Due to use of point estimates of parameters and predictions in classical deep learning models, these models are too confident about their output and decision without taking into consideration the uncertainty associated with data and the model (Ghahramani 2015; Abdullah et al. 2022). Yet, In CAD system, the reliability of automated decisions and quantifying such uncertainties are quite important to quantify our confidence about the predictions of the models (Filos et al. 2019; Leibig et al. 2017). To mitigate this risk, probabilistic approaches are used to estimate the uncertainty in the model predictions (Jospin et al. 2022). Bayesian deep learning (BDL) uses a combination of the Bayesian probability theory with classical deep learning to provide a prediction and extract the uncertainty values. The uncertainty is estimated as a probability density over outcomes through this process, known as inference in BDL. However, exact inference is analytically intractable and hence approximate inference has been applied instead (Wang and Yeung 2021). Gal and Ghahramani (2015) proposed a new framework that use dropout in deep neural networks as approximate Bayesian inference while mitigate the problem of quantifying uncertainty in deep learning without suffering computational complexity or decreasing test accuracy.

Our literature review identified some research gaps related to the importance of uncertainty quantification in the breast cancer classification tasks. On the other hand, to the best of our knowledge, there are very few studies that have used both types of images (Ultrasound and Mammography Images) simultaneously. Therefore, to achieve high accuracy as well as reliability in breast cancer classification task in Ultrasound and Mammography Images simultaneously, in this paper, we employed an uncertainty-aware deep convolutional neural network model as a CAD system.

Therefore, to achieve high accuracy as well as reliability in breast cancer classification task simultaneously, in this paper, we employed an uncertainty-aware deep convolutional neural network model as a CAD system. Differently from most of existing studies, which just used one type of images in breast cancer classification, we present a simple but efficient deep learning model, which is able to classify accurately different class of images, namely, mammograms and ultrasound. The main contributions of this study are as follows:

• We quantified the uncertainty in our proposed deep CNN model using the MC Dropout technique as Bayesian deep learning.

• We proposed a simple but efficient deep learning model to breast cancer classification task using Ultrasound and Mammography Images.

• We used some pretrained architectures for breast Classification such as Xception, VGG16, ResNet50V2 and DenseNet121 combining a CNN model as a feature extractor.

• To demonstrate the importance of having high accuracy in the diagnosis of each class, especially the class of malignancy, we computed the performance of the proposed model for each class separately.

• We indicated that the proposed deep CNN model shows high uncertainty in misclassifications. we suggested to leverage the uncertainty information to increase the performance of the CAD system by selecting appropriate subsets as suspicious cases for further inspection.

2. Related works

Researchers present several deep learning‐based CAD systems using mammogram as well as ultrasound images which provide good performance in breast cancer classification (Murtaza et al. 2020; Hassan et al. 2022). Noushiba Mahbub et al. (2022) proposed a modified CNN and fuzzy analytical hierarchy process for diagnosis of breast cancer. Ahmed Roni et al. (2022) used the Xception model and a generative adversarial network to improve the algorithm’s precision for breast cancer binary classification. Al-masni et al (2018) proposed a CAD model based on YOLO (You Only Look Once) techniques. This YOLO-based CAD system can distinguish between benign and malignant lesions. Ragab et al (2019) presented a model where an AlexNet is used and fine-tuned to classify two classes. Duggento et al (2019) described a CAD based on an ad hoc CNN architecture specialised in breast lesion classification. Chougrad et al (2018) proposed a CAD system based on deep Convolutional Neural Networks (CNN) and fine-tuning strategy.

Similarly, different CAD systems based on machine and deep learning have been proposed for breast cancer classification on ultrasound images. Moon et al (2020) have proposed a model for tumour diagnosis using an image fusion method combined with different image content representations and an ensemble of different CNN architectures on private and public US images. This CNN-based model includes VGGNet, ResNet, and DenseNet. Byra (2021) has proposed a model that employs transfer learning with convolutional neural networks and used deep features through scaling layers for breast mass classification in ultrasound images. Sadad et al (2020) proposed a CAD system by Hilbert transform for reconstructing B-mode images from the raw data followed by the marker-controlled watershed transformation to segment the lesion and hybrid feature set which developed after the extraction of shape-based and texture features from the breast lesion. In this model, the decision tree, k-nearest neighbour and ensemble decision tree model via random under-sampling with Boost (RUSBoost) are utilised to breast cancer classification. Mishra et al (2021) proposed a machine learning-radiomics based classification pipeline. In this scheme shape, texture and histogram-oriented gradients related features have been used and various machine learning models such as random forest, gradient boosting and AdaBoost classifiers are considered. Cao et al. (2020) proposed an approach called noise filter network (NF-Net) to address the problem of noisy labels by incorporating two SoftMax layers for classification and a teacher-student module for distilling the knowledge of clean labels. Zhuang et al. (2020) employed image decomposition to obtain fuzzy enhanced and bilateral filtered images to enrich input information of breast lesions which are presented as inputs to a pre-trained VGG16 model. Al-Dhabyani et al. (2019) used Generative Adversarial Network (GAN) for data augmentation and CNN with Transfer Learning approach have proposed to be used for breast ultrasound image classification. Acevedo and Vazquez (2019) used k-means and GLCM algorithms to segment ultrasound images and a support vector machine algorithm to breast cancer classification.

Additionally, some researchers present different CAD models which use multi-modal breast cancer datasets like mammograms and ultrasound images. Muduli et al. (2021) proposed a deep convolutional neural network model for breast cancer classification from a different class of images, namely, mammograms and sonograms. Assari et al. (2022b) proposed a bimodal deep residual learning model for solid breast mass classification. In the other work (Assari et al. 2022a), these authors used a bimodal GoogLeNet-Based CAD system and two-step transfer learning strategy for solid breast mass classification by combining mammograms and corresponding sonograms data.

Although most CAD models are looking for good performance results in different applications, but it is not enough. The CAD models require both high accuracy as well as reliability in breast cancer classification. So, we need uncertainty quantification in classification for achieving a reliable CAD model. Few researchers have worked on Uncertainty Quantification in medical image classification. Song et al. (2021) proposed a Bayesian deep network capable of estimating uncertainty to assess oral cancer image classification. Filos et al. (2019) employed Bayesian deep learning based on the VGG model for diabetic retinopathy classification. Abdar et al. (2023) proposed an uncertainty-aware deep learning feature fusion model for COVID-19 Detection. Thiagarajan et al. (2022) proposed a technique to utilise the uncertainties provided by the Bayesian CNN for the classification of breast histopathological images.

Motivated by these shortcomings in the field of uncertainty estimation of the breast cancer classification task, we develop a reliable uncertainty-aware deep learning-based model for breast cancer classification from a different class of images, namely, mammograms and ultrasound.

3. Materials and methods

In this paper, an uncertainty-aware deep learning model is utilised for binary breast cancer classification using mammogram as well as sonograms. For better understanding, summary of this study is shown in Figure. 1

Figure 1. A general overview of the proposed uncertainty-aware deep learning-based CAD system in breast cancer classification.

3.1. Datasets

In this study, simulation has been carried out to classify the image as benign or malignant in both mammogram dataset as well as ultrasound datasets. So, we have utilised mammograms and ultrasound images as breast cancer datasets. For mammograms, we have considered a public dataset containing mammography with benign and malignant masses, namely DDSM (Heath et al. 2000; Lina and Huang 2020; Huang and Lina 2020). In the case of sonograms, we have utilised two public datasets, namely, BUSM (Rodrigues 2017), and BUSI (Al-Dhabyani et al. 2020). Figure 2 shows some random image samples from the DDSM, BUSM and BUSI datasets and details of each dataset used in our study have been tabulated in Table 1.

Figure 2. Some random image samples from the mammogram and sonogram datasets were considered in our study. benign and malignant images from (a) DDSM dataset, (b) BUSI dataset and (c) the BUSM dataset. (M is malignant and B is benign).

Table 1. A description of the number of mammogram and sonogram datasets and the number of samples of each dataset in the training and testing from two categories benign(B) and malignant(M).

Datasets	Total number of images		Training		Testing
	B	M	B	M	B	M
DDSM	995	1193	798	952	197	241
BUSM	100	150	74	126	26	24
BUSI	437	210	346	171	91	39

3.2. Preprocessing

3.2.1. Histogram Equalization

First, an image preprocessing method, namely, Contrast Limited Adaptive Histogram Equalization (CLAHE) was used on the original images.

3.2.2. Resizing and Normalization

In this study, each image was resized to 224 × 224 pixels and its intensity values were linearly normalised to the interval [0, 1].

3.2.3. Data augmentation

The data augmentation method can be used to address the problem of over-fitting due to the small training datasets by using some transformations on them (Al-Dhabyani et al. 2019; Shorten and Khoshgoftaar 2019; Oza et al. 2022). To overcome the over-fitting issue, we used the data augmentation technique. In the training images, the utilised augmentation methods are image rotation and flipping horizontally and vertically using an API Keras Augmentor.

3.3. Model architectures

The considered CNN model consisted of 18 layers such as an input layer in the size of 224 × 224 pixels, convolutional layers, batch normalisation layers, pooling layers, Monte Carlo Dropout layers and fully connected layers. More details of the considered CNN model can be found in Figure 1.

3.4. Learning method

The ADAM optimiser with initial learning rate of 0.00005 and batch size of 32 is used for training all models. We used the Binary cross-entropy loss function to select the optimal weights. The hyper-parameters of the proposed CNN model are listed in Table 2.

Table 2. The value of proposed deep CNN model hyperparameters.

Hyper parameter Type	values
Batch size	32
Number of epochs	100
Initial learning rate	0.00005
Beta1	0.9
Beta2	0.999
epsilon	1e-07

3.5. Transfer learning and fine-tuning

Due to the limited training samples, to train the proposed model we used transfer learning. transfer learning is a method that involves reusing a pre-trained CNN model to learn features as the starting point for a model on a new task (Chang Shin et al. 2016; Zhuang et al. 2021). By applying this method, the proposed model can achieve higher performance and less uncertainty (Hendrycks et al. 2019).

To train the proposed CNN model, we performed transfer learning methods based on some popular pre-trained CNN models that were trained with the ImageNet and we transferred the pre-trained model to the studied datasets for fine-tuning to represent two classes as benign and malignant. In this case, we used the architecture of the pre-trained model combining the proposed CNN model and retrained entire it according to the considered datasets.

Also, we utilised two transfer learning and fine-tuning strategies. In the first strategy, we froze the pre-trained model in its original form and then used its outputs to feed the proposed CNN model. In the second strategy, we trained the entire model. In this case, we used the architecture of the pre-trained model combining the proposed CNN model and trained it according to the considered datasets. Figure 3 shows two utilised transfer learning and fine-tuning strategies.

Figure 3. The first and second transfer learning and fine-tuning strategies.

3.6. Uncertainty quantification method

In this study, the Monte Carlo Dropout approach was used for uncertainty estimation. Dropout is an effective technique that is being used in deep learning models for regularisation purposes, which solves over-fitting problems in models. During the training step, dropout is discarding some neurons of the Neural Network with a certain probability, called the dropout rate, for each iteration (Srivastava et al. 2014). This process is called a stochastic forward pass.

Gal and Ghahramani (Gal and Ghahramani 2015) developed a framework casting dropout in deep neural networks as approximate Bayesian inference in deep Gaussian processes. In this method, the dropout can be activated not only during training but also at test time. In another word, this method used T stochastic forward passes on the test time and collected them. It is equivalent to a form of variational inference used to estimate posterior distribution in the BDL (Gal and Ghahramani 2016).

The main idea behind the BDL is to consider a posterior distribution over the space of parameters $\mathit{w}$ given training inputs X = {x₁, … , x_N} and their corresponding outputs Y = {y₁, … , y_N}, instead of a single value and then is looked for the posterior distribution by using Bayes theorem which is written in eq. (1).

(1) $\mathit{p}\left(\mathit{w}|\mathit{X},\mathit{Y}\right)=\frac{\mathit{p}\left(\mathit{Y}|X,w\right)p\left(w\right)}{\mathit{p}\left(\mathit{Y}|X\right)}$(1)

Using the BDL model, the predictive probability distribution can calculate an output for a given test sample $x^{\ast }$ by integrating, as shown in eq. (2).

(2) $\mathit{p}\left(y^{\ast }|x^{\ast },\mathit{X},\mathit{Y}\right)=\int \mathit{p}\left(y^{\ast }|x^{\ast },\mathit{w}\right)\mathit{p}\left(\mathit{w}|\mathit{X},\mathit{Y}\right)\mathit{dw}$(2)

However,$\mathit{p}\left(\mathit{w}|\mathit{X},\mathit{Y}\right)$ cannot be computed analytically and should be approximated by various approximation methods such as variational inference (Graves 2011; Sun et al. 2019). So, according to the variational inference method, in order to find the predictive distribution, can define an approximating variational distribution $\mathit{q}\left(\mathit{w}\right)$ instead of true posterior distribution as shown in eq. (3).

(3) $\mathit{p}\left(y^{\ast }|x^{\ast },\mathit{X},\mathit{Y}\right)=\int \mathit{p}\left(y^{\ast }|x^{\ast },\mathit{w}\right)\mathit{p}\left(\mathit{w}|\mathit{X},\mathit{Y}\right)\mathit{dw}\approx \int \mathit{p}\left(y^{\ast }|x^{\ast },\mathit{w}\right)\mathit{q}\left(\mathit{w}\right)\mathit{dw}$(3)

In practice, when the dropout method is used, the predictive distribution can be estimated using a Monte Carlo integration. In this way, it is equivalent to perform T stochastic forward passes through the network and averaging the results. It is shown in eq. (4).

(4) $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)\approx \frac{1}{T}\sum _t\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },w_t\right)$(4)

Where $w_t\left(\mathit{w}\right)$, $\mathit{q}\left(\mathit{w}\right)$ is a Bernoulli variational distribution or a dropout variational distribution, c is all possible classes that y can take and $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right)$ is a SoftMax likelihood for classification task which is written in eq. (5).

(5) $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(f_c^w\right)}{{\sum }_{\mathit{c}{\mathit{ }}^{\mathrm{\prime }}}\exp \left(f_{\mathit{c}{\mathit{ }}^{\mathrm{\prime }}}^{\mathit{w}}\right)}$(5)

Where $f_c^w$ denotes a network function parametrized by the variables $\mathit{w}$. This means that a network already trained with dropout is a BDL model (Gal and Ghahramani 2015). It is an effective method to add uncertainty handling in deep learning models.

Finally, we can use three approaches to evaluate the uncertainty within breast cancer classification task:

1. Predictive entropy

2. Mutual information

3. Predictive variance and standard deviation of the model

In the result subsections, we added the details of each approach by uncertainty quantification metrics.

3.7. Uncertainty quantification metrics

We can measure three metrics to evaluate uncertainty within the classification task: (i) predictive entropy, (ii) mutual information and (iii) the predictive variance of the model.

3.7.1. Predictive entropy

We quantified the uncertainty of our binary classification predictions by predictive entropy (Shannon 1984; Gal 2016). Predictive entropy is the entropy of predictive distribution $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)$, in test points $x^{\ast }$, which captures the average of the information in the predictive distribution. The predictive entropy defined as eq. (6):

(6) ${\mathrm{H}}_{p\left(y^{\ast }|x^{\ast },\mathit{X},\mathit{Y}\right)}\left[y^{\ast }\right]=-\sum _{y^{\ast }=\mathit{c}}\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)log\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)$(6)

Where $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)$ indicates the Mc dropout approximated predictive distribution obtained by T stochastic forward passes through the model, defined as eq. (4). Due to the predictive entropy capturing epistemic and aleatoric uncertainty, this quantity is high when either the aleatoric or epistemic uncertainty is high.

3.7.2. Mutual information

The mutual information (Shannon 1984; Gal 2016) between the posterior over the model parameters $\mathit{w}$ and model output $y^{\ast }$on input $x^{\ast }$offers a different measure of uncertainty, as shown in eq. (7).

(7) $\mathit{MI}\left(y^{\ast },\mathit{w}|\mathit{X},\mathit{Y},x^{\ast }\right)={\mathrm{H}}_{p\left(y^{\ast }|x^{\ast },\mathit{X},\mathit{Y}\right)}\left[y^{\ast }\right]-E_{\mathit{p}\left(\mathit{w}|X,Y\right)}\left[{\mathrm{H}}_{p\left(y^{\ast }|x^{\ast },\mathit{w}\right)}\left[y^{\ast }\right]\right]$(7)

Where ${\mathrm{H}}_{p\left(y^{\ast }|x^{\ast },\mathit{X},\mathit{Y}\right)}\left[y^{\ast }\right]$ is corresponded predictive entropy. The mutual information can be approximated in our setting in a similar way to the predictive entropy approximation, which is shown in eq. (8):

(8) $\mathit{MI}\left(y^{\ast },\mathit{w}|\mathit{X},\mathit{Y},x^{\ast }\right)=-\sum _{y^{\ast }=\mathit{c}}\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)log\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)+\int \mathit{p}\left(\mathit{w}|\mathit{X},\mathit{Y}\right)\sum _{y^{\ast }=\mathit{c}}\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right)log\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right)$(8)

The Mc dropout approximated predictive distribution $\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)$ defined as eq. (4). So, using MC approximation we have:

(9) $\mathit{MI}\left(y^{\ast },\mathit{w}|\mathit{X},\mathit{Y},x^{\ast }\right)\approx -\sum _{y^{\ast }=\mathit{c}}\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)log\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y}\right)+\frac{1}{T}\sum _{t,y^{\ast }=\mathit{c}}\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right))log\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{w}\right)$(9)

The mutual information captures only epistemic uncertainty. So, this quantity is high when the epistemic uncertainty is high.

3.7.3 Predictive variance and standard deviation

We can obtain the approximated predictive distribution using the Monte Carlo dropout approach. So, we can consider the predictive variance from the approximated predictive distribution as associated uncertainty. The variance of the predictive distributions can be calculated by eq. (10).

(10) $\mathit{variance}=\frac{1}{T}\sum _t(\mathit{p}\left(y^{\ast }=\mathit{c}|x^{\ast },w_t\right)-\mathit{p}{\left(y^{\ast }=\mathit{c}|x^{\ast },\mathit{X},\mathit{Y})\right)}^2$(10)

Also, the standard deviation is the square root of the variance in eq. (10).

4. Results

4.1. Evaluation metrics

In this work, a variation of five-fold cross-validation (CV), called Stratified five-fold method, has been used to evaluate all of the deep CNN models. It divides the original datasets in to five folds so that each fold contains approximately the same percentage of samples of each target class. The detailed Stratified five-fold cross-validation process is shown in Figure 4 and dataset divisions have been tabulated in Table 1.

Figure 4. Description of stratified five-fold cross validation and distribution of samples for each split.

4.2. Performance metrics

For a binary classification problem, the output comes under one among the four categories: True Positive (TP), False Negative (FN), False Positive (FP), and True Negative (TN).

To evaluate the performance of the proposed model in predicting the type of breast cancer, recall (Sensitivity), precision, accuracy and F1 score were used. The calculation formulas are written in eq. (11–14).

(11) $\mathit{Accuracy}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}}$(11)

(12) $\mathit{Recall}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$(12)

(13) $\mathit{Precision}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(13)

(14) $\mathit{F}1-\mathit{score}=2\times \frac{\mathit{Recall}\times \mathit{Precision}}{\mathit{Recall}+\mathit{Precision}}=\frac{2\times \mathit{TP}}{2\times \mathit{TP}+\mathit{FP}+\mathit{FN}}$(14)

4.3. The proposed deep CNN model results

In our first experiment, to determine the best model for constructing the proposed deep CNN model in mammograms and ultrasound images, we practiced four different pre-trained models combing the proposed CNN model, including Xception (Chollet 2016), VGG16 (Simonyan and Zisserman 2015), ResNet50V2 (He et al. 2016) and DenseNet121 (Huang et al. 2018). The models were compared in the metrics of the accuracy and the number of trainable parameters.

The obtained results of 5-fold cross-validation are presented from Tables 3–5. In Tables 3–5, it can be perceived that in both of mammograms and ultrasound images, the pre-trained Xception combining the CNN model provides the proper performance than the other CNN models with average classification accuracy of 99.95%, 92.11% and 95.60% in the DDSM, BUSI and BUSM datasets, respectively. It should be noted that the pre-trained Xception combining the CNN model had a comparable performance to the single Xception model, while the number of trainable parameters was less. So, we selected pre-trained Xception combining the CNN model for constructing the proposed deep CNN model. Since, we also considered the impact of the uncertainty quantification method on the performance metric of the proposed model, our experiments have been conducted with and without applying uncertainty quantification.

Table 3. Classification performance in (%) of CNN and various pre-trained CNN model with mammograms (DDSM dataset). Results obtained without considering uncertainty.

				Fold1	Fold2	Fold3	Fold4	Fold5	Average
	Pretrained model		# of trainable parameters	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy
	CNN		25,800,018	100	99.77	99.32	100	99.77	99.77 (± 0.25%)
Transfer learning	Xception		72,261,738	100	100	100	100	100	100 (± 0.00%)
	S 1	Xception+CNN	437,154	97.72	98.63	98.40	91.76	96.34	96.57 (± 2.53%)
	S 2	Xception+CNN	21,244,106	100	100	100	100	99.77	99.95 (± 0.09%)
	ResNet50V2		74,974,146	100	100	100	100	100	100 (± 0.00%)
	S 1	ResNet50V2+CNN	437,154	99.77	99.32	100	99.54	99.54	99.63 (± 0.23%)
	S 2	ResNet50V2+CNN	23,956,514	99.77	98.17	98.63	100	100	99.32 (± 0.76%)
	DenseNet121		32,718,530	99.54	100	98.40	99.54	100	99.50 (± 0.58%)
	S 1	DenseNet121+CNN	289,698	99.09	98.86	99.77	98.86	94.05	98.12 (± 2.06%)
	S 2	DenseNet121+CNN	7,243,554	100	99.32	100	99.31	99.77	99.68 (± 0.31%)
	VGG16		27,634,306	94.98	100	98.86	53.78	89.93	87.51 (± 17.23%)
	S 1	VGG16+CNN	215,970	45.43	54.57	50.68	50.34	54.46	51.10 (± 3.35%)
	S 2	VGG16+CNN	14,930,658	99.08	99.31	100	100	100	99.68 (± 0.40%)

S1, S2: The first and second transfer learning and fine-tuning strategies.

Table 4. Classification performance in (%) of CNN and various pre-trained CNN model with ultrasound images (BUSI dataset). Results obtained without considering uncertainty.

				Fold1	Fold2	Fold3	Fold4	Fold5	Average
	Pretrained model		# of trainable parameters	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy
	CNN		25,800,018	81.54	73.08	79.84	72.09	79.07	77.12 (± 3.80%)
Transfer learning	Xception		72,261,738	92.31	93	89.92	83.72	89.15	89.64 (± 3.30%)
	S 1	Xception+CNN	437,154	73.85	78.46	76.74	76.74	79.84	77.13 (± 2.01%)
	S 2	Xception+CNN	21,244,106	90.77	96.15	94.57	86.82	92.25	92.11 (±3.23%)
	ResNet50V2		74,974,146	89.23	92.31	90.7	83.72	86.05	88.40 (± 3.12%)
	S 1	ResNet50V2+CNN	437,154	83.85	80.77	83.72	79.07	79.07	81.30 (± 2.12%)
	S 2	ResNet50V2+CNN	23,956,514	91.54	80.77	79.84	84.5	88.37	85.00 (± 4.45%)
	DenseNet121		32,718,530	9.77	90	87.60	83.72	84.50	87.32 (± 2.83%)
	S 1	DenseNet121+CNN	289,698	85.38	80	80.62	75.97	80.62	80.52 (± 2.99%)
	S 2	DenseNet121+CNN	7,243,554	87.69	86.15	88.37	86.82	82.17	86.24 (± 2.17%)
	VGG16		27,634,306	67.69	78.46	86.05	39.53	65.89	67.53 (± 15.80%)
	S 1	VGG16+CNN	215,970	69.23	30.77	68.22	67.44	31.78	53.49 (± 18.15%)
	S 2	VGG16+CNN	14,930,658	82.95	86.05	84.5	87.6	87.5	85.72 (± 1.79%)

S1, S2: The first and second transfer learning and fine-tuning strategies.

Table 5. Classification performance in (%) of CNN and various pre-trained CNN model with ultrasound images (BUSM dataset). Results obtained without considering uncertainty.

	Pretrained model		# of trainable parameters	Fold1	Fold2	Fold3	Fold4	Fold5	Average
				Accuracy	Accuracy	Accuracy	Accuracy	Accuracy	Accuracy
	CNN		25,800,018	96	98	88	98	96	95.20 (± 3.71%)
Transfer learning	Xception		72,261,738	100	100	100	98	96	98.80 (± 1.60%)
	S 1	Xception+CNN	437,154	88	88	78	82	84	84.00 (± 3.79%)
	S 2	Xception+CNN	21,244,106	96	98	100	84	100	95.60% (± 5.99%)
	ResNet50V2		74,974,146	94	100	96	100	100	98.00 (± 2.53%)
	S 1	ResNet50V2+CNN	437,154	92	96	90	88	86	90.40 (± 3.44%)
	S 2	ResNet50V2+CNN	23,956,514	84	100	98	100	100	96.40 (± 6.25%)
	DenseNet121		32,718,530	100	100	98	98	98	98.80 (± 0.98%)
	S 1	DenseNet121+CNN	289,698	88	84	90	86	86	86.80 (± 2.04%)
	S 2	DenseNet121+CNN	7,243,554	92	90	100	98	98	95.60 (± 3.88%)
	VGG16		27,634,306	18	94	98	60	22	58.40 (± 34.04%)
	S 1	VGG16+CNN	215,970	40	60	76	64	60	60.00 (± 11.59%)
	S 2	VGG16+CNN	14,930,658	82	84	88	84	90	85.60 (± 2.94%)

S1, S2:The first and second transfer learning and fine-tuning strategies.

The summary outline of the considered model is listed in Table 6. Also, Figure 5 (a-e) show accuracy and loss curves over epochs of the models obtained using datasets for training phase.

Figure 5. Training accuracy and loss of proposed model versus number of epochs for (a) fold 1, (b) fold 2, (c) fold 3, (d) fold 4 and (e) fold 5.

Table 6. The detail configuration of the proposed CNN architecture.

Layer (type)	Output shape	# of parameter
input_1 (InputLayer)	(None, 224, 224, 3)	0
xception (Functional)	(None, 3,3, 2048)	20861480
conv2d (Conv2D)	(None, 3, 3, 16)	294928
conv2d_1 (Conv2D)	(None, 3, 3, 16)	2320
max_pooling2d (MaxPooling2D)	(None, 2, 2, 16)	0
dropout (Dropout)(MC)	(None, 2, 2, 16)	0
conv2d_2 (Conv2D)	(None, 2, 2, 32)	4640
conv2d_3 (Conv2D)	(None, 2, 2, 32)	9248
batch_normalization (BatchNormalization)	(None, 2, 2, 32)	128
max_pooling2d_1 (MaxPooling 2D)	(None, 1, 1, 32)	0
dropout_1 (Dropout)(MC)	(None, 1, 1, 32)	0
conv2d_4 (Conv2D)	(None, 1, 1, 64)	18496
batch_normalization_1 (BatchNormalization)	(None, 1, 1, 64)	256
max_pooling2d_2 (MaxPooling 2D)	(None, 1, 1, 64)	0
dropout_2 (Dropout)(MC)	(None, 1, 1, 64)	0
flatten (Flatten) dense (Dense)	(None, 64) (None, 512)	0 33280
dense (Dense)	(None, 128)	65664
dense_1 (Dense)	(None, 64)	8256
dropout_3 (Dropout)(MC)	(None, 64)	0
dense_2 (Dense)	(None, 2)	130
Total params:		21,298,826
Trainable params:		21,244,106
Non-trainable params:		54,720

4.3.1. Breast cancer classification without considering uncertainty

First, we investigated the other performance metrics of the proposed deep CNN model without considering uncertainty. The obtained results are presented in Table 7 for the sonogram and mammogram datasets.

Table 7. Average classification results of the proposed deep CNN model in breast cancer classification in (%) for BUSM, BUSI and DDSM datasets on 5 folds: results obtained without considering uncertainty quantification.

dataset	Precision	Recall	F-Measure	Accuracy
BUSM	95.60	95.60	95.60	95.60
BUSI	92.11	92.11	92.11	92.11
DDSM	99.95	99.95	99.95	99.95

4.3.2. Breast cancer classification considering uncertainty

The results obtained from the proposed deep CNN model without considering uncertainty show proper classification performance. But it is not enough and the uncertainty estimates should accompany such deep learning models. To estimate the uncertainty of our deep CNN model predictions, we used an uncertainty quantification method, called MC dropout.

The obtained results are presented in Table 8 for the mammogram and sonogram datasets by the proposed deep CNN model considering uncertainty. To demonstrate the importance of having high accuracy in the diagnosis of each class, especially the class of malignancy, we computed the performance of the proposed model for each class, separately.

Table 8. Classification performance in (%) of the proposed deep CNN model in breast cancer classification using BUSM, BUSI and DDSM datasets: results obtained with considering MC dropout uncertainty quantification (the performance of each class is shown, separately).

		FOLD 1				FOLD 2				FOLD 3				FOLD 4				FOLD 5
dataset	Class	Precision	Recall	F-Measure	Accuracy	Precision	Recall	F-Measure	Accuracy	Precision	Recall	F-Measure	Accuracy	Precision	Recall	F-Measure	Accuracy	Precision	Recall	F-Measure	Accuracy
BUSM	Benign	100	100	100	-	100	100	100	-	100	100	100	-	100	80	88.89	-	100	100	100	-
	Malignant	100	100	100	-	100	100	100	-	100	100	100	-	88.44	100	93.75	-	100	100	100	-
	Average	100	100	100	100	100	100	100	100	100	100	100	100	94.12	90	91.32	92	100	100	100	100
BUSI	Benign	96.39	90.91	93.57	-	95.56	97.73	96.63	-	96.55	96.55	96.55	-	90.48	87.36	88.89	-	92.94	9.80	91.86	-
	Malignant	82.98	92.86	87.64	-	95	9.048	92.68	-	92.86	92.86	92.86	-	75.56	80.95	78.16	-	81.82	85.71	83.72	-
	Average	89.68	91.88	90.60	91.54	95.28	94.10	94.66	95.38	94.70	94.70	94.70	95.35	83.02	84.15	83.52	85.27	87.38	88.26	87.79	89.15
DDSM	Benign	100	100	100	-	100	100	100	-	100	100	100	-	100	100	100	-	100	995	99.75	-
	Malignant	100	100	100	-	100	100	100	-	100	100	100	-	100	100	100	-	99.58	100	99.79	-
	Average	100	100	100	100	100	100	100	100	100	100	100	100	100	100	100	100	99.79	99.75	99.77	99.77

As shown in Table 9, the proposed deep CNN model considering uncertainty provides an accuracy of 99.95%, 91.34% and 98.40% in the case of DDSM, BUSI and BUSM datasets, respectively.

Table 9. Average classification results of the proposed deep CNN model in breast cancer classification in (%) for BUSM, BUSI and DDSM datasets on 5 folds: results obtained with considering MC dropout uncertainty quantification.

dataset	Precision	Recall	F-Measure	Accuracy
BUSM	98.82	98	98.26	98.40
BUSI	9.01	90.62	90.25	91.34
DDSM	99.96	99.95	99.95	99.95

The results obtained using the proposed deep CNN model with and without uncertainty quantification show that the model with uncertainty quantification method has had slightly better performance than the model without uncertainty quantification in the DDSM and BUSM datasets. In general, the result shows well classification performance of the Bayesian Inference and indicates the proposed deep CNN model did not sacrifice accuracy while can estimate the uncertainty values.

Based on an image, the proposed deep CNN model can be certain (Figure 6(a)) or uncertain (Figure 6(b)) about its predictions. Figure 5 presents examples of breast cancer images and corresponding approximated predictive probability distribution produced by the Bayesian Inference in mammograms. Some images have high and some ones have low uncertainty in prediction, where in Figure 6, the uncertainty indicated by the standard deviation of the predictive distribution calculated with eq. (14). The mammogram of Figure 6(a) is confidently classified as benign by the proposed deep CNN model since the uncertainty value (standard deviation) is 0.01. Whereas, Figure 6(b) shows a case of the model is uncertain. In this case, the predicted label is benign, but with wider predictive distribution, and the uncertainty value (standard deviation) is 0.156.

Figure 6. Examples of breast cancer images and corresponding approximate predictive probability distribution produced by the bayesian inference. (a) image with low uncertainty in prediction from DDSM as certain benign and (b) image with high uncertainty in prediction from DDSM as benign. This case is predicted benign but slightly uncertain. The corresponding standard deviation (STD) as uncertainty is shown.

4.3.3. Relationship between misclassification cases and their uncertainty values

In this part, we investigate the relationship between uncertainty and incorrect prediction cases for increasing the performance of the CAD system by selecting appropriate subsets for further inspection, in order to mimic the human clinical workflow that they usually obtain more information when their predictions are uncertain (Leibig et al. 2017).

Some studies have illustrated that there is a relationship between uncertainty and the misclassification rate of a classifier explicitly and this determine that the misclassification rate increases with the growing up of uncertainty in some cases (Leibig et al. 2017; Zhou et al. 2020; Song et al. 2021). We found that the misclassified cases have higher uncertainty values than correct predictions in mammograms as well as ultrasound images. The uncertainty values for DDSM and BUSI test images were plotted and grouped by correct and incorrect predictions (see Figure 7 (a,b)). Figure 7 shows incorrect cases have higher uncertainty. Because high uncertainty was indicated of incorrect predictions, therefore this information could be used to find some cases for further inspection and leverage to increase the performance of the breast cancer classification CAD system.

Figure 7. Distribution of uncertainty value for all test data grouped by correct and incorrect model predictions from (a) mammograms (DDSM) and (b) ultrasound images (BUSI).

For example, Figure 8 presents some examples of the mammogram images with high uncertainty (≥0.1) in prediction which can be selected as appropriate subsets for further inspection.

Figure 8. Some examples of breast cancer images with high uncertainty (≥0.1) in predictions from mammograms which can be selected as appropriate subsets for further inspection. Predictive entropy is considered as the model uncertainty. (M and B are predicted labels as malignant and benign, respectively).

5. Discussion

Breast cancer is the most common malignant tumour in women. Using of medical imaging is one of the most effective methods for the diagnosis of breast cancer. Thus, the development of computer-aided diagnosis systems that are capable of accurately and reliably classifying breast cancer images is of immediate priority. Deep learning algorithms have become a good performance model for analysing medical images. In spite of the advantages, most deep learning-based approaches cannot quantify the uncertainty of their predictions. Uncertainty in the predicted results might endanger people’s life in these crucial steps for clinical diagnostics. Therefore, leveraging uncertainty information is imperative to ensure safe and reliable use of these novel tools. Accordingly, to have an accurate and reliable CAD system simultaneously, here, we provide an uncertainty-aware deep learning-based CAD system.

In this study, we proposed an uncertainty-aware deep learning-based CAD system for breast cancer classification task using Ultrasound and Mammography Images.

Differently from most of existing studies, which just used one type of images in breast cancer classification, we present a simple but efficient deep learning model, which is able to classify accurately different class of images, namely, mammograms and ultrasound.

In this work, four pre-trained deep learning models combining the CNN model for breast cancer classification are proposed. The different proposed models were compared according to accuracy and trainable parameters. As listed in Tables 3–5, the obtained results show that the Xception combining the CNN model gave the best classification performance in all of the datasets on average. So, this model was selected as the proposed deep CNN model.

We also estimated uncertainty values for each case using the Monte Carlo Dropout method as Bayesian Inference in the proposed deep CNN model. As the results showed in Table 8, the proposed deep CNN model could distinguish benign and malignant cases with high accuracy and reliability for breast cancer classification and indicate the proposed deep CNN model as a BNN has slightly better performance than the model without uncertainty quantification (Table 7) and does not sacrifice accuracy while can estimate the uncertainty values. From Table 10, it seems that our proposed deep CNN model, not only can quantify uncertainty values, but also the proposed approach appears to have comparable performance to some competent CAD models applied to the same datasets in both sonograms and mammograms.

Table 10. Comparison of the uncertainty-aware deep learning model with other existing methods tested on the used datasets for breast cancer classification.

Medical Image	Dataset	Study	Method	Accuracy (%)	Uncertainty quantification
Mammograms	DDSM	(Chougrad et al. 2018)	Inceptionv3+CNN	97.35	No
	DDSM	(Muduli et al. 2021)	CNN	90.68	No
	DDSM	(Al-Masni et al. 2018)	YOLO+CNN	97	No
	DDSM	(Ragab et al. 2019)	AlexNet +SVM	80.5	No
	DDSM	(Maqsood et al. 2022)	TCNN	99.08	No
	DDSM	(Kumar Pattanaik et al. 2022)	DenseNet121+ELM	99.14	No
	DDSM	Ours	Uncertainty-aware deep learning (xception+CNN+Mc droupout)	99.86	Yes
Sonograms	BUSM	(Al-Dhabyani et al. 2019)	CNN-AlexNet	80	No
	BUSM	(Al-Dhabyani et al. 2019)	TL- VGG16	82	No
	BUSM	(Al-Dhabyani et al. 2019)	TL- Inception	80	No
	BUSM	(Acevedo and Vazquez 2019)	k-means+SVM	94	No
	BUSM	Ours	Uncertainty-aware deep learning (xception+CNN+Mc droupout)	91.60	Yes
	BUSI	(Al-Dhabyani et al. 2019)	CNN-AlexNet	78	No
	BUSI	(Al-Dhabyani et al. 2019)	TL- VGG16	88	No
	BUSI	(Al-Dhabyani et al. 2019)	TL- Inception	85	No
	BUSI	(Cao et al. 2020)	NF-Net	73	No
	BUSI	(Ayana and Choe 2022)	TL-ResNet50	86.2	No
	BUSI	Ours	Uncertainty-aware deep learning (xception+CNN+Mc droupout)	88.57	Yes

TCNN: the Transferable Texture Convolutional Neural Network, ELM: Extreme Learning Machine, TL: Transfer Learning, NF-Net: noise filter network.

Finally, since we found that the proposed deep CNN model shows high uncertainty in misclassifications, so we used the uncertainty values to find appropriate subsets for further inspection, in order to mimic the human clinical workflow that they usually obtain more information when their predictions are uncertain.

5.1. Major advantages and disadvantages

In summary, there are some advantages and disadvantages. The main salient features of our method are given below:

• Achieved superior performance for breast cancer classification using Ultrasound and Mammography Images.

• Prediction of the uncertainty

• Analyzed of results in each class of datasets to ensure the transparency in diagnosis of each class.

The limitations and disadvantages of our proposed method are also presented below:

• Unable to return and investigate uncertain samples to medical experts due to lack of a medical team.

• Yields better performance using large medical data than for smaller data.

• The performance of the model is lower in unbalanced datasets.

6. Conclusion

Uncertainty in the predicted results from a Computer-Aided Diagnosis (CAD) system might endanger people’s life in these crucial steps for breast cancer classification. Therefore, Quantifying the uncertainty will enable the development of safe and reliable models for breast cancer classification.

In this study, we proposed an uncertainty-aware deep learning-based CAD system for breast cancer classification task. Differently from most of existing studies, which just used one type of images in breast cancer classification, we present a simple but efficient deep learning model, which is able to classify accurately different class of images, namely, mammograms and ultrasound.

The proposed deep CNN model achieved 99.95%, 91.34%, and 98.40% accuracy on DDSM, BUSI and BUSM datasets, respectively. Therefore, the results indicate the proposed deep CNN model does not sacrifice accuracy while can estimate the uncertainty values.

The limitations of the proposed uncertainty-aware deep learning-based CAD system will be addressed in our future studies. Thus, in the future, we intend to: (i) improve the generalisation power of the model using various datasets, (ii) the development of the proposed model using multi-modal data, and (iii) investigation of the uncertain samples by medical experts.

Disclosure statement

No potential conflict of interest was reported by the author(s)

Additional information

Notes on contributors

Mohaddeseh Chegini

Mohaddeseh Chegini is a PhD student at Tarbiat Modares university supervised by Dr.Ali Mahlooji Far. Mohaddeseh received her Bachelor’s degree from the Islamic Azad University and then completed her MS degree at Iran University of Science and Technology in biomedical engineering.

Ali Mahlooji Far

Dr. Ali Mahlooji Far is Associate Professor of biomedical engineering in Tarbiat Modares university. His research has been focused on medical Image Processing.