Leveraging bone age assessment via a novel joint decomposition teacher–student learning paradigm from X-ray images

ABSTRACT

Advancements in technology has accelerated the evolution of bone age assessment (BAA) methodologies, one of which is deep learning algorithms, which overcome the drawbacks of conventional approaches. In spite of excellent effectiveness of deep neural networks in detection of the correct class for bone age, they have a significant degree of complexity due to the numerous parameters they employ for each region of interest (ROI). In this paper, we propose a BAA method using a hybrid knowledge distillation (KD) paradigm in order to conquer this difficulty by mapping different ROIs into a single ROI. In this regard, the student receives knowledge from a teacher network that has been pre-trained on six ROIs including bones of five fingers and the wrist, transfers the knowledge of its final response layer and internal layers to the student. Then, six student models each of which is constructed based on just one of these ROIs, while receiving the information of the teacher model. Empirical results on digital hand atlas report that our student model trained on one ROI obtains 95% accuracy on 19 classes of bone age.

KEYWORDS:

• Bone age assessment
• knowledge distillation
• region of interest extraction
• deep learning
• edge device

1. Introduction

In recent years, bone age assessment (BAA) which is one of the significant medical measurements for bone maturity in radiological estimation, particularly in paediatric radiology has received a lot of attention. It detects the difference between bone age and chronological age, which could be a sign of anomalies in skeletal growth (Liu et al. 2019). BAA can correctly indicate level of maturity in human because bone age is a major indicator of the actual human growth, and the human skeleton is continuously growing (Bui et al. 2019). Paediatric endocrine problems and growth abnormalities are frequently diagnosed with BAA. Furthermore, a precise skeletal age assessment facilitates the estimation of height in adolescents, and diagnoses a number of diseases, including idiopathic dwarfism and early puberty (Ren et al. 2019; Deshmukh and Khaparde 2022). The BAA technique can help effectively in other similar research like calculating bone fracture risk through evaluation of cortical bone fracture resistance curves (Liu et al. 2019). The most popular approach to identify the level of skeletal maturity is the analysis of the left hand and wrist X-ray images due to its low operational cost, short exposure time for the subject to X-rays, and simplicity (An Ding et al. 2020; Saini et al. 2021). Traditionally, the Greulich-Pyle (GP) and Tanner-Whitehouse (TW) methods have been used to determine the age of bones. The GP method compares an X-ray image of a patient with an atlas made up of a number of standard bone age images to identify the patient’s skeletal age. The TW approach, in contrast, uses a scoring system that evaluate the degree of skeletal maturity in certain region of interest (ROI) in the bones of the hand and wrist. Depending on the maturity levels, a numerical score is then given to each ROI, and finally these scores are added to determine the overall maturity score. Despite the simplicity of the GP, and accuracy of TW, they both depend on the expertise of radiologists, and there is always a substantial intra and inter-observer variability (Simu et al. 2017; Chen et al. 2020).

Ongoing progress in computer technology accelerates the development of techniques resolving these problems; thus, numerous automated BAA approaches have been proposed based on image processing and computer vision techniques (Mouloodi et al. 2020; Salim and Ben Hamza 2021; Rzecki et al. 2021). They often deal with BAA as a classification or regression problem containing fundamental stages as hand segmentation, ROI detection, feature extraction and classifier or regressor model (Liu et al. 2019). Deep neural network methods especially convolutional neural networks (CNNs) are one of the promising solutions for a variety of medical image analysis applications so as it can be advantageous to BAA issue. To address BAA problem, a number of deep learning-based works have been introduced recently that learn distinguished attributes from a collection of training images automatically (Bui et al. 2019; Liang et al. 2019; Wu et al. 2019).

In Bui et al. (2019), a deep learning-based technique is presented by combining the TW3 and CNN based on extracted ROI. CNNs are trained to learn the maturity stage from each ROI of the TW3 approach in order to extract the feature engineering. A deep learning strategy for BAA is suggested in Han and Wang (2020), where a Resnet extract features from 17 important regions in the left-hand X-ray image, and the CNN is utilised for automated determination of the bone age. An Ding et al. (2020) suggest an automatic BAA approach from images of carpal radiography by extracting 14 ROIs, and classifying with a CNN. In Lu et al. (2021), 14 bones from the left hand are chosen as ROI, and a trained CNN determines the likelihood that each bone is in a particular developmental stage using each bone’s ROI. Chen et al. (2020) introduce a TW3-based BAA system using deep neural networks that classify levels of the skeletal maturity by extracting 13 ROIs. Although the aforementioned approaches attain high accuracy, they have complicated architecture as utilising multiple convolutional and pooling layers in their structure which impose a huge processing power on BAA systems because of computing millions of parameters. This volume of calculation requires high computational capacity and large storage that bring about difficulties in internet of medical things (IoMT). Since IoMT devices have constricted computing power and low capacity memory, these deep learning based BAA methods cannot be implemented on them. To deal with these constraints, several compression and acceleration techniques have been introduced recently like parameter pruning and sharing, transferred compact convolutional filters and knowledge distillation (KD). KD is one of the effective options among them in which a student network is trained under monitoring a progressive teacher network. Indeed, the student attempts to emulate the teacher’s action in a straightforward way in KD framework with least amount of loss (Fu et al. 2021; Sepahvand and Abdali-Mohammadi 2022).

This paper presents a novel BAA method with the help of KD paradigm using several ROIs extracted from X-ray images. To this end, a teacher network that is trained ahead with different ROIs and offers highly accurate results, supervises the training mechanism of a student network which is performed using one ROI with capability of easy deployment on healthcare internet of things (IoT). The teacher model is composed of advanced CNN design, whereas the student model is formed of a plain CNN structure makes it suitable for implementing in IoMT real time devices. Teacher model as source of knowledge provides information about different ROIs for the student in knowledge transfer phase in order to have more similar acting like the teacher by conveying knowledge to the student. In this paper, the transferred knowledge is a combination of the final response layer and intermediate feature map knowledge which result in performance improvement. Several tests are conducted on digital hand atlas (DHA) Database for evaluating our proposed method one of which is the complexity comparison between the teacher and the student model. This dataset is a comprehensive evaluation tool for automated bone age estimating methods which consists of digital left hand radiographs collected from children between the ages of 0 and 18 and different races. Outcomes reveal that student model estimate bone age with average accuracy of 85%, which is slightly less accurate than teacher models but far less complex. Other findings demonstrate our method’s superiority over other cutting-edge BAA techniques based on deep learning. Finally, the main contributions in this study are listed below.

• Developing a novel BAA method that employs KD paradigm with the minimum accuracy reduction rather than high complicated deep neural network models.

• Using feature maps from intermediate layers in combination with final layer output in teacher model in order to use as new knowledge that is conveyed to student model.

• This model is appropriate for deploying in IoMT due to its great accuracy and lightweight structure.

The rest of the paper is arranged as follows. Section 2 reviews recent studies carried out in BAA using deep neural networks. In Section 3, our proposed method is explained in particularity, while in Section 4, the experimental results are related. Finally, a conclusion is given in Section 5.

2. Related work

BAA using radiographic images of the left wrist is an important subject in machine learning due to its simplicity and numerous ossification centres. It has many applications, such as diagnosing growth disorders, human trafficking and competitive sports (Crawley 2007). In recent years, deep learning has presented various image processing algorithms, leading to major breakthroughs in related practical fields, such as medical imaging. In BAA with radiographic images, deep learning has overcome the need for expert knowledge and manual feature extraction with remarkable results. BAA studies based on the deep learning algorithm are explained in the following. In He and Jiang (2021), a compression module with 15 convolution layers and 4 pooling layers for big data compression and combined SE (squeeze-and-excitation) and ResNet with over 16 feature extraction and classification layers has presented. Spampinato et al. (2017) have introduced a 15-layer convolutional network called BoNet for BAA. Both methods (Spampinato et al. 2017; He and Jiang 2021) have compared the proposed networks’ performance with pre-trained networks such as VGG, R-CNN, GoogLeNet and OxfordNet, demonstrating the proposed methods’ superiority. In Li et al. (2022), an unsupervised learning method has been used with convolution layers to extract the largest area as the hand region. Then, the MobileNetV3 model was implemented for feature extraction and age detection as the head and backbone of multi-layer perceptron prediction. The model proposed in Deshmukh and Khaparde (2022) covers three main stages: (a) segmentation using Otsu’s method, (b) feature learning based on Faster R-CNN and (c) classification using the RNN algorithm. The method in Salim and Ben Hamza (2021) used a Mask R-CNN network for segmentation by first drawing a bounding box around the hand image. The proposed BAA model uses the VGG-19 CNN. Although none of these methods segmented hand components, segmenting the different parts of the hand image and using more networks and layers can help extract higher-level features and improve learning from data. In An Ding et al. (2020) and Lu et al. (2021), 14 ROI (finger and wrist joint) ossification centers from the hand image have been extracted in the preprocessing stage. After extracting the joints, An Ding et al. (2020) used the pre-trained VGG-19, GoogleNet and AlexNet to classify models. For classification, the proposed method in Lu et al. (2021) has used an AlexNet network to check the growth state of each ROI and determine bone age based on the total score of the 14 ossification centers and the age comparison table. Studies in Bui et al. (2019) and Son et al. (2019), Chen et al. (2020) have used Faster R-CNN to segment and extract the sensitive ossification regions in hand images. Bui et al. (2019) have employed the Inception architecture to extract six ROI regions and has investigated the growth state of each by integrating three continuous convolutions and max-pooling layers, a global average pooling layer, and two logits and softmax layers. For final age detection, each Inception network sent 1536 neurons from the last feature map as input for the support vector regression (SVR) network for age estimation. Chen et al. (2020) have classified the R-CNN network’s output vectors using a ST-Resnet network. In Han and Wang (2020), the features have been extracted and then the age of each ROI has been determined using a VGG-13 network, and the final results were obtained by an ensemble between VGG models using voting.

Segmenting the different parts of hand image and using deep learning models separately for each part has improved learning from data and the accuracy of models. In addition to improving accuracy, the greater number of neural networks has significantly increased the number of parameters and the model implementation complexity. For example, study in Son et al. (2019) has used 13 VGG-13 networks for extracting 13 ROIs for age detection; A VGG-13 network with 112 $\times$ 112 input images has $\sim 29\mathit{M}$ generated parameters. By using double precision storage (8-byte) for each sample, the volume of this model equals $\sim 29\mathit{M}\times 8\mathit{B}=232$ MB, or about $220$ MB for each ROI segment. This size increases further for larger images, more ROIs and deeper networks. Thus, although implementing BAA methods based on segmentation and deep learning can improve feature extraction and remove redundant image areas while providing greater accuracy than models without preprocessing and segmentation, they are inadequate due to their massive number of parameters and large computational burden. It is important to note that new fields such as IoMT and edge intelligence have tried to distribute some of the computational and recognition process to smartphones (with limited computing power) for efficient server use. However, this is infeasible with deep learning models due to their large size. Table 1 represents some of the BAA methods that have used deep learning approaches while their advantages and disadvantages are given, too.

Table 1. Review of some recent BAA methods.

Reference	Dataset	Method	Advantages	Disadvantages
He and Jiang (2021)	Private – 12,611 Image	SE-ResNe	Presented an image compression module without loss of important features	The model database only included Chinese children
Spampinato et al. (2017)	Public-DHA − 1380 Image	CNN-BoNet	Tested several existing pre-trained CNNs on a dataset	No image preprocessing
Deshmukh and Khaparde (2022)	RSNA – 12,611 Image	FR-CNN, RNN, AF-SFO	Good performance by the proposed Otsu segmentation model and BAA model compared to other machine learning algorithms	The proposed model’s complexity for implementation and medical use
Salim and Ben Hamza (2021)	RSNA – 12,611 Image	Mask R-CNN and VGG-19	The proposed model also performed well on the chest X-ray dataset	Using hand annotation for detecting hand image bounds
An Ding et al. (2020)	Public-DHA − 1380 Image	VGG-19, GoogleNet AlexNet	Segmented images and tested multiple deep learning methods on the data to achieve better results	Lower accuracy than the proposed model with the same dataset
Lu et al. (2021)	DR – 1666 Image	AlexNet	Improved age detection in the [−1, 0, 1] age error interval by up to 96%	Limited by the large number of neural network parameters and low computational resources
Bui et al. (2019)	Public-DHA − 1380 Image	R-CNN and Inception-SVR	The model database includes children of four different races and improves accuracy	This classification method is limited to assessment of six small ROI segments
Chen et al. (2020)	Private – 12,536 Image	R-CNN and ST-Resnet	The proposed model is (ST-Res) based on Res-Net and spatial transformer	The database is specific to Chinese children. The model is limited by low computational resources and big data
Son et al. (2019)	Private – 12,536 Image	R-CNN and VGG-13	Better image segmentation with good accuracy	The model database was specific to Korean children. High computational burden due to the large number of parameters
Wang et al. (2023)	Private – 12,536 Image	Interpretable CNN	Ability to interpret deep architecture used for bone age detection	High computational complexity of building the proposed architecture
Liu et al. (2022)	Private – 12,536 Image	Ensemble CNNs	Applying the idea of collective intelligence to increase the accuracy of bone age diagnosis	The model database was specific to Korean children. High computational burden due to the large number of parameters
Deshmukh and Khaparde (2022)	Private – 12,536 Image	Faster R-CNN	Better image segmentation with good accuracy	No image preprocessing

3. Proposed method

3.1. Preprocessing

In this section, we will describe all the steps of selecting six ROIs from the X-ray hand images in detail. First of all, the input image (Figures 1(1)) is converted to the binary ones by applying a threshold based approach on it, according to Equations 1(1) $\mathit{B}(\mathit{x},\mathit{y})=\left(\begin{array}{cc}1,& \mathit{if}\mathit{x}\ge T_{\mathit{x},\mathit{y}}\\ 0,& \mathit{otherwise}\end{array}\right.$(1) and 2(2) $T_{\mathit{x},\mathit{y}}=\sum _{\mathit{x}=0}^m\sum _{\mathit{y}=0}^n\mathit{f}(\mathit{x},\mathit{y})/(\mathit{m}\times \mathit{n})$(2) . The threshold (T_x,y) is defined as the average intensity of the image, and then binary image (B) is achieved by comparing each pixel of the image to the threshold (Figures 1(2)).

Figure 1. Steps of six ROIs’ extraction from an X-ray image from DHA database.

(1) $\mathit{B}(\mathit{x},\mathit{y})=\left(\begin{array}{cc}1,& \mathit{if}\mathit{x}\ge T_{\mathit{x},\mathit{y}}\\ 0,& \mathit{otherwise}\end{array}\right.$(1)

(2) $T_{\mathit{x},\mathit{y}}=\sum _{\mathit{x}=0}^m\sum _{\mathit{y}=0}^n\mathit{f}(\mathit{x},\mathit{y})/(\mathit{m}\times \mathit{n})$(2)

In these equations, $\mathit{m}$ and $\mathit{n}$ are height and width of the image, respectively and $\mathit{f}(.)$ is the pixel intensity. In the second step, hand part is extracted from image B with regionprops MATLAB function and then the largest region is obtained which belongs to hand. Next step aims to extract wrist bones region in such a way that first edges of the image B is detected using Sobel edge detector and the coordination of edges is stored in a vector (Figures 1(3)). Next, Hough algorithm is applied, and then a straight line is drawn from the middle of the wrist while rotating the image relative to this line so as to obtain hand image in a straight direction. At last, two lines are drawn vertically, parallel to the edges of the wrist, and a horizontal line is then drawn to the center of the hand that segments the hand (Figures 1(4)). Lastly, the wrist segment is elicited from these segments (Figures 1(5)). In the last step, finger ROIs are extracted using convex hull algorithm. In this algorithm, hand is converted to a polygon that fingertips are located at the vertices of the convex region (Figures 1(6)), and to determine their coordinates, the coordinates of the vertices of the convex region must be determined. For this purpose, a line slope calculation idea is utilised in such a way that coordination of convex regions is kept in a vector, and then line slope of consecutive points are calculated and store in a vector likewise. Points with maximum slope are fingertips or so close to the fingertips (Figures 1(7)). In the end, two points are specified as marker points from the fingertip to the center of the hand or from the fingertip to the center of the wrist, and a line is drawn between these two points (Figures 1(8)). Then, the image is rotated on the basis of this line to pose the intended finger in a straight position (Figures 1(9)). The region associated with each finger is identified according to the fingertip’s coordinate. This process is repeated for all five fingers. Finally, when all ROIs are determined, cropping is performed on the image B to extract ROIs (Figures 1(10)).

3.2. KD module

The design of the KD model used in BAA using DHA database is depicted in Figure 2. There are a teacher model consists of an advanced CNN with $\mathit{I}$ convolution layers and a simple student model with $\mathit{J}$ convolution layers in its CNN network, both of which followed by a fully connected layer and a softmax layer. While the teacher has been trained by six ROIs extracted from DHA images beforehand, the student will be trained with one of these extracted ROI. In KD, the student model attempts to emulate the teacher behavior so that provides comparable performance with respect to the teacher model besides using lightweight architecture. This can be accomplished by using the information of the teacher network known as dark knowledge, during training which is transmitted to the student model as a procedure called knowledge transfer. As it can be observed from the figure, two different types of knowledge, i.e. response-based and feature-based are combined and transferred from the teacher to the student, and at last the total error of the method is defined by a hybrid error comprises response-based KD loss $(L_{\mathit{Res}})$ and the feature-based KD loss $(L_{\mathit{Feat}})$ that is shown in Equation (3).(3) $L_{\mathit{total}}=\mathit{\gamma }{L}_{\mathit{Feat}}+(1-\mathit{\gamma })L_{\mathit{Res}}$(3) In this equation, gamma is a hyper-parameter that controls the relative importance of $(L_{\mathit{Feat}})$ and $(L_{\mathit{Res}})$ coefficients. The two following subsections analyse the performance of the proposed method in terms of each of these losses.

Figure 2. Diagram of student’s learning procedure based on hybrid KD model.

(3) $L_{\mathit{total}}=\mathit{\gamma }{L}_{\mathit{Feat}}+(1-\mathit{\gamma })L_{\mathit{Res}}$(3)

3.2.1. Response-based KD loss

The objective behind response-based knowledge is to directly transfer the output of the last layer such as soft targets, which come in the form of a logit vector, to the student side in order to mimic the teacher model’s final prediction. Thus, to measure how well the student can imitate the teacher’s manner, distillation loss is computed as:

(4) $L_{\mathit{Res}}=\mathit{\alpha }{L}_{\mathit{Soft}}+(1-\mathit{\alpha })L_{\mathit{Hard}}$(4)

$L_{\mathit{soft}}$ represents the discrepancy in the logits between the student and teacher models that is formulated as:

(5) $L_{\mathit{Soft}}=-\sum _{\mathit{c}=1}^N{p}_c^{T}log{p}_c^S$(5)

$p_c^T$ and $p_c^S$ indicate the likelihood that input x matches to the $\mathit{c}$th class in the teacher and student network, respectively, which are calculated through a softmax function as:

(6) $p_c^T=\frac{e^{z_c^t}/\mathit{T}}{{\sum }_{d=1}^N{e}^{z_d^t}/\mathit{T}}$(6)

(7) $p_c^S=\frac{e^{z_c^s}/\mathit{T}}{{\sum }_{d=1}^N{e}^{z_d^s}/\mathit{T}}$(7)

Where $z_c^t$ and $z_c^s$ refers to the logit vector of teacher and student for the $\mathit{c}$th class, correspondingly. The teacher model uses $p_c^T$ to predict soft targets which contains the dark knowledge used as a supervisor to transmit knowledge to the student. Unlike hard targets (ground-truth labels) that are used in conventional classification and feature on-hot coding, soft targets are created by dividing hard targets by a controlling variable called temperature (T), where a greater temperature results in a softer probability distribution among classes. In particular, the probability for each class would be identical by increasing T to infinity, and the soft targets change into hard labels as T nears to zero value (Hinton et al. 2015). Therefore, $(L_{\mathit{Soft}})$ will be achieved by cross-entropy loss function according to Equation 5(5) $L_{\mathit{Soft}}=-\sum _{\mathit{c}=1}^N{p}_c^{T}log{p}_c^S$(5) which measures the distance between the teacher and student’s logits in the final layer. Distillation conveys valuable information to the student by the teacher through the logits comparison with higher values of $\mathit{T}$. In addition to soft targets, the ground-truth label is important for improving the student model’s accuracy. So, the student’s error $(L_{\mathit{Hard}})$ is considered in distillation loss $(L_{\mathit{Res}})$, and is calculated by the cross-entropy loss between the soft error of the student and the hard target as below:

(8) $L_{\mathit{Hard}}=-\sum _{\mathit{c}=1}^N{y}_{c}log{p}_c^S$(8)

Where $y_c$ is the label of the input x, and $p_c^S$ is calculated as Equation 6(6) $p_c^T=\frac{e^{z_c^t}/\mathit{T}}{{\sum }_{d=1}^N{e}^{z_d^t}/\mathit{T}}$(6) except that here the value of T is considered 1. Back to Equation 4(4) $L_{\mathit{Res}}=\mathit{\alpha }{L}_{\mathit{Soft}}+(1-\mathit{\alpha })L_{\mathit{Hard}}$(4) , $L_{\mathit{Hard}}$ and $L_{\mathit{Soft}}$ are calculated separately and then integrated with each other by a hyper-parameter $(\mathit{\alpha })$. It regulates the proportion of each of the two coefficients’ effect.

3.2.2. Feature-based KD loss

Deep neural networks have performed well in learning features at different levels of their internal layers by increasing abstraction. This is known as representational learning; Therefore, both the output of the last layer and the output of the intermediate layers, i.e. feature maps, can be used as knowledge to supervise the training of the learner model. In particular, knowledge based on feature extraction from intermediate layers is a good generalisation of response-based knowledge. Especially for training shallow networks by deep networks. The teacher model consists of $\mathit{X}$ number of layers and the student model consists of $\mathit{Z}$ number of layers. From both teacher and student models, the number $\mathit{G}$ of the selection layer and their feature mapping matrix are extracted. The knowledge distance of these layers is checked with each other and the optimisation function of this method tries to minimise this distance.

For teacher and student deep neural networks, which are composed of $\mathit{X}$ and $\mathit{Z}$ layers, respectively, the output of each layer is transferred to the next layer. If the output of each layer is a function of $\mathcal{Y}=\mathit{F}(\mathcal{W}\mathit{K}\pm \mathit{b})$, then the output of the final layer for each of the models is equal to Equation 9(9) $\begin{array}{cc}\mathcal{Y}=& { }^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}⟩}{\mathcal{W}}^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}-1⟩}{\mathcal{W}}^{⟨\mathit{X}-1⟩}\mathit{F}{(.}^{⟨\mathit{X}-2⟩}{\mathcal{W}}^{⟨\mathit{X}-2⟩}\mathit{F}(.\cdots {(.}^{⟨1⟩}\mathit{F}(\\ \mathit{\hspace{0.17em}}& {.}^{⟨1⟩}\mathcal{W}\mathbf{K}{+}^{⟨1⟩}\mathit{b}.).)\cdots .){+}^{⟨\mathit{X}-2⟩}\mathit{b}.){+}^{⟨\mathit{X}-1⟩}\mathit{b}.){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(9) :

(9) $\begin{array}{cc}\mathcal{Y}=& { }^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}⟩}{\mathcal{W}}^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}-1⟩}{\mathcal{W}}^{⟨\mathit{X}-1⟩}\mathit{F}{(.}^{⟨\mathit{X}-2⟩}{\mathcal{W}}^{⟨\mathit{X}-2⟩}\mathit{F}(.\cdots {(.}^{⟨1⟩}\mathit{F}(\\ \mathit{\hspace{0.17em}}& {.}^{⟨1⟩}\mathcal{W}\mathbf{K}{+}^{⟨1⟩}\mathit{b}.).)\cdots .){+}^{⟨\mathit{X}-2⟩}\mathit{b}.){+}^{⟨\mathit{X}-1⟩}\mathit{b}.){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(9)

where ${ }^{⟨\mathit{x}⟩}\mathit{b}$ and ${ }^{⟨\mathit{x}⟩}\mathcal{W}$ refer to the bias and weight tensor of the xth layer, respectively. The structure of tensors is complicated and difficult to calculate the difference between tensors due to their high dimensions. For this reason, the proposed model tries to break down this complexity by using Tucker’s decomposition and create a simpler example of tensors for faster calculations. To decompose the tensors related to the teacher and student layers, in the proposed model, the weight tensor ${ }^{⟨\mathit{x}⟩}\mathcal{W}$ of the layers is first decomposed. Equation 10(10) ${ }^{⟨\mathit{x}⟩}\mathcal{W}{\cong }^{⟨\mathit{x}⟩}\mathcal{Q}{{\times }_1}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(1)}{{\times }_2}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{N})}\text{break}{=}^{⟨\mathit{x}⟩}\mathcal{Q}{,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(1)}{,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(2)},\cdots {,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{N})}$(10) refers to the decomposed form of a weight tensor ${ }^{⟨\mathit{x}⟩}\mathcal{W}$ using the Tucker decomposition:

(10) ${ }^{⟨\mathit{x}⟩}\mathcal{W}{\cong }^{⟨\mathit{x}⟩}\mathcal{Q}{{\times }_1}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(1)}{{\times }_2}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{N})}\text{break}{=}^{⟨\mathit{x}⟩}\mathcal{Q}{,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(1)}{,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(2)},\cdots {,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{x}⟩}{\mathbf{M}}^{(\mathit{N})}$(10)

where ${ }^{⟨\mathit{x}⟩}\mathcal{Q}\in R^{I_1\times I_2\times \cdots \times I_N}$ represents the weight kernel tensor of the $\mathit{x}$th layer, $M^{(\mathit{n})}$ to the factor matrix in the $\mathit{n}$th order and ${\times }_n(\mathit{n}=1,2,\dots ,\mathit{N})$ to the product between Kernel tensor and factor matrices are pointed. Also, each example of Equation 9(9) $\begin{array}{cc}\mathcal{Y}=& { }^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}⟩}{\mathcal{W}}^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}-1⟩}{\mathcal{W}}^{⟨\mathit{X}-1⟩}\mathit{F}{(.}^{⟨\mathit{X}-2⟩}{\mathcal{W}}^{⟨\mathit{X}-2⟩}\mathit{F}(.\cdots {(.}^{⟨1⟩}\mathit{F}(\\ \mathit{\hspace{0.17em}}& {.}^{⟨1⟩}\mathcal{W}\mathbf{K}{+}^{⟨1⟩}\mathit{b}.).)\cdots .){+}^{⟨\mathit{X}-2⟩}\mathit{b}.){+}^{⟨\mathit{X}-1⟩}\mathit{b}.){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(9) can be expressed in the form of equation 11(11) ${ }^{〈\mathit{x}〉}{W}_{r_1,\cdots ,r_N}={\displaystyle \sum _{i_1=1}^{I_1}{\displaystyle \sum _{i_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{i_N=1}^{I_L}〈\mathit{x}〉q_{r_1,\cdots ,r_N}{ }^{〈\mathit{x}〉}{M}_{\left(X_1,i_1\right)}^{\left(1\right)}〈\mathit{x}〉M_{\left(X_2,i_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{x}〉}{M}_{\left(X_N,i_N\right)}^{\left(\mathit{N}\right)}$(11) :

(11) ${ }^{〈\mathit{x}〉}{W}_{r_1,\cdots ,r_N}={\displaystyle \sum _{i_1=1}^{I_1}{\displaystyle \sum _{i_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{i_N=1}^{I_L}〈\mathit{x}〉q_{r_1,\cdots ,r_N}{ }^{〈\mathit{x}〉}{M}_{\left(X_1,i_1\right)}^{\left(1\right)}〈\mathit{x}〉M_{\left(X_2,i_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{x}〉}{M}_{\left(X_N,i_N\right)}^{\left(\mathit{N}\right)}$(11)

The decomposed deep model for the teacher model is obtained by substituting Equation 11(11) ${ }^{〈\mathit{x}〉}{W}_{r_1,\cdots ,r_N}={\displaystyle \sum _{i_1=1}^{I_1}{\displaystyle \sum _{i_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{i_N=1}^{I_L}〈\mathit{x}〉q_{r_1,\cdots ,r_N}{ }^{〈\mathit{x}〉}{M}_{\left(X_1,i_1\right)}^{\left(1\right)}〈\mathit{x}〉M_{\left(X_2,i_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{x}〉}{M}_{\left(X_N,i_N\right)}^{\left(\mathit{N}\right)}$(11) in Equation 9(9) $\begin{array}{cc}\mathcal{Y}=& { }^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}⟩}{\mathcal{W}}^{⟨\mathit{X}⟩}\mathit{F}{(.}^{⟨\mathit{X}-1⟩}{\mathcal{W}}^{⟨\mathit{X}-1⟩}\mathit{F}{(.}^{⟨\mathit{X}-2⟩}{\mathcal{W}}^{⟨\mathit{X}-2⟩}\mathit{F}(.\cdots {(.}^{⟨1⟩}\mathit{F}(\\ \mathit{\hspace{0.17em}}& {.}^{⟨1⟩}\mathcal{W}\mathbf{K}{+}^{⟨1⟩}\mathit{b}.).)\cdots .){+}^{⟨\mathit{X}-2⟩}\mathit{b}.){+}^{⟨\mathit{X}-1⟩}\mathit{b}.){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(9) as follows:

(12) $\begin{array}{cc}\mathcal{Y}& =\mathit{F}(.\left[{ }^{⟨\mathit{X}⟩}\mathcal{Q}{,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(1)}{,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\mathit{F}(.\left[{ }^{⟨\mathit{X}-1⟩}\mathcal{Q}{,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(1)}{,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\cdots (.\mathit{F}(.\left[{ }^{⟨1⟩}\mathcal{Q}{,}^{⟨1⟩}{\mathcal{M}}^{(1)}{,}^{⟨1⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨1⟩}{\mathcal{M}}^{(\mathit{N})}\right]\mathit{X}\\ \mathit{\hspace{0.17em}}& {+}^{⟨1⟩}\mathit{b}.).).){+}^{⟨\mathit{X}-1⟩}\mathit{b}.).){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(12)

In the following, if ${ }^{⟨\mathit{g}⟩}\mathcal{J}$ refers to the tensor of the feature mapping of the $\mathit{g}$th layer obtained from each function ${ }^{⟨\mathit{g}⟩}\mathit{F}$ in the teacher model, the resulting Tucker format for the tensor of each layer of the teacher model is as follows:

(13) ${ }^{⟨\mathit{g}⟩}\mathcal{J}{\cong }^{⟨\mathit{g}⟩}\mathcal{T}{{\times }_1}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{{\times }_2}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}=\left[\left[{ }^{⟨\mathit{g}⟩}\mathcal{T}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}\right]\right]$(13)

${ }^{⟨\mathit{g}⟩}\mathcal{T}\in R^{I_1\times I_2\times \cdots \times I_N}$ refers to the decomposed kernel tensor from the $\mathit{g}$th layer of teacher’s model. Also, ${\mathbf{V}}^{(\mathit{n})}$ refers to the factor matrix in the $\mathit{n}$th order of the teacher’s model. Each case of Equation (13)(13) ${ }^{⟨\mathit{g}⟩}\mathcal{J}{\cong }^{⟨\mathit{g}⟩}\mathcal{T}{{\times }_1}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{{\times }_2}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}=\left[\left[{ }^{⟨\mathit{g}⟩}\mathcal{T}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}\right]\right]$(13) is represented as Equation (14)(14) ${ }^{〈\mathit{g}〉}{J}_{k_1,\cdots ,k_N}={\displaystyle \sum _{j_1=1}^{I_1}{\displaystyle \sum _{j_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{j_N=1}^{I_L}〈\mathit{g}〉T_{k_1,\cdots ,k_N}{ }^{〈\mathit{g}〉}{V}_{\left(k_1,j_1\right)}^{\left(1\right)}〈\mathit{g}〉V_{\left(k_2,j_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{g}〉}{V}_{\left(k_N,j_N\right)}^{\left(\mathit{N}\right)}$(14) .

(14) ${ }^{〈\mathit{g}〉}{J}_{k_1,\cdots ,k_N}={\displaystyle \sum _{j_1=1}^{I_1}{\displaystyle \sum _{j_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{j_N=1}^{I_L}〈\mathit{g}〉T_{k_1,\cdots ,k_N}{ }^{〈\mathit{g}〉}{V}_{\left(k_1,j_1\right)}^{\left(1\right)}〈\mathit{g}〉V_{\left(k_2,j_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{g}〉}{V}_{\left(k_N,j_N\right)}^{\left(\mathit{N}\right)}$(14)

The deep student model consists of $\mathit{Z}$ number of layers, the output of each layer is transferred to the next layer. If the output of each student layer is a function of $\mathcal{H}=\mathit{F}(\mathcal{W}\mathit{K}\pm \mathit{b})$, then Equation 15(15) $\begin{array}{cc}\mathcal{H}& =\mathit{F}(.\left[{ }^{⟨\mathit{Z}⟩}\mathcal{L}{,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(1)}{,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\mathit{F}(.\left[{ }^{⟨\mathit{Z}-1⟩}\mathcal{L}{,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(1)}{,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\cdots (.\mathit{F}(.\left[{ }^{⟨1⟩}\mathcal{L}{,}^{⟨1⟩}{\mathcal{A}}^{(1)}{,}^{⟨1⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨1⟩}{\mathcal{A}}^{(\mathit{N})}\right]\mathit{K}\\ \mathit{\hspace{0.17em}}& {+}^{⟨1⟩}\mathit{b}.).).){+}^{⟨\mathit{Z}-1⟩}\mathit{b}.).){+}^{⟨\mathit{Z}⟩}\mathit{b}.)\end{array}$(15) refers to the deep decomposed student model:

(15) $\begin{array}{cc}\mathcal{H}& =\mathit{F}(.\left[{ }^{⟨\mathit{Z}⟩}\mathcal{L}{,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(1)}{,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨\mathit{Z}⟩}{\mathcal{A}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\mathit{F}(.\left[{ }^{⟨\mathit{Z}-1⟩}\mathcal{L}{,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(1)}{,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨\mathit{Z}-1⟩}{\mathcal{A}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\cdots (.\mathit{F}(.\left[{ }^{⟨1⟩}\mathcal{L}{,}^{⟨1⟩}{\mathcal{A}}^{(1)}{,}^{⟨1⟩}{\mathcal{A}}^{(2)},\cdots {,}^{⟨1⟩}{\mathcal{A}}^{(\mathit{N})}\right]\mathit{K}\\ \mathit{\hspace{0.17em}}& {+}^{⟨1⟩}\mathit{b}.).).){+}^{⟨\mathit{Z}-1⟩}\mathit{b}.).){+}^{⟨\mathit{Z}⟩}\mathit{b}.)\end{array}$(15)

The resulting Tucker format for the tensor of each layer of the student model is in the form of Equation 16(16) ${ }^{⟨\mathit{g}⟩}\mathcal{K}{\cong }^{⟨\mathit{g}⟩}\mathcal{S}{{\times }_1}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(1)}{{\times }_2}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{N})}\text{break}{=}^{⟨\mathit{g}⟩}\mathcal{S}{,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(1)}{,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(2)}\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{N})}$(16) :

(16) ${ }^{⟨\mathit{g}⟩}\mathcal{K}{\cong }^{⟨\mathit{g}⟩}\mathcal{S}{{\times }_1}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(1)}{{\times }_2}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{N})}\text{break}{=}^{⟨\mathit{g}⟩}\mathcal{S}{,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(1)}{,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(2)}\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{D}}^{(\mathit{N})}$(16)

where $\mathcal{H}=\mathit{F}(\mathcal{W}\mathit{K}\pm \mathit{b})$ is the feature mapping tensor of the $\mathit{g}$th layer obtained from each function ${ }^{⟨\mathit{g}⟩}\mathit{F}$ in the student model. Also, ${ }^{⟨\mathit{g}⟩}\mathcal{S}$ and ${\mathbf{D}}^{(\mathit{n})}$, respectively, refer to the kernel tensor resulting from the analysis of the feature mapping and the factor matrix in the $\mathit{n}$th order. Each example of this equation can be rewritten in the following form:

(17) ${ }^{〈\mathit{g}〉}{K}_{k_1,\cdots ,k_N}={\displaystyle \sum _{r_1=1}^{I_1}{\displaystyle \sum _{r_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{r_N=1}^{I_N}〈\mathit{g}〉S_{k_1,\cdots ,k_N}{ }^{〈\mathit{g}〉}{D}_{\left(k_1,r_1\right)}^{\left(1\right)}〈\mathit{g}〉D_{\left(k_2,r_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{g}〉}{D}_{\left(k_N,r_N\right)}^{\left(\mathit{N}\right)}$(17)

After extracting the tensors of the decomposed core of the feature maps from the middle layers of the student and teacher models, the last step in knowledge transfer is based on the feature of knowledge transfer from teacher to student in the form of a transfer module. For this reason, the function ${ }^{⟨\mathit{g}⟩}\mathit{F}$ is defined based on Euclidean soft $L_2$, which monitors the performance of the student model by using the $\mathit{\alpha }$ coefficient. In general, the error of this method can be formulated as follows:

(18) $\begin{array}{c}\mathit{Los}{s}_{\mathit{IF}}=\sum _{\mathit{g}=1}^G{L}_2\left({ }^{⟨\mathit{g}⟩}\mathcal{T},{}^{⟨\mathit{g}⟩}\mathcal{S}\right)\end{array}$(18)

3.3. Student model training

In this section, training process of the student model under supervision of the pre-trained teacher is expressed in the form of algorithm 1. The pre-trained teacher $M_t$ and one ROI extracted from DHA dataset in the form of $\mathit{D}=\mathit{X},\mathit{Y}$ are fed into this algorithm as input. Each of the steps of training procedure is performed on all of the samples in X. To this end, first, the response based loss $(L_{\mathit{Res}})$ is computed by calculating $p_c^t$ and $p_c^s$ in order to obtain $L_{\mathit{hard}}$ and $L_{\mathit{soft}}$ as explained in Equations 4(4) $L_{\mathit{Res}}=\mathit{\alpha }{L}_{\mathit{Soft}}+(1-\mathit{\alpha })L_{\mathit{Hard}}$(4) –8(8) $L_{\mathit{Hard}}=-\sum _{\mathit{c}=1}^N{y}_{c}log{p}_c^S$(8) After that, error of feature based $(L_{\mathit{Feat}})$ student is calculated as follows. Four internal layers are derived from teacher and student models, and are given to four modules to compute the distance between their feature maps tensors by measuring Euclidean norm of their mode-n unfolding form through Equation 12.(12) $\begin{array}{cc}\mathcal{Y}& =\mathit{F}(.\left[{ }^{⟨\mathit{X}⟩}\mathcal{Q}{,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(1)}{,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨\mathit{X}⟩}{\mathcal{M}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\mathit{F}(.\left[{ }^{⟨\mathit{X}-1⟩}\mathcal{Q}{,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(1)}{,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨\mathit{X}-1⟩}{\mathcal{M}}^{(\mathit{N})}\right]\\ \mathit{\hspace{0.17em}}& \times (.\cdots (.\mathit{F}(.\left[{ }^{⟨1⟩}\mathcal{Q}{,}^{⟨1⟩}{\mathcal{M}}^{(1)}{,}^{⟨1⟩}{\mathcal{M}}^{(2)},\cdots {,}^{⟨1⟩}{\mathcal{M}}^{(\mathit{N})}\right]\mathit{X}\\ \mathit{\hspace{0.17em}}& {+}^{⟨1⟩}\mathit{b}.).).){+}^{⟨\mathit{X}-1⟩}\mathit{b}.).){+}^{⟨\mathit{X}⟩}\mathit{b}.)\end{array}$(12) Then, these differences are summed to obtain $(L_{\mathit{Feat}})$. In the last step, overall loss of our KD model is computed as weighted sum of $(L_{\mathit{Res}})$ and $(L_{\mathit{Feat}})$ according to Equation 3(3) $L_{\mathit{total}}=\mathit{\gamma }{L}_{\mathit{Feat}}+(1-\mathit{\gamma })L_{\mathit{Res}}$(3) and the parameters are optimised through the backpropagation process in each iteration.

Table

Algorithm 1: Student Model Training Procedure with Single ROI
1: $\mathbf{Input}\mathbf{:}\mathit{D}=\left\{\mathit{X},\mathit{Y}\right\}\in {\mathbb{R}}^{I_1\times I_2\times I_3}//\mathrm{single}\mathrm{ROI}$
2: $\mathrm{with}:$
3: $M_t//\mathrm{Pre}-\mathrm{trained}\mathrm{teacher}\mathrm{model}$
4: $\mathbf{TraintheStudentModel}\mathit{\hspace{0.17em}}{M}_s:$
5: for $\mathit{i}=1$ to len$(\mathit{X})$ do
6: $p^{\mathcal{T}}=\mathcal{T}(\mathit{X}[\mathit{i}])$
7: $p^{\mathcal{S}}=\mathcal{S}(\mathit{X}[\mathit{i}])$
8: Compute $L_{\mathit{Soft}}$ using Equation 5(5) $L_{\mathit{Soft}}=-\sum _{\mathit{c}=1}^N{p}_c^{T}log{p}_c^S$(5)
9: Compute $L_{\mathit{Res}}$ using Equation 4(4) $L_{\mathit{Res}}=\mathit{\alpha }{L}_{\mathit{Soft}}+(1-\mathit{\alpha })L_{\mathit{Hard}}$(4)
10: Choose four feature maps from teacher $(\mathcal{T})$ and student $(\mathcal{S})$
11: for $\mathit{e}=1$ to $4$ do
12: Convert feature map tensors from previous step to matrices by applying mode-n unfolding ${ }^{⟨\mathit{e}⟩}{\mathcal{T}}_{(\mathit{n})}$ and ${ }^{⟨\mathit{e}⟩}{\mathcal{S}}_{(\mathit{n})}$
13: Compute Euclidean norm $\left(L_2\left({ }^{⟨\mathit{e}⟩}{\mathcal{T}}_{(\mathit{n})}{,}^{⟨\mathit{e}⟩}{\mathcal{S}}_{(\mathit{n})}\right)\right)$ with Equation 18(18) $\begin{array}{c}\mathit{Los}{s}_{\mathit{IF}}=\sum _{\mathit{g}=1}^G{L}_2\left({ }^{⟨\mathit{g}⟩}\mathcal{T},{}^{⟨\mathit{g}⟩}\mathcal{S}\right)\end{array}$(18)
14: Calculate $L_{\mathit{dist},\mathit{e}}\left({ }^{⟨\mathit{e}⟩}{\mathcal{T}}_{(\mathit{n})}{,}^{⟨\mathit{e}⟩}{\mathcal{S}}_{(\mathit{n})}\right)$ according to Equation 17(17) ${ }^{〈\mathit{g}〉}{K}_{k_1,\cdots ,k_N}={\displaystyle \sum _{r_1=1}^{I_1}{\displaystyle \sum _{r_2=1}^{I_2}\cdots }}{\displaystyle \sum }_{r_N=1}^{I_N}〈\mathit{g}〉S_{k_1,\cdots ,k_N}{ }^{〈\mathit{g}〉}{D}_{\left(k_1,r_1\right)}^{\left(1\right)}〈\mathit{g}〉D_{\left(k_2,r_2\right)}^{\left(2\right)}{\cdots }^{〈\mathit{g}〉}{D}_{\left(k_N,r_N\right)}^{\left(\mathit{N}\right)}$(17)
15: end for
16: Compute $(L_{\mathit{Feat}})$ using Equation 13(13) ${ }^{⟨\mathit{g}⟩}\mathcal{J}{\cong }^{⟨\mathit{g}⟩}\mathcal{T}{{\times }_1}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{{\times }_2}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {{\times }_n}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})}\cdots {{\times }_N}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}=\left[\left[{ }^{⟨\mathit{g}⟩}\mathcal{T}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(1)}{,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(2)}\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{n})},\cdots {,}^{⟨\mathit{g}⟩}{\mathbf{V}}^{(\mathit{N})}\right]\right]$(13)
17: Use Equation 3(3) $L_{\mathit{total}}=\mathit{\gamma }{L}_{\mathit{Feat}}+(1-\mathit{\gamma })L_{\mathit{Res}}$(3) to obtain total loss $(L_{\mathit{total}})$ for the student model $M_s$
18: Execute Backward propagation for $M_s$
19: end for

4. Experimental result

This section illustrates the effectiveness of our suggested strategy using the DHA database while comparing it with other state-of-the-art approaches. Moreover, simulation and execution process are briefly discussed.

4.1. Dataset

DHA¹ serves as a thorough and public evaluation tool for automated bone age estimation techniques. It is made up of 1400 digital left hand radiographs taken from uniformly distributed normally developing Caucasian (CA), Asian (AS), African-American (AA), and Hispanic (HI) male (M) and female (F) children, aged 0 to 18 years old. Additionally, each image includes two bone age estimates provided by two radiologists. The entire process of experiments and analysis has been implemented on a computer with an Intel i7-9750 H processor, 64 GB of RAM, and an Nvidia RTX 2060 GDDR6 SLI GPU with 12 GB of memory. Through the Adam optimiser, the weights for the student and teacher networks were determined. Each of experiments is executed with 200 iterations with a learning rate of 0.006 for the student model, and in each quarter, a rate of 0.01 was increased. The value of $\mathit{\gamma }$ and $\mathit{\alpha }$ is set to 0.9. In order to gain reliable performance in evaluation, dataset is partitioned into train, validation and test sets in proportions of 70%, 15% and 15%, accordingly. The criteria used to assess the outcomes include accuracy, precision, recall, specificity and mean absolute error (MAE). These metrics’ calculation are as below:

(19) $\mathit{Accuracy}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}}$(19)

(20) $\mathit{Precision}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}$(20)

(21) $\mathit{Recall}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$(21)

(22) $\mathit{Specificity}=\frac{\mathit{TN}}{\mathit{FP}+\mathit{FN}}$(22)

(23) $\mathit{F}-\mathit{Score}=\frac{2\times \mathit{Precision}\times \mathit{Recall}}{\mathit{Precision}+\mathit{Recall}}$(23)

(24) $\mathit{MAE}=1\text{overm}\sum _{\mathit{i}=1}^m\left|y_i-{\hat{y}}_{\mathit{i}}\right|$(24)

4.2. Performance of student model

As discussed in the preceding section, the teacher model uses all six DHA ROIs – fingers 1 through 5 and the wrist – in training process while the student model only uses one of these ROIs. There are, in fact, six student models, referred to as student-1, student-2, student-3, student-4, student-5 and student-6, each of which has been trained on a different ROI. Table 2 presents the findings of the evaluation of these models and the teacher model. It demonstrates that the student model could, on average, attain accuracy of 85%, precision of 56%, recall of 56%, F-score of 56% and specificity of 88% for 19 age categories at each ROI. In contrast, the teacher model performed on average at accuracy of 97%, precision of 76%, recall of 76%, F-score of 76%, specificity of 98% and MAE of 0.013. These results reveals that the student model performs almost as well as its teacher, with a slight decrease of about 12% in accuracy, 20% in precision, 20% in recall, 20% in F-score, 10% in specificity, and 13% in MAE. Furthermore, Table 2 suggests that student-6 which is built based on wrist region higher competence than other students with accuracy of 95%, precision of 65%, recall of 65%, F-score of 65%, specificity of 87% and MAE of 0.022 than other students. Therefore, student-6 is taken into account to do a more thorough analysis with the teacher and other methods.

Table 2. Evaluation results of six students and the teacher model.

Model	ROI	Accuracy	Precision	Recall	F-Score	Specificity	MAE
Student model	Student-1	0.84 $\pm$ 0.00	0.53 $\pm$ 0.47	0.55 $\pm$ 0.00	0.53 $\pm$ 0.98	0.86 $\pm$ 0.25	0.034
	Student-2	0.85 $\pm$ 0.56	0.55 $\pm$ 0.67	0.54 $\pm$ 0.12	0.54 $\pm$ 0.50	0.90 $\pm$ 0.69	0.398
	Student-3	0.84 $\pm$ 0.58	0.54 $\pm$ 0.80	0.54 $\pm$ 0.76	0.54 $\pm$ 0.00	0.90 $\pm$ 0.84	0.345
	Student-4	0.81 $\pm$ 0.07	0.54 $\pm$ 0.14	0.55 $\pm$ 0.87	0.54 $\pm$ 0.50	0.88 $\pm$ 0.85	0.030
	Student-5	0.83 $\pm$ 0.96	0.55 $\pm$ 0.86	0.54 $\pm$ 0.87	0.54 $\pm$ 0.50	0.88 $\pm$ 0.47	0.031
	Student-6	0.95 $\pm$ 0.54	0.65 $\pm$ 0.01	0.65 $\pm$ 0.69	0.65 $\pm$ 0.00	0.87 $\pm$ 0.58	0.022
	Average	0.85 $\pm$ 0.45	0.56 $\pm$ 0.49	0.56 $\pm$ 0.55	0.56 $\pm$ 0.00	0.88 $\pm$ 0.61	0.143
Teacher model	All six ROIs	0.97 $\pm$ 0.03	0.76 $\pm$ 0.42	0.76 $\pm$ 0.54	0.76 $\pm$ 0.00	0.98 $\pm$ 0.87	0.013

The complexity and size of the networks in the train and test sets for the student-6 and teacher models are compared in Table 3. The results clearly show that the student model is less sophisticated than the teacher model; during the training phase, the student has about 2.34 million parameters, and classification is completed in 2.57 billion floating-point operations per second (GFLOPs) while 1.43 G of graphics processing unit (GPU) memory is used. Also, it has a 124.34 compression rate, making it lightweight. In addition to offering high accuracy, this feature makes it appropriate for deployment in edge devices.

Table 3. Comparison of the student-6 and the teacher regarding complexity and size of network.

Phase	Model	Number of parameters	Computation (GFLOPs)	GPU memory	Compression rate
Train	Student-6	$\sim$ 2.34 $\mathit{M}$	$\sim$ 0.43	0.08 $\mathit{G}$	$\times$124.34
	Teacher (All six ROIs)	$\sim$ 9.46 $\mathit{M}$	$\sim$ 2.57	1.43 $\mathit{G}$	$\times$32.01
Test	Student-6	$\sim$ 0.09 $\mathit{M}$	$\sim$ 0.01	0.01 $\mathit{G}$	$\times$66.12
	Teacher (All six ROIs)	$\sim$ 0.65 $\mathit{M}$	$\sim$ 0.12	0.12 $\mathit{G}$	$\times$3.09

The confusion matrices for the student-6 and teacher models for 19 age classes of age 0 to age 18 are illustrated in Figure 3. The main diagonal of the confusion matrix shows the number of samples that our desired targets are classified correctly which called TP and it is has significant influence on evaluation metrics. For the student confusion matrix, the student could identify the most of the classes correctly for each of 19 classes, and the most of the misclassified samples are those having closer bone age. For instance, for age-16 category, the student could predict 74 samples out of 105 correctly as age 16 class. Moreover, in classes age 12 and age 13, the proposed method outperformed than other age categories regarding TP. In addition, comparing these numbers with that of teacher’s confusion matrix, one can conclude that they are quite close suggesting comparable performance of student and its teacher.

Figure 3. The confusion matrices for age classes of (a) teacher model and (b) student-6 model.

Tables 4 and 5 represent the performance evaluation results of the student-6 and teacher, respectively, for each of 19 classes in terms of accuracy, precision, recall, specificity and MAE. According to Table 4, our student is able to classify each of bone age classes with high degree of certainty with the following results: accuracy of 94.68%, precision of 64.73%, recall of 64.94%, F-score of 64.50%, specificity of 97% and MAE of 0.022. By contrasting Tables 4 and 5, it can be recognised that how closely the performance of the proposed student model matches that of the teacher. For example, in class age 6, performance of student is lower than teacher’s performance as little as 1 $\pm$ 76% in accuracy, 1 $\pm$ 0.63 in precision, 0.09 $\pm$ 0.36 in recall and 0.0 $\pm$ 0.04 in specificity, whereas its MAE is 0.002 more than teacher. Hence, it can be inferred that our student can provide approximately the same performance as the teacher while it is much less complex than teacher model.

Table 4. Performance of the student-6 model for all 19 bone age classes.

Category	Accuracy	Precision	Recall	F-Score	Specificity	MAE
Age 0	0.96 $\pm$ 0.36	0.34 $\pm$ 0.63	0.32 $\pm$ 0.28	0.32 $\pm$ 0.97	0.98 $\pm$ 0.79	0.011
Age 1	0.96 $\pm$ 0.68	0.62 $\pm$ 0.89	0.53 $\pm$ 0.69	0.57 $\pm$ 0.15	0.98 $\pm$ 0.94	0.110
Age 2	0.96 $\pm$ 0.59	0.60 $\pm$ 0.08	0.54 $\pm$ 0.55	0.56 $\pm$ 0.84	0.98 $\pm$ 0.68	0.009
Age 3	0.95 $\pm$ 0.78	0.57 $\pm$ 0.07	0.44 $\pm$ 0.39	0.49 $\pm$ 0.66	0.98 $\pm$ 0.13	0.012
Age 4	0.96 $\pm$ 0.36	0.60 $\pm$ 0.90	0.61 $\pm$ 0.61	0.60 $\pm$ 0.50	0.98 $\pm$ 0.72	0.014
Age 5	0.95 $\pm$ 0.20	0.66 $\pm$ 0.53	0.71 $\pm$ 0.78	0.68 $\pm$ 0.41	0.97 $\pm$ 0.11	0.009
Age 6	0.97 $\pm$ 0.08	0.72 $\pm$ 0.10	0.80 $\pm$ 0.33	0.75 $\pm$ 0.79	0.98 $\pm$ 0.11	0.009
Age 7	0.95 $\pm$ 0.77	0.63 $\pm$ 0.82	0.65 $\pm$ 0.60	0.63 $\pm$ 0.98	0.97 $\pm$ 0.64	0.009
Age 8	0.96 $\pm$ 0.20	0.71 $\pm$ 0.33	0.74 $\pm$ 0.74	0.72 $\pm$ 0.47	0.97 $\pm$ 0.32	0.015
Age 9	0.95 $\pm$ 0.38	0.67 $\pm$ 0.29	0.64 $\pm$ 0.10	0.65 $\pm$ 0.47	0.97 $\pm$ 0.65	0.017
Age 10	0.93 $\pm$ 0.55	0.70 $\pm$ 0.74	0.71 $\pm$ 0.12	0.70 $\pm$ 0.50	0.96 $\pm$ 0.74	0.017
Age 11	0.93 $\pm$ 0.22	0.67 $\pm$ 0.01	0.72 $\pm$ 0.54	0.69 $\pm$ 0.41	0.96 $\pm$ 0.58	0.011
Age 12	0.92 $\pm$ 0.64	0.68 $\pm$ 0.04	0.70 $\pm$ 0.48	0.68 $\pm$ 0.99	0.95 $\pm$ 0.74	0.014
Age 13	0.92 $\pm$ 0.48	0.70 $\pm$ 0.66	0.65 $\pm$ 0.89	0.67 $\pm$ 0.41	0.96 $\pm$ 0.23	0.019
Age 14	0.93 $\pm$ 0.15	0.75 $\pm$ 0.60	0.66 $\pm$ 0.79	0.70 $\pm$ 0.21	0.97 $\pm$ 0.73	0.008
Age 15	0.93 $\pm$ 0.78	0.67 $\pm$ 0.52	0.60 $\pm$ 0.73	0.63 $\pm$ 0.31	0.97 $\pm$ 0.97	0.091
Age 16	0.95 $\pm$ 0.10	0.68 $\pm$ 0.72	0.69 $\pm$ 0.05	0.68 $\pm$ 0.50	0.97 $\pm$ 0.86	0.017
Age 17	0.95 $\pm$ 0.29	0.63 $\pm$ 0.70	0.76 $\pm$ 0.07	0.68 $\pm$ 0.89	0.96 $\pm$ 0.08	0.019
Age 18	0.96 $\pm$ 0.23	0.70 $\pm$ 0.78	0.87 $\pm$ 0.08	0.77 $\pm$ 0.58	0.97 $\pm$ 0.36	0.014

Table 5. Performance of the teacher model per each 19 bone age classes.

Category	Accuracy	Precision	Recall	F-Score	Specificity	MAE
Age 0	0.97 $\pm$ 0.67	0.52 $\pm$ 0.25	0.46 $\pm$ 0.17	0.48 $\pm$ 0.82	0.98 $\pm$ 0.10	0.009
Age 1	0.97 $\pm$ 0.69	0.77 $\pm$ 0.29	0.64 $\pm$ 0.42	0.69 $\pm$ 0.90	0.99 $\pm$ 0.61	0.010
Age 2	0.97 $\pm$ 0.06	0.72 $\pm$ 0.61	0.67 $\pm$ 0.09	0.69 $\pm$ 0.41	0.98 $\pm$ 0.77	0.008
Age 3	0.97 $\pm$ 0.25	0.70 $\pm$ 0.26	0.70 $\pm$ 0.59	0.70 $\pm$ 0.00	0.98 $\pm$ 0.42	0.010
Age 4	0.97 $\pm$ 0.22	0.75 $\pm$ 0.82	0.66 $\pm$ 0.47	0.70 $\pm$ 0.21	0.99 $\pm$ 0.09	0.010
Age 5	0.96 $\pm$ 0.66	0.73 $\pm$ 0.98	0.77 $\pm$ 0.69	0.74 $\pm$ 0.95	0.98 $\pm$ 0.26	0.007
Age 6	0.98 $\pm$ 0.84	0.82 $\pm$ 0.73	0.89 $\pm$ 0.69	0.85 $\pm$ 0.36	0.98 $\pm$ 0.15	0.007
Age 7	0.96 $\pm$ 0.34	0.73 $\pm$ 0.34	0.73 $\pm$ 0.63	0.73 $\pm$ 0.00	0.98 $\pm$ 0.28	0.006
Age 8	0.97 $\pm$ 0.78	0.82 $\pm$ 0.58	0.84 $\pm$ 0.03	0.82 $\pm$ 0.99	0.98 $\pm$ 0.44	0.012
Age 9	0.96 $\pm$ 0.67	0.73 $\pm$ 0.10	0.72 $\pm$ 0.06	0.72 $\pm$ 0.50	0.98 $\pm$ 0.52	0.014
Age 10	0.96 $\pm$ 0.00	0.77 $\pm$ 0.90	0.81 $\pm$ 0.31	0.78 $\pm$ 0.95	0.97 $\pm$ 0.45	0.013
Age 11	0.96 $\pm$ 0.60	0.76 $\pm$ 0.87	0.83 $\pm$ 0.53	0.79 $\pm$ 0.35	0.97 $\pm$ 0.87	0.009
Age 12	0.95 $\pm$ 0.38	0.81 $\pm$ 0.81	0.76 $\pm$ 0.65	0.78 $\pm$ 0.42	0.98 $\pm$ 0.51	0.009
Age 13	0.96 $\pm$ 0.91	0.83 $\pm$ 0.26	0.82 $\pm$ 0.40	0.82 $\pm$ 0.50	0.98 $\pm$ 0.94	0.004
Age 14	0.96 $\pm$ 0.00	0.84 $\pm$ 0.59	0.75 $\pm$ 0.81	0.79 $\pm$ 0.25	0.98 $\pm$ 0.63	0.008
Age 15	0.96 $\pm$ 0.46	0.78 $\pm$ 0.02	0.70 $\pm$ 0.71	0.73 $\pm$ 0.78	0.98 $\pm$ 0.95	0.091
Age 16	0.97 $\pm$ 0.42	0.81 $\pm$ 0.42	0.83 $\pm$ 0.96	0.81 $\pm$ 0.99	0.98 $\pm$ 0.24	0.011
Age 17	0.97 $\pm$ 0.46	0.77 $\pm$ 0.31	0.87 $\pm$ 0.53	0.81 $\pm$ 0.70	0.98 $\pm$ 0.67	0.009
Age 18	0.97 $\pm$ 0.77	0.77 $\pm$ 0.16	0.93 $\pm$ 0.32	0.84 $\pm$ 0.25	0.98 $\pm$ 0.28	0.003

4.3. Ablation study

In this section, in order to investigate effectiveness of our suggested hybrid KD method, some experiments are carried out to compare the proposed KD with three pure KD models based on accuracy, precision, recall and specificity metrics. To this end, the following three KD models are taken into account for comparison with student-6:

1. OKD (Hinton et al. 2015): In this study, which the concept of KD was introduced for the first time, a student is constructed by utilising the response-based knowledge provided from a teacher network.

2. FitNet (Romero et al. 2014): This method was the first attempt to use the intermediate information for training the student model. Indeed, feature maps of internal layers of the teacher is passed on to the student.

3. TAKD (Mirzadeh et al. 2020): In this approach, a number of teachers supervise training the student by transferring a combined knowledge comprising the information of the last output and the inner layers of each teacher. It is noted that assignment of each teacher to a group of layers of students is random.

The performance of these three methods and our student model is shown in Table 6. According to the table, student-6 yielded accuracy of 95.16%, precision of 65.16%, recall of 65.34%, F-score of 65.24% and specificity of 87.36% which is a significant improvements compared to the three mentioned methods. Actually, our student model enhances the performance of the Mirzadeh et al. (2020) model which is the best among the three studies, with 5.63%, 3.17% and 4.45% improvements in accuracy, precision and recall respectively. Altogether, student-6 which is constructed based on wrist bone outperformed the other previously introduced KD methods.

Table 6. Comparing analysis of student-6 results with the student model from three other KD methods.

Methods	(%)	Precision (%)	Recall (%)	F-Score (%)	Specificity (%)
O-KD Hinton et al. (2015)	74.67 $\pm$ 0.78	59.56 $\pm$ 0.90	58.67 $\pm$ 0.47	59.11 $\pm$ 0.17	78.56 $\pm$ 0.17
FitNet Romero et al. (2014)	78.35 $\pm$ 0.99	58.69 $\pm$ 0.85	57.54 $\pm$ 0.08	58.10 $\pm$ 0.93	77.03 $\pm$ 0.99
T-KD Mirzadeh et al. (2020)	89.53 $\pm$ 0.08	61.99 $\pm$ 0.70	60.89 $\pm$ 0.43	61.43 $\pm$ 0.51	89.86 $\pm$ 0.84
Proposed KD	95.16 $\pm$ 0.54	65.16 $\pm$ 0.01	65.34 $\pm$ 0.69	65.24 $\pm$ 0.99	87.36 $\pm$ 0.48

To show that results of Table 7 are reliable and superiority of our model over those three KD methods has not been obtained by chance, a t-test is used in which a null hypothesis would be rejected or not. A t-test is an inferential statistic used to determine if there is a significant difference between the means of two groups and how they are related. For this purpose, we execute t-test for all measurement accuracy, precision, recall, specificity and F-score between our proposed method and other three KD ones, and outcomes is depicted in Table 7. Here, null hypothesis is that there is no difference between our KD and each of the three methods. This hypothesis is rejected either $T_{\mathit{obs}}<T_{\mathit{critical}}$ or $\mathit{p}\mathit{value}<0.05$ with 95% confidence level. Results of Table 7, for all of the criteria satisfies these two conditions which implies the null hypothesis is rejected. This means that there is a significant difference between our method and three KD approaches, and the proposed method outperforms rather than three other studies.

Table 7. Results of comparison the paired t-test of the proposed method and three other KD approaches.

Criteria	Methods	${\mathit{T}}_{\mathit{obs}}$	${\mathit{T}}_{\mathit{critical}}$	$\mathit{p}\mathbf{\_}\mathit{Value}$
Precision	O-KD Hinton et al. (2015)	−4.5685	4.5844	0.0065
	FitNet Romero et al. (2014)	−4.8540	1.3737	0.0173
	T-KD Mirzadeh et al. (2020)	−2.6867	1.5854	0.0455
Sensitivity	O-KD Hinton et al. (2015)	−4.4857	3.8885	0.0068
	FitNet Romero et al. (2014)	−5.4757	2.3722	0.0185
	T-KD Mirzadeh et al. (2020)	−0.4746	1.0004	0.0365
F-Score	O-KD Hinton et al. (2015)	−6.4740	3.5833	0.0055
	FitNet Romero et al. (2014)	−2.3731	1.0830	0.0056
	T-KD Mirzadeh et al. (2020)	−2.4742	1.5730	0.0301
Specificity	O-KD Hinton et al. (2015)	−6.4573	2.6664	0.0036
	FitNet Romero et al. (2014)	−3.6733	0.8522	0.0687
	T-KD Mirzadeh et al. (2020)	−0.8883	1.7842	0.0343
Accuracy	O-KD Hinton et al. (2015)	−8.5329	3.5855	0.0074
	FitNet Romero et al. (2014)	−6.4742	2.9433	0.0890
	T-KD Mirzadeh et al. (2020)	−1.0459	1.6750	0.0315

4.4. Comparison of proposed method with other BAA studies

Performance of our proposed BAA method is compared to some of other BAA techniques in favour of examining effectiveness of utilising KD paradigm in BAA schemes. For this purpose, Table 8, demonstrates performance of six BAA techniques and our introduced one in terms of accuracy and MAE. These six methods have exploited different approaches from classic machine learning algorithms to convolutional deep neural networks, and all of them has been executed on DHA dataset while classifying 0–18 age classes. The results clearly show that our BAA method outperforms all the other methods by a large margin such that our strategy can classify all the 19 bone age classes with accuracy of 95.16% and MAE of 0.022 which is much higher than other approaches. For example, in Kashif et al. (2016), the SVM classifier could classify bone age with MAE of 0.605, whereas our BAA method providing MAE of 0.22 which is 0.583 higher than this study. Despite exploiting CNN, which is a powerful classifier (An Ding et al. 2020), can classify bone age categories with MAE of 0.590 that is 0.568 less than our proposed one. Moreover, in Bui et al. (2019), although deep neural network has been employed in combination with SVR, it could not categorise bone age classes with accuracy more than 67%. Instead, our introduced BAA approach offers an accuracy of nearly 95% with a lightweight structure.

Table 8. Performance evaluation of the student-6 and some state-of-the-art studies.

Reference	Method	No. of samples	Age	Accuracy (%)	MAE
Kashif et al. (2016)	SVM	1100-DHA	0–18	–	0.605
Gertych et al. (2007)	Fuzzy Classifiers	1400-DHA	0–18	79	–
Mansourvar et al. (2012)	Histogram technique	1100-DHA	0–18	–	0.170
Spampinato et al. (2017)	CNN	1391-DHA	0–18	79	–
An Ding et al. (2020)	CNN	1400-DHA	0–18	–	0.590
Bui et al. (2019)	CNN + SVR	1400-DHA	0–18	67	–
Wang et al. (2023)	Interpretable CNN	1400-DHA	0–18	82.5	0341
Liu et al. (2022)	Ensemble CNNs	1400-DHA	0–18	83.8	–
Deshmukh and Khaparde (2022)	Faster R-CNN	1400-DHA	0–18	89.1	0.24
Proposed Method	Knowledge Distillation	1400-DHA	0–18	95.16	0.022

5. Conclusion

In this paper, a new BAA technique was introduced by employing a hybrid KD paradigm maintaining high accuracy with a lightweight structure. In this method, a teacher network with an advanced CNN structure supervises the learning procedure of a student model made of simple CNN architecture which leads the student to imitate behaviour of the teacher model. Student model is trained using transmitted knowledge of the teacher which contains data on the feature maps of intermediary layers of teacher as well as its last output layer. Exploiting these helpful insight, the student model can mimic the manner of the teacher with high resemblance, making the student more accurate rather teacher while keeping the architecture far less complex. In the experimental results, performance of the student in classifying the bone age 0 to age 18 was analysed through a variety of examinations. According to the results, student model trained on the wrist ROI was able to attain a higher level of accuracy (accuracy of 95%) than the other five students. Moreover, other comparison tables indicate superiority of our proposed method over other cutting-edge techniques.

The diversity in skeleton shape, texture and morphology among individuals poses a challenge when interpreting bone X-ray images. Consequently, the diversity in the statistical learning models that were developed using these images becomes evident. Diversity refers to modified behaviour of statistical learning models under different circumstances. Here, model generalisation (generalisability) is diminished in dealing with unseen data. Although the proposed KD model performs well, one of its drawbacks is that it fails to examine the diversity problem. Applying the collective intelligence approach to learning models is one way to address the diversity problem. Using an ensemble of multi-teacher models instead of single-teacher models, the proposed KD model reduces behavioural diversity and converges to uniform behaviour. Thus, a future work can be presenting a multi-teacher version of the proposed KD model forBAA from X-ray images, where the diversity problem and generalisation preservation are taken into account.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Majid Sepahvand

Majid Sepahvand Graduated from Razi University with an PhD in computer engineering. His doctoral dissertation entitled decomposed knowledge distillation from teacher to student for efficient biomedical data processing was supervised by Dr. Fardin Abdali-Mohammadi from Razi University and Amir Taherkordi from the University of Oslo. His research areas of interest include signal & image processing, deep learning, evolutionary computation, and edge intelligence.

Fardin Abdali-Mohammadi

Fardin Abdali-Mohammadi was born in KhorramAbad, Lorestan, Iran, in 1979. He received the B.Sc. degree in computer engineering from the Sharif University of Technology in 2001, the M.Sc. degree in software engineering from the Amirkabir University of Technology in 2003, and the Ph.D. degree from Isfahan University in 2012. He is currently an Associate Professor with the Department of Computer Engineering and Information Technology, Razi University, Kermanshah, Iran. His research interests include Medical Informatics, Biomedical signal and image processing, Data mining and Bioinfo.

Maytham N. Meqdad

Maytham N. Meqdad was born in Babylon, in 1987. He received Ph.D. degree with the Department of Computer Engineering and Information Technology, Razi University, Kermanshah, Iran in 2023. Currently he is working as assistant lecturer in Al-Mustaqbal University, Babylon, Iraq since 2013. His research interests include deep learning, evolutionary algorithms, and biomedical.

Notes

1. http://www.ipilab.org/BAAweb/