Ultrasound-based 3D bone modelling in computer assisted orthopedic surgery – a review and future challenges

Abstract

Computer-assisted orthopedic surgery requires precise representations of bone surfaces. To date, computed tomography constitutes the gold standard, but comes with a number of limitations, including costs, radiation and availability. Ultrasound has potential to become an alternative to computed tomography, yet suffers from low image quality and limited field-of-view. These shortcomings may be addressed by a fully automatic segmentation and model-based completion of 3D bone surfaces from ultrasound images. This survey summarizes the state-of-the-art in this field by introducing employed algorithms, and determining challenges and trends. For segmentation, a clear trend toward machine learning-based algorithms can be observed. For 3D bone model completion however, none of the published methods involve machine learning. Furthermore, data sets and metrics are identified as weak spots in current research, preventing development and evaluation of models that generalize well.

Keywords:

• Ultrasound
• review
• bone model completion
• segmentation
• 3D

1. Introduction

Computer assisted orthopedic surgery (CAOS) provides a number of benefits to the patient, surgeon and clinic. By CAOS, the accuracy and repeatability in placement of e.g. screws, osteotomies or implants can be improved [1–3]. The automated data processing enables personalized solutions like patient-specific instruments and implants [4], where the 3D geometry is required to design rigidly fitting cutting guides and to assess the patient’s J-curve, respectively, among other aspects. These methods potentially improve mechanical alignment [5] or other functional parameters [6,7] and decrease net-costs in total knee arthroplasty (TKA) [8,9], respectively. E.g. implant templating in 3D as opposed to 2D increases tibial size selection accuracy from 28.7% to 93.1% [10]. Finally, personalized biomechanical simulation and optimization based on 3D modeling may correlate with patient-related outcomes [11–14], and thus help to improve clinical outcomes.

Therefore, for many applications in CAOS, a patient-specific 3D representation of bony anatomy is fundamental. The accuracy of this representation is of high importance: mismatch between the bone and implant geometries is associated with poor clinical outcome. Chung et al. state that femoral component overhang of over 4 mm correlates with lower maximum flexion angle (p = 0.005) [15]. Mahoney et al. report frequent overhang exceeding 3 mm in 40% of men and 68% of women and found an almost twofold increased risk of knee pain [16]. Conventionally, computed tomography (CT) and magnetic resonance imaging (MRI) are the gold standard for the acquisition of 3D bony geometry. These techniques provide a comprehensive 3D image of the region of interest at a high resolution, with in-plane voxel spacing of current systems in the range of 0.3–0.5 mm [17], and is influenced by external factors, like the tissue properties. While CT allows for comparably easy segmentation of bony structures, MRI enables visualization of the surrounding soft tissues. However, both systems come with high costs. Yet another alternative is intra-operative 2D or 3D fluoroscopy. While it is rather cheap compared to CT or MRI, it exposes the patient and possibly the surgeon to ionizing radiation.

In contrast to computed tomography and fluoroscopy, ultrasound is a radiation-free imaging technique. Based on the reflection of ultrasound waves at impedance interfaces, it is able to visualize various tissues and structures, for example muscles, blood vessels, cartilage and bones. Its real-time capability allows for an interactive morpho-functional examination including e.g. doppler sonography or elastography. It is widely available and represents a cheap alternative [18]. The physical limit of spatial resolution depends on the imaging frequency and ranges from 0.1 mm to 1 mm for frequencies of 20 to 1 MHz. However, ultrasound comes with a number of limitations. The most apparent one is the so-called speckle noise: The interference of numerous small diffuse reflections creates the characteristic noise commonly found in ultrasound images. Accordingly, the signal-to-noise ratio (SNR) is quite low. Furthermore, bone interfaces cause a strong specular reflection of the ultrasound wave, with almost no echo of structures in the ‘bone shadow’. As such, visualization of structures behind a bone surface is not possible in general. For a number of bony anatomies this is highly relevant, as vital parts, e.g. retropatellar surfaces of the knee joint, cannot be imaged by ultrasound. The specular reflection furthermore induces a high dependence of the signal strength on the inclination angle between bone surface and ultrasound beam: In an orthogonal setting, the sound energy is reflected in direction of the probe, producing a strong response in the image. As the angle becomes more acute, less sound energy is reflected to and measured by the probe, causing a small or even no response of the surface in the image. In a fully orthogonal setting on the other hand, echo artifacts may occur. Imaging parameters need to be set manually, even if some progress on automation has been made by ultrasound vendors (Clarius, Vancouver, Canada). Similarly, the data acquisition itself is a highly manual process. Attempts to automate it with robots are still in its infancy [19]. All of the above leads to a steep learning curve for sonographers, who require years of training and practice to be able to distinguish relevant features from artifacts [20]. Most importantly, these limitations induce the need for a modeling of occluded parts of the bone. This holds especially true for narrow joint gaps, e.g. in the wrist, knee or hip. Two examples of relevant medical applications in this regard are TKA, where the distal cut may be defined by an offset to the distal articulating surface and total hip arthroplasty (THA), where the cup needs to be positioned in the joint socket such that it is covered by bone.

The ultrasound signal recorded by the probe is referred to as radio frequency (RF) data. Commonly, the signal envelope of the RF data is computed and a grey-scale Brightness-Mode (B-Mode) image is constructed from it. All papers reviewed in this work base their computation on B-Mode images, even though the original data may be used as well [21]. Ultrasound signals can be measured for a single 1D beam, a 2D image or a 3D volume image. The latter can be generated by combining multiple 2D image slices into one volume. This requires information on the pose of the probe, which can be provided by a mechanically swiped 1D piezo-element array, by a phased 2D piezo-element array, or by tracking a 2D probe with optical or electromagnetic tracking systems. Given the limited field-of-view of common ultrasound probes, this technique becomes highly relevant to depict large structures in a single volume. Note, that the image interpolation process is often referred to as reconstruction, which is not a topic of this survey. Similarly, (multi-)modal registration of (volumetric) ultrasound images to itself, other imaging modalities or atlases is not discussed Instead, we refer the reader to [22] and [23] for reviews of these topics.

Computer-assistance may be capable of handling the aforementioned limitations and allow the wide-spread use of ultrasound for 3D bone modeling. In general, two tasks need to be accomplished: First, the delineation of the bone surface in the ultrasound image, typically done via segmentation, and second, the bone model completion of the partial surface. A recent review on the first topic is given by Pandey et al. [24], presenting 56 works published over a 29-year period. This work strives to continue their work by adding another 44 publications published in the past three years to the list, underlining the significance of the topic. We furthermore extent their work by providing a method tutorial and investigating bone model completion. The review most closely related to this topic by Heimann et al. mentions just a single publication involving ultrasound imaging [25].

This leads to the main objectives and contributions of this work: (1) Review of the state-of-the-art in segmentation of bone surfaces in ultrasound images. (2) Review of the state-of-the-art in completion of partial bone models derived from ultrasound. From these contributions, the reader should get a comprehensive overview and detailed understanding of the methodology used. (3) Identification and discussion of past limitations in order to (4) make recommendations for future research. This work primarily addresses researchers in the fields of total knee and hip arthroplasty (TKA/THA) and biomechanics, that require full 3D models as input for their methods.

2. Methods

We followed the ‘Preferred Reporting Items for Systematic Reviews and Meta-Analysis’ (PRISMA) paradigm when applicable. As the review at hand is a tutorial review, this excluded data collection, bias assessment or syntheses. The review was not registered beforehand. Our research protocol followed a multi-stage approach: First, we researched related review papers with similar scope. This allowed us to identify the need for a more recent survey publication and to hint the reader to high quality reviews of prior work. The keywords required in the title or abstract were either a combination of 3D, ultrasound and survey, or ultrasound, survey and segmentation. For each word, alternative notations and synonyms were included (see Table 1 for a complete list). Two literature databases, Google Scholar and PubMed were used. Examining 520 survey papers, this research yielded five relevant surveys on segmentation of ultrasound images, that provide a comprehensive literature review [26–29]. We want to highlight the work by Pandey et al. that provides a high quality overview up to the year 2019 [24]. Three relevant surveys on shape completion of partial 3D ultrasound models were found, providing an overview of the state of the art up to 2009 [25,30,31]. The most relevant survey on that topic is the work by Heimann et al. [25]. This concludes the first stage, where other survey papers were reviewed. All papers included in these survey papers were excluded from our search.

Table 1. List of alternative keywords used in the keyword-based literature research.

3D	Ultrasound	Survey	Segmentation	Orthopedic	Model	Completion
3-d	Ultrasonic	Review	Localization	Bone	Models	Inpainting
volumetric	sonography		Surface	Femur	Modeling	Interpolation
Volume	medical		Mesh	Tibia	Shape	Reconstruction
3-dimensional				Pelvis	Shapes	Extrapolation
Three-dimensional				Spine	Point cloud
				Knee
				Hip Wrist Ankle

In the second stage, we limited our research to more recent work published after 2019 and 2009, respectively, and searched for original articles using a similar pattern: For segmentation related papers, the keywords were a combination of 3D, ultrasound, segmentation and orthopedics. For bone model completion, the keywords were either model, orthopedics and completion or model, 3D and ultrasound. The papers title and abstract were scanned to decide upon selection. The papers included had to present methods not published previously. As mentioned beforehand, methods on registration and volume-reconstruction were excluded. Furthermore, manual or semi-manual techniques were excluded, as well as publications that were not accessible. Based on this method, we added 39 relevant out of 2000 examined papers to our list.

After this keyword-based analysis, we checked all authors that published more than one paper in the list so far. Their research groups were determined and all publications of their respective head of research were checked. Based on this, we added another 8 papers to our list.

Finally, we searched through all citations of the publications found so far and selected related works, adding another 8 publications. During revision, three more paper were pointed out by the reviewer, such that this review included 58 papers in total.

3. Methodological overview

As mentioned before, most of the publications addressed one of the two sub-tasks, segmentation and bone model completion. Therefore, we analyzed the respective problems separately. In the following subchapters, we present the most popular and successful methods.

3.1. Segmentation

3.1.1. Shadow peak

One of the most challenging tasks in the interpretation of ultrasound images is the distinction between bone surfaces from other soft tissue interfaces. A helpful feature for identifying bone surfaces is the acoustic shadow occurring behind interfaces with high difference in impedance, the so-called bone shadow: The largest part of incoming sound energy is reflected, only a very small part is transmitted. Mostly, bone shadow information is used as an additional feature for the segmentation of bone surfaces. One example is the Shadow Peak algorithm [32], where a shadow confidence map S is computed based on the integration of normalized and Gaussian-filtered intensities along ultrasound scan lines: $$S\left(x,y,z\right)=1-\sqrt{\frac{{\sum }_{i=1}^{i=y}{I}_{x,i,z}}{{\sum }_{j=1}^{j=Y}{I}_{x,j,z}}}$$

Here, $I_{x,y,z}$ denotes the image intensity $y$ the vertical image axis. This shadow confidence map (see Figure 1) can then be used for the enhancement of the original ultrasound image by element-wise multiplication.

Figure 1. Original ultrasound image (left), the shadow confidence map (Middle) computed by the shadow Peak algorithm, and its multiplication, the masked image (right). Note, that while soft tissue interfaces are damped, the bone surface remains barely discernable.

3.1.2. Phase symmetry

Intensity-based segmentation techniques suffer from the unnormalized nature of ultrasound images. While these techniques may work well for CT images, the actual intensity observed in ultrasound is influenced by a number of variables, e.g. the wave amplitude, attenuation of the surrounding tissue or the inclination angle of the reflecting interface to the US beam. However, one may use another feature inherent to bone responses: their symmetry. The ultrasound pulse generated by the ultrasound probe has a certain spatial extent. When reflecting off a surface, its waveform may change, e.g. the pulse length may increase due to a reflection off an inclined surface, but its symmetrical nature will remain [33].

In an image, this feature can be searched for by transforming it into the frequency domain and conducting a wavelet analysis. Suitable wavelets include log-Gabor filters, that exhibit a frequency and a directional component. By varying the frequency component, symmetric features of different sizes can be determined. Similarly, filters for different directions can be combined. In ultrasound segmentation, a filter bank of $m$ filters is built that exhibits rotation-invariance. The even $e_m(x)$ and odd $o_m\left(x\right)$ part of the filter response relate to the symmetry and asymmetry of the image signal. Constants $T$ and $\epsilon$ account for noise and prevent division by zero. Along with the inherent intensity-invariance of the individual filters, bone responses in the image can be detected reliably. This is computed by the so-called Phase Symmetry (PS), shown in Figure 2 [34]. $$\mathit{\text{PS}}\left(x\right)=\frac{{\sum }_m{|e}_m\left(x\right)|-|o_m\left(x\right)|-T}{{\sum }_m\sqrt{{{(e}_m\left(x\right))}^2+{{(o}_m\left(x\right))}^2}+\epsilon }$$

Figure 2. An example of an ultrasound image and its respective phase symmetry (PS) image. Note, that the bone surface on the left has a rather asymmetric appearance due to the hyperechoic soft tissue on top. This is why the bone surface is barely recognized.

Soft-tissue and other interface responses also exhibit a symmetric structure and are thus enhanced alongside the bone surfaces. However, bone surfaces are expected to have a sheet-like shape, which may differ from other interfaces like blood vessels. By conducting an eigen analysis of the PS image, one can assess the local eigenvalues. For a sheet like structure, the eigenvalues ${\lambda }_i$ are ${\lambda }_1\approx {\lambda }_2$ and ${\lambda }_3\approx 0.$ These requirements can be incorporated into the PS-formula as following, to obtain the Structured Phase Symmetry (SPS) [35]: $$\mathit{\text{SPS}}\left(x\right)=\{\begin{array}{l}0,{\lambda }_3>0\\ e^{{-R}_{\mathit{\text{sheet}}}^2}(1-e^{{-R}_{\mathit{\text{blob}}}^2})(1-e^{{-R}_{\mathit{\text{noise}}}^2}),o\mathit{\text{therwise}}\end{array}$$ where $R_{\mathit{\text{blob}}}=(|2\left|{\lambda }_3\right|-\left|{\mathrm{\lambda }}_2\right|-\left|{\lambda }_1\right||)/{\lambda }_3$ describes a blob-eliminating term, $R_{\mathit{\text{sheet}}}=\left|{\lambda }_2/{\lambda }_3\right|$ a sheet-enhancing term and $R_{\mathit{\text{noise}}}=\sqrt{{\lambda }_1^2+{\lambda }_2^2+{\lambda }_3^2}$ a noise-canceling term.

Another feature inherent to bone surfaces is the aforementioned bone shadow. It can be incorporated in the SPS by computing an attenuation map, similar to the shadow peak feature. The confidence-weighted structed phase symmetry (CSPS) map is defined as the product of the SPS with the shadow feature map. By thresholding the CSPS map, one can segment the bone surface [35]. Note that the terms as well as the exact formulas used by the authors publishing about these methods slightly changed over the years. $$\mathit{\text{CSPS}}\left(x\right)=p_{\mathit{\text{shadow}}}\left(x\right)\hspace{0.17em}\mathrm{*}\hspace{0.17em}\mathit{\text{SPS}}\left(x\right)$$

While phase symmetry and its derivatives were among the most popular segmentation techniques prior to the era of convolutional neural networks, only two recent publication [36,37] fully rely on this technique for segmentation of bones. Another publication [38] however utilizes symmetry filters for detecting not just the bones surface, but the full spinal column profile. In [39], phase symmetry is compared to a trimmed version utilizing just the bone shadow and scan-line derivative in order to speed up the computation.

3.1.3. Convolutional neural networks

The aforementioned methods rely on handcrafted features and heuristics. In contrast, the following methods belong to the domain of deep learning.

With the success of the AlexNet [40], convolutional neural networks (CNNs) became the gold standard for most vision tasks, including segmentation. See Figure 3 for an exemplary ultrasound image segmentation. In the medical domain, the U-Net [41] establishes a gold standard architecture. However, various improvements have been adopted.

Figure 3. An example of an ultrasound image and its respective segmentation by a DeepLabv3 CNN architecture.

Most of the works analyzed in this paper rely on the original U-Net [42–53], or its 3D counterpart [54–57], or the succeeding U-Net++ [58], which densely populates the skip connections with convolutional blocks. Three publications incorporate dropout, typically used during training in order to improve the networks robustness, into the inference scheme [54,59,60]. Random neurons, or filters, are set to zero (‘dropped’) in order to introduce non-determinism into the inference process. This allows to measure the uncertainty of the segmentation. If a pixel is reliably segmented over a number of runs, even though different features were dropped, the network has a high certainty regarding its segmentation. The computational burden increases linearly with the number of inferences run. This technique can also be used at run-time to identify voxels with uncertain segmentation and to focus the weight adaption on these [55]. Auxiliary phase-symmetry images can either be predicted by a second branch [61] or incorporated as additional feature [62–64]. Similarly, Tang et al. propose to combine two U-Nets processing B-mode and shadow enhanced B-mode images simultaneously, fuzing their respective outputs to obtain the final segmentation [56]. Several contributions do not rely on the U-Net, but on the DeepLabv3 + [65–68], Mask R-CNN [69], a two-stage detector, and the Pyramid Attention Network (PAN) [70]. The former utilizes strided convolutions to retain high-resolution feature maps and combine multi-scale features. The latter adds an attention mechanism to the skip connections. When combining the low-level features of the skip connection into the decoder, the high-semantic features are utilized to focus the attention to relevant parts. Additionally, a multi-scale pyramid pooling is added to the highest semantic layer. Jiang et al. follow this concept and stack multi-scale features to guide attention in the final classification step [71]. Banerjee et al. introduce a number of changes to the U-Net, that were inspired by other architectures [72]. Their ‘Light-convolution Dense Selection’ architecture (LDS-U-Net) employs depthwise-separable convolution for a reduced number of parameters, selection gates and residual connections. Similarly, the ‘BoneNet’ augments the original U-Net with ‘squeeze-and-excite’ blocks, that introduce an attention mechanism, too, and depthwise-separable convolution [73]. For the segmentation of volumetric images, spatial information of the neighboring slices may be incorporated. Next to 3D convolutions, this can be achieved by combining two encoders in a Siam-U-Net [74], where the known region-of-interest of the previous slice is processed by a siamese encoder and combined with the encoding of the current frame via cross-correlation. This concept can also be formulated as a recurrent neural network on the basis of a gated recurrent unit as proposed by Jiang et al. [63].

3.1.4. Generative adversarial networks

Generative adversarial networks (GANs) [75] consist of two competing subnetworks: a generator network which learns to create synthetic samples from random distributions, and a discriminator network which learns to differentiate between synthetic and real samples. Both subnetworks are trained simultaneously in a min-max fashion. One extension are conditional generative adversarial networks (cGANs) [76], where the generator is conditioned on input samples and learns to create corresponding output samples; thus, cGANs can be used for image-to-image translation [77].

In [78], the authors compare the pix2pix cGAN with the U-Net for the semantic segmentation of radial metaphysis, radial epiphysis and carpal bones in ultrasound images. Furthermore, they investigate a combination of U-Net and pix2pix, with the cGAN trained to improve UNet predictions.

Apart from direct segmentation, cGANs can be used to generate additional input features. In [79], the authors condition a cGAN on ultrasound B-mode images to generate corresponding bone shadow segmentation maps. These are subsequently fed to a multi-feature guided CNN for the segmentation of femur, tibia, radius, knee, and spine in ultrasound images.

Another use of GANs is the synthesis of ultrasound images and corresponding segmentation masks for data augmentation, as has been shown in [51] for ultrasound images of radius, femur, spine and tibia. Similarly, in [53] the authors use the pix2pix cGAN for the synthesis of ultrasound images of the humerus.

3.1.5. Orientation-guided graph convolutional networks

A feature specific to bone surfaces in ultrasound images is their connectivity. Typically, there are no sudden jumps or isolated patches. Rahman et al. leverage this knowledge by building a U-Net like network, but introducing graph convolution [80]. The input feature map of the image is mapped to a subspace on which a learned similarity matrix is applied. This process imposes a learnable graph structure on the image, promoting continuous segmentation. Furthermore, the network predicts the surface’s orientation, adding another clue helping with continuous segmentation. This is of course limited to structures exhibiting a single closed surface, which is not the case for, e.g. spine images or cross-sections of radius and ulna.

3.1.6. Vision transformer

Large Transformer models are the basis of the recent advances in natural language processing. By computing an attention measure between all pairs of words in a sentence, their relative relevance is determined. This concept can be transferred to images by dividing the image into patches. In contrast to the convolution operation, which operates locally, global image context can be accessed in a single self-attention layer. In order to incorporate features that extend over multiple patches, shifted and hierarchically scaled patches may be established. On the hand, Transformer models are known to require large training data sets, but on the other hand, they are able to benefit strongly from additional data. Broessner et al. confirm this when comparing a shifted-window vision transformer to the U-Net [81].

3.2. Bone model completion

All publications on ultrasound-based bone modeling reviewed in this work rely on statistical shape models (SSM) or similar approaches, typically built from CT image data bases. For a detailed description of the basic concept, we refer the reader to Heimann et al. [27]. In general, the SSM has two components, the mean shape $\overline{x}$ and the modes matrix $P.$ The model can adapt to any shape used in the training data by adding a linear combination of the modes with weights $b$ to the mean shape: $$x=\overline{x}+\mathit{\text{Pb}}$$ apart from adapting the shape, all methods involve some kind of pose registration, either simultaneously or alternating to the shape adaption. As the model can only adapt its shape in the statistical limits of the data base used for its construction, some methods involve a final free-form deformation scheme.

3.2.1. Correspondence-based

In order to find the weights that adapt the model to the incomplete patient geometry, a mismatch function needs to be defined. This can be achieved by computing the squared distances between corresponding points. As the mapping from model to patient geometry is unknown, most publications define the nearest neighbors as the corresponding points [82–84]. To adopt an image-based correspondence search, one can search along the vertex normal for high intensity pixels in either the original ultrasound image or the segmented one [70,85,86]. For optimization of the mismatch function, the system of linear equations needs to be solved, e.g. by a QR decomposition.

3.2.2. Gaussian-mixture-model fitting

In a different approach, a Gaussian-Mixture-Model (GMM) is built from the model point cloud, where the vertices define the centroids of the GMM. Then the model is fitted to the incomplete patient geometry by assuming it was sampled from the GMM. This is similar to the Coherent Point Drift algorithm [87], and eliminates the need to define correspondences. The target points may also be weighted according to their strength in the segmentation process. Model fitting may involve an expectation-maximization scheme [88], a quasi-newton optimization [89,90] or covariance matrix adaption evolution strategy [91].

3.2.3. Ultrasound simulation and correlation

Another measure that can be optimized is the correlation of a simulated ultrasound image with the recorded ultrasound image. For this, the current instance of the SSM is sliced according to the current transformation, and an ultrasound image is simulated. During simulation, rays are casted in the direction of the simulated ultrasound wave, to find intersection points with the model, that cast a bone shadow. By this process, one can eliminate the need for a segmentation, but needs to replace it with a simulation. Ghanavati et al. simultaneously optimize shape and pose by minimizing Linear Correlation by Linear Combination ${\mathit{\text{LC}}}^2.$ Their ultrasound image simulation includes a gray gradient image as background [92,93]. Khallaghi et al. in contrast alternate between optimizing shape and pose to optimize ${\mathit{\text{LC}}}^2.$ This reduces the number of parameters optimized simultaneously and speeds up the computation. By moving this highly parallelizable process to the GPU, computation time is again reduced. Furthermore, instead of point clouds or meshes they use a data base of CT images to build the SSM [94].

4. Results

Tables 2 and 3 give an overview over all publications regarding segmentation and 3D bone modeling, respectively. The remainder of this paper is structured as followed. We aggregate the publications according to different aspect to provide insight over the state of the art. Following, we report quantitative evaluations and discuss their limitations. Finally, we suggest directions for future research.

Table 2. Overview of the publications addressing segmentation.

Publication	Year	Application	Eval	Annot	Dim	Algorithm	Data Set Sizes
							Train	Val	Test
Alsinan [64]	2019	CAOS	In-vivo	Surface	2D	PS, U-Net	415		131
Asaduz [43]	2019	Tumor	In-vivo	Surface	2D	U-Net	6000	2000	2000
Duong [44]	2019	Periodontal	Ex-vivo	Surface	2D	U-Net	35		15
El-Hariri [45]	2019	DDH	In-vivo	Surface	2D	SP, PS, U-Net	439		103
Desai [37]	2019	Cartilage Monitoring	In-vivo	Other	2D	PS, Other			200
Jiang [95]	2019	Scoliosis	In-vivo	Other	VPI	PS, Other			111
Kompella [69]	2019	ROS: Arthroscopy	In-vivo	Other	2D	Other CNN	435		77
Alsinan [79]	2020	CAOS	In-vivo	Shadow	2D	GAN	1000		235
Antico [42]	2020	ROS: Arthroscopy	In-vivo	Other	2D	U-Net	15531		3124
Antico [59]	2020	ROS: Arthroscopy	In-vivo	Other	2D	U-Net, Dropout	14000		3000
Banerjee [47]	2020	Scoliosis	In-vivo	Other	VPI	U-Net	30		30
Dunnhofer [74]	2020	ROS: Arthroscopy	In-vivo	Other	2D	Other CNN	14622		2656
Hohlmann [67]	2020	TKA	In-vivo	Surface	2D	U-Net, Other CNN	3089		618
Huang [48]	2020	Scoliosis	In-vivo	Other	VPI	U-Net	80		29
Kannan [54]	2020	DDH	In-vivo	Surface	3D	U-Net, Dropout	133
Ungi [46]	2020	Scoliosis	In-vivo	Surface	2D	U-Net	3290		1892
Wang [61]	2020	CAOS	In-vivo	Surface	2D	PS, U-Net	834		383
Zhou [38]	2020	Scoliosis	In-vivo	Other	VPI	SP, Other			99
Luan [73]	2020	CAOS	In-vivo	Surface	2D	U-Net, Other CNN	511	128	47
Nguyen [50]	2020	Periodontal	In-vivo	Surface	2D	U-Net	700	200	200
Zaman [51]	2020	Tumor	In-vivo	Surface	2D	U-Net, GAN	3104	560	870
Alsinan [49]	2020	CAOS	In-vivo	Surface	2D	U-Net, GAN	700		535
Banerjee [72]	2021	Scoliosis	In-vivo	Other	VPI	Other CNN	79		30
Hohlmann [66]	2021	Fracture	In-vivo	Surface	2D	U-Net, Other CNN	1620	405	405
Kannan [55]	2021	DDH	In-vivo	Surface	3D	U-Net, Dropout	73*	18*	24*
Quader [36]	2021	DDH	In-vivo	Surface	3D	SP, PS	13*		12*
Tang [56]	2021	CAOS	In-vivo	Surface	3D	SP, U-Net	9*	1*	4*
Tu [39]	2021	CAOS	In-vivo	Surface	2D	SP			1393
Cengizler [96]	2021	Spina Bifida	In-vivo	Surface	2D	Other			10*
Oghli [64]	2021	Fetal Biometry	In-vivo	Surface	2D	SP, PS, Other CNN	1354		453
Broessner [65]	2022	Fracture	In-vitro	Surface	2D	Other CNN	1782		594
Du Toit [58]	2022	Cartilage Monitoring	In-vivo	Other	2D	U-Net	160	40	50
Huang [97]	2022	Scoliosis	In-vivo	Other	VPI	U-Net, GAN	87		22
Pandey [60]	2022	CAOS	In-vivo	Surface	2D	U-Net, Dropout	11976		1711
Rahman [80]	2022	CAOS	In-vivo	Surface	2D	Other	1036		208
Rahman [98]	2022	CAOS	In-vivo	Both	2D	Other CNN, SP, PS	834	208	185
Zhou [78]	2022	Fracture	In-vivo	Surface	2D	U-Net, GAN	10500		1154
Broessner [81]	2022	TKA	In-vivo	Surface	2D	Other ML	7155		1011
Hohlmann [57]	2022	TKA	In-vivo	Surface	3D	U-Net	10315		2456
Jiang [71]	2022	Scoliosis	In-vivo	Surface	2D	Other CNN	2000
Pan [53]	2022	Periodontal	Ex-vivo	Surface	2D	U-Net	274		353
Jiang [63]	2023	Lumbar punction	In-vitro	Surface	2D	SP, PS, Other ML	2357		1010
Li [52]	2023	ROS: Pedicle Screw	Ex-vivo	Surface	2D	U-Net	1000		150
Broessner [68]	2023	Fracture	In-vitro	Surface	3D	Other CNN	9072	1134	1134

Abbreviations: Evaluation (eval), annotation (annot), volume projection image (VPI), Shadow Peak (SP), Phase symmetry (PS), validation (val). Data set size is given in number of 2D image slices, if provided. Otherwise, the values are marked by *.

Table 3. Summary of publications on 3D bone model completion. Algorithms categorized according to the methods section.

Publication	Year	Application	Evaluation	Algorithm
Ghanavati [93]	2010	THA	Ex-vivo	Simulation
Ghanavati [92]	2011	THA	In-vitro	Simulation
Khallaghi [94]	2011	Spine injection	In-vitro	Simulation
Rasoulian [91]	2012	Spine injection	In-vivo	GMM-like
Schumann [83]	2012	THA	In-vitro	Correspondence
Hacihaliloghlu [88]	2013	Spine injection	In-vivo	GMM-like
Rasoulian [90]	2013	Spine injection	In-vivo	Correspondence
Hacihaliloghlu [89]	2014	Spine injection	In-vivo	GMM-like
Haenisch [85]	2015	TKA	In-vitro	Correspondence
Haenisch [99]	2017	TKA	In-silico	Correspondence
Hohlmann [70]	2019	TKA	In-vivo	Correspondence
Hohlmann [86]	2019	TKA	In-vivo	Correspondence
Mahfouz [82]	2021	TKA	Ex-vivo	Unknown
Hohlmann [84]	2023	TKA	In-vivo	Correspondence

Figure 4 (left) shows the different medical applications the publications address. Several rather technical papers do not mention a specific application and are summarized under ‘CAOS’. Of those that do state an application, diagnosis of Scoliosis and Developmental Dysplasia of the Hip (DDH) are most common. Further diagnostic applications include fractures, bone tumors and periodontal tissue assessment and cartilage monitoring, among others. Finally, implant planning in TKA and robotic surgery complete the list. Figure 4 (right) shows the body parts involved during either training or testing. Here, knee and spine represent two thirds of all applications. Other relevant body parts are the hip, arm and wrist.

Figure 4. Addressed medical application (left) and targeted body parts (right) of publications on segmentation. Publications can have multiple mentions.

These findings demonstrate the wide range of medical applications that can profit off ultrasound imaging.

Figure 5 presents the applications addressed in publications on bone model completion. Only three applications are addressed involving surgery of the hip, knee and spine. In TKA, patient-specific bone models are sought for implant planning. In THA, the local bone coordinate system is searched for to determine the correct cup position. Note that in surgery of the spine, solely epidural needle placement is addressed and that there are no publications on pedicle screw placement.

Figure 5. Addressed medical application of publications on bone model completion.

Depending on the application, the proportion of accessible bone surface varies strongly, but no publication reports a quantitative evaluation in this regard. In TKA, just the articulating surfaces in the joint gap need to be determined by the bone model. All THA related publications model the pelvis on the basis of just three patches, located at the iliac crest and pubic symphysis. These patches cover only a very small portion of the pelvis. Similarly, during spine imaging, just the posterior surface of the vertebral bodies is visible. Furthermore, they overlap.

Most publications on segmentation are developed for and evaluated on in-vivo data. For 3D bone modeling, half of the publications are yet to be tested on in-vivo data. Evaluation under laboratory conditions of course differs from the actual scenario. See Figure 6 for a detailed overview. The medical applications and testing domain demonstrate that ultrasound segmentation is a well-developed field of research, which does not hold true for bone model completion.

Figure 6. Testing domain for publications on segmentation (left) and bone model completion (right).

Investigating Table 2, the data set sizes differ strongly, ranging from a few dozen images up to 20.000. In total, almost 130.000 images were annotated just in recent years. This assumes disjunct data sets. Furthermore, the number of slices for some volume images are based on estimates.

Figure 7 shows the adopted methods for segmentation and bone model completion. For segmentation, all but four publications rely on deep learning algorithms. Seventeen works incorporate variants of Shadow Peak and Phase Symmetry into their hybrid method, but CNNs dominate this chart.

Figure 7. Methods applied for segmentation (left) and bone model completion (right). Publications can have multiple mentions.

Categorization of bone model completion algorithms is not as straight-forward. All published methods involve statistical shape models and none include machine learning. While seven utilize a direct correspondence-based optimization, three follow a GMM-like approach. Three simulate ultrasound images to establish a common framework and one paper does not reveal its algorithm. Accordingly, correspondence-based optimization is the most active research direction.

Figures 8–10 present a ranking regarding quantitative performance in terms of Dice, average surface distance error (SDE) and computation time. We normalize the reported processing times by the number of pixels, to account for varying image sizes. In case the image size is not reported, a default value of 256 × 256 pixels is assumed. Overall, a rather poor performance is reported in terms of Dice, when comparing to other medical imaging task, with 0.79 on average and 0.95 being the highest value. In terms of surface distance error, however, the performance is surprisingly good with 0.59 mm on average and 0.11 mm being the lowest value. Regarding processing speed, extremely large differences of a factor of 150 can be found, but real-time processing, i.e. processing of images at 25 Hz, is frequently achieved.

Figure 8. Quantitative performance analysis regarding segmentation. Note that the Dice metric differs strongly depending on the size of the object: Segmenting the bone shadow (marked by asterisk) significantly improves the Dice compared to segmenting just the bone response.

Figure 9. Quantitative performance analysis regarding segmentation. Note that the metric differs strongly depending on its definition: the surface distance error can be either directed or symmetric, computed on point clouds or surface meshes and either the absolute or the root-mean-square distance.

Figure 10. Analysis of computation time normalized to time per 1.000.000 pixel. Note that some images sized are based on estimates (marked by *), and that these times were reported for varying hard- and software.

A fair quantitative evaluation of the segmentation accuracy of the proposed methods is not possible based on the reviewed publications. This would require common test sets and identical metric definitions and evaluation scripts. In the current state, the evaluation procedures vary greatly. Furthermore, for distance-based metrics, even the definitions are inconsistent, e.g. by being either directional or symmetric, by computing the distance to a point cloud or a surface mesh or by computing the absolute or the root-mean-squared error. These variations have a huge influence on the evaluation but are rarely reported. Performance in terms of processing time suffers from similar shortcomings, as varying soft- and hardware renders direct comparison impossible.

Regarding 3D bone modeling a comparison becomes even harder, as various metrics may be adopted. While Dice is not reported at all, surface distances are complemented with target registration errors (TRE) and coordinate-system (CoS) mismatch. As image sizes are rarely reported for 3D applications, we refrain from normalizing the computation time by the number of pixels. See Figures 11–14 for a ranking according to time, CoS, TRE and SDE. Overall, we see larger errors mostly above 1 mm and much slower computation in the range of minutes instead of milliseconds.

Figure 11. Quantitative evaluation of bone model completion algorithms regarding the average surface distance error in mm.

Figure 12. The target registration error describes the deviation of the computed to the ground truth transformation.

Figure 13. Similarly to TRE, the coordinate system error describes the deviation of the computed coordinate system to the ground truth coordinate system in degree.

Figure 14. Computation time of bone model completion algorithms in seconds reported as the common logarithm.

5. Challenges and trends

Over recent years, a clear trend toward machine learning can be noticed. A number of publications compare traditional to machine learning methods and independently confirm the superiority in terms of robustness, accuracy and processing times [39,45,56,62,79], although some show benefits of hybrid approaches. Presumably, the trend to machine learning will continue. However, transferring this success from the field of research into everyday clinical use will bring great challenges: Machine learning models resemble ‘black boxes’. But verification, transparency and interpretability of these models is of high importance in clinical application. Few works address this problem [54,55,59,60] by analyzing the model uncertainty. Furthermore, domain shifts have proved to be an issue, especially when working on small data sets. While the scientific community annotated a huge number of images, they are not published along with the papers. In terms of accuracy, this is not much of an issue as the test data closely resembles the training data in most cases. However, this strongly limits our ability to build models that generalize well. Most publications do not employ separate validation and tests sets. Some utilize cross-validation, but the remaining models hyperparameters are tuned on the data they are tested on. While this improves the apparent accuracy, it limits the generalizability.

Currently, there is no consensus on the best metrics for validation of either segmentation or bone model completion algorithms. Intersection-based metrics like the Dice metric struggle with the thin bone surface annotations: Even small shifts can cause a significant drop in terms of this metric. To counter this problem, some authors dilate the annotations and predictions, improving the metrics apparent performance, but hampering comparability. One advantage of intersection-based metrics over surface distance-based ones is their derivability. One can define a loss function approximating the Dice. By this, the model is optimized on its actual task, improving its accuracy. Furthermore, it can be computed for images that do not show bones, allowing to quantify false positive segmentations.

For surface distance-based metrics, the directionality from prediction to ground truth and vice versa varies. Yet, in order to detect overcomplete (false positive) as well as undercomplete segmentation (false negative), the metric needs to be symmetric. Other variants of this metric found in literature include measuring the distance along the scanline direction only or even ignoring distances above 1 mm, artificially improving the metrics apparent performance. In some cases, the root-mean-squared surface distance error is reported. This penalizes large errors stronger, but hampers interpretability. When bone is visible only in either the image or the reference, the surface distance becomes undefined. This is especially a problem when evaluating 2D data, where empty image slices are frequent. By analyzing volumetric images, even for 2D models, this problem may be solved. In contrast to intersection-based metrics, surface distance errors are easy to interpret and thus facilitate communication with medical personnel.

In summary, a single comprehensive metric is yet to be found. Pandey et al. [24] recommend to report at least one distance- and one overlap-based metric, potentially along other, task-specific, metrics. Given the highly variable definitions of the metrics as discussed above, this is not sufficient from our point of view. Instead, for evaluating overall segmentation quality, we recommend reporting the average symmetric surface distance on volume images containing at least one bone fragment. This metric combines all relevant aspects into a single, easily interpretable scalar. Furthermore, it is affected by both, false positive and false negative segmentations. This allows for direct comparison of algorithms by a single metric. By computation in 3D, the surface distance is defined in all cases, even if some slices do not contain bone surfaces. This mandatory metric can be supplemented with Dice or application-specific metrics. Any high-quality analysis of a segmentation algorithm should involve separate investigation of individual aspects, e.g. tendency to false positive or false negative segmentation, for example by investigating directed distances. Note, that the SSDE does not reflect computational efficiency, which should be evaluated separately, too.

In accordance with Pandey et al. we think that for fair comparison of algorithms, a common benchmark that provides a shared training set as well as a unified evaluation is essential. We want to promote their attempt to build such a benchmark [100].

We cannot recommend one method or the other for either segmentation or bone model completion for the aforementioned reasons. Taking a closer look at the three best-performing segmentation algorithms in terms of mean surface distance error, their respective data sets appear to be small (1000 images or less) and the error computed is directed instead of symmetric [49,62,79]. Still, we can rule out purely traditional methods and see a slight favor for hybrid ones. However, some of these require heavy computational effort. For bone model completion, the two best-performing methods are evaluated in-vitro or do not publish the underlying algorithm [82,85].

The ultrasound vision community heavily relies on a single convolutional neural network architecture: The U-Net. This is in sharp contrast to the non-medical domain. A direct comparison of architectures developed for the medical and non-medical domains is rare. Considering high quality semantic segmentation challenges like ADE20k [101], we see impressive performance gains with architectures that do not resemble the U-Net ‘fully convolutional encoder-decoder with skip connections’-concept. We hypothesize, that great potential lies in a better scholarly exchange between the general and ultrasound vision communities. E.g. the vision transformer has proven its capabilities on non-medical data. While a first analysis did not show any advantage over CNNs on a small-scale data set [81], future research may benefit from additional data published by the medical ultrasound research community [100]. Furthermore, cross-modal models may utilize CT and MRI images. Coming back to ADE20k, the current leading model ‘ONE-PEACE’ builds not only on images, but audio and text, too.

Regarding 3D bone modeling, the total number of publications is rather small. Also, in contrast to publications on segmentation, only half of the evaluations are performed in-vivo. While being discussed for decades, the topic is paid little attention. Even worse, there has been little to no investigation of the effect of different segmentation strategies on the final bone model. While Hacihaliloglu et al. [88,89] find their proposed monogenic filters to be superior to Phase Symmetry and Shadow Peak-based segmentation, no such investigation was performed for machine learning methods. The tradeoff between a lower coverage of the bone model and a more reliable segmentation remains unknown. Future research could build upon the segmentation uncertainty provided by ensemble models or such using dropout. Alternatively, in- or decreasing the segmentation threshold constitutes an even simpler experimental setup, compatible with all presented segmentation algorithms. The final threshold may be aspect to machine learning and adopted locally, too.

While not being in focus, bone model completion is crucial to enable modeling of joints using ultrasound. With this paper, we want to promote future research on this topic by providing an up-to-date review. Most importantly, all of the reviewed publications rely on statistical shape models and none employ machine learning algorithms. We hypothesize, that the success of point-based models in the non-medical domain can be transferred to the medical domain, similar to the task of segmentation. The Completion3D data set constitutes one example for which the surface distance error was halved in recent years [102]. While consisting of man-made objects that differ from natural geometry, the advances of point-based models may be transferred to the medical domain. This appears especially probable as the underlying architecture of the best-performing GRNet (at the time of publication) utilizes 3D convolutions that also proved to be effective in medical image processing [103].

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.