Monocular 3D object detection with thermodynamic loss and decoupled instance depth

Abstract

Monocular 3D detection is to obtain the 3D information of the object from the image. The mainstream methods mainly use L1 loss or L1-like loss to control the instance depth prediction. However, these methods have not achieved satisfactory results. One of the main reasons is that L1 loss or L1-like loss does not accurately reflect the fit between the predicted instance depth and the corresponding ground truth. Another of the main reason is that the instance depth on the RGB image hard to be directly learned in the network. In order to solve the above problems, a novel thermodynamic loss based on the principle of free energy minimisation and a novel depth decoupling method are proposed in this paper. The proposed method is called the monocular 3D object detection network with thermodynamic loss and decoupled instance depth (TDN). In TDN, the optimisation of the instance depth prediction is regarded as the thermodynamic process. Therefore, the thermodynamic loss is designed according to the principle of free energy minimisation. TDN decouples the instance depth into three different depths. By combining the thermodynamic loss and the different types of depths, we can obtain the final instance depth.

KEYWORDS:

• Thermodynamic loss
• monocular 3D detection
• depth estimation
• computer vision

1. Introduction

3D detection aims to locate and identify objects in 3D space, and is mainly used to help machines perceive and understand the surrounding environment. 3D detection is one of the foundations of 3D perception. In order to precisely locate the position of an object in 3D space, some methods use LiDAR sensors (Liu et al., 2023; Shin et al., 2019; Wang et al., 2023; Yan et al., 2018; Yang et al., 2018; Zhou & Tuzel, 2018) or stereo matching (Chen et al., 2018, 2023; Li et al., 2019; Xu & Chen, 2018) to obtain accurate 3D information of objects. Although 3D detection methods based on LiDAR or stereo matching can achieve higher detection accuracy, the sensors used by these methods are expensive, power-consuming and the overall system is relatively complex. Monocular 3D object detection is a branch of 3D detection. Compared with the above detection methods, the monocular detection method only uses the camera to complete the 3D detection task. Because monocular 3D detection has the advantages of low cost, easy maintenance and strong generalisation ability, it has attracted more and more attention from industry and academia and has become a research hotspot.

Since the camera cannot directly obtain the 3D information of the objects, monocular 3D object detection is the challenging task. Depth estimation (Wang et al., 2021a) is the core task of monocular 3D detection. Since the image only provides 2D visual information and lacks direct depth information, it is difficult to accurately estimate the depth information of the objects. According to the different depth estimation methods adopted, the monocular 3D detection methods are mainly divided into the methods based on geometric constraints and the methods based on pseudo lidar (Jiang et al., 2022; Miao et al., 2023). The geometric constraint-based methods mainly use keypoint regression (Liu et al., 2020), geometry constraints (Lu et al., 2021) and shape constraints (Liu et al., 2021) to estimate the depths of objects. The methods based on pseudo-radar mainly use the depth maps to generate pseudo point cloud, and then estimate the depths of the objects through the pseudo point cloud data (Chen et al., 2022; Liu et al., 2022). Although these methods have achieved some results, the accuracy of the depths estimated by these methods still cannot meet the actual requirements.

Currently, monocular 3D object detection methods mainly use L1 loss or L1-like loss to control the prediction of the instance depth. Since L1 loss or L1-like loss only calculates the distance between the predicted instance depth and the ground truth (GT), it ignores the implicit relationship between the predicted instance depth and GT. Hence, it is difficult for L1 loss or L1-like loss to truly reflect the fitting degree between the predicted instance depth and the ground truth. It is necessary to design a new loss that reflects the actual fitting situation. Since the instance depth on the image is not intuitive, it is difficult for the network to learn effectively. It is necessary to design a new representation of the instance depth to reduce the learning pressure of the network.

The instance depth prediction is a complex optimisation problem and it is difficult to find the optimal solution. Since the network learning situation reflected by L1 loss or L1-like loss is different from the actual situation, it further increases the difficulty of finding the optimal solution, and the optimisation algorithm is more likely to fall into a local optimum. The optimisation process has certain similarities with the thermodynamic process. During the optimisation process, the behaviour of optimising the loss function is similar to the behaviour of reducing the internal energy in a thermodynamic system. The goal of optimisation is to minimise loss, while the goal of reducing internal energy in thermodynamics is to bring the system to an equilibrium state. These two goals are very similar. Therefore, the theories from thermodynamics can be used to guide the optimisation process of the network.

Since the distribution of the predicted instance depth is relatively complex, directly predicting instance depth will bring greater network learning pressure to the network, making it difficult for the network to predict accurately. According to the principle of divide and conquer method, the instance depth is decoupled into multiple different depths for prediction respectively, that is, the complex distribution is decomposed into multiple distributions for separate learning, which reduces the learning pressure of the network and improves the accuracy of network prediction.

Based on the above considerations, this paper proposes a novel thermodynamic loss based on the principle of free energy minimisation and a novel depth decoupling method. The proposed method is called the monocular 3D object detection network with thermodynamic loss and decoupled instance depth (TDN). In TDN, the instance depth optimisation process is analogised to the thermodynamic process and follows the principle of free energy minimisation. On this basis, we propose a novel thermodynamic loss based on the principle of free energy minimisation to control the instance depth optimisation process. The proposed thermodynamic loss uses free energy to measure the fit of the predicted instance depth and GT. Compared with L1 loss or L1-like loss, the proposed thermodynamic loss can effectively improve the accuracy of the instance depth prediction. In order to enhance the learning ability of the network to the instance depth, TDN proposed a new depth decoupling method. In TDN, the instance depth is decoupled into the instance surface depth (surface depth), the surface offset depth (offset depth) and the instance error depth (error depth). TDN uses different branches to predict the corresponding depth. The multi-branch prediction method effectively reduces the learning pressure of the network.

We conduct experiments on the KITTI 3D object detection dataset (Geiger et al., 2012). Experimental results demonstrate that TDN can achieve state-of-the-art (SOTA) performance on the monocular 3D object detection and bird's eye view (BEV) tasks.

The contributions of this paper are summarised as follows:

• We analogise the instance depth optimisation process to a thermodynamic process and introduces the principle of free energy minimisation to design the instance depth loss. We proposed a new thermodynamic loss based on the principle of free energy minimisation to allow the network to more accurately fit the instance depth.

• We proposed a new depth decoupling method. Our method decouples the instance depth into three different depths, and the network uses three branches to learn the corresponding depth distributions respectively. Our method effectively reduces the pressure of network learning.

2. Related works

Monocular 3D object detection aims to predict the 3D bounding boxes of objects from a single image. Recently, monocular 3D object detection has been extensively studied with many results. Monocular 3D object detection can be roughly divided into geometric constraint-based methods and depth-based methods.

2.1. Geometric constraints-based method

Since monocular 3D object detection is difficult to directly predict the instance depth, some methods introduce geometric constraints to predict depth. The geometric prior mainly uses the key point information of the object or the projection of the object on the plane to construct the depth information. By introducing geometric prior constraints, the depth prediction is transformed into the regression problem.

Movi-3D (Simonelli et al., 2020) used geometric information to generate virtual views to assist detection tasks, reducing the variation in visual size due to distance. M3D-RPN (Brazil & Liu, 2019) proposed the depth-aware convolution and sets a novel 3D anchor point to alleviate the problem of 3D parameter prediction and improve the accuracy of 3D detection. MonoPair (Chen et al., 2020) obtains more accurate object pose and shape information through the geometric spatial variation relationship between objects in two adjacent frames.

But the above methods rely heavily on feature engineering, which hinders their further extension to general scenarios. To solve this problem, some methods are proposed. MonoCon (Liu et al., 2022) adds the auxiliary task of learning monocular context to training to aid 3D object detection. In the tasks, a set of well-posed projected 2D supervision signals are used for supervised training. M3OD (Li et al., 2022a) obtains the multiple depths based on different attributes of the object and selects the optimal depth from them. With the introduction of uncertainty, most current depth prediction methods are based on uncertainty. GupNet (Lu et al., 2021) proposes the geometry uncertainty projection module to reduce the aggravation of the depth error amplification caused by the height error in the geometric projection formula. DID-M3D (Peng et al., 2022a, 2022b) decouples the instance depth into the visual surface depth and the attribute depth. This depth decoupling method can improve the accuracy of depth prediction by associating the depths with visual cues provided by images. DCD (Li et al., 2022b) uses the key point information of the object on the 2D image to constrain the relationship between the depth values of different parts of the object in 3D space, and then generates multiple depth candidates to improve the 3D detection accuracy.

2.2. Depth-based methods

Depth-based methods can also be called pseudo-lidar methods. The accuracy of the depth map has a direct impact on the3D detection accuracy. Many approaches use the depth maps directly in the framework to enhance the 3D object detection accuracy. The depth-based methods (Ma et al., 2020; Reading et al., 2021; Wang et al., 2020) use the depth maps to introduce depth information of objects in monocular 3D detection. Among these methods, the depth map effectively improves the 3D detection accuracy. Ding et al. (2020) proposed depth-guided dynamic-depth-wise-dilated LCN, which overcomes the limitation of conventional 2D convolutions and narrows the gap between image representation and 3D point cloud representation. MonoDAJD (Lei et al., 2021) used the consistency-aware joint detection mechanism to detect objects in the image and objects in the depth map, and utilises localisation information in the depth detection to optimise detection results. ROI-10D (Manhardt et al., 2019) uses the extract fused feature maps from the 2D detector and the monocular depth prediction network to regress 3D bounding boxes. UR3D (Shi et al., 2020) proposed the distance-normalised unified representation which reduces the nonlinearity of distance by fusing the images and the estimated depth maps to significantly eases the distance learning. Ouyang et al. (2021) proposed the adaptive depth-guided instance normalisation layer and the dynamic depth transformation module to estimate the 3D properties of objects and improve the accuracy of depth prediction. Wang et al. (2021b) combined the generated 2D bounding boxes and the estimated depth maps as the input of the 3D detector to achieve 3D detection. CaDDN (Reading et al., 2021) obtains more accurate depth estimation by predicting the category of each object and the depth distribution within each category, and uses the gridded depth prediction to generate high-quality BEV images for 3D detection. Pseudo-Stereo-3D constructed the pseudo-stereo image by using the left view to generate the virtual right view, and proposed the disparity dynamic convolution to generate the feature of the virtual right view. Then, Pseudo-Stereo-3D used the stereo 3D Detector to detect the constructed pseudo-stereo image.

SGM3D (Zhou et al., 2022) utilised 3D features extracted from the stereo images to enhance the features learned from monocular images. SGM3D also proposed the multi-granularity domain adaptation module (MG-DA) and the IoU matching-based alignment module (IoU-MA) to enforce the network to mimic the stereo representations based on monocular images. Although these depth-based methods can improve the performance of the network, these methods is relatively complex and requires the additional network to generate the depth maps. Hence, these methods usually require more training data and longer inference time.

It can be seen that many geometric constraints methods and depth-based methods have been proposed and achieve impressive results in monocular 3D detection task. As a comparison, TDN introduces thermodynamic theory as a design guide for the instance depth loss, and proposes a novel depth decoupling method to improve the learning efficiency of the network. It has great advantages on improving the accuracy of 3D detection.

3. Method

3.1. Overview

For monocular 3D detection, inaccurate depth estimation is the main reason for the decline in detection accuracy. Hence, the depth estimation is the most critical issue in monocular 3D detection. Since it is difficult to infer 3D information from 2D image, improving the accuracy of depth estimation is a challenging task. In order to improve the accuracy of depth prediction, this paper proposes the monocular 3D object detection network with thermodynamic loss and decoupled instance depth (TDN). The core concept of TDN is to use the principle of free energy minimisation to guide depth estimation. Therefore, a novel thermodynamic loss based on the principle of free energy minimisation and the corresponding depth decoupling method are proposed in TDN. The new loss function can better reflect the fitting degree of the prediction depth and GT. The decoupled depth also improves the learning efficiency of the network. These two methods effectively improve the accuracy of depth prediction. The overview of the proposed TDN model is shown in Figure 1.

Figure 1. The framework of TDN.

As shown in Figure 1, the overall network architecture can be divided into a two-branch network: the 2D detection branch and the 3D detection branch. The 2D detection branch is used to generate 2D bounding boxes, and the 3D detection network uses the RoI (regions of interest, RoI) features to generate 3D bounding boxes.

The keypoints of our approach are described in detail as follows.

3.2. 2D detection branch

The 2D detection backbone of TDN is built on CenterNet (Zhou et al., 2019). Like CenterNet, TDN also uses DLA-34(Yu et al., 2018) as the backbone to extract the features of the objects. As shown in Figure 1, TDN includes three detection branches to detect the information of 2D bounding boxes: (1) the heatmap branch is used to indicate the coarse locations and confidences of the objects in the given image; (2) the offset branch is used to refine the coarse locations to the accurate keypoint; (3) the 2D size branch predicts the size of each box. Different from CenterNet, the keypoint for the 2D detection backbone of TDN is defined as the centre of the 3D bounding box’s projection on the image, rather than the centre of the 2D bounding box. Since the 2D information of the object needs to be used to restore the 3D information of the object, the 2D information of the object must be strongly correlated with the 3D information of the object. Compared with the centre of the 3D bounding box’s projection on the image, the centre of the 2D bounding box does not have a strong relationship with the 3D bounding box, which results in a larger error in the recovered 3D bounding box. The loss functions of the 2D detection are the same as those of CenterNet.

To obtain the object-level features, RoIAlign (Ren et al., 2017) is used to crop and resize the RoI features. For monocular the monocular depth estimation, the RoI features lack location and size cues (Van Dijk & De Croon, 2019). Therefore, the normalised coordinate map is concatenated with each RoI feature in a channel-wise manner to generate the input for the 3D detection branch. The coordinate map is the coordinates formed by projecting the upper left corner and lower right corner of the bounding box generated by CenterNet into 3D space. The main function of the coordinate map is to generate the coarse information of the 3D bounding box based on the information of the bounding box generated by CenterNet. This coarse information is beneficial to the subsequent generation of the accurate information of the 3D bounding box.

3.3. 3D detection branch

3D detection branch is used to predict the information of the 3D bounding boxes of the objects in 3D space. The information of 3D bounding box includes the 3D centre location of 3D bounding box ($x_{3d},y_{3d},z$), dimension $(h,w,l)$ and the orientation $\theta$, where $z$ is the instance depth, $h$ is the height of 3D bounding box, $w$ is the width of 3D bounding box, $l$ is the length of 3D bounding box. The 3D bounding box is parameterised as $\left(x_{3d},y_{3d},z,h,w,l,\theta \right)$.

Let $\left(u_{2d},v_{2d}\right)$ denote the coordinates of 2D bounding box generated by 2D detection branch, $\left(u_{3d},v_{3d}\right)$ denote the coordinates of the centre of the 3D bounding box’s projection on the image. The 3D projected centre offset branch is used to predict the offset $\left({\delta }_{3\text{d}}^{\text{u}},{\delta }_{3\text{d}}^{\text{v}}\right)$ of the centre of the 3D bounding box’s projection on the image. $\left(u_{3d},v_{3d}\right)$ is formulated as follows: (1) $\{\begin{array}{l}{\text{u}}_{3\text{d}}={\text{u}}_{2\text{d}}+{\delta }_{\text{x}3\text{d}}^u\\ {\text{v}}_{3\text{d}}={\text{v}}_{2\text{d}}+{\delta }_{\text{y}3\text{d}}^v\end{array}$(1)

Let $(x_{3d},y_{3d},\text{z}{)}^{\text{T}}$ denote the coordinates of the centre of the 3D bounding box in the camera coordinate system. The camera intrinsic parameter matrix $K$ is used to project the point $\left({\text{u}}_{3\text{d}},{\text{v}}_{3\text{d}}\right)$ to the camera coordinate system. The 3D point $(x_{3d},y_{3d},\text{z}{)}^{\text{T}}$ is defined as: (2) $\{\begin{array}{l}{x}_{3d}=\frac{\left(u_{3d}-c_x\right)\ast z}{f_x}\\ y_{3d}=\frac{\left(v_{3d}-c_y\right)\ast z}{f_y}\\ z=\mathrm{depth}\end{array}$(2) where $\left({\text{c}}_{\text{x}},{\text{c}}_{\text{y}}\right)$ is the pixel location of the camera centre, ${\text{f}}_{\text{x}}$ and ${\text{f}}_{\text{y}}$ represent the horizontal and vertical focal lengths of the camera, respectively.

The loss function $L_{\mathrm{offset}3d}$ of the 3D projected centre offset branch can be defined as: (3) $L_{\mathrm{offset}3d}={\text{L}}_1\left(\text{o},{\text{o}}_{\text{gt}}\right)$(3) where ${\text{o}}_{\text{gt}}$ is the ground-truth offset, $\text{o}$ is the predicted offset, ${\text{L}}_1(\bullet )$ represents ${\text{L}}_1$ loss and “•” indicates the input.

For the orientation, the network predicts the observation angle and uses the multi-bin loss (Mousavian et al., 2017).$L_{\theta }$. The 3D size branch predicts the size of 3D bounding box. Like SMOKE (Liu et al., 2020), a pre-calculated category-wise average size $[\overline{h},\overline{w},\overline{l}{]}^T$ is used to recover object size, where $\overline{\mathrm{h}}$ is the average height of all of 3D bounding boxes in the whole dataset, $\overline{\mathrm{w}}$ is the average width of all of 3D bounding boxes in the whole dataset and $\overline{\mathrm{l}}$ is the average length of all of 3D bounding boxes in the whole dataset. In fact, the 3D size branch outputs the residual size offset $[{\delta }_{\text{h}},{\delta }_{\text{w}},{\delta }_{\text{l}}{]}^{\text{T}}$. TDN utilises the residual size offset and the pre-calculated category-wise average size to generate the size of 3D bounding box. The size of 3D bounding box is formulated as follows: (4) $\left[\begin{array}{c}h\\ w\\ l\end{array}\right]=\left[\begin{array}{c}\overline{\mathrm{h}}+{\delta }_{\text{h}}\\ \overline{\mathrm{w}}+{\delta }_{\text{w}}\\ \overline{\mathrm{l}}+{\delta }_{\text{l}}\end{array}\right]$(4) The loss function $L_{\mathrm{size}3d}$ of the 3D size branch can be defined as: (5) $L_{\mathrm{size}3d}=L_1\left(w,w_{\mathrm{gt}}\right)+L_1\left(h,h_{\mathrm{gt}}\right)+L_1\left(l,l_{\mathrm{gt}}\right)$(5) where $w_{\mathrm{gt}}$, $h_{\mathrm{gt}}$ and $l_{\mathrm{gt}}$ are the size of the ground-truth 3D box, $w$, $h$ and $l$ are the size of the predicted 3D box.

3.4. Decoupled depth

In the monocular 3D detection task, the instance depth is the depth of the centre point of the 3D bounding box. TDN uses the depth value in the depth map at the centre point coordinates of the 2D bounding box to assist depth estimation. The pixel information in the depth map is expressed as the depth distance from the camera to the surface of the object, which is slightly different from the distance from the camera to the centre of the object. This error will lead to a decrease in the accuracy of depth estimation.

Therefore, inspired by DID-M3D, we propose a new depth decoupling method. In our method, the instance depth is decoupled into three parts: the instance surface depth patch (surface depth) $d_s$, the surface offset depth patch (offset depth) $d_o$ and the instance error depth patch (error depth) $d_e$, $d_s,d_e,d_o\in {\mathbb{R}}^{7\ast 7}$. The sum of the surface depth patch and the offset depth patch constitutes the depth value at the centre point coordinates of the 3D bounding box on the depth map. The error depth patch is the error between the depth value at the centre point coordinates of the 3D bounding box and the real depth of the centre point of the 3D bounding box. The decoupled depth is shown in Figure 2. The instance depth patch is defined by: (6) $d_{\mathrm{ins}}=d_s+d_o+d_e$(6) where $d_{\mathrm{ins}}\in {\mathbb{R}}^{7\ast 7}$.

Figure 2. Visualisation of decoupled depth.

The three different detection heads in the 3D detection branch are used to regress these depths. Since the distribution of the instance depth is a coupling of multiple different depth distributions, using only a single detection head will cause greater network learning pressure and make it difficult to effectively learn the complex distribution. After decoupling the instance depth, the corresponding pixel in the depth map has a stronger correlation with the decoupled depth, and each detection head only requires to learn the depth with a single distribution, which reduces the learning pressure of the network and improves the learning efficiency of the network.

Since the surface depth is difficult to directly regress, the surface depth is discretized. Like CaDDN, linear-increasing discretization (LID; Tang et al., 2021) is used to discretize the surface depth. LID is defined as: (7) $d_c=d_{\min }+\frac{d_{\max }-d_{\min }}{D\left(D+1\right)}\ast d_i\left(d_i+1\right)$(7) where $d_c$ is the continuous surface depth value, $\left(d_{\min },d_{\max }\right)$ is the full surface depth range to be discretized, $D$ is the number of the depth bins and $d_i$is the depth bin index. In fact, LID divides the full surface depth range into $D$ bins, and $d_c$ represents the surface depth corresponding to the$\mathrm{ith}$ bin. With the LID method, the surface depth prediction is transformed from a regression task to a classification task. Hence, the surface depth head is used to predict the bin to which the surface depth belongs and output the surface depth corresponding to the bin.

Since each bin contains a range of the surface depth values and $d_c$ is the numerical representation corresponding to the bin, there is a big difference between the surface depth value obtained by classification and the surface depth value obtained by regression. Therefore, it is necessary to convert the surface depth value obtained by classification into an actual predicted surface depth value. The conversion formula can be expressed as follows: (8) $d_s=d_{\min }+\frac{d_{\max }-d_{\min }}{D\ast \left(D+1\right)}\ast \mathrm{dc}\ast \left(\mathrm{dc}+1\right)$(8) According to Equation (8), $d_c$ is converted to the actual predicted surface depth$d_s$.

The offset depth patch $d_o$refers to the offset of the predicted surface depth patch $d_s$ relative to the corresponding depth in the depth map. The surface offset depth patch $d_o$ is defined by: (9) $d_o=d_{\mathrm{depth}}-d_s$(9) where $d_{\mathrm{depth}}$ is the depth (the depth value is padded to $7\ast 7$) in the depth map.

The error depth patch $d_e$ refers to the error between the centre point of the 3D bounding box and the corresponding depth in the depth map. The error depth patch $d_e$ is defined by: (10) $d_e=d_{\mathrm{gt}}-d_{\mathrm{depth}}$(10) where $d_{\mathrm{gt}}$ is the ground truth instance depth. The offset depth patch head and the error depth patch head are used to directly regress $d_o$ and $d_e$.

We employ an uncertainty-aware regression loss to control the regression at all depths after decoupling. (11) $L_d=\sqrt{2}\ast \frac{|d_i-{d_{\mathrm{gt}}}_i|}{{\sigma }_i}+\log {\sigma }_i$(11) where $d_i$ is the $\mathrm{ith}$ depth, ${d_{\mathrm{gt}}}_i$ is the $\mathrm{ith}$ GT depth patch corresponding to the $\mathrm{ith}$ depth patch and ${\sigma }_i$ is the uncertainty patch corresponding to the $\mathrm{ith}$ depth patch. Let ${\sigma }_{\mathrm{patch}}$ is the uncertainty patch of the instance depth patch (${\sigma }_{\mathrm{patch}}$ is the sum of three uncertainty patches corresponding to the three depth patches). In order to obtain the final instance depth, ${\sigma }_{\mathrm{patch}}$needs to be converted into the confidence patch $p_{\left(\mathrm{patch}\right)}$. The probability matrix is defined by: (12) $p_{\left(\mathrm{patch}\right)}=\exp \left({\sigma }_{\mathrm{patch}}\right)$(12)

The instance depth patch outputted by TDN is then combined with the corresponding confidence score patch to convert to the final instance depth. The final instance depth is defined by: (13) $d=\sum \frac{d_{\mathrm{ins}}\ast p_{\mathrm{patch}}}{\sum p_{\mathrm{patch}}}$(13)

The confidence corresponding to the instance depth is defined by: (14) $\begin{array}{rl}{p}_{3d}& =\sum \frac{p_{\mathrm{patch}}\ast p_{\mathrm{patch}}}{\sum p_{\mathrm{patch}}}\end{array}$(14) (15) $\begin{array}{rl}\mathit{\text{score}}& =p_{3d}\times p_{2d}\end{array}$(15) where the $p_{2d}$ is the 2D confidence.

3.5. Thermodynamic loss based on the principle of free energy minimisation

Due to the lack of the depth information in the detection process, the L1 loss or L1-like loss is difficult to accurately reflect the fit of the network. This makes it difficult for the network to accurately predict depth information. In order to solve the above problems, TDN analogises the depth prediction to a thermodynamic process and proposes a new thermodynamic loss function based on the principle of free energy minimisation.

In TDN, the optimisation of depth prediction is analogous to a closed system that exchanges energy with its surroundings. In a closed system, the exchange of energy can be expressed as a change in free energy. The free energy $F$ of the closed system is calculated by: (16) $F=E-\mathrm{HT}$(16) where $E$ is the energy of the system, $H$ is the thermodynamic entropy of the system, and $T$ is the temperature. The free energy follows the principle of free energy minimisation. The principle of free energy minimisation is briefly introduced as follows:

For a closed system that exchanges heat with the outside while maintaining a constant temperature, the state of the system always changes spontaneously in the direction of decreasing free energy and the system reaches equilibrium when the free energy reaches a minimum.

According to the principle of free energy minimisation, any change in the system can be regarded as the result of competition between $E$ and $H$, and the temperature $T$ determines the relative weights of the two. This competition mechanism effectively coordinates the conflict between $E$ and $H$, allowing the system to eventually reach a stable state of low energy and low entropy.

Like a closed system, the optimisation of depth prediction also follows the principle of free energy minimisation. Therefore, the principle of free energy minimisation id used to design the instance depth loss function.

The thermodynamic loss function based on the principle of free energy minimisation is formulated as follows: (17) $L_{\mathrm{tl}}=E-\frac{T}{\frac{{||d-d_{\mathrm{gt}}||}^2}{s}+1}$(17) where $E$ represents the energy in the optimisation process, $T$ is temperature parameter and is a constant, $d$ is the predicted instance depth, $s$ is the instance depth uncertainty, $d_{\mathrm{gt}}$ is the GT depth.

According to the energy-based model (EBM; Joseph et al., 2021), we design a novel the energy calculation method. The energy calculation method is formulated as follows: (18) $E=-\mathrm{Tlog}\sum _{i=1}^n\exp \left(|d_{\mathrm{gt}}-d_i|\right)$(18) Where $d_i$ belongs to the depth value in the instance depth patch $d_{\mathrm{ins}}$. Since the uncertainty s of the instance depth is used to indicate the quality of the instance depth prediction, it is similar to the role of entropy in thermodynamics. The uncertainty s is formulated as follows: (19) $s=\mathrm{mean}\left({\sigma }_{\mathrm{patch}}\right)$(19)

The $s$ is used to design the entropy in the thermodynamic loss. The entropy is formulated as follows: (20) $H=\frac{1}{\frac{{||d-d_{\mathrm{gt}}||}^2}{s}+1}$(20)

When the predicted instance depth and the GT instance depth gradually approach, the entropy gradually decrease and the energy also gradually decreases. In this case, the thermodynamic loss also reduced. It shows that thermodynamic loss obeys the principle of free energy minimisation. Figure 3 shows the relationship between the energy and the entropy.

Figure 3. Visualisation of optimisation process. Darker colours represent higher energy, lighter colours represent lower energy.

As shown in Figure 3, the depth values in the left patch are quite different and far from the GT values. This shows that the left patch has greater energy. As optimisation proceeds, the patch releases energy to the outside and the entropy gradually decreases. The depth values in the right patch tend to be consistent and close to the GT value. This shows that the right patch has greater energy. This means that the right patch is already in a stable state of low energy and low entropy.

The overall loss of TDN is: (21) $L=L_{\mathrm{Heatmap}}+L_{\mathrm{offset}2d}+L_{\mathrm{size}2d}+L_d+L_{\mathrm{tl}}+L_{\mathrm{size}3d}+L_{\mathrm{offset}3d}+L_{\theta }$(21)

Although our approach is inspired by DID-M3D, our approach is significantly different from DID-M3D. The main differences between our approach and DID-M3D are summarised as follows:

• Our method introduces the principle of free energy minimisation in thermodynamics as a guide for the instance depth optimisation process. Under the guidance of this concept, our method proposes the thermodynamic loss function based on the principle of free energy minimisation and uses it to optimise the instance depth prediction. In contrast, DID-M3D did not introduce any theories from other disciplines to guide instance depth optimisation, and still used the traditional the uncertainty-aware regression loss to optimisation the instance depth prediction.

• Although our method also uses the idea of decoupling, the details of our implementation are completely different from DID-M3D. Our approach decouples the instance depth into three different depths while DID-M3D decouples the instance depth into two different depths. For the surface depth, our method uses LID to divide the depth of objects in the depth map into bins, and then regresses the label value of the bin to obtain the surface depth while DID-M3D directly uses the depth map to regress the surface depth. Our method reduces the difficulty of the surface depth regression and relieves the pressure of network learning. In order to further improve the accuracy of the surface depth prediction, we also designed the offset depth to fine-tune the predicted surface depth. In contrast, DID-M3D does not have the fine-tuned surface depth operations.

4. Experiment

Experiments are conducted to evaluate the performance of the proposed monocular 3D object detection method on the benchmark dataset. In this section, the experimental setup, dataset and metrics followed by the discussion of results are described.

4.1. Implementation details

Our experiments were conducted on an NVIDIA 2080Ti GPU using the pytorch framework. We train the network with 200 epochs and employ the Hierarchical Task Learning (HTL) training strategy. The batch size is 10 and the K in the HTL is set to 5. We use the ADAM solver (Kingma & Ba, 2015) with the initial learning rate $1e-5$. The learning rate increases to $1\text{e}-3$ in the first 5 epochs by employing the linear warm-up strategy and decays in epoch 100 and 150 with rate 0.1. T is a constant and is set to 3.8918.

4.2. Dataset and metrics

We conduct experiments on the KITTI 3D detection dataset. The KITTI dataset is the most commonly used dataset in the monocular 3D object detection task, consisting of 7481 training images and 7518 testing images. The KITTI dataset contains three categories: pedestrian, cyclist and car. The KITTI dataset only exposes the training sample labels and the testing sample labels are not available in the KITTI website. The testing sample labels are only used for online evaluation. To conduct ablations, the training samples are divided into 3712 training images and 3769 validation images. This data split is widely adopted by most previous works. We conduct ablation studies based on this split and also report the final results with the model trained on all 7481 images and tested by KITTI official server. The objects in the KITTI dataset are divided into the easy, moderate and hard level according to the object 2D box height, occlusion and truncation levels.

The 40-point interpolated average precision $\left(\mathrm{AP}40\right)$(Qian et al., 2022) with IoU (intersection over union) at 0.7 is used as the evaluation metric. For 3D detection task, the average precision of 3D bounding boxes $\left(\mathrm{AP}{40}_{3D}\right)$ and the average precision of BEV $\left(\mathrm{AP}{40}_{\mathrm{bev}}\right)$ are used for comprehensive evaluation. The expression of AP40 is defined as follows: (22) $\mathrm{AP}=\frac{1}{|R|}\sum _{r\in R}{p}_{\mathrm{interp}}\left(r\right)$(22)

Where $R$ is the predefined set of recall positions and $r$ is the recall. $p_{\mathrm{interp}}\left(r\right)$ is the interpolation function.

4.3. Results of car category on the KITTI test set

We compare our approach with some state-of-art the monocular 3D object detection methods in recent year. These methods are effective methods and have achieved some significant results in monocular 3D object detection. For a fair comparison, we use the implementation provided by the authors to train the selected baseline model.

The comparison results with other state-of-the-art monocular 3D object detection methods are shown in Table 1. The best results are shown in bold. As shown in Table 1, our method achieves superior results of Car category over previous methods. The $3D$ detection and $\mathrm{BEV}$ detection of TDN are (25.59, 34.70), (16.45, 22.89) and (13.42, 19.42) on the easy, medium and hard settings, respectively. Although DID-M3D is slightly higher than our method on the hard settings of $3D$ detection and $\mathrm{BEV}$ detection, TDN gives the relative improvements of (1.19, 1.75), (0.16, 0.13) on the easy and medium settings of 3D and BEV. For $3D$ detection and $\mathrm{BEV}$ detection, TDN gives the relative improvements of (0.9, 2.11) and (0.31, 1.63) on the easy and medium settings compared with MonoJSG (Lian et al., 2022). For the hard settings of AP40BEV, TDN gives the relative improvements of 1.24. In contrast, MonoJSG only outperforms TDN by 0.22 points on the hard settings of $\mathrm{AP}403D$. It demonstrates that TDN is still promising and comparable with those state-of-the-art models. The thermodynamic loss and the decoupled instance depth can more accurately control depth prediction and effectively improve the accuracy of 3D detection.

Table 1. The results of car category on the KITTI test set.

		3D@IoU = 0.7			BEV@IoU = 0.7
Method	Venue	Easy	Moderate	Hard	Easy	Moderate	Hard
ROI-10D	CVPR19	4.32	2.02	1.46	9.78	4.91	3.74
MonoPSR	CVPR19	10.76	7.25	5.85	18.33	12.58	9.91
M3D-RPN	ICCV19	14.76	9.71	7.42	21.02	13.67	10.23
SMOKE	CVPR20	14.03	9.76	7.84	20.83	14.49	12.75
MonoPair	CVPR20	13.04	9.99	8.65	19.28	14.83	12.89
AM3D	ICCV19	16.50	10.74	9.52	25.03	17.32	14.91
PatchNet	ECCV20	15.68	11.12	10.17	22.97	16.86	14.97
D4LCN	CVPR20	16.65	11.72	9.51	22.51	16.02	12.55
MonoDle	CVPR21	17.23	12.26	10.29	24.79	18.89	16.00
CaDDN	CVPR21	19.17	12.32	9.65	26.19	18.27	14.05
MonoRCNN	ICCV21	18.36	12.65	10.03	25.48	18.11	14.10
GUPNet	ICCV21	22.26	15.02	13.12	30.29	21.19	18.20
MonoJSG	CVPR22	24.69	16.14	13.64	32.59	21.26	18.18
DID-M3D	ECCV22	24.40	16.29	13.75	32.95	22.76	19.83
TDN		25.59	16.45	13.42	34.70	22.89	19.42

Both metrics are evaluated by AP40 at 0.7 IoU threshold.

4.4. Results of car category on the KITTI validation set

The comparison results on the KITTI validation set are shown in Table 2. As can be seen from Table 2, TDN outperforms all baseline methods on all evaluated protocols. Compared to the best competing method DID-M3D, our method improves (3.12, 4.12), (1.34, 2.07) and (0.41, 1.58) on the easy, medium and hard settings of 3D detection and BEV detection, respectively. Combined with the results in 3D/BEV detection, it can be concluded that the overall performance of TDN is better than that of the state-of-the-art methods in terms of the detection accuracy.

Table 2. The results of car category on the KITTI validation set.

	3D@IoU = 0.7			BEV@IoU = 0.7
Method	Easy	Moderate	Hard	Easy	Moderate	Hard
M3D-RPN	14.53	11.07	8.65	20.85	15.62	11.88
MonoPair	16.28	12.30	10.42	24.12	18.17	15.76
MonoDle	17.45	13.66	11.68	24.97	19.33	17.01
GUPNet	22.76	16.46	13.72	31.07	22.94	19.75
DID-M3D	22.98	16.12	14.03	31.10	22.76	19.50
TDN	26.10	17.46	14.44	35.22	24.83	21.08

Both metrics are evaluated by AP40 at 0.7 IoU threshold.

4.5. Effect of each component of TDN

In this section, a set of ablation experiments are to verify the effectiveness of each component of TDN. The components that affect the performance of TDN are mainly the thermodynamic loss and the decoupled instance depth. In these experiments, we show the contribution of these components to the overall performance of TDN. For TDN, it should be shown that all components are useful for the final results. All ablation studies were performed on the KITTI validation set for the car category, and the main results are summarised in Table 3. In experiment (a), the decoupled instance depth and the thermodynamic loss are not used. The traditional L1 loss is used for depth prediction instead of the thermodynamic loss.

Table 3. Effect of each component on the performance of TDN.

			3D@IoU = 0.7			BEV@IoU = 0.7
D.dep Ther.loss			Easy	Moderate	Hard	Easy	Moderate	Hard
(a)	–	–	18.01	12.94	11.29	26.43	19.33	17.25
(b)	✓	–	23.46	15.77	13.67	31.00	22.07	18.85
(c)	–	✓	22.55	15.06	12.31	30.03	21.67	18.24
(d)	✓	✓	26.10	17.46	14.44	35.22	24.83	21.08

Both metrics are evaluated by AP40 at 0.7 IoU threshold. “D.dep”: Decoupled Depth; “Ther.loss”: thermodynamic loss.

From Table 3, it can be seen that the thermodynamic loss and the decoupled instance depth have a powerful influence on the performance of TDN. Compared with (b) and (a), the decoupled instance depth brings the relative improvements of (5.45, 2.83, 2.38) on the three settings of 3D detection and (4.57, 2.74, 1.6) on the three settings of BEV detection. Compared with (c) and (a), the thermodynamic loss brings the relative improvements of (4.54, 2.12, 1.02) on the three settings of 3D detection and (3.60, 2.34, 0.99) on the three settings of BEV detection. By comparing settings (b$\to$a) and (c$\to$a), we can find the decoupled instance depth and the thermodynamic loss can effectively and stably improve the overall performance.

In TDN, the decoupled instance depth enables the different branches of the network to learn the corresponding depth distributions, thereby effectively reducing network learning pressure and making training more stable. To further investigate the impact of the decoupled instance depth on network optimisation performance, we compared the convergence of the decomposed approach and the non-decomposed approach. Figure 4 shows the convergence curves for TDN and TDN using the coupled instance depth. As shown in the Figure 4, it can be seen that the total depth loss using the decoupled instance depth converges slightly faster than the total depth loss using the coupled instance depth. It shows that decoupled instance depth mainly improves the accuracy of the depth prediction and slightly accelerates the convergence of the network.

Figure 4. The convergence curves for TDN and TDN using the coupled instance depth. The X-axis represents the epoch, and the y-axis represents the loss value.

Compared with the traditional L1 loss, the thermodynamic loss investigates the depth prediction process from an energy perspective and can better reflect the rules of the predicted depth distribution. Hence, the thermodynamic loss can more accurately control the depth prediction process. To further investigate the impact of different losses on network optimisation performance, we compared the convergence of the thermodynamic loss and the L1 loss. Figure 5 shows the convergence curves for different losses. As shown in the Figure 5, it can be seen that the total depth loss using the thermodynamic loss converges faster than the total depth loss using the L1 loss. The thermodynamic loss significantly accelerates the convergence rate of the total depth loss and can get the optimal value within the given generations. Consequently, the thermodynamic loss is more suitable for the optimisation of depth prediction than the L1 loss.

Figure 5. The optimisation process of thermodynamic loss and L1 loss. The X-axis represents the epoch, and the y-axis represents the loss value.

4.6. Qualitative results

Figure 6 shows the qualitative results on KITTI validation set. In Figure 6, the qualitative results on 3D detection and BEV detection are provided. The detection objects contain cars, pedestrians and cyclists. We can observe that for most cases, the model predictions are quite precise. Our method achieves high detection performance for both distant objects and slightly occluded objects.

Figure 6. Qualitative results on KITTI validation set. Red represents the GT value, green represents the predicted value.

However, our method has lower depth estimation accuracy for heavily occluded objects or small objects. This is a common dilemma for most monocular works. This is because the effect of 3D detection is mainly affected by the effect of 2D detection in monocular 3D detection.

5. Discussions

In monocular 3D detection, the depth prediction is the core task. Therefore, improving the accuracy of depth prediction can effectively improve the accuracy of 3D detection. In TDN, the proposed thermodynamic loss and the proposed decoupled instance depth are used to improve the accuracy of depth prediction. Experiments show that TDN can effectively improve the detection accuracy.

TDN regards the optimisation process of depth estimation as a process of energy exchange between a closed system and the outside. The proposed thermodynamic loss investigates the depth prediction process from an energy distribution perspective. L1 loss or L1 type loss only calculates the distance between the predicted depth and GT depth, without investigating the difference between the two from a distribution perspective. Since depth prediction is essentially a distribution prediction, the thermodynamic loss can more accurately reflect the actual situation of depth prediction.

Due to the lack of sufficient depth information in monocular imagery, it is difficult to directly regress the instance depth information. And there is a certain difference between the depth observed on the image and the actual depth. These circumstances increase the learning pressure of the network and make network training unstable. The decoupled instance depth splits the instance depth into 3 different depths and uses 3 network branches to predict these 3 depths respectively. This allows each network branch to fit only a single distribution, effectively reducing the learning pressure of the network. The decoupled depth can also highlight the error between the actual depth and the predicted depth in more detail. The ablation experiments have proven that the thermodynamic loss and the decoupled instance depth can effectively improve 3D detection accuracy.

For commonly used dataset, TDN can obtain better results than other baseline models. It shows that TDN has better detection ability and performs better than some state-of-the-art 3D object detection networks.

6. Conclusion

In this paper, we proposed TDN to solve the inaccurate depth estimation in for monocular 3D object detection. TDN introduces the concept of thermodynamics to design the thermodynamic loss function to control the depth regression and proposed a novel depth decoupling method to decouple the instance depth into three different depths. The combination of the thermodynamic loss and the decoupled instance depth investigate depth regression from a new perspective. The experimental results indicate that TDN can generate more accuracy depth information and detection accuracy. Comparisons with some state-of-the-art baseline methods, it demonstrates that our method is more effective and efficient in terms of the detection quality in most cases. The ablation experiments show that the effectiveness of each component of the model. In future work, we will further improve the depth prediction mechanism to achieve more accurate detection results.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Hubei University of Technology Graduate Research Innovation Project (4306.22019). The work described in this paper was support by National Natural Science Foundation of China Foundation No.61300127.