Multi-source point cloud registration for urban areas using a coarse-to-fine approach

ABSTRACT

The foundational step in constructing a digital twin involves transforming the physical world into an immersive 3D virtual environment. Unmanned aerial vehicle (UAV)-based photogrammetry, mobile mapping system (MMS), and terrestrial laser scanning (TLS) serve as powerful tools for 3D reconstruction of urban environments. Given the distinctive strengths and weaknesses of each sensor, a multi-source point cloud registration technique is imperative to achieve a detailed and precisely geolocated 3D urban model. Hence, we developed coarse-to-fine approach methods, called building exterior wall-based (BEWB) and building outline-based (BOB) algorithms, to register point clouds captured by diverse sensors in urban scenes. The BEWB algorithm coarsely registers urban point clouds by extracting building exterior walls, establishing correspondence points, and effectively removing outliers within this correspondence point set. The BOB algorithm precisely registers the urban point clouds acquired from multiple sensors by leveraging building outlines and points corresponding to the ground of the point clouds. To validate the proposed algorithms, we initially registered a UAV-based photogrammetry point cloud with the MMS point cloud and subsequently registered the TLS point clouds. Comprehensive quantitative and qualitative analyzes of the results demonstrate that our algorithms outperform existing methods in achieving precise registration. By successfully registering three distinct point clouds, we generated a comprehensive urban scene point cloud characterized by enhanced precision, point density, and geolocation accuracy.

KEYWORDS:

• Multi-source
• cross-source
• urban
• point cloud registration
• coarse-to-fine

1. Introduction

With the advent of the fourth industrial revolution, the concept of a smart city has emerged as a future urban platform capable of addressing urban challenges and enhancing the overall quality of life. A critical component in the realization of a smart city is the establishment of a real-time connection between the physical and virtual worlds through a digital twin (Deren, Wenbo, and Zhenfeng 2021; Jiang et al. 2021). The initial step toward creating a digital twin involves reconstructing the physical world in an immersive 3D virtual environment.

Unmanned aerial vehicle (UAV)-based photogrammetry, mobile mapping system (MMS), and terrestrial laser scanning (TLS) are powerful tools for 3D urban reconstruction (Chiabrando et al. 2019; Mohammadi et al. 2021; Previtali, Brumana, and Banfi 2022). UAV-based photogrammetry produces extensive 3D reconstructions with rich textures and comprehensive coverage. However, the resulting photogrammetric point clouds have a relatively low point density, and the use of ground control points (GCPs) is essential to ensure accurate geolocation (Nota, Nijland, and de Haas 2022; Padró et al. 2019). MMS excels at capturing precise and dense point clouds along roads, and the increasing prevalence of high-definition (HD) maps makes it more convenient to access accurately georeferenced MMS point cloud data (Won et al. 2019; Q. Xu et al. 2023). Nevertheless, MMS has limitations in capturing geospatial information beyond roadways, including areas such as roofs that are not visible on the road. TLS is adept at capturing precise and dense point clouds in inaccessible areas via MMS. However, TLS is constrained to a limited range with one station and relies on local coordinate systems. Each sensor exhibits distinct strengths and weaknesses, emphasizing the need for a multi-source point cloud registration technique to comprehensively reconstruct urban environments in a 3D model (Choi and Song 2022; Kim, Chen, and Cho 2018; Park et al. 2019; Zhao et al. 2023). This approach ensures enhanced precision, a higher point density, and superior geolocation accuracy in the final representation (Chiabrando et al. 2019; Previtali, Brumana, and Banfi 2022; J. Li, Hu, and Ai 2021).

The iterative closest point (ICP) algorithm (Besl and McKay 1992) is the predominant choice for point cloud registration (PCR). Nevertheless, the performance of the ICP algorithm relies significantly on the initial alignment because it determines correspondence points through proximity in each iteration. When confronted with a suboptimal initial alignment, the algorithm tends to converge toward the local minima or in more challenging scenarios, fails to converge. Consequently, the prevailing strategy for PCR is to adopt a coarse-to-fine approach. This involves roughly aligning the two point clouds and then progressively refining the registration for high-precision results.

Coarse registration can be performed using either local features extracted from points or clusters of primitives, or global features derived from the entire point cloud. Methods employing global features, such as the 3D normal distribution transform (NDT) (Magnusson, Lilienthal, and Duckett 2007) and frequency domain-based approaches (R. Huang et al. 2021), typically require significant overlap ratios. While effective for registering point clouds of similar scales, global feature methods are inadequate for registering between small-scale point clouds and large-scale point clouds. Therefore, this study focused on coarse registration utilizing local features because it was necessary to register small-scale TLS point clouds to a large-scale UAV-based photogrammetry point cloud.

Coarse registration utilizing local features entails extracting features, such as points, lines, and planes, from two point clouds and resolving the transformation problem through correspondence features. In a study by S. Li et al. (2021), straight lines were extracted from MMS and UAV-based photogrammetry point clouds, and the two point clouds were registered using the correspondence lines. This process involved projecting both point clouds onto a horizontal plane and extracting straight lines using a line segment detector (LSD) algorithm (Wang, Neumann, and You 2009). Studies by B. Yang et al. (2015) and Cheng et al. (2018) employed building outlines and ground surfaces to register airborne laser scanner (ALS) point clouds and TLS point clouds. Yang et al. conducted horizontal registration by aligning the building outlines using geometric constraints and Laplacian matrices, while vertical registration was accomplished by utilizing the average elevation difference between points surrounding buildings of two point clouds. Cheng et al. achieved horizontal registration by utilizing the four-point congruent sets method on the extracted building outline, while vertical registration was accomplished by dividing the ground into a grid and establishing correspondence ground planes. Xu et al. (2022) extracted 3D line features from two TLS point clouds and combined them to form 2-line congruent sets (2LCS). The transformation problem was solved using correspondence-congruent sets with the highest registration score in random sample consensus (RANSAC). Additionally, Y. Xu et al. (2019) extracted voxel-based 4-plane congruent sets (V4PCS) from two TLS point clouds and tackled the transformation problem using correspondence-congruent sets with the highest registration score in RANSAC. Lines and planes inherently contain more geometric information than individual points do. Consequently, when lines and planes are accurately extracted from the two point clouds and the correspondence features are accurately established, solving the transformation problem becomes more straightforward compared to point-based coarse PCRs (Dong et al. 2020).

However, extracting identical lines or planes from the same structure in cross-source point clouds poses considerable challenges. Cross-source point clouds tend to contain more noise and outliers than same-source point clouds and exhibit different point densities and precisions, further complicating feature extraction and corresponding. Moreover, to increase the chances of success, both point clouds must comprehensively contain structures such as buildings (X. R. Huang et al. 2021).

The point-based coarse PCR provides a high level of generalizability and flexibility (J. J. Li, Hu, and Ai 2021; J. Li, Hu, and Ai 2021). Consequently, it has emerged as a more adaptable solution for cross-source point clouds that typically present more constraints than their same-source counterparts. In point-based coarse PCR, distinctive points among the surrounding points are first identified as keypoints using keypoint detectors, such as intrinsic shape signatures (ISS) (Yu 2009), local surface patches (LSP) (H. Chen and Bhanu 2007), and KeypointNet (Suwajanakorn et al. 2018). Subsequently, keypoint descriptors such as the fast point feature histogram (FPFH) (Rusu, Blodow, and Beetz 2009), radius-based surface descriptor (RSD) (Marton et al. 2010), unique shape context (USC) (Tombari, Salti, and Di Stefano 2010), and PPFNet (Deng, Birdal, and Ilic 2018) were employed to articulate the keypoints as feature vectors. The keypoints from the two point clouds are compared based on the similarity of their feature vectors, leading to the establishment of correspondence points.

However, existing 3D keypoint methods are encumbered by limitations that result in numerous outliers in the correspondence point set. The initial correspondence point set often exhibits an outlier ratio exceeding 95% (Bustos and Jun Chin 2018), underscoring the need for a method to achieve successful registration, even when there is a significant number of outliers within the set of correspondence points. Zhou, Park, and Koltun (2016) introduced fast global registration (FGR), which effectively removed outliers using the Geman-McClure cost function and graduated nonconvexity (GNC). Yang, Shi, and Carlone (2021) proposed truncated least-squares estimation and semidefinite relaxation plus plus (TEASER++), which decomposes the registration problem into scale, rotation, and translation subproblems and estimates a transformation matrix by removing outliers using a truncated least-squares cost function and adaptive voting. Similarly, J. Li, Hu, and Ai (2021) decomposed the registration problem into three subproblems, and the outliers were addressed using Tukey’s biweight cost function and one-point RANSAC. Furthermore, they emphasized that their method exhibited a lower computational complexity than TEASER++. According to the experimental results (H. Yang, Shi, and Carlone 2021; J. Li, Hu, and Ai 2021), FGR demonstrated robustness with up to 80% outliers, whereas the TEASER++ and Li et al.’s algorithms achieved robustness with up to 99% outliers. However, the challenge intensifies in the context of cross-source urban scene point clouds, characterized by their large-scale and diverse noise such as clutter objects. The outlier ratio in the correspondence point set often exceeds 99%, leading to the failure of existing algorithms in achieving successful registration. Consequently, there is a crucial demand for a point-based coarse PCR method that is specifically tailored to urban scene cross-source point clouds.

The ICP algorithm is the de facto standard for fine registration, boasting numerous variants. To enhance the registration accuracy of the conventional ICP algorithm, researchers have proposed several alternative error metrics to replace point-to-point error distances. These alternatives include point-to-line (Censi 2008), point-to-plane (Y. Chen and Medioni 1992), and plane-to-plane metrics (Segal, Haehnel, and Thrun 2009). Addressing the challenge of outliers within a correspondence point set is crucial for the ICP algorithm. In this context, Babin, Giguere, and Pomerleau (2019) introduced a robust ICP algorithm that mitigates the impact of outliers on registration. They achieved this by assigning significant weights to inliers and minimal weights to outliers using an M-estimator. Despite these advancements, the ICP algorithm and its variants struggle to perfectly filter outliers in the correspondence point set, particularly in the face of the challenges posed by cross-source urban scene point clouds. This limitation hampers the attainment of precise PCR. Consequently, there is a need for robust fine registration techniques specifically tailored to urban scene cross-source point clouds.

The findings of the analysis of related research aimed at addressing large-scale cross-source urban point cloud registration are summarized as follows: urban point clouds can be georeferenced using local or global coordinates. Therefore, to register two point clouds that use different coordinate systems, a coarse-to-fine approach involving coarse registration followed by fine registration is required. Coarse registration methods can be divided into line-, plane-, and point-based methods. Among these, point-based coarse registration with a high level of generalizability and flexibility is suitable for cross-source point cloud registration. The primary challenge in point-based coarse registration is a significantly large number of outliers in a set of putative correspondence points. However, the existing algorithms lack robustness when the outlier rate within the set of putative correspondence points exceeds 99%. In the registration of cross-source urban point clouds characterized by their large-scale and diverse noise such as clutter objects, it is common for the ratio of outliers within a set of putative correspondence points to exceed 99%. This high rate of outliers makes it challenging to achieve successful registration using the existing algorithms. The sensors used to acquire urban point clouds have two inherent advantages: they provide accurate gravity direction and have absolute scale. Therefore, by leveraging these two advantages, it is necessary to develop a point-based coarse registration method that surpasses the robustness of existing algorithms for large-scale cross-source urban point cloud registration. In fine registration, the ICP-type algorithm is commonly used as the de facto standard. However, point clouds in urban scenes often contain various types of noise, including clutter objects. Moreover, the point clouds obtained from the different sensors exhibit differences in point density and precision. Consequently, ICP-type algorithms face challenges in achieving precise registration of key structures. Therefore, by leveraging the two advantages inherent in urban point clouds, a fine registration method is required that extracts only the key structures from both point clouds and employs them as reference points.

In this study, we propose a robust registration method for coarse PCR in cross-source urban scenes, called a building exterior wall-based (BEWB) algorithm. The BEWB algorithm capitalizes on the distinctive role of the exterior walls of buildings in defining urban landscapes. We specifically focused on registering point clouds with four degrees of freedom (DOF), benefiting from the accurate gravity direction obtainable from most sensors employed to capture urban environments. Additionally, this paper presents a building outline-based (BOB) algorithm for fine registration. The BOB method utilizes building outlines and points corresponding to the ground of point clouds to achieve precise registration between the two point clouds.

To validate the performance of the proposed algorithms, we registered UAV-based photogrammetry and MMS point clouds. Subsequently, for areas requiring enhanced 3D modeling intricacy, we registered TLS point clouds with the MMS+UAV-based photogrammetry point cloud. The obtained results were analyzed quantitatively and qualitatively to assess the accuracy and reliability of PCR.

2. Methodology

2.1. Overview

As shown in Figure 1, this study registered UAV-based photogrammetry and TLS point clouds with an MMS point cloud, anticipating that the accessibility of accurately georeferenced MMS point clouds will increase with the advancement of the autonomous driving industry. The initial alignment of the MMS and UAV-based photogrammetry point clouds relied on the global navigation satellite system (GNSS) information from both platforms, eliminating the need for coarse registration. Consequently, only fine registration was performed using the BOB algorithm.

Figure 1. Schematic workflow.

Subsequently, for areas requiring more detailed 3D reconstruction, the TLS point clouds were registered with the MMS+UAV-based photogrammetry point cloud. As the TLS point clouds were in a local coordinate system, coarse registration was conducted using the BEWB algorithm, followed by fine registration using the BOB algorithm. The results were subjected to comprehensive analysis incorporating qualitative and quantitative evaluations, as described in detail in the following section.

2.2. Coarse registration using BEWB algorithm

Sensors employed for urban environment capture, such as UAV-based photogrammetry, MMS, and TLS, offer accurate gravity direction and have absolute scale. Considering these advantages, the BEWB algorithm was introduced, as illustrated in Figure 2.

Figure 2. BEWB algorithm flowchart.

In registration problems, it is essential to establish the correct correspondence points. Therefore, the BEWB algorithm attempts to overcome the above issue in two ways: by reducing the proportion of outliers within the set of putative correspondence points and by removing outliers within the set of putative correspondence points.

First, The BEWB algorithm aims to reduce the proportion of outliers within a set of putative correspondence points. Exterior building walls play a pivotal role in defining urban landscapes. Therefore, the probability that correct correspondence points will be generated among the exterior wall points is high. The BEWB algorithm utilizes the normal vectors of the points to identify the points corresponding to the exterior walls of the building and generates putative correspondence points among them using FPFH. In this process, effectively detecting keypoints from cross-source point clouds is challenging given the differences in point density and precision between the two datasets. To overcome this challenge, the BEWB algorithm substitutes keypoint detection with downsampling, specifically by adjusting the voxel size to 1 m. Subsequently, the points of the downsampled point cloud were described in a keypoint vector format using the FPFH, thus establishing putative correspondence points.

Second, BEWB effectively removes outliers within the set of putative correspondence points. As both point clouds share the same absolute scale, the first round of outlier removal based on the scale values was applied to the putative correspondence points. Subsequently, during rotation matrix estimation using an annealing maximum likelihood-type (AM)-estimator, a second round of outlier removal was executed from the putative correspondence points. After a substantial reduction in outliers, the remaining putative correspondence points were subjected to the RANSAC algorithm to estimate the translation vector. Finally, the estimated transformation matrix is employed for the coarse registration of the two point clouds.

2.2.1. Building exterior wall extraction

Building exterior walls are typically perpendicular to the ground, and most sensors utilized for 3D mapping of urban environments offer accurate gravity direction. Hence, the BEWB algorithm extracts points corresponding to the building exterior walls by leveraging the Z-axis value of the normal vector of the points.

According to the guidelines for the safety and maintenance of facilities (Korea Authority of Land & Infrastructure Safety 2022) in Korea, building exterior walls with tilts greater than 1/150 (0.38${ }^{\circ }$) are classified as structures that necessitate emergency reinforcement, prohibition of use, or demolition. Surfaces that meet safety standards are likely to be walls. Therefore, points constituting a surface with a tilt lower than 1/30 (1.91°), which is five times the safety standard, were classified as walls, to account for point cloud noise and precision. Consequently, points with Z-axis values of the normal vectors smaller than sin ($1.91{ }^{\circ }$) were extracted as walls. To reduce noise after extracting the exterior walls, we applied a statistical outlier removal (SOR) algorithm (Rusu et al. 2008).

Deriving putative correspondence points exclusively from points corresponding to the exterior walls of buildings has two notable advantages. First, building exterior walls play a pivotal role in shaping urban landscapes. Limiting the consideration to these points significantly diminishes the outlier ratio within the correspondence point set, unlike when using the entire point cloud. Second, this approach significantly reduces the overall number of putative correspondence points, resulting in a substantial reduction in the computational workload during subsequent processing.

2.2.2. Filtering outliers of correspondence vectors using scale value

For two given point clouds, $\mathit{A}=\{a_t{\}}_1^{N_A}$ and $\mathit{B}=\{b_t{\}}_1^{N_B}$, if the set of correspondence points ($\mathit{C}{P}_I=\{\left(a_i,b_i\right){\}}_1^{N_I}$) consists only of inliers, then $a_i$ and $b_i$ adhere to Equation (1)(1) $b_i=\mathit{sR}{a}_i+\mathit{t}+e_i$(1) , where $\mathit{s}>0$ is a scale factor, $\mathit{R}\in \mathit{SO}\left(3\right)$ represents the rotation matrix, $\mathit{t}\in {\mathbb{R}}^3$ denotes the translation vector, and $e_i$ accounts for the measurement noise. Variables $\mathit{s}$, $\mathit{R}$, and $\mathit{t}$ are unknown and must be estimated. The measurement noise $e_i$, follows a zero-mean Gaussian noise with isotropic covariance, reflecting the precision of the sensors.

(1) $b_i=\mathit{sR}{a}_i+\mathit{t}+e_i$(1)

This study aims to register point clouds acquired from MMS, UAV-based photogrammetry, and TLS, which provide gravity direction and have absolute scale. To obtain Equation (2)(2) $b_i=R_k{a}_i+\mathit{t}+e_i$(2) , we set $\mathit{s}$ to 1 and fixed the $\mathit{\omega }$ and $\mathit{\phi }$ components of the rotation matrix $\mathit{R}$ to 0, solely estimating the $\mathit{\kappa }$ component. Here, $R_k$ is a 1DOF rotation matrix.

(2) $b_i=R_k{a}_i+\mathit{t}+e_i$(2)

However, the correspondence point set established by the keypoint descriptor contains a significant number of outliers. Consequently, a point-based coarse PCR formulation becomes a robust estimation problem, as expressed in Equation (3)(3) $\underset{R_k,\mathit{t}}{\mathit{argmin}}\sum _{\mathit{i}=1}^{{\mathit{N}}_{\mathit{I}+\mathit{O}}}\mathit{\rho }\left(||b_i-\left(R_k{a}_i+\mathit{t}\right)||\right)$(3) . Here, $\mathit{\rho }\left(\cdot \right)$ is a robust function, $||\cdot ||$ denotes $l_2-\mathit{norm}$, and $\mathit{C}{P}_{\mathit{I}+\mathit{O}}=\{\left(a_i,b_i\right){\}}_1^{{\mathit{N}}_{\mathit{I}+\mathit{O}}}$ represents a correspondence point set composed of inliers and outliers.

(3) $\underset{R_k,\mathit{t}}{\mathit{argmin}}\sum _{\mathit{i}=1}^{{\mathit{N}}_{\mathit{I}+\mathit{O}}}\mathit{\rho }\left(||b_i-\left(R_k{a}_i+\mathit{t}\right)||\right)$(3)

RANSAC is widely used for estimating the parameters of a mathematical model from a dataset containing outliers. However, a limitation of RANSAC is the exponential increase in computational time with higher outlier ratios. Therefore, a substantial portion of outliers should be preprocessed and removed before applying RANSAC. This was facilitated by leveraging the fact that the sensors employed in this study provide absolute scale.

To achieve translation invariance in Equation (2)(2) $b_i=R_k{a}_i+\mathit{t}+e_i$(2) , Equation (2)(2) $b_i=R_k{a}_i+\mathit{t}+e_i$(2) for the $\mathit{i}$^th correspondence point is subtracted from Equation (2)(2) $b_i=R_k{a}_i+\mathit{t}+e_i$(2) for the $\mathit{j}$^th correspondence point to yield Equation (4)(4) ${\vec{b}}_{\mathit{ij}}=R_k{\vec{a}}_{\mathit{ij}}+{\vec{e}}_{\mathit{ij}}$(4) , which effectively eliminates the translation vector. Here, ${\vec{a}}_{\mathit{ij}}=a_j-a_i$ and ${\vec{b}}_{\mathit{ij}}=b_j-b_i$ represent the correspondence vectors, and ${\vec{e}}_{\mathit{ij}}=e_j-e_i$ is the noise vector.

(4) ${\vec{b}}_{\mathit{ij}}=R_k{\vec{a}}_{\mathit{ij}}+{\vec{e}}_{\mathit{ij}}$(4)

To attain rotation invariance, we apply the $l_2-\mathit{norm}$ to Equation (4)(4) ${\vec{b}}_{\mathit{ij}}=R_k{\vec{a}}_{\mathit{ij}}+{\vec{e}}_{\mathit{ij}}$(4) and leverage the triangle inequality to formulate Equation (5)(5) $-||{\vec{e}}_{\mathit{ij}}||\le ||{\vec{b}}_{\mathit{ij}}||-||R_k{\vec{a}}_{\mathit{ij}}||\le ||{\vec{e}}_{\mathit{ij}}||$(5) .

(5) $-||{\vec{e}}_{\mathit{ij}}||\le ||{\vec{b}}_{\mathit{ij}}||-||R_k{\vec{a}}_{\mathit{ij}}||\le ||{\vec{e}}_{\mathit{ij}}||$(5)

To remove outliers by leveraging the absolute scale of the point clouds, we divide $||{\vec{a}}_{\mathit{ij}}||$ in Equation (5)(5) $-||{\vec{e}}_{\mathit{ij}}||\le ||{\vec{b}}_{\mathit{ij}}||-||R_k{\vec{a}}_{\mathit{ij}}||\le ||{\vec{e}}_{\mathit{ij}}||$(5) to formulate Equation (6)(6) $1-\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}\le s_{\mathit{ij}}\le 1+\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}$(6) . Subsequently, we estimate the set of inliers (${\widetilde{\mathit{CV}}}_{\mathit{sI}}=\{\left({\vec{a}}_{\mathit{p}},{\vec{b}}_{\mathit{p}}\right){\}}_1^{{\mathit{N}}_{\mathit{cv}\_\mathit{sI}}}$) composed of the correspondence vectors that satisfy Equation (6)(6) $1-\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}\le s_{\mathit{ij}}\le 1+\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}$(6) , where, $s_{\mathit{ij}}=\frac{||{\vec{b}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}$ represents the scale value of point cloud $\mathit{B}$ relative to point cloud $\mathit{A}$.

(6) $1-\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}\le s_{\mathit{ij}}\le 1+\frac{||{\vec{e}}_{\mathit{ij}}||}{||{\vec{a}}_{\mathit{ij}}||}$(6)

2.2.3. Rotation matrix estimation

Utilizing the scale value for outlier removal can significantly reduce the proportion of outliers within the set of correspondence vectors, which typically reaches approximately 80%. However, the outlier ratio is still relatively high to apply RANSAC, which relies on randomness. Therefore, this study employs an AM-estimator (S. Z. Li, Wang, and Soh 1998), which is more reliable and faster than RANSAC, as shown in Equation (7)(7) $\underset{R_k}{\mathit{argmin}}\sum _{\mathit{p}=1}^{{\mathit{N}}_{\mathit{cv}\_\mathit{sI}}}\mathit{\rho }\left(||{\vec{b}}_{\mathit{p}}-R_k{\vec{a}}_{\mathit{p}}||\right)$(7) , to estimate the rotation matrix. Here, $\mathit{\rho }\left(\cdot \right)$ represents the cost function. Rather than directly solving this problem, it can be implemented as an iteratively reweighted least-squares (IRLS) problem, as shown in Equation (8)(8) $\underset{R_k^i}{a\mathrm{rgmin}}\sum _{\mathit{p}=1}^{{\mathit{N}}_{\mathit{cv}\_\mathit{sI}}}\mathit{w}\left(r_p^{i-1}\right)||{\vec{b}}_{\mathit{p}}-R_k^i{\vec{a}}_{\mathit{p}}|{|}^2$(8) . In this context, $\mathit{w}\left(\cdot \right)$ is the weight function, and $r_p^{i-1}=\parallel {\vec{b}}_{\mathit{p}}-{\tilde{R}}_{\mathit{k}}^{\mathit{i}-1}{\vec{a}}_{\mathit{p}}\parallel$ represents the residual calculated using the rotation matrix estimated at $\left(\mathit{i}-1\right)$^th iteration. The iterations are continued until convergence occurs.

(7) $\underset{R_k}{\mathit{argmin}}\sum _{\mathit{p}=1}^{{\mathit{N}}_{\mathit{cv}\_\mathit{sI}}}\mathit{\rho }\left(||{\vec{b}}_{\mathit{p}}-R_k{\vec{a}}_{\mathit{p}}||\right)$(7)

(8) $\underset{R_k^i}{a\mathrm{rgmin}}\sum _{\mathit{p}=1}^{{\mathit{N}}_{\mathit{cv}\_\mathit{sI}}}\mathit{w}\left(r_p^{i-1}\right)||{\vec{b}}_{\mathit{p}}-R_k^i{\vec{a}}_{\mathit{p}}|{|}^2$(8)

The cost and weight functions of the widely employed AM-estimator are listed in Table 1, where $\mathit{u}$ is the scale parameter governing the shape of the cost and weight functions, and $\mathit{r}$ is the residual calculated using the rotation matrix estimated in the previous iteration. Figure 3 illustrates the shape of the function corresponding to different values of $\mathit{u}$. As $\mathit{u}$ gradually decreased, the width of the weight function narrowed. The AM-estimator systematically reduced the value of $\mathit{u}$ at a fixed rate ($\mathit{u}=\mathit{u}/\mathit{d}$) during each iteration of the IRLS calculation. This approach steadily diminishes the influence of the outliers at each iteration, thereby facilitating a robust estimation of the rotation matrix. Here, $\mathit{d}$ represents the rate at which the value of $\mathit{u}$ is reduced at each iteration of the AM-estimator. Consequently, a higher $\mathit{d}$ value leads to a more aggressive reduction in the width of the weight function, resulting in the assignment of smaller weights to data considered outliers. However, overly aggressive narrowing of the graph width of the weight function may also assign small weights to inliers. Therefore, it is essential to determine the appropriate magnitude of parameter $\mathit{d}$. In this study, the optimal cost function type and $\mathit{d}$ value were determined through simulations, and the detailed results are presented in Section 2.2.5.

Figure 3. Shape of the cost and weight functions according to $\mathit{u}$.

Table 1. Cost and weight functions of the AM-estimator.

	Cost function	Weight function
Fair	$u^2\left(\frac{\left|\mathit{r}\right|}{u}-log(1+\frac{\left|\mathit{r}\right|}{u})\right)$	$\frac{1}{1+\left|\mathit{r}\right|/u}$
Cauchy	$\frac{u^2}{2}log\left(1+(\mathit{r}/\mathit{u}{)}^2\right)$	$\frac{1}{1+{(\mathit{r}/u)}^2}$
Welsch	$\frac{u^2}{2}\left(1-\mathit{exp}(-{(\mathit{r}/\mathit{u})}^2)\right)$	$exp\left(-(\mathit{r}/\mathit{u}{)}^2\right)$
Tukey $\left\{\begin{array}{c}\mathit{if}\left|\mathit{r}\right|\le \mathit{u}\\ \mathit{if}\left|\mathit{r}\right|>\mathit{u}\end{array}\right.$	$\left\{\begin{array}{c}\frac{u^2}{6}\left(1-(1-(\mathit{r}/\mathit{u}{)}^2{)}^3\right)\\ u^2/6\end{array}\right.$	$\left\{\begin{array}{c}{(1-{(\mathit{r}/u)}^2)}^2\\ 0\end{array}\right.$

2.2.4. Translation vector estimation

The translation vector was estimated using the one-point RANSAC method, following the approach adopted by J. Li, Hu, and Ai (2021). In the final iteration of the rotation matrix estimation process, the correspondence vectors with residuals smaller than $\mathit{u}$ can be identified and considered as a new set of inliers (${\widetilde{\mathit{CV}}}_{\mathit{rI}}=\{\left({\vec{a}}_{\mathit{p}},{\vec{b}}_{\mathit{p}}\right){\}}_1^{{\mathit{N}}_{\mathit{cv}\_\mathit{rI}}}$). As the correspondence vector consists of a pair of correspondence points, an inlier set of correspondence points (${\widetilde{\mathit{CP}}}_{\mathit{rI}}=\{\left(a_i,b_i\right){\}}_1^{{\mathit{N}}_{\mathit{rI}}}$) can be derived from ${\widetilde{\mathit{CV}}}_{\mathit{rI}}$. Given the substantial removal of outliers in the preceding two steps, the one-point RANSAC facilitated a robust estimation of the translation vector.

2.2.5. Synthetic data experiments

Experiments were conducted using synthetic data to identify the optimal cost function type and $\mathit{d}$ value. The source correspondence points were generated by creating randomly distributed N ($\mathit{number}\mathit{of}\mathit{inliers}/\left(1-\mathit{the}\mathit{rate}\mathit{of}\mathit{outliers}\right)$) points within cubes ${\left[-\mathit{N},\mathit{N}\right]}^3$. Random $\mathit{\kappa }$ of the rotation matrix in the interval $\left[-\mathit{\pi },\mathit{\pi }\right]$ and the random translation vector within the cube ${\left[-\mathit{N},\mathit{N}\right]}^3$ were applied to the inliers of the source correspondence points to derive the corresponding inliers of the target correspondence points. Subsequently, outliers of the target correspondence points were generated by creating additional $\left(\mathit{N}-\mathit{number}\mathit{of}\mathit{inliers}\right)$ randomly distributed points within the cube ${\left[-\mathit{N},\mathit{N}\right]}^3$ and applying the previously generated random $\mathit{\kappa }$ of the rotation matrix and random translation vector. We then added zero-mean Gaussian noise with an isotropic covariance to the source and target correspondence points. The number of inliers was set to 10. We gradually increased the proportion of outliers and ascertained the success rate of PCR. For every outlier ratio, we conducted 1000 simulations and calculated the PCR success rates ($\left(\left(\mathit{Number}\mathit{of}\mathit{successes}\right)/1000\right)\times 100$).

The PCR success rates according to the cost function type are listed in Table 2. As depicted in Figure 3, the width of the weight function diminishes more significantly in the sequence of Fair, Cauchy, Welsch, and Tukey as the value of $\mathit{u}$ decreases. The simulation results indicated that, for Welsch and Tukey, the PCR success rate significantly decreased when the outlier ratio exceeded 97%. Similarly, the PCR success rate of the Fair cost function decreased significantly when the outlier ratio exceeded 99.5%. In contrast, the Cauchy function maintained a PCR success rate exceeding 99%, even with an outlier ratio of 99.98%. Therefore, we conclude that the Cauchy function exhibits the most suitable width reduction rate for estimating a 1DOF rotation matrix in the BEWB algorithm.

Table 2. PCR success rate according to cost function type and according to $\mathit{d}$ with Cauchy cost function.

Outlier ratio	Success rate according to cost function type $\left((\left(\mathit{Number}\mathit{of}\mathit{successes}\right)/1000\right)\times 100)$ (%)				Success rate according to $\mathit{d}$ with Cauchy cost function $\left((\left(\mathit{Number}\mathit{of}\mathit{successes}\right)/1000\right)\times 100)$ (%)
	Fair	Cauchy	Welsch	Tukey	$\mathit{d}$=1.1	$\mathit{d}$=1.2	$\mathit{d}$=1.3	$\mathit{d}$=1.4	$\mathit{d}$=1.5	$\mathit{d}$=1.6
99.98%	56.1	99.1	0	0	24.5	99.1	99.3	97.9	97.8	97.8
99.50%	99.9	100	1.7	2.5	100	100	100	100	100	100
99.00%	99.5	100	20.7	24.3	100	100	100	100	100	100
97.00%	100	100	91.2	91.8	100	100	100	100	100	100

The PCR success rates according to $\mathit{d}$ with the Cauchy cost function are shown in Table 2. The experiments were conducted by incrementally increasing $\mathit{d}$ from 1.1 to 1.6 in steps of 0.1 while monitoring the success rate of the PCR. When the outlier ratio was 99.50% or lower, a 100% PCR success rate was observed regardless of the $\mathit{d}$. However, when the outlier ratio exceeded 99.98%, a success rate of 24.5% was observed at $\mathit{d}$ of 1.1. As $\mathit{d}$ increased, the PCR success rate also increased, peaking at 99.3% for a $\mathit{d}$ of 1.3. However, from 1.4 onwards, the PCR success rate started to decline. Therefore, based on the experimental results, we set the value of $\mathit{d}$ for the BEWB algorithm to be 1.3, which yielded the highest PCR success rate.

2.3. Fine registration using BOB algorithm

Point clouds in urban scenes often contain various types of noise, including clutter objects. Moreover, the point clouds obtained from the different sensors exhibit differences in point density and precision. Consequently, ICP-type algorithms struggle to achieve an accurate alignment of key structures, such as roads and building façades, during the fine registration of cross-source urban scene point clouds. To address this challenge, we propose the BOB algorithm, which utilizes building outlines and points corresponding to the ground of point clouds to achieve precise registration of urban point clouds acquired from multiple sensors. A flowchart of the BOB algorithm is shown in Figure 4. Splitting the 3D registration problem into 2D (XY-plane) and 1D (Z-axis) allows the ICP algorithm within each step of BOB to establish the correct corresponding points.

Figure 4. BOB algorithm flowchart.

2.3.1. Building outline extraction and estimating XY-plane transformation matrix

The building exterior walls were extracted using the method employed in the BEWB algorithm to generate the building outlines. Subsequently, we set the Z-axis values of the extracted building exterior wall points to zero, creating a 2D representation of the building outline. We applied an SOR algorithm to reduce noise in the point cloud at each intermediate step. Subsequently, the ICP algorithm was applied to the 2D building outline point clouds to estimate the transformation matrix for the XY-plane.

2.3.2. Z-axis offset compensation

Even after transforming the source point cloud onto the XY-plane, the Z-axis offset persisted. Hence, we extracted points representing the ground surface with slopes of less than 30${ }^{\circ }$ and utilized them for the Z-axis offset correction. By employing a simple morphological filter (SMRF) algorithm (Pingel, Clarke, and McBride 2013), which is a widely used point cloud ground extraction technique, we identified points corresponding to the ground surface. However, utilizing the SMRF algorithm alone for ground extraction can result in the inclusion of nonground parts, such as the lower sections of buildings. To extract the actual ground surface exclusively, we filtered the SMRF results using a normal vector to extract points with slopes of less than 30${ }^{\circ }$, and noise removal was performed using the SOR method.

This approach excludes inaccurately extracted lower parts of buildings using the SMRF algorithm. In addition, it enables the extraction of pristine ground surfaces with slopes of less than 30${ }^{\circ }$ from both the source and reference point clouds. This also reduces the number of points utilized in the Z-axis offset correction calculations, decreasing the computational workload. Because the goal is solely to correct the Z-offset, we modified the ICP algorithm to create a constrained ICP algorithm that corrects only the Z-axis values of the source points. Finally, we registered the two point clouds using the Z-offset value estimated using the constrained ICP.

3. Experiments

3.1. Study area

Yonsei University, Seoul, Korea, was selected as the study area. The study area encompasses approximately 0.2 km², and the area features elevations ranging from 30 to 50 m. It includes both fixed-ground objects, such as buildings, playgrounds, roads, and trees, and non-fixed objects, such as people and vehicles. The tallest buildings within this area have 10 floors, and the spacing between the buildings ranges from 10 to 30 m. Vehicular roads cut through the central part of the study area.

3.2. Datasets

3.2.1. MMS point cloud

The MMS point cloud was acquired on 15 September 2020, along the road cut through the study area using a Leica Pegasus: Two. The geolocation accuracy of this system is 2 and 1.5 cm for the horizontal and vertical root mean square errors (RMSE), respectively, under open-sky conditions (Elhashash, Albanwan, and Qin 2022). The geolocation accuracy of the MMS point cloud was assessed using seven checkpoints (CKPs), resulting in an RMSE of 0.058 m. Further refinement was performed using seven GCPs to enhance the geolocation accuracy, resulting in an RMSE of 0.039 m. The refinement process using GCPs improved the MMS geolocation accuracy by approximately 2 cm. Both GCPs and CKPs were acquired using a Leica GS18T GNSS network real-time kinematic (RTK) rover, with horizontal and vertical accuracy of the RTK mode of 8 and 15 mm, respectively (Czaplewski et al. 2020). The distributions of the GCPs and CKPs are shown in Figure 5. The number of points and the point density of the MMS point cloud are 111,299,614 points and 7,816 points/m², respectively.

Figure 5. MMS point cloud and distribution of the GCPs and CKPs.

3.2.2. UAV-based photogrammetry point cloud

In UAV-based photogrammetry, combining nadir and oblique images has been proven effective in achieving comprehensive building reconstruction (Vacca, Dessì, and Sacco 2017). A total of 542 UAV images were captured on 24 February 2020, in nadir view, and 247 UAV images were captured on 8 February 2019, from an oblique view. The nadir images were captured using a Firefly 6 Pro, a fixed-wing UAV with a Sony Alpha ILCE-A6000 camera. Oblique images were obtained at a 55° angle relative to the vertical axis using a DJI Phantom 4 Pro, a rotary-wing UAV with a 1” CMOS sensor. The generation of a UAV-based photogrammetry point cloud employs the Pix4D Mapper program, and direct georeferencing was performed without GCPs. Figure 6 illustrates the point cloud generated through UAV-based photogrammetry and the distribution of the CKPs employed for geolocation accuracy assessment after PCR. The collection of CKPs was conducted using a Leica GS18T GNSS RTK rover for the precise assessment of geolocation accuracy. The number of points and the point density of the UAV-based photogrammetry point cloud are 32,805,778 points and 150 points/m², respectively.

Figure 6. UAV-based photogrammetry point cloud and CKP distribution.

3.2.3. TLS point clouds

TLS point clouds, focusing on five specific areas requiring refinement after the registration of the MMS and UAV-based photogrammetry point clouds, were obtained using a Leica BLK360 imaging laser scanner. Each area consisted of 6–13 stations, and the stations within each area were meticulously registered using Leica Cyclone software. Figure 7 shows the five TSL point clouds and the distribution of CKPs. CKPs were collected using a Leica GS18T GNSS RTK rover for a precise geolocation accuracy assessment. The number of points and point densities of the TLS point clouds are summarized in Table 3.

Figure 7. TLS point clouds and CKP distribution.

Table 3. Number of points and point densities of TLS point clouds.

	TLS#1	TLS#2	TLS#3	TLS#4	TLS#5
Number of points (points)	126,855,642	167,293,720	109,311,406	206,923,032	466,657,939
Point density (points/m^2)	107,546	89,023	111,806	107,568	335,545
Acquisition date	19 Nov. 2022				4 Jul. 2023

3.3. MMS point cloud and UAV-based photogrammetry point cloud registration

The initial coarse alignment of the MMS and UAV-based photogrammetry point clouds utilized the GNSS information gathered from both platforms. Consequently, our attention was directed exclusively toward fine registration. We performed fine registration on the two point clouds using the BOB algorithm and compared the results with those obtained using the ICP, point-to-plane ICP, and robust ICP algorithms. However, the BOB algorithm cannot rectify errors in the Z-axis orientation. To address this limitation, we applied the ICP algorithm in conjunction with the alignment achieved by the BOB algorithm and compared the results with those of other methods. Because the two point clouds had already undergone fine registration using the BOB algorithm, we constrained the maximum distance for correspondence points in the ICP to 1 m.

Figure 8 shows the building outlines after fine registration. The blue lines represent the building outlines of the reference MMS point cloud, whereas the yellow lines depict the building outlines of the UAV-based photogrammetry point cloud before the fine PCR. The building outlines of the UAV-based photogrammetry point clouds after applying the BOB or BOB+ICP algorithm were aligned precisely with those of the MMS point cloud. However, ICP, point-to-plane ICP, and robust ICP algorithms faced challenges in achieving precise alignment with the building outlines of the MMS point cloud. In particular, the ICP and point-to-plane ICP algorithms resulted in building outlines that were further offset from the reference than before fine PCR.

Figure 8. Building outlines of the UAV-based photogrammetry point clouds after the fine registration processes.

Figure 9 presents the road and building cross-sections following the fine registration process. All methods displayed improved vertical alignment with the reference MMS point cloud compared to that before the fine PCR. Specifically, UAV-based photogrammetry point clouds registered using BOB+ICP or robust ICP achieved precise vertical alignment with the MMS point clouds. The application of the BOB algorithm achieved an accurate vertical alignment with the MMS point cloud in certain areas, while displaying slight discrepancies in others. These discrepancies can be attributed to errors in the gravity direction provided by the sensor.

Figure 9. Road and building cross-sections of the UAV-based photogrammetry point clouds after the fine registration processes.

Multiscale model-to-model cloud comparison (M3C2) is a point cloud-based state-of-the-art 3D change-detection algorithm that calculates the orthogonal distance between a pair of point clouds (Lague, Brodu, and Leroux 2013; Winiwarter, Anders, and Höfle 2021). In the case of precisely aligned pairs of point clouds, M3C2 distances near zero are observed for stationary objects such as roads or buildings. Consequently, we employed the M3C2 distance to analyze the registration results thoroughly. Figure 10 illustrates the M3C2 distance between the MMS point cloud and UAV-based photogrammetry point clouds. Similar to prior assessments, registration using the BOB or BOB+ICP algorithm yielded M3C2 distances close to zero for the building exterior walls, whereas registration using BOB+ICP or robust ICP resulted in M3C2 distances close to zero for the ground surface.

Figure 10. M3C2 distances between the MMS and UAV-based photogrammetry point clouds.

The RMSE values for the absolute geolocations of the UAV-based photogrammetry point clouds calculated using the 12 CKPs indicated in Figure 6 are listed in Table 4. Notably, BOB+ICP stood out as the top performer in terms of absolute geolocation accuracy, followed by the standalone BOB algorithm. Although the robust ICP demonstrated superior vertical geolocation accuracy, its horizontal geolocation accuracy was lower than that of the BOB algorithm, resulting in diminished absolute geolocation accuracy compared to the latter. Moreover, the absolute geolocation accuracy of point-to-plane ICP deteriorated even further compared to that before fine PCR.

Table 4. Geolocation RMSE calculated by CKPs of UAV-based photogrammetry point clouds after fine PCR.

	Before fine PCR		BOB		BOB+ICP		ICP		Point-to-plane ICP		Robust ICP
	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)
P-CKP #1	1.66	5.44	0.22	0.41	0.10	0.13	2.45	0.44	13.07	0.04	0.96	0.12
P-CKP #2	1.49	5.53	0.13	0.53	0.14	0.04	2.61	0.08	13.64	0.79	1.00	0.15
P-CKP #3	1.91	4.62	0.28	0.38	0.22	0.12	2.07	0.85	10.35	0.80	0.60	0.03
P-CKP #4	2.17	4.42	0.31	0.61	0.27	0.18	2.10	0.98	9.97	1.05	0.54	0.02
P-CKP #5	1.61	4.97	0.22	0.03	0.17	0.02	2.17	0.66	9.38	0.40	0.69	0.08
P-CKP #6	1.57	5.53	0.06	0.51	0.10	0.15	2.76	0.26	10.66	0.34	1.03	0.10
P-CKP #7	1.49	5.42	0.11	0.45	0.05	0.17	2.73	0.29	8.58	0.34	0.93	0.06
P-CKP #8	1.42	5.47	0.06	0.47	0.11	0.14	2.78	0.16	8.78	0.54	1.01	0.02
P-CKP #9	1.71	4.97	0.03	0.03	0.08	0.09	2.61	0.63	5.33	0.26	0.88	0.11
P-CKP #10	1.79	4.54	0.14	0.46	0.06	0.02	2.37	0.88	3.16	0.74	0.66	0.14
P-CKP #11	1.75	4.71	0.10	0.29	0.16	0.04	2.90	0.44	5.19	0.01	0.87	0.07
P-CKP #12	1.92	4.34	0.18	0.66	0.23	0.19	2.93	0.50	5.46	0.19	0.97	0.17
P-CKP#1–12	1.72	5.01	0.18	0.44	0.16	0.12	2.56	0.58	9.18	0.56	0.86	0.10
	5.30		0.48		0.20		2.62		9.20		0.87

3.4. MMS+UAV-based photogrammetry point cloud and TLS point clouds registration

3.4.1. Coarse registration

TLS point clouds do not possess an inherent initial alignment with the MMS+UAV-based photogrammetry point cloud because TLS point clouds are referenced in local coordinates. Therefore, we conducted a coarse registration between the TLS point clouds and MMS+UAV-based photogrammetry point cloud using the BEWB algorithm. We used an MMS+UAV-based photogrammetry point cloud registered using the BOB+ICP algorithm with the highest absolute geolocation. The registration results of the BEWB algorithm were compared to those obtained using Li et al.’s algorithm (J. Li, Hu, and Ai 2021) and TEASER++ (H. Yang, Shi, and Carlone 2021).

To establish the putative correspondence points employed as inputs for Li et al. s algorithm and TEASER++, we replaced keypoint detection with a process involving downsampling of the source and reference point clouds with a voxel size of 1 m. Furthermore, we utilized the FPFH algorithm to describe the keypoints. Equations (9)(9) $e_t=||t_{\mathit{GT}}-t_e||$(9) and (10(10) $e_R=\mathit{arccos}\left(\frac{\mathit{tr}({\mathit{R}}_{\mathit{GT}}{(R_e)}^{\mathit{T}}-1}{2}\right)$(10) ) were employed as metrics for quantitative assessments, where $e_t$ represents the translation error, $t_{\mathit{GT}}$ is the ground-truth translation vector, and $t_e$ is the estimated translation vector obtained using the algorithm. $e_R$ denotes the rotation error, $R_{\mathit{GT}}$ is the ground-truth rotation matrix, and $R_e$ is the estimated rotation matrix derived from the algorithm. Figure 11 presents the coarse PCR results of each algorithm for the five TLS point clouds and MMS+UAV-based photogrammetry point cloud. Table 5 presents the coarse PCR success rates based on the permissible error thresholds.

(9) $e_t=||t_{\mathit{GT}}-t_e||$(9)

(10) $e_R=\mathit{arccos}\left(\frac{\mathit{tr}({\mathit{R}}_{\mathit{GT}}{(R_e)}^{\mathit{T}}-1}{2}\right)$(10)

Figure 11. Coarse PCR results of each algorithm for the five TLS point clouds and MMS+UAV-based photogrammetry point cloud. The TLS point clouds are tinted blue to differentiate them from the MMS+UAV-based photogrammetry point cloud.

Table 5. Success rates of coarse PCR.

	Success rates
	$e_R<1^{\circ }$; $e_t<2\mathit{m}$	$e_R<3^{\circ }$; $e_t<5\mathit{m}$	$e_R<5^{\circ }$; $e_t<7\mathit{m}$
Li et al.s algorithm	0% (0/5)	0% (0/5)	0% (0/5)
TEASER++	20% (1/5)	60% (3/5)	80% (4/5)
BEWB	80% (4/5)	100% (5/5)	100% (5/5)

Experimental results demonstrated that the BEWB algorithm achieved the highest coarse PCR success rate. The algorithm of Li et al. failed in all the PCR, whereas TEASER++ achieved a PCR success rate of 80% when the permissible error thresholds were increased to $e_R<5^{\circ }$ and $e_t<7\mathit{m}$. However, TEASER++ consistently exhibited larger errors than the BEWB algorithm, ultimately leading to PCR failure for TLS#3.

3.4.2. Fine registration

After the coarse PCR using the BEWB algorithm, fine registration was performed using the BOB, ICP, point-to-plane ICP, robust ICP, and BOB+ICP algorithms. However, point-to-plane ICP and robust ICP failed to register TLS#1, TLS#2, TLS#3, and TLS#5 and even exacerbated the misalignment between the two point clouds. The point-to-plane ICP algorithm is particularly sensitive to the initial alignment, rendering it less robust than the point-to-point ICP (P. P. Li et al. 2020). Consequently, point-to-plane ICP and robust ICP based on point-to-plane ICP were considered unsuccessful in cases with relatively large errors in the coarse PCR, except for TLS#4, which exhibited the smallest error in the coarse PCR. Therefore, our analysis was conducted solely based on the results of fine registration using the BOB, ICP, and BOB+ICP algorithms.

Figure 12 shows the building outlines after the fine registration process. Compared with before fine PCR, all methods demonstrated improved horizontal alignment with the reference MMS+UAV-based photogrammetry point cloud. Following the application of the BOB+ICP algorithm, the building outlines of the TLS point clouds were aligned almost precisely with those of the reference MMS+UAV-based photogrammetry point cloud. The building outline of the pink TLS point cloud subjected to the ICP algorithm appeared slightly thicker. This phenomenon can be attributed to a misalignment in the gravity direction of the TLS point clouds resulting from the ICP process. In contrast, the thicknesses of the building outlines in the TLS point clouds remained nearly unchanged when the BOB+ICP algorithm was applied. This suggests that the accurate initial alignment obtained using the BOB algorithm prevents the ICP from tilting erroneously in the wrong gravity direction.

Figure 12. Building outlines of the TLS point clouds after the fine registration processes.

Figure 13 depicts the cross-sections of the ground surface and buildings following the fine registration processes, whereas Figure 14 illustrates the M3C2 distance between the MMS+UAV-based photogrammetry and TLS point clouds. The TLS point clouds subjected to the BOB+ICP algorithm exhibited precise vertical alignment with the reference MMS+UAV-based photogrammetry point cloud. The M3C2 distance also exhibited values close to zero for the exterior walls of the buildings and ground surfaces when registered using the BOB+ICP algorithm.

Figure 13. Ground surface and building cross-sections of the TLS point clouds after the fine registration processes.

Figure 14. M3C2 distances between the MMS+UAV-based photogrammetry point cloud and the TLS point clouds.

However, the TLS point cloud with only the BOB algorithm applied displayed some discrepancies in certain areas, as shown in Figure 14(b), with the M3C2 distance indicating values close to zero for the building exterior walls but varying for the ground surfaces. The TLS point clouds with only the ICP application exhibited substantial overall disparities with the reference point cloud, and the M3C2 distance was generally larger for both building exterior walls and ground surfaces than for the other two methods.

The RMSE values for the absolute geolocations of the TLS point clouds calculated using the 15 CKPs shown in Figure 7 are listed in Table 6. The absolute geolocation accuracies of the TLS point clouds registered using the BEWB+BOB+ICP and BEWB+BOB algorithms were both calculated to be 0.24 m. Although the BEWB+BOB+ICP algorithm produced the most accurate registration outcomes, the absolute geolocation accuracy of certain TLS point clouds was the highest when the BEWB+BEWB+BOB algorithm was applied. This observation can be attributed to the reference MMS+UAV-based photogrammetry point cloud with an RMSE of 0.2 m.

Table 6. Geolocation RMSE calculated by the CKPs of the TLS point clouds after fine PCR.

		BEWB		BEWB + BOB		BEWB + BOB+ICP		BEWB + ICP
		XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)
TLS#1	T-CKP#1	0.51	0.65	0.07	0.20	0.22	0.38	0.36	0.66
	T-CKP#2	0.03	0.63	0.10	0.19	0.19	0.23	0.22	0.39
	T-CKP#3	0.31	0.62	0.08	0.18	0.18	0.15	0.21	0.26
	T-CKP#1–3	0.35	0.64	0.08	0.19	0.19	0.27	0.27	0.47
		0.72		0.21		0.33		0.54
TLS#2	T-CKP#4	0.79	1.09	0.13	0.11	0.16	0.29	0.20	0.81
	T-CKP#5	0.51	1.18	0.10	0.19	0.16	0.27	0.19	0.71
	T-CKP#6	0.71	1.14	0.13	0.16	0.23	0.02	0.21	0.34
	T-CKP#4–6	0.68	1.14	0.12	0.16	0.19	0.23	0.20	0.66
		1.32		0.20		0.30		0.69
TLS#3	T-CKP#7	0.26	0.25	0.14	0.19	0.12	0.02	0.12	0.03
	T-CKP#8	0.18	0.28	0.15	0.22	0.12	0.09	0.09	0.14
	T-CKP#9	0.26	0.23	0.19	0.16	0.17	0.06	0.25	0.09
	T-CKP #7–9	0.24	0.26	0.16	0.19	0.14	0.06	0.17	0.10
		0.35		0.25		0.15		0.20
TLS#4	T-CKP#10	0.31	0.23	0.20	0.04	0.02	0.09	0.32	0.26
	T-CKP#11	0.32	0.29	0.20	0.03	0.03	0.15	0.36	0.19
	T-CKP#12	0.39	0.26	0.25	0.01	0.09	0.11	0.41	0.16
	T-CKP#10–12	0.34	0.26	0.22	0.03	0.06	0.12	0.37	0.21
		0.43		0.22		0.13		0.42
TLS#5	T-CKP#13	0.69	0.94	0.32	0.04	0.21	0.02	0.87	0.53
	T-CKP#14	0.65	1.00	0.28	0.02	0.20	0.09	0.86	0.72
	T-CKP#15	0.56	0.95	0.32	0.03	0.24	0.06	0.89	0.78
	T-CKP#13–15	0.64	0.96	0.31	0.03	0.22	0.07	0.87	0.68
		1.15		0.31		0.23		1.11
T-CKP#1–15		0.48	0.74	0.19	0.14	0.17	0.17	0.46	0.48
		0.89		0.24		0.24		0.66

4. Discussion

This study introduces a multi-source point cloud registration method tailored for urban environments, employing a coarse-to-fine approach to achieve a comprehensive 3D reconstruction of the urban landscape. We successfully registered the MMS, UAV-based photogrammetry, and TLS point clouds using the proposed method. The results of these registrations are shown in Figure 15. Figure 16 compares UAV-based photogrammetry and MMS + UAV-based photogrammetry + TLS point cloud. The outcome of registering the point clouds obtained from the three sensors demonstrated a substantial improvement in point density compared to the UAV-based photogrammetry point cloud, as shown in Figure 16.

Figure 15. Complete view of the MMS + UAV-based photogrammetry + TLS point cloud.

Figure 16. Comparison of the UAV-based photogrammetry point cloud and the MMS + UAV-based photogrammetry + TLS point cloud.

To verify the generality of the proposed algorithms, we conducted additional experiments on supplementary datasets representing typical urban areas, comprising MMS point cloud, UAV-based photogrammetry point cloud, and TLS point clouds. The results from these additional experiments, documented in Appendix A(see Appendix Figure A1, Figure A2, Figure A3, Figure A4, Figure A5, Figure A6, Figure A7, Table A1, Table A2, Table A3, Table A4), are consistent with those presented in Section 3.

The applicability of the proposed methodology to the 3D reconstruction of an entire city is a crucial issue. However, applying the proposed methodology to an entire city is a challenging owing to data collection efficiency. In this methodology, comprehensive urban scenes are acquired via UAV-based photogrammetry. UAVs are constrained in coverage because of issues such as battery performance, making it challenging to map extensive areas with limited costs and time. TLS surveying also requires significant time and costs. Therefore, utilizing conventional aerial photogrammetry with manned aircraft to initially construct a 3D representation of the entire urban landscape, followed by the sequential application of this methodology to regions requiring enhanced precision, point density, and geolocation accuracy, could be a viable approach. To more effectively utilize this methodology in the 3D reconstruction of an entire city, future research needs to develop a framework to identify and prioritize areas within digital twins where high-quality 3D models are necessary.

Another critical issue is whether the proposed algorithm can be applied to complex urban areas, such as when the spacing between buildings is narrow. Because the proposed algorithms execute registration by extracting points corresponding to exterior walls, the presence of several overlapping exterior walls between two point clouds becomes a crucial factor influencing the success of the algorithms. To capture exterior walls using an MMS, it is crucial to ensure the presence of accessible roads for vehicles around the building. UAV-based photogrammetry can better capture the exterior walls of buildings by lowering the UAV altitude and utilizing oblique images captured from four directions: east, west, south, and north (Lee et al. 2024). The TLS can install equipment even in narrow spaces between buildings, allowing the capture of exterior walls without the issue. However, in cases where the space between buildings is narrow, the survey range achievable from one station is limited, necessitating surveys to be conducted at multiple stations. Given that surveying with TLS may not be efficient in narrow spacings between buildings, one potential approach is to acquire a point cloud using a backpack-mounted MMS (Bao et al. 2022).

With the rapid progress in UAV LiDAR systems, high-density UAV LiDAR point clouds have emerged as potent data sources for urban 3D modeling (B. B. Yang and Li 2022). Therefore, to further verify the generality of the proposed algorithms, the registration results of the UAV LiDAR point cloud and TLS point clouds are added to the Appendix B (see Appendix Figure B1, Figure B2, Figure B3, Figure B4, Table B1, Table B2, Table B3). Despite the partial occlusion of the exterior walls of the buildings in the UAV LiDAR point cloud, coarse registration proved successful in the majority of cases (see Appendix Table B2). However, in cases where only one overlapping exterior wall exists between the UAV LiDAR point cloud and TLS point cloud, and substantial occlusion is present on that exterior wall in the UAV LiDAR data (registration between UAV LiDAR point cloud and TLS3_part3), the registration process failed. As mentioned in the previous paragraph, the proposed algorithms perform registration by extracting points corresponding to exterior walls. Therefore, it is reiterated that the presence of several overlapping exterior walls between two point clouds is a crucial factor in determining the success of the algorithms.

The main contributions of this study are as follows:

1. We proposed a coarse registration algorithm called the BEWB algorithm, which is tailored to coarsely register urban scene point clouds acquired from multiple sensors. As a point-based algorithm, the BEWB algorithm exhibits remarkable flexibility in handling diverse point cloud sources. It coarsely registers urban point clouds by extracting exterior walls of buildings, establishing correspondence points, and efficiently removing outliers within the set of correspondence points.

2. We proposed a fine registration algorithm called the BOB algorithm, which is tailored to precisely register urban scene point clouds acquired from multiple sensors. The BOB algorithm achieved precise registration in cross-source urban point clouds by leveraging building outlines and points corresponding to the ground of point clouds.

3. We proposed a methodology for generating a comprehensive urban scene point cloud with enhanced precision, higher point density, and superior geolocation accuracy. This was achieved by registering MMS, UAV-based photogrammetry, and TLS point clouds. With the growing prominence of the autonomous driving industry, accessibility to accurately georeferenced MMS point clouds is expected to increase significantly. We leveraged this by registering the UAV-based photogrammetry point cloud with the MMS point cloud, thereby improving geolocation accuracy. In addition, for areas requiring detailed 3D reconstruction, the registration of the TLS point clouds with the MMS+UAV-based photogrammetry point cloud enhanced the detail and comprehensiveness of the point cloud.

The limitations of this study are as follows. First, both the BEWB and BOB algorithms are applicable only to 4DOF transformation problems. They operate under the assumption that the Z-axis orientation in the point cloud is accurate and extract the points corresponding to the exterior walls of buildings based on this assumption. Consequently, their applicability is constrained when a significant error exists in the Z-axis orientation of point clouds. However, minor errors in the Z-axis can be rectified by applying the proposed algorithms, followed by the application of ICP, as demonstrated in our experiments.

Second, the scope of the BEWB and BOB algorithms is confined to areas with buildings, relying on information from the exterior walls of buildings for point cloud registration. This implies that these algorithms are unsuitable for nonurban or open landscapes. For both BEWB and BOB, the presence of several overlapping exterior walls, even with some occlusion, between the two point clouds is essential for successful registration.

Finally, as shown in Figure 16(a), the UAV-based photogrammetry point cloud had a relatively low point density, with an interpoint spacing of 15–20 cm. Despite registering it with the MMS point cloud, the enhancement of the geolocation accuracy of the UAV-based photogrammetry point cloud was limited. Consequently, achieving a substantial improvement in the geolocation accuracy of the registered TLS point clouds was hindered. To address this issue, future studies should explore methods for obtaining a more precise and denser UAV-based photogrammetry point cloud involving approaches such as lowering UAV altitudes or using advanced optical sensors.

5. Conclusion

This study introduced a coarse-to-fine approach with BEWB and BOB algorithms, designed to register urban scene point clouds acquired from multiple sensors. These algorithms capitalize on the inherent gravity direction and absolute scale provided by commonly used sensors in urban mapping, such as MMS, UAV-based photogrammetry, and TLS. By recognizing the significance of building exterior walls and ground surfaces in shaping urban landscapes, we utilized these elements for precise registration.

To assess the performance of the proposed algorithms, experiments were conducted involving the registration of UAV-based photogrammetry point cloud with MMS point cloud and subsequent registration of TLS point clouds. The experimental results demonstrate that the proposed algorithms outperform existing methods, achieving a more precise registration of urban scene point clouds. Furthermore, this approach facilitated the generation of a comprehensive urban scene point cloud with improved precision, higher point density, and superior geolocation accuracy.

Acknowledgments

We would like to thank Editage (www.editage.co.kr) for the English language editing.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data are not publicly available because they are only accessible to Korean citizens, owing to legal restrictions.

Supplementary Material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/15481603.2024.2341557

Additional information

Funding

This work is supported by the Korea Agency for Infrastructure Technology Advancement [KAIA] grant R&D program of Digital Land Information Technology Development funded by the Ministry of Land, Infrastructure and Transport (MOLIT) [Grant RS-2022-00142501].

Appendix A.

Additional experiments on the registration of MMS point cloud, UAV-based photogrammetry point cloud, and TLS point clouds

1. Datasets

Figure A1. MMS point cloud, UAV-based photogrammetry point cloud, and CKPs distribution: the MMS point cloud, captured by a Leica Pegasus: Two, underwent refinement utilizing five GCPs. The achieved horizontal and vertical geolocation accuracy is 3.2 cm and 3.6 cm, respectively. A UAV-based photogrammetry point cloud was generated using nadir images captured by a Firefly 6 Pro, Sony Alpha ILCE-A6000. The collection of CKPs was conducted using a Leica GS18T GNSS RTK rover.

Figure A2. TLS point clouds, and CKPs distribution: TLS point clouds were captured using a Leica BLK360 imaging laser scanner. Each area consisted of 7–14 stations and registered using Leica Cyclone software. The collection of CKPs was conducted using a Leica GS18T GNSS RTK rover.

Table A1. Number of points and point densities of MMS point cloud, UAV-based photogrammetry point cloud and TLS point clouds.

	MMS	UAV-based photogrammetry	TLS2 #1	TLS2 #2	TLS2 #3
Number of points (points)	44,262,396	25,536,999	533,661,465	273,063,027	209,390,111
Point density (points/m^2)	2,574	269	127,973	387,398	117,032
Acquisition date	15 Sep. 2020	24 Feb. 2020	14 Mar. 2024		18 Mar. 2024

2.

2. Fine registration between MMS point cloud and UAV-based photogrammetry point cloud

Figure A3. Building outlines of the MMS point cloud and UAV-based photogrammetry point cloud after the fine registration processes.

Figure A4. Road and building cross-sections of the MMS point cloud and UAV-based photogrammetry point cloud after the fine registration processes.

Table A2. Geolocation RMSE calculated by the CKPs of the UAV-based photogrammetry point cloud after fine PCR.

	Before fine PCR		BOB		BOB+ICP		ICP		Point-to-plane ICP		Robust ICP
	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)
P2-CKP #1	1.14	3.80	0.11	0.11	0.07	0.31	0.82	0.29	5.69	0.00	0.62	0.40
P2-CKP #2	1.18	3.79	0.16	0.12	0.13	0.23	0.82	0.21	5.70	0.06	0.60	0.32
P2-CKP #3	0.97	2.63	0.08	1.28	0.12	0.02	2.09	0.11	5.83	0.14	0.71	0.03
P2-CKP #4	1.34	4.45	0.07	0.54	0.05	0.12	1.86	0.15	6.36	0.23	0.94	0.17
P2-CKP #5	1.99	5.74	0.20	1.83	0.15	0.12	4.96	0.85	7.49	0.60	1.48	0.14
P2-CKP #6	2.07	5.68	0.23	1.77	0.12	0.17	5.04	0.88	7.59	0.63	1.58	0.19
P2-CKP #7	1.81	4.29	0.13	0.38	0.02	0.06	4.32	0.58	7.45	0.36	1.46	0.09
P2-CKP #8	1.77	2.96	0.30	0.95	0.09	0.11	3.82	0.20	7.33	0.04	1.43	0.07
P2-CKP #9	1.43	2.26	0.25	1.65	0.05	0.14	2.96	0.06	6.81	0.06	1.19	0.13
P2-CKP #10	1.24	2.06	0.26	1.85	0.06	0.16	2.75	0.06	6.42	0.18	1.02	0.15
P2-CKP #11	1.00	1.71	0.20	2.20	0.10	0.24	3.02	0.01	6.05	0.21	0.86	0.22
P2-CKP#1–11	1.50	3.81	0.19	1.36	0.09	0.17	3.27	0.43	6.65	0.31	1.14	0.20
	4.10		1.38		0.20		3.30		6.66		1.15

3.

3. Coarse and fine registration between MMS+UAV-based photogrammetry point cloud and TLS point clouds

Table A3. Results and success rates of coarse PCR between MMS+UAV-based photogrammetry point cloud and TLS point clouds.

	TLS2 #1		TLS2 #2		TLS2 #3		Success rates (${\mathit{e}}_{\mathit{R}}<3^{\circ }$; ${\mathit{e}}_{\mathit{t}}<5\mathit{m}$)
	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)
Li et al.’ algorithm	84.92	262.65	1.66	0.70	174.08	322.93	33.3% (1/3)
	Failure		Success		Failure
TEASER++	45.81	38.84	0.18	0.30	132.79	431.72	33.3% (1/3)
	Failure		Success		Failure
BEWB	1.60	4.36	0.19	0.46	1.51	1.34	100% (3/3)
	Success		Success		Success

Figure A5. Building outlines of the MMS+UAV-based photogrammetry point cloud and TLS point clouds after the fine registration processes.

Figure A6. Ground surface and building cross-sections of the MMS+UAV-based photogrammetry point cloud and TLS point clouds after the fine registration processes.

Table A4. Geolocation RMSE calculated by the CKPs of the TLS point clouds after fine PCR.

		BEWB		BEWB + BOB		BEWB + BOB+ICP		BEWB + ICP
		XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)
TLS2 #1	T2-CKP#1	0.61	3.57	0.12	0.34	0.09	0.20	0.29	0.78
	T2-CKP#2	0.88	3.60	0.14	0.30	0.11	0.19	0.32	0.80
	T2-CKP#3	2.20	3.76	0.01	0.14	0.04	0.12	0.31	0.38
	T2-CKP#1–3	1.41	3.65	0.11	0.27	0.08	0.17	0.31	0.68
		3.91		0.29		0.19		0.75
TLS2 #2	T2-CKP#4	0.44	0.48	0.18	0.12	0.07	0.11	0.07	0.22
	T2-CKP#5	0.32	0.49	0.19	0.14	0.11	0.11	0.21	0.23
	T2-CKP#6	0.30	0.50	0.17	0.14	0.09	0.11	0.22	0.22
	T2-CKP#4–6	0.36	0.49	0.18	0.13	0.09	0.11	0.18	0.22
		0.61		0.22		0.14		0.28
TLS2 #3	T2-CKP#7	0.95	0.31	0.09	0.09	0.06	0.01	0.68	0.43
	T2-CKP#8	0.79	0.08	0.08	0.14	0.04	0.01	0.54	0.05
	T2-CKP#9	0.76	0.33	0.08	0.11	0.04	0.05	0.62	0.61
	T2-CKP#7–9	0.84	0.27	0.09	0.12	0.04	0.03	0.62	0.43
		0.88		0.14		0.05		0.75
T2-CKP#1–9		0.97	2.13	0.13	0.19	0.08	0.12	0.41	0.48
		2.34		0.23		0.14		0.63

Figure A7. Complete view of the MMS + UAV-based photogrammetry + TLS point cloud.

Appendix B.

Additional experiments on the registration between UAV LiDAR point cloud and TLS point clouds

Figure B1. UAV LiDAR point cloud, TLS point clouds, and CKPs distribution: the UAV LiDAR point cloud, captured by a DJI Matrice 300 L1, was georeferenced using six GCPs. Horizontal and vertical geolocation accuracies are 5.6 cm and 2.8 cm, respectively. TLS point clouds were captured using a Leica BLK360 imaging laser scanner. TLS3 is comprised of 14 TLS stations registered using Leica Cyclone software. TLS3_part1–3 refers to one of the stations that constitute TLS3. The collection of CKPs was conducted using a Leica GS18T GNSS RTK rover.

Table B1. Number of points and point densities of UAV LiDAR point cloud and TLS point clouds.

	UAV LiDAR	TLS3	TLS3_part1	TLS3_part2	TLS3_part3
Number of points (points)	309,665,263	454,794,292	31,870,108	32,655,113	28,782,291
Point density (points/m^2)	4,591	437,861	413,063	427,885	377,296
Acquisition date	24 Oct. 2023	30 Nov. 2023

Table B2. Results and success rates of coarse PCR between UAV LiDAR point cloud and TLS point clouds.

	TLS3		TLS3_part1		TLS3_part2		TLS3_part3		Success rates (${\mathit{e}}_{\mathit{R}}<3^{\circ }$; ${\mathit{e}}_{\mathit{t}}<5\mathit{m}$)
	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)	${\mathit{e}}_{\mathit{R}}$ (°)	${\mathit{e}}_{\mathit{t}}$ (m)
Li et al.’ algorithm	177.55	75.74	160.69	63.58	150.21	68.20	174.51	80.08	0% (0/4)
	Failure		Failure		Failure		Failure
TEASER++	0.28	0.47	3.42	0.94	148.81	73.51	157.39	144.83	25% (1/4)
	Success		Failure		Failure		Failure
BEWB	0.13	3.28	0.49	0.80	1.32	0.66	129.32	169.04	75% (3/4)
	Success		Success		Success		Failure

Figure B2. Building outlines of the UAV LiDAR point cloud and TLS point cloud after the fine registration processes.

Figure B3. Ground surface and building cross-sections of the UAV LiDAR point cloud and TLS point cloud after the fine registration processes.

Table B3. Geolocation RMSE calculated by the CKPs of the TLS point clouds after fine PCR.

		BEWB		BEWB + BOB		BEWB + BOB+ICP		BEWB + ICP
		XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)	XY (m)	Z (m)
TLS3	T3-CKP#1	2.88	1.84	0.12	0.22	0.13	0.12	5.39	0.21
	T3-CKP#2	2.90	1.74	0.10	0.11	0.11	0.04	5.47	0.33
	T3-CKP#3	2.97	1.65	0.09	0.02	0.08	0.04	9.10	1.38
	T3-CKP#4	2.93	1.77	0.07	0.15	0.06	0.14	9.08	0.43
	T3-CKP#5	2.86	1.90	0.08	0.28	0.08	0.22	9.03	0.56
	T3-CKP#6	2.82	1.91	0.10	0.29	0.11	0.17	6.18	0.79
	T3-CKP#1–6	2.89	1.81	0.10	0.20	0.10	0.14	7.57	0.73
		3.41		0.22		0.17		7.61
TLS3_part1	T3-CKP#7	0.17	0.77	0.06	0.04	0.06	0.03	0.26	0.16
		0.78		0.07		0.07		0.31
TLS3_part2	T3-CKP#8	0.44	0.61	0.11	0.10	0.11	0.05	0.27	0.07
		0.75		0.15		0.12		0.28
T3-CKP#1–8		2.51	1.60	0.10	0.18	0.10	0.12	6.56	0.63
		2.98		0.20		0.15		6.59

Figure B4. Complete view of registration result between UAV LiDAR point cloud and TLS point cloud.