Point cloud semantic segmentation of grotto scenes using the knowledge-guided deep learning method

ABSTRACT

In recent years, the deep learning-based semantic segmentation for point clouds has demonstrated remarkable capabilities in processing 3D urban scenes for applications such as three-dimensional reconstruction, semantic modeling, and augmented reality. However, research on grottoes scenes is very limited. It is currently unclear how existing neural architectures for point cloud semantic segmentation perform in grotto scenes, and how to effectively incorporate the unique characteristics of grotto scenes to enhance the performance of deep neural networks. This study proposed a method for point cloud semantic segmentation of grotto scenes, combining knowledge with deep learning approaches. The method adopted knowledge to guide the creation of benchmark datasets, the design of a neural network called GSS-Net, and the correction of segmentation errors in the results of deep learning. The results show that the proposed method outperforms four existing mainstream models without the correction of segmentation results. Moreover, a set of ablation studies verified the effectiveness of each proposed module. This method not only improves the accuracy of point cloud semantic segmentation in grotto scenes but also enhances the interpretability of network designs. It provides new insights into the application of knowledge-guided deep learning models in grotto scenes.

KEYWORDS:

• Grotto scene
• cultural heritage
• point cloud
• semantic segmentation
• knowledge-guided deep learning

1. Introduction

Grottoes, as cultural treasures with immense historical and artistic significance, embody a wealth of historical details and cultural connotations (Li 2023; Runze 2023). Some are recognized as the World Heritage by the United Nations Educational, Scientific, and Cultural Organization (UNESCO), such as the Longmen Grottoes,¹ Yungang Grottoes,² and Dazu Rock Carvings.³ The intricate Buddha sculptures and delicate geometric patterns are carved into cliffs at these grottoes, narrating Buddhist tales, historical events, and slices of social life. Consequently, the three-dimensional (3D) digital documentation of these invaluable grottoes has become integral to cultural heritage preservation. Photogrammetry and 3D laser scanning technologies are extensively employed to capture the sophisticated geometry of the grottoes (Ahn and Wohn 2015; Diao et al. 2019; Keqina et al. 2008). However, unstructured point clouds consist of spatial coordinates with limited attributes (e.g. color and reflectivity), and are lack of semantic information for scene representation. This makes it difficult to be directly used for applications and services beyond visualization (Camuffo, Mari, and Milani 2022; Weinmann 2016). Semantic segmentation assigns meaningful semantic labels to each point, enabling the transformation of point clouds from visual formats into interpretable resources (Malinverni et al. 2019; Xie, Tian, and Zhu 2020; Yang, Hou, and Li 2023). Point cloud semantic segmentation has become a fundamental step in 3D reconstruction with semantics, scene interpretation and knowledge sharing (Colucci et al. 2021; Croce et al. 2020; Croce, Caroti, De Luca et al. 2021; Moyano et al. 2022). Furthermore, the semantic segmentation of basic geometric primitives in point clouds, coupled with the integration of relevant cultural heritage information, significantly enhances the expression of 3D models (Moyano et al. 2021; Poux, Neuville, Hallot et al. 2017; Poux, Neuville, Van Wersch et al. 2017). Therefore, semantic segmentation is essential in transforming raw point cloud data into semantically rich 3D models in the 3D reconstruction process (Apollonio, Gaiani, and Sun 2013; Croce, Caroti, Piemonte et al. 2021). A notable implementation is its application in Scan to Building Information Modeling (BIM) processes (Bosché et al. 2015; Brumana et al. 2020; Rocha et al. 2020).

In this study, the grotto scene is classified into ten classes, including cliff wall, cliff wall footing, niche eaves, Buddha statues, small niches, pedestals, steles, cement pavements, railings and low vegetation. Since mountains are blanketed by low vegetation, they are considered as the same class (Yang and Hou 2023; Yang, Hou, and Fan 2024). Due to the limited research, the semantic segmentation of grotto scenes still relies on manual processing, which is often time-consuming and labor-intensive. In recent years, data-driven deep learning methods have made remarkable advancements in point cloud semantic segmentation, particularly in large-scale urban and indoor scenes (Liu et al. 2019; Zhang et al. 2019; Zhang et al. 2023). To evaluate the performance of these methods, many point cloud benchmark datasets such as Semantic3D (Hackel et al. 2017), Toronto-3D (Tan et al. 2020), Stanford Large-Scale 3D Indoor Spaces (S3DIS) (Armeni et al. 2016), SemanticKITTI (Behley et al. 2019), and SensatUrban (Hu, Yang, Khalid et al. 2021) have been available to the public. Moreover, 3D semantic segmentation has been focused on specific urban sub-scenes and industrial scenarios, such as agricultural landscapes (Chen et al. 2021), bridge and railway infrastructures (Grandio et al. 2022; Lee, Park, and Ryu 2021; Xia, Yang, and Chen 2022), shield tunnels (Li et al. 2023), and architectural façades (Maru et al. 2023). In the field of cultural heritage, most of the point cloud semantic segmentation-related studies place focus on historical buildings (Grilli and Remondino 2019; Matrone et al. 2020; Morbidoni et al. 2020; Pierdicca et al. 2020). However, significant differences exist between the grotto scenes and the afore-mentioned scenes. In the grotto scene, geometric primitives with the same semantics vary in shape, such as the size, posture, and the appearance of different Buddha statues, which are often neglected by existing deep learning methods. Therefore, the performance of existing deep learning methods in the grotto scenes is unclear. Additionally, how to modify the existing network modules to improve segmentation accuracy in grotto scenes remains a problem that needs to be solved.

To address the above-mentioned problems, we proposed a method for point cloud semantic segmentation of grotto scenes through combining knowledge with deep neural network. The proposed method first summarized the point clouds characteristics, color characteristics, local geometric features, global spatial distribution patterns, the threshold for the size of grotto components and grotto components (referred to as class labels in grotto scenes) as knowledge. The knowledge is used to guide the production of benchmark dataset, design network modules, and correct incorrect segmentation results. With regard to the neural network design, we proposed a K-random neighbors method (called K-R method in this paper) based on K-Nearest Neighbors (KNN) approach to select neighboring points, addressing the issue of uneven distribution of point clouds in multiple directions. A total of six eigen-features were then used to build the point cloud attribute extension module. Through adopting the multi-layer perceptron (MLP) as the fundamental unit, we developed the local spatial hybrid encoding (LSHE) module, the parallel attention pooling (PAP) module, and the local-global feature aggregation (LGFA) module. These modules were integrated through concatenation and stacking to implement the Grotto Semantic Segmentation Network (GSS-Net). Additionally, the spatial distribution patterns and the size threshold of grotto components were used as constraint conditions to correct segmentation results. The performance of our model was further validated through comparative experiments which took four classical methods as baselines. A series of ablation studies demonstrated the effectiveness of the proposed modules.

The main contributions of this work are as follows:

• The knowledge of grotto scenes is summarized, laying foundation for building benchmark dataset, customizing neural network and improving the results through knowledge guidance.

• A neural network model specific for grottoes scenes is developed under the knowledge guidance, thereby enhancing the interpretability of the model design.

• We quantitatively evaluate the performance of our proposed method by comparing it with four classical deep learning-based methods across five distinct grotto scenes. The results demonstrate the superior effectiveness of our method.

The rest of this paper is organized as follows. Section 2 reviews related work in this field. Section 3 elaborates on the proposed methods. Section 4 presents experiment design, benchmark dataset, and the experiment results. In Section 5, the influence and effect of each module of the proposed method and some limitations are discussed. The conclusion of this work is finally summarized in Section 6.

2. Related work

Deep learning-based methods can be categorized into projection-based (Boulch et al. 2018; Lawin et al. 2017), voxel-based (Huang and You 2016), and point-based methods (Hu, Yang, Khalid et al. 2021; Qi, Su et al. 2017). To capture the geometric details in grotto scenes, the point clouds usually have a high density with a large number of points, making them highly suitable for point-based methods. These methods process raw point cloud data directly and avoid the loss of geometric fidelity inherent in the viewpoint selection or voxelization processes, which are limitations observed in projection-based and voxel-based approaches. Consequently, we mainly focus on reviewing the point-based methods in the following.

2.1. Point-based semantic segmentation

Point-based methods can be grouped into four types, including pointwise MLP (Chen et al. 2019; Engelmann et al. 2018; Hu, Yang, Xie et al. 2021; Jiang et al. 2018; Qi, Su et al. 2017; Qi, Yi et al. 2017; Xie et al. 2018; Yang et al. 2019; Yin et al. 2023; Zeng and Gevers 2018; Zhang, Hua, and Yeung 2019; Zhao et al. 2019), point convolution (Engelmann, Kontogianni, and Leibe 2020; Hua, Tran, and Yeung 2018; Li et al. 2018; Li et al. 2022; Ma et al. 2019; Park et al. 2023; Thomas et al. 2019; Wang, Suo et al. 2018), RNN-based (Engelmann et al. 2017; Huang, Wang, and Neumann 2018; Ye et al. 2018), and graph-based methods (Du, Ye, and Cao 2022; Jiang et al. 2019; Landrieu and Simonovsky 2018; Liang et al. 2019; Pan, Chew, and Lee 2020; Wang et al. 2019). The pointwise MLP method, which uses shared MLP as the units of the artificial neural network, is suitable for integrating with other networks such as pooling layers or feature aggregation layers. This facilitates the construction of network architectures for specific tasks (Guo et al. 2020).

PointNet (Qi, Su et al. 2017) pioneers the pointwise MLPs by directly processing point clouds. It employs MLPs to extract features, addresses point cloud permutation invariance using a symmetric function, and aggregates point features through max-pooling. However, PointNet is limited in capturing local contextual information. Qi, Yi et al. (2017) subsequently proposed PointNet++, which uses PointNet as a foundational module for hierarchical feature extraction. This method allows the network to capture multi-scale information within the local neighborhood of the point cloud, thereby improving its capability to express local contextual information. Inspired by the 2D shape descriptor Scale-Invariant Feature Transform (SIFT), Jiang et al. (2018) developed the PointSIFT module. This module constructs directional encoding units in eight directions and stacks them for comprehensive feature representation. Zhao et al. (2019) proposed the PointWeb approach with an Adaptive Feature Adjustment (AFA) module for efficient local contextual feature extraction, though its edge segmentation capabilities remain limited. Hu, Yang, Xie et al. (2021) proposed RandLA-Net, a cost-effective and lightweight network for large-scale point cloud segmentation. It utilizes random point sampling to enhance computational efficiency and incorporates advanced residual network modules to effectively capture multi-scale geometric features, thus compensating for the potential information loss that is typically associated with random sampling. Point Attention Transformers (PATs) (Yang et al. 2019) process points by taking the absolute and relative positions of points into consideration and employ MLPs for high-dimensional feature learning. The DCNet (Yin et al. 2023) presents novel local feature aggregation schemes to further improve local contextual feature aggregation.

2.2. Knowledge-guided semantic segmentation

Combining knowledge with deep learning for point cloud semantic segmentation has shown promising results. Weinmann et al. (2015) demonstrated that incorporating common local geometric features into model training enhanced the accuracy of semantic segmentation in point clouds. As using too many features can be counterproductive for the deep learning algorithm (Weinmann and Weinmann 2019), González-Collazo et al. (2024) explored the use of 1D Convolutional Neural Networks (1D CNNs) for the semantic segmentation of urban point clouds gathered through mobile laser scanning, utilizing local geometric features for training. Grilli et al. (2023) proposed a knowledge-enhanced neural network that added local geometric features and logical knowledge as a new layer into Transformer architecture (Zhao et al. 2021), but it did not change the network structure and modules. Maru et al. (2023) segmented building facades based on RandLA-Net, which applied corrected laser intensity as an additional attribute. Xia, Yang, and Chen (2022) developed a local geometric feature descriptor for bridge structural components, providing novel insights for small sample learning. Ballouch et al. (2022) employed image classification as prior knowledge, and then mapped these raster values onto point clouds, expanding the attributes in point clouds. Ponciano et al. (2021) compared the performance of deep learning-based method and knowledge-based method in urban scenes of point cloud semantic segmentation. The deep learning-based proved more effective for processing those elements with complex, non-standard geometric features, while the knowledge-based method excelled in segmenting the elements with regular geometric shapes. In order to improve semantic segmentation results using a knowledge-based approach, extensive and precise knowledge sets need to be predefined.

Currently, the performance of existing point cloud semantic segmentation methods in grotto scenes remains uncertain. Therefore, this study aims to summarize the characteristics of grotto scenes into knowledge for guiding neural network design and employ additional constraints to improve the deep learning results. To accomplish this objective, we build a point cloud benchmark dataset comprising five grotto scenes and selected four deep learning methods as baselines. Subsequently, we proposed a knowledge-guided deep learning approach to elevate the accuracy of point cloud semantic segmentation in grotto scenes.

3. Methodology

This section begins with an overview of the overall research approach. Accordingly, the details of the neural network modules are highlighted, following which the results based on knowledge constraint are introduced.

3.1. Overall research approach

The overall research framework is depicted in Figure 1. Firstly, knowledge of grotto scenes is summarized to guide the deep learning-based semantic segmentation process. The knowledge includes point cloud density knowledge, visual color knowledge, local geometric feature knowledge, global spatial distribution knowledge, threshold for the size of grotto components knowledge and grotto components knowledge. Guided by the above-mentioned knowledge, a benchmark dataset is constructed, addressing the challenge faced by data processors who are not in the field of cultural heritage and find it difficult to understand class labels in grottoes. Subsequently, a neural network taking the characteristics knowledge of grotto scenes into account is designed, trained, verified and tested with the benchmark dataset. Finally, the erroneously segmented areas are corrected by applying the constraints derived from the afore-mentioned knowledge. More details regarding how the summarized knowledge guides semantic segmentation of point clouds in grotto scenes are elaborated in the following.

Figure 1. The overall research approach of semantic segmentation using the knowledge-guided deep learning method.

The grotto scene knowledge (GSK) is summarized below.

• GSK 1: Point cloud density knowledge applied for grotto scenes

In scenarios with uneven point cloud density, it is crucial to select nearest neighbor points that encompass information from multiple directions. This approach better represents local geometric features. The variations in the proportions of different components within the point clouds can result in insufficient learning for those classes with smaller sample size, such as stone tablets, railings in the scene.

• GSK 2: Visual color knowledge applied for grotto scenes

Color variations are notably significant among different components in grotto scenes. For example, vegetation typically appears green, while the ground is usually cementing gray. The color of cliff walls varies depending on the type of rock. For instance, in the Dazu Rock Carvings, the predominantly red sandstone results in red-colored cliff walls. These obvious color differences among grotto components aid in distinguishing various classes.

• GSK 3: Local geometric feature knowledge applied for grotto scenes

Different components within grotto scenes possess unique local geometric shapes. For example, the ground and niche eave are relatively flat and oriented horizontally, and cliff walls are nearly vertical with minimal local curvature changes. The geometric shapes of Buddha statues are complex with significant curvature variations. Additionally, the sizes and orientations of the Buddha statues vary. Thus, the rotation as well as the scale-invariant local geometric feature descriptors are essential for their description.

• GSK 4: Global spatial distribution knowledge applied for grotto scenes

The types of components in grotto scenes are closely associated with their relative elevations within the scene. There is a distinct vertical stratification, with components arranged from the lowest to the highest, i.e. the ground, the cliff wall footing, pedestal, the cliff wall and Buddha statue, niche eave, and the mountain and vegetation. Consequently, the relative height can effectively serve as a constraint for identifying and categorizing global features.

• GSK 5: Threshold for the size of grotto components knowledge

The actual size range of various components in grottoes and the size of segmentation results can be used to identify erroneously segmented regions. Therefore, the threshold for the size of grotto components can be used to improve results.

• GSK 6: Grotto components knowledge

Accurate identification of semantic categories (i.e. labeling classes) within grotto scenes requires expertise in cultural heritage, which plays as basis for the construction of point cloud benchmarks dataset.

In the following subsections, we provide an exploration of how knowledge guides the design of modules as listed in Table 1. Furthermore, we have conducted ablation studies to validate the effectiveness of each module. The correspondences among GSKs, network modules, and ablation studies are also illustrated in Table 1.

Table 1. The correspondence between module, grotto scene knowledge, and ablation study.

Module	Knowledge	Ablation study
Neighborhood selection module (Sect. 3.2.1)	GSK 1	Ablation of the K-R method for selecting neighbor points (Sect. 5.1.5)
Point cloud attribute extension module (Sect. 3.2.2)	GSK 2, GSK 3	Ablation of point cloud attribute extension (Sect. 5.1.2)
Local spatial hybrid encoding module (Sect. 3.2.3)	GSK 3	Ablation of local spatial encoding (Sect. 5.1.3)
Parallel attention pooling module (Sect. 3.2.4)	-	Ablation of overall framework (Sect. 5.1.1)
Local-global feature aggregation module (Sect. 3.2.5)	GSK 4	Ablation of local-global feature aggregation (Sect. 5.1.4)
Network architecture (Sect. 3.2.6)	GSK 1, GSK 6	-
Results improvement with knowledge constraints (Sect. 3.3)	GSK 4, GSK 5	Comparison of experimental results (Sect. 4.3)

3.2. Knowledge-guided point cloud semantic segmentation of grotto scene

3.2.1. Neighborhood selection

In order to describe the local geometric features of a given point $p$ with its surrounding nearest neighborhood in the point clouds, several neighborhood definition methods include the spherical approach (Lee and Schenk 2002), the cylindrical approach (Filin and Pfeifer 2005), and the K-Nearest Neighbors (KNN) method (Connor and Kumar 2010) have been commonly used. The spherical and cylindrical approaches require predefined parameters, such as radius or cylinder height, which highly depend on the density of the point cloud. In contrast, the KNN method, which relies on a fixed number of neighboring points $K$, is independent of point cloud specifics.

As described in GSK 1, the KNN method may not ensure the inclusion of neighboring points from a variety of directions in point clouds with uneven distribution, potentially leading to imprecise representation of local geometric features. This limitation is particularly noticeable in high-density point clouds with a smaller $K$ value. To address this, we proposed the K-random neighbors method (K-R method) for selecting nearest neighbors in the point clouds. The K-R method initially expands the neighborhood by increasing $K$ to include a broader range of points and then selects $k$ points through random sampling. Figure 2 shows the results of the KNN method and the K-R method in selecting neighbor points to the given point $p$. The red points in Figure 2(a) and the blue points in Figure 2(b) are the neighbor points selected by the two methods The K-R method considers neighbor points in various directions relative to the given point, thereby enhancing the ability to discern local geometric features.

Figure 2. Comparison of KNN and K-R method.

3.2.2. Point cloud attribute extension

The integration of the local features of point clouds as extended attributes into the neural network has been shown to enhance the performance of semantic segmentation (Grilli et al. 2023; Maru et al. 2023; Weinmann et al. 2015). Guided by GSK2 and GSK3, the Point Cloud Attribute Extension (PCAE) module extends a new scalar field for each point, which is composed of color features and eigen-features (Lin et al. 2014; Weinmann, Jutzi, and Mallet 2014).

The covariance matrix is a fundamental statistical tool that captures the spatial variance of a set of points in 3D space, reflecting the spread and orientation of the data. We used the weighted principal component analysis (PCA) method with geometric median (Lin et al. 2014) to extract local neighborhood eigen-features for each point. The covariance matrix is formulated as (1) $\mathit{C}={\displaystyle \frac{{\sum }_{p_i\in P}{w}_{p_{\mathrm{gm}}}(p_i-p_{\mathrm{gm}}){(p_i-p_{\mathrm{gm}})}^T}{{\sum }_{p_i\in P}{w}_{p_{\mathrm{gm}}}(p_i)}}$(1) where $p_i\in P$ is the point $p_i$ belongs to point set $P$ and $w_{p_{\mathrm{gm}}}(p_i)$ is the weight of point $p_i$ with respect to the geometric median $p_{\mathrm{gm}}$.

The calculation of the covariance matrix presents three non-negative eigenvalues, denoted as ${\lambda }_1\ge {\lambda }_2\ge {\lambda }_3\ge 0$. Upon normalizing these eigenvalues, a set of features is computed by formula (2)-(7) to describe the local geometric characteristics of a point cloud and indicate the 2D planarity, 3D divergence, surface variation, and the order or disorder of 3D points within the local 3D neighborhood. The eigen-features selected in this study include planarity $P_{\lambda }$, scattering $S_{\lambda }$, omnivariance $O_{\lambda }$, anisotropy $A_{\lambda }$, eigenentropy $E_{\lambda }$, and change of curvature $C_{\lambda }$. Those formulas are as follows: (2) $P_{\lambda }={\displaystyle \frac{{\lambda }_2-{\lambda }_3}{{\lambda }_1}}$(2) (3) $S_{\lambda }={\displaystyle \frac{{\lambda }_3}{{\lambda }_1}}$(3) (4) $O_{\lambda }=\sqrt[3]{{\lambda }_1{\lambda }_2{\lambda }_3}$(4) (5) $A_{\lambda }={\displaystyle \frac{{\lambda }_1-{\lambda }_3}{{\lambda }_1}}$(5) (6) $E_{\lambda }=-\sum _{i=1}^3{\lambda }_i\ln {\lambda }_i$(6) (7) $C_{\lambda }={\displaystyle \frac{{\lambda }_3}{{\lambda }_1+{\lambda }_2+{\lambda }_3}}$(7)

3.2.3. Local spatial hybrid encoding

Effective local spatial feature encoding can represent the spatial patterns and the structure of points related to their neighbors, thereby providing improved contextual information to the neural network (Chen et al. 2022; Fan et al. 2021; Huang, Xu, and Stilla 2021; Li et al. 2020). Guided by GSK3, we adopt a hybrid approach that combines the Euclidean distance in Cartesian coordinates with the Laplacian (Laplace operator) in differential coordinates (Sorkine 2005) to construct the local spatial hybrid encoding (LSHE) module. The design details of this module are shown in Figure 3.

Figure 3. The local spatial hybrid encoding module.

The Laplacian is defined as the divergence ($\mathrm{\nabla }\cdot f$) of the gradient ($\mathrm{\nabla }f$). This operator exhibits rotational invariance, scale invariance, making it effective for describing the difference between the function value of a point and the average value of its neighborhood. It can be used to approximately express the local geometry shape based on the nearest neighbor in a simple, efficient and accurate way (Cao et al. 2010; Liang et al. 2012; Sharp and Crane 2020), which is very important for point cloud processing. The Laplacian ($\mathrm{\Delta }f$) corresponding to any twice-differentiable function $f$can be expressed as: (8) $\mathrm{\Delta }f={\mathrm{\nabla }}^{2}f=\mathrm{\nabla }\cdot \mathrm{\nabla f}$(8) Given that point clouds are spatially discrete, this module uses the discretization of the cotangent Laplace operator to express local geometric feature around a point. As shown in Figure 4, one-ring triangles are constructed with the given point $p_i$ and its nearest neighbors $n$ is selected by the K-R method. The value of Laplacian (i.e. differential coordinate vector) with the cotangent weighting is calculated as follows (Bunge et al. 2020; Sorkine 2006). (9) ${\delta }_i={\displaystyle \frac{1}{n}\sum _{j\in n}(\cot {\alpha }_{\mathrm{ij}}+\cot {\beta }_{\mathrm{ij}})(p_i-p_j)}$(9) where $p_i$ is the given point, $n$ is the total number of neighboring points, $p_j$ is one of the neighboring points, and ${\alpha }_{\mathrm{ij}}$ and ${\beta }_{\mathrm{ij}}$ are denoted as the two angles opposite of edge ($p_i,p_j$).

Figure 4. Cotangent discretization of the Laplacian operator.

Finally, spatial encoding is performed for each point set $\{p_i^1\dots p_i^k\dots p_i^K\}$ formed by K-R neighbors. The formula for spatial encoding is as follows. (10) $r_i^k=\mathrm{MLP}({\delta }_i\oplus (p_i-p_i^k)\oplus \Vert p_i-p_i^k\Vert )$(10) where ${\delta }_i$ represents the Laplacian operator of the given point $p_i$, and $p_i^k$ are the absolute coordinates of the point cloud in the Euclidean coordinate system, $\oplus$ denotes the concatenation operation, and $\Vert \cdot \Vert$ is the Euclidean distance between the central point and its neighbors. $r_i^k$ is the spatial encoding of discrete spatial coordinate points, which is used for local feature learning.

3.2.4. Parallel attention pooling

The pooling layers are used to select prominent features from the high-dimensional feature vectors of neighbor points (Er et al. 2016; Lu et al. 2022; Wang et al. 2023). Features within each neighborhood are aggregated using methods such as maximum pooling (Qi, Su et al. 2017), average pooling (Wang, Zhang et al. 2018), or combination of both (Li et al. 2023). However, these pooling approaches are limited to capturing partial information, ignoring other valuable features (Yang et al. 2020). The attention scores are gained through adopting a ‘scoring method’ to assign attention to features rather than simply ignoring some of them. In order to minimize the loss of feature selection, we use MLP transformation on the input, calculate Softmax scores, and ultimately multiply the input features. Considering that new scalar fields have been expanded in the PCAE module, there exist a risk that the premature mixing of point cloud attribute information with spatial encoding information leads to the loss of significant features. As shown in Figure 5, this study develops a parallel attention pooling (PAP) module, which computes different attention scores for spatial information and extended attributes. Subsequently, the two sets of features are concatenated and combined through weighted summation, with feature aggregation achieved using the shared MLP.

Figure 5. The parallel attention pooling module.

The PAP module computes attention scores separately for spatial encoding $S_i=\{S_i^1\dots S_i^k\dots S_i^K\}$ and attribute features $A_i=\{A_i^1\dots A_i^k\dots A_i^K\}$ based on a shared function $g(\cdot )$, ensuring that the sum of attention scores in each row equals to 1. The function $g(\cdot )$ consists of a shared MLP followed by a Softmax operation. The formula is as follows. (11) $\{\begin{array}{c}S{s}_i^k=g(S_i^k,W_S)\\ A{s}_i^k=g(A_i^k,W_A)\end{array}$(11) where $W$ represents the learnable weights of the shared MLP, and $S{s}_i^k$ and $A{s}_i^k$ are the attention scores of spatial encoding and attributes, respectively.

Subsequently, it adopts attention scores as weights of feature selection for features aggregation to enhance important features while weakening irrelevant features. The pooled feature for each point is determined by the weighted sum of its local features. The result is represented by a feature vector that encapsulates the enhanced and aggregated features, which is computed as follows. (12) $\widetilde{f_i}=\sum _{k=1}^K((S_i^k\cdot S{s}_i^k)\oplus (A_i^k\cdot A{s}_i^k))$(12) where $\cdot$ denotes the Hadamard product, which is the element-wise multiplication of two vectors, forming a new vector of the same size as the original vectors.

The PAP module captures the most salient features from neighboring points, enabling the model to handle complex spatial dependencies effectively. Therefore, for a given point $p_i$ in point cloud $P$, its geometric patterns and attributes in a local area are expressed by the vector $\widetilde{f_i}$ after learning with the LSHE and PAP modules. This vector encapsulates all the important features about one point and its neighbors.

3.2.5. Local-global feature aggregation

The feature aggregation integrates an extensive range of local and global contextual information, thereby enriching each point with a comprehensive set of features (Xie et al. 2024; Zhou, Huang, and Fang 2021). In accordance with GSK4, the global features are obtained through computing a global pattern by multiplying the horizontal projection density $(\rho )$ of local point sets with their relative elevation $(h)$. The formula for this process is as follows: (13) $f_g=\mathrm{MLP}(p_i\oplus (\rho \cdot h))$(13) where $p_i$ denotes the spatial coordinates of points within the point cloud, $\rho$ represents the projection density of the point set in the local neighborhood on the horizontal plane, and $h$ signifies the relative elevation of the point within the aggregate point cloud.

By stacking the LSHE and PAP modules twice, the local features across multiple spatial scales are incorporated. The fusion of global and local features forms the Local-Global Feature Aggregation (LGFA) module, which is inspired by the residual module principles in ResNet (He et al. 2016) and RandLA-Net (Hu, Yang, Xie et al. 2021). The proposed LGFA unit consists of the following steps as depicted in Figure 6.

Figure 6. The local-global feature aggregation module.

3.2.6. Network architecture

As illustrated in Figure 7, the GSS-Net adopts the Encoder-Decoder architecture with skip connections. The input of this network is point cloud data composed $N$ points. Each point with $d$-dimensional$(N,d)$ consists of spatial coordinates, color information, and extended attributes. Initially, a Fully Connected (FC) layer is utilized to expand the feature dimension to $(N,16)$. The encoding phase involves stacking four layers. Each layer comprises a LGFA module and a random sampling operation. After each encoding step, the number of points is down sampled to 25% of the previous layer, while the feature dimensions are increased to 32, 128, 256, and 512, respectively. This down-sampling strategy, based on random sampling (Hu, Yang, Xie et al. 2021), is complemented by the LGFA module, which aims to mitigate potential information loss incurred during the process.

Figure 7. The architecture of GSS-Net. FC: Fully Connect layer, LGFA: Local-Global Feature Aggregation Module, RS: Random Sampling, US: Up-sampling, MLP: shared Multi-Layer Perceptron, DP: Dropout.

In the decoding layers, the nearest neighbor interpolation is employed for up-sampling, progressively restoring the point cloud to its original size. Meanwhile, the feature dimension of each layer is gradually reduced. The features generated in each decoding layer are aligned and concatenated with those from the corresponding LGFA modules in the encoding layers. This ensures continuity and prevents information loss during the up-sampling. Finally, the network concludes with three FC layers that generate the prediction results. The network outputs point clouds comprising $N$ points, where each point is labeled with a 1-dimensional class label $n_{\mathrm{class}}$.

The implementation details of the shared MLP in this work can be represented as (14) $\mathrm{MLP}=w(\mathrm{BN}(\mathrm{conv}(x)))$(14) where $w$ represents the activation function, $\mathrm{BN}$ denotes batch normalization, $\mathrm{conv}$ refers to a convolution operation with a 1 × 1 kernel, and $x$ is the input for the $\mathrm{MLP}$.

In addition, guided by GSK1 and GSK6, the weighted cross-entropy loss function (Ho and Wookey 2019) is used to handle unbalanced datasets in classification tasks. This approach assigns different weights to various categories, which not only addresses the imbalance but also accelerates the convergence speed in the model training process.

3.3. Results improvement with knowledge constraints

The process of improving segmentation results with knowledge constraints is shown in Figure 8. After processing by GSS-Net, the point clouds are divided into different semantic categories according to semantic labels. With regard to the same labels, the spatial clustering algorithm based on Euclidean distance is used to cluster and constitute multiple point cloud patches. Under the constraints of GSK4, GSK5 and the number of outlier points, each point cloud patch is assessed to determine whether it is an outlier (i.e. a set of points that is not segmented correctly). For example, regarding the detection of outlier points in Figure 8, the number and the size of point set in the red box are obviously different from the circular Buddhist niche, so it is judged to be the incorrect area for segmentation. In such cases, the boundary of the outlier points is calculated using the alpha-shape operator (Wei 2008). The outlier points are merged into the semantic label with which they share the most boundaries.

Figure 8. The method of improving results with knowledge constraints.

3.4. Evaluation metrics

Currently, there is a lack of point cloud benchmark datasets for grotto scenes in the field of cultural heritage. To facilitate a quantitative evaluation of the method proposed in this work, we use CloudCompare software⁴ to manually segment and assign semantic labels to point cloud of five grotto scenes. GSK 6 provides knowledge guidance for manually adding class labels when making benchmark datasets. The description of the ten classes is provided in detail in Table 2.

Table 2. The class labels in grotto scenes.

Types	Class labels	Description
Natural elements	Low vegetation and mountains	Natural mountains and low vegetation covering them.
Grotto components	Cliff wall	Man-made cliff face adorned with grotto art.
	Cliff wall footing	A protruding part at the base of the cliff, like the plinth of modern architecture.
	Niche eave	A protruding section at the top of the cliff wall, designed to prevent rainwater from directly eroding the cliff, like the eaves of modern architecture.
	Buddha statue	Buddha statues carved into the cliff wall.
	Small niche	Small niches carved into the cliff wall. In this study, the types of small niches are mainly circular niche.
	Pedestal	The pedestal of the Buddha statue.
	Stele	Stone steles inscribed with text.
Man-made elements	Cement pavement	Cement pathways constructed for visitor access.
	Railing	Railings constructed alongside the mountain for the safety of visitors.

In this study, three quantitative indicators are used to evaluate the experimental results, which include the Overall Accuracy (OA), Intersection-over-Union (IoU) for each class, and mean Intersection-over-Union (mIoU). Among them, OA is the ratio of the number of correctly predicted points to the total number of points. The formulas for calculating IoU and mIoU are as follows. (15) $\mathrm{Io}{U}_i={\displaystyle \frac{p_{\mathrm{ii}}}{p_{\mathrm{ii}}+{\sum }_{j\ne i}{p}_{\mathrm{ij}}+{\sum }_{k\ne i}{p}_{\mathrm{ki}}}}$(15) (16) $\mathrm{mIoU}={\displaystyle \frac{{\sum }_{i=1}^N\mathrm{Io}{U}_i}{N}}$(16) where $p_{\mathrm{ii}}$ is the number of correctly predicted points for class $i$, $p_{\mathrm{ij}}$ represents the number of points that belong to class $i$ but are misclassified as class $j$, and $p_{\mathrm{ki}}$ indicates the number of points that belong to class $k$ but are misclassified as class $i$. $N$ is the total number of classes. $\mathrm{Io}{U}_i$ represents the IoU for class $i$.

4. Experiments and results

4.1. Experiment design

In this section, two sets of experiments were designed to verify the effectiveness of the proposed method. A total of four classical point-based deep learning models for point cloud semantic segmentation were selected as baselines to compare with our proposed method for performance validation based on the benchmark dataset of grotto scenes. Figure 9 presents the detailed experimental design, including grotto scene datasets used in different experiments, the division of the grotto scenes for training, validation and testing, and the comparative experiments for each experimental configuration.

Figure 9. The design of experiments.

In Experiment 1, each of the five grotto scenes was bisected into left and right sections, which alternately served as training set, validation set and testing set. A two-fold cross-validation approach was used to validate the methods proposed in this study. For example, when the left side of scene 1 is used for training and validation, the right side is used for testing, and then the left and right sides are swapped again. Grotto scenes 1–5 are selected for the comparative analysis in the intra-scene by using four baseline approaches: PointNet (Qi, Su et al. 2017), PointNet++ (Qi, Yi et al. 2017), KPConv (Thomas et al. 2019), and RandLA-Net (Hu, Yang, Xie et al. 2021).

Experiment 2 is to verify the performance in the inter-scene. The number of points, the number of semantic labels and the complexity of grotto scene 2 are moderate among the five scenes. As such, grotto scenes 1, 3, 4, and 5 were selected as the training set and validation set, while grotto scene 2 was selected as the testing set. As there exist different number of labels in the five grotto scenes, these experiments focused on using five types of class labels: Buddha statues, cliff walls, niche eaves, low vegetation, and circular niches, which are almost included in each scene.

The hardware environment for these experiments consisted of a 13th i7-13700KF 3.40 GHz processor and a RTX 4070Ti GPU with 12G of memory. The software environment included Ubuntu 22.04, the TensorFlow 2.0 deep learning framework, CUDA 12.2, CUDNN 8.3.0, and Python 3.6. An Adam optimizer with default settings was also used. The initial learning rate was set as 0.01, with a 5% reduction after each epoch. The batch size for the experiments was configured at 4. The K-R method initially set 32 nearest neighbors, and then randomly selected 16 points. The impact of different values on experiments is shown in Section 5.1.5. The code employed for the comparative experiments was obtained from the open-source repositories hosted on GitHub, as shown in Table 3. All source codes were adapted to the TensorFlow 2.0 framework without other adjustments. The network architecture and implementation proposed in this work was referred to the code of RandLA-Net.

Table 3. The code source of comparative experiments.

Methods	Code sources	Code publication time (the latest version)
PointNet	https://github.com/charlesq34/pointnet	Commits on Sep 23, 2022
PointNet++	https://github.com/charlesq34/pointnet2	Commits on Nov 20, 2018
KPConv	https://github.com/HuguesTHOMAS/KPConv	Commits on Feb 23, 2021
RandLA-Net	https://github.com/QingyongHu/RandLA-Net	Commits on Jul 2, 2021

4.2. Point cloud benchmark dataset of grotto scenes

The benchmark dataset includes five distinct grotto scenes located in Baoding Mountain, a part of the Dazu Rock Carvings in Chongqing, China. Figure 10 illustrates the point clouds of the five scenes with class labels. From scene 1 to scene 5, it exhibits a progressive increase in the semantic categories and complexity of grotto components, showcasing a range of geometric structure that evolves from simple to intricate forms. The point clouds were generated by intensive image matching between the UAV oblique photography and close-up photos in the Context Capture software. To be specific, the drone was DJI mini 4 Pro, of which the image sensor was 1/1.3-inch CMOS with 48 million effective pixels. The SONY A7R4 camera was used to take supplementary photos closer to the grottoes, with a lens resolution of 62 megapixels and a 35 mm full-frame Exmor R CMOS backlit image sensor. Table 4 provides the detailed information of the number of points across various semantics (i.e. ten labeled classes) within each scene.

Figure 10. The visualization of the grotto scenes point clouds.

Table 4. The number of points of each element in the grotto scene point cloud benchmark dataset (Bud.sta.: Buddha statue, cli. wal.: cliff wall, nic. eav.: niche eave, low veg.: low vegetation, cir.nic.: circular niche, c. w. foo.: cliff wall footing, cem.pav.: cement pavement, rai.: railing, ped.: pedestal, ste.: stele).

Scene	Bud. sta.	cli. wal.	nic. eav	low veg.	cir. nic.	c. w. foo.	cem. pav.	rai.	ped.	ste.
Scene 1	323050	276140	99502	25583	/	/	/	/	/	/
Scene 2	910188	1187292	179002	12441	446888	/	/	/	/	/
Scene 3	1848049	1123757	551957	17613	627104	/	/	/	/	/
Scene 4	1346130	1742420	77266	27124	/	248476	98870	20532	/	/
Scene 5	1198088	672119	414568	36919	904720	342702	28449	27747	118740	17146

The characteristics and details of each scene are as follows.

1. Grotto scene 1 features simple semantic categories with orderly arranged components. The scene spans approximately 4.5 meters in height and 10.7 meters in length. It features a Buddha statue approximately 1.6 meters tall, with a total number of points of 724,275.

2. Grotto scene 2 is more complex compared to scene 1, which features circular niches and Buddha statues of various geometric shapes. It spans approximately 20 meters in length and 7.3 meters in height. The central Buddha statue is approximately 4 meters tall, with other statues averaging around 1.6 meters in height. The circular niches are about 1.8 meters high and 1.4 meters wide. The total number of points in this scene is 2,735,811.

3. Grotto scene 3 extends about 20 meters in length and approximately 10 meters in height, characterized by its double-layered niche eaves which divide the cliff wall into upper and lower sections. The upper section predominantly consists of circular niches, each measuring approximately 1.5 meters by 1.7 meters. The lower section displays small Buddha statues approximately 1.6 meters tall and a central large statue about 4 meters in height. The total number of points in this scene is 4,168,480.

4. Grotto scene 4, approximately 14 meters high and 18 meters long, features a 4.5-meter-tall half-body Buddha statue and various smaller statues scattered around. The classes are expanded to include cliff wall footing, railings, and cement pavement. This scene encompasses a total of 3,560,818 points.

5. Grotto scene 5, stretching approximately 16 meters in length and about 14.5 meters in height, encompasses the most diverse range of 10 classes. This scene features three prominent Buddha statues, each standing 6 meters tall. The cliff walls adorned with circular niches approximately 0.75 meters in diameter. The total number of points in this scene reaches 3,761,198.

4.3. Results and analysis

1. Experiment 1 – grotto scene 1

Table 5 illustrated the quantitative results from OA, mIoU and IoU of grotto scene 1, including the results obtained by the four baseline methods, the results of our proposed GSS-Net method, and the results of adding knowledge constraints on the GSS-Net results. The RGB colored input point clouds, ground truth, the results obtained by the GSS-Net and the improved results with knowledge constraints are visualized in Figure 11. Overall, it obtained excellent results in scenarios with fewer types of classes and regularly arranged Buddha statues. The result of GSS-Net segmentation achieved OA of 95% and mIoU of 90.23%. After improving the segmentation results with knowledge constraints (see Figure 11(c) and (d)), the OA increased by 1.59%, and the mIoU improved by 0.7%.

(2)	Experiment 1 – grotto scene 2

Figure 11. The visualization results of segmentation in grotto scene 1.

Table 5. The results of OA, mIoU and IoU of different approaches in grotto scene 1.

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav.	low veg.
PointNet	88.24	82.61	78.03	75.56	90.27	86.56
PointNet++	88.65	83.23	78.74	76.23	90.81	87.13
KPConv	92.91	87.95	86.69	84.67	91.52	88.92
RandLa-Net	94.55	89.96	88.14	86.51	92.92	92.25
GSS-Net (Ours)	95.00	90.23	88.53	87.78	92.65	91.96
Improved results	96.59	90.93	89.10	87.83	93.11	93.68

In grotto scene 2, it was challenging to segment the main Buddha statue, particularly at its boundary area, due to its distinctive shape. As shown in Figure 12(c and d), the improved results with knowledge constraints effectively corrected errors in the boundaries of Buddha statues, circular niches, and niche eave. Furthermore, those areas with relatively regular geometric features, such as circular niches, achieved good segmentation results. The mIoU of the GSS-Net in this scene reached 86.98%. After the improvement, there was a notable enhancement in segmentation results, with the OA increasing by 1.2% and the mIoU increasing by 0.76%, as detailed in Table 6.

(3)	Experiment 1 – grotto scene 3

Figure 12. The visualization results of segmentation in grotto scene 2.

Table 6. The results of OA, mIoU and IoU of different approaches in grotto scene 2

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav.	low veg.	cir. nic
PointNet	87.34	79.56	76.20	82.12	83.34	80.67	75.45
PointNet++	88.51	80.49	77.89	82.45	84.12	80.34	77.67
KPConv	92.15	85.99	85.12	86.52	92.20	84.67	81.45
RandLa-Net	92.78	86.18	85.34	85.78	91.41	85.12	83.23
GSS-Net (Ours)	93.42	86.98	85.28	86.10	93.16	85.64	84.74
Improved results	94.62	87.74	85.75	86.74	93.76	87.06	85.38

In grotto scene 3, the mIoU for the Buddha statue and niche eave was relatively low. This was primarily attributed to their complex geometric shapes, leading to some horizontal areas being mistakenly identified as niche eave. A notable example was observed on the left side of the experimental area, where a large section part of the cliff wall was erroneously classified as circular niches (see Figure 13 (c and d)). Since the number of points and the size of these misclassified areas did not correspond with actual circular niches, the improvement process was able to reclassify these areas into cliff wall, particularly focusing on the sections with the most boundary contact. The visualized segmentation results for grotto scene 3 are depicted in Figure 13. The detailed quantitative results are shown in Table 7.

(4)	Experiment 1 – grotto scene 4

Figure 13. The visualization results of segmentation in grotto scene 3.

Table 7. The results of OA, mIoU and IoU of different approaches in grotto scene 3.

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav.	low veg.	cir. nic
PointNet	84.43	77.85	73.30	69.85	82.45	81.14	82.50
PointNet++	85.10	78.84	73.80	71.92	83.23	82.45	82.78
KPConv	90.30	86.57	80.45	77.21	91.87	89.43	93.90
RandLa-Net	91.01	87.72	81.34	80.56	91.45	90.56	94.67
GSS-Net (Ours)	92.60	88.23	82.32	80.15	92.00	92.01	94.66
Improved results	93.61	88.80	83.29	81.26	92.15	92.05	95.24

In grotto scene 4, the Buddha statues were distributed more sporadically across the cliff wall and varied in posture, while the shape of the cliff wall footing was irregular. These factors contributed to lower segmentation results for the Buddha statues, cliff wall, and cliff wall footing, compared to other classes. The visualization and quantitative segmentation results for this scene are illustrated in Figure 14 and Table 8, respectively. It is noted that the mIoU of segmenting railings, which own regular geometric shapes, achieved 95.09%.

Figure 14. The visualization results of segmentation in grotto scene 4.

Table 8. The results of OA, mIoU and IoU of different approaches in grotto scene 4.

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav	low veg.	c. w. foo.	cem. pav.	rai.
PointNet	79.34	69.10	45.12	56.23	80.45	78.56	72.34	68.45	82.56
PointNet++	80.78	70.31	48.78	56.90	82.12	79.34	72.45	68.93	83.67
KPConv	85.24	79.29	66.89	71.12	85.67	84.23	83.12	73.56	90.45
RandLa-Net	86.36	80.08	67.45	71.20	88.56	85.12	83.23	74.67	90.33
GSS-Net (Ours)	87.59	81.23	69.17	70.89	88.71	86.45	82.59	75.74	95.09
Improved results	88.55	81.83	69.75	71.43	89.02	87.57	83.55	75.99	95.51

The knowledge constraints played a crucial role in refining and improving the segmentation results. A significant portion of the main Buddha statue, which was initially misclassified as part of the cliff wall, was correctly classified as Buddha statue after applying the knowledge constraints. Such correction improved the segmentation results of the main Buddha statue. Consequently, as shown in Table 8, the OA increased by 0.96% and the mIoU increased by 0.6%.

(5)	Experiment 1 – grotto scene 5

In grotto scene 5, the comparative results of the proposed method in this study against the other four baseline methods are depicted in Figure 15. Table 9 presents the quantitative results comparing our proposed method with others. These results demonstrate that the GSS-Net outperforms the second best in four baselines in both OA and mIoU, with improvements of 0.18% and 1.65%. The Buddha statue, niche eave, low vegetation, cliff wall footing and stele achieved the best IoU scores. After improving the results with knowledge constraints, the OA and mIoU reached 93.69% and 85.41%.

(6)	Experiment 2

Figure 15. Comparison of different approaches in grotto scene 5.

Table 9. The results of OA, mIoU and IoU of different approaches in grotto scene 5.

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav	low veg.	cir. nic.	c. w. foo.	cem. pav.	rai.	ped.	ste.
PointNet	78.45	68.09	64.44	35.93	89.70	86.85	68.93	73.28	73.93	88.38	45.95	53.47
PointNet++	84.87	72.96	82.54	62.87	87.91	88.73	78.41	74.80	70.15	73.58	46.00	64.60
KPConv	91.40	81.43	89.68	75.01	94.83	91.36	90.46	79.92	80.76	88.94	60.15	63.15
RandLa-Net	92.56	82.18	92.64	81.74	87.74	92.87	91.51	77.56	76.09	93.37	58.66	69.64
GSS-Net (Ours)	92.74	83.83	93.41	79.71	95.90	95.55	89.92	81.33	80.66	91.93	51.68	78.23
Improved results	93.69	85.41	93.43	80.92	96.50	96.24	89.94	82.87	81.68	92.00	61.47	79.02

In experiment 2 that compared our method with four baseline methods, our proposed method also achieved convincing result as shown in Figure 16 and Table 10. It can be observed that, without optimization, GSS-Net achieved the best performance in OA and mIoU. Buddha statues, low vegetation, and circular niches achieved the higher IoU scores. The differences in shapes among various Buddha statues pose a segmentation challenge. However, GSS-Net is able to consider the characteristics of grotto scenes in feature expression and aggregation, our method achieved an IoU of 77.76% for Buddha statues, which is an improvement of 2.85% over the second-best method.

Figure 16. Comparison of different methods in grotto scene 2.

Table 10. The results of OA, mIoU and IoU of different methods in grotto scene 2.

Methods	OA (%)	mIoU (%)	Bud. sta.	cli. wal.	nic. eav.	low veg.	cir. nic
PointNet	75.47	60.65	65.77	62.79	54.39	57.49	62.82
PointNet++	78.07	63.34	70.02	68.16	48.36	60.91	69.25
KPConv	82.09	67.78	74.91	69.32	58.07	63.92	72.70
RandLa-Net	82.87	69.68	74.60	76.16	77.08	60.33	60.21
GSS-Net (Ours)	83.53	71.22	77.76	69.31	55.24	75.68	78.11
Improved results	90.28	76.47	78.40	72.48	76.17	75.99	79.32

Our method showed weaker performance in segmenting niche eaves and cliff walls. It is potentially because there is a small horizontal protrusion in the middle of the cliff wall, which was intended to differentiate the lower layer of Buddha statues from the upper layer of circular niches. Our method misclassified it as niche eaves, leading to numerous incorrectly segmented points. However, guided by the GSK5 of relative height, these areas were correctly reclassified as part of the cliff wall in the improvement process. Figure 17 illustrates the incorrectly segmented areas that were identified through spatial clustering. These areas were corrected under the constraints of relative height and the threshold for the size of grotto components.

Figure 17. The identification of incorrectly segmented areas in grotto scene 2 (red boxes indicate incorrectly segmented areas).

5. Discussion

5.1. Ablation studies

We design the following ablation studies to illustrate the effectiveness of each sub-module in GSS-Net designed under the knowledge guidance and its influence on the segmentation results. For these ablation studies, grotto scene 1, 3, 4, and 5 were chosen as the training and validation set, while grotto scene 2 was selected as the testing set. The mIoU score was selected as the metric to validate the comparison results in ablation studies, since it can reflect the segmentation effect of all class labels in the grottoes.

5.1.1. Ablation of the overall framework

The network architecture of GSS-Net comprises several key modules, including the local spatial hybrid encoding (LSHE) module, the parallel attention pooling (PAP), and the local-global feature aggregation (LGFA) module. To evaluate the individual contribution of each sub-module to the overall performance of the framework, six ablation studies were conducted. These studies involved adjusting the network by selectively disabling or modifying certain modules to assess their impact on the network performance. Table 11 shows the details of each ablation study and presents the mIoU scores for all the adjusted networks compared to the full implementation of GSS-Net.

Table 11. mIoU for different module ablation studies in the overall framework.

Ablated networks	Description	mIoU (%)
Remove LSHE	The LSHE module was directly removed, and the point cloud was directly fed into subsequent modules	61.88
Replace with max-pooling	Replace the attention pooling of both channels in PAP with max-pooling	68.16
Replace with mean-pooling	Replace the attention pooling of both channels in PAP with mean-pooling	66.48
Replace with sum-pooling	Replace the attention pooling of both channels in PAP with sum-pooling	66.74
Simplify LGFA	The LGFA module was simplified to consist of only one LSHE and one LGFA module stacked	60.42
Standard full framework	The standard network architecture proposed in this work	71.22

From Table 11, the findings of this ablation study revealed the following results.

1. Impact of Simplifying the LGFE Module: The simplification of this module hindered its ability to aggregate multi-scale spatial features and global features of local areas by increasing the receptive field. This limitation led to a significant decrease in segmentation effectiveness. Hence, establishing a local neighborhood that encompasses multi-scale spatial context and effectively linking it with global information is essential for semantic segmentation.

2. Removal of the LSHE Module: The LSHE module had the second most significant impact on the overall performance of the model. The LSHE module is primarily responsible for the representation of local geometric pattern. Without this module, the subsequent network modules are difficult to effectively learn and interpret the context of each point.

3. Replacement of the Attention Pooling Module: The PAP module assigns a weighted attention score to different features, facilitating a more careful feature selection. Alternative methods used in place of attention pooling did not provide clear weight interpretation, resulting in less detailed attention to different feature channels.

5.1.2. Ablation of point cloud attribute extension

The point cloud attribute extension module conducted four ablation experiments to examine the impact of various combinations of spatial coordinates, color information and eigen-features. The different combinations of attributes and mIoU results are shown in Table 12.

Table 12. The mIoU scores of different point cloud attributes for ablation studies.

Ablated point cloud attributes	mIoU (%)
Only spatial coordinates	68.63
Spatial coordinates and color	70.16
Spatial coordinates and eigen-features	70.35
All attributes	71.22

The experimental findings demonstrate the effectiveness of incorporating color information and expanding point cloud attributes in the point cloud semantic segmentation of grotto scenes. Within these scenes, some components exhibit pronounced color patterns, such as ground, vegetation, and cliff walls. The integration of color information into the model provides more distinctive features, aiding in the differentiation between various classes. Additionally, through inputting eigen-features as additional attributes into the network, the representation of local geometry around each point is significantly enhanced. This enhancement is particularly noticeable when comparing the uniform geometric shapes of the ground and cliff walls with the more varied curvature found in Buddha statues. The point cloud attribute extension module improves the network to accurately identify the local geometric characteristics of different objects, resulting in an increase of segmentation accuracy.

5.1.3. Ablation of local spatial encoding

In this work, we introduced a novel spatial encoding method that integrates the Laplacian operator ${\delta }_i$, the relative position between a given point and its neighboring points $(p_i-p_i^k)$, as well as the Euclidean distance $p_i-p_i^k$. To assess the impact of different types of spatial information on the spatial encoding units, we tested seven distinct combinations of these encoding units. The results in Table 13 indicate that, among the three sub-encoding methods, the relative position information between a given point and its neighbors contributes most to the model. This is because this information intuitively provides the local geometric pattern of the point. Although the Laplacian operator was also used to describe local geometric features, it was designed resist to rotation and scale variations in an implicit manner. The contribution of the Euclidean distance is the least. The potential reason is that it fails to capture the subtle differences in local geometric shapes. Ultimately, combining these three sub-encoding methods together achieved the best performance.

Table 13. The mIoU scores of different spatial coding for ablation studies.

Ablated spatial coding	mIoU (%)
${\delta }_i$	52.75
$(p_i-p_i^k)$	60.04
$p_i-p_i^k$	45.90
${\delta }_i\oplus (p_i-p_i^k)$	60.38
${\delta }_i\oplus p_i-p_i^k$	66.84
$(p_i-p_i^k)\oplus p_i-p_i^k$	68.46
${\delta }_i\oplus (p_i-p_i^k)\oplus p_i-p_i^k$	71.22

5.1.4 Ablation of local-global feature aggregation

The role of the local-global feature aggregation module is to encompass global geometric features while increasing the receptive field. To evaluate the impact of LSHE, PAP and the concatenation of global feature representations on the overall network, we designed four ablation studies as shown in Table 14.

Table 14. The mIoU scores of ablated LGFA.

Ablated LGFA	mIoU (%)
One set of the LSHE module concatenated with the PAP module, without the addition of global feature	60.42
One set of the LSHE module concatenated with the PAP module, with the addition of global feature	61.14
Two sets of LSHE and PAP modules, integrated with global feature representations (standard module in GSS-Net)	71.22
Three sets of LSHE and PAP modules, integrated with global feature representations	66.10

As indicated by the results in Table 14, one set of the LSHE module concatenated with the PAP module fails to effectively increase the receptive field, leading to insufficient model performance in perceiving local multi-scale spatial context. Meanwhile, stacking three sets of LSHE and PAP modules increases the network complexity, but it requires more samples and longer training time for model convergence and is prone to overfitting, thus reducing its generalization ability. This explains why the performance of three stacked sets is inferior to two stacked sets. Moreover, integrating global features into the LGFA module enhances segmentation performance. This is mainly due to the correlation between different components in grotto scenes and their relative heights.

5.1.5. Ablation of the K-R method for selecting neighbor points

To compare the effectiveness of the K-R method with the KNN method, and the impact of the values of $K$ and $k$ on segmentation results, eight ablation studies with different parameter settings were conducted. As illustrated in Figure 18, the K-R method influences the effectiveness of semantic segmentation. However, selecting a larger value of $k$ can lead to overfitting issues due to the over broad selection area. Conversely, a too small value of $k$ may not sufficiently represent local geometric patterns. In this study, the best segmentation results were achieved when $K=32$ and $k=16$. This is because the K-R method considers more directions when selecting neighbor points and performs a dropout operation through random sampling to prevent overfitting caused by involving all points in the calculation. It is important to note that the optimal values for $K$ and $k$ are closely related to the density of the point clouds.

Figure 18. The influence of different parameter on mIoU by using the K-R method.

5.2. Limitations

Although extensive experimentation has demonstrated the superior efficacy of the proposed method in point cloud semantic segmentation for grotto scenes, there are still limitations to be addressed. The purpose of point cloud semantic segmentation of grotto scenes is modeling grottoes in 3D with semantic information for knowledge dissemination, which requires higher precision of segmentation and is less sensitive to segmentation efficiency. Therefore, compared with the segmentation accuracy, the computational efficiency of network modules is seldom considered in this study. In the expression of geometric patterns, local information often proves more crucial than global spatial information. The effective representation of point cloud contextual information significantly enhances the results of semantic segmentation. This is because global features are more likely to be affected by application scenarios. Therefore, based on existing semantic segmentation models, incorporating more global features to enhance the perception of global patterns can improve the segmentation accuracy in certain scenes. Additionally, it is effective to adopt knowledge as guidance to improve the segmentation results after deep learning. This approach requires the predefined knowledge specific to grotto scenes, such as the threshold for the size of grotto components and relative elevation. Such strategy holds the potential to evolve into an independent and more universal module for improving the segmentation results, and is applicable as a post-processing step in deep learning as well. Lastly, grottoes are not solely man-made artistic creations, but their geometric shapes are also influenced by natural environment. Therefore, the applicability and performance of our proposed method across different types of grottoes require further validation.

6. Conclusion

In this study, we proposed a method of point cloud semantic segmentation of grotto scenes with good performance, which is based on knowledge guidance and a deep learning model. The knowledge, including point cloud density knowledge, visual color knowledge, local geometric feature knowledge, global spatial distribution knowledge, threshold for the size of grotto components and grotto components knowledge, guided the construction of the grotto scene point cloud benchmark dataset, the design and improvement of neural network, and the improvement of segmentation results. With the guidance of these knowledge and the reference of RandLA-Net, we built a deep learning model named GSS-NET to make it more suitable for grotto scenes. This model more effectively captures the local and global spatial geometric patterns in grotto scenes and incorporates neural network modules for efficient selection and aggregation of point clouds contextual information. The comparative experiments with four deep learning models on our self-constructed point cloud benchmark dataset demonstrate the advanced performance of our proposed method. The knowledge constraints further improve the segmentation results. In the inter-scene experiment, it achieved an OA of 83.53% and a mIoU of 71.22% with the GSS-Net. Through applying knowledge constraints for result improvement, the OA and the mIoU increased to 90.28% and 76.47%. A set of ablation studies reveal the effectiveness of model design and improvement under knowledge guidance, aiding explain the contribution of each module to the entire deep neural architecture for point cloud sematic segmentation. The GSS-Net proposed in this study adopts Pointwise MLP as the basic unit. The spatial coding layer, pooling layer, and feature aggregation layer as sub-modules in our proposed model can be used in other MLP-Based methods. For example, the Laplace operator is capable of representing the local geometric properties of the point clouds. The parallel attention pooling can capture contextual information in a soft way. The global features enable the neural network to perceive the spatial distribution of features. In addition, the first four of the six GSKs we summarized in this study are applicable to grotto scenes and have certain generalization to deal with those scenes that have similar features, such as complex geometry, uneven density distribution, color being important, and relative height being correlated with category. In the future, our goal is to expand the benchmark dataset of grotto scenes to evaluate the generalizability of this method across a broader range of grotto scenes.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The point cloud used in the study cannot be shared publicly due to restrictions.

Additional information

Funding

This research was funded by the National Natural Science Foundation of China [grant number 42171444].

Notes

1 https://whc.unesco.org/en/list/1003/ (accessed on 15 December 2023).

2 https://whc.unesco.org/en/list/1039/ (accessed on 15 December 2023).

3 https://whc.unesco.org/en/list/912/ (accessed on 15 December 2023).

4 https://www.cloudcompare.org/ (accessed on 14 January 2024).