3D Path prediction of moving objects in a video-augmented indoor virtual environment

ABSTRACT

Augmented virtual environments (AVE) combine real-time videos with 3D scenes in a Digital Earth System or 3D GIS to present dynamic information and a virtual scene simultaneously. AVE can provide solutions for continuous tracking of moving objects, camera scheduling, and path planning in the real world. This paper proposes a novel approach for 3D path prediction of moving objects in a video-augmented indoor virtual environment. The study includes 3D motion analysis of moving objects, multi-path prediction, hierarchical visualization, and path-based multi-camera scheduling. The results show that these methods can give a closed-loop process of 3D path prediction and continuous tracking of moving objects in an AVE. The path analysis algorithms proved accurate and time-efficient, costing less than 1.3 ms to get the optimal path. The experiment ran a 3D scene containing 295,000 triangles at around 35 frames per second on a laptop with 1 GB of graphics card memory, which means the performance of the proposed methods is good enough to maintain high rendering efficiency for a video-augmented indoor virtual scene.

KEYWORDS:

• Augmented virtual environment (AVE)
• Digital Earth platform
• indoor
• moving object
• 3D path prediction

1. Introduction

Augmented virtual environments (AVE) combine real-time videos with 3D scenes in a Digital Earth System. The combination provides dynamic information about the reality being represented by the virtual scene simultaneously by fusing video images with 3D models and visualizations of moving objects (Neumann et al. 2003; Jian et al. 2017). A wide field of view AVE created by fusing several separate or overlapping videos into the same virtual scene helps give a complete global view and makes it easier to understand the relationships between these videos and the changing situation in the real scene without using separate monitors to display videos. The fusion of dynamic images and 3D models can also display objects that are not part of the static 3D scene, such as trees, lamp poles, cars, pedestrians, and even their 3D trajectories. In addition to the fusion of video textures and visualization of moving objects in AVEs, object-oriented analysis is gaining more attention in practical applications to provide solutions in the real world, such as continuous tracking of moving objects, camera scheduling, and path planning. The moving objects’ images, 3D coordinates, and trajectories generated in an AVE are all crucial for object identification, 3D location, motion analysis, and 3D spatial analysis, which allow us to do 3D video surveillance, resource management, and decision making in a Digital Earth System.

3D path analysis is a useful method to search for and visualize the optimal path in a 3D scene and is commonly used in path planning, path routing, and navigation of pedestrians, vehicles, and UAVs. Combining path analysis methods and the objects’ coordinates generated in a video-augmented scene can help find the shortest or optimal path dynamically by taking an object’s position as a start or end point. This combination will play a role in predicting moving objects’ future trajectories and scheduling the related cameras in a 3D virtual scene, which is important in saving decision time, camera resources, and computational resources when assisting with decision support for 3D object tracking in an actual environment.

Based on the information in an AVE (foreground images, 3D coordinates, trajectories, object IDs, annotations, etc.), this paper focuses on analyzing the 3D motion of a moving object and predicting the paths it might take in a video-augmented virtual scene so as to schedule surveillance cameras in the real word and track objects continuously. This paper is organized as follows. The next section introduces several related studies, including research on AVEs, path analysis, path prediction and camera scheduling. Section 3 introduces the type, usage, and processing of the data used in this study. Section 4 describes the proposed methods in detail, including 3D motion analysis of moving objects, multi-path prediction and hierarchical visualization, and path-based multi-camera scheduling. Section 5 presents the results of multi-path prediction of a moving object and camera scheduling in an AVE, and analyzes the methods’ efficiency. Finally, the conclusion and future improvements are elaborated in Section 6.

2. Related work

The path prediction in an AVE to be carried out in this research involves video fusion, object-related 3D visualization, path analysis, path prediction, and camera scheduling.

2.1. Video fusion and object visualization in AVEs

Research on augmented virtual environments mostly focuses on the fusion of 3D scenes with video images and moving objects. The concept of AVE was first presented to fuse dynamic imagery with 3D models to support information extraction and analysis for an improved understanding of the scene (Neumann et al. 2003). A GIS-based system for augmented video surveillance (Milosavljević, Dimitrijević, and Rančić 2010) was developed to register geospatial data to outdoor video surveillance systems and retrieve information about viewed geospatial objects. A further integration of 3D GIS and video surveillance systems using augmented reality (AR) techniques has also been discussed (Milosavljević et al. 2016; Ma et al. 2018). Two models of integration (GIS-augmented video and video-augmented GIS) have been defined and a prototype has been implemented to serve as a basis for analyzing possible applications of real-world systems based on the integration of GIS and video. A topology-accounting method for multi-video fusion with 3D GIS scenes (Li et al. 2021) was proposed to project videos to topological models and rapidly identify rendered objects in 3D scenes.

As an important part of video, moving objects are extracted from videos and visualized as separate models in a 3D scene to prevent video distortions and give a better comprehension of both videos and virtual scenes. In an AVE system (Sebe et al. 2003), the moving objects are extracted and visualized as pseudo-3D elements to help observers comprehend videos and images from arbitrary views of the 3D scene. The moving objects were rendered to view-dependent texture-mapped billboards in a multiple sparse camera-based free view video system (Silva, Santos, and Morimoto 2011) so as to observe real scenes in a 3D virtual environment from any desired viewpoint. A realistic AVE with a wide field of view (Jian et al. 2017) was established by fusing multiple video textures together, visualizing a moving object as a 3D rectangle and drawing its trajectory along with the 3D object in a 3D scene. A prototype of the GIS and Moving Objects (GIS–MOV) system (Xie et al. 2017) was constructed using motion detection, moving object extraction, spatial mapping, and fusion patterns for GIS and moving object techniques. The integration of multi-camera video moving objects with GIS (Xie et al. 2019) can assist users in browsing video objects from multiple cameras.

The current developments of AVE-related video fusion and moving object visualization are quite good. Most research focuses on how to integrate the 3D scenes with video images or moving objects, subsequently visualizing what they have fused. However, there is a lack of analysis related to moving objects to serve practical solutions in the real world.

2.2. Path analysis and path prediction

Path analysis is a useful procedure in GIS for finding an optimal route between two points through continuous space that minimizes costs, and it can arise in various applications such as indoor and outdoor navigation, emergency evacuation, and path planning (Atila, Karas, and Abdul-Rahman 2013; Lin et al. 2013; Yang et al. 2016). Numerous path analysis algorithms have been implemented, especially in pgRouting, such as Dijkstra, Bi-directional Dijkstra, A* and Bi-directional A* (pgRouting Contributors 2018). With rich geospatial information, GIS has inherent resource advantages to carry out spatial path analysis for prediction of moving objects (pedestrian, cyclist, vehicle, metro flows, etc.) and mode choice (Cooper and Chiaradia 2020), urban travel behavior analyses and route prediction (Sun et al. 2014), and GIS vehicle path selection and routing (Wang and Wang 2014; Vu et al. 2019). Furthermore, path prediction methods in the field of computer vision have been mainly applied to predict the trajectory of moving objects in camera view for better obstacle avoidance, automatic driving and autonomous navigation (Russakovsky et al. 2015; Serrao et al. 2015; Li et al. 2017; Kooij et al. 2019).

However, the practice of using AVEs to conduct path analysis and path prediction remains limited. Combined with moving objects’ trajectories and 3D path networks, path analysis methods can play an important role in a video-augmented virtual scene, such as equivalent cost path analysis and optimal path planning for moving objects.

2.3. Camera scheduling

Camera scheduling is a key part of surveillance systems because the target in a specific area cannot be fully monitored even by a set of smart cameras without camera scheduling decisions at a given point in time (Jaenen et al. 2013). Especially in intelligent video surveillance systems that require different kinds of information processing within a strict time limitation, camera scheduling can allow system designers to flexibly assign execution times and application-derived utilities to tasks, track objects simultaneously through multiple cameras, and finally maximize total system utility (de Niz et al. 2013; El Maadi and Djouadi 2016). In the field of computer vision, camera scheduling methods have been widely used in multiple-object tracking (Ward and Naish 2009; Shih et al. 2012; Hong et al. 2020), surveillance area coverage selection (Guo et al. 2015), resource cost minimization (Gao et al. 2015; Aghdasi and Abbaspour 2016), and effective surveillance (Lin et al. 2016; Kumari et al. 2020). However, there are few studies on camera scheduling in AVEs and the integration of video and GIS. An AVE has 3D spatial analysis and simulation capabilities, as well as rich data resources such as 3D models and path networks, making it an innovative and helpful tool to schedule cameras and optimize resources in a video-augmented scene.

3. Data preparation and processing

The method in this study begins after the 3D visualization of a moving object, and focuses on motion analysis, path prediction, and camera scheduling guidance. The data used in 3D visualization of a moving object and the output data (coordinates, trajectories, etc.) are all taken as input data for this study. The introduction, usage, and processing of these data are listed in Table 1.

Table 1. Introduction, usage, and processing of the data used in this study.

Data	Description	Usage	Processing
3D models	3D models of the test site, created by modeling software with photos, CADs and field measurement data	Basic virtual scene	Add the 3D models to a Digital Earth System according to their 3D geo-coordinates
Cameras and videos	Surveillance cameras at the test site and the videos they captured	3D object visualization, camera scheduling	Store camera parameters (positions, orientations, attributes) into database; create camera models and add to 3D scene; moving object extraction and 3D visualization
3D path network	3D paths made up of indoor corridors, stairs, and elevator shafts	3D path analysis	Store the path network into a database and create topology for the paths
3D coordinates of the moving object	3D coordinates of the object calculated during 3D object localization and visualization	Object localization, 3D path analysis	Take the coordinates as the start or center point when searching nearest nodes and paths
3D trajectories	3D trajectories of the moving object	3D motion analysis	Draw them as 3D lines in the virtual scene

The outdoor and indoor scenes of the test site were digitized into 3D models and loaded on a Digital Earth system according to their geographical coordinates. The surveillance cameras’ parameters (positions, orientations, attributes) were stored in a database and the camera models and annotations were also added to the 3D scene. Figure 1 shows the 3D virtual scene, camera annotations, and the captured video examples on the 5th and 8^th floors.

Figure 1. 3D virtual scene, camera annotations, and video examples. (a) 3D models of the test site and the camera annotations. (b) A video frame on the 5th floor. (c) A video frame on the 8th floor.

Extracting a moving object and visualizing it in the 3D virtual scene are basic preparations for 3D motion analysis and path prediction. In this study, a moving object was extracted from surveillance videos by a fast video object tracking method composed of a multi-threshold improved Gauss Mixed Model background difference algorithm and a foreground optimization algorithm based on HOG features and the SVM classifier. Then the extracted moving object was visualized as a rectangular polygon model and relocated in the augmented virtual scene by a moving object visualization method, including 3D modeling for the object, 3D coordinate calculation, model transformation, object texture projection, 3D position updates, and drawn trajectories (Jian et al. 2017). As shown in Figure 2, the moving object model with a foreground image was located and updated according to the video data captured from the same place as Figure 1(b). The sequential geographic coordinates were drawn out as a 3D line (trajectory) while updating the 3D position of the object model, and the object name was annotated along with the object model.

Figure 2. 3D visualization of a moving object. (a) Horizontal view. (b) Top view.

In order to apply 3D path analysis and path prediction, the 3D indoor paths were drawn in the virtual scene and imported into PostGIS (PostGIS Development Group 2021), which was used to create the topology of the indoor paths. This indoor path network contained 101 edges and 88 nodes, as shown in Figure 3 with 3D lines and points.

Figure 3. Indoor path network in the 3D virtual scene. (a) Overall view. (b) Edges and nodes.

4. Methodology

4.1. 3d motion analysis of moving objects

Trajectories generated during the 3D visualization of moving objects contain a lot of motion information, such as the 3D geo-coordinates, velocity, and direction of every moment, which can be used for motion analysis and path prediction of moving objects. As shown in Figure 4, a 3D trajectory t is composed of n geo-coordinates v_i (i=1, 2, … , n) and v_n is the 3D geo-coordinates when the moving object left the camera view. The velocity $|V_n|$ and the vector of movement direction $\overrightarrow{V_n}$ are calculated as follows: (1) $|V_n|={\displaystyle \frac{\sqrt{{(v_n.x-v_{n-1}.x)}^2+{(v_n.y-v_{n-1}.y)}^2}}{T_n-T_{n-1}}}$(1) (2) $\overrightarrow{V_n}=v_n-v_{n-1}$(2) where x, y are the values of longitude and latitude and T_i (i=1, 2, … , n) is the time of v_i. The vector of movement direction $\overrightarrow{V_n}$ is represented as a dotted arrow line in Figure 4.

Figure 4. Example of motion analysis of a trajectory.

The angle of a movement direction in a 3D scene is represented the same way as a 3D model: the object’s geo-coordinates are taken as the origin, the East direction as the X axis, and the North direction as the Y axis. An angle of a movement direction ranges within [−180, 180], as shown in Figure 5.

Figure 5. Angle representation of a movement direction.

4.2. Multi-path prediction and hierarchical visualization

Multi-path prediction aims to calculate some possible paths that a moving object may take after leaving the camera view. The 3D path network plays an important role in path prediction because it contains all the places that people can go (e.g. corridors, stairs, and elevator shafts). Combining the 3D trajectories, 3D path network, and results of the motion analysis, this paper proposes a multi-path prediction and hierarchical visualization method to predict some possible paths that a moving object may take after leaving the camera, sort these paths by their probabilities, and visualize them with different colors.

The workflow of multi-path prediction and the hierarchical visualization method is shown in Figure 6, mainly composed of four parts: nearest neighbor node search, equivalent cost path analysis, angle calculation and sorting of multiple paths, and hierarchical visualization.

1. Nearest neighbor node search.

Figure 6. Workflow of multi-path prediction and hierarchical visualization.

Path analysis needs a start node that exists in the path network while the last point of a 3D trajectory (last place when the object leaves the camera view) is generally not in the path network. It is necessary to search the node closest to the last point of the trajectory—a 3D nearest neighbor search problem. Nearest neighbor is an important query and analysis method for spatial applications in computer science and geospatial science, and numerous techniques have been developed due to its popularity and potential (Suhaibah et al. 2016). In this study the last point of the 3D trajectory is taken as a center and one or more neighbor nodes of the path network is searched in the 3D space. The closest one is kept as the nearest neighbor node.

• (2) Equivalent cost path analysis.

Taking the nearest neighbor node as the starting point, the possible paths and nodes that satisfy the condition of equal cost (equal time, equal distance, etc.) are searched in all directions. Since the nearest neighbor node is connected to edge(s) in the path network, there is at least one path that satisfies the condition.

• (3) Angle calculation and sorting of multiple paths.

The orientation angles are calculated for the searched paths that start from the nearest neighbor node. The angle representation method is the same as 3D motion analysis of moving objects. The intersection angles between the movement direction and each orientation angle of the possible paths are calculated, and the multiple paths are sorted according to these intersection angles.

• (4) Hierarchical visualization.

A hierarchy of classification and visualization is defined for the multiple possible paths. For example, the paths can be classified according to the intersection angles and visualized with different colors.

Hierarchies of classification and visualization for the multiple paths are shown in Figure 7 and Table 2. According to the intersection angle between the movement direction and the path orientation, the multiple paths are classified into three groups (Classes S1, S2, and S3) and visualized with different colors (red, magenta, and orange) that indicate the possibility of a path being adopted.

Figure 7. A hierarchy of classification and visualization for multiple paths.

Table 2. A coloring scheme according to intersection angles.

Class	Intersection angles	Example	Coloring scheme
S1	0° ∼ 60°	$\overrightarrow{OA}$	Red (255,0,0)
S2	60° ∼ 120°	$\overrightarrow{OB}$	Magenta (255,0,255)
S3	120° ∼ 180°	$\overrightarrow{OC}$	Orange (255,192,0)

An example of multi-path prediction and hierarchical visualization is shown in Figure 8, where V is the last point of the trajectory t when a moving object left the camera view; $\overrightarrow{VF}$ is the vector of the movement direction; node O is the nearest neighbor node to point V; $\overrightarrow{OP}||\overrightarrow{VF}$; and $\overrightarrow{OA}$, $\overrightarrow{OB}$ and $\overrightarrow{OC}$ are the results of equal cost path analysis, that is, the possible paths which may be taken by the object. According to the classification and visualization rules, the intersection angle of $\overrightarrow{OA}$ and $\overrightarrow{OP}$ is less than 60 degrees (Class S1), so $\overrightarrow{OA}$ is the most likely path that the object may take, visualized as a red line. The intersection angle of $\overrightarrow{OB}$ and $\overrightarrow{OP}$ is between 60 and 120 degrees so it is classified as Class S2 and colored with magenta. In the same way, $\overrightarrow{OC}$ is classified as Class S3 and colored with orange, which means it is the least likely to be chosen.

Figure 8. Example of multi-path prediction and hierarchical visualization. (a) Nearest neighbor node search, equal cost path analysis, and intersection angle calculation; (b) hierarchical visualization of multiple paths.

4.3. Path-based multi-camera scheduling

Camera scheduling and continuous object tracking are important tasks in video surveillance. A good camera scheduling method will help economize resources and track moving objects. Combining the predicted paths and cameras’ 3D coordinates, this paper proposes a path-based multi-camera scheduling method to predict which cameras the object may pass through and dispatch these cameras to detect moving objects specifically.

1. Establish the visibility relationships of camera positions and nodes of the 3D path network. Calculate which path nodes can be seen by a camera according to its 3D coordinates, attitude angles and field of view, and store the relations of node IDs, camera ID and camera preset angle in a database table. The main attribute fields of the relation table are shown in Figure 9.

2. Search the cameras that the moving object may go through. These camera IDs can be queried from the relation table according to the possible path nodes calculated by the multi-path prediction and hierarchical visualization method.

3. Dispatch the chosen cameras and apply object tracking methods to them. For the static cameras, the attitude angles are fixed; the cameras with pan, tilt, and zoom (PTZ) are adjusted according to the preset angles in the relation table.

Figure 9. The main attribute fields of the relation table.

A diagram of this path-based multi-camera scheduling method is shown in Figure 10. C1 to C6 are cameras and the fields of view (FOV) are drawn as light green; the blue lines and points are paths and nodes; the thick cyan line is a moving trajectory; the vector of movement direction is drawn as a black dashed arrow; and the predicted paths and nodes are colored with red. According to the predicted nodes and the visual relations between cameras and path nodes, the cameras that the moving object may go through are C1, C2, and C3, which are marked as chosen cameras and listed as the focus of object tracking.

Figure 10. Diagram of path-based multi-camera scheduling.

5. Result and discussion

5.1. Motion analysis and multi-path prediction

This study predicted multiple possible paths a moving object might take after leaving a camera’s view using the proposed 3D motion analysis, multi-path prediction, and hierarchical visualization methods, so that they can be used for camera scheduling and continuous object tracking.

Figure 11 illustrates the process of 3D motion analysis of a moving object at a top view of Figure 2(a). The cyan polyline is the moving object’s trajectory composed of n geo-coordinates in which $V_{n-1}$ and $V_n$ are the last two points. The vector of the movement direction was carried out by ($V_{n-1}$, $V_n$) and shown as a yellow arrow, which is 22.3 degrees in the 3D scene.

Figure 11. 3D motion analysis of a moving object.

Taking the last point $V_n$ as the center, a 3D nearest neighbor node search method was used to get the 3D path node closest to the center, which is point O shown in Figure 11. With point O as the starting point (N₂), all nodes and paths that meet the condition of equal time cost in all directions were searched in the 3D path network. All angles of the searched paths around point N₂ were calculated and compared with the vector of the movement direction, then visualized with different colors by the proposed path sorting and hierarchical visualization method. The paths from point N₂ to the end nodes in each direction are listed in Table 3 as well as the intersection angles and coloring schemes. As shown in Figure 12, the position where the moving object left the camera’s view is highlighted by a yellow cylinder; the blue point annotations are the end points of the predicted paths that the object might take; and the paths are visualized with red, magenta, and orange according to the angles between the movement direction and start paths (from N₂ to the second point in each path). The start angle of red paths (from N₂ to an elevator node) is similar to the movement direction so they are most likely to be taken by the moving object, meaning the person will take an elevator up or down. The start angle of magenta paths (to go up or down the stairs) is approximately perpendicular to the movement direction so their possibilities come next. As the yellow paths are opposite to the movement direction, they are the least likely to be chosen.

Figure 12. Hierarchical visualization for multiple predicted paths.

Table 3. Details of multiple predicted paths and coloring schemes.

ID	Path	Intersection angles	Coloring scheme
1	P (N2, N9, … , N53)	0.5°	Red
2	P (N2, N9, … , N55)
3	P (N2, N9, … , N57)
4	P (N2, N9, … , N95)
5	P (N2, N9, … , N97)
6	P (N2, N9, … , N99)
7	P (N2, N9, … , N139)
8	P (N2, N9, … , N143)
9	P (N2, N9, … , N184)
10	P (N2, N9, … , N186)
11	P (N2, N9, … , N188)
12	P (N2, N9, … , N191)
13	P (N2, N9, … , N229)
14	P (N2, N9, … , N231)
15	P (N2, N9, … , N234)
16	P (N2, N9, … , N275)
17	P (N2, N9, … , N279)
18	P (N2, N9, … , N361)
19	P (N2, N3)	91.5°	Magenta
20	P (N2, N1, … , N82)	88.5°
21	P (N2, N1, … , N261)
22	P (N2, N12, … , N41)	179.5°	Orange
23	P (N2, N12, … , N48)

5.2. Camera scheduling and continuous object tracking

According to the visibility relationships of camera positions and 3D path nodes, the camera nodes that can see the predicted nodes are queried and given special surveillance for moving objects. As shown in Figure 13, the red icon annotations are the chosen cameras along the predicted paths and nodes.

Figure 13. The cameras corresponding to the predicted path nodes.

In order to clearly show the visibility relationships between a camera's field of view and a 3D path, this paper uses a translucent pyramid to represent a camera's visual range, and displays it together with the 3D path model created by a quaternion-based piecewise 3D modelling method for indoor path networks (Jian et al. 2019). As shown in Figure 14, the translucent pyramids are created by the cameras’ 3D coordinates and attitude parameters, and the four pyramids on the right represent the cameras monitoring the elevators on the 1^st, 3^rd, 5th, and 8^th floors.

Figure 14. The visibility relationship between the camera's visual range and the 3D path model.

These chosen cameras were arranged to see the predicted paths and nodes, then the object tracking methods were applied to them. When the moving object is detected again in one of these cameras, it is continuously tracked and visualized using the proposed 3D motion analysis, multi-path prediction, hierarchical visualization, and path-based multi-camera scheduling methods. For example, after the object walked out of view of the camera on the 5th floor (Figure 2), it was detected and re-identified by the camera on the 8^th floor, and visualized in the 3D scene as shown in Figure 15. The above steps are repeated to predict the possible paths and track the moving object continuously, forming a closed loop of 3D path prediction and continuous tracking of moving objects in a video-augmented indoor virtual environment.

Figure 15. Continuous tracking and 3D visualization of the moving object. (a) Camera’s view. (b) Side view.

5.3. Efficiency analysis

Accuracy and time consumption of path analysis as well as the performance of the 3D path prediction methods will affect the fluency of object tracking and the 3D rendering efficiency in a video-augmented indoor virtual environment.

In this study, four path search algorithms, Dijkstra, Bi-directional Dijkstra (bdDijkstra), A* and Bi-directional A* (bdA*), were tested to predict multiple paths under three conditions when the cost equaled 50, 60, and 70 meters. The results showed that all four algorithms searched the same paths in the same cost situation, as shown in Figure 16(a–d), and obtained the appropriate paths according to different conditions, as shown in Figure 16(a,e,f). This is because once a weight was assigned to each path in the network, the result of the equivalent cost path analysis could be mathematically certain, and different algorithms could consume different time to obtain the same result.

Figure 16. Multiple path search by four algorithms under three cost conditions.

For the time consumption of the path analysis, the algorithms were compared in two situations when elevators were available and unavailable. As shown in Figure 17, the algorithms cost less than 1.3 ms to get the optimal path, which means the path search algorithms can guarantee accuracy and real-time optimal path searches and not increase the burden of 3D scene rendering.

Figure 17. Time consumption of path search algorithms.

The frame rates of a video-augmented indoor scene with 295,000 triangles were recorded when running the proposed methods on a laptop with an NVIDIA GeForce GT 730M graphics card with 1 GB of memory. As shown in Figure 18, the frame rate remained around 35 fps during 3D visualization and motion analysis of a moving object, prediction and hierarchical visualization of multiple paths, path-based multi-camera scheduling, and continuous object tracking. This result shows the proposed 3D motion analysis, 3D path prediction, and camera scheduling methods can meet the needs of real-time, efficient rendering of a video-augmented indoor virtual scene.

Figure 18. Frame rates of 3D path prediction and continuous tracking of moving objects in a video-augmented indoor virtual scene with 295,000 triangles.

6. Conclusion

This paper proposed a novel approach for 3D path prediction of moving objects in a video-augmented indoor virtual environment, which includes 3D motion analysis of moving objects, multi-path prediction and hierarchical visualization, and path-based multi-camera scheduling. These methods take video and 3D trajectories of a moving object as input data, analyze the moving object’s motion, predict and visualize the possible paths the moving object may take, and schedule the cameras in the virtual scene to provide solutions for continuous object tracking in the real world. The results show that these methods can give a closed-loop process, from 3D visualization of a moving object to 3D motion analysis, path prediction, camera scheduling, and 3D visualization of the moving object again. The path analysis algorithms proved to be accurate and fast, and the performance of the proposed method is good enough to maintain high rendering efficiency of a video-augmented indoor virtual scene.

The proposed approach was implemented with a test in this study, so there is still a long way to go before it can be applied in practice. As input conditions, the precision of the 3D path network and the accuracy of moving object localization determine the reliability of the predicted path. In practical applications, it is necessary to ensure that the 3D path network can cover all the entrances and exits and indoor paths, and that the algorithms for tracking and extracting moving objects from video are as accurate and efficient as possible. Furthermore, multiple objects should be taken into consideration because it is a common situation to track multiple objects at the same time or to track specific targets in a crowd. The authors are currently referring to mask-based deep learning object tracking methods such as Mask R-CNN (He et al. 2017) and SiamMask (Wang et al. 2019) to track and extract multiple objects accurately, and this will be discussed in future work.

Acknowledgements

The authors would like to thank the editors and anonymous reviewers for their valuable comments and suggestions. We also would like to thank Xiaoping Du, Jian Liu, Qin Zhan and Junjie Zhu for helpful discussions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

Supplemental data for this article can be accessed at https://doi.org/10.6084/m9.figshare.14999175.v2.

Additional information

Funding

Research presented in this paper was supported by the National Natural Science Foundation of China [grant number 41901328 and 41974108], the Strategic Priority Research Program of the Chinese Academy of Sciences [grant number XDA19080101], the National Key Research and Development Program of China [grant number 2016YFB0501503 and 2016YFB0501502].