UAV photogrammetry for red tide monitoring and geo-localization via onboard GNSS/IMU data

Abstract

As one of the worst ecological disasters in the world’s oceans, red tide seriously threatens marine ecology. Red tide monitoring is essential for preserving the ocean ecosystem. However, traditional photogrammetric processing workflows, e.g. SfM (Structure from Motion) and MVS (Multi-view Stereo) based image orientation and dense matching, do not apply to offshore images due to the low texture of ocean color. The primary contribution of this study is a UAV (Unmanned Aerial Vehicle)-based photogrammetric solution for red tide monitoring and direct geo-localization. First, a direct geo-localization model has been created, which solely uses onboard sensor data for the 3D coordinate calculation of 2D image targets based on the collinear equation. Second, two UAVs are chosen as photogrammetric systems that can supply the necessary data for the direct geo-localization model, including high-resolution images, camera intrinsic and extrinsic parameters. Third, a human annotation technique has been utilized to identify red tide regions whose ground polygons can be further estimated using the direct geo-localization model, taking into account how difficult it is to detect red tides automatically. Finally, the accuracy of direct geo-localization and the viability of the proposed solution has been evaluated and verified using real UAV datasets. The experimental results show that the direct geo-localization precision is better than 20 m when only onboard sensor data is used. The suggested workflow is appropriate for monitoring red tides, which can provide critical information for managing marine ecosystems. The executable toolkit would be made publicly available at https://github.com/json87/PhotoDigitizer.

Keywords:

• Unmanned aerial vehicle
• oblique photogrammetry
• red tide monitoring
• direct geo-localization
• structure-from-motion
• 3D reconstruction

1. Introduction

Red tide, also referred to as algal blooms, is one of the major ecological disasters in the world’s oceans and are brought on by the frenzied growth of bacteria, protozoa, or phytoplankton. It poses a serious threat to the world’s marine ecology (Maze et al. 2015). As a result, it is critical to regularly monitor red tide in order to assess, forecast, and maintain the security of the marine ecosystem and its ability to support economic activity.

As seen in Figure 1, the alteration in ocean color is primarily to blame for the occurrence of red tide. It offers optical image-based remote sensing (RS) red tide monitoring. Due to the rapid advancement of onboard sensors, such as the Chinese GaoFen-1 Wide Field of View (GF-1 WFV) and the HaiYang 1D Coastal Zone Imagery (HY-1D CZI), and the capability of high-orbit satellite platforms for large-scale observation, satellite optical images have been used extensively in the literature for red tide monitoring (Liu et al. 2022). The main concept is to create a mapping model that links the spectrum reflectance of RS photos to the presence of red tide. Sakuno et al. (2019) created a red tide detection model using the Sentinel-2 MSI (Multispectral Instrument) images by merging the data of bands 4 and 5. They then validated the model for red tide monitoring over a period of five months. Shin et al. (2021) adopted the PlanetScope images with a spatial resolution of 3 m for red tide monitoring in consideration of the constrained spatial resolutions of low and medium RS sensors. An U-Net based deep learning model was trained and used in research to forecast the location of red tides, and the findings are encouraging when compared to traditional hand-crafted techniques. Similarly, in order to overcome the problem of adapting deep learning models for red tide detection from low spatial resolution to medium spatial resolution images, Zhao et al. (2021) created the RDU-Net CNN (Convolutional Neural Network) model based on the U-Net architecture. HY-1D CZI images were used to train and validate the model. However, satellite platforms and used images have some limitations. The fixed orbits of satellites limit their capacity to respond quickly enough to acquire data effectively. At the same time, their highly high height reduces the spatial resolution of RS images. For the purpose of monitoring red tides, it is therefore necessary to increase data acquisition efficiency and image resolution.

Figure 1. RS images captured from the HY-1D satellite and UAV platforms.

Over the past decade, Unmanned Aerial Vehicles (UAVs) have undergone significant advancements in photogrammetry and remote sensing, primarily due to their cost-effectiveness and versatile data acquisition capabilities. The use of these applications with UAVs facilitates the construction of 3D models. It enables comprehensive observation and analysis of various types of ranging from forest resource evaluation (Lin et al. 2015; Ecke et al. 2022) to the geological disasters that occur worldwide, such as debris flow, rockfall, flooding, earthquakes, coseismic ruptures, and landslides (Francioni et al. 2020; Cirillo et al. 2024), and transmission line inspection (Jiang et al. 2017). UAV imagery has emerged as one of the most valuable sources of remote sensing (RS) data, particularly notable for its application in red tide monitoring (Shin et al. 2021). The availability of UAVs equipped with consumer-grade cameras has revolutionized data acquisition in photogrammetry and remote sensing, offering cost-effective solutions for diverse applications. Through the comprehensive utilization of various photographic lenses, along with oblique photography techniques capturing images from multiple angles, high-resolution imagery can be rapidly acquired and precisely geolocated using RTK and PPK photogrammetric surveys in certain cases (Štroner et al. 2021; Cirillo et al. 2022). These methodologies enable the generation of 3D point clouds with exceptional accuracy, as well as orthophoto maps and digital terrain models (James and Robson 2012), effectively minimizing systematic errors in topographic models (James and Robson 2014). Cheng et al. (2020) suggested a UAV-based solution to monitor coastal algal blooms to avoid the impact of weather conditions on satellite images. High-resolution UAV images were spatially registered to create an overview of test sites, and the radiometric correction was carried out to generate preferred pixel values for algal bloom regression. High spatial resolution UAV images were employed in Jiang et al. (2020b) to identify green tide, and a green-tide index, known as RGB-FAI (red-green-blue floating algae index) was created by exploiting the red and blue bands of RGB images. Kim et al. (2020a) evaluated the potential of UAV platforms and multispectral sensors to map red tide intensity, in which a 5-band camera was utilized to take images, and the RBR (red-to-blue band ratio) algorithm was used to calculate the intensity values. To address the problem of band registration of multispectral images, Kim et al. (2020b) used multispectral sensors in water monitoring, in which images were captured using a UAV fitted with a multispectral camera, and the correspondences between image bands were determined using the radiometric distribution of pixels. To overcome the limitations of satellite-based systems, UAVs show tremendous possibilities for effective data collecting and quick emergency reactions in red tide monitoring.

Although UAV platforms feature flexible data acquisitions and high spatial resolutions, there are several clear drawbacks when it comes to extensive red tide monitoring. On the one hand, the energy sources used, such as lithium batteries, limit the endurance of UAV platforms, which are usually less than 30 min for the majority of market-available rotor-wing UAVs. For the collection of outdoor data, many flying campaigns may be necessary; on the other hand, the configured dynamical system typically restricts the flight height of UAV platforms, which is close to 100 m for commonly used rotor-wing UAVs (Colomina and Molina 2014). When compared to satellite images, collected images have a limited amount of ground coverage. In the absence of these two elements, the gathered UAV images cannot pinpoint the location of red tide since they are not geo-referenced (Helgesen et al. 2019). For extensive red tide monitoring, image registration and geo-localization are therefore necessary.

SfM (Structure from Motion) (Jiang et al. 2020a) and MVS (Multi-view Stereo) (Dall¡¯Asta et al. 2017) are the commonly used techniques in the fields of photogrammetry and computer vision to resume camera poses and generate dense point clouds, which are then used to achieve image registration and geo-localization of UAV images. To successfully align photos and perform dense matching, it is necessary to establish enough and precise correspondences between overlapping images. However, because most of UAV images used for red tide monitoring are obscured by low-texture ocean water, as seen in Figure 1(b), it becomes difficult to monitor red tides. In other words, red tide monitoring cannot use the traditional workflow for UAV images. Fortunately, POS (Positioning and Orientation system), which can be used to implement camera extrinsic orientation, is integrated with the majority of market-available UAV platforms. Additionally, pre-calibrated parameters can be used to establish camera intrinsic orientation. As a result, this study proposes a workflow for monitoring and geo-localizing red tides using UAV-based photogrammetry. In conclusion, the main contributions of this study are as follows: (1) we design a workflow to achieve the image orientation of UAV images and the geometric calculation of red tide regions by using onboard POS data and camera parameters. It does not rely on the SfM-MVS based workflow and avoids the low-texture drawback of recorded images; (2) we analyze the accuracy of the direct geo-referencing solution by using widely used UAV platforms, and evaluate the geo-referencing accuracy for the direct geo-localization of the proposed solution; and (3) we put the proposed solution to red tide monitoring and geo-localization in practice, and verify its validation and response ability for red tide monitoring. The establish workflow for monitoring red tides is shown in Figure 2.

Figure 2. The workflow for red tide monitoring and geo-localization.

This paper presents a UAV-based photogrammetric solution for red tide monitoring. This paper is organized as follows. Section 2 gives the details of the proposed solution in terms of the direct geo-localization model design, the selection of UAV platforms, and the calculation of red tide geometry. By using real UAV datasets, the precision of the direct geo-localization model and the applicability of the solution are evaluated and verified in Section 3, which is followed by the discussion in Section 4. Finally, Section 5 presents the conclusions of this study and further studies in the future.

2. Methodologies

This paper proposes a workflow for red tide monitoring and direct geo-localization based on onboard POS data and camera parameters. To achieve this goal, three major issues should be carefully addressed, i.e. the direct geo-localization model, UAV photogrammetric system, and geometric calculation of red tide. The details of each case are presented as follows.

2.1. Direct geo-localization model

The direct geo-localization model is based on the principle of the collinear equation in the field of photogrammetry, which describes the projection relationship from 3D scene points to 2D image points through camera projection centers (Mikhail et al. 2001). The principle of the direct geo-localization model is illustrated in Figure 3, and the formula of the direct geo-localization model is presented in Equation (1)(1) $$\{\begin{array}{c}X=(Z_0-Z_S)\frac{a_1(x-x_0)+a_2(y-y_0)-a_{3}f}{c_1(x-x_0)+c_2(y-y_0)-c_{3}f}+X_S\hfill \\ Y=(Z_0-Z_S)\frac{b_1(x-x_0)+b_2(y-y_0)-b_{3}f}{c_1(x-x_0)+c_2(y-y_0)-c_{3}f}+Y_S\hfill \end{array}$$(1) (1) $$\{\begin{array}{c}X=(Z_0-Z_S)\frac{a_1(x-x_0)+a_2(y-y_0)-a_{3}f}{c_1(x-x_0)+c_2(y-y_0)-c_{3}f}+X_S\hfill \\ Y=(Z_0-Z_S)\frac{b_1(x-x_0)+b_2(y-y_0)-b_{3}f}{c_1(x-x_0)+c_2(y-y_0)-c_{3}f}+Y_S\hfill \end{array}$$(1) where, (x, y) is the 2D image point of the target; $(x_0,y_0,f)$ are camera intrinsic parameters, i.e. the principle point $(x_0,y_0)$ and the focal length f; $a_i,b_i,c_i(i=1,2,3)$ are the elements of the rotation matrix that is calculated from camera orientation parameters, i.e. omega, phi, and kappa $(\omega ,\Phi ,\kappa );$ $X_S,Y_S,Z_S$ is the camera position parameters; Z₀ is the elevation of the plane that the target locates at; and (x, y) is the calculated 3D scene point of the target. $(\omega ,\Phi ,\kappa )$ and $X_S,Y_S,Z_S$ consist of camera extrinsic parameters. According to the direct geo-localization model, three items should be given, including 2D image points of red tide, the camera intrinsic parameters and extrinsic parameters, and the elevation of the sea level. The elevation of the sea level Z₀ can be obtained from hydrological stations or calculated by in-field surveying.

Figure 3. The principle of the direct geo-localization model.

2.2. UAV photogrammetric system

2.2.1. UAV platform selection

Considering the required data for direct geo-localization, the DJI Phantom 4 RTK (Real-Time Kinematic) and the DJI M300 UAVs have been selected as the photogrammetric system for data acquisition. For photogrammetry, the used camera is critical to the final productions. In this study, the default cameras configured with the UAVs have been used due to two main reasons. On the one hand, it enables the application of the proposed solution; on the other hand, the configured cameras can provide all necessary data.

For the DJI Phantom 4 RTK UAV, the maximum flight speed and wind speed resistance are 14.0 m/s and 10.0 m/s for the UAV aircraft, which is essential for offshore data acquisition campaigns. Due to the usage of batteries, the flight duration is approximately 30 min for one flight. The Phantom 4 RTK UAV is equipped with one FC6310R camera with a resolution of 5472 by 3648 pixels and a high-precision positioning sensor that supports multiple GNSS (Global Navigation Satellite System) systems. The nominal positioning accuracy with the RTK-enabled mode is 1.0 and 1.5 cm in the horizontal and vertical directions, respectively. The DJI M300 RTK has the same GNSS configuration and positioning accuracy, while the other items differ. A Zenmuse P1 camera with a resolution of 8192 by 5460 pixels has been used for image recording. These configurations enable the collection of high-resolution images with necessary POS data.

2.2.2. Calculation of camera intrinsic parameters

According to the direct geo-localization model, the camera intrinsic parameters $(x_0,y_0,f)$ are mandatorily required to determine the position and orientation of imaging rays relative to the camera coordinate system (c-system), which are recorded in the EXIF (Exchangeable Image File Format) data of UAV images. For the used DJI UAVs, camera intrinsic parameters are provided under laboratory pre-calibration, along with the well-known Brown distortion coefficients. In addition, there are two types of working modes, i.e. dewarp mode and non-dewarp mode. The former uses pre-calibration parameters to undistort images; the latter stores original images along with distortion coefficients. In this study, we use the Brown camera model (Fryer and Brown 1986), as shown in Equation (2)(2) $$\{\begin{array}{c}\Delta x=(x-x_0)(k_1{r}^2+k_2{r}^4+k_3{r}^6)+p_1[r^2+2{(x-x_0)}^2]+2p_2(x-x_0)(y-y_0)\hfill \\ \Delta y=(y-y_0)(k_1{r}^2+k_2{r}^4+k_3{r}^6)+p_2[r^2+2{(y-y_0)}^2]+2p_1(x-x_0)(y-y_0)\hfill \end{array}$$(2) , with the corresponding parameters for the intrinsic orientation of imaging rays (2) $$\{\begin{array}{c}\Delta x=(x-x_0)(k_1{r}^2+k_2{r}^4+k_3{r}^6)+p_1[r^2+2{(x-x_0)}^2]+2p_2(x-x_0)(y-y_0)\hfill \\ \Delta y=(y-y_0)(k_1{r}^2+k_2{r}^4+k_3{r}^6)+p_2[r^2+2{(y-y_0)}^2]+2p_1(x-x_0)(y-y_0)\hfill \end{array}$$(2) where, (x, y) is coordinate in the distorted image; $r=\sqrt{{(x-x_0)}^2+{(y-y_0)}^2}$ is the offset from (x, y) to $(x_0,y_0);$ $(\Delta x,\Delta y)$ is the correction value of (x, y); $(k_1,k_2,k_2)$ and $(p_1,p_2)$ are respectively the radial and tangent distortion parameters. For the non-dewarp mode, the distortion parameters are obtained from EXIF data; for the dewarp mode, their values are set as zero since images have been undistorted.

2.2.3. Calculation of camera extrinsic parameters

For geo-localization, extrinsic parameters $(X_S,Y_S,Z_S,\omega ,\Phi ,\kappa )$ are required to determine the position and orientation of imaging rays relative to the surveying coordinate system (E-system). For the used UAV, onboard POS system records camera positions $(\varphi ,\lambda ,\eta )$ in the WGS-84 coordinate system and orientation angles $(\psi ,\Theta ,\Phi )$ in the navigation system (n-system). Thus, the original data cannot be directly used for geometric calculation.

In the field of photogrammetry, orientation angles $(\omega ,\Phi ,\kappa )$ are defined in the E-system, which differ from the recorded orientation angles $(\psi ,\Theta ,\Phi )$ in the n-system (Zhao et al. 2013). By using the Earth Centered Earth Fixed coordinate system (e-system), orientation angles $(\psi ,\Theta ,\Phi )$ can be converted to the E-system as represented by Equation (3)(3) $$R_E^c={(R_e^E\times R_{n_i}^e\times R_b^{n_i}\times R_c^b)}^T$$(3) (3) $$R_E^c={(R_e^E\times R_{n_i}^e\times R_b^{n_i}\times R_c^b)}^T$$(3) where $R_b^{n_i}$ is the rotation matrix from the n-system to the IMU body system (b-system) using the recorded angles $(\psi ,\Theta ,\Phi );$ $R_{n_i}^e$ is the rotation matrix from the e-system to the n-system, which is determined by the recorded latitude and longitude $(\varphi ,\lambda );$ $R_e^E$ is the rotation matrix from E-system to the e-system, which is determined by the origin $({\varphi }_0,{\lambda }_0)$ of the E-system. is the rotation matrix from the c-system to the E-system, which defines the orientation angles $(\omega ,\Phi ,\kappa )$ of cameras in the E-system. For the details of the above-mentioned transformations, the readers can refer to Jiang and Jiang (2017). Finally, $R_c^b$ is the rotation matrix from the b-system to the c-system. For the used DJI UAVs, their relationship is shown in Figure 4 and can be calculated by using Equation (4)(4) $$R_c^b=R_Z(90)\times R_X(90)$$(4) (4) $$R_c^b=R_Z(90)\times R_X(90)$$(4)

Figure 4. The camera and body coordinate systems of the DJI Phantom 4 RTK UAV.

Besides, camera positions $(\varphi ,\lambda ,\eta )$ in the WGS-84 coordinate system are also converted to the E-system with the origin $({\varphi }_0,{\lambda }_0,0).$ The transformation is expressed by Equation (5)(5) $$V_E=R_e^E(T_{\mathit{\text{ell}}}^e\left[\begin{array}{c}\varphi \\ \lambda \\ \eta \end{array}\right]-T_{\mathit{\text{ell}}}^e\left[\begin{array}{c}{\varphi }_0\\ {\lambda }_0\\ 0\end{array}\right])$$(5) (5) $$V_E=R_e^E(T_{\mathit{\text{ell}}}^e\left[\begin{array}{c}\varphi \\ \lambda \\ \eta \end{array}\right]-T_{\mathit{\text{ell}}}^e\left[\begin{array}{c}{\varphi }_0\\ {\lambda }_0\\ 0\end{array}\right])$$(5) where, $T_{\mathit{\text{ell}}}^e$ is the transformation from the WGS-84 coordinate system to the e-system; $V_E=(X_S,Y_S,Z_S)$ is the converted position in the e-system. In this study, the origin $({\varphi }_0,{\lambda }_0)$ of the E-system is defined by using the recorded latitude and longitude $(\varphi ,\lambda )$ of each image as images are processed individually. After the conversion of camera positions $(\varphi ,\lambda ,\eta )$ and orientation angles $(\psi ,\Theta ,\Phi ),$ camera extrinsic parameters $(X_S,Y_S,Z_S,\omega ,\Phi ,\kappa )$ can be obtained and utilized for the direct geo-localization of 2D image points.

2.3. Geometric calculation for red tide

After obtaining the elevation of sea level Z₀ and camera intrinsic parameters $(x_0,y_0,f)$ and camera extrinsic parameters $(X_S,Y_S,Z_S,\omega ,\Phi ,\kappa ),$ the geographic coordinates of red tide can be calculated using the direct geo-localization model. In practice, except for the high-resolution RGB images, the geo-location and overlap region of red tide are interesting to the users, which provide the basic geographic data for red tide monitoring. To calculate this information, red tide regions must be completely and accurately extracted from the RGB images. Although sea color changes obviously, there are large proportions of regions with light and mixed color, as illustrated in Figure 5(a). It becomes difficult to detect these regions using automatic methods, e.g. the U-Net model (Ronneberger et al. 2015).

Figure 5. The comparison of the automatic and manual labels of red tide.

In contrast, to using the automatic methods, this study tends to the manual operation for red tide detection since it can be a more practical solution to solve the issue. In this study, the well-known labeling tool, termed Labelme (Labelme 2022), is selected for the manual annotation of red tide regions, as shown in Figure 5(b). Labelme allows users to draw the regions of red tide using polygons and store the endpoints in the JSON (JavaScript Object Notation) file format. After annotating the regions of red tide, the geographic coordinates of endpoints are calculated based on the direct geo-localization model, which is converted into the Shapefile format and can be rendered and analyzed in the GIS (Geographic Information System) software package ArcMap (ArcGIS 2022) for further application, e.g. area calculation and overlap analysis.

2.4. Implementation of the proposed solution

Based on the above-mentioned techniques, this study implements the overall solution for red tide monitoring, as shown in Figure 6. The proposed solution is achieved by using the C++ programming language. The details are described as follows:

Figure 6. The implementation of the proposed solution for red tide monitoring.

1. For outdoor data acquisition, the materials of test sites are first analyzed, such as the wind speed and temperature of offshore and the rough location of red tide. These data facilitate the settings for acquisition campaigns. By setting proper flight trajectories, UAV images can be collected under automatic or manual operations.

2. For indoor data processing, images that cover the region of red tide are annotated by using the Labelme software package, and the coverage polygons with geo-reference information are generated in a batch manner. For the used DJI Phantom 4 RTK UAV, the required camera intrinsic and extrinsic parameters are stored in the EXIF data of images. The used tag labels are presented in Appendix A.

3. To obtain the overall ground overlap, the individually generated polygons are further merged by using the ArcMap GIS software packages. The generated polygons could be utilized for further GIS analysis, e.g. area calculation of red tide regions, overlap analysis with other data for visualization, and trend prediction.

3. Experiments and results

The experiments are conducted from the aspects of precision analysis and application of the proposed solution for red tide monitoring. In the precision analysis for direct geo-localization, one test site on the shore has been selected, and the digital orthographic map (DOM) and digital surface map (DSM) have been generated with the aid of ground control points, which provide ground-truth data for precision comparison. In the application for red tide monitoring, two UAV acquisition campaigns have been executed to collect high-resolution images, and the proposed solution has been used for the geometric calculation of red tide. The details of the tests are presented in the following sections.

3.1. Precision analysis for direct geo-localization

3.1.1. Test sites and datasets

This study has chosen one test site on the shore for precision analysis. The test site locates in Minqing County, Fujian province, which is close to the Minjiang River. The majority of the test site is covered by bare ground, and in the central region, there exists a country that consists of some low-height village houses.

For data acquisition, two UAV platforms have been used, i.e. the DJI Phantom 4 RTK and DJI M300 RTK. The former is equipped with the DJI FC6310R camera with dimensions of 5472 by 3648 pixels; the latter carries the DJI Zenmuse P1 camera with dimensions of 8192 by 5460 pixels. For comparison, these two acquisition campaigns are conducted by using the same flight height, which is 300 meters relative to the ground. It is determined by UAV platforms and flying environments, such as wind speed and carrying capacity. The collected images are presented in Figure 7, which are further processed by using the classical photogrammetric software package, termed Agisoft Metashape (Agisoft 2022), to generate the DOM and DSM.

Figure 7. The illustration of collected UAV images by using the selected UAV platforms.

For the precision of the direct geo-localization model, five ground targets are selected and annotated from the DOM data presented in Figure 8(a), including two targets on the ground and three targets on house roofs. The coordinates of these ground targets are manually measured from the geo-referenced DOM image. The targets are located on varying elevation planes, whose values are obtained from the DSM image shown in Figure 8(b).

Figure 8. The DOM and DSM for precision analysis.

3.1.2. Results and analysis

Based on the proposed direct geo-localization model, the footprints of the selected targets are calculated and presented in Figure 9. The images in the first row indicate the annotation of selected targets on the photos, and the images in the second and third rows show the calculated footprints of these targets by using data from the DJI Phantom 4 RTK and M300 RTK UAVs. It is clearly shown that an obvious offset exists between the calculated footprints and selected targets due to the errors from camera intrinsic and extrinsic parameters. However, using the direct geo-localization model, we can still obtain the footprints of the selected targets that are close to the ground-truth locations, as presented in Figure 9.

Figure 9. Polygon annotation and geometry calculation from UAV images.

To present a quantitative evaluation for direct geo-localization, this study has conducted the statistic of geometry calculation for these two UAV platforms, including offsets between vertex coordinates and differences between footprint areas. The results are presented in Tables 1 and 2, in which Table 1 shows the statistic of geometry calculation for the DJI Phantom 4 RTK UAV, and Table 2 shows the statistic of geometry calculation for the DJI M300 RTK UAV. We can conclude: (1) based on the proposed direct geo-localization model, the offset between vertex coordinates is approximately 20 m in the XY plane for these two UAV platforms; (2) the differences between footprint areas are less than 9.2% and 7.7% for the DJI Phantom 4 RTK and M300 RTK UAVs; (3) the area difference for the targets locating on house roofs is smaller than that of targets locating on the ground since targets on the house roofs have more consistent altitude, which is also applied to red tide monitoring on the sea. In a word, the used UAV platforms, i.e. DJI Phantom 4 RTK and M300 RTK UAVs, can achieve an accuracy of about 20 m for direct geo-localization and the residual better than 9.2% for area estimation in red tide monitoring based on the direct geo-referencing model.

Table 1. The statistic of geometry calculation for the DJI Phantom 4 RTK UAV images. The coordinate and area differences are respectively in the unit of meters and square meters; the percentage is the ratio of area difference to the ground-truth area.

Label 1
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.991366	26.19846	118.991394	26.198583	−3.12	−13.69	14.04
2	118.991547	26.198618	118.991581	26.198751	−3.78	−14.81	15.28
3	118.991624	26.19855	118.991667	26.198677	−4.79	−14.14	14.93
4	118.991439	26.19838	118.991470	26.198504	−3.45	−13.80	14.23
Mean					−3.78	−14.11	14.61
Area	353.747		389.797		−36.050 (9.2%)
Label 2
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.993490	26.199936	118.993601	26.200053	−12.36	−13.02	17.95
2	118.993910	26.200338	118.994058	26.200451	−16.48	−12.58	20.73
3	118.994183	26.200098	118.994341	26.200185	−17.59	−9.68	20.08
4	118.993722	26.199720	118.993839	26.199820	−13.02	−11.13	17.13
Mean					−14.86	−11.61	18.97
Area	2733.674		3008.416		−274.742 (9.1%)
Label 3
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.992192	26.200497	118.99218	26.200638	−1.34	15.70	15.75
2	118.992399	26.200664	118.992395	26.200813	−0.45	16.57	16.59
3	118.992450	26.200614	118.992465	26.200743	1.67	14.36	14.46
4	118.992530	26.200537	118.992525	26.200679	−0.56	15.81	15.82
5	118.992316	26.200370	118.99231	26.200502	−0.67	14.69	14.71
6	118.992239	26.200442	118.992246	26.200568	0.78	14.03	14.05
Mean					−0.09	15.20	15.20
Area	665.449		715.234		−49.785 (7.0%)
Label 4
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.988737	26.198247	118.988599	26.19834	15.36	−10.35	18.52
2	118.988956	26.198317	118.988824	26.198407	14.69	−10.02	17.78
3	118.988957	26.19825	118.988852	26.198329	11.69	−8.79	14.63
4	118.989008	26.198164	118.988879	26.198253	14.36	−9.91	17.45
5	118.988785	26.198103	118.988652	26.198186	14.81	−9.24	17.45
6	118.988740	26.198183	118.988627	26.198260	12.58	−8.57	15.22
Mean					13.91	−9.48	16.84
Area	492.009		529.373		−37.364 (7.1%)
Label 5
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.992357	26.201299	118.992509	26.201255	−16.92	4.90	17.62
2	118.992524	26.201446	118.992662	26.201401	−15.36	5.01	16.16
3	118.992658	26.201329	118.992795	26.201288	−15.25	4.56	15.92
4	118.992602	26.201279	118.99274	26.201239	−15.36	4.45	15.99
5	118.992559	26.201313	118.992702	26.201272	−15.92	4.56	16.56
6	118.992448	26.201218	118.992599	26.201176	−16.81	4.68	17.45
Mean					−15.94	4.69	16.61
Area	431.464		404.510		26.954 (6.7%)

Table 2. The statistic of geometry calculation for the DJI M300 RTK UAV images. The coordinate and area differences are respectively in the unit of meters (m) and square meters (m2); the percentage is the ratio of area difference to the ground-truth area.

Label 1
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.991432	26.198693	118.991392	26.198583	4.45	12.25	13.03
2	118.991626	26.198858	118.99158	26.19875	5.12	12.02	13.07
3	118.991703	26.198781	118.99166	26.198677	4.79	11.58	12.53
4	118.991509	26.198609	118.991474	26.198503	3.90	11.80	12.43
Mean					4.56	11.91	12.76
Area	392.772			385.452	7.320 (1.9%)
Label 2
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.993686	26.200221	118.9936	26.200054	−9.57	−18.59	20.91
2	118.994161	26.200639	118.994057	26.200451	−11.58	−20.93	23.92
3	118.994436	26.200351	118.994345	26.200183	−10.13	−18.70	21.27
4	118.993929	26.199965	118.993837	26.199821	−10.24	−16.03	19.02
Mean					−10.38	−18.56	21.28
Area	3266.198		3033.193		233.005 (7.7%)
Label 3
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.98875	26.198461	118.988598	26.19834	−16.92	−13.47	21.63
2	118.988977	26.198525	118.988822	26.198407	−17.25	−13.14	21.69
3	118.988993	26.198484	118.988851	26.198329	−15.81	−17.25	23.40
4	118.989026	26.19837	118.98888	26.198254	−16.25	−12.91	20.76
5	118.988798	26.198302	118.988651	26.198185	−16.56	−14.19	21.81
6	118.988766	26.198412	118.988626	26.198261	−16.47	−14.37	21.86
Mean					0.09	−15.20	15.20
Area	536.538696		522.52573		14.013 (2.6%)
Label 4
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.988750	26.198461	118.988598	26.19834	16.92	13.47	21.63
2	118.988977	26.198525	118.988822	26.198407	17.25	13.14	21.69
3	118.988993	26.198484	118.988851	26.198329	15.81	17.25	23.40
4	118.989026	26.19837	118.98888	26.198254	16.25	12.91	20.76
5	118.988798	26.198302	118.988651	26.198185	16.56	14.19	21.81
6	118.988766	26.198412	118.988626	26.198261	16.47	14.37	21.86
Mean					16.54	14.22	21.86
Area	536.539		522.526		14.013 (2.9%)
Label 5
	Calculated		Ground-truth		Difference
No.	Longitude	Latitude	Longitude	Latitude	$\Delta X$	$\Delta Y$	$\Delta \mathit{\text{XY}}$
1	118.992357	26.201299	118.992509	26.201255	−16.92	4.90	17.62
2	118.992524	26.201446	118.992662	26.201401	−15.36	5.01	16.16
3	118.992658	26.201329	118.992795	26.201288	−15.25	4.56	15.92
4	118.992602	26.201279	118.99274	26.201239	−15.36	4.45	15.99
5	118.992559	26.201313	118.992702	26.201272	−15.92	4.56	16.56
6	118.992448	26.201218	118.992599	26.201176	−16.81	4.68	17.45
Mean					−15.94	4.69	16.61
Area	431.464		404.510		26.954 (6.7%)

3.2. Application for red tide monitoring

3.2.1. Test sites and datasets

By using the proposed solution, this study has conducted two practical tasks for red tide monitoring over the sea near Fujian province, as shown in Figure 10. In April 2022, there was some red tide occurring near the Pingtan sea region in Fujian province, and we conducted two acquisition campaigns to collect UAV images. The first one occurred on April 22, 2022, near the Fengtian village, in which red tide is featured with bronzing color and located from the direction of the northeast to southwest. As shown in Figure 11(a), the boundary between red tide and seawater is evident, which can facilitate manual annotation. The second one occurred on April 13, 2022, near Changjiang Ao, which locates from the direction of north to south. In contrast to the first one, red tide in this test site has a low density and light color, as presented in Figure 11(b), which causes difficulties in separating them from seawater. For data acquisition, the DJI Phantom 4 RTK UAV has been utilized in these two test sites. The imaging angle of onboard cameras is set as negative 20 degrees to obtain large ground coverage.

Figure 10. The location of these two test sites.

Figure 11. The illustration of collected UAV images for red tide monitoring.

3.2.2. Results and analysis

In the first step of the proposed solution, red tide regions are annotated using the Labelme software package. Due to the redundancy of captured images, a subset of images are extracted from original UAV images, which are annotated for red tide regions, as shown in Figure 12. For test site 1, the boundary of red tide is obvious, and the regions can be easily annotated, as presented in Figure 12(a). For test site 2, the boundary of red tide is not obviously clear since the light color causes mixing with seawater. However, due to the high resolution of collected UAV images, red tide regions can still be successfully annotated, as shown in Figure 12(b). In this step, there are 15 images selected from the original datasets, and the time costs for the annotation of red tide regions are approximately 30 min for these two datasets. The generated individual ground-coverage polygons are shown on the right side in Figure 12. We can see that the shape of these polygons is consistent with the annotation in the corresponding images.

Figure 12. The annotation and geometric calculation of UAV images.

After the geometric calculation of all annotated images, the ground-coverage polygons are then merged to generate the overall region of red sides. Considering that the precision of direct geo-localization is about 20 m for the proposed solution, a buffer operation is executed for the merged polygon to ensure enough coverage of red tide. The results are presented in Figure 13. From the generated map, we can clearly see that: (1) for test site 1, the area of red tide regions intruded by the red tide is about 2.49 square kilometers, whose spatial spans in the north-south and west-east directions are 3.69 kilometers and 2.44 kilometers, respectively; (2) for test site 2, the area of red tide regions is near 4.61 square kilometers, which locates along the north-south direction with the spatial span of 5.86 kilometers. Noticeably, the sea farms in the right-bottom region have been polluted by red tide. In a word, the proposed UAV-based photogrammetric solution and the designed software packages can be a suitable way for the regular monitoring of red tide and provide useful data for the management of marine ecosystems.

Figure 13. The generated map for red monitoring of the two test sites.

4. Discussion

4.1. UAV platform and direct geo-localization precision

The proposed solution depends on the camera intrinsic and extrinsic parameters to build the direct geo-localization model and calculate the geometry of annotated red tide regions. The camera intrinsic parameters mainly consist of the focal length and principal point; the camera extrinsic parameters include the positioning and orientation data of captured images. For the used DJI Phantom 4 RTK and DJI M300 RTK, onboard cameras have been well calibrated in the laboratory, which can be directly read from the EXIF data from images. Besides, almost market-available UAVs are equipped with POS sensors for automatic navigation. In this study, the used UAVs can achieve high-precision positioning due to the usage of RTK. As presented in Section 3.1, the precision of the direct geo-localization model is about 20 m in the horizontal direction, mainly because of the low orientation accuracy of onboard IMU (Inertial Measurement Unit) instruments. For red tide monitoring, the precision is acceptable due to two main reasons. On the one hand, the red tide region changes rapidly due to varying ocean environments; on the other hand, the red tide region is usually determined via human checking, whose error is generally large. Thus, the geo-localization error of 20 m is accepted for red tide monitoring considering these complex factors.

To improve the precision of geometric calculation, two solutions can be used from the aspects of sensor integration and data processing. On the one hand, high-precision and lightweight IMU can be used in UAV photogrammetric systems. This solution requires high economic costs and the redesign of UAV platforms; on the other hand, bundle adjustment can be executed to resume accurate orientation angles instead of the usage of low-precision data from onboard IMU instruments (Jiang et al. 2022). This solution requires that enough correspondences can be matched, which is difficult for offshore UAV images. Considering the first solution, this study has developed a software package for geometric calculation, which support parsing parameters from EXIF data or reading from external files (Figure 14).

Figure 14. The software package developed for geometric calculation in this study.

4.2. Manual annotation and automatic detection of red tide region

In the workflow of the proposed solution, red tide regions are manually annotated in the used Labelme software package ¹ https://github.com/labelmeai/labelme. Although red tide causes the color change of sea waters, the density of colors and the mixing with waters make it difficult for the automatic detection of red tide regions. To assess the ability of automatic detection, this study has also used deep learning techniques for red tide detection because of the successful usage of CNN models. First, training images captured from different locations and with varying conditions samples are collected, which are used to create training samples using annotation in Labelme, as presented in Figure 15(a). Second, the well-known CNN model, termed U-Net (Ronneberger et al. 2015), is trained from scratch using the labeled training images. Finally, the trained model is applied to some other images, and the segmentation results are shown in Figure 15(b). We can see that red tide with noticeable color changes, e.g. the first and fourth columns, can be well segmented from seawater. However, other red tides with light color would be ignored in the segmentation, e.g. the results shown in the second and third columns. Thus, this study uses manual annotation to detect red tide.

Figure 15. The annotation of training data and automatic segmentation.

4.3. Calculation complexity and its replicability for related applications

The proposed solution only uses UAV images and onboard POS data for the geo-localization of red tides. In contrast to the typical SfM-MVS based methods, the calculation complexity is negligible since it only involves the intersection of 3D points based on the colinear equation. Within the proposed solution, the only time-consuming step is the annotation of red tide regions from UAV images, as described in Section 4.2. In practice, this issue can be resolved from two aspects. On the one hand, oblique photogrammetry can be used as it can cover a larger region in one image; on the other hand, the UAV trajectory can be elegantly designed to record as less images as possible to cover the whole red tide region. Thus, the proposed solution can easily scale to large regions and perform red tide monitoring efficiently.

Apart from red tide monitoring, the proposed solution can also be extended to other applications. The core idea of the proposed solution is to use onboard sensor data for direct geo-localization. In practice, many other situations require direct geo-localization ability, such as landslide monitoring in geology. These applications need the automatic detection and efficient geo-localization of interesting targets. In general, these requirements are identical to the purpose of red tide monitoring. In a word, the proposed solution can also extend to other related applications for direct geo-localization.

5. Conclusion

This study proposes an overall workflow for red tide monitoring from the aspects of direct geo-localization model design, UAV platform selection, and geometric calculation. Because the classical photogrammetric processing solution is not appropriate for offshore UAV images, the direct geo-localization model depends sorely on onboard data for the geometric calculation, i.e. pre-calibrated camera intrinsic parameters and converted camera extrinsic parameters. To assess the precision of the direct geo-localization model, two commonly used UAV platforms have been evaluated, including the DJI Phantom 4 RTK and M300 RTK UAVs, by using images from one ground test site. The results demonstrate that the precision is approximately 20 m. Two practical tasks based on the defined workflow have already been completed, demonstrating that the proposed UAV-based photogrammetric workflow is appropriate for red tide monitoring and can deliver valuable data for marine ecosystem management. However, the proposed solution still uses manual annotation for red tide regions. In order to improve the efficiency of the proposed solution, more research would concentrate on the automatic detection of red tide regions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

National Natural Science Foundation of China (Grant No. 42371442, 42301514), and the Hubei Provincial Natural Science Foundation of China (Grant No. 2023AFB568).

Appendix A.

Abbreviations

The following abbreviations are used in this manuscript:

UAV	=	Unmanned Aerial Vehicle
SfM	=	Structure from Motion
MVS	=	Multi-view Stereo
RS	=	Remote Sensing
CNN	=	Convolutional Neural Network
RTK	=	Real-Time Kinematic
GNSS	=	Global Navigation Satellite System
POS	=	Positioning and Orientation system
EXIF	=	Exchangeable Image File Format
JSON	=	JavaScript Object Notation
GIS	=	Geographic Information System
DOM	=	Digital Orthographic Map
DSM	=	Digital Surface Map
IMU	=	Inertial Measurement Unit
RBR	=	Red-to-Blue Band Ratio

Appendix B.

Meta data in EXIF

Table B.1. The used tag labels in the EXIF data of UAV images.

Tag name	Description
CalibratedFocalLength	Camera focal length f
CalibratedOpticalCenterX	Camera principal point cx
CalibratedOpticalCenterY	Camera principal point cy
DewarpFlag	Image undistortion flag [0/1]
DewarpData	Calibration data in format $(f_x,f_y,c_x,c_y,k_1,k_2,p_1,p_2,k_3)$
GpsLatitude	Camera latitude ϕ
GpsLongtitude	Camera longitude λ
AbsoluteAltitude	Camera altitude η
GimbalRollDegree	IMU roll angle Ψ
GimbalPitchDegree	IMU pitch angle Θ
GimbalYawDegree	IMU yaw angle Φ