A global multimodal flood event dataset with heterogeneous text and multi-source remote sensing images

ABSTRACT

With the increasing frequency of floods, in-depth flood event analyses are essential for effective disaster relief and prevention. Satellite-based flood event datasets have become the primary data source for flood event analyses instead of limited disaster maps due to their enhanced availability. Nevertheless, despite the vast amount of available remote sensing images, existing flood event datasets continue to pose significant challenges in flood event analyses due to the uneven geographical distribution of data, the scarcity of time series data, and the limited availability of flood-related semantic information. There has been a surge in acceptance of deep learning models for flood event analyses, but some existing flood datasets do not align well with model training, and distinguishing flooded areas has proven difficult with limited data modalities and semantic information. Moreover, efficient retrieval and pre-screening of flood-related imagery from vast satellite data impose notable obstacles, particularly within large-scale analyses. To address these issues, we propose a Multimodal Flood Event Dataset (MFED) for deep-learning-based flood event analyses and data retrieval. It consists of 18 years of multi-source remote sensing imagery and heterogeneous textual information covering flood-prone areas worldwide. Incorporating optical and radar imagery can exploit the correlation and complementarity between distinct image modalities to capture the pixel features in flood imagery. It is worth noting that text modality data, including auxiliary hydrological information extracted from the Global Flood Database and text information refined from online news records, can also offer a semantic supplement to the images for flood event retrieval and analysis. To verify the applicability of the MFED in deep learning models, we carried out experiments with different models using a single modality and different combinations of modalities, which fully verified the effectiveness of the dataset. Furthermore, we also verify the efficiency of the MFED in comparative experiments with existing multimodal datasets and diverse neural network structures.

KEYWORDS:

• Flood event
• multimodal dataset
• deep learning
• multi-source remote sensing data
• internet data

1. Introduction

The increasing frequency of flood disasters severely threatens human society (Schumann, 2014). Early analysis and accurate estimation of flood events’ causes, social impacts, duration, and extent are essential for disaster relief and prevention. Traditional analysis methods encounter significant hurdles due to relying on limited hazard maps (Bernhofen et al., 2018), especially when scaling to large regions. Instead, the enormous amount of remote sensing data has become the main source of flood event analysis data (Bhatt et al., 2013; Rahman & Di, 2017). Accordingly, existing flood event datasets typically include remote sensing imagery datasets, like the FloodNet dataset (Rahnemoonfar et al., 2021) and the Multi3Net dataset (Rudner et al., 2019), and meteorological-information-enriched datasets (Gourley et al., 2013) exemplified by the European Flood Database (Hall et al., 2015), which included victim counts, flood causes, water levels, and spatiotemporal range information. Additionally, there are also datasets collected from social media platforms, such as CHF2015 (Layek et al., 2016).

However, notwithstanding the various types of flood event datasets and the available remote sensing imagery, flood event analysis remains greatly challenged by the geographically uneven distribution of data, the scarcity of time series data, and the limited availability of flood-related information. For instance, runoff data and flood disaster records collected by the Global Runoff Data Center (GRDC) are sufficient in Europe and the United States, but this is not the case in developing countries (Sampson et al., 2015). Accordingly, flood event analyses that typically adopt flood event data modeling (Hallegatte et al., 2013; Sampson et al., 2016), such as the river-basin-centered CaMa-Flood model (Yamazaki et al., 2011) and the Global Flood Model (GFM) (Wood et al., 2011), are not able to employ global flood event data for large-scale flood event evaluations. Likewise, Wilson et al. (2007) restricted their flood event assessment to Bangladesh, and Hirabayashi et al. (2013) quantified the impact of 10-year and 100-year floods instead of annual analyses, attributed to the shortage of time series data. Moreover, hydrological data, along with pertinent textual information (Jongman et al., 2015; Winsemius et al., 2016), including factors like casualties, geographical location, and economic data, are essential for thoroughly evaluating flood events. However, their acquisition without efficient methods and expertise is rather cumbersome and challenging. Hence, to establish global flood event datasets, amalgamating imagery data with hydrological or textual information is also critical to facilitate enhanced flood event analyses.

The Global Flood Database (Tellman et al., 2021), based on data from the Dartmouth Flood Observatory (DFO), is a comprehensive archive of satellite-based flood event datasets across the globe over 15 years. These datasets contain not only satellite image data but also hydrological information. Deep learning models have recently become increasingly adopted for flood event assessments (Islam et al., 2020; Luppichini et al., 2022; Yang & Cervone, 2019). Image data in the Global Flood Database, namely the flood map products derived from remote sensing images, are generally used for model verification but are not fit for deep learning model training. Practically, remote sensing imagery data are used to train deep-learning-based models for flood event analysis, typically confined to single-category classification or semantic segmentation of floods. Nevertheless, precisely distinguishing flooded areas from permanent water bodies remains a substantial challenge, primarily attributed to the dynamic characteristics inherent in these features. Even when leveraging high-resolution remote sensing imagery for improved image features like shape, color, and texture (Blaschke, 2010; Cheng et al., 2017), determining the appropriate partitioning thresholds in different scenarios remains a formidable challenge. The DFTC datasets (Islam et al., 2020) from the data fusion technical committee are multimodal flood training datasets containing SPOT-5 and radar images to support flood detection. Thus, resorting to multi-source imagery to fully take advantage of the complementarity of different sensors, like optical and radar sensors, could be substantially beneficial in flood event analyses.

It is worth noting that intrinsic semantic correlations exist between images regarding flood causes, watershed changes, and temporal variations. These correlation factors influence image features in flood event classification (Wood et al., 2011). In practice, by leveraging the textual information in a dataset, semantic details can be extracted from image features for a more refined classification of flood scenery. Therefore, establishing high-quality global flood event datasets, which cover diverse regions and amalgamate imagery data with semantic-rich textual information, is critical for high-accuracy model training and validation (Bernhofen et al., 2018) and facilitates enhanced flood event assessments considering both the visual and semantic factors. Aside from deep-learning-based flood event analyses, global flood datasets are vital for empowering fast and accurate large-scale flood event image retrieval. Traditional retrieval methods face challenges when determining precise spatiotemporal searching ranges (Wood et al., 2011). Some methods extract approximate flood regions from online reports or social media data (Li et al., 2018; Vieweg et al., 2010). At the same time, new attempts employ semantic search techniques (Li et al., 2012) like fuzzy retrieval using the flood type or level as a keyword. The need for enhanced semantic information in global flood datasets is emphasized in complex data retrieval scenarios (Deng et al., 2011). However, traditional screening methods are labor-intensive and require expertise. Enriching global flood event datasets with important semantic information is vital for effectively retrieving and pre-screening flood-related imagery.

To solve the abovementioned issues, we built a Multimodal Flood Event Dataset (MFED) for deep-learning-based flood event analysis and retrieval. It comprises multi-source remote sensing images and heterogeneous textual information from various flood-prone regions worldwide spanning the years 2000–2018. Firstly, the MFED contains multi-source remote sensing data, incorporating optical and radar images as the image modalities of flood event datasets. Here, satellite data from Landsat-5/7/8 and IRS-P6 are used as the optical image modality, while Sentinel-1 and ENVISAT-1/2 data are used for the radar image modality. By exploiting the features of optical and radar satellite sensors, the correlation and complementarity between distinct image modalities in the MFED can be harnessed to capture the nuances of pixels in flood images. Diverging from single-mode datasets, the presented multimodal dataset introduces much richer information with more dimensions. Secondly, based on the Global Flood Database, the MFED employs heterogeneous textual information as the text modality of the flood event dataset. It systematically incorporates satellite-based flood image data and hydrological information from the Global Flood Database. Then, through a meticulous data cleaning process, the structured flood-related textual information can be filtered from these gathered flood data. With these flood-related text modal data in the MFED, including geographical extents, global flood event causes, watershed changes, and temporal variations, the flood model can extract semantic features to provide semantic support for flood event classification and analysis. The geographical extent and temporal information related to the flood event can also be exploited to enable automatic remote sensing data retrieval for model training. Additionally, information from news reports on the web is also harnessed as another text modality. In the case of vague or imprecise flood regions, the semantic information extracted from retrieved online news can provide more precise geographical information on flood event locations. Accordingly, semantic information may supplement the correlated image data and support the comprehensive flood event analysis. Following this, benefiting from the global multimodal flood-related remote sensing imagery and semantic-rich text offered by the MFED, we can expect an enhanced efficiency of flood event analyses and flood-related image retrieval. In this paper, we verify the advantages of multimodality by comparing multimodality and single-modality models. We also compare the MFED with existing multimodal remote sensing datasets to verify its effectiveness for flood event analysis.

2. Data and method

In this section, we first introduce our data sources. For image modalities, optical images from Landsat series satellites on the public remote sensing platform and radar images from the Sentinel-1 satellite were selected according to the requirements for the spatiotemporal range and the ease of downloading data. For the text modality, we used comprehensive search engines to source news records and the Global Flood Database as a source of hydrological data. Secondly, through the API provided by the remote sensing platform and internet crawler technology, we retrieved data according to the collection principle and downloaded the remote sensing images and text data that met the requirements in bulk. Finally, the data processing of different remote sensing images included band synthesis, fringe elimination, image acquisition, etc. Structured textual data processing included information extraction based on keywords and data wrangling. For unstructured text, data processing included data filtering and normalization. Figure1 shows the principal data sources in our study and the processes involved in collecting and processing distinct data modalities.

Figure 1. Data sources, data collection methods, and data processes for the MFED.

2.1. Data source

To facilitate batch data downloads and address the worldwide need for spatiotemporal observations, we used multiple platforms such as the European Space Agency (ESA), the United States Geological Survey (USGS), and the Aerospace Information Research Institute at the Chinese Academy of Sciences as remote sensing imagery sources. To ensure reliable and comprehensive text data sources, we chose Google and Baidu search engines and the Global Flood Database as the text modal data sources.

We chose to use optical and radar images in the MFED, combining the spectral characteristics of water bodies in optical images (Bhagat & Sonawane, 2011; Feyisa et al., 2014; Li et al., 2015) with the advantage that radar images are not influenced by weather conditions (Gan et al., 2012; Giustarini et al., 2012; Westerhoff et al., 2013). Considering that the selected satellite measurements are global and the target time frame of the Global Flood Event Database is 2000–2018, we selected Landsat-5, Landsat-7, and Landsat-8 satellites as our optical image sources. To facilitate bulk data downloads, we utilized the Landsat Collection 2 Level-2 dataset, available on the Microsoft Planetary Computer platform. This dataset covers the period from August 1982 to the present and is sourced from data providers such as NASA and USGS. Furthermore, to enhance the diversity of data types in our multi-source remote sensing imagery, we add IPS-P6 as an additional optical image source.

To match the spatio-temporal range of the optical images and to simplify data downloading, we chose the Sentinel-1 satellite as the source of radar imagery. We used the Copernicus Open Access Hub as a data source, which provides complete, free, and open access to Sentinel-1, Sentinel-2, Sentinel-3, and Sentinel-5P products. The Copernicus Open Access Hub provides an API and creates a script to automatically search for and download Sentinel data. However, the first Sentinel-1 satellite was launched in April 2014, and our dataset extends back to 2000. Thus, our data sources also encompassed ENVISAT-1 and ENVISAT-2 satellites.

Hydrological information can serve as a trusted record for flood event analyses (Lehner & Grill, 2013; Sampson et al., 2016) and network data have gradually been playing an increasingly important role in disaster management analysis and evaluation (Alam et al., 2020; Kruczkiewicz et al., 2021; Vogel & Arumugam, 2016). We chose hydrological information and news reports taken from the internet as text modalities. The hydrological information in the Global Flood Database combines over 15 years of flood data and major flood events recorded by the DFO since the beginning of satellite records, creating the first comprehensive satellite-based resource for global flood risk management and mitigation. To ensure the acquisition of precise news reports on flood events, we opted for Google and Baidu, two widely utilized search engines, as our data sources.

Table 1 shows the data sources for all flood events in the MFED. For image modality, we obtained 1100 satellite images of Landsat-5/7/8 from the USGS platform, 198 satellite images of Sentinel-1 from the Copernicus Open Access Hub, and 121 satellite images of IRS-P6, ENVISAT-1 and ENVISAT-2 satellites from the Institute of Aerospace Information of the Chinese Academy of Sciences. In the text modality, we obtained 8,000 news reports from the Google search engine, 1,000 news records from the Baidu search engine, and 913 flood event records from the Global Flood database.

Table 1. Data source statistics of flood events in MFED.

Text Modality Source	Number	Image Modality Source	Number
Baidu	1000	United States Geological Survey	1100
Google	8000	Copernicus Open Access Hub	198
Global Flood Database	913	Aerospace Information Research Institute	121

2.2. Data collection

To acquire remote sensing imagery and textual information about flood events, we leveraged the spatiotemporal range provided by the Global Flood Event Database for data retrieval. We obtained multi-source remote sensing imagery and heterogeneous text data using an application programming interface in Python libraries that supports remote sensing image retrieval. First, we needed to collect global flood event information from the Global Flood Event Database as the basis for building the dataset. Therefore, we collected data on 913 flood events from flood-prone regions worldwide spanning the years 2000–2018, encompassing 29 countries. For each flood event, we established an analysis pair comprising flood images and corresponding event records. Each pair consists of an optical and a radar image, alongside hydrological details and an associated news record. Figure 2 shows the number of flood events we collected by country.

Figure 2. Collected flood event statistics for flood-prone areas around the world. Between 2000–2018, there were over 15 flood events in each of these countries.

When collecting remote sensing images, we used the density and frequency of the region, the number of clouds, and the proportion of the flood area as the criteria for data collection. We used automated downloading tools to download remote sensing images in batches from major data platforms and preprocessed different types of images to obtain image modalities. For each pair, image modalities were acquired with the same spatiotemporal extent. The images in the Global Flood Database are processed flood data products, and we can retrieve thousands of remote sensing images by their latitude and longitude. Therefore, we used the extension of the center point coordinates instead of the four-corner latitude and longitude coordinates of remote sensing product images as the search range.

For optical images, we used Python scripts to acquire remote sensing images in batches using the PyStac¹ tool, which is a library for working with SpatioTemporal Asset Catalogs (STACs) in Python 3. Using the STAC API, Landsat images were acquired from the Microsoft Planetary Computer platform (Source et al., 2022). We set the search scope according to the spatiotemporal extent of the flood event and limited the maximum cloud cover of the crawled data to 20%. The remote sensing imagery results obtained in the search area were sorted according to the cloud cover size, and the data with the lowest cloud cover were selected.

For radar, we use the Sentinelsat² (Valgur et al., 2019) API based on the HTTPS protocol in Python to perform batch searches and automatic downloads of Sentinel-1 data on Copernicus Open Access Hubs. It should be noted that we could use the Sentinelsat API to access the latest data and historical data simultaneously. Offline historical data in the database needed to be activated before downloading. Due to restrictions on downloading remote sensing images of other regions, we have chosen to utilize remote sensing images specifically covering China in order to diversify the range of image types within the MFED dataset. The IRS-P6 satellite served as an additional optical imagery source, enriching the image modality, while ENVISAT-1 and ENVISAT-2 were introduced as additional radar imagery sources. The data are sourced from the Remote Sensing Data Service SystemRemote Sensing Data Service System³ operated by the Aerospace Information Research Institute at the Chinese Academy of Sciences. For model training with the MFED, we utilized the event database to supply accurate spatiotemporal ranges as positive samples. Concurrently, we designated spatiotemporal ranges beyond its scope as negative samples, achieved through latitude and longitude deviations. All image types used are listed in Table 2.

Table 2. Types of remote sensing images collected in our dataset and the corresponding numbers of each type.

Optical		Radar
Name	Number	Name	Number
Landsat-5	266	ENVISAT-1	44
Landsat-7	704	ENVISAT-2	39
Landsat-8	130	Sentinel-1	198
IRS-P6	38	Total	1419

The structured information in the MFED is hydrological information on flood events taken from the Global Flood Dataset. It includes the main cause of the flood event, the longitude and latitude coordinates of the center point of the flood area, the start and end time, the main country where the flood event occurred, and the coordinates of the rectangular frame used to retrieve the image. For unstructured information, we used the country and time of flood as search keywords to search for relevant titles and abstracts on Google and Baidu as positive samples. At the same time, we used water as a keyword to search for information not related to floods as negative samples. By leveraging web crawler techniques like link analysis and content parsing, we ensure a meticulous and comprehensive extraction of relevant data. Details of the text modality are shown in the Table 3.

Table 3. Structured information of our text modality, such as the specific time, major countries, and reason for the flood event, and the news reports crawled from the internet, including summaries and titles.

Country	Title	Content	Main Cause	Time
Thailand	Floods leave Pai village pleading for relocation	Residents of a village in Mae Hong Son’s Pai district have called for local agencies to move their village out of high-risk areas for flash floods	Monsoonal rain	17 Aug 2016
Philippines	Storms claim 11 in Philippines, 34 in Vietnam	8, 2007, 9:14 AM PDT/Source: The Associated Press. Tropical Storm Pabuk churned across the Philippines on Wednesday, triggering deadly landslides	Heavy rain	8 Aug 2007

2.3. Data processing

The images we downloaded couldn’t be directly incorporated into the dataset due to quality issues such as cloud interference, speckle noise, et cetera, and the proportion of the flood area in the image will affect the quality of the model training. Hence, processing the data for each modality in the MFED was imperative. We download the images from the Microsoft Planetary Computer platform using the STAC API with an upper cloud coverage of 20% and conduct an optimal data selection based on cloud coverage through a batch data selection script. Then, we further check the cloud coverage and speckle noise of the data manually one by one to ensure that the flood area could be identified. In addition, based on the image characteristics, we adopted different data processing methods for radar and optical images. Most of the optical images we collected were multi-band data. We chose three bands for color synthesis to adapt to the RGB image format in the deep learning model. Based on the characteristics of different satellite bands, we selected bands that best reflect the characteristics of water bodies, such as red bands, near-infrared bands, and short-wave infrared bands, for synthesis. For Landsat-5, bands 4, 5, and 3 were chosen for synthesis; for Landsat-7, bands 5, 4, and 3 were chosen; for Landsat-8, bands 5, 6, and 4 were chosen; and for IRS-P6, bands 5, 4, and 3 were chosen.

Due to the Landsat-7 ETM+ onboard scanning line corrector (SLC) failure, the Landsat-7 images we acquired had stripes. We used the official landsat_gapfill.sav plug-in to eliminate these stripes. Since the flood area does not necessarily occupy the entire image, we also cropped the flood area while preserving the original image as much as possible. Figure 3 shows our results of processing optical images.

Figure 3. MFED performs multi-band color synthesis for optical images. For the images with stripes after synthesis, the stripes are eliminated. To make the flood area account for a larger proportion of the overall image, we cropped the flood area.

Radar image processing was relatively simple. For Sentinel-1, the product type we chose was Level-1 Ground Range Detected (GRD), which consists of focused radar data that have been detected, multi-looked, and projected to the ground range using an Earth ellipsoid model. Radar images do not consist of multiple bands like optical images. Different polarization methods are also important in addition to frequency, amplitude, and phase. The images we downloaded are dual-polarized: HH (horizontal transmission horizontal receiving) and HV (horizontal transmission vertical directly). Sentinel-1 scans the same area with a longer cycle and larger coverage than optical sensors, so we cropped the flood area into multiple images. Sentinel-1 imagery processing is shown in Figure 4. Figure 5 shows our image processing results for all types.

Figure 4. Images from Sentinel-1 collected in our dataset in HH and HV polarization. Due to the wide range of observations in radar images, we clipped flood areas with intervals into multiple instances.

Figure 5. Intermediate and final results during the processing of all image types in our dataset. It is worth noting that optical image processing involves two more steps of image synthesis and streak elimination.

We performed data wrangling for structured information. We extracted key fields suitable for deep learning training, classified some small classes in the fields into broad categories (such as Typhoons Damrey and Saola are converted to Typhoons), unified different descriptions of the same type, transformed the spelling, and so on. The unstructured information included the title and abstract information from internet news reports. However, the HTML crawled from the internet has problems such as format errors, repeated information, and invalid information. Our text cleaning work included removing special characters and web page tags using regular expressions, formatting data to make it easy to process and analyze, and removing invalid data. Finally, we converted the text and image modalities into JSON formats and recorded various information types through key-value in a hierarchical structure.

Figure 6 shows eight sample pairs in the MFED; the four groups on the left are positive examples, and the four groups on the right are negative examples. In the event record, the “Main_cause” field denotes the cause of the flood event, the “Time” field indicates the time of occurrence, the “Country” field specifies the country affected, and the “Flood” field contains a value of 1 for positive examples and 0 for negative examples. The MFED dataset is available for download from the Science Data Bank. The download link has been provided in the data availability statement section. The MFED dataset’s positive pairs mean both the text and image modality data are related to flood events. The negative pairs mean that the text and image are unrelated to the flood. For image modality, the geographical location and time of the images in the negative pairs are offset by the flood events, making the negative pairs’ image modality irrelevant to flood events. For text modality, we use “water” as a keyword in search engines to get news records unrelated to floods. In addition, although the textual information from Internet news records does not contain abundant information related to flood events, the data pairs could still be considered positive since the image modality data covers the flood’s location. The image modality includes radar and optical images. Since the purpose of our dataset is to retrieve images of flooded areas through different types of remote sensing imagery, there is no strong correlation between the images, but the latitude and longitude ranges of the images are similar. While both optical and radar images depict the flood area, the ability to discern flowing areas from other terrains is evident. Yet, distinguishing permanent bodies of water from flooded areas solely through imagery remains challenging. Take the last group as an example, i.e. the positive and negative sample pairs from Australia in 2016; the information obtained on the image modality is limited. From the text modality, we can not only learn the specific cause of the flood but also determine the location and time of the flood. To determine data quality, we used a detection program on all the image files to ensure that all files were readable and usable, and we randomly selected 50% of the images for detection. Finally, the MFED contains 1138 scenes of optical images, 281 scenes of radar images, 9000 pieces of news data related to flood events, and 913 pieces of flood event records after all data processing.

Figure 6. Processed sample pairs in the dataset. The image modality includes an optical image and a radar image, and the text modality includes unstructured news records and structured hydrological information.

3. Data validation

In this section, we apply the global flood classification model as an example to verify the validity of the MFED. We use a basic model architecture for classification experiments on the MFED, a variety of common neural network algorithms for testing, and comparison experiments on a common multimodal dataset. We verify that the MFED can successfully aid in the detection of flooded areas and demonstrate that multimodal data are more expressive than single-modality data.

3.1. Deep-learning-based flood model settings for the MFED

We verified the effectiveness of the MFED in flood classification using different model structures. To better apply multimodal data, we used a classical image and text model structure to verify the validity of the data.

Since multimodal remote sensing data cannot be directly applied in neural networks, we used a simplified multimodal model for model training. Our multimodal model primarily comprises an image and text feature encoder. The image feature encoder utilizes a typical deep learning network for image feature extraction, such as ResNet152 or VGG16. The text feature encoder utilizes the Bert network. We use the image encoder to extract features from multi-source remote sensing images and the text encoder to extract features from text modality. These features are combined using a vector concatenation method, resulting in the joint representation of multimodal features through a fully connected layer. The model structure is shown in Figure 7. Taking ResNet152 as an example for image modal feature extraction, the input image size is [224, 224, 3]. Given the diverse resolutions and band numbers of remote sensing images, we devised an adaptive sampling method. These approaches enable us to effectively read and sample various remote sensing images, ensuring their suitability for deep neural networks. The sampled images are passed through the ResNet152 model structure, undergoing convolution and pooling operations within the network. In our model training process, we initially load the pre-trained raw weights of ResNet152. Subsequently, we train our dataset based on these pre-trained weights. This approach accelerates our model training process. Consequently, the output of the optical image is [1, 1, 2048]. For Text modality coding, we utilize the Bert network to encode text features, setting the maximum number of words to 256. The input word vector size is computed within each mini-batch. We calculate the longest vector size in each batch as the input dimension. Short sentences are padded to match the maximum length. Additionally, we set the hidden layer size to 768, resulting in a feature vector length of 768 for the output after text modality encoding. We merge the multimodal image and text features through feature vector concatenation, then feed them into a fully connected layer for model prediction. We use the cross-entropy function as the loss function, which measures the difference between the predicted probability distribution and the actual distribution of the labels to effectively train our model by penalizing incorrect predictions and encouraging convergence towards the correct classification.

Figure 7. The simple multimodal model architecture used for training the multimodal data in our dataset. In the multimodal model, there are two processing parts, image modal encoder and text modal encoder, in which different deep learning models are used as coding structures. Finally, the features obtained from the image modality and text modality are fused and passed into the fully connected layer to obtain the final judgment result of the model.

During the experiments, we used different deep learning models for model training to test the applicability of the MFED. The image modality comparison analysis involved ResNet, VGG16, SqueezeNet, and other models. For the text modality, since Bert has proven to have excellent classification results in the text field, we choose Bert as the text model.

We utilize the JSON format to store the multimodal data records of MFED. Following the conventional size settings for training, validation, and test set partitions, we randomly scrambled the file records storing the MFED dataset information. Subsequently, we generated the training, validation, and test sets by parsing the record file in a ratio of 60:20:20. We train the model using the training set and evaluate its classification performance using the validation set, which fine-tunes the model’s hyperparameters and conducts an initial assessment. Finally, we gauge the model’s classification effectiveness based on the results obtained from the test set. In our analysis, we employed diverse evaluation metrics to assess the performance of our model comprehensively. These metrics include F1 score, Average Precision (AP), Precision, Recall, and Area Under the Curve (AUC), allowing us to understand its effectiveness in classification tasks.

3.2. Single modality validation of the MFED

To verify the validity of each modality in the MFED, we tested each single modality and combined text and image modalities to compare their performance. We use single-modality data to train different deep-learning models. For the text modality, the Bert network was used for training. For the two image modalities, we used VGG16 and ResNet152 for training. The best experimental results were finally obtained by continuously adjusting the model parameters and are shown in Table 4.

Table 4. Single modality training results. The Bert model structure is selected for the text modality, and the VGG16 and ResNet models are selected for the image modality. We chose the accuracy rate as the best performance indicator of the model and recorded the results of each indicator in the best model.

	TEXT	Radar		Optical
Metric\Model	BERT	VGG16	ResNet152	VGG16	ResNet152
F1	0.91176	0.84454	0.78992	0.73109	0.81933
AP	0.95730	0.8458	0.81313	0.81067	0.87602
Precision	0.98148	0.85	0.8191	0.82659	0.89881
Recall	0.89831	0.96045	0.9209	0.80791	0.85311
ROC	0.92456	0.7343	0.66537	0.65805	0.78721

For radar, the effect of VGG16 is better than that of the ResNet152 model; the F1 value is 0.84454, the AP is 0.8458, and the Recall is 0.96045. For optical images, the ResNet152 model is better than VGG16; its F1 value is 0.81933, higher than the value of VGG16 but slightly lower than the effect of radar in the VGG16 model. For models trained in image modality, the AP and precision on the test set did not exceed 90% in ResNet152 and VGG16 networks. Model training using text modality also exhibits low F1 values and recall rates. Despite the experiment confirming that single modality data have classification effectiveness for model training, the performances are unsatisfactory. Additionally, as the primary objective of the MFED dataset is to analyze flood images, we prioritize evaluating the effects of the imaging modality.

3.3. Multimodal validation of the MFED

The use of multimodal data takes advantage of the benefits of different types of remote sensing imagery (Knyaz, 2019; Lahat et al., 2015; Yang et al., 2019). To prove that using multimodal data in the MFED classification effect is better than using single-modality data, we used pairwise combinations of different modalities and all modalities as input data with the same parameters and models for comparative experiments. We chose the combination of optical images and text, radar and text, and optics, radar, and text to conduct comparative experiments in the same network structure. The experimental results are shown in the Table 5. We found that the classification effect of multimodal data is much better than that of single-modality data. The combination of text and radar data and the combination of text and optics data significantly improved the classification effect compared to the single-image modality.

Table 5. Multimodal training results. We experimented on multimodal datasets with the same model structure and training method. The single modality is replaced by a combination of different image modalities and text modalities, and the best experimental results under different models are recorded.

	TEXT+Radar		TEXT+Optical		TEXT+Optical+Radar
Metric\Model	VGG16+Bert	ResNet152+Bert	VGG16+Bert	ResNet152+Bert	VGG16+Bert	ResNet152+Bert
F1	0.92857	0.84874	0.94118	0.82773	0.97899	0.93277
AP	0.95521	0.85271	0.95954	0.88201	0.98044	0.95665
Precision	0.97059	0.85787	0.9711	0.90476	0.98315	0.97076
Recall	0.9322	0.9548	0.94915	0.85876	0.98870	0.93785
AUC	0.92512	0.74789	0.93359	0.79823	0.96976	0.92794

When using a combination of optical images and text in the VGG16 model, compared to using optical images, the precision increased by 17.48%, the F1 value increased by 28.74%, and the AUC value increased by 41.87%. When using a combination of radar images and text in the VGG16 model, compared to using radar images, the precision increased by 14.19%, the F1 value increased by 9.95%, and the AUC value increased by 25.99%. For the ResNet152 model, the precision increased by 4.73%, the F1 value increased by 7.45%, and the AUC value increased by 12.4%. As demonstrated in the experiment, the VGG16-based image feature extraction network trained with a combination of radar image and text data outperforms the model trained with only text modality in terms of F1 value, Recall value, and ROC. While the model trained with the combination of optical image and text only shows a slight discrepancy in AP compared to that trained with only text modality, other indicators are higher than the model trained with simply text modality. Unfortunately, on the ResNet image feature extraction network, the combination of a single image source and text failed to exceed the training results of the text modalities. This suggests that the features extracted by the ResNet network do not achieve effective fusion with the text features, thereby impacting the overall classification effectiveness to a certain extent.

When using a combination of optics, radar, and text modalities, the classification effect of the model was best. For the VGG16+Bert model, the F1 score was 0.98799, the AP was 0.98044, and Recall was 0.9887. Even the model classification based on ResNet152 as an image feature extraction network has been significantly improved, which demonstrated that a combination of three modalities was better than a single modality and a combination of two modalities.

3.4. Adaptability of the MFED on different deep-learning-based flood models

To verify the adaptability of the MFED to different deep learning models, we tested more model structures. We consistently used the Bert network for the text modality and different deep learning networks for the image modality, such as ResNet50, ResNet152, VGG16, Densenet169, and SqueezeNet, for performance comparison.

In Table 6, it’s evident that the model effect of multi-source remote sensing image as image modality in the MFED dataset has achieved good results in various network structures. Among the six network models, AP and Precision are all higher than 0.95, achieving a good classification effect. The effect of ResNet50 is better than ResNet152. This could be due to the higher proportion of non-flooded areas in remote sensing images. As a result, deeper network structures may emphasize irrelevant information and potentially interfere with the classification process. We found that VGG16 achieved the best classification effect, even though VGG16 does not have many network layers. VGG16 uses multiple convolutional layers with smaller convolutional kernels instead of convolutional layers with larger convolutional kernels. In this way, the flood areas in the remote sensing images are captured more effectively, thus achieving better classification results.

Table 6. Comparison of experimental results between the MFED and other multimodal datasets.

	MFED (Image (Optical)+TEXT)					RSITMD (Image+TEXT)
Network\Indicators	F1	AP	Precision	Recall	AUC	F1	AP	Precision	Recall	AUC
VGG16+Bert	0.94118	0.95954	0.9711	0.94915	0.93359	0.95522	0.90625	0.90625	1.00000	0.96053
ResNet152+Bert	0.82773	0.88201	0.90476	0.85876	0.79823	0.97015	0.93548	0.93548	1.00000	0.97368
ResNet50+Bert	0.95798	0.97725	0.98830	0.95480	0.96101	0.89552	0.80556	0.80556	1.00000	0.90789
Densenet169+Bert	0.92437	0.95376	0.97041	0.92655	0.92229	0.91045	0.82857	0.82857	1.00000	0.92105
SqueezeNet+Bert	0.86555	0.88815	0.90055	0.92090	0.81291	0.94030	0.87879	0.87879	1.00000	0.94737
	MFED (Image (Optical+Radar)+TEXT)					RSICD (Image+TEXT)
Network\Indicators	F1	AP	Precision	Recall	AUC	F1	AP	Precision	Recall	AUC
VGG16+Bert	0.97899	0.98044	0.98315	0.98870	0.96976	0.95749	0.94358	0.96169	0.95802	0.95746
ResNet152+Bert	0.93277	0.95665	0.97076	0.93785	0.92794	0.98583	0.97959	0.98479	0.98855	0.98565
ResNet50+Bert	0.94538	0.97291	0.98810	0.93785	0.95253	0.99393	0.99063	0.99240	0.99618	0.99378
Densenet169+Bert	0.94958	0.95856	0.96610	0.96610	0.93387	0.96154	0.94715	0.96198	0.96565	0.96127
SqueezeNet+Bert	0.95378	0.9562	0.96111	0.97740	0.93132	0.95547	0.94179	0.96154	0.95420	0.95555

The ROC curves are shown in Figure 8, which shows that when multiple models are tested with the same parameters, the effect of using multi-source remote sensing imagery in multimodal data is superior to the combination of single remote sensing imagery and a text modality. The excellent classification effect of the MFED for various deep learning models fully verifies its applicability.

Figure 8. ROC curves of different modality combinations for each model.

We compared the performance of MFED (Image (Optics)+TEXT) and MFED (Image (Optics+Radar)+TEXT), both of which train models based on the MFED dataset, except that the former image modality uses only optical images, while the latter image modality uses radar and optical images. Table 6 shows that the six multimodal models using different image feature extraction networks all have good classification effects. The classification effect of multimodal far exceeds that of a single modality, which indicates the advantage of multimodal data. The impact of using radar and optical image as image modality in the five models is better than that of using optical image only as image modality. The effect on the ResNet50 model is not exceeded, which may be because the deep network structure is not entirely suitable for mining multimodal features. We will further explore the more suitable network structure for mining multimodal features.

3.5. Comparative analysis of the MFED with traditional multimodal remote sensing datasets

To verify the validity of the MFED as a multimodal remote sensing dataset, we compared it with traditional multimodal remote sensing datasets. Traditional multimodal remote sensing datasets encompass a limited variety of remote sensing imagery. ISPRS (Cramer, 2010) includes Infrared-Red-Green (IRRG) images and DSM data extracted from the Lidar point cloud. The RIT-18 dataset (Kemker et al., 2018) includes six VNIR spectral bands and two NIR bands not included in ISPRS benchmarks. The image modality data in the MFED contains seven different types of remote sensing satellite images. In addition, the MFED’s text modality includes news reports with richer semantic information.

The MFED dataset is designed to provide samples for deep learning models and verify its effectiveness for training various deep learning models. To achieve this, we evaluate MFED’s performance on different models. To validate the validity of the data evaluation method, we also used other multimodal datasets (RSITMD and RSICD) to train the models on the same model structure. We choose the RSICD and the RSITMD for comparative experiments with the MFED. Considering that the RSICD and the RSITMD only have a single image modality and a text modality, we experimented with the combinations of optical images and text and optical images, radar, and text on different models. The MFED was devised for flood event classification, while the RSICD and RSITMD are aimed at multi-classification tasks and include various remote sensing imagery types, such as rivers, airports, mountains, cities, etc. To assess the effectiveness of the classification model structure with multimodal data, we selected positive and negative examples from the RSITMD and the RSICD, focusing on river shapes and bridges that approximate the shape of rivers. This choice validates the suitability of our chosen classification model structure for multimodal data. However, this reduces the available data in RSICD and RSTIMD, thereby increasing the likelihood of overfitting in the model.

Table 6 shows the performance of the MFED, RSICD, and RSITMD datasets in the same deep learning model structure for training. RSICD (Image+TEXT) represents the model classification effect in different networks using Image and TEXT modalities data in the RSICD dataset. Similarly, RSITMD (Image+TEXT) represents the model classification effect in the RSITMD dataset. The basic model architecture we used achieved good classification results on both RSITMD and RSICD datasets, especially in the RSCID dataset. The feature extraction network based on ResNet50 and Bert achieved more than 99% results in almost every index. This validates that our proposed model structure can be used to verify the validity of multimodal datasets.

Additionally, it is observed in the experiment of the RSITMD dataset that the Recall value of the test set achieves 1 in six image feature extraction networks. This phenomenon arises due to the limited number of samples in the RSITMD dataset. To verify the universality of the model structure, only two categories in the RSITM dataset are selected for the classification experiment. Consequently, there are fewer samples available for model training. Moreover, the text modality of RSITMD contains simple, similar statements, enabling Bert networks to rapidly learn text features for category differentiation, leading to the overfitting of deep learning models.

The images in the RSITMD are in GeoTIFF format, so the training results are close to the MFED. The images in the RSICD are in JPEG format, and the data are read in RGB format. Due to the low resolution, the deep, complex network structure failed to achieve better results. For the combination of a single image modality and text modality, the classification effect of our proposed MFED is better than the RSITMD. We have verified that the combination of optical images, radar, and text can lead to better classification results in multiple models, which fully proves the validity of multimodal data.

4. Discussion

Using trained deep learning models has proven to be instrumental in enhancing the efficiency of flood event analyses. Our goal is to provide training and validation data for deep-learning-based global flood models and to compensate for the lack of semantic information in traditional multimodal remote sensing datasets. The MFED offers a worldwide repository of multi-source remote sensing images, including optical and radar data, and integrates diverse remote sensing data with authentic textual information. The MFED ensures rapid retrieval based on text semantics and provides a multi-dimensional reference for flood event analyses. Notably, the MFED extends across a broader temporal and spatial spectrum and offers more detailed and authentic textual records. It allows for an in-depth analysis of flood events in multiple dimensions and serves as a valuable source of model training data and model validation support for global flood models.

To verify the multimodal advantage of the MFED in model training, we compared multimodal data with single-modal data. Our experiments involved various models utilizing both single modality data from MFED and different combinations of modalities, fully verifying the effectiveness of the dataset. In the comparison experiment between single modality and combined modalities, we maintained uniform parameters and employed identical deep learning model structures to compare the outcomes of model training. The results show that using multimodal data significantly improves classification performance compared with using a single modality. When we combine all modalities, the multimodal dataset achieves a better classification performance on all models. For the VGG16 model, the precision was the highest at 0.98315, which is 18.94% higher than when using optical images alone and 15.66% higher than when using radar alone. This validates that the classification effect of multimodal data is better than that of a single modality and the combination of two simple modalities. Remote sensing images carry geographic information that can potentially correlate with textual semantics. Different satellite images emphasize varied observation aspects, resulting in complementarity and correlation among multi-source remote sensing images. While visual features and spatial details are provided by the image modality, contextual nuances and semantic descriptions are offered by the text modality. Integrating these modalities and exploring their internal correlations can enhance the performance of classification and retrieval tasks. Multimodality data allows us to explore hidden connections between textual semantics and visual features in remote sensing images, thereby advancing the application of multimodal data analysis.

To verify the MFED’s utility as a multimodal dataset, we compared its effectiveness in model training with that of traditional multimodal remote sensing datasets. We used different model architectures and explored combinations of different modalities for training to perform a comprehensive comparison. The results show that the MFED performs as well as other multimodal remote sensing datasets in model training. The applicability of the MFED as a model training dataset was verified through the application to the global flood classification model. The structured hydrological information in the MFED can also be used as validation information to evaluate model and algorithm effectiveness. In the experiment, we use the multimodal feature vector concatenation and a full connected layer to combine these features, which is insufficient for mining the correlation between multimodal features. In addition, we chose two categories of water and bridge in RSITMD and RSICD for classification experiments, which resulted in less training data. The experimental results are prone to overfitting. Our subsequent research will also use more complex model structures to explore the correlation between multimodal features further.

Multimodal data can fully exploit the advantages of different types of remote sensing imagery; therefore, research on multimodal remote sensing datasets is increasing. We also compared the MFED with traditional multimodal remote sensing datasets. Table 7 displays an overview of the recently published public datasets within remote sensing. The MFED dataset is a multimodal flood event dataset comprising multi-source remote sensing images and text modality rich in flood-related semantics. It offers sufficient semantic rich flood event related text modality data abstracted from both Global Flood Database and news reports on the Internet. Furthermore, it retains abundant geographic information on the multi-source imageries while maintaining the imageries with their original spatial resolution. Practically, the typical multimodal remote sensing datasets consist of different types of remote sensing imageries. However, the absence of textual data makes these multimodal image data impractical in flood event analysis due to the shortage of flood-related semantic information. Generally, most existing multimodal remote sensing datasets contain some text modality data. Still, these data are usually a simple self-description of the image data scene instead of flood correlated semantic information. For instance, textual descriptions within the RSITMD multimodal dataset are simply straightforward data depictions, like “There are some trees outside the playground.” In contrast, our MFED dataset takes full advantage of textual semantics in flood event analysis by applying flood-related text modality, especially the geographic information of flood location and the cause, casualties, and description of the flood. These flood-related textual data are abstracted from Global Flood Database and flood related news reports on the Internet. The text modality data abstracted from the authentic news articles are not limited to the temporal and spatial scope information of floods, but also the detailed information of the flood event, like the cause of the flood and the disaster situation, the countries primarily affected, the duration of the event, etc. Take flood news from the Internet, for example, which could state “People in Myanmar appear to have been warned that a cyclone was approaching, (10 feet) high storm surge that flooded and destroyed entire villages.”. The information abstracted from this news as a text modality of our MFED datasets may encapsulate more elaborate semantic details, including the cause of the flooding (cyclone) and the location (Myanmar). Following this, the text modality of our MFED enriched with textual representations harbor deeper semantic associations and could finally enable the discovery of latent relationships between text and visual modalities. Furthermore, to fit deep learning model training, some existing multimodal datasets typically convert imageries into some specific format at sampled lower resolution with fixed sizes such as 256$\times$256$\times$3. Instead of trivial data format conversion and data sampling, MFED datasets retain abundant geographic information (like coordinate system, geographic location, and projection parameters) of the multi-source imageries while maintaining the imageries with their original spatial resolution. Therefore, for MFED datasets, the information loss and degradation during data converting could be avoided.

Table 7. Comparison of publicly available multimodal remote sensing imagery datasets.

Dataset	Source	The number of instances	Data type	Application scenario	The number of Modalities
SEN12MS-CR (Ebel et al., 2020)	ESA	122,218	Sentinel-1, Sentinel-2	Cloud removal	2
SEN12MS (Schmitt et al., 2019)	Google Earth	180,000	Sentinel-1, Sentinel-2, MODIS	Classification	3
MiniFrance-DFC22 (Hänsch et al., 2022)	IGN BD ORTHO, IGN RGE ALTI	5,016	VHR aerial, DEM, Land use/cover maps	Semantic segmentation	3
Track MSD (Malkin et al., 2021)	NAIP, Google Earth, NLCD	2,250	NAIP data, Landsat-8, NLCD data	Change detection and classification	3
HOUSTON 2018 DATASET (Prasad et al., 2020)	NCALM at the University of Houston	1,000	LIDAR, Color imager, Hyperspectral imager	Classification	3
ISPRS-2D (Cramer, 2010)	BSF Swissphoto	71	Infrared-Red-Green (IRRG) images, DSM	Detection and reconstruction	2
RSICD (Lu et al., 2017)	Google Earth, Baidu Map, Tianditu	10,921	JPG, TXT	Remote sensing image captioning	2
RSITMD (Yuan et al., 2022)	Google Earth, Baidu Map, Tianditu	4743	Geotiff, TXT	Remote sensing image captioning	2
MFED	Google, ESA, USGS	4,083	Radar, Optical images, TXT	Classification	3

The Global Flood Database was initially adopted for data collection since it could provide the spatiotemporal range of locations where the flood event occurred. Nevertheless, the MFED is still a relatively small sample dataset, and the amount of images is impractical for a global-scale flood event analysis. Therefore, the MFED dataset also continually collect the flood related news reports from the Internet using a web crawler during model training. The flood-related semantic information from news reports is adopted as text modality and employed to capture the image data. In the following work, we also lay an effort to train a retrieval model to continuously augment the datasets by incorporating news records that have been meticulously retrieved and authenticated by domain experts.

5. Conclusion

The significance of flood event analysis is paramount in the context of increasingly frequent flood disasters. With the application of deep learning models in flood event analysis, high-quality flood event datasets are very important. To fulfill this requirement, we propose MFED, a flood event dataset with global coverage. MFED aims to mitigate the inherent limitations of traditional flood datasets, such as uneven geographical distribution, reliance on a single image type, and inadequate semantic information. The MFED integrates multi-source remote sensing imagery and textual information to provide a multimodal dataset of flood-prone regions worldwide, which can be utilized for training deep learning models. By utilizing multi-source remote sensing images like optical and radar, we can access a wider range of visual semantic features in flood-related remote sensing images. Furthermore, MFED incorporates news reports and flood event information with rich semantic information as text modalities, enhancing flood event analysis by providing richer and more realistic textual semantic features. We conducted full comparative experiments on single modality and multimodal combinations, proving that the MFED is effective for training multiple models. This comparative analysis confirms that multimodal data are better than single-modality data for flood classification. In the future, we will further improve the amount and types of remote sensing imagery and authentic text information in the MFED and explore the potential relationships between multimodal data at a deeper level.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The MFED dataset is available for download from the Science Data Bank at the following link: https://doi.org/10.57760/sciencedb.14826.

Additional information

Funding

This research was jointly supported by the National Natural Science Foundation of China [Grant No. 42071413] and the GHfund C [Grant No. 202302039381].

Notes on contributors

Zhixin Zhang

Zhixin Zhang received a B.E. in Network Engineering from the China University of Geosciences (Wuhan) in 2021 and is currently pursuing a master’s degree in Signal and Information Processing at the University of Chinese Academy of Sciences.

Yan Ma

Yan Ma is an Associate Professor at the Aerospace Information Research Institute, Chinese Academy of Sciences. Her research interests are focused on remote sensing, big data computing, data center cloud computing, cluster computing, and performance optimization.

Peng Liu

Peng Liu is an Associate Professor at the Aerospace Information Research Institute of the Chinese Academy of Sciences. His research interests include remote sensing imagery processing and deep learning.

Notes

1. PyStac document: https://pystac.readthedocs.io/en/latest/.

2. Sentinelsat document: https://Sentinelsat.readthedocs.io/en/stable/.

3. URL: http://eds.ceode.ac.cn/nuds/freedataquery#.