A Low-Cost Mobile Mapping System (LCMMS) for field data acquisition: a potential use to validate aerial/satellite building damage assessment

Abstract

Among the major natural disasters that occurred in 2010, the Haiti earthquake was a real turning point concerning the availability, dissemination and licensing of a huge quantity of geospatial data. In a few days several map products based on the analysis of remotely sensed data-sets were delivered to users. This demonstrated the need for reliable methods to validate the increasing variety of open source data and remote sensing-derived products for crisis management, with the aim to correctly spatially reference and interconnect these data with other global digital archives. As far as building damage assessment is concerned, the need for accurate field data to overcome the limitations of both vertical and oblique view satellite and aerial images was evident. To cope with the aforementioned need, a newly developed Low-Cost Mobile Mapping System (LCMMS) was deployed in Port-au-Prince (Haiti) and tested during a five-day survey in February–March 2010. The system allows for acquisition of movies and single georeferenced frames by means of a transportable device easily installable (or adaptable) to every type of vehicle. It is composed of four webcams with a total field of view of about 180 degrees and one Global Positioning System (GPS) receiver, with the main aim to rapidly cover large areas for effective usage in emergency situations. The main technical features of the LCMMS, the operational use in the field (and related issues) and a potential approach to be adopted for the validation of satellite/aerial building damage assessments are thoroughly described in the article.

Keywords:

• Mobile Mapping System
• building damage assessment
• earthquake
• low cost
• validation
• crisis management
• Digital Earth
• geospatial data

1. Introduction

Among the major natural disasters that occurred in 2010, the Haiti earthquake was a real turning point concerning the availability, dissemination and licensing of a huge quantity of geospatial data. In a few days several damage assessment map products based on the analysis of remotely sensed data-sets were delivered to users, confirming that rapid impact assessment after a catastrophic event is crucial for initiating effective emergency response actions (Brunner et al. 2010).

The acquisition of field data, supporting the aforementioned impact assessment, in areas hit by severe earthquakes is indeed a hard task, especially in the first hours after the event. To overcome the accessibility and time constraint issues, the use of earth observation (EO) data in earthquake contexts, especially for damage assessment purposes, has been widely proposed and a number of results have been presented after every event, mostly based on optical data and manual interpretation (Polli et al. 2010). Furthermore, the availability of virtual globe geo-browsers allows us to easily and quickly display EO data and perform integrated geospatial analysis, going into the direction of a truly global, collaborative linking of systems, as mentioned in the Digital Earth vision.

As far as satellite-based impact assessment is concerned, the main issue that requires an in-depth investigation is the reliability of the provided information, i.e. the expected accuracy of the damage assessment according to the technical features of the available satellite/aerial imagery (e.g. sensor type, ground sample distance, off-nadir angle and spectral resolution).

Recent studies (Saito et al. 2010) highlight that vertical imagery (and in certain conditions also oblique ones) may be limited in discriminating the level of damage of some buildings. Furthermore, in the summary ¹ of the ‘2nd International Workshop on Validation of geo-information products for crisis management’ (JRC, Ispra–Italy, 12–13 October 2010), it is explicitly reported that a validation of a joint damage assessment (using airborne images) performed with around 6000 geo-tagged photos collected in the field gave an overall accuracy of only 60%. It is therefore crucial to rely also on information acquired in the field, especially by means of Global Positioning System (GPS) devices that allow the acquired information to be geo-tagged.

The goal of this article is to describe the technical features of a GPS-based device Low-Cost Mobile Mapping System (LCMMS) aimed at acquiring movies and single georeferenced frames. This is done according to three crucial keywords related to crisis management activities: timeliness (data acquired in the shortest time possible after the event), geo-localisation (acquisition of the geographical coordinates of the data) and open access (data are distributed without licensing constraints). Apart from the use of the LCMMS-acquired data for damage assessment purposes, the article is focused on the potential use in validation protocols. The specific goal is to estimate the accuracy of building damage assessments based on high-resolution aerial images, performing a thorough comparison between field observations (LCMMS) and remotely sensed ones. The results of a validation exercise carried out on more than 150 buildings surveyed in Haiti after the January 2010 earthquake will be shown and discussed.

2. Geomatics supporting field data acquisition

Several methods and technologies can be adopted to acquire geospatial data in a short time frame. That means recording not only the acquired information (i.e. surface geometry, temperature and brightness) but also the spatial location of the data. Among those technologies, modern mass market devices for navigation purposes had a remarkable improvement in these last years with increased performance in terms of raw data precision and quality. Devices such as GPS and Inertial Measurement Unit (IMU) equipped mobile phones and digital cameras (e.g. Ricoh G700SE² and Sony DSC-HX9V ³ ), personal GPS navigators, as well as more professional devices integrating a rugged PDA, GPS, camera, laser range finder or digital compass (e.g. ikeGPS ⁴ ) are now available on the market, but they cannot be used for systematic, continuous surveys on wide areas.

The commercial investment in mass market devices had a positive spillover on low-cost GPS receivers and IMUs. Additionally, modern GPS receivers and IMU platforms allow estimation of a real-time solution (e.g. PVT: Position Velocity and Timing). Other possibilities include calculus algorithms such as the Kalman filter or integration algorithm such as the loosely coupled algorithm. These modern GPS receivers are also able to collect raw data to improve the positioning accuracy through a post-processing step. This procedure, also known as ‘relative positioning’, could be realised using different parts of the GPS signal: the pseudorange (i.e. C/A, the most noisy part of the GPS signal) and the carrier-phase (i.e. L1, the least noisy part of the GPS signal). A centimetric level of accuracy could be obtained if the carrier phase data are used (Groves 2008a). This feature is particularly interesting since a reliable real-time solution is provided (Piras et al. 2010). In confirmation of this trend, nowadays several commercial off the shelf (COTS) digital cameras are equipped with GPS receivers, adding geo-tagging capabilities to the device. Geographical identification metadata are consequently stored along with the digital image itself.

2.1. Mobile Mapping Systems

One of the approaches recently developed to achieve the aforementioned goals is based on Mobile Mapping Systems (MMS), terrestrial (usually) vehicles equipped with several sensors as digital cameras, Terrestrial Laser Scanners, odometers, GPS receivers and Inertial Navigation Systems (INS), to acquire georeferenced data.

The aim of MMS is to acquire three-dimensional information using different sensors, capturing all data at the same time by means of a triggering system. GPS time is usually considered the reference time (El-Sheimy 1996, Groves 2008b). The acquired data can be georeferenced through a GPS–INS integration approach (Škaloud 1999) which allows estimation of the vehicle position and the attitude epoch-by-epoch, storing a high number of georeferenced data in a short period. Unfortunately, some features of traditional MMSs may be limiting for some applications (including crisis management), specifically:

• high costs (>300 k€);

• skilled users are required;

• not easily adaptable to any vehicle.

The aforementioned considerations led the Geomatics group of the Politecnico di Torino to start a new research line aimed at developing a proof-of-concept MMS, using only low-cost sensors. The first output of such research was a low-cost version of an MMS tailored to road cadastre requirements. Three geodetic Global Navigation Satellite System (GNSS) receivers, two IMU and three webcams were adopted for this solution, with the aim to keep the price lower than 30 k€ and to achieve a positioning accuracy better than 0.1 m (Piras et al. 2008). Response to natural disasters has different needs, especially in terms of horizontal positional accuracy that is certainly less stringent than the aforementioned case. A metric level of accuracy (<10 m) generally fits end user requirements: e.g. in the case of road accessibility assessment after earthquake events, it is important to correctly identify which element of the road network is not practicable, and not the exact location of the cause of the impracticability (e.g. presence of debris). Immediately after a disaster, the prompt availability of low-cost systems devoted to geometric surveys that can easily be fit to almost any type of vehicle is crucial. Keeping the cost as low as possible allows deployment of a larger number of systems, consequently increasing the survey speed and minimising the cost implication of a potential loss of devices (e.g. due to incidents or riots related to post-emergency situations).

2.2. Low-Cost Mobile Mapping System

The system has been further developed to create a low-cost solution, decreasing the accuracy of the acquired data (2–5 m but still in a range suitable for emergency response purposes as well as for validation purposes) and focusing on the ease of use of the device. The main technical features of the LCMMS are listed below:

• compact device, easily transportable;

• fast installation on almost any terrestrial vehicle;

• user friendly (can be used also by local relief organisation with a few hours of training);

• very low-cost (<2500 €) solution;

• suitable geometric accuracy (2–5 m);

• positioning is obtained in real-time (without GPS post-processing).

Specifically, the current version of the system is composed of four commercial webcams (Logitech Quickcam C905 with Carl Zeiss lens) and one low-cost GPS receiver (u-blox AEK 4 Antaris, which allows carrier phase on L1 to be collected).

All sensors are embedded in a beacon light with a magnetic support (Figure 1), allowing a quick and easy installation over a vehicle's roof. Manual rotation of each camera allows the field of view to be corrected according to the vehicle's roof shape and specific goals of the survey.

Figure 1. LCMMS system: four webcams and Global Positioning System (GPS) antenna embedded in a beacon light with a magnetic support.

The LCMMS requires a connection to a high-performance laptop during the acquisition phase; technical features of the PC used during the test field are reported in Table 1 (and can be considered the minimum requirements). The laptop represents the most expensive component of the system.

Table 1. Technical features of the PC used in the field test.

PC feature
CPU	Intel Core (TM) 2 Extreme 32-bit
CPU clock	2.80 GHz
RAM	3.50 GB
Hard Disk	250 GB
Operating System	Windows XP SP2

2.2.1. Time synchronisation

As mentioned in the previous section, one of the MMS’ crucial characteristics is to ensure a reliable time synchronisation between the installed instruments (e.g. laptop and GPS receiver), allowing each measurement to be coupled with its position in an absolute spatial reference system. To cope with the time synchronisation issue, expensive triggering systems are adopted in commercial MMSs, which are not suitable for a low-cost approach. An ad hoc solution was therefore developed aimed at ensuring synchronisation between the two different time scales of the LCMMS, specifically between the PC and GPS.

PC clock stability depends on the CPU, since a different RAM percentage is allocated to the PC clock according to other ongoing CPU tasks, and is therefore far less accurate than GPS time. Unfortunately an easy clock synchronisation carried out at the beginning of the survey is not a suitable solution, since generally the time offset is not constant, as is clearly shown in Figure 2.

Figure 2. GPS-PC time offset trend (GPS time as reference).

The diagram highlights that the time offset average trend is approximately 2 s/h. Considering for example a 2 hour survey, which is the minimum acquisition time of the field test surveys, the maximum horizontal displacement caused by the time offset error is about 56 m (when adopting a survey speed of 50 km/h). To minimise displacements due to time delays it is necessary to synchronise the PC time to the GPS. This issue was solved by implementing a software procedure that stores both GPS time [broadcasted from the GPS receiver by a National Marine Electronics Association (NMEA) message] and PC time with a sample rate equal to 1 Hz. The time delay is estimated considering two consecutive GPS epochs. The PC acquisition time for each frame is corrected considering the offset calculated for the current epoch.

2.3. LCMMS software

2.3.1. Acquisition software

The software devoted to the data acquisition (Figure 3), developed by ITHACA (Lingua et al. 2009), has two main components: the Sample Grabber and GPS data logger modules.

Figure 3. Acquistion software graphical interfaces. (a) Sample grabber, (b) GPS Data Logger, (c) Acquisition windows.

2.3.1.1. Sample Grabber module

The module is devoted to the acquisition of images from each webcam, storing the acquired frames in the local drive.

The laptop graphics card acquires the data flow sent by the webcam via a USB connection and converts it into a continuous video. The Sample Grabber module intercepts the output of the graphics card and extracts single frames at a regular time interval by means of DirectShow APIs, ⁵ a multimedia framework that allows to perform various operations with media files or streams; they were originally distributed within the DirectX libraries (version 11, the Microsoft API developed for gaming purposes ⁶ ). Each frame is stored on a local drive and is flagged with the PC time and unique code of the webcam. Up to six instances of this module can run at the same time, each one working with a different webcam linked to the same PC.

The resolution of the acquired frames can be customised (up to 1280×1024 pixels) for each webcam independently, as well as the acquisition rate (up to 15 fps).

The best balance of resolution, acquisition rate and number of modules running simultaneously has to be defined, mainly depending on the purpose of the acquisition and on the laptop performance (in terms of CPU, RAM, graphic card features and hard disk latency). Based on field acquisition experience in an urban environment, a set of pre-defined configurations have been identified: an application set-up procedure guides the user step-by-step, allowing the parameters to be set in a few minutes.

2.3.1.2. GPS data logger module

This module is devoted to the acquisition of the NMEA navigational message transmitted by the GPS receiver to the PC through a USB connection. Exploiting the availability of the GPS receiver drivers, the software extracts the navigational string and stores it in a text file together with the timestamp reporting the PC time (refer to Section 2.2.1 for details). The frequency of the acquisition depends on the characteristics of the GPS receiver (1 Hz for the LCMMS).

Specifically, raw GPS data acquisition is done using the U-blox software (U-blox 2011). This software allows data acquisition (including carrier phase on L1), to monitor crucial parameters, e.g. satellite visibility, DOP and SNR.

The GPS data logger module also records specific, predefined information (tags) during the survey that are expressed as a Boolean datum at the end of the standard NMEA format together with the Point of Interest (PoI) category. This information can be exploited in the post-processing phase.

2.3.2. Post-processing software

The post-processing step is independent from the acquisition. It is aimed at extracting the value added information and the subsequent storage in a DBMS, exploiting the PoI tagged during the survey, if available. This operation would obviously benefit from the availability of an up-to-date digital map or remotely sensed imagery used as a backdrop to display the position of the camera. It was therefore a natural decision to develop the post-processing software in a GIS environment.

In this specific case, the solution was developed by extending ESRI ArcMap base functionalities exploiting ESRI ArcObjects through C# scripts (Zeiler 2001). Similar post-processing tools can be developed in an open source environment, with the benefit of reducing software-related costs. The application implemented a specific window devoted to managing the acquired data, displaying the data on a map, extracting the features of interest (automatically georeferencing the camera position) and storing the selected information in an ESRI feature class.

From an operational point of view, a single window (which can be opened from the ArcMap toolbar) is sufficient for the management of the aforementioned operations. The working window (Figure 4) has the following main elements:

• four Image Boxes showing the acquired frames;

• several buttons to navigate the frames forward and backward or to activate the animation mode;

• different Text Boxes to fill and store details and notes.

Figure 4. Post-processing software interface.

From a logical point of view, the software executes instructions at different levels: from simple image management with a pan/zoom option to the creation of point features in a feature class and filling the attribute table with details and notes entered by the user. The core of the software is the determination of the coordinates of the camera position for each acquired frame. This goal is achieved by estimating a linear regression of the spatial coordinates acquired by the GPS data logger module, which are time-dependent variables. The time synchronisation between the GPS and PC time is carried out by calculating the time offset as described in Section 2.2.1.

An additional capability of the software is to generate the geometry of the track followed by the vehicle during the survey, by means of point interpolation and line simplification ESRI tools.

3. LCMMS field test. The 2010 Haiti earthquake experience

On 12 January 2010, Haiti was hit by a catastrophic earthquake of magnitude 7.0 Mw, with an epicentre near the town of Léogâne, approximately 25 km west of Port-au-Prince, Haiti's capital. The earthquake caused major damage in Port-au-Prince, Jacmel, Léogâne and other settlements in the region. Amongst the widespread devastation and damage throughout Port-au-Prince and elsewhere, vital infrastructures necessary to respond to the disaster were destroyed or severely damaged.

A few weeks after the earthquake, UN WFP requested that ITHACA make available a small team of experts to be deployed in Port-au-Prince. The duty of the team was to join the UN WFP staff and, in general, support the whole humanitarian mission with specific competencies on data acquisition, processing, analysis and output.

The team was requested to be, as far as possible, autonomous in terms of instruments (hardware and software), with only electricity, network connectivity and printing facilities being granted in the Log Base, the place inside the fenced-in Port-au-Prince International Airport area where most of the agencies and NGOs were concentrated. The ITHACA team, composed of three experts, left Italy on 14 February 2010 with:

• three laptops with GIS/RS software installed ⁷ ;

• GPS devices, standalone or integrated with a digital camera; and

• two complete LCMMS sets, including dedicated laptops.

Safe storage, weights and volumes of the instruments were a major constraint because of the long travel, especially on the connection from Santo Domingo, Dominican Republic, to Port-au-Prince, granted by UN flights allowing only a limited freight weight (20 kg per person, including personal items). Exceeding that limitation would imply a longer stay in Santo Domingo, limiting the operational support capabilities of the team.

The team arrived in Port-au-Prince on 17 February and in condition to be operational starting from the following day. One of the main tasks was to acquire geospatial data in the field, mainly aimed at assessing the damage to buildings. Two months after the earthquake, at the end of the field mission, it was still common to see groups of civil engineers walking on the streets of Port-au-Prince escorted by fully-armed soldiers, compiling files on each single building and deciding if it should be demolished. A semi-automated general survey of the city would have helped in better prioritising the assessment zones (in respect to the initial satellite-based damage maps) and consequently optimising efforts during such surveys.

As far as the LCMMS use is concerned, system flexibility and ease of use indeed represented a great advantage. The only way to travel around the city was to use the car rental service provided by UN WFP, reserving it a day in advance (with no possibility to have the same car or the same driver always assigned). So, the adaptability of the LCMMS to almost any type of vehicle was crucial (Figure 5).

Figure 5. LCMMS device installed on a WFP car in Port-au-Prince.

The system was used during a five-day survey in Port-au-Prince and nearby localities (Figure 6); thanks to it, it was possible to acquire about 490,000 georeferenced frames (~39 GB, 4 webcams at 960×720 pixels at 7 fps) and 58,000 GPS points (1 Hz). Table 2 shows the survey calendar and the technical details of the acquired data, while Figure 7 shows some samples of the acquired frames.

Figure 6. LCMMS survey tracks in Port-au-Prince.

Figure 7. Examples of LCMMS georeferenced frames acquired during the survey. (a) Collapsed building, (b) Restricted road, (c) Temporary shelter.

Table 2. Acquisition survey in Haiti – Statistics.

Date	No. of frames	No. of GPS points	Size (GB)
24 February 2010 (am)	49,578	3689	3.03
24 February 2010 (pm)	53,529	5432	4.10
27 February 2010 (am)	86,952	9807	7.33
27 February 2010 (pm)	43,846	5897	2.23
28 February 2010 (am)	27,107	8171	2.37
1 March 2010 (am)	93,737	9563	6.95
2 March 2010 (am_1)	14,377	1511	1.32
2 March 2010 (am_2)	69,975	7186	6.95
2 March 2010 (pm)	50,980	6600	4.80
Total	490,081	57,856	39.08 GB

The survey was carried out by a team of two people: one in charge of the LCMMS operational status (controlling the system, acquiring tags, starting and stopping it) and one providing route indications to the driver and acquiring complementary georeferenced data by means of a stand-alone GPS device and a camera with an embedded GPS.

3.1. Main field test outcomes

As far as the acquisition phase is concerned, there are three main comments:

• limitations in survey capability must be considered, not only in relation to what can be surveyed from the road (normally limited to one or two building facades) but also with specific conditions that are quite common while operating in emergency situations, such as blocked or impracticable roads due to physical or security limitations. Several Port-au-Prince areas were accessible only accompanied by military escort (not easy to obtain);

• accurate planning of each survey was required maximise survey efficiency and avoid excessive data redundancy;

• natural illumination conditions may strongly affect the quality of surveyed data and the capability of controlling the acquisition, since some areas of the image may be saturated, not allowing information extraction from the affected pixels. The limitation particularly evident in Port-au-Prince was generally strong illumination conditions, with high variability related to cloud coverage variability. If a pre-defined, fixed camera sensitivity configuration is employed, it has to be regularly adapted during the survey—not an easy task in a continuously shaking, 4-wheel drive car on unpaved or damaged roads. Automatic adaptation to light conditions may result in inhomogeneous colour saturation of frames acquired by the four webcams.

4. Potential use of LCMMS data for building damage assessment validation

Apart from the use of LCMMS field data for damage assessment purposes, one of the goals of the research is to verify the potential use in a validation protocol context. The specific goal was to estimate the accuracy of building damage assessments based on high-resolution aerial images, performing a thorough comparison between field and remotely sensed information.

Reliable estimates of the semantic accuracy of remote sensing-based assessment will allow extrapolation of the expected accuracy of the damage assessment vs. the technical features of the available satellite/aerial imagery (e.g. sensor type, ground sample distance, off-nadir angle, spectral resolution).

4.1. High-resolution aerial imagery damage assessment

A subset of the data acquired with LCMMS in Haiti has been used to set up a possible approach for the validation of assessments based on high-resolution aerial images. This imagery, characterised by a 0.15 m spatial resolution, was acquired on 17 January 2010 and made immediately and universally accessible as base layers in Google earth and Google Maps by the Google Crisis Response Team.

A few blocks of Port-au-Prince covered by both the LCMMS survey and the high-resolution aerial imagery were chosen (Figure 8). Given the nature of LCMMS imagery, only buildings facing the roads were taken into account. Those buildings have been analysed by an operator in a GIS environment and classified according to the EMS-98 damage grades. The European Macroseismic Scale (EMS) is the basis for evaluation of seismic intensity in European countries and is also used in a number of countries outside Europe. Damage grades in the range of 1–5 should ideally represent a linear increase in the strength of shaking. They do this only approximately, and are heavily influenced by the need to describe classes of damage which can be readily distinguished by the operator.

Figure 8. Car survey ground track (red lines) and building damage assessment (based on high-resolution aerial imagery) considered for the validation test.

Specifically, the following damage grades, adapted to an interpretation based on vertical imagery, have been adopted in the aerial imagery classification step:

• Grade 1: negligible damage

• Grade 3: substantial to heavy damage

• Grade 4: very heavy damage

• Grade 5: destruction

In detail, ‘Grade 2-moderate damage’ has been excluded since it was not possible to detect this type of damage using vertical imagery. As far as the ‘Grade 3: substantial to heavy damage’ is concerned, an a priori classification approach was adopted, attributing all buildings attached to heavily damaged buildings (classified as Grade 4) to the Grade 3 category.

The output of the high-resolution imagery classification step consists of a point features data-set, as shown in Figure 8.

4.2. LCMMS data analysis

The same analysis carried out on the high-resolution images had to be carried out on the LCMMS datasets. Exploiting the functionalities of the post-processing software described in a previous section, the vehicle position and related frames were overlaid onto the aerial orthoimages. The information about the heading of the vehicle is derived automatically by the post-processing software module and is exploited to change the orientation of the vehicle symbol on the screen. Such an approach supports the georeferencing of the features of interest identified by means of visual interpretation, matching the elements on the pictures to the same elements on the orthoimages.

To better navigate through the LCMMS images, a set of checkpoints was created and used as a ‘bookmark’ to easily move from a specific road to another. Only a few checkpoints were actually needed to efficiently move through the sample area. A visual comparison of the LCMMS images and the orthoimages allows common points to be identified and consequently to verify if the data integration was carried out correctly.

The damage assessments based on aerial images and LCMMS were compared adopting the same aforementioned EMS-98 damage grades and considering only the buildings that are present on both assessments. The LCMMS survey is therefore limited to roadside buildings.

Figure 9 shows a clear example of a building whose damage estimate was shown to be accurate, with the LCMMS data interpretation and the aerial image classification being the same (Grade 5 – Destruction).

Figure 9. Comparison of a building damage classification on the aerial orthoimage and on the LCMMS frames – Classification agreement example.

Figure 10 shows an example of an erroneous classification of a building. The two concrete-roof buildings have been classified as completely destroyed. By means of a thorough analysis of LCMMS data, it is clear that the perception was completely obscured by the left house's shadow. What looked like a collapsed roof was in fact an irregular shadow on an otherwise still standing building. Obviously only exploiting the point of view from the ground, as provided by the LCMMS system installed on a car, it was possible to determine the wrong classification which otherwise would have been unnoticed.

Figure 10. Comparison of a building damage classification on the aerial orthoimage and on the LCMMS frames – Erroneous classification example.

In some cases, high fences or other obstacles blocking the view from the car made it impossible to verify the damage and validate the remotely sensed data. This intrinsic limitation of the LCMMS could be partially overcome by means of a system upgrade (i.e. using a higher point of view and lenses with a wider field of view), but this approach contrasts with some of the system requirements (ease of use and low cost) as well as with its applicability in an emergency response context.

4.3. Validation outcomes

The comparison between the LCMMS and aerial assessments was carried out by calculating a confusion matrix (Table 3).

Table 3. Low Cost Mobile Mapping System (LCMMS) vs. aerial assessments confusion matrix.

		Aerial assessment
		Grade 5	Grade 4	Grade 3	Grade 1	Total
LCMMS-based	Grade 5	34	2	0	1	37
assessment	Grade 4	1	1	0	2	4
	Grade 3	0	0	1	0	1
	Grade 1	7	7	32	80	126
	Total	42	10	33	83	168
	Class Accuracy (%)	80.95	10.00	3.03	96.39	69.05 (global)

Generally speaking, a confusion matrix contains information about actual and predicted classifications done by a classification system. In our case, a four-class confusion matrix was built. Each column represents classes derived from remotely sensed images while each row represents classes derived using LCMMS data. The main diagonal contains information about the number of damaged buildings correctly assessed while the rest of the values are erroneously predicted cases.

The Class Accuracy (CA) was calculated using Equation (1):

(1)

where:

a=matrix element

i=matrix column index

j=matrix row index

n= number of matrix columns=number of matrix rows (square matrix)

As can be seen from the confusion matrix, damage Grade 1 (negligible damage) was extremely accurate (96%) with only three cases where an aerial-based assessment turned out to be wrong. Also, damage Grade 5 (destruction) had an 80% accuracy rate, with only a few building damage grades overestimated. Almost all the buildings that were predicted to have sustained Grade 3 damage as described in Section 4.1 presented on LCMMS data inspection little or no damage at all. It is therefore confirmed that Grade 3 damages are almost impossible to detect using vertical imagery and it can be highlighted that an a priori approach (to automatically assign Grade 3 to all the buildings attached to Grade 4 ones) is not feasible. In fact, in the present case it would have been better to ‘declassify’ all the buildings marked as damaged to the third grade to first grade damage, considering in the analysis stage that about 5% of the Grade 1 buildings presented some damage.

A similar consideration can be made for Grade 4, which was highly overestimated and again confirms that incompletely collapsed buildings are difficult to detect from vertical imagery.

The Equations (2) and (3) allow calculation of Damage Underestimation (DU) as well as Damage Overestimation (DO):

(2)

(3)

where:

a=matrix element

i=matrix column index

j=matrix row index

n= number of matrix columns=number of matrix rows (square matrix)

The confusion matrix shows that the upper diagonal is almost empty, meaning that there was no significant underestimation of damage (DU<3%), or, differently said, except in a few rare cases, the damage grade assigned on the aerial images turned out to be accurate or with a lower level of damage. About 28% (DO) of the building have been overestimated, i.e. assigned to a greater damage grade.

5. Conclusion

The main goal of the article was to show the result of an operational field test of a newly developed LCMMS device as well as to use the acquired data for damage assessment validation purposes.

As far as the LCMMS field test is concerned, the system proved to be suitable for operational use during crisis situations, mainly thanks to its portability, versatility and ease of use. Some issues were related to the fact that the test was performed in a real emergency scenario, with blocked or impracticable roads. The integration of a road navigator in the acquisition software, showing in real-time the vehicle position and the pre-defined survey plan, will definitively optimise the survey efficiency in terms of time vs. the covered area's size. The main limitation that emerged during the test in Port-au-Prince was the influence of natural illumination conditions that may strongly affect the quality of the acquired imagery. To cope with this issue specifically, the research team is testing low-cost digital sensors with wider lenses and improved automatic light balance capabilities, to replace the webcams currently adopted.

Furthermore, the potential use of new GPS receivers able to track new signals and constellations (e.g. GLONASS, GALILEO, new L5 GPS) will ensure better georeferencing performance than the current receiver.

From an operational emergency response point of view, on the basis of the field experience it could be advised to deploy similar systems immediately after the Search and Rescue (SAR) phase, to make updated geospatial data available to the humanitarian community as soon as possible but without interfering with the SAR operations. Such data can be used for different purposes, providing information for both emergency Response and Recovery phases, such as prioritising the areas to be assessed on the ground by expert teams as well as providing reference maps for the reconstruction plan.

As for the use of LCMMS data, the building damage assessment exercise highlights that a point of view on the ground is crucial, especially for earthquake emergency response and recovery purposes, overcoming the limit of vertical satellite/aerial imagery. In some cases vertical imagery does not allow verification of the presence of damage on building facades or identification of vertical movements of the buildings, even if their magnitude has an order of a few meters (the case of soft storey collapse or ‘pancake’ effect). LCMMS imagery has indeed a very high spatial resolution which allows better recognition of objects and their shape, but on the other side the LCMMS does not allow performance of a complete survey of all the affected buildings because of accessibility issues when mounted on a vehicle (e.g. the internal side of a city block, a courtyard or simply parts of the city isolated by inaccessible roads).

The identification and interpretation of features of interest, e.g. buildings and their damage grade, on LCMMS frames is a task that can be carried out by an average PC user familiar with the EMS-98 classification grades. The georeferencing of such information requires a visual comparison with the available map used as a backdrop in the GIS environment, though it is facilitated by the automatic positioning of the camera centre on the map. To facilitate this step, ad hoc procedures, based on photogrammetric procedures, have been developed by the Politecnico di Torino group. The new version of the post-processing software will allow 3D measurements (points, distance, angle, etc.) on the acquired frames and consequently to automatically georeference the features of interest identified on the LCMMS imagery, avoiding the manual identification of the same on the reference map/orthoimage. Specifically, with each element (e.g. damaged building) captured in several sequential frames, it is possible to apply a photogrammetric approach exploiting the availability of different images covering the same area (along-track overlapping). Furthermore it is possible to generate a rectified image (e.g. a geometrical projection on a defined plane) applying a homographic transformation.

The field data analysis step is indeed time consuming: each single framed building has to be visually interpreted, allowing grade damage assessment of a few buildings per minute. According to the Haiti earthquake experience, it is clear that, in major events, only a volunteer-based approach allows delivery of the damage assessment information within a few days after image availability.

Finally, the validation exercise conducted on more than 168 buildings highlighted that the overall semantic accuracy of a building damage assessment based on high-resolution aerial orthoimages is approximately 70% considering a four-grade damage classification. The confusion matrix shows that the main issue when analysing vertical images is the identification of intermediate-level damages, while the accuracy related to the identification of destroyed or undamaged buildings is, respectively, 81% and 96%.

Acknowledgements

The authors would like to thank Prof. Horea Bendea, who practically realised the LCMMS prototype in a very short time, in order to be operational during the Haiti emergency and Dr Fabio Giorgi, who collaborated on the development of the acquisition and post-processing software.

Notes

1. http://globesec.jrc.ec.europa.eu/workshops/valgeo2010/Summary_of_VALgEO2010.pdf

2. http://www.ricoh.com/r_dc/g/g700se/

3. http://www.sony.co.uk/product/dsc-h-series/dsc-hx9v

4. http://www.ikegps.com/

5. http://msdn.microsoft.com/en-us/library/windows/desktop/dd375454(v=vs.85).aspx

6. http://msdn.microsoft.com/en-us/directx

7. Thanks to the ESRI licensing policy applied in the specific case of the Haiti emergency.