Detecting shadows in multi-temporal aerial imagery to support near-real-time change detection

Abstract

Multi-temporal aerial imagery captured via an approach called repeat station imaging (RSI) facilitates post-hazard assessment of damage to infrastructure. Spectral-radiometric (SR) variations caused by differences in shadowing may inhibit successful change detection based on image differencing. This study evaluates a novel approach to shadow classification based on bi-temporal imagery, which exploits SR change signatures associated with transient shadows. Changes in intensity (brightness from red–green–blue images) and intensity-normalized blue waveband values provide a basis for classifying transient shadows across a range of material types with unique reflectance properties, using thresholds that proved versatile for very different scenes. We derive classification thresholds for persistent shadows based on hue to intensity ratio (H/I) images, by exploiting statistics obtained from transient shadow areas. We assess shadow classification accuracy based on this procedure, and compare it to the more conventional approach of thresholding individual H/I images based on frequency distributions. Our efficient and semi-automated shadow classification procedure shows improved mean accuracy (93.3%) and versatility with different image sets over the conventional approach (84.7%). For proof-of-concept, we demonstrate that overlaying bi-temporal imagery also facilitates normalization of intensity values in transient shadow areas, as part of an integrated procedure to support near-real-time change detection.

Keywords:

• shadow classification
• shadow normalization
• shadow detection
• post-hazard damage assessment
• change detection

Introduction

Timely information regarding land surface features that are damaged, modified, or relocated by natural hazards may improve strategies and reaction times of emergency responders. Visual or automated analysis of digital aerial frame images acquired before and after an event are a basis for detecting changes to assess the condition of critical infrastructure, including bridges, electric power stations, dams, hospitals, and major roads. Visual image interpretation may be inefficient when rapid assessment of many different sites is required. Image pre-processing and change detection software are therefore valuable, yet must be efficient for applications in which the timeliness of information is critical (Lippitt and Stow 2015).

Development of (1) robust point matching algorithms which enable automated geometric co-registration of multi-temporal imagery, (2) missions designed to replicate view geometry by repeat station image (RSI) acquisition which minimizes parallax distortion effects in co-registration (Coulter, Stow, and Baer 2003; Coulter et al. 2015), and (3) unpiloted aircraft system (UAS) technologies increase the potential of high spatial resolution, multi-temporal aerial imagery for change detection purposes. High spatial resolution images from IKONOS, GeoEye, QuickBird, and WorldView satellite systems also enable detection of fine-scale features on land surfaces and critical infrastructure (Wu and Bauer 2013).

Key sources of radiometric variability between multi-temporal aerial frame images include atmospheric and illumination conditions, camera exposure settings, and shadows that change in orientation or extent. Shadows may lead to errors or insensitivities in change detection, image classification, and geometric co-registration (Benediktsson & Sveinsson 1997; Wang et al. 1999). Such issues stem from spectral-radiometric (SR) differences that are similar or greater in magnitude to those of the changed features of interest, time-varying illumination effects on image texture, or limited dynamic range of digital number (DN) values in shadow areas.

Binary classification (also termed detection) of pixels in shadow is the basis for either masking (removing) or radiometrically normalizing (brightening) shadowed areas of an image. Shadow normalization (also termed compensation or restoration) refers to the process of adjusting DN values of pixels within shadows in order to approximate directly-illuminated conditions (Li, Zhang, and Shen 2014). Shadow classification methods may be categorized as: (1) model-based techniques which involve a priori knowledge of sun angle, feature geometries, surface elevations, and other variables – such techniques are generally not preferable due to the amount of information required (Jiang et al. 2013); (2) pixel-based techniques that exploit the SR properties of shadows including color and intensity, and do not rely on knowledge of scene geometry or structure (Arévalo, González, and Ambrosio 2008); and (3) object-based image analysis (OBIA), which may leverage spatial relationships as well as SR information (Guo, Dai, and Hoiem 2011; Liu and Yamazaki 2012; Huang and Bu 2015).

Employing OBIA in shadow classification presents the following difficulties: (1) segmentation parameters depend on the scale and SR properties of features, which vary greatly between scenes; (2) over-segmentation produces an excess of fragments, which require aggregation based on particular criteria; and (3) computational time can be lengthy (Huang and Bu 2015), while OBIA requires considerable human effort and expertise. Normalizing compound shadows (which are cast upon several contiguous material types) based on OBIA requires image classification on a material type basis, in order associate shadowed and directly-illuminated materials of similar reflectance, color, and texture (cf. Li, Zhang, and Shen 2014). Developing appropriate segmentation and object classification parameters is time expensive, and often overlooked as an efficiency-limiting factor in applied remote sensing (Laliberte, Browning, and Rango 2012). Often in conjunction with OBIA, image matting has been utilized in many shadow restoration studies to account for reflectance variability within shadow segments (Wu et al. 2007; Guo, Dai, and Hoiem 2011, 2013; Gong and Cosker 2016; Khan et al. 2016; etc.). Though it reduces artifacts in shadow normalization, image matting may obscure textural irregularities (e.g., fractures) that are targets in change detection. For the reasons above, radiometric algorithms and pixel-based operations that are computationally efficient (cf. Sanin, Sanderson, and Lovell 2012) seem preferable to OBIA for shadow classification and normalization in time-sensitive, emergency response applications.

Hue–saturation–intensity (HSI) transforms can be derived from color (RGB) images and utilized to enhance shadows, which tend to exhibit low intensity (I) and high hue (H) values (Tsai 2006; Li, Zhang, and Shen 2014). The ratio of hue to intensity (H/I) is a widely used index for classifying shadows. Histogram thresholding has been used in previous studies in order to classify shadowed (S) and directly-illuminated (DI) pixels based on H/I, intensity, and other color space data (Otsu 1975; Shu and Freeman 1990; Shettigara and Sumerling 1998; Dare 2005; Chen et al. 2007; Besheer and Abdelhafiz 2015). A technique for classifying shadows by thresholding at the inflection point between two prominent modes (associated with S and DI regions) of a histogram was developed by Otsu (1975) and applied in subsequent studies (Tsai 2006; Zhou et al. 2009; Huang and Bu 2015). However, the Otsu (1975) technique requires that the shadow index image has exactly two modes of central tendency, and assumes that the value of least occurrence between these modes is the optimal threshold. Shadow classifications that rely on intensity-correlated indices and unary thresholds may exhibit inevitable errors, especially for bright objects in shadows or dark objects in directly-illuminated areas (Liu and Yamazaki 2012). Our preliminary work to classify shadows based on H/I images of complex urban scenes confirmed this limitation.

Knowledge of the SR effects of shadowing is relevant for improving shadow classification criteria. Based on in situ spectral radiance measurements of various materials under S and DI conditions at multiple points in time, Liu and Yamazaki (2012) observed that the SR effects of shadowing vary with season and atmospheric conditions (e.g., haziness). Shadowing has the least effect on radiance at shorter visible wavelengths, due to light scattering and diffuse atmospheric illumination, which are highest in blue wavelengths of visible light (Huang et al. 2004). The fraction of solar irradiance reflected in a given wavelength (i.e., spectral reflectance) depends on material composition (Liu and Yamazaki 2012). We infer that two general effects of shadowing occur with all material types: net decrease in visible radiance (correlated with image brightness), and relative increase of radiance in blue compared to green and red wavelengths. Multi-temporal imagery may reveal inherent SR changes associated with transient shadows (i.e., shadows present in only one image of a co-registered pair), and thus may provide a reliable basis for classifying as well as normalizing shadows.

Traditional approaches to shadow classification and normalization may be unsuitable to support rapid, accurate, and reliable change detection based on multi-temporal aerial frame imagery. Variability stemming from the reflectance properties and spatial arrangement of surficial materials represents a key challenge in shadow classification, regardless of the technique that is used. We evaluate the utility in exploiting SR differences between RSI pairs as a basis for shadow classification and normalization, in order to support detailed image change detection for heterogeneous scenes. We address the following objectives:

• empirically assess the SR effects of shadowing on various material types, in order to identify trends which are relevant to shadow classification and normalization;

• exploit SR differences in bi-temporal image sets in order to classify pixels associated with shadows that have changed in orientation or extent, which we refer to as “transient” shadows;

• determine how statistical data from transient shadow areas can be used in classifying “persistent” shadows (that are present at time-1 and time-2) based on H/I images, in a semi-automated manner;

• compare the accuracy and versatility of the proposed approach to that of classifying mono-temporal H/I images separately using histogram-based thresholds (cf. Otsu 1975); and

• demonstrate the potential in normalizing transient shadows by overlaying multi-temporal imagery (applying basic mean and standard deviation adjustments), as proof-of-concept.

Methods

Aerial frame images and geometric processing

Aerial frame images of sites in San Diego, California and Albuquerque, New Mexico (USA) were acquired in October of 2011 and 2015, respectively. The scenes covered by the four bi-temporal image sets used in this study include a hospital facility under construction, a suburban office park, a portion of San Diego State University campus, and a highway overpass bridge (Figure 1(a–h)). Each image pair contains shadows that changed from time-1 to time-2 (hereafter, t-1 and t-2). Prevalent features in the scenes include roads, parking lots, sidewalks, vegetation, motor vehicles, and buildings of various types.

Figure 1. Bi-temporal aerial frame image sets used in this study. (a and b) Hospital facility at morning and afternoon; (c and d) office park at morning and afternoon; (e and f) campus of San Diego State University during the afternoon with ~15 min between image acquisitions; and (g and h) a bridge overpass in Albuquerque, NM, imaged in the morning and afternoon. The top of each frame is oriented northward, except for (g) and (h), in which rightward is northward.

Images of the San Diego sites were captured using a Canon 5D Mark II camera (50 mm lens), and those of the Albuquerque site using a Nikon D800 (50 mm lens). All of the images were acquired at approximate spatial resolutions of 0.10–0.13 m, in 8-bit, true-color (RGB) format. The image pairs were captured at nearly identical (X, Y, and Z) coordinates with the guidance of a global positioning system, using the RSI approach (Coulter, Stow, and Baer 2003; Coulter and Stow 2005, 2008; Coulter et al. 2012, 2015; Stow et al. 2016).

Images were co-registered using a scale-invariant feature transform (SIFT) routine to generate prospective control points (Lowe 2004), and random sample consensus (RANSAC) (Fischler and Bolles 1981) to eliminate dissimilar control points and align the images based on second-order polynomial transformations. In this RANSAC procedure, we used 90% of the refined control point set for transformation, and the remaining 10% to compute route mean square error (RMSE). The RMSE values for the co-registered image sets were generally less than 0.3 m (~2 pixels). We acknowledge that widespread land surface changes in a scene (e.g., following a major disaster) could prevent image co-registration based on SIFT algorithms; the procedures of this study are applicable to scenes containing moderate changes, such as partial building failures or fractures in road surfaces due to earthquakes. The color image pairs that we captured and co-registered are shown in grayscale form in Figure 1(a–h).

Radiometric characteristics and color transformations

Detecting land surface dynamics based on changes in image brightness depends on radiometric consistency between images, and can require image normalization based on features of stable reflectance (Schott, Salvaggio, and Volchok 1988; Song et al. 2001). We observed that pixels associated with unchanged, flat surfaces (mainly sunlit concrete and asphalt) tend to exhibit differences of 2–6% when comparing DN values of the t-1 versus t-2 images. These variations in DN values could stem from differences in solar irradiance, camera exposure settings, or detector gain, and produced slight differences in the histogram distributions. We deemed histogram matching to be inappropriate for radiometric pre-normalization of the image pairs because changed features (mainly transient shadows) represent considerable portions of each scene.

We used ERDAS Imagine™ software for image processing. H and I images were derived using formulas given by Jensen (2009) (Equations (1) and (2)). The intensity images were re-scaled to the range of 0–1, as is consistent with the conventional HSI transformation. Blue (B) band values were also re-scaled from 0 to 1 then divided by intensity (yielding B/I ratio images), in order to represent the relative value of the blue band compared to the red–green–blue (RGB) average. The I and B/I image sets were used as a basis for classifying shadows and estimating their SR effects. The H/I ratio was computed for each image, and the H/I images were re-scaled to the range of 0–1, in order to render the H/I values more inter-comparable among scenes:

(1)

(2)

Characterization of shadow effects

Shadow effects on image intensity were characterized by sampling small groups of pixels associated with a range of material types in each scene, under both S and DI conditions. We extracted and averaged pixel values in 4 m² (square) polygons, which we delineated in pairs on opposite sides of shadow boundaries and on corresponding material types. The spatial unit of 4 m² was small enough to fall within the objects of interest, and large enough to minimize the influence of micro-scale texture on the pixel samples. The pixel samples used to assess shadow effects were extracted from each image individually (rather than from the same areas at t-1 and t-2), to preclude the influence of radiometric differences that are unrelated to shadowing. From the eight study images we selected a total of 137 sample pairs corresponding to asphalt, cement, soil, grass, foliar vegetation, rooftops, and painted surfaces. I and B/I values were sampled and averaged from each polygon element. Inherent radiometric differences in the image pairs (unrelated to shadowing) were assessed by extracting samples at both t-1 and t-2 from 40 of the sample polygons (mentioned above) that represent unchanged, persistently illuminated areas.

Classification of transient shadows

We utilized temporal ratios of intensity (I_ratio) and intensity-normalized blue (B/I_ratio) as a basis for classifying transient shadows, as defined in Equations (3) and (4). These temporal ratios were computed “forward” as well as “backward” in time (i.e., t-2/t-1 and t-1/t-2, respectively). This allowed us to use SR measures and thresholds consistently, irrespective of the direction of change through time:

(3)

(4)

where I_time‐1 and I_time‐2 are the intensity values at t-1 and t-2, and B_time−1 and B_time−2 are the re-scaled blue band values at t-1 and t-2.

Visual observations confirmed that transient shadow pixels exhibit negative values in the I_ratio images and positive values in the B/I_ratio images, respectively. Though some changed land surface features show similar I_ratio or B/I_ratio values as the transient shadows, the presence of negative I_ratio values and (conjointly) positive B/I_ratio values appear to be characteristic of transient shadows, and not prevalent in changed objects or surfaces. Hence, we employed a dual-criteria (I_ratio and B/I_ratio) classification approach. Figure 2(a–e) provides an example of a transient shadow classification derived by thresholding the I_ratio and B/I_ratio images.

Figure 2. Examples of images associated with the transient shadow classification procedure. Transient shadows are visible in the intensity images at (a) time-1 and (b) time-2. This figure shows the time-2/time-1 ratios of (c) intensity (I_ratio) and (d) intensity-normalized blue (B/I_ratio), wherein higher values correspond to brighter areas in grayscale. Threshold classification of I_ratio and B/I_ratio images was used to derive (e) the bi-temporal transient shadow map. The clockwise arrangement of frames (a)–(e) reflects the workflow order.

We applied the same I_ratio and B/I_ratio thresholds for classifying shadows in each of the images. Iterative classification trials showed that declines of ≥10% in intensity (I_ratio ≤ 0.9), and increases of ≥10% in intensity-normalized blue (B/I_ratio ≥ 1.1), are the most versatile criteria for classifying transient shadows in all of the images used in this study. Although slightly more accurate shadow classifications of some images were possible by varying the thresholds within 5% of the selected general/universal criteria, we applied the same thresholds to all of the images in order to evaluate their robustness and efficiency for time-sensitive applications.

Classification of persistent shadows

In a second phase of the procedure, we classified all shadows separately for t-1 and t-2 by thresholding the H/I ratio images. (Afterward, we distinguished persistent shadows from transient shadows by overlaying of the t-1 and t-2 binary classification products.) The H/I threshold values were determined statistically based on the transient shadow areas. Pixel sets of the H/I images were extracted from the transient shadows associated with t-1 and t-2, and then statistically characterized by global mean (µ_H/I) and standard deviation (σ_H/I) values. To account for the likely case that one image of a pair could lack transient shadows, we computed the µ_H/I and σ_H/I based on transient shadow pixel sets that were merged (combined) from t-1 and t-2. Figure 3(a,b) shows a scene-wide H/I histogram and one associated with the transient shadow region, from which σ_H/I and µ_H/I values are computed and used to derive an H/I classification threshold.

Figure 3. Illustration of the approach for deriving a hue/intensity ratio (H/I) classification threshold value based on the transient shadow region of an image: (a) histogram distribution of H/I pixels extracted from the transient shadow region, showing mean and standard deviation measures and (b) histogram of original H/I image and position of the threshold from (a) used to classify shadow pixels in the time-1 and time-2 images. Histograms (a) and (b) are from the hospital facility scene.

We applied multiplicative coefficients to the σ_H/I values and then subtracted the resulting scalar values from the µ_H/I values (formulas are shown in Table 1), in order to derive the H/I thresholds for each image pair. The coefficients were optimized by varying them at intervals of 0.1 σ_H/I, and visually evaluating the resultant shadow classifications. These coefficients were identified after 3–5 trials, which required less than 1 min in total per scene. Though the coefficients varied substantially between scenes, the same coefficient was applicable to t-1 and t-2 for each scene, as shown in Table 1.

Table 1. Hue-intensity classification coefficients based on the mean (µ) and standard deviation (σ) of the transient shadow regions in each image.

Scene	Image	Threshold formula
Hospital facility	t-1	µ – 1.3σ
	t-2	µ – 1.3σ
Office park	t-1	µ – 0.8σ
	t-2	µ – 0.8σ
University campus	t-1	µ – 0.1σ
	t-2	µ – 0.1σ
Bridge overpass	t-1	µ – 0.5σ
	t-2	µ – 0.5σ

We combined the transient shadow classification products (based on I_ratio and B/I_ratio) with the products of the H/I based shadow classifications using spatial unions, such that pixels associated with shadow in either classification inherit a map value of 1. The transient shadow maps exhibit greater coverage of low magnitude shadows than the H/I based maps, as well as more omission errors in the form of gaps within large shadows. The H/I based shadow maps contain fewer erroneous gaps, and hence are beneficial to join with the transient shadow maps. Figure 4 illustrates the workflow used to generate the final classification products (hereafter referred to as shadow maps) based on our approach.

Figure 4. Workflow diagram outlining the procedure for shadow classification used in this study. Items with dashed perimeters signify image data and classification products, linked to procedural step items with gray backgrounds and solid perimeters. Arrows indicate the flow of data.

We compared the shadow maps based on our approach to ones that we generated by thresholding the H/I images based on the Otsu (1975) histogram threshold selection technique. The reason for evaluating our procedure relative to that of Otsu is that, like ours, it could be implemented in a near-real-time workflow (unlike other approaches including OBIA that require intensive processing and analyst interaction). As most of the H/I histograms contained more than two modes of central tendency, it was not feasible to implement the automated thresholding approach used in other studies, which requires a bi-modal distribution. Instead, we interactively selected thresholds at the inflection point (or valley bottom) between the two modes (or peaks) of highest value in each histogram. The H/I thresholds derived from each approach are given in Table 2. Generally, the histogram-based H/I thresholds are very close to those of our procedure.

Table 2. Hue-to-intensity (H/I) threshold values used to classify shadows based on visual identification of the inter-modal inflection points, and those based upon statistics derived from the transient shadow regions.

Scene	Time	Inter-modal inflection point	Threshold based on transient shadows
Hospital facility	1	0.63	0.62
	2	0.62	0.62
Office park	1	0.47	0.49
	2	0.55	0.49
University campus	1	0.51	0.55
	2	0.53	0.55
Bridge overpass	1	0.57	0.56
	2	0.50	0.56

Post-processing and accuracy assessment of shadow maps

Multi-temporal shadow maps are necessary to normalize shadows in multi-temporal image sets. The shadow maps based our approach and those based on histogram thresholding were used to generate bi-temporal shadow maps, overlaying the classification products from t-1 and t-2. These shadow maps contain the following four classes: shadowed (t-1)/shadowed (t-2) (S–S), shadowed (t-1)/directly-illuminated (t-2) (S–DI), directly-illuminated (t-1)/shadowed (t-2) (DI–S), and directly-illuminated (t-1)/directly-illuminated (t-2) (DI–DI).

We assessed accuracy for the bi-temporal shadow maps stemming from each approach. Four-hundred verification points (pixel locations) were generated in a stratified random manner for each scene. In order to minimize bias to either shadow map set, half of the random pixels were stratified based on the shadow classes (S–S, S–DI, DI–S, and DI–DI) from each approach (i.e., fifty pixels were randomly distributed within each class of each bi-temporal shadow map). The shadow class of each verification pixel was determined based on visual interpretation of the t-1 and t-2 RGB images, independent of (without reference to) the shadow maps. We generated confusion matrices and estimated user’s, producer’s, and overall accuracy measures based on the aggregate of 1600 verification pixels.

Shadow normalization and image differencing

Bi-temporal shadow maps based on our approach provided a basis for normalizing the intensity of pixels in transient shadow areas. Prior to shadow normalization, like-class pixel aggregation (clump) and small clump filtering (sieve) operations were used to eliminate groups of S and DI pixels of contiguous areas smaller than 4 m². As smaller shadows have higher perimeter-to-area ratios, this filtering reduced the prevalence of artifacts from shadow normalization that are typical at shadow edges.

We overlaid the t-1 and t-2 intensity images in order to calculate the global statistical means and standard deviations within each transient shadow class (S–DI and DI–S) for each image. We used these statistics to normalize intensity values of the pixels classified as transient shadows, according to Equation (5) (cf. Liu and Yamazaki 2012):

(5)

where DN_i is the input pixel value, µ_S and µ_DI represent the shadowed and directly-illuminated mean values, σ_S and σ_DI represent the shadowed and directly-illuminated standard deviation values, and DN_r is the pixel value after shadow normalization.

We generated arithmetical difference images (t-2 minus t-1) based on the normalized and original (non-normalized) intensity image pairs. We applied smoothing filters (moving averages based on 9 × 9 pixel kernels) to the difference images that were based on the normalized intensity images (referred to hereafter as normalized difference images) along shadow edges, in order to reduce visible artifacts. Correcting for shadow edge (penumbra) artifacts directly and quantitatively is beyond the scope of this study.

Results

Shadow map products and assessment of shadow effects

We interpret SR sample data used to estimate shadow effects on a variety of material types and radiometric differences unrelated to shadowing, based on Table 3. Observed differences of I and B/I in the image pairs which stem from image capture or illumination variability (unrelated to shadowing) are less than 11%, and for most samples less than 5%. The average I_ratio and B/I_ratio values associated with shadowing range from 0.48 to 0.67 and from 1.13 to 1.28 (respectively) among the scenes, indicating substantially greater effects from shadowing than other radiometric variations in the image pairs.

Table 3. Summary statistics of shadowing effects on intensity and intensity-normalized blue for the paired pixel block samples, and spectral-radiometric differences observed in directly-illuminated areas of the image pairs. Bold, italicized values are compared with the I_ratio and B/I_ratio thresholds used in shadow classification: 0.9 and 1.1, respectively.

Image		Shadowed (S) and directly-illuminated (DI) areas of interest								Differences between t-1 and t-2
		Ishadowed /Idirectly illuminated (Iratio)				B/Iratio				Iratio	B/Iratio
		min	max	μ	σ	min	max	μ	σ	μ	μ
Hospital facility	t-1	0.42	0.89	0.59	0.15	1.06	1.35	1.21	0.08	0.06	0.08
	t-2	0.26	0.89	0.46	0.22	1.06	1.38	1.23	0.10
Office park	t-1	0.35	0.63	0.45	0.08	1.17	1.32	1.23	0.04	0.03	0.05
	t-2	0.35	0.74	0.52	0.09	1.09	1.29	1.18	0.05
University campus	t-1	0.36	0.75	0.54	0.10	1.11	1.31	1.21	0.05	0.09	−0.11
	t-2	0.54	0.83	0.67	0.07	1.05	1.21	1.13	0.05
Bridge overpass	t-1	0.32	0.67	0.48	0.11	1.05	1.38	1.18	0.08	0.01	0.04
	t-2	0.38	0.67	0.49	0.08	1.11	1.42	1.28	0.08

The I_ratio threshold of 0.90 (negative 10% change) used for shadow classification is greater than the maximum I_ratio value observed in the SR sample data on shadow effects (0.89). Hence, this classification threshold corresponds to shadows that are relatively low in magnitude (i.e., with lesser effect on intensity than all of the shadows we sampled). Conversely, the B/I_ratio threshold of 1.1 (positive 10% change) approximates the minimum B/I_ratio values observed in each image based on the SR samples. The ranges and standard deviations in I_ratio and B/I_ratio values associated with each scene indicate considerable variability in the SR effects of shadowing among the different material types that we sampled. In summary, the I_ratio and B/I_ratio thresholds we applied are inclusive of shadows in various material types, do not lead to classification errors by capturing other radiometric variations in the image sets, and are versatile for the different scenes.

Bi-temporal shadow maps based on our procedures are shown in Figure 5(a–d). Each of the scenes contains extensive shadows occupying 25–50% of the area, and cast upon a variety of material types with unique reflectance properties. The classifications derived by histogram-based thresholds coincide closely with those shown in Figure 5(a–d), and thus were not included. Differences between the shadow map sets from each approach are described quantitatively based on accuracy assessment.

Figure 5. Bi-temporal shadow classification maps of the four study scenes, corresponding to (a) frames (a and b) (hospital facility), (b) frames (c and d) (office park), (c) frames (d and e) (university campus), and (d) frames (f and g) (bridge overpass) shown in Figure 1. The shadow maps shown were filtered using clump and sieve operations.

Accuracy assessment

The accuracy of shadow maps based on the Otsu approach ranges from 79% to 86% (producer’s) and from 78% to 92.3% (user’s) among the shadow classes, with an overall accuracy of 84.7%. Our approach resulted in substantially higher accuracies, ranging from 87.2% to 96.6% (producer’s) and from 88.6% to 97.7% (user’s), with overall accuracy of 93.3% (Table 4). Accuracy of the transient shadow classes is most critical for our purposes relating to change detection. Our approach resulted in 9–17% higher map accuracy in the S–DI and DI–S classes, as compared to the Otsu approach. Lower accuracy in transient shadows based on the Otsu approach is evident despite the fact that unique H/I thresholds were applied at t-1 and t-2 (refer to Table 2). Variation in producer’s accuracy between the S–DI and DI–S classes is only ~2% for our approach, compared to ~6% based on the Otsu approach (Table 4).

Table 4. Accuracy assessment results for (a) maps based on the Otsu thresholding approach and (b) maps based on the novel approach evaluated in this study.

	Based on verification				Accuracies
	No shadow	Shadow t-1	Shadow t-2	Persistent shadow	Producer’s accuracy	User’s accuracy
(a) Based on classification
No- shadow	358	50	45	6	88.2	78.0
Shadow t-1-1	4	303	2	29	79.1	89.6
Shadow t-2-2	43	5	309	28	85.4	80.3
Persistent shadow	1	25	6	384	85.9	92.3
				Average:	84.6	85.1
(b) Based on classification				Overall accuracy:		84.7
No- shadow	390	8	15	4	96.1	93.5
Shadow t-1-1	7	372	3	38	96.6	88.6
Shadow t-2-2	8	0	341	15	94.2	93.7
Persistent shadow	1	5	3	390	87.2	97.7
				Average:	93.5	93.4
				Overall accuracy:		93.3

Shadow normalization and image differencing

The bi-temporal shadow maps provide a basis for normalizing intensity values within the transient shadow regions, using the mean and variance equalization defined in Equation (5). The original (non-normalized) and normalized intensity image pairs of the Hospital facility scene are shown in Figure 6(a–f), along with arithmetical difference images based on the original and normalized intensity image pairs. The brightness variation resulting from shadows in the original images leads to the false appearance of change in the difference image, which reduces detectability of actual physical changes. The difference image based on the shadow-normalized products reduces the shadow-related artifacts and renders actual land surface changes (e.g., construction features and translocated vehicles) more apparent.

Figure 6. Comparison of intensity image difference products based on the normalized versus original intensity images. The original intensity images at (a) time-1 and (b) time-2 were subtracted to generate (c) a difference image, wherein neutral (no or little change) is gray while black and white shades represent high negative and positive intensity changes, respectively. The frames labeled (d) and (e) represent the time-1 and time-2 intensity images, which were normalized in transient shadow areas, and frame (f) shows their arithmetical difference.

Finer-scale results of shadow normalization can be observed in Figure 7(a), which shows a cement surface in transient shadow that was normalized to approximate DI conditions with high fidelity. Artifacts of commission error in the transient shadow classification are evident through the shadow normalization in Figure 7(b), where some rays of light penetrating a tree canopy were misclassified as shadow and over-brightened. Figure 7(c) and 7(d) illustrates cases where the shadow normalization partially reduced (yet did not fully compensate for) the effects of shadowing. Variability in shadow normalization effectiveness within and among scenes is expected (based on this simple global mean and standard deviation adjustment), due to variation in spectral reflectance which modulates the shadowing effect in complex ways. In general, the normalized difference images in Figures 6(a–f) and 7(a–d) contain fewer and less prominent visual artifacts related to shadow than those based on the original intensity images.

Figure 7. Magnified subsets of the intensity and arithmetical difference products, based on normalized versus original intensity images. The subsets are located within the (a) hospital facility, (b) office park, (c) bridge overpass, and (d) university campus scenes. Subset images are co-organized as: [1] original intensity image at time-1 and [2] time-2 used to generate [3] an arithmetical difference image; subsets [4] and [5] represent the time-1 and time-2 intensity images which were normalized in transient shadows, and [6] their arithmetical difference. Neutral (no or little change) is gray while black and white shades represent negative and positive changes, respectively.

Discussion

The rationale, objectives, and procedures of this study were developed to support post-hazard damage assessment based on high spatial resolution RSI pairs. The problem of shadows in change detection necessitates solutions that are efficient and suitable for disaster response applications. In this context, we suggest that shadow classification procedures require: (1) parameters and thresholds that are versatile or can be determined quickly; (2) computational efficiency; (3) a capacity to accommodate shadow index (e.g., H/I) images with more than two histogram modes; and (4) high classification accuracy (especially in transient shadows) for a range of material types with varying reflectance properties, as are typical in built environments. The key objective of this study is to assess whether differences of intensity and intensity-normalized blue values in multi-temporal image sets can be used effectively as criteria for classifying transient shadows. Second, we explored how statistical information extracted from transient shadow areas could be used to determine thresholds for classifying shadows in each image separately, as a basis for differentiating persistent and transient shadows.

The general shadow map accuracy range of 87–97% based on our approach is comparable to that reported by Tsai (2006) (89–97%), whom had applied the Otsu thresholding approach to classify H/I images. This result is notable given the greater variety of materials with unique reflectance properties in our images (compared to those used by Tsai 2006). Based on preliminary research, we found that H/I thresholds could not be optimized to capture all shadows without considerable errors of commission or omission, and that classification accuracy often differs in t-1 versus t-2. Integration of the transient shadow maps (capturing low magnitude shadows) and the H/I based shadow maps (capturing mainly deep, dark shadows) was helpful in addressing this problem, which stems from differential reflectance among material types and from temporal variation in illumination conditions. The assessment of 1600 verification points demonstrated that our approach yielded shadow maps at generally 8% higher accuracy than the conventional (Otsu 1975) approach for the same scenes. Importantly, we observed the greatest increase in map accuracies in the transient shadow classes (9–17%) compared to the histogram-based thresholding approach.

Conclusions

This study demonstrates the utility of a novel, efficient approach to shadow classification, which exploits spectral-radiometric variations in bi-temporal aerial image sets. Our shadow detection procedure has several advantages over the histogram-based thresholding technique introduced by Otsu (1975), including: (1) higher overall classification accuracy of 8% based on our experimental imagery; (2) accuracies ranging from 9–17% higher in the transient shadow areas; and (3) it does not require a bi-modal distribution in the shadow index (H/I) image. Given these accuracy improvements and the capacity to accommodate multi-modal histograms, our approach is implicitly more effective for scenes containing diverse material types with unique reflectances.

We attribute the effectiveness of this procedure to several factors. Changes in intensity and intensity-normalized blue values associated with transient shadows tend to exhibit characteristic minima (i.e., change by at least ten percent) in a range of material types commonly found in built environments, even under variable diffuse illumination conditions. Inclusion of the intensity-normalized blue criterion also reduces misclassification of changed features as shadows, according to our observations. The transient shadow maps and those based on H/I image thresholding helped to improve coverage of low magnitude shadows and false occlusions within shadows, respectively. A key advantage of our procedure is that only one varying parameter must be determined for each scene (the H/I thresholding coefficient). This coefficient may be identified prior to a disaster using a baseline (pre-disaster) bi-temporal image set, and thereafter applied in overlay with post-disaster imagery to classify shadows in a more automated fashion.

Reasonably accurate shadow maps facilitated normalization of transient shadows based on a simple mean and standard deviation stretch, using values obtained through overlaying the bi-temporal image sets. This normalization is efficient, fully automated, and appears beneficial for change detection based on image differencing. We acknowledge that further work is required to resolve the complex effects of differential material reflectance for the purpose of shadow normalization. A clearer understanding of this phenomenon (as it relates to multi-temporal imagery and change detection) will be critically informative, even for more complex approaches including OBIA and processing in the true-color domain. We highlight the particular utility in data obtained via overlay of multi-temporal imagery, which support an efficient, integrated procedure for classifying as well as normalizing shadows. This general line of research will facilitate more accurate and efficient change detection, and thereby improve the reliability of post-hazard emergency response efforts.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

The work was partially funded by the National Science Foundation Directorate of Engineering, Infrastructure Management and Extreme Events (IMEE) program (Grant Number G00010529), United States Department of Transportation (USDOT) Office of the Assistant Secretary for Research & Technology (OST-R) Commercial Remote Sensing and Spatial Information (CRS&SI) Technologies Program (Cooperative Agreement Number OASRTRS-14-H-UNM), and by the Department of Geography at San Diego State University.