Spatially simplified scatterplots for large raster datasets

Figures & data

Figure 1. Scatterplot and a GAM fitted line for Normalized Difference Vegetation Index (NDVI) (Y) and ETM + Band 4 (X). Data are standardized. The R package mgcv was used to fit the GAM (Wood 2006; Hastie and Tibshirani 1990).

Figure 2. Alpha blending scatterplot of a subset (1000 × 1000) of the same data in Figure 1, with alpha value set to 0.05. The alpha value specifies the level of transparency for the data points, ranging from 1 for opaque to 0 for black. Even at such a low alpha level, excessive overplotting remains a problem.

Figure 3. Two-dimensional binned kernel smoothing scatterplot.

Notes: Black dots are bins with a single data point. Data are the same as Figure 2.

Figure 4. Scatterplot with contours. Due to the lack of sufficient memory to render the plot with the large data-set in Figure 2, 10,000 data points are randomly generated from a standard normal distribution.

Figure 5. Binned scatterplot with the same data in Figure 2.

Notes: A 30-by-30 grid was used to bin the original data (1000 × 1000). Brightness symbolizes the counts of data points in each bin. Changing the bin size can achieve different levels of generalization. The shape of the bins can be either rectangular or hexagonal.

Figure 6. Nested lattice hexagon binning with the same data in Figure 2.

Notes: For the hexagons, size of the inner point represents counts, while hue represents the hierarchical categories and serves as borders between the hexagons. The graph was generated with the R package hexbin.

Table 1. Summary statistics of the simulated data-set.

	Mean	Standard Deviation	Skewness	Kurtosis	Moran coefficient	ρ	ρxy
X	−0.0193	8.2652	0.033	3.000	0.7705	0.9658	0.4688
Y	176.1985	3689.0297	0.443	7.432	0.2221	0.5136	0.4688

Figure 7. Simulated data-set X (left) and Y (right).

Notes: Y = 0.5X + X² + X³. The size of the image is 100 × 100 cells. See Table 1 for the statistical summaries.

Figure 8. Scatterplot with a local regression fit for the simulated data-set X and Y.

Figure 9. Remote sensing data-set used in the experiment.

Notes: Landsat ETM Band 4 (left) and derived NDVI (right). Image dimension is 3000 rows by 3000 columns, with a spatial resolution of 30 m. Data were obtained on 6 July 2002, near Palisades, Idaho. Figures 1–6 are various scatterplots for ETM Band 4 and NDVI.

Table 2. Summary statistics of the remote sensing data-set.

	Mean	Standard deviation	Skewness	Kurtosis	Moran coefficient	ρ	ρxy
NDVI	0.4375	0.1729	−0.55	3.547	0.9174	0.9852	0.2281
NIR	90.25	24.26	0.90	6.820	0.8940	0.9840	0.2280

Figure 10. Illustrations of sample sites for three sampling schemes: random (left), regular (center), and hexagon stratified random (right). The background image is for the variable X, with 100 rows by 100 columns. Variable Y is not shown here. Effective sample size () was calculated based on ρ_x = 0.9658, ρ_y = 0.5136, ρ_xy = 0.4688. Due to geometric restrictions, actual sample sizes are 2809, for regular sampling, and 2438, for hexagon stratified random sampling.

Table 3. Summary statistics for simulated variables from the resampled sets.

Parameter	Mean (CI)	Standard deviation(CI)	Skewness	Kurtosis	Moran coefficient	D (p-value)	ρ	ρxy (CI)
X	−0.02 (−0.18, 0.14)	8.27 (8.16, 8.38)	0.033	3.0	0.77		0.9658	0.47 (0.45, 0.49)
Random	−0.06 (−0.36, 0.24)	8.15 (7.94, 8.36)	−0.010	2.92	0.56	0.012 (0.91)	0.8482	0.44 (0.41, 0.47)
Regular	−0.03 (−0.33, 0.28)	8.30 (8.08, 8.52)	0.039	3.02	0.55	0.011 (0.95)	0.8881	0.46 (0.43, 0.49)
Hex stratified	−0.02 (−0.35, 0.30)	8.28 (8.05,8.52)	0.025	2.94	0.52	0.013 (0.85)	0.8540	0.45 (0.42, 0.49)
Hex2	0.10 (−0.36, 0.55)	8.38 (8.06, 8.70)	0.065	3.08	0.44	0.017 (0.85)	0.7611	0.51 (0.47, 0.56)
Y	176.20 (103.89, 248.50)	3689.03 (3637.90–3740.15)	0.443	7.43	0.22		0.5136	0.47 (0.45, 0.49)
Random	99.05 (−35.11, 233.2)	3612.31 (3517.42,3707.17)	0.445	8.78	0.13	0.017 (0.52)	0.3930	0.44 (0.41, 0.47)
Regular	225.41 (87.37, 363.46)	3732.9 (3635.26, 3830.51)	0.461	8.33	0.12	0.018 (0.46)	0.3570	0.46 (0.43, 0.49)
Hex stratified	165.03 (20.3, 309.8)	3646.07 (3543.70, 3748.4)	0.740	9.43	0.12	0.016 (0.68)	0.3490	0.45 (0.42, 0.46)
Hex2	78.73 (−124.3, 281.8)	3772.66 (3629.00, 3916.24)	0.025	6.43	0.13	0.018 (0.82)	0.3590	0.51 (0.47, 0.56)

Notes:

n = 10, 000, , , . Ninety five percentage confidence intervals for selected statistics are recorded in the parentheses. For the K–S test, the p-values accompanying D are recorded in the parentheses. The row Hex2 reports the results for the second round of resampling (n = 2438, n^*=1326).

Figure 11. Spatially simplified scatterplot consisting of the point cloud and local/global fitting lines overlays on the same graphic features from the original data.

Notes: Grey dots are the original data (n = 10,000 data points). Black dots are data points from the resampled sets (left: random sampling, n^* = 2785; center: regular sampling, n^* = 2809; right: hexagon stratified random sampling, n^* = 2438). The red curve (with a yellow error band) and the red dashed line are from the sampled set. The white curve (with a blue error band that appear grey due to color mixing with the yellow band) and the green line are from the original data.

Figure 12. Spatially simplified scatterplots (point clouds with local/global fits) from three resampling schemes (left: random sampling; center: regular sampling; right: hexagon stratified sampling).

Figure 13. A random sample (n = 1326, right) of the hexagon stratified random sample set (n = 2438, left).

Note: Sample sites are shown as yellow dots.

Figure 14. Spatially simplified scatterplot of the simulated data using a two-stage resampling scheme.

Notes: In the left graph, the fitted curves for the original data are colored in white (with a blue error band) and green. The red curve (with a yellow error band) and the red dashed line are from the sampled set. The original data points are colored in light grey. In the right graph, the LOESS line is red with a blue error band. The dashed line is the OLS fit.

Table 4. Summary of resampling statistics for the remote sensing data-set.

Parameter	Mean (CI)	Standard deviation(CI)	Skewness	Kurtosis	Moran coefficient	D (p-value)	ρ	ρxy (CI)
NDVI	0.4375 (0.4373, 0.4376)	0.1729 (0.1728, 0.1730)	−0.550	3.547	0.9174		0.9852	0.2281 (0.2275, 0.2282)
Hex stratified	0.4374 (0.4366, 0.4382)	0.1728 (0.1722, 0.1733)	−0.545	3.496	0.6729	0.0012 (0.94)	0.8579	0.2571 (0.2527, 0.2615)
Random	0.4379 (0.4363, 0.4396)	0.1727 (0.1715, 0.1739)	−0.550	3.511	0.5974	0.004 (0.66)	0.8028	0.2638 (0.2546, 0.2730)
Regular	0.4383 (0.4366, 0.4400)	0.1733 (0.1721, 0.1745)	−0.5682	3.640	0.5738	0.004 (0.45)	0.8035	0.2525 (0.2433, 0.2617)
Rec stratified	0.4379 (0.4363, 0.4396)	0.1723 (0.1711, 0.1735)	−0.550	3.500	0.5834	0.003 (0.76)	0.8098	0.2579 (0.2487, 0.2672)
NIR	90.25 (90.23, 90.27)	24.26 (24.25, 24.27)	0.900	6.820	0.894		0.9840	0.2281 (0.2275, 0.2282)
Hex stratified	90.26 (90.145, 90.373)	24.24 (24.16, 24.32)	0.900	6.820	0.5556	0.001 (0.99)	0.8084	0.2571 (0.2527, 0.2615)
Random	90.18 (89.95, 90.42)	24.37 (24.21, 24.53)	0.930	6.940	0.4558	0.003 (0.73)	0.7245	0.2638 (0.2546, 0.2730)
Regular	90.28 (90.05, 90.51)	24.39 (24.23, 24.55)	0.920	6.930	0.4297	0.001 (1.0)	0.7165	0.2525 (0.2433, 0.2617)
Rec stratified	90.30 (90.07, 90.53)	24.19 (24.02, 24.35)	0.890	6.840	0.4371	0.004 (0.62)	0.7246	0.2579 (0.2487, 0.2672)

Notes:

Stage-one effective sample size based on ρ_NIR = 0.9840, ρ_NDVI = 0.9852, n = 9000,000, ρ_NIR−NDVI = 0.2281, . Actual hexagons constructed for stratified sampling: 174,212. Stage-two (in italic type) effective sample size based on ρ_NIRa = 0.8084, ρ_NDVI = 0.8579, n = 174,212, ρ_NIR−_NDVI = 0.2571, . Actual sample sizes: random, 42,128; regular, 42,025; stratified, 42,047.

Figure 15. An overlay of scatterplots from the original data and the resampled set.

Notes: Light grey point cloud: original data (3000 × 3000 points). Darker grey point cloud: hexagon stratified random sampled data (174,212 points). Black wavy curve: GAM fitted line from the original data. Violet wavy curve: GAM fitted line from sample data. Blue curve: a cubic polynomial fit to the sample data. Red curve: a quartic polynomial fit to the sample data. Green curve: a quartic polynomial fit to the original data. Black dash: linear fit to the original data. Violet dash: linear fit to the sample data.

Figure 16. Scatterplot of data from the stage-two regular resampling.

Notes: Light grey point cloud: stage-one hexagon random sampled data (174,212 data points). Darker grey point cloud: stage-two regular sampled data (42,025 data points). Black wavy curve: GAM fitted line from stage-one data. Violet wavy curve: GAM fitted line from stage-two resampled data. Blue curve: a cubic polynomial fit to the stage-two resampled data. Red curve: a quartic polynomial fit to the stage-two resampled data. Green curve: a quartic polynomial fit to the stage-one data. Black dash: linear fit to the stage-one data. Violet dash: linear fit to the stage-two resampled data. X axis = normalized Band 4 radiance, Y axis = normalized NDVI, NDVI = [(Band 4 − Band 3)/(Band 4 + Band 3)]. NDVI is a simple algebraic combination of remotely sensed spectral information (eg Bands 4 and 3 from Landsat Enhanced Thematic Mapper) that provides meaningful information about vegetative structure (leaf/cellular structure) and condition (chlorophyll content). Generally, high NDVI values indicate that photosynthetically active plant biomass is present.

Wood, S. 2006. Generalized Additive Models: An Introduction with R. London: CRC Press.

 Google Scholar

Hastie, T. J., and R. J. Tibshirani. 1990. Generalized Additive Models. vol. 43. London: CRC Press.

 Google Scholar