Application of Near-Infrared Hyperspectral Imaging for Detection of External Insect Infestations on Jujube Fruit

Abstract

A hyperspectral imaging system has been built for detecting external insect damage and acquiring reflectance images from jujubes in the near-infrared region of 900–1700 nm. Spectral information was extracted from each jujube, and six optimal wavelengths (987, 1028, 1160, 1231, 1285, and 1464 nm) were obtained using principal component analysis. The first principal component images (PC-1) using the selected six wavelengths were obtained for further image processing. The detection algorithm was then developed based on principal component analysis and two-band ratio (R1160/R1464) coupled with image subtraction algorithm (R1160-R1464). An identification accuracy of 93.1% for insect-infested jujubes and 100% classification rate for the intact ones were achieved. The results of this research demonstrated that it is feasible to discriminate insect-infested jujubes from intact jujubes using the near-infrared hyperspectral imaging technology.

Keywords:

• “Lingwu Changzao” jujube
• Hyperspectral imaging
• Insect infestation
• Principal component analysis
• Band ratio
• Image subtraction

INTRODUCTION

Zizyphus jujuba Mill.cv. “Lingwu Changzao,” also called Ningxia Changzao, is one of the fine fresh-eating varieties with over 800 years history of cultivation in China. It is favored because of its thin edible skin, sweet whitish flesh, and pleasant flavor. As the origin of jujube fruit, the planting area and production of “Lingwu Changzao” in Ningxia ranks first in China. Over the past ten years, Lingwu City in Ningxia Hui Autonomous Region has increased its fruit export to many countries. Long-distance transportation requires delivering higher quality and more consistent fresh jujube fruit from the origin in order to meet the quality standards upon arrival at the destination.^[1,^2]

Accurate evaluation of jujube quality is challenging because the individual fruit is small and has great variations in external as well as internal quality characteristics. Insect damage consists of internal and external insect infestations. The internal infesting insects lay their eggs within the jujube, so their larvae can feed on the inside, external infesting insects feed on the outside of the jujube. The external defects such as mildew, insect damage, bird pecks, and phytotoxicity seriously affect the quality of red jujubes, and thus, external insect damage is a key indicator for assessing external quality of jujube fruits. So it is necessary to remove the defect ones from the normal ones. Insects cause severe damages to jujube fruits by darkening the skin color of the fruits and making holes on the surface. Carposina niponensis walsingham is a particularly problematic pest that causes external damage to jujubes. Its larvae often bore holes from the jujube fruits’ surface into the interior of the jujube where they are fed until maturity, which will result in the lost of the commodity value of the jujube fruit completely.^[3] Moreover, the presence of a few jujube fruits with insect damage in a shipment may render the whole shipment unmarketable. So it is of particular interested for the producers or industries to detect such surface defect on insect-infested fruits and eliminate the problematic ones before shipment. Manual removal of insect-infested jujubes is tedious, time-consuming, and inefficient. Therefore, it is important to develop a system that can efficiently identify insect damage and assure fruit quality and safety, which would increase the marketability and enhance or maintain the shelf life of jujubes.^[4]

In the past 20 years, non-destructive methods of detecting the fruit and vegetable quality have received increasing attention, such as machine vision for shape classification and defect detection,^[5−7] x-ray imaging for inspection internal quality,^[8,^9] and near-infrared (NIR) spectroscopy for prediction internal quality of fruit.^[10−12] However, machine vision has been found to be unreliable because other kinds of surface damage may leave marks that are similar to holes.^[13] X-ray imaging is not practical for online detection.^[14] NIR spectroscopy can’t be used to collect spatial information of fruits quality as a point-source detection technique.

Hyperspectral imaging is a rapid, non-destructive technique which combines conventional spectroscopy and imaging techniques to acquire both spectral and spatial information from the fruits simultaneously.^[15] It has more advantages for evaluating fruit quality in an in-line system based on multispectral imaging where only the optimal wavelengths are inspected. Example applications of hyperspectral imaging include detecting bruises and other defects on apples,^[16−19] identifying citrus canker in grapefruit and resulting in 100% classification,^[20] discriminating internal infestation in Montmorency tart cherry,^[21−23] distinguishing insect-damaged wheat kernels,^[24] and detecting insect-damaged vegetable soybeans.^[1] Numerous research articles have been published about discriminating insect damage in many fruit cultivars. However, relatively few studies have been specifically targeted on external insect damage in the jujube. Therefore, the goal of this research is to combine image processing algorithm with spectral information to identify insect-infested jujubes from intact jujubes.

The objectives of this research were to: (1) build a NIR hyperspectral imaging system in the spectral range of 900–1700 nm and collect the hyperspectral image data, (2) determine the optimal wavelength for detecting of external insect infestations on jujube fruits, and (3) develop an image processing algorithm and discriminate insect-infested jujubes from intact jujubes.

MATERIALS AND METHODS

Jujube Samples

A total of 80 intact jujubes (Zizyphus jujuba Mill. cv. “Lingwu Changzao”) and 160 external insect-infested jujubes were collected from two different commercial orchards in Ningxia, China on October 8, 2012. The morphological properties of each sample were measured and recorded before spectroscopic measurements (Table 1). All jujube samples were stored at refrigerated temperature (4°) and used for the experiments within one week. Jujubes were removed from cold storage three hours before the experiments started in order to let them reach room temperature (22°). Representative images for intact jujubes and external insect-infested jujubes were shown in Fig. 1.

TABLE 1 Morphological properties of the jujube samples studied

	Diameter (mm)	Height (mm)	Transverse area (mm^2)
Range	18.82–26.88	28.12–55.10	414.99–1162.65
Mean value	21.95	43.32	746.44
Std. dev.	1.80	3.67	96.59

FIGURE 1 Typical surface conditions of jujube sample: (a) the intact jujubes, (b) the external insect-infested jujubes.

Hyperspectral Imaging System

A hyperspectral imaging system with a spatial resolution of approximately 5 nm was developed for acquiring reflectance images from the jujube samples, and its schematic diagram was shown in Fig. 2. The system consists of five major parts: a high-performance back-illuminated CCD camera (Zelos-285GV, Kappa optronics GmbH, Gleichen, Germany) with 1392 effective pixels; an imaging spectrograph (Imspector N17E, Spectral Imaging Ltd., Oulu, Finland) coupled with a standard C-mount zoom lens (V23-f/2.4, Specim Ltd., Finland) in the spectral region of 900–1700 nm; a sample transportation plate (PSA200-11-X, Zolix Instruments Co. Ltd., Beijing, China); four 35 Watt (W) tungsten halogen lamps as light source (HSIA-LS-TDIF, Zolix instruments Co., Ltd, Beijing, China); and a computer (Lenovo Inter (R) Core [email protected] GHz, RAM 4.00G). The spectral range of the hyperspectral camera is from 900 to 1700 nm with 3 nm spectral intervals, which results in 256 spectral bands.

FIGURE 2 Schematic diagram of the hyperspectral imaging system: (a) a CCD camera; (b) an imaging spectrograph with a standard C-mount zoom lens; (c) a sample transportation plate; (d) a tungsten halogen lighting unit; (e) a computer.

Hyperspectral Image Acquisition and Calibration

The hyperspectral imaging system is a push broom and line-scan based imaging system. The laboratory-based system was operated in a dark inspection chamber where the halogen lights were the only light source. The exposure time was adjusted to 10 ms and the speed of fruit movement was adjusted to 1.3 cm/s throughout the test. The object distance was set at 42 cm. The camera and spectrograph were used to scan the jujube fruit line-by-line as the translation plate moved the jujube through the field of view of the optical system. After finishing the scans on one entire jujube fruit, the spatial-by-spectral matrices were combined to construct a three-dimensional (3D, 320 pixels × 300 pixels × 256 bands) spatial and spectral data cube. The hyperspectral images were calibrated with a white and a dark reference images. The corrected image (R) was calculated using Eq. (1).^[25−27]

(1)

where, R_o is the recorded hyperspectral image, R_d the dark image (with 0% reflectance) recorded by turning off the lighting source with the lens of the camera completely closed, and R_r is the white reference image (Teflon white board with 99% reflectance). The corrected images will be the basis for the subsequent image analysis.

Data Processing and Analyzing

All data processing and analyzing were performed using ENVI 4.6 (Research System Inc., Boulder, CO, USA) and Matlab2008a (The MathWorks Inc., Natick, USA) with the image processing toolbox. Principal component analysis (PCA) on the hyperspectral images of jujubes was performed with ENVI 4.6. The PCA is a technology to reduce spectral dimensionality of the hyperspectral reflectance images, and to aid in visualizing the hyperspectral data. Then several optimal spectral wavelengths were chosen according to corresponding weighing coefficients of the PCA. The first principal components was adopted to eliminate the effects of bright spots and enhance the spectral differences among the bands. Finally, the PC-1 scores images by performing PCA on the selected optimal wavelengths were used for further image processing (masked image, image negation, thresholding, dilation, connectivity analysis) to discriminate external insect damage from jujube fruits.

Band ratio and image subtraction were evaluated to improve the detection rates of insect-infested jujubes by reducing the number of false positives. A band ratio algorithm could effectively enhance the contrast between different regions and produce more uniform responses across the jujube surface.^[28] The two-band ratio was calculated using Eq. (2):

(2)

where, Q_t/k represents a quotient of spectral reflectances, and R_t and R_k are reflectance intensities at t nm and k nm, respectively. Image subtraction was a process whereby the digital numeric value of one pixel or whole image was subtracted from another image. The difference between two images f (x,y) and h (x,y), expressed as:

(3)

Equation (3) is obtained by computing the difference between all pairs of corresponding pixels from f and h.^[29]

RESULTS AND DISCUSSION

Hyperspectral Reflectance Spectra

Hyperspectral imaging data contained both spatial and spectral information from jujubes, each pixel on a jujube fruit corresponded to one spectrum. The average spectra of insect-infested jujubes were obtained by choosing 3 × 3 pixels of each jujube around the insect-holes and outside of the insect-holes. The average spectra of intact jujubes were selected 3 × 3 pixels of each jujube, the corresponding area of insect-infested jujubes (Fig. 3). The average spectral curves of the insect-infested jujubes and intact jujubes showed a downward trend and the potential peaks and valleys appeared in the wavelength range between 900 and 1700 nm. The noise spectrum was obvious below 975 nm and above 1650 nm. Therefore, the representative regions of interest (ROIs) in the wavelength range of 975–1650 nm were used to perform PCA for further analysis. Moreover, the reflectance for insect-infested jujubes had a much lower value than that for the intact fruits, especially for the wavelength regions below 1400 nm. The jujube surface was damaged by insects, which provided with more chances for light to interfer with the cellular material. Therefore, lights would be more absorbed and scattered. As expected, the insect-infested jujubes had a higher absorbance than the intact fruits due to the decreased reflectance of the insect-infested jujubes surface.

FIGURE 3 Average spectral curves (900–1700 nm) for intact jujubes and insect-infested jujubes.

PCA in the NIR Region of the Spectrum

The first three PC scores images (denoted by PC-1 to PC-3) of jujube samples obtained from PCA using all 227 wavebands in the region from 975 to 1650 nm were shown in Fig. 4. In the PC-1 images, the differences of gray value were smaller between insect parts and the surrounding area, they could not provide more unique features than the original untransformed hyperspectral images. The PC-2 images indicated that the gray value showed significant differences between the insect part and the surrounding area. The insect-infested regions could be clearly identified. In the PC-3 images, the transformed image no longer possessed meaningful information, and they were not as useful as PC-2 for the detection of external insect damage from jujube fruits. Therefore, PC-2 images were chosen for the selection of optimal spectral wavelengths.

FIGURE 4 First three principal component images obtained using the spectral region from 975 to 1650 nm for insect-infested jujubes. PC-1 to PC-3 are the scores images of first, second, and third principal components, respectively.

Selection of Optimal Spectral Wavelengths

The weighing coefficients plot for the PC-2 images from PCA on the 975–1650 nm spectral region was shown in Fig. 5. In this article, the selection of optimal spectral wavelengths was based primarily on the following two principles: (1) the absolute values of weighing coefficients at wavelengths were in larger ones; (2) The wavelength should appear the potential peaks and valleys in the weighing coefficients curve.^[30] Six wavebands from 975 to 1650 nm were chosen, which were centered at around 987, 1028, 1160, 1231, 1285, and 1464 nm, respectively. Example images of jujube at each optimal wavelength were shown in Fig. 6.

FIGURE 5 Weighing coefficients for PC-2 images on a insect-infested jujube using the spectral region of 975–1650 nm.

FIGURE 6 Images of jujube at six optimal wavelengths.

PCA on Selected Optimal Wavelengths

The uneven intensity of light could be formed in different bands because the jujubes are oval fruits. Therefore, it was of great significance to adopt PCA method on selected optimal wavelengths to eliminate the influence of the high brightness area on image processing. PCA was carried out with only six optimal wavelengths. The PCA scores images of jujube were shown in Fig. 7. The contrast between the sound and damaged tissue appeared differently for each PCA scores image.

FIGURE 7 Six principal component images by PCA based on the six optimal wavelengths (987, 1028, 1160, 1231, 1285, and 1464 nm). PC-1 to PC-6 were the first, second, third, fourth, fifth, and sixth principal components, respectively.

As shown in Fig. 7, the insect-infested regions in PC-3 images were not as evident as those in PC-1 and PC-2 images due to the lack of contrast. The PC-4 to PC-6 might be inadequate for reliable detection of insect-infested jujube due to the noisy information. Based on the inspection and the comparisons above, the PC-1 images showed greater potential for indentifying insect-infested jujubes from the intact jujubes, compared to PC-2. Therefore, PC-1 images were chosen for performing the images processing.

The detailed procedures for detecting insect-infested jujubes were illustrated in Fig. 8. First, a single-band image at 1455 nm was used to build a mask template. Images at six selected wavelengths were masked analysis to remove the background that could impact results. Second, the PCA was performed on spectral imaging data at these six wavelengths. Then, the PC-1 scores images were chosen for determining the quality of a jujube.

FIGURE 8 Flow chart of the key steps involved in detection algorithm of insect-infested jujubes.

Two-Band Ratios Algorithm

Based on the spectral curves in Fig. 3, the two-band ratio using 987 and 1464 nm pair, 1160 and 1464 nm pair and 1285 and 1464 nm pair could be potential to detect insect-infested jujubes because the reflectance spectra of insect-infested tissue exhibited local minimum value at 1464 nm, and local maximum values at 987, 1160, and 1285 nm. The ratio images were shown in Fig. 9. By visual observation, the R1160/R1464 ratio images distinguished the insect-infested region clearly from the sound surface (white dot area in Fig. 9), where two strong absorption peaks at 1160 and 1464 nm may be associated with a combination of second overtone of band C-H and overtones of band O-H in H₂O, and first overtone of bond O-H, respectively.^[31] The ratio images with R987/R1464 and R1285/R1464 did not work the same effectively as R1160/R1464. A global threshold value of 0.7 was applied to the ratio image of R1160/R1464 to separate insect-infested jujubes from the intact jujubes. The flowchart based on band ratio algorithm was shown in Fig. 10 and the detection rate was given in Table 2.

TABLE 2 Test results for developed algorithms based on 240 independent samples with intact jujubes and insect-infested jujubes

		Misclassified			Accuracy (%)
Class	Number	PCA	PCA+BR	PCA+BR+IS	PCA	PCA+BR	PCA+BR+ IS
		2	0	0	97.5	100	100
Insect-infested jujubes	160	33	20	11	79.4	87.5	93.1
Total	240	35	20	11	85.4	91.7	95.4

FIGURE 9 Example two-band ratio images (R987/R1464, R1160/R1464, and R1285/R1464).

FIGURE 10 Flow chart of the key steps involved in two-band ratios algorithm and image subtraction algorithm of insect-infested jujubes.

Image Subtraction Algorithm

Image subtraction algorithm was used to identify small changes between images of insect-infested jujubes and intact jujubes. Like two-band ratios algorithm, the R987–R1464 and R1285 -- R1464 subtraction images were ineffective in producing images for distinguishing insect-infested jujubes. However, the R1160 -- R1464 subtraction images showed a clear insect-infested region (black area in Fig. 11), but jujube surface exhibiting whiter features. Two wavebands images at 1160 and 1464 nm were effective for detecting insect damage through many experiments. Therefore, the images at 1160 and 1464 nm were first used to produce subtraction image (R1160 -- R1464). The flowchart based on image subtraction algorithm was shown in Fig. 10.

FIGURE 11 Representative two-band ratio images (R987–R1464, R1160–R1464, and R1285 -- R1464).

Identification Results

The algorithms for multispectral image processing and classification described above were evaluated using 240 independent jujube samples. The test results for intact jujubes and insect-infested jujubes were shown in Table 2. A total of 240 jujubes were divided into two classes: intact jujubes including 80 samples, and insect-infested jujubes including 160 samples. All samples were evaluated using three detection methods: The first method was PCA method on selected optimal wavebands; the second method was PCA and two-band ratio (R1160/R1464); and the third method was PCA and two-band ratio (R1160/R1464) combined with image subtraction algorithm (R1160 -- R1464). As shown in Table 2, the overall detection accuracy for the tested samples of insect-infested jujubes was 79.4, 87.5, and 93.1%, and 97.5, 100, and 100% for the intact jujubes based on three methods, respectively.

CONCLUSIONS

The goal of this research was to investigate the feasibility of using hyperspectral imaging technique for detecting insect-infested jujubes in the wavelength range between 900 and 1700 nm. PCA was used to reduce high spectral dimension to a few optimal wavelengths centered at 987, 1028, 1160, 1231, 1285, and 1464 nm. PCA and two-band ratio (R1160/R1464) coupled with image subtraction algorithm (R1160 -- R1464) achieved the best 93.1% insect-infested jujubes identification accuracy. Although this study demonstrated that hyperspectral imaging technique is promising for detection of intact jujubes and insect-infested jujubes in practical applications, the selected optimal wavelengths at 987, 1028, 1160, 1231, 1285, and 1464 nm may not always the best to develop algorithm for all jujube fruit due to the limited sample size. Therefore, future research will focus on (1) increasing the sample size to make the algorithm more robust, (2) integrating PCA on selected optimal wavelengths and two-band ratio combined with image subtraction algorithm into a new algorithm for simplifying algorithms and increasing detection accuracy, and (3) developing an efficient multispectral imaging system for the on-line detection of insect-infested jujubes from intact jujubes.

ACKNOWLEDGMENT

The authors gratefully acknowledge the guidance from Professor He for this experiment.

FUNDING

The authors are grateful for the financial support from the National Natural Fund Committee (31060233), Key Project in the National Science & Technology Pillar Program during the Twelfth Five-year Plan Period (2012BAF07B06), and Technologies R & D Program of Ningxia Hui Autonomous Region in 2011.

Additional information

Funding

The authors are grateful for the financial support from the National Natural Fund Committee (31060233), Key Project in the National Science & Technology Pillar Program during the Twelfth Five-year Plan Period (2012BAF07B06), and Technologies R & D Program of Ningxia Hui Autonomous Region in 2011.