Rapid detection and visualization of slight bruise on apples using hyperspectral imaging

ABSTRACT

During harvesting and storage, slight bruises on apple surface caused by impact, compression, vibration, or abrasion are inevitable. To find an appropriate method to identify the bruised apples at five stages (1 min, 1 day, 2 days, 3 days and 4 days after bruising), 108 Fuji apples were collected as samples. Hyperspectral images of apples covering the wavelength between 400 and 1000 nm were acquired by the SOC710-VP hyperspectral imaging system. The standard normal variate (SNV) method was utilized for smoothing and denoising of the original hyperspectral data. Classification models, including Extreme Learning Machine (ELM), Partial Least Squares Linear Discriminant Analysis (PLS-DA) and Classification and Regression Tree (CART), coupled with a variable selection method named Minimum Redundancy Maximum Relevance (mRMR), were built to identify the bruised apples. The results showed that the ELM models exhibited the best classification capability, with the mean correct classification rate of 95.97%. The bruised samples are easier to be identified over time. Minimum noise fraction (MNF) method was implemented to classify the bruised region of apples based on the selected wavelengths. The overall classification accuracy of MNF is 92.9%, which indicates that MNF is an effective method for identifying bruised regions of apples.

KEYWORDS:

• Bruised apples
• hyperspectral images
• minimum redundancy maximum relevance
• extreme learning machine
• minimum noise fraction

Introduction

During harvesting and storage, slight bruises on apple surface caused by impact, compression, vibration, or abrasion are inevitable^[1], which are similar to the color and texture of healthy apples and hard to be recognized with the naked eyes for a moment. Over time, the bruised tissues become darkened, soften, and dry. Damages on apple surface not only affect the appearance and taste, but also may infect other normal fruit, causing great economic losses.^[2] At present, manual sorting is the main method of fruit quality detection. It is time-consuming and greatly influenced by the subjective factors. In the manual sorting system, bruise is the primary reason why apples are classified as inferior. Hence, it is necessary to develop fast, nondestructive and efficient bruising detection methods.

In the past two decades, a range of nondestructive techniques, such as electrical impedance^[3], X-rays^[4], thermal imaging^[5], have been applied for bruise detection. However, these methods acquire data from a small area on a sample, which may lead to incorrect recognition. Machine vision^[6] is a branch of artificial intelligence (AI) and developing rapidly. It has the advantages of accurate detection of appearance features of samples including size, color, shape, and texture. More and more scholars and experts work at the domain of machine vision, yet there are still some challenges in terms of the detection of minor injuries, which cannot be found easily. Hyperspectral imaging^[7] is an emerging technology in recent years, which has the ability of providing both spatial and spectral information. It is advanced and efficient to not only detect non-obvious bruise of samples but also assess internal attributes of fruits such as soluble solid content, firmness, pH value. ‘Golden Delicious’ and ‘Pink lady’ apples were used by Ferrari et al.^[8] to detected early bruise. The spectral range is 400–1000 nm, and the modeling results equal to 92.42% and 94.04%, respectively. The detection of the number of days after bruising of apples using hyperspectral images (400–2500 nm) and linear logistic regression neural network models was researched by Baranowski et al.^[9] After pretreated by the second derivative method, the prediction accuracies higher than 90% were achieved. ElMasry et al.^[10] used PLS and stepwise discrimination analysis to select characteristic wavelengths, then the damaged areas were segmented by the image processing method. In addition to apples, other agricultural products such as white mushroom^[11], sliced potato^[12] and strawberry^[13] are also tested using hyperspectral technology.

From the above-mentioned works, it is obvious that there was very little attention on apple bruise at different damage time. The major objective of this study is to find an appropriate method to identify the bruised apples at five stages (1 min, 1 day, 2 days, 3 days, and 4 days after bruising). The research was conducted through:

1. Acquire hyperspectral images of sound apples and bruised apples at different time after bruising by hyperspectral image system covering the wavelength between 400 and 1000 nm.

2. Select characteristic wavelengths applying Minimum Redundancy Maximum Relevance (mRMR) method.

3. Develop Extreme Learning Machine (ELM), Partial Least Squares Discriminant Analysis algorithm (PLS-DA) and Classification and Regression Tree (CART) models based on full wavelengths and characteristic wavelengths, respectively, to classify bruised samples and sound samples.

4. Visualize the bruised area of apples based on Minimum noise fraction (MNF).

Materials and methods

Apple samples

One hundred and eight Fuji apples with an equatorial diameter of approximately 9 cm and no obvious rottenness were purchased from a Fruit Wholesale Market in Wuxi China in July 2017. From the sample set, 54 samples were bruised in the middle area between stem and calyx artificially, and the rest were sound samples. Jiang et al.^[14] studied wavelength selection methods for detection of slight bruise on Korla pear based on hyperspectral imaging. Referring to their experiments, the apple was dropped from a height of 40 cm to a horizontal iron plate, and the slight bruise was made near the equatorial position of apple. The texture and color of the controlled early bruised areas were similar to those of the sound areas, which was difficult to recognize by naked eyes (Figure 1). After being bruised, all samples were numbered and stored until the experiment was conducted. The temperature of the air-conditioned laboratory was controlled at 22°C and the relative humidity was 75%. Hyperspectral images at 1 min, 1 day, 2 days, 3 days, and 4 days after damaging were acquired for simulating the bruise cases at different times.

Figure 1. Sound and bruised samples of apples

Hyperspectral image acquisition system

Figure 2 depicts the schematic of the hyperspectral imaging system and hyperspectral 3D spectral image. The hyperspectral imaging system composed by the following three parts: (1) Imaging unit. A SOC710-VP portable visible light/near-infrared hyperspectral imager, adopting holographic diffraction technology and owning both unique built-in scanning and single lens dual CCD design, was used to collect hyperspectral images of samples. (2) Light source unit. Four halogen lamps (150 W) were fixed on four inner sides of the box to provide enough light for the experimental samples. (3) Transport unit. The height of the platform can be adjusted manually to get clear hyperspectral images. Furthermore, a computer with data acquisition software HyperScanner2.0 was indispensable. To reduce the interference from outside, hyperspectral image acquisition experiments were carried out in a light-lock chamber.

Figure 2. Schematic of the hyperspectral imaging system and hyperspectral 3D spectral image

Hyperspectral images are stored in computers in the form of three-dimensional data ($x,y,\lambda$). Spatial location of a pixel is determined by (x,y), and λ represents the wavelength. From the perspective of images, hyperspectral image is composed of a set of gray-scale images. Each wavelength represents a 2D image. Based on the analysis of spectroscopy, every pixel is reflected as a continuous spectral response curve.

The hyperspectral imaging system collected spectral images of apples covering the wavelength between 400 and 1000 nm, and the spectral resolution is approximately 4.68 nm. Thus, each spectral image possesses 128 bands. A grayscale image at a given wavelength has 1392 × 1040 pixels. After adjustment, the exposure time of the camera and the scanning speed of the spectrograph were set at 25 ms and 30 lines per second, respectively.

Hyperspectral image calibration

For lowering the impact of dark current noise and inhomogeneity illumination intensity of the hyperspectral imaging system, relative reflectance of the original hyperspectral image was acquired using white and dark images based on Formula (1).

(1) $R(\lambda )=\frac{R_0(\lambda )-R_d}{R_w-R_d}\ast 100\mathrm{\%}$(1)

The R₀(λ) and R(λ) in Formula (1) were the original uncorrected hyperspectral data and the calibrated hyperspectral data at λ wavelength, respectively. R_d was the full dark image (with 0% reflectance) obtained with the camera lens covered. R_w was the monochrome image with a reflectance of 99%.

Data processing and analysis

Data extraction and preprocessing: After the hyperspectral images were corrected, we used ENVI 5.3 software to extract spectral data. The grayscale images at the wavelength of 811 nm were used for the data extraction, which showed the optimal contrast visually. For sound samples, a rectangle around 100 pixels at the equator of the apples was selected as an ROI.^[15] For bruised samples, the damage area is more than 100 pixels. Thus, a rectangle with an area of 100 pixels in the bruised region was selected as an ROI. For each band, the spectral reflectance of each pixel from ROI was extracted and averaged. Eventually, we got five datasets, named as Bruise_1min, Bruise_1day, Bruise_2days, Bruise_3days, and Bruise_4days. Every dataset contains 54 sound samples and 54 bruised samples. In addition to the essential information of apples, spectral noise is unavoidable, for instance, the effect of the instrument, operation, and the environment. Eliminating interference information is critical to improve the performance of the model. In this paper, standard normal variate (SNV) method^[16] was used as a pretreatment for correcting spectral errors triggered by solid particle size and surface astigmatism.

Characteristic wavelength selection by mRMR: In the above sections, the average spectrum of apples was extracted and analyzed. Hyperspectral image combines traditional image with spectrum, which can provide a lot of information. The hyperspectral image used in our experiments covers the wavelength between 400 and 1000 nm and possesses 128 bands. However, many of these wavelengths are highly correlated. In order to reduce the amount of calculation and improve the accuracy of classification, characteristic wavelength selection is indispensable. Many methods for characteristic wavelength selection have been proposed, such as PLS, stepwise discrimination analysis^[10], correlation-based feature selection (CFS), mRMR^[17], combined competitive adaptive reweighted sampling (CARS)^[18], successive projection algorithm (SPA)^[19], uninformative variable elimination (UVE).^[20]

Minimum Redundancy Maximum Relevance is a filtering feature selection method that takes mutual information (MI) to define the relevance between the features.^[21] Peng et al. have proved that mRMR is an effective feature selection method.^[22] Given the features x_i and the classification variable c, the mutual information of x_i and c is computed according to the Formula (2):

(2) $MI\left(x_i;c\right)=\sum _{ }^{ }\sum _{ }^{ }p(x_i,c)log\left(\frac{p(x_i,c)}{p(x_i)\ast p(c)}\right)$(2)

where p(x_i), p(c) are the marginal probabilities and p(x_i,c) is the joint probability distribution of x_i and c. However, the combination of features which provides large dependency on the target class does not increase the performance of the classifier. It is possible that the selected features are highly correlated, which leads to redundancy and affects the performance of the classifier.

The key idea of mRMR is to maximize the correlation between features and classification variables and minimize the correlation between features.^[23] Suppose that n-1 features have been selected and put into a set ${\mathrm{X}}_{\mathrm{n}-1}$, the n^th feature from the set $\left\{\mathrm{X}-{\mathrm{X}}_{\mathrm{n}-1}\right\}$ is defined as via MI:

(3) $\underset{x_i\in X-X_{n-1}}{max}\left(MI(x_i;c)-\frac{1}{n-1}\sum _{x_j\in X_{n-1}}^{ }MI(x_j;x_i)\right)$(3)

where MI(x_i;c) is the relevance of the individual feature x_i to the class c alone,$MI\left(x_j;x_i\right)$ is the redundancy between two features ${\mathrm{x}}_{\mathrm{j}}$ and x_i.

Classification algorithm for sound and damaged apples: In this paper, ELM, PLS-DA, and CART were tested to determine which classification method provides the best performance. ELM^[24,25] is a feedforward neural network with a single hidden layer. In the learning process, there is no need to adjust hidden layer nodes. PLS-DA^[26] is a multivariate statistical analysis method, and it can reduce the influence of multicollinearity among variables. CART is a binary decision tree, which produces only two sub-nodes after each feature segmentation operation. The performance of the model was evaluated with accuracy.

MNF for discrimination between bruised and sound tissue: MNF^[27] is a statistical analysis method designed to de-correlate and segregate any apparent noise in the noise-contained data based on noise covariance estimation. The principal components after MNF are sorted in descending order according to the signal-to-noise ratio, which overcomes the influence of noise on image quality. Baranowski et al.^[5] suggested MNF transform can get better segmentation effect and its components are suitable for image segmentation.

Results and discussion

Overview of spectra and images

The average spectral reflectance curves of sound apples and bruised apples at five stages (10 min, 1day, 2days, 3days and 4days after bruising) are shown in the left part of Figure 3. It is noted that the trend of spectral reflectance curves of damaged samples is similar to that of sound samples. In general, the longer the sample is damaged, the lower the spectral reflectance is. In this spectral region (680–740 nm) the change of spectral reflectance is obvious, which is due to the red effect of plants. There is an absorption peak at 680 nm, this might be ascribed to the absorption of chlorophyll on the surface of the apple. 970 nm is a typical water absorption band, which is caused by the water content of the apple samples.^[28]

Figure 3. Average spectra reflectance and grayscale images at 811 nm of sound and bruised apples

The right part in Figure 3 shows the grayscale images at 811nm of sound and bruised apples. The grayscale images at 811nm provided clear visual contrast between apple and background, because the difference between the reflectance of apples and that of background is the largest. In these two images, it can be found that the color of the bruise region is darker slightly than the sound region. Therefore, it is feasible to segment the damaged and sound areas of apples with a few image processing methods.

Optimal wavelength selected by the mRMR algorithm

Each pixel in a hyperspectral image has 128 values, representing reflectance at different wavelengths, which provides a wealth of information for the classification of the damaged apples. Nevertheless, the spectral wavelengths are relatively continuous and there is a remarkable resemblance between adjacent wavelengths. The duplicate data will not only cause redundancy but also affect the timeliness and accuracy of the multivariate analysis.

After preprocessing, we made use of the mRMR algorithm to obtain optimal wavelengths. mRMR selects those features with the highest relevance to the target class, and at the same time, they are also maximally dissimilar to the rest of the classes.^[21] Cen et al.^[23] detected internal defects of pickling cucumbers based mRMR waveband selection and the classification performance is well. In practice, ELM with strong generalization ability was used to build the classification model to determine the optimal wavelength subset. Figure 4 indicates the changing trend of the classification performance of the test set with the increase of selected bands. It can be seen from Figure 4 that, with the increase of the number of selected bands, there is a sharp rise in classification accuracy at first, then declines moderately, and finally increases slowly and tends to be stable. When the band number is 11, the classification accuracy of the test set is 97.78%. Although the classification accuracy is higher when the number of bands is 74, more calculations are needed. The selected wavelengths by mRMR are: 421.9, 481.8, 491.8, 603.3, 634.1, 639.2, 644.4, 649.5 659.9, 696.1, and 945.2 nm.

Figure 4. Classification performance vs. number of bands selected by mRMR

Classification by ELM, PLS-DA, and CART algorithm

We first divided the 108 samples into the training set and test set via Kennard-Stone (KS) method in a proportion of 2 to 1. Then the ELM, PLS-DA, and CART were applied to establish models to distinguish between the damaged apples and the sound apples. In practice, there are 20 neurons in the hidden layer, and we used the Sigmoid function as activation function. Besides, we used the genetic algorithm (GA) to optimize parameters to get better classification results. The number of latent variables (LVs) in PLS-DA models was determined by Monte Carlo Cross-validation (MCCV) method.

The classification results on full wavelengths are given in Table 1. ELM can get higher accuracy, and the mean correctly identified accuracy of the test set is 95.97%. Other two classifiers have slightly lower performance and give more than 90% of the overall accuracy. We note that even in the early stage of damage, the ELM model can achieve better classification accuracy. In addition, there is little difference in classification accuracy between different data sets when ELM was used as a modeling method. Table 2 shows the classification accuracy on the optimum wavelengths by ELM, PLS-DA, and CART, and the mean accuracy of the test set is 95.94%, 93.33%, and 81.67%, respectively. Comparing with the results in Table 1, the mean accuracy of the ELM model and PLS-DA model fluctuated slightly. For Beuise_1min dataset, the accuracy of ELM increased from 94.86% to 95.69 while that of PLS-DA model is from 83.33% to 86.11%, raised by about 3%. CART models using full wavelengths showed higher precision than using optimum wavelengths. The average recognition accuracy of CART, modeling with optimum wavelengths, is 81.67%. It proves that CART is not good at dealing with bruised apple identification. Among the models based on five datasets, the ELM models exhibited the best classification capability. Mean accuracy of training set and test set of ELM models are 97.82% and 95.94%, respectively.

Table 1. Classification of samples based on ELM, PLS-DA and CART models using full wavelengths

	ELM		PLS-DA		CART
Model dataset	Training set	Test set	Training set	Test set	Training set	Test set
Bruise_1min	97.50	94.86	98.61	83.33	90.28	83.33
Bruise_1day	97.71	95.28	95.83	91.67	100	94.44
Bruise_2days	97.85	97.08	93.06	100	98.61	94.44
Bruise_3days	98.12	96.53	94.44	88.90	100	94.44
Bruise_4days	97.78	96.61	98.61	97.22	98.61	97.22
Mean	97.80	95.97	96.11	92.22	97.50	92.77

Table 2. Classification of samples based on ELM, PLS-DA and CART models using optimum wavelengths

	ELM		PLS-DA		CART
Model dataset	Training set	Test set	Training set	Test set	Training set	Test set
Bruise_1min	97.92	95.69	87.50	86.11	90.28	80.56
Bruise_1day	97.85	95.97	95.83	97.67	94.44	75.00
Bruise_2days	97.92	96.25	97.22	100	91.67	86.11
Bruise_3days	97.64	95.42	97.22	91.67	95.83	80.56
Bruise_4days	97.78	96.39	95.83	97.72	91.67	86.11
Mean	97.82	95.94	94.92	93.33	92.78	81.67

From both Tables 1 and 2 we can find that the majority of models did not perform very well for classifying the samples just bruised. The results are in agreement with the conclusions of many researchers.^[29] The chemical composition of apple with early bruise showed a slight change, in the meanwhile, the spectral reflectance had no noticeable change. With time going by, the content of reducing vitamin C, total sugar and sugar steadily decreased, which contributed to the change on spectral reflectance.^[29] Hence, the bruised samples are easier to be identified over time.

Visualization of bruised area based on MNF

Figure 5 depicts the key steps for the whole experimental procedure. First, the hyperspectral images were collected by hyperspectral imaging system and calibrated using white and dark images based on Formula (1). Then we extracted the spectral data from ROIs. Nevertheless, the spectral wavelengths are relatively continuous and there is a remarkable resemblance between adjacent wavelengths. Therefore, the mRMR method was used to obtain optimal wavelengths, which contained most of the information provided by all wavelengths. Eleven wavelengths were selected, and the modeling results showed that they were efficient for apple bruise discrimination. The next operation was to apply the MNF method to classify the bruised region of apples based on the optimal wavelengths. The grayscale images at 811nm, which gave a clear visual contrast between apple and background, can be used to build a mask to eliminate the information of background. After that, a total of 11 characteristic images without background information were analyzed by the MNF method.

Figure 5. Flow diagram of the experimental procedure

Figure 6 shows the first three principal component images (denoted by MNF1 to MNF3) of the bruised sample after MNF transformation and the bruised area extracted from MNF3. MNF1 images of Sample 1 and Sample 2 clearly show the outline of the sample. And we note that the color of the middle area of the sample is different from that of the surrounding area, which is caused by the illumination inhomogeneity. Therefore, MNF1 images mainly described the outline information and uneven illumination distribution. MNF2 illustrated the appearance features, such as spots and textures. Compared with the sound region, the grayscale values of the damaged region are lower. However, there are a lot of grayscale values similar to it. Thus, it is difficult to segment the damaged area using MNF2. From MNF3 images can clearly distinguish the bruised tissues from sound tissues, which were used to segment bruised region.^[30] Then, we applied OSTU to MNF3 images, the bruised area was extracted. Overall, the classification accuracy of MNF is 92.9%, which indicates MNF is an efficient method to identify the bruised regions of apples. Zhang et al.^[31] have studied the early detection rottenness in the apples by MNF, and the identification accuracy is 97.7%.

Figure 6. The first three principal component images (MNF1 to MNF3) gained using MNF

Conclusion

In this study, we find an appropriate method to identify the bruised apples at five stages (1 min, 1 day, 2 days, 3 days and 4 days after bruising). One hundred and eight Fuji apples were collected as samples. Hyperspectral images of both sound and damaged apples covering the wavelength between 400 and 1000 nm were acquired by the SOC710-VP hyperspectral imaging system. SNV was utilized for smoothing and denoising of the original hyperspectral data. For reducing data redundancy and computation overhead, mRMR was applied to selected characteristic wavelengths. After that, three different algorithms, ELM, PLS-DA, and CART, were tested to determine which classification method achieves the best performance. Among the models based on five datasets, the ELM models exhibited the best classification capability with the accuracy of 95.97% on the full wavelengths and 95.94% on the optimal wavelengths, respectively. Our experimental results are in agreement with many researchers’ viewpoints. It is that with the prolongation of damaging time, the classification accuracy increases gradually. To visualize the bruised area of apples, MNF and other traditional image processing methods were used. We found that MNF3 images can clearly distinguish the damaged area from the healthy area, offering high visual contrast. The overall classification accuracy of MNF is 92.9%, which indicates that MNF is an effective method for identifying bruised regions of apples. Overall, it is feasible to classify slight bruise on apples using hyperspectral images. In the research, we learned that the degree of apple bruise is an important indicator of apple quality evaluation. We will consider the degree of bruise as the research content in the follow-up experiments.

Disclosure statement

The authors declare no conflict of interest.

Additional information

Funding

This study was supported by the Key Project of the Jiangsu Provincial Research and Development [No. BE2016627]; and the Fundamental Research Funds for the Central Universities [No. RP51635B].