Flower segmentation with level sets evolution controlled by colour, texture and shape features

Abstract

This work proposes a pre-informed Chan vese based level sets algorithm. Pre information includes objects colour, texture and shape fused features. The aim is to use this algorithm to segment flower images and extract meaningful features that will help is classification of floral content. Shape pre-information modelling is handled manually using advance image processing tools. Local binary patterns features makeup texture pre-information and Red, Green and Blue colour channels of the object provide colour pre-information. All pre-defined object information is fused together to for high dimension subspace defining object characteristics. Testing of the algorithm on flower images datasets show a jump in information content in the resulting segmentation output compared to other models in the category. Segmentation of flowers is important for recognition, classification and quality assessment to ever increasing volumes in floral markets.

Keywords:

• flower image segmentation
• texture priors
• local binary patterns
• shape priors
• level sets

View correction statement:

Erratum

Public Interest Statement

Flower segmentation and classification is a challenging task for computer vision engineers. A tool is required that can capture the full scale of parameters for classification. This work is the first step towards the development of the iFlower tool for recognition of floral species. This mobile based application can greatly help people understand the natural environment and contribute their part in making it nourish and prevent them from going extinct. This work focused on a segmentation model for flowers in natural environment using their color, texture and shape information. Shape is extracted manually in this work. In future a fully automated process is being developed by our team.

1. Introduction

Flowers induce instantaneous and elongated effects on emotions, mood, behaviours and memory of both male and female human beings (Haviland-Jones, Rosario, Wilson, & McGuire, 2005). The authors studied extensively about the reactions flowers cause during their contact with humans in three different ways and concluded that human happiness is directly linked to flowers. This is the reason for a 30% increase in world floriculture market every year and a 25% in India per annum (Oommen, 2015). The other side of the story is the losses incurred as they don’t last long after they are cut form the plant. The storage, temperature, sorting, packaging and transportation are some of the causes for a market loss of nearly 25% every year (van Meeteren, 2009).

Computer vision based algorithms are capable of determining the quality of flower during its journey from blossoming to final consumer market. In this work we limit ourselves to the first stage of development of a complete floral quality tester using computer vision models. The first and most complicated task is to extract the flower to lower dimensional subspace for classification. The binary segmentation of the flower is performed by using a higher dimensional feature subspace consisting of colour, texture and shape characteristics of the image objects. The proposed method is evaluated on flower database available at oxford http://www.robots.ox.ac.uk/~vgg/data/flowers/.

The world consists of close to 250,000 species of flowers. Classification of these species is largely at the discretion of the botanists. Even the people involved in floral trade are unable to classify them correctly. An image is enough to classify the floral content with the help a guide book and an expert botanist. People still find it difficult to know a particular flower species when capture a picture of it using a digital camera. If they have a name of the flower it is easy to find information about the flower species using Google search engine. But the link between the photographed flower picture and the name of the flower is missing. Hence, this paper investigates the first step in the process of automatic classification of flora for images of flowers captured by digital cameras.

Vision computing applications is growing at an enormous pace in the last decade and agriculture (Gomes & Leta, 2012) is no exception. Pest detection, grading (Effendi, Ramli, Ghani, & Yaakob, 2009), lesion estimation (Niphadkar, Burks, Qin, & Ritenour, 2013), yield prediction (Dana & Ivo, 2008) and flower quality estimation (Benmehaia, Khedidja, & Bentchikou, 2016) leading to good harvesting are the major areas (Sharma, Sharma, & Srivastava, 2016). For floral image processing, quite a few research is published in the past decade. The problem is being approached from two different perspectives. They are knowledge based and data based approaches. Data based approaches use the information in the digital picture such as color, shape, features and resolution as the parameters for segmentation. Knowledge based models use some form pre information about the object being segmented.

In (Nilsback & Zisserman, 2007), the authors use an image specific color distribution which detects the shape of the flowers in the image automatically. The algorithm starts with color descriptor and transforms into a foreground/background segmentation initializing a generic shape model. The generic shape model is applicable across multiple classes and viewpoints. Previously Das, Manmatha, and Riseman (1999) used color of the flowers as domain knowledge for segmentation. The algorithm also learns image background model from the periphery of the image. This model works well when the flower in the image is different from background by over 50%.

But the oxford database which is being used in our work does not follow this rule and the creators of the database propose the algorithm in Nilsback and Zisserman (2006). Here the authors develop a visual vocabulary of flowers based on color, texture and shape (CTS) information. This model overcomes the ambiguities that arise during flower classifications due to ambient lighting variations, petals shape deformations, color changes and occlusion during image capture. Our work in this paper is also focused on pre knowledge of shape, color and texture of the flower.

The methods in literature for flower segmentation project the segmented flower on to color plane. This way it becomes difficult for classification using an automatic classifier. In this work we make a flower feature database that can be used for classification using Hidden Markov Model (HMM’s) or Artificial Neural Networks (ANN’s). The oxford data-set consists of 17 different flower species. Flower classification is a challenging task even for human eyes. This can be observed from Figure 1, where the flower images from the data-set in http://www.robots.ox.ac.uk/~vgg/data/flowers/ are stacked together for reference.

Figure 1. Sample data-set.

Source: http://www.robots.ox.ac.uk/~vgg/data/flowers/.

The scale of difficulty in extracting useful segments for classification from these randomly captured images is an uphill task for computer vision scientists. Here we propose to use CTS of the flower as pre information for the segmentation cost function. Literature used all these parameters as features for segmentation. But the disadvantage lies with the algorithm which fail to extract fine lines on the surface of the flowers as shown in Figure 2.

Figure 2. Detail information that is missing in the flower segmentation algorithms.

Shape only or texture only classifications of flowers produce ambiguous results. Figure 3 shows some of the shape prior, texture priors form gray level covariance matrix (GLCM) (Kishore & Rajesh Kumar, 2012) and Gabor filter in 4 different orientations (Kishore, Prasad, & Kishore, 2013). The texture models produce excellent texture segmentation outputs for most of the image processing problems. But from the Figure 3 it becomes difficult to produce the required information shown in Figure 2. The shape model gives only the global region of flowers which alone cannot handle the segmentation process.

Figure 3. Shape, GLCM and gabor texture features of a flower.

The features from GLCM and Gabor textures show information loss that is vital for flower classification. In this work we introduce a mixed feature as pre information for the level set function. The mixed feature is made up of shape, texture and color. For color Red, Green and Blue (RGB) planes are featured. Shapes are hand modelled from the original images of flowers. For texture we use Local Binary Patterns (LBP) (Ahonen, Hadid, & Pietikainen 2006; Guo, Zhang, & Zhang, 2010; Zhao & Pietikainen, 2007) features instead of GLCM or Gabor features. The mixed feature image of a flower from the data-set is shown in Figure 4.

Figure 4. Mixed feature using CTS images.

The pre information related to flower in CTS form is provided as constraints in the formulation of Level sets. Level sets were first introduced by Osher and Fedkiw (2006) and later popularized by Vese and Chan (2002). A number of versions of level sets with shape (Saito, Nawano, & Shimizu, 2016) and texture priors (Wu, Gan, Lin, Zhang, & Chang, 2015) were very popular with the image processing research community. A combination of CTS features are used exclusively by computer vision researchers for complex image segmentation (Cremers, Osher, & Soatto, 2006; Cremers, Rousson, & Deriche 2007; Hu et al., 2013).

All the previous level set models used the features separately during level set formulation. Here we propose to fuse all the pre information to the level set and provide them as a single constraint instead of three separate functions. Section 2 introduces the proposed model. Section 3 discuss the results and Section 4 concludes the work.

The flow chart describes the methodology used for segmenting flower images is shown in Figure 5.

Figure 5. Flow chart describing the proposed flower segmentation model.

2. Multi feature level set formulation

CTS features form a knowledge base for the level set function to operate on the image plane. Previous algorithms used three different functions for level set formulation from these features. In this work we propose to use a single term for all these features to be incorporated in a level set. A brief review about the level sets and feature extraction models make up this section.

2.1. Color feature

Previous works extracted RGB vectors from the image plane to construct a size (RGB) × 3 vector feature subspace. Other methods involve converting RGB space to Lab or HIS or HSV color spaces and extracting them as color features. The novel model proposed in this work saves computing power initially by avoiding this step. The idea is to use each of the R, G and B planes separately during contour propagation. The level set is formulated on these 3 sub planes, which will be elaborated in the level sets section.

2.2. Texture features

Nature creates different varieties of flowers based on colors and textures. Quite a few models in texture extraction were ideated for flower segmentation. They are Gray Covariance Matrix (GLCM), Gabor Filters, Wavelet Filters and LBP. Results of our analysis in Figures 3 and 4 show that LBP features in RGB plane provide us with good texture of the flower compared to the other three.

LBP compares each pixel in a pre-defined neighbourhood to summarize the local structure of the image. For an image pixel I(x, y) ∊ ℜ⁺, where (x, y) gives the pixel position in the intensity image. The RGB image is I(x, y, N) ∊ ℜ⁺, where N represents RGB color planes. The neighbourhoods of a pixel can vary from 3 pixels with radius r = 1 or a neighbourhood of 12 pixels with r = 2.5. The value of pixels using LBP code for a centre pixel $\left(x_{\text{c}},y_{\text{c}},N\right)$ is given by(1) $$\text{LBP}\left(x_{\text{c}},y_{\text{c}},N\right)=\sum _{i=1}^N\sum _{j=1}^{P}s\left(g_{\text{p}}-g_{\text{c}}\right)2^P$$(1) (2) $$s(x)=\left\{\begin{array}{cc}1\hfill & \forall x\ge 0\hfill \\ 0\hfill & \text{Otherwise}\hfill \end{array}\right.$$(2)

where g_c is intensity gray value of centre pixel at $\left(x_{\text{c}},y_{\text{c}}\right)$ and g_p is gray value around the neighbourhood of g_c. The value of N represents color plane and P gives the number pixels in the neighbourhood of g_c. Figure 6 gives the LBP calculations on the flower image in Figure 4.

Figure 6. (a) Original flower image, (b) LBP process for 3 × 3 window and (c) the resultant LBP resulting in a color texture image.

The LBP texture features are invariant to monotonic gray level changes that are predominant in flower images. Computational simplicity makes the LBP ideal for this work as the next stage involves more computationally intensive algorithm in the form of level sets. This color texture flower data forms the pre-information to the level set function.

2.3. Shape features

Shape features are deciding factor for object classification. Shape segmentation is the most challenging concept with respect to flower images. Pre-information on shapes is obtained using hand segmented flower images. From the data-set of flower images, external shapes of flowers are segmented resulting in binary shape filters as shown in Figure 3.

2.4. Level set formulation

The level set model introduced by Caselles, Kimmel, and Sapiro (1997) and few others is an image segmentation model based on a closed contour spreading in the image plane adhering to object edges. An implicitly defined contour of arbitrary shape Θ in the image plane $\text{O}:\{o(x,y)\subset {\mathbb{R}}^2\}$ as the zero level set of an embedding function $\mathit{\varphi }:\text{O}\to \mathbb{R}:$(3) $$\mathrm{\Theta }=\left\{x,y\in \text{O}|\mathit{\varphi }\left(x,y\right)=0\right\}$$(3)

Level set function ϕ evolves instead of contour Θ itself giving advantages in the form of no marker control and immune to topological deformations of the active contour in the image plane. Here we focus on Chan Vese (CV) (Vese & Chan, 2002) level set functional formulated from Chan and Vese (2001) in the image plane $I:\mathrm{\Theta }\to {\mathbb{R}}^{+}$as(4) $$E^{\text{Cv}}\left(\mathit{\varphi }\right)=\underset{\mathrm{\Theta }}{\int }{\left(I(x)-C^{+}\right)}^{2}H\left(\mathit{\varphi }\left(x\right)\right)\text{d}x+\underset{\mathrm{\Theta }}{\int }{\left(I(x)-C^{-}\right)}^2\left(1-H\left(\mathit{\varphi }\left(x\right)\right)\right)\text{d}x+\mathit{\lambda }\underset{\mathrm{\Theta }}{\int }\left|\mathrm{\nabla }H\mathit{\varphi }\left(x\right)\right|\text{d}x$$(4)

$I(x,y)$ is a 2D image plane represented as I(x). Here $H\left(\mathit{\varphi }\left(x\right)\right)$ is a Heaviside step function. C⁺ represents average intensity of pixels considered constant in positive ϕ region and C⁻ represents negative ϕ region constant. The last term in the Equation (4) tries to keep a smoothing contour during evolution and $\mathit{\lambda }$ is proportionality constant deciding on the minimum amount of separation needed between boundaries. The first two parts constitute external energy representing error between the image and piecewise constant approximations of the evolving level set functional. Gradient descent minimization of the level set ϕ gives a curve evolution expression(5) $$\frac{\mathit{\partial }\mathit{\varphi }}{\mathit{\partial }t}=-\frac{\mathit{\partial }{E}^{\text{Cv}}}{\mathit{\partial }\mathit{\varphi }}=\mathit{\delta }\left(\mathit{\varphi }\right)\left[\mathit{\lambda }\left(\mathrm{\nabla }.\left(\frac{\mathrm{\nabla }\mathit{\varphi }}{\left|\mathrm{\nabla }\mathit{\varphi }\right|}\right)\right)-{\left(I(x)-C^{+}\right)}^2\right.\left.+{\left(I(x)-C^{-}\right)}^2\right]$$(5)

CV model proposed the $\mathit{\delta }\left(\mathit{\varphi }\right)$ term which helps level set detect even the internal edges. Applying this level set model in (5) to segment flower images in various unformatted images results in segmentation outputs shown in Figure 7.

Figure 7. (a) Original flower imagem (b) level set evolution and (c) segmented flowers using CV model.

The outputs in Figure 7 shows incomplete flower segments after an average of 1000 iterations. A shape and texture prior model for level set as a learning basis will focus on segmenting the flower in RGB sub planes which is useful in post processing recognition. Cremers et al. (2006) introduced the signed distance function for shape encoded level sets which is modified in this work incorporating texture information. To establish a unique relationship between its surrounding level set ϕ and a pre-defined shape, texture, color (STC) model φ^STC, it will be assumed that $\mathit{\varphi }<0,\text{inside}{\mathit{\phi }}^{\text{STC}}$, $\mathit{\varphi }>0,\text{outside}{\mathit{\phi }}^{\text{STC}}$ and $\left|\mathit{\varphi }\right|=1$ everywhere else. The model φ^STC is defined as a combination of shape and texture models taken across RGB color planes. The color model in this case is user dependent and can be either HSV or Lab color space. The combination level set function is(6) $${\mathit{\phi }}^{\text{STC}}=\sum _{N=3}{\mathit{\phi }}^S+{\mathit{\phi }}^T$$(6)

There are many ways to define this signed distance function (Kraft, 2016; Laadhari, Saramito, & Misbah, 2016; Sussman, Smereka, & Osher, 1994), out of which we use the most widely applied with constrains towards scaling, rotation and translational properties. In this work, we propose to use initial contour ϕ and shape prior φ^STC contour to compute level set area difference as in Cremers et al. (2006):(7) $$d^2\left(\mathit{\varphi },{\mathit{\phi }}^{\text{STC}}\right)=\sum _{i=1}^N\underset{\mathrm{\Theta }}{\int }{\left(H\left(\mathit{\varphi }\left(x\right)\right)-H\left({\mathit{\phi }}^{\text{STC}}\left(x\right)\right)\right)}^2\text{d}x$$(7)

where N is the number of color sub planes the image is defined. The defined distance function is image size independent, nonnegative, symmetrical and satisfies triangle inequality. Local energy minimization between $\left({\mathit{\varphi }}_0,{\mathit{\phi }}^{\text{STC}}\right)$ maximizes the possibility of finding correct shape in the cluttered backgrounds. The affine transformations are defined by current STC ϕ₀. The curve evolution expression is obtained by applying Euler–Lagrange equation on (7) as(8) $$\frac{\mathit{\partial }{\mathit{\varphi }}_0}{\mathit{\partial }t}=\sum _{i=1}^{N}2\mathit{\delta }\left({\mathit{\varphi }}_0\right)\times \left(H\left({\mathit{\phi }}^{\text{STC}}\right)-H\left({\mathit{\varphi }}_0\right)\right)$$(8)

where $\mathit{\delta }\left(\bullet \right)$ is delta function and t is artificial time step. Finally combining STC prior energy term in (7) and CV level set function in (2), we get the total energy function of the level set as(9) $$E^{\text{T}}=\sum _{i=1}^N\mathit{\zeta }{E}^{\text{C}}+(1-\mathit{\zeta })E^{\text{STC}}$$(9)

Here ζ controls the effect of STC prior energy on the image energy. For single shape priors the energy functional used for algorithm development is derived from evolution Equations in (5) and (8) is(10) $$\frac{\mathit{\partial }\mathit{\varphi }}{\mathit{\partial }t}=\sum _{i=1}^N\mathit{\zeta }\mathit{\delta }\left(\mathit{\varphi }\right)\left[\mathit{\lambda }\left(\mathrm{\nabla }.\left(\frac{\mathrm{\nabla }\mathit{\varphi }}{\left|\mathrm{\nabla }\mathit{\varphi }\right|}\right)\right)-{\left(I(x)-C^{+}\right)}^2\right.\left.+{\left(I(x)-C^{-}\right)}^2\right]+2\left(1-\mathit{\zeta }\right)\times \left(H\left({\mathit{\phi }}^{\text{STC}}\right)-H\left(\mathit{\varphi }\right)\right)$$(10)

where C⁺ and C⁻ are updated iteratively in each discrete time step using the expressions(11) $$C^{+}=\sum _{i=1}^N\frac{\underset{\mathrm{\Theta }}{\int }I\left(H\left(\mathit{\varphi }\right)\right)\text{d}x}{\underset{\mathrm{\Theta }}{\int }\left(H\left(\mathit{\varphi }\right)\right)\text{d}x}$$(11) (12) $$C^{-}=\sum _{i=1}^N\frac{\underset{\mathrm{\Theta }}{\int }I\times \left(1-H\left(\mathit{\varphi }\right)\right)\text{d}x}{\underset{\mathrm{\Theta }}{\int }\left(1-H\left(\mathit{\varphi }\right)\right)\text{d}x}$$(12)

The following proposed level set model is tested for different flower images in the data-set (http://www.robots.ox.ac.uk/~vgg/data/flowers/).

3. Results and discussion

The following parameters of the proposed level set functional needs adjustments during the simulation trials. The STC effect term ζ is set based on the complexity in the image planes. This term controls the contour movement distributions on the sub planes related to shape and texture. The ζ is around 0.634 for images captured under lesser ambient lighting or with dominating backgrounds compared to flower foregrounds. For high quality images this value is kept very low around 0.12. In this work the value of ζ ranges from 0.1 to 0.7.

The level set stopping criteria is set using the gradient descent algorithm. Contour evolution stops when the error between the current iteration and previous iteration reaches a pre-defined threshold. The error value is set 0.0001. The performance of the proposed method is estimated using structural similarity index (SSIM) (Kishore, Sastry, & Rahman, 2016) against the other methods such as Gabor texture prior (GTP) (Shah & Chauhan, 2016) and gray covariance matrix texture priors (GLCMTP) (Lloyd, Marshall, Moore, & Rosin, 2016).

The segmentation results for various flower images in the data-set are presented in Figure 8 along with their shape and texture priors. Local binary pattern based texture is used in all the simulations.

Figure 8. Flower segmentation process.

Notes: FI: flower image, TCFI: texture color FI, SFI: shape flower image, IC: initial contour, 29 itr: 29th iteration captured, 81 itr: 81th iteration captured, 181 itr: 181th iteration captured, FS: segmented flower.

Figure 8 shows initial contour in yellow and final contour in green. Both the contours are projected on to the image under IC heading. The final iteration count for each flower image is different, even though the images collected in Figure 8 are shown for common iteration count. The last column shown the binary segmented flower image that can be used for classification process. Visual analysis of the segmented flower indicates the quality of the proposed method. A very clear picture is obtained about the flower showing internal texture and its shape. The internal stem of the flower in Anthurium is fully extracted under the cluttering influence of the flower texture. This near perfect segmentation can achieve good classification rates. Figure 9 shows more test results on flower images from the data-set in http://www.robots.ox.ac.uk/~vgg/data/flowers/.

Figure 9. Segmentation results of the proposed algorithm on flowers of different sizes, shapes, textures and colors.

Figure 9 gives the performance of the algorithm when tested on images of different color, shape and texture. The resulting flower images can be presented to a classifier for classification. For qualitative performance measurement we use SSIM as a measure between the segmented output and the hand segmented ground truth. The ground truth is extracted using expert craftsman in Photoshop.

Superiority of the proposed Local binary pattern Texture (LBPT) segmentation is established by comparing its performance with GTP (Shah & Chauhan, 2016) and GLCMTP (Lloyd et al., 2016). The performance indicators are SSIM and number of iterations. SSIM measures the local and global similarity between pixels between images. We use reference image as the hand segmented image and the test image is represented by segmented flower. The local similarity 3D map for a flower image in the last row of Figure 8 is shown in Figure 10.

Figure 10. Showing local SSIM using a 3D mesh plot.

The clarity of segmentation result for the last row is questionable and its closeness to actual segmentation can be seen from the plot of Figure 10. There is a small variation at the centre of the flower as the stem in the flower is masked by the flower itself. Except for that one centre region, the other portions of the flower exactly match to that of the hand segmented flower image.

Table 1 gives global SSIM values for flowers in the database between LBPT, GTP and GLCMTP. SSIM values are in the range of [0, 1]. Is SSIM score is ‘0’ indicates no match and ‘1’ indicates a perfect match.

Table 1. Performance indicators for the flower segmentation algorithms using level sets

Flower name	No of images/flower	LBPT − LS		GTP − LS		GLCMTP − LS
		Average SSIM	Average iteration	Average SSIM	Average iteration	Average SSIM	Average iteration
Anthurium	10	0.9235	181	0.7856	220	0.7863	222
Barbeton daisy	10	0.9189	186	0.7765	236	0.7789	226
Bishop of llandaff	10	0.9356	192	0.7535	242	0.7513	232
Californian Poppy	10	0.9525	188	0.8022	224	0.8008	234
Frangipani	10	0.9203	184	0.7869	229	0.7898	219
Hibiscus	10	0.8725	177	0.7425	211	0.7489	221
Lotus	10	0.9425	187	0.7958	236	0.7911	226
Rose	10	0.8526	171	0.7369	209	0.7307	219
Thorn apple	10	0.8925	179	0.7612	219	0.7602	229
Tree mallow	10	0.8659	173	0.7501	213	0.7522	223

The average SSIM value for all the images in the data-set from the proposed algorithm (LBPT − LS) is around 0.9282 and the average number of iterations is 189. The average SSIM values for GTP − LS and GLCMTP − LS are 0.7798 and 0.7801 respectively. Similarly, the average iteration count is 221 and 228 for GTP and GLCMTP. Further these values can be improved using a combination of these methods and more complicated flower image structures.

4. Conclusion

A novel method for flower image segmentation is attempted in this work. Flower images are the most complicated structures nature creates for humans which are difficult to understand. The task is accomplished using level sets that are pre informed about the color, shape and texture related to the flower. LBP make texture information related to the flower. Shape is hand segmented and the level set is evolved in the RGB color plane using the shape and texture information. SSIM and number of iterations are used as a performance measure for comparing the proposed level set with Gabor texture filter based level set and GLCM based level sets. The proposed method using LBP textures outperforms the other two texture models for flower classification. In future, a more comprehensive model is required to segment flowers and extract meaningful features in real time. A deep learning model is being developed for classifying a large data-set of flowers.

Cover image

Source: Authors.

Erratum

This article was originally published with errors. This version has been corrected. Please see Erratum https://doi.org/10.1080/23311916.2017.1333255.

Additional information

Funding

Funding. The authors received no direct funding for this research.

Notes on contributors

Syed Inthiyaz

Syed Inthiyaz was born in Vijayawada, Krishna (District), AP, India. He received BTech in ECE from JNT University Hyderabad, AP, India. MTech from JNT University, Kakinada. His research interested areas include Image Processing and Video Processing.

B.T.P. Madhav

B.T.P. Madhav was born in India, AP, in 1981. He received the BSc, MSc, MBA, MTech degrees from Nagarjuna University, AP, India in 2001, 2003, 2007, and 2009 respectively. He got his PhD in the field of antennas from K.L. University. He currently working as Professor in Electronics and Communication Engineering department of ECE. He has published more than 240 papers in International, National journals and Conferences. His research interests include antennas, liquid crystals applications and wireless communications.

P.V.V. Kishore

P.V.V. Kishore (MIEEE’07) received his PhD degree in Electronics and Communications Engineering from Andhra University College of Engineering in 2013. He received MTech from Cochin University in the year 2003. He is currently full professor and Multimedia and Vision Computing Research Chair at K.L. University, ECE Department.