An Optimized Image Segmentation Approach Based on Boltzmann Machine

Figures & data

Figure 1. Flowchart of the proposed approach.

Figure 2. “0027” image from subset1, (a) GT image, (b) segmented mask image by BM, (c) segmented mask image by GF2T, (d) original image, (e) extracted object image by BM, and (f) extracted image by GF2T.

Figure 3. “0049” image from subset2, (a) GT image, (b) segmented mask image by BM, (c) segmented mask image by GF2T, (d) original image, (e) extracted object image by BM, and (f) extracted image by GF2T.

Figure 4. “0002” image from subset3, (a) GT image, (b) segmented mask image by BM, (c) segmented mask image by GF2T, (d) original image, (e) extracted object image by BM, and (f) extracted image by GF2T.

Figure 5. “0037” image from subset4, (a) GT image, (b) segmented mask image by BM, (c) segmented mask image by GF2T, (d) original image, (e) extracted object image by BM, and (f) extracted image by GF2T.

Figure 6. “0023” image from subset5, (a) GT image, (b) segmented mask image by BM, (c) segmented mask image by GF2T, (d) original image, (e) extracted object image by BM, and (f) extracted image by GF2T.

Table 1. BoMISe’s algorithm.

Step 0. Begin Step 1. Eliminate the possible noises or artifacts from given image by the use of MS filtering with the parameters Sp and Sr determined by the end-user Step 1.1. Construct the smoothed image Step 2. Use GF2T method to select strongest corners from given smoothed image according to the parameter value of the number of maximum corners Step 2.1. To find strongest corners, all corners in the image below quality level are rejected Step 2.2. Sort the remaining corners based on quality in the descending order Step 2.3. Take the first strongest corner, throw away all the nearby corners in the range of minimum Euclidean distance Step 2.4. Return n strongest corners as a set of the coordinates of salient features, Step 3. If BM is selected for optimization process Step 3.1. To form n^2 layout matrix of TSP, assign n cities’ coordinates of TSP from the coordinates of salient features’ set in Step 2.4 as position of the city in the tour and the index of that city Step 3.2. At starting temperature T0 = 100°C, start to run the BM net with current layout matrix of TSP Step 3.3. Repeat throughout the annealing process from Step 3.3 to Step 3.8, update randomly and asynchronously the BM units (i.e., neurons) according to annealing (i.e., cooling) schedule to compute the current tour with weighted connections of units as distances between cities of TSP Step 3.4. In every step of annealing process, decrease T0 value by multiplying the previous temperature by 0.99 to reach the ending temperature Tf = 0.000001 Step 3.5. In every step of annealing process, calculate the energy difference between previous state and current state of the system (i.e., machine) Step 3.6. According to the Metropolis criterion, compute the given unit’s activation probability and thus this unit is updated with the probability of becoming (or remaining) active (i.e., state is “on”) Step 3.7. Bring the BM net to thermal equilibrium condition, thus compute the TSP tour length and path by consider the distances between cities Step 3.8. If the stopping condition is met (i.e., feasible solution with a valid tour for consensus function) or Tf is reached, then the optimization process is finished, exit from the loop of Step 3.3; therefore, the junction points of final valid TSP tour’s path are accepted as convex spline junction points Step 4. If BM is not selected for optimization process, apply simply the convex hull encapsulation method to select outermost coordinates of salient points to construct a convex spline with its junction points Step 5. Use the GrabCut method for automatic image segmentation as given Step 5.1 to Step 5.12 Step 5.1. Identify automatically an initial ellipse which is formed by the convex spline of convex hull encapsulation of Step 4 or the convex spline from BM process as best fitted ellipse Step 5.2. Form the shape mask from this ellipse around the OOI Step 5.3. Mark the region inside that ellipse as unknown, also, pixels outside of this ellipse will then be marked as known background. Therefore, there is a trimap: the background, foreground, and unknown part of the given image Step 5.4. As unknown pixels placed in the foreground class and all known background pixels placed in the background class, initialize the segmentation of image Step 5.5. Assign GMM components to pixels of foreground and background regions Step 5.6. In the foreground GMM, assign every pixel in the foreground as most probable Gaussian component and, also, every pixel in the background as most probable Gaussian component in the background GMM Step 5.7. Learn GMM parameters from data (i.e., pixel sets) that were created in Step 5.6 Step 5.8. Apply Scharr operator over the image to obtain gradient image Step 5.9. Set-up the graph using the vertices in the convex spline (i.e., optimized by BM) Step 5.10. Compute the energy using gradient image intensity, the mean intensities, and shape mask (i.e., best fitted ellipse) Step 5.11. Calculate the minimum cut of graph with GMMs and abovementioned pixel data Step 5.12. Repeat from Step 5.5, until the classification convergence Step 5.13. Construct the final bi-level segmented image according to the classification result Step 6. Display extracted OOI image Step 7. End

GMM: Gaussian mixture model; TSP: traveling salesman problem; OOI: object of interest.

Table 2. The quantitative evaluation of performance based on the precision metric. The results shown in bold in the table indicate the best performance values.

Subset ID	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Subset1	0.9940	0.9522	0.9325	0.9195
Subset2	0.9907	0.9147	0.9572	0.8817
Subset3	0.9339	0.8965	0.8588	0.9097
Subset4	0.9041	0.9590	0.8486	0.9267
Subset5	0.8079	0.9063	0.6590	0.8716

BM: Boltzmann machine; GF2T: Good Features to Track.

Table 3. The quantitative evaluation of performance based on the recall metric. The results shown in bold in the table indicate the best performance values.

Subset ID	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Subset1	0.8122	0.8207	0.7857	0.8811
Subset2	0.4899	0.6179	0.4371	0.6992
Subset3	0.4019	0.4224	0.3669	0.4455
Subset4	0.6516	0.7608	0.5994	0.8217
Subset5	0.6904	0.7280	0.6398	0.7964

BM: Boltzmann machine; GF2T: Good Features to Track.

Table 4. The quantitative evaluation of performance based on the F-measure metric. The results shown in bold in the table indicate the best performance values.

Subset ID	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Subset1	0.8901	0.8564	0.8492	0.8754
Subset2	0.6411	0.7058	0.5896	0.7491
Subset3	0.5235	0.4791	0.4780	0.5086
Subset4	0.7300	0.8176	0.6726	0.8406
Subset5	0.7061	0.7614	0.6049	0.7856

BM: Boltzmann machine; GF2T: Good Features to Track.

Table 5. The quantitative evaluation of performance based on the Jaccard metric. The results shown in bold in the table indicate the best performance values.

Subset ID	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Subset1	0.8084	0.7773	0.7439	0.8066
Subset2	0.4860	0.5637	0.4283	0.6138
Subset3	0.3749	0.3550	0.3343	0.3893
Subset4	0.5884	0.7258	0.5206	0.7580
Subset5	0.5620	0.6625	0.4646	0.6922

BM: Boltzmann machine; GF2T: Good Features to Track.

Table 6. The quantitative evaluation of performance based on the kappa metric. The results shown in bold in the table indicate the best performance values.

Subset ID	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Subset1	0.3352	0.3304	0.2481	0.2639
Subset2	0.1634	0.1970	0.1020	0.1510
Subset3	0.1623	0.1603	0.1120	0.1243
Subset4	0.2866	0.3139	0.2056	0.2579
Subset5	0.2593	0.2646	0.1764	0.2094

BM: Boltzmann machine; GF2T: Good Features to Track.

Table 7. The overall performance results obtained for five image subsets. The results shown in bold in the table indicate the best performance values.

Metric	GF2T result based on the bounding boxes	BM result based on the bounding boxes	GF2T result based on the pixel regions	BM result based on the pixel regions
Overall precision	0.9261	0.9257	0.8512	0.9018
Overall recall	0.6092	0.6699	0.5657	0.7287
Overall F-measure	0.6981	0.7240	0.6388	0.7518
Overall Jaccard	0.5639	0.6168	0.4983	0.6519
Overall kappa	0.2413	0.2532	0.1688	0.2013

BM: Boltzmann machine; GF2T: Good Features to Track.