ASM-VoFDehaze: a real-time defogging method of zinc froth image

Abstract

When the ambient temperature is low, a large amount of water mist and dust will inevitably appear around the zinc flotation cell, forming haze, which seriously affects the extraction of flotation froth image features. General defogging methods of natural image are difficult to obtain satisfactory results for such industrial haze image. Therefore, we propose a real-time defogging method based on ASM-VoFD (Atmospheric Scattering Model and the Variable-order Fractional Differential). First, the dark pixel ratio is used to detect fog in froth image, which solves the redundant calculation caused by unnecessary defogging operations. Second, the linear transformation of the atmospheric scattering model is used to calculate the initial transmission map, and the gaussian filter is used to optimize the initial transmittance, and the haze-free image is restored with atmospheric light estimation. Finally, a variable order fractional differential operator is used to enhance the edges and texture details of the restored froth image, which solves the problems of blurred edges and low contrast. The experiments show that the algorithm has a good defogging effect on the industrial images, enhance the edges of the image, and can be effectively implemented in O(N) time to meet the application requirements of real-time flotation monitoring.

Keywords:

• Real-time defogging of zinc froth image
• atmospheric scattering model
• linear transformation
• Gaussian filtering
• the variable-order fractional differential operator

1. Introduction

Froth flotation is a beneficiation method to separate minerals according to the physical and chemical properties of the mineral surface. The main process of froth flotation is that bubbles rise in the pulp with selective but adhered mineral particles at the gas–liquid interface, and then the froth formed on the pulp surface is scraped to achieve the purpose of beneficiation. The surface visual characteristics of these froths (such as froth size, colour, texture, and flow rate) are closely related to process indicators, working conditions, and operating variables. These characteristics can be used as an important basis for judging the effect of mineral separation operations. The accurate extraction of froth image features is the premise of the machine vision flotation production process (Aldrich et al., 2010). However, the complex flotation industry has a harsh environment with a large amount of dust and fog, as well as uneven illumination, resulting in serious pollution in images collected by surveillance video (Jinping et al., 2010). In particular, when the temperature in the plant is low (below 5${}^{\circ }$C), The water mist produced by the mineral particles and water droplets splashed during the froth defoaming is very large. This condition causes a thick water mist that covers the entire flotation cell. The existence of these water mists seriously affects the accurate extraction of bubble characteristics. Therefore, dehazing preprocessing on the froth image is an important task in obtaining froth parameters.

The flotation state monitoring system based on machine vision (as shown in Figure 1(a)) not only obtains traditional process parameters, but also collects a large amount of visual image information related to working conditions. Then, these parameters are fused with characteristic information to realise the intelligent identification and accurate trend forecast of multi-modal conditions in the flotation process. The collection and processing of data is very important to the analysis results (Liang et al., 2019). Long-term practice observation shows that the flotation plant suffers from insufficient light at night and a lot of floating dust, especially when the temperature is low, the haze will appear, as shown in Figure 1(b). Different flotation tanks (zinc-roughing, -refining, and -scavenging) carried different zinc content of the froth, and the froth showed different grey-brown. Figures 1(c), 1(d) and 1(e) are the foggy froth images in the three flotation tanks, respectively. In particular, the quality of fogged froth images (as shown in Figure 1(e)) taken in dark weather or low light at night is very low, which urgently requires real-time and effective defogging algorithms to improve the clarity of froth images.

Figure 1. (a) Flotation site industrial camera; (b) flotation plant covered by mist; (c) hazy image of the zinc-roughing tank; (d) hazy image of the zinc-refining tank; (e) hazy image of the zinc-scavenging tank.

Figure 3 shows the clear images, hazy images and the corresponding histogram in the zinc-roughing tank. Figure 3 shows that the grey value range of the foggy froth image is greatly reduced, indicating a single peak state. The reason is that a layer of haze is added to the bubble map, and the image is off-white as a whole. Compared with natural images, froth images have the following characteristics:

1. In the froth image, only a small amount of black and dark small pieces are seen, especially the high mineral content carried by the zinc-refining froth, which is not easy to break, and finding pixels with a grey value less than 50 is difficult. The defogging algorithm based on dark channel prior (He et al., 2010; C. Li et al., 2020; J. B. Wang et al., 2015; Wei et al., 2020) has a very good effect in natural images, but it is not suitable for flotation froth image defogging, which will result in high estimated transmittance and the recovered image cannot truly reflect the froth characteristics.

2. Due to the fact that the flotation froth is sporty during the shooting process, it is impossible to find the corresponding clear image from the collected blurred image. This limits the application of some advanced and effective defogging algorithms. For example, the method of estimating scene depth information by obtaining multiple images of the same scene (Narasimhan & Nayar, 2000, 2002), and the supervised deep learning network method (Cai et al., 2016; B. Li et al., 2017).

3. The distance between the froth surface and the industrial camera is relatively stable, that is, the depth of field of the froth image is basically the same. Except for the reflected light-bright spots on the top of the froth, there is no large colour jump at the edges between the foams, which is more conducive to the restoration of hazy images based on the linear transformation of the atmospheric scattering model. For example, the linear transformation method of the atmospheric scattering model can achieve a better defogging effect when the depth of the field is basically the same (Alajarmeh et al., 2018; Ge et al., 2015; Ju et al., 2017; W. Wang et al., 2017). The key is that it has a great advantage in time efficiency.

4. The light source when collecting the froth image is mainly the high-brightness incandescent set in the installation box of the industrial camera. Auxiliary light sources are from atmospheric light and factory lighting. However, froth images taken in dark weather and night environment are low-light images with unclear edges, so many good defogging algorithms cannot achieve the ideal defogging effect.

The purpose of this study is to develop a suitable defogging method of flotation froth image, improve the accuracy of feature extraction of flotation froth image, and meet the application requirements of real-time monitoring based on machine vision. In order to achieve this goal, we first analyse the different characteristics between clear froth image and fogged image, and identify foggy froth image with average brightness ratio and dark pixel ratio. Secondly, we estimate the transmission image by the prior information of local image energy of gradient (EOG) and the linear transformation of atmospheric scattering model (ASM), optimise the initial transmittance by Gaussian filtering, and combine the estimated atmospheric light value to reconstruct the clear image. Finally, for the edge blur of the restored low-light image at the froth-highlights, we use the variational fractional differential operator to enhance the edges, texture details and brightness to obtain the optimal defogging performance.

The main contributions of this research work are as follows:

1. Before the image defogs, we detect the froth image by extracting a dark pixel ratio, which can effectively identify the foggy image and distinguish it from other froth blur images (motion blur and defocus blur). This provides a reference for the recognition of blurred images in the flotation industry.

2. In the defogging algorithm, the maximisation of the local energy gradient function is adopted as a prior information, which is easy to realise and independent of colour information. Therefore, such information is more suitable for the defogging of the zinc flotation froth image (monochrome colour, greyish brown).

3. Through qualitative and quantitative analysis in the experiment, this method is superior to the five advanced defogging methods in image recovery performance and time efficiency. At the same time, edge enhancement is performed on the defogging results of low-light fogged images, which improves the applicability of the algorithm.

2. Related work

At present, dehazing algorithms are mainly divided into two categories: image enhancement and image restoration. The image enhancement method does not consider the causes of image degradation and has wide applicability. This method can effectively improve the contrast of the foggy image, enhance the image details, and improve the visual effect of the image, but it may cause some loss of information in the prominent part. Typical methods include histogram equalisation (Stark, 2000), retinex (Zeng et al., 2014; Zhou & Zhou, 2013), the wavelet transform (Khmag et al., 2018), and contrast enhancement algorithms (Kim et al., 2011). The defogging method based on the physical model mainly establishes a degradation model by analysing the principle of light scattering by the atmosphere and uses the model to obtain a clear fog-free image. This kind of method has strong pertinence and natural dehazing and generally involves no information loss. The key point of processing is the estimation of the parameters in the model. McCartney (1976) first used this principle (Mie Scattering Theory) to establish an atmospheric scattering model in 1977. In 1998, Oakley and Satherley (1998) began to use The Mie scattering theory to carry out some restoration studies on images taken under severe weather. Subsequently, model-based defogging methods have attracted more and more attention and have now become a research hotspot in the field of image processing. Fattal (2008) estimated the reflectivity of the scene and the transmittance of the medium under the assumption that the transmittance and the surface shadow were locally unrelated. This method has a good effect on the processing of light haze images, but may fail for images with heavy haze. Tarel and Hautiere (2009) introduced a method based on contrast enhancement to eliminate haze. The algorithm is simple and efficient, but the parameters in the algorithm cannot be adjusted adaptively. The best and well-known algorithm is single image dehazing based on dark channel prior theory proposed by He et al. (2010). However, the optimisation process takes more time in estimating the transmittance map. Subsequently, extensive studies have been conducted on this theory (Fujita & Fukushima, 2017; Khmag et al., 2018). Kim et al. (2013) enhanced the image by maximising the contrast block by block, minimised the information loss caused by pixel truncation, and minimised the colour distortion of the restored image as much as possible. W. Wang et al. (2017) proposed a fast single-image haze removal method based on linear transformation. The algorithm is effective in removing fog in images with little change in depth of field. But in the low light fog image, the edge of the restored image is blurred. Wu et al. (2021) proposed a method of atmospheric scattering linear transformation based on local image entropy maximisation prior to estimate the transmittance of image blocks and optimised the transmittance by guided filtering. The algorithm has good defogging performance and high efficiency, which can meet the needs of industrial applications. The algorithm (Kim et al., 2013; Tarel & Hautiere, 2009; W. Wang et al., 2017; Wu et al., 2021) uses some prior information to transform the atmospheric scattering model, which can obtain the estimation of transmittance. These algorithms are suitable for foggy images with little change in depth of field and have high efficiency.

With the development of neural networks and deep learning, researchers have begun to replace traditional image dehazing methods with methods based on deep learning. Tang et al. (2014) proposed a novel transmission estimation method through a learning-based method; the researchers analysed the relevant characteristics of multi-scale haze in a regression framework based on random forest, but this feature fusion method relies heavily on dark channel features. Cai et al. (2016) proposed an end-to-end training model that used neural networks to estimate $t(x)$ in the atmospheric degradation model. The model inputs the hazy map, outputs the transmission rate $t(x)$ map, and then uses the atmospheric degradation model to restore the haze-free image. B. Li et al. (2017) proposed an image defogging model established by a convolutional neural network (CNN), called all-in-one defogging network (AOD-net). AOD net does not estimate the transfer rate and atmospheric light as previous models do, but directly generates a de-fogging image. These learning-based algorithms require many haze-free and hazy images as training data. If the training sample is not large enough, then it cannot display the true depth information of the image, especially the edge area. In the process of capturing the flotation froth image, the foggy image and the corresponding non-fog image cannot be obtained due to the froth movement and the fog mask. At the same time, the change of illumination makes the extracted fogged characteristics also affect the recovery result.

3. Dehazing algorithm

The method of removing the haze in the flotation froth monitoring video consists of the following steps: (1) foggy image recognition (dark pixel ratio features); (2) defogging method based on linear transformation of the atmospheric scattering model; (3) image edge enhancement of the restoration result. First, the detailed content of the foggy image recognition algorithm in the first part is described.

3.1. Foggy image recognition

In the haze image, the human eye's recognition ability is lower than that of normal weather, and it is more sensitive to the lightness information than the colour information of the image. Therefore, the dark pixel ratio is selected as the identified features of the foggy image. Figure 2 shows a flowchart of the froth foggy image recognition.

Figure 2. The flowchart of froth foggy image recognition.

Figure 3. Dark pixel ratio of blur and clear image: (a) clear image; (b) zinc-roughing foggy image; (c) motion blur image; (d) defocus blur image.

Under the same illumination conditions, the lightness of clear froth image and foggy image in the same flotation cell is very different. This is mainly reflected in the histogram of its lightness component. The histogram of the clear image is evenly distributed and smooth, while the histogram of the foggy froth image becomes narrower and thinner and moves to the right as a whole, as shown in Figure 3. Therefore, compared with clear images, the lightness value of dark pixels increases in foggy images, resulting in a significant reduction in the dark pixel ratio.

Perform histogram statistics for lightness component image. Dark pixel ratio D is denoted as (1) $D=l_{\mathrm{dark}}/l_{\mathrm{all}}=\sum _{k=0}^{k=100}{H}_k/\sum _{k=0}^{k=255}{H}_k,$(1) where $H_k$ represents the histogram value with a grey level of k. Dark pixel $l_{\mathrm{dark}}$ refers to the corresponding pixel to the grey value range [0,100]. The statistical result of the dark pixels $l_{\mathrm{dark}}$ is $\sum _{k=0}^{k=255}{H}_k$.

We choose the froth image of the zinc-roughing flotation tank for comparative analysis. Experimental results are shown in Figure 3. The dark pixel ratio of clear images is larger than that of motion blur images and defocus blur images, while the dark pixel ratio of foggy images is much smaller than that of the first three images, almost 10 times the difference. Thus, this dark pixel ratio method can be used to classify foggy froth images.

3.2. Zinc flotation froth image defogging

3.2.1. Atmospheric scattering model of flotation industry site

The atmospheric scattering models are as follows: (2) $I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right),$(2) where I is the intensity of the haze image observed by the imaging device, J is the intensity of the haze-free image to be restored, A is the atmospheric light, and t is the transmittance that reflects the ability of light to penetrate the haze. In Equation (2(2) $I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right),$(2) ), $J(x)t(x)$ represents the part of the reflected light from the target that enters the camera after atmospheric scattering and attenuation; this part decays exponentially as the depth of the scene increases. $A(1-t(x))$ is the part of the camera that enters the camera after the atmospheric light is scattered; this part increases with the depth of the scene, which causes blurring and colour shift distortion. Parameters $t(x)$ and A have to be estimated to restore the haze-free image. The solution to this problem is ill-conditioned. The atmospheric scattering model of the flotation froth foggy image is presented in Figure 4.

Figure 4. Atmospheric scattering model of flotation foam atomisation image.

The clear processing of the flotation froth foggy image requires strong prior information or a reasonable structure to solve the model. The algorithm proposed in this chapter aims to perform linear transformation on the scattering model, and use the local image EOG and the fidelity for the constraint to restore the haze-free froth image. This method not only obtains a good dehazing effect but also retains the textural details of the image. Furthermore, the transmittance is quickly optimised, thereby meeting real-time industry requirements.

3.2.2. Estimation of atmospheric light

The accurate estimation of atmospheric light A value in the atmospheric scattering model is directly related to the accurate calculation of transmittance, and ultimately affects the colour and brightness of the restored image. Generally, in the dehazing of natural images, if the atmospheric light value A is extremely large, the overall image is darker and the visibility is low. When the atmospheric light value A is low, some parts of the image (such as bright areas, light edges, and white objects) have obvious colour distortion.

According to Figure 1, the main light source of the froth image video comes from the light-emitting diode installed at the top of the box. Due to the physical and characteristics of the froth, the froth that adheres to the mineral particles automatically burst after a certain period under the action of gravity. As a result, the mineral content on the top of the froth slowly decreases until only a water film is found at the end, which bursts immediately. The reflection of the light source at the water-air interface is total reflection, so in the froth image we obtain, bright spots are seen at the top of the froth. In the non-fog state, the brightness of the froth highlights is extremely high and is basically pure white. In the foggy state, the froth highlights are shrouded in haze, which is grey and foggy. In the image of the flotation froth with fog, the bright spots on the froth reflect the atmospheric light value of the corresponding area of the image. In this section, the froth image is first greyed out, and then the threshold is set to extract the highlight pixels of the froth image. The average value of these pixels is taken as the estimated result of the atmospheric light A value. Many experiments have proved that the estimated results of atmospheric light A value in different flotation cell froths are different. The atmospheric light A of the zinc-roughing tank is approximately 220, the zinc-refining tank is approximately 175, and the zinc-scavenging tank is approximately 190. The result can effectively reflect the real atmospheric light conditions and restore the texture details of the fog-free image.

3.2.3. Estimation of transmission map

When atmospheric light A is known, the accurate estimation of the transmittance $t(x)$ determines the effect of image dehazing. This study proposes that in the real-time dehazing of a linear transformation of the atmospheric scattering model, the local image EOG and fidelity maximisation prior is used in estimating the transmission map.

(a) Linear transformation of atmospheric scattering model

According to Equation (2(2) $I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right),$(2) ), the mathematical model of a single dehazing image $J(x)$ can be derived as follows: (3) $J\left(x\right)=\frac{I\left(x\right)-A}{t\left(x\right)}+A.$(3) Based on the assumption that in a small local image, the depth of the image is the same (in fact, the foam layer formed by the bubbles floating up to the surface of the slurry is basically the same as the depth of the camera above the flotation tank), which means the transmission rate $t(x)$ of each small block is a constant. Then, according to Equation (2(2) $I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right),$(2) ), each pixel p in the image block $\mathrm{\Omega }(y)$ can be expressed as (4) $\underset{p\in \mathrm{\Omega }\left(y\right)}{J\left(p\right)}=a\cdot I\left(p\right)+b,$(4) where a is the slope and b is the intercept. They are expressed as (5) $a=\frac{1}{t\left(p\right)},b=A-\frac{A}{t\left(p\right)}.$(5) Equation (4(4) $\underset{p\in \mathrm{\Omega }\left(y\right)}{J\left(p\right)}=a\cdot I\left(p\right)+b,$(4) ) shows that in a partial image block, a linear relationship exists between the haze-containing image $I(p)$ and the clear haze-free image $J(p)$. The slope a and intercept b change with the change of the transmission rate t.

(b) The maximisation prior of EOG–fidelity objective function

The clarity of flotation froth foggy image was evaluated by the energy of the gradient function. This function takes the squares sum of the squares of the difference between the grey values of adjacent pixels in the x and y directions as the gradient value of each pixel, and then accumulates the gradient values of all pixels as an evaluation of image sharpness. For the image $f(x,y)$, the definition of the energy gradient function is expressed as follows: (6) $F=\sum _x\sum _y{G}_x^2(x,y)+G_y^2(x,y)$(6) In this forum: (7a) $\begin{array}{rl}{G}_x(x,y)& =f(x,y)\otimes T_x,\end{array}$(7a) (7b) $\begin{array}{rl}{G}_y(x,y)& =f(x,y)\otimes T_y.\end{array}$(7b) The symbol ⊗ represents the spatial filtering of the image under the mask template. The size of $T_x$ and $T_y$ is (8) $T_x=\left[\begin{array}{cc}-1& 0\\ 1& 0\end{array}\right]T_y=\left[\begin{array}{cc}-1& 1\\ 0& 0\end{array}\right].$(8) The pixel neighbourhood corresponding to the mask template is (9) $\left[\begin{array}{cc}f(x,y)& f(x+1,y)\\ f(x,y+1)& f(x+1,y)\end{array}\right].$(9) Therefore, the expression of the image energy of the gradient function is (10) $F_{eog}=\sum _x\sum _y^{}\{{[f(x+1,y)-f(x,y)]}^2+{[f(x,y+1)-f(x,y)]}^2\}.$(10) Equation (2(2) $I\left(x\right)=J\left(x\right)t\left(x\right)+A\left(1-t\left(x\right)\right),$(2) ) shows that if the value of the transmittance $t(x)$ is small, then the value of $J(x)$ may appear more than the upper limit of grey value 255. If the value of the transmittance $t(x)$ is large, the value of $J(x)$ less than the lower limit of the grey value 0. In this case, the grey value beyond the range of $[0,255]$ is automatically cropped, which causes colour distortion in the restored fog-free image. Therefore, this study uses the local image energy gradient maximisation prior as the constraint to estimate the optimal transmittance $t(x)$, and also designs a subfunction (Fidelity function), that is, the image colour metric sub-function composed of information fidelity, such as (11) $\{\begin{array}{c}{F}_{\mathrm{fidelity}}(t)={\displaystyle \underset{c\in \{r,g,b\}}{min}{S}^c(t),}\\ \\ S^c(t)={\displaystyle \frac{1}{N}\sum _{p=1}^N\delta (p),}\\ \\ \delta (p)=\{\begin{array}{cc}1,& 0\le J^c(p)\le 255,\\ 0,& \mathrm{otherwise},\end{array}\end{array}$(11) where p represents a certain pixel in the restored image, N is the total number of pixels, and $S^c(t)$ represents the ratio of pixels in the range of $[0,255]$ each colour channel of the restored haze-free image J when the transmittance $t(x)$ is known. When the pixel ratio to be cropped is smaller, the value of the information fidelity function $F_{fidelity}$ is larger. The maximum value of this function can be used to control the pixel ratio that is cut off, which maximises the EOG of the local image to the greatest extent and makes the restored image clearer while retaining its original colour.

The proposed algorithm designs an objective function based on the energy of the gradient fidelity to constrain the estimated optimal transmittance $t(x)$ as follows: (12) $F_{obj}=F_{eog}\times F_{fidelity},$(12) where $F_{eog}$ represents the image energy gradient function, and $F_{fidelity}$ represents the image colour metric subfunction. The optimal estimated value of the transmittance t based on the energy gradient-fidelity objective function proposed in this paper is (13) $\begin{array}{rl}{t}^{\ast }& =\arg max\left\{F_{obj}\left(t\right)\right\}\\ & =\arg max\{F_{eog}\left(t\right)\times F_{fidelity}\left(t\right)\},\end{array}$(13) where $F_{obj}(t)$, $F_{eog}(t)$, and $F_{fidelity}(t)$ represent when the transmittance t is known, the EOG–fidelity of the restored haze-free image J function value, the EOG function value, and the fidelity function value.

As shown in Figure 5, the lines with the highest and lowest peak values, respectively, represent the changes in local energy of gradient and fidelity, and the middle line represents the changes in the objective function $F_{obj}$. In this example, when the transmittance t is less than 0.3, the objective function value is relatively low, and the fidelity is much less than 1, indicating that within this range, few points exist in the image with pixel values between 0 and 255, which does not meet our needs for results. When the transmittance t exceeds 0.6, the fidelity is close to 1, which means that almost no pixels are cut off. At the same time, when the transmittance t approaches 1, $F_{eog}$ gradually decreases. Therefore, the optimal transmittance should be determined between 0.5 and 0.6 to ensure that the energy gradient-fidelity objective function reaches the maximum value.

Figure 5. The objective function for finding the optimal transmission value.

Through the aforementioned method, the transmittance of a small image can be estimated. To obtain the best transmittance, we divide the image into $8\times 8$, $16\times 16$, $24\times 24$, and other small blocks. Then, we use the step size as the 1/2 sliding window of the block size to estimate, which can reduce the degree of block artefacts in the restored image to a certain extent. The preceding steps are repeated to obtain the initial estimated transmission map.

3.2.4. Refining the transmission map

The transmission map obtained by the method in Section 3.2.3 is pixel-based, and the transmission rate is the same in a small area. Theoretically, the transmission rate of each pixel is different in the image, especially at the image edge or where the depth of field jumps. At this time, the restored image may have halo or block artefacts. To solve this problem, we use Gaussian blur filtering to refine the block-based transmission mapping, reduce block artefacts, and enhance image details. The Gaussian blur method uses the weighted average method. At the image scale of $M\times N$, the Gaussian function represented by pixel $(x,y)$ is as follows: (14) $G(p,q)=\frac{1}{2\pi {\delta }^2}{\mathrm{e}}^{\frac{(x-m/2{)}^2+(y-n/2{)}^2}{2{\delta }^2}}.$(14) The larger the standard deviation $\delta$ of the normal distribution, the more blurred is the image. The distribution map after Gaussian filtering is obtained after convolving two non-zero matrices: (15) $t^{\mathrm{\prime }\mathrm{\prime }}(x)=t^{\mathrm{\prime }}(x)\times G.$(15) By comparing the results of Gaussian, average, anisotropic, median, and guided filtering on the image processing results, we find that the effect of Gaussian filter processing is better than that of mean filter and median filter processing, and the effect of guided filtering is best. Considering the processing time, Gaussian filtering and average filtering take less time. Figure 6 shows the process of estimating the transmission rate, refining the transmission map, and generating a fog-free image. We analyse the impact of various filtering algorithms on the results in Section 4.2.

Figure 6. Calculate the transmission map and dehaze processing effect map for the foggy image.

3.3. Recovered image enhancement based on variable-order fractional differential operator

In order to solve the problems of unclear edge and low contrast of the recovered image, the variable-order fractional differential operator (Xu et al., 2015) is used to enhance the image. The flow chart of this method is shown in Figure 7.

Figure 7. The flow chart of froth images enhancement.

In this paper, the Riemann–Liouville fractional derivative is adopted and its definition is as follows: (16) ${}_a{D}_x^{-v}f(x)=\frac{1}{\mathrm{\Gamma }(v)}{\int }_a^x(x-s{)}^{v-1}f(s)\mathrm{d}s,$(16) where the $\mathrm{\Gamma }(\cdot )$ represents the Euler's gamma function. Then the expressions of right and left R-L fractional derivatives are shown below, respectively: (17) $\begin{array}{rl}{}_a{D}_x^{v}f(x)& =\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}[{}_a{D}_x^{-(n-v)}f(x)]=\frac{1}{\mathrm{\Gamma }(n-v)}\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}{\int }_a^x(x-s{)}^{n-v-1}f(s)\mathrm{d}s,\end{array}$(17) (18) $\begin{array}{rl}{}_a{D}_b^{v}f(x)& =\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}[{}_x{D}_b^{-(n-v)}f(x)]=\frac{1}{\mathrm{\Gamma }(n-v)}\frac{{\mathrm{d}}^n}{\mathrm{d}{x}^n}{\int }_x^b(x-s{)}^{n-v-1}f(s)\mathrm{d}s,\end{array}$(18) where $v\in (n-1,n),n\in N$. And the mathematical expression for the discretisation of R-L fractional derivative at point $x=x_j$ is shown below: (19) ${[{}_a{D}_x^{v}f(x)]}_{x=x_j}=\frac{1}{h^v}\sum _{j=0}^n{C}_j^{v}f\left(x_{n-j}\right)+O\left(h^p\right),$(19) where the v is the order of the fractional derivative. And the h represents the step size, the $C_j^v$ is the p-order discrete coefficient of the R-L fractional derivative. In addition, we use the fractional backward difference formula of order p (FBDF-p) to generate discretisation coefficients. The discretisation coefficient $C_j^v$ can be obtained by the following recurrence formulas: (20) $\begin{array}{rl}& C_j^v=\frac{1}{j{u}_0}\sum _{t=0}^{j-1}[v(j-i)-i{]}_i^v{u}_{j-t},\end{array}$(20) (21) $\begin{array}{rl}& \sum _{j=1}^p\frac{1}{j}(1-z{)}^j=\sum _{j=1}^p{u}_j{z}^j,\end{array}$(21) where the discrete order p satisfies $p=\{1,2,3\}$.

In this paper, a template structure based on eight symmetric directions is used to construct a variable-order fractional differential mask, as shown in Figure 8.

Figure 8. Fractional differential mask construction.

The size of the template is $(2m+1)\times (2m+1)$, and all elements in the template are normalised, as shown below: (22) $Sum=\sum _{j=0}^m{C}_j^v(j=0,1,\dots ,m).$(22) Then, the final enhanced froth images can be obtained through convolution based on the variable-order fractional-order template M, which can be expressed by the following formula: (23) $J(x,y)=M\otimes f(x,y)=\sum _{i=-m}^m\sum _{j=-m}^{m}M(i,j)f(x+i,y+j).$(23) The enhanced image J can be finally obtained by sliding the template size window from left to right and from top to bottom to conduct convolution operation on the froth image.

4. Experimental analysis

Experiments were carried out on flotation froth images based on the above proposed defogging and enhancement algorithm, which proved the effectiveness of the proposed method for the clearness of flotation froth foggy images.

4.1. Date sets

The visual data of the froth layer on the pulp in the flotation tank were collected by the industrial camera installed in the flotation field, and the foggy froth image data set was obtained for the defogging pretreatment. The camera is installed above the flotation tank, as shown in Figure 1(a). In the zinc flotation process, the life of froth from generation to deformation and then to rupture is a complex physical and chemical process. At the same time, the whole froth layer on the pulp has dynamic changes such as displacement and collapse due to the surge of pulp and scraping-bubble. Therefore, even for flotation froth manufactured in the laboratory, the corresponding clear image of the foggy image cannot be obtained with the artificial addition of fog. In the experiment, we captured froth images of different illumination conditions, different flotation tanks and different haze levels in the flotation field, and established two image data sets named Image_nor (normal illumination) and Image_low (low illumination) (90 $320\times 240$ images, respectively). Each data set contains three groups of froth foggy images (30 images per tank) from different flotation tanks (zinc-refining tank, zinc-roughing tank and zinc-scavenging tank), among which the images in each group have fog images of light, medium and heavy fog levels (10 images per stage).

4.2. Qualitative validation experiments

In this part, we will verify the performance of the algorithm proposed in this paper from the qualitative aspect, mainly from fog concentration, lighting conditions, transmission rate optimisation, fog removal algorithm to verify the effect of image restoration.

4.2.1. Influence of various filtering methods on dehazing effect

In refining the transmission map, various filtering methods have different processing effects. We, respectively, perform Gaussian, average, guided, bilateral (Liang et al., 2021) and median filtering on the initial transmission map of the hazy image of the zinc flotation cell. The processing effects of filtering are shown in Figure 9, where the VER value is the image quality evaluation standard without reference (see Section 4.3 for specific calculation), and the TIME value is the execution time of optimised transmission graph.

Figure 9. Comparison of different filtering dehazing effects.

In the recovered image, the effects of the guided filter, bilateral filter and Gaussian filter on optimising the transmission map are better than those of the average filter and median filter. However, because the refinement process of guided filter and bilateral filter is highly complex and time-consuming, it is not conducive to real-time industrial scenes. The median and average filter have similar effects, while the result of the Gaussian filtering is closer to the true haze-free image than the two other types, with clearer details and less time consumption. The reason is that simple average filter and median filter do not consider the weight of adjacent pixels in the smoothing operation, which leads to blurred edges. In the Gaussian filter, the closer the relationship between adjacent pixels, the higher the weight. Otherwise, the weight is lower. The bilateral filter effect is more effective but also more time-consuming. Therefore, under comprehensive consideration, we select the Gaussian filter as the method to refine the transmission map.

4.2.2. The enhancement effect of the initial recovered image

The hazy froth image was selected as the experimental object in the zinc-roughing tank, and the initial restored image was obtained by the defogging algorithm based on linear transformation in Section 3.2, and then the final defogging result was obtained by edge enhancement of the restored image. According to the definition of the visibility edge ratio in Section 4.3, we can obtain the visibility edge image of fog image, initial recovered image and final defogging image, as shown in Figure 10. The visible edge of the enhanced image is significantly more than that of the original restored image.

Figure 10. Visible edges of haze image and recovered image.

4.2.3. Dehazing effect under different lighting conditions

In the zinc flotation site, the change of ambient light will affect the brightness of the foam image. In this paper, the method proposed for low light foam atomisation image enhances the edge and texture of the restored image after defogging, and obtains a good defogging effect. As shown in Figure 11, the fog removal effects of other classical methods (Berman et al., 2016; Cai et al., 2016; He et al., 2010; Petro et al., 2014; Tarel & Hautiere, 2009; W. Wang et al., 2017) are compared.

Figure 11. Comparison chart of dehazing effect under different lighting conditions.

This paper uses prior information and linear transformation on the atmospheric scattering model to obtain the transmission rate, and the dehazing result after optimising the initial transmission rate is satisfactory. However, in the sharpening result of low-light dense fog images, the edges and the highlights of the froth are blurred. Therefore, the variational fractional differential operator is used to enhance the edges and texture structure of the recovered image, and the energy of gradient value of the image is improved (as shown in Table 1). It can be seen from Figure 11 that the method proposed in this paper can obtain restored images that are most consistent with the colour and texture characteristics of the froth itself under normal or low light conditions.

Table 1. Comparison of dehazing time of different algorithms (time unit: s).

Image	DCP	Retinex	NLDe	Visibresto2	DehazeNet	Proposed method
$320\times 240$	0.7677	0.2389	0.7185	0.5507	0.3118	0.0481
$640\times 480$	1.0592	0.3214	0.8338	0.6217	0.4654	0.1193

4.2.4. The defogging effect of each algorithm under different fog densities

To test the dehazing performance of the algorithm under different fog densities, we select three fog density of low, medium, and high foggy froth images to process the foggy images in the zinc-roughing tank, zinc-refining tank and zinc-scavenging tank. As shown in Figure 12, the proposed algorithm can effectively dehaze images with different densities. The only disadvantage is that when dealing with images with high fog density, the resulting image becomes distorted.

Figure 12. Comparison of dehazing effects of different fog densities.

The DCP algorithm (He et al., 2010) is an effective algorithm in the current image dehazing effect. However, it has some prerequisites, that is, after statistical analysis, many rich colours or shadows are in the natural image. Then, the intensity value of most of the pixels in its dark channel image is extremely low or close to zero. However, the flotation froth image is different from the natural image. It has a single colour, and most of the pixel values of the corresponding dark channel image are at least greater than 50. If the DCP algorithm is used to remove the fog, the restored fog-free image cannot truly describe the colour and brightness of the froth. Retinex algorithm (Petro et al., 2014) is an enhancement dehazing method that has a certain dehazing effect visually. The algorithm is efficient, but the phenomenon of image distortion appears in certain places. NLDe algorithm (Berman et al., 2016) is effective in removing smog, but the result is often excessive enhancement and colour distortion.Visibresto2 algorithm (Tarel & Hautiere, 2009) has a certain effect on outdoor images, but it seems useless for industrial foggy images such as flotation froth. DehazeNet algorithm (Cai et al., 2016) can remove haze in images with slight and moderate haze levels. However, as the smog becomes denser, its defogging ability is very limited.

The algorithm proposed in this paper has a better effect than other methods in low-light especially. The dehazing method based on the linear transformation of the atmospheric scattering model and Gaussian filter optimising has a shorter processing time and high efficiency, and can satisfy real-time industrial dehazing needs. The enhanced algorithm to restore the image solves the problem of edge blur.

4.3. Quantitative validation experiments

In the process of obtaining a flotation foam image, froth deformation, collapse, merger, rupture and irregular motion changes occur at any time. These changes are unpredictable. Therefore, it is impossible to obtain a foggy froth image and a corresponding clear image at the same time. Here, we choose two non-reference standards (EOG described in Section 3.2.3 and visibility edge ratio (Hautiere et al., 2008; Sun et al., 2021) to evaluate the performance of the proposed algorithm.

The difference in image pixel brightness is a direct reflection of spatial contrast. Hautiere et al. (2008) introduced a contrast definition that is very suitable for digital images and gave the contrast definition between two pixels x and y of an image f, which is defined by (24) $C_{x,y}=\frac{M|f(x)-f(y)|}{M-min(f(x)-f(y))},$(24) where M is the maximum brightness of the image. Let $F(s)$ be the set of all couples $(x,y)$ separated by s. The optimum thresholds $S_0$ are set, and if 2 $C(S_0)>0.05$, $F(S_0)$ is considered to be a visible edge. More detail about this method can be found in references (Hautiere et al., 2008; Yu et al., 2011). According to the contrast value and threshold control of each pixel, we can obtain the visible edge image. Let the set of points on the visible edge be expressed as $\phi$, the number of visible edge pixels be denoted as $N_{view-edge}$, and the number of all edge pixels be represented as $N_{all-edge}$. Finally, the visibility edge ratio of the image is defined as (25) $E=\lambda \log \sum _{x=\phi }c(x),$(25) where $\lambda =N_{view-edge}/N_{all-edge}$.

In order to verify the effectiveness of the proposed algorithm, we conducted a qualitative comparison of each algorithm on froth images of different fog levels in the three flotation tanks in Figure 12, as shown in Figure 13. The results of VER and EOG show that the proposed algorithm (marked by the line with the highest VER value and EOG value), Retinex and NLDe algorithm can obtain better visibility edge ratio values than other algorithms, and the proposed algorithm can obtain higher energy of gradient values than other algorithms. Combined with the dehazing effect of Figures 11 and 12, the Retinex algorithm has a certain defogging effect on light fog images, but the restored images have serious colour distortion. The NLDe algorithm has a strong defogging ability, but no matter how the parameters are adjusted, the restored image is over-enhanced, the colour is dark, and the greyish brown of the froth cannot be restored.

Figure 13. Performance comparison of different algorithms.

In the process of froth image defogging, the first step is fog detection. When the ambient temperature is low, the air near the flotation cell is humid and easy to saturate, forming haze. This situation is relatively low compared to the seasonal change of the year. Therefore, it is necessary to detect the haze before defogging the froth image. In order to improve the detection efficiency, we set the temperature parameter. When the temperature is less than 5 ${}^{\circ }$C, the system detects whether the froth image is foggy every hour, and the operation efficiency of the detection process is $O(MN)$(MN is the image size). The second step is the dehazing algorithm based on linear transformation of the atmospheric scattering model. The estimated operating efficiency of the transmission diagram is $O(MN)$. In the refinement of the transmission diagram, the time of Gaussian filtering is lower than that of other filtering, which is shown in Figure 10. Thirdly, when the edge enhancement is performed on the initial restored image, the mask size of the differential operator is $7\times 7$, and the fractional order is 0.5, the details of the froth image are enhanced without introducing redundant noise. We convert an RGB image into HIS space and only enhance the edge of brightness component, which improves the efficiency of operation without causing colour loss of image.

We compared the running times of different algorithms, as shown in Table 1. The size of the test froth image is $320\times 240$, $640\times 480$. The parameters of the computing platform are CPU3.5 GHz, 64-bit Intel(R) Xeon(R), 32 GB RAM, Matlab2018a. The classical DCP dehazing algorithm consumes a lot of time, and the time efficiency is improved after many improvements by many researchers. Retinex algorithm is used for defogging from the aspect of image enhancement, with low time cost, but its deficiency is that the colour of the restored image seriously deviates from the true colour of the froth. Visibresto2 defogging algorithm takes about 0.5 s. This algorithm is not suitable for industrial image defogging, and the effect is very poor. DehazeNet algorithm can recover high-quality froth images under normal light with no colour distortion. However, its defogging performance is greatly reduced for low-light foggy images. The algorithm proposed in this paper has great advantages in time efficiency, the restored image can truly describe the characteristics of the froth itself, and the image texture is rich in details.

5. Conclusions

In the intelligent flotation automation system based on machine vision, defogging is only a pretreatment operation to provide clear froth images for subsequent feature extraction, and it must meet the real-time requirements of the industry. Therefore, this paper aims to solve this problem by proposing a fast and effective method to remove haze for flotation industrial froth images. The energy gradient of the image reflects the sharpness of the image. In the experiment, the maximum local image energy gradient is used as a prior constraint to obtain the initial transmittance. The proposed method is simple in operation and high in calculation efficiency, and does not need a lot of data to participate in the network training for a long time. However, in view of the results of removing low-light fog, the blur phenomenon exists at the edges in the high spots of bubbles. Therefore, the method based on the variational fractional differential is adopted to enhance the edges of images and enrich the details of images. The quantitative and qualitative analysis shows that the defogging performance and computational efficiency of this method are better than those of the other five methods, and it can meet the demand of high-quality samples extracted from flotation froth features. However, in the captured froth image, there are various mixed-blur images. In the future, our goal is to study an algorithm that can remove multiple blurs or mixed blurs, and consider applying the deep neural network method (Sun et al., 2021; Xi et al., 2021) and some algorithms of image enhancement (Y. Li et al., 2018; Zhang et al., 2021) to the new computing environment of the Internet of Things-smart terminal devices (Liang et al., 2021) (such as edge computing, fog computing, etc.) to meet the requirements of high efficiency and low power consumption in the flotation industry. We will extract froth image features, industrial process data for information integration, fusion and other processing (Sandor et al., 2019; Yu et al., 2021), and finally, realise the intelligent automatic control of flotation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the National Natural Science Foundation of China [61771492, 62171476] and National Natural Science Foundation of China Guang-dong Joint Fund [U1701261].