Recognition of expiry data on food packages based on improved DBNet

Abstract

To prevent products with missing character information from reaching the market, manufacturers need an automatic character recognition method. One of the key problems of this recognition method is to recognise text under complex package patterns. In addition, some products use dot matrix characters to reduce printing costs, which makes text extraction more difficult. We propose a character detection algorithm using DBNet as the base network, combined with the Convolutional Block Attention Module (CBAM) to improve its feature extraction of characters in complex contexts. After the character area has been located by the detection algorithm, it is intercepted and fed into a fully convolutional character recognition network to achieve print character recognition. We use ResNet as the backbone network and CTC loss for training. In addition, the CBAM module was added to the backbone network to enhance its recognition of dot matrix characters. The algorithm was finally deployed on the jetson nano. The experimental results show that the character detection accuracy reaches 97.9%, an improvement of 1.9% compared to the original network. As for the character recognition algorithm, the inference speed is doubled when deployed to the nano platform compared to the CRNN network, with an accuracy of 97.8%.

KEYWORDS:

• Optical character recognition (OCR)
• DBNet
• CBAM
• expiry data
• deep learning

1 Introduction

During the production of food and beverages, information such as expiry data are often printed on the surface of the packaging to monitor the quality of the product and to provide consumers with important information about the quality of the product. In the food packaging industry, some types of ink-jet codes use dot matrix character printing as it is more cost-effective than inkjet printing and can convey information efficiently. Dot matrix characters consist of dots printed in a specific order. Although this is more cost-effective, the text quality will be inferior to other methods and the character characteristics will not be as pronounced as with other characters. Therefore, dot matrix characters are more difficult to recognise. At the same time, with the rapid growth of the country’s food industry, companies have designed many beautiful food packages to attract consumers to buy their products, and the packaging patterns are becoming more and more diverse, which will make character recognition more difficult. In the process of spraying character information due to abnormalities in the spraying equipment or environmental interference problems, which can lead to missed spraying, wrong spraying and other situations. Failure to detect a product with a defective print in time and bring it to market can have an impact on a company’s reputation and severely reduce the consumer’s desire to buy the product. The number of products verified in these industries daily can reach a very large figure. There is therefore a need to establish a robust and accurate optical character recognition method that can improve the accuracy of recognising characters against complex background patterns.

The rise of deep learning techniques in recent years has made it a leading research method in several fields, especially in the field of vision. Deep learning is very powerful and can automatically filter the most effective features by learning from the data to better perform various tasks. Many methods developed based on convolutional neural networks enable end-to-end coding character recognition and better detection of coding in complex backgrounds. The OCR we use is divided into two phases: the text detection phase and the text recognition phase. The text detection network uses a modified DBNet (Liao et al., 2020) network for detecting where the characters on the package are located and boxing them out with rectangular boxes. The character recognition network uses resnet18 (He et al., 2016) as the backbone network and a network with full convolution trained by CTC loss to recognise characters in rectangular boxes. In addition, for the problem of skewed characters, the affine transformation process is performed before inputting into the character recognition network. Experiments show that the method used in this paper has high accuracy and good robustness. We propose to solve the problem of optical character recognition in the industrial detection of good’s expiry dates. Combined with embedded devices, the OCR system is portable, can be quickly applied in production, and can reduce costs.

The main contributions of our research are:

1. Proposed the improved backbone network of detection and recognition models, which address the problem of complicated food package images, thus effectively improving the performance of the model.

2. Adopt affine transformation to address the skewed text lines that appear during printing characters.

3. Deploying the improved model to embedded devices reduces hardware costs and reduces the size to improve portability.

The rest of this paper is organised as follows: Section III describes the detection and recognition models used in this paper, Section IV gives the experimental results and analysis of the detection and recognition models, Section V describes the deployment of the model and its performance on embedded devices, and Section VI concludes the paper.

2 Relate work

In the field of optical character recognition, there are many different solutions for extracting characters from images. The traditional character recognition method (Inunganbi et al., 2020; Wang et al., 2020) mainly consists of the following steps: (1) pre-processing the image; (2) locating the character area and extracting the ROI; (3) segmenting and extracting individual characters in the ROI area; (4) recognising individual character pieces; (5) finally comparing with the correct character and judging whether it is qualified or not. Although traditional character recognition is characterised by high accuracy and speed in some simple or task-specific scenarios, it requires the design of high-precision templates, and the usage scenarios have greater limitations and less robustness. With the development of deep learning techniques, some of the shortcomings in traditional methods have been effectively solved and are widely used in major classical vision tasks (Chen, 2019): image classification, target detection, text recognition, etc.

The first CNN-based text recognition method was proposed by Jaderberg et al. (2014; Jaderberg et al., 2016), which classifies words into predefined sets of dictionary words. Before the recognition phase, a combination of Edge Boxes (Zitnick & Dollár, 2014) and an aggregated channel feature (ACF) detector framework (Dollár et al., 2014) is used to detect candidate locations for the words. Following Jaderberg’s work, many new approaches have recently emerged. For text detection, there are EAST (Zhou et al., 2017), TextSnake (Long et al., 2018), PSENet (Li et al., 2020), etc., and text recognition models such as CRNN (Shi et al., 2017), ASTER (Shi et al., 2018), NRTR (Sheng et al., 2019), etc. However, these methods are rarely used in practical applications. However, these methods are rarely used in practical applications.

With the increase in retail merchandise, an efficient and accurate OCR system has become an urgent need. Deep learning methods are very effective in these tasks (Gong et al., 2018; Maitrichit & Hnoohom, 2020; Muresan et al., 2019; Wei et al., 2018). These methods use a combination of deep learning and traditional machine vision. As a result, they all have limitations in terms of usage scenarios. A better solution would be to adopt a deep learning approach altogether. In (Ai & Tao, 2020), a deep learning approach was used to detect and recognise characters on the steel surface. The backbone was changed to a lightweight MobileNet network based on the target detection model SSD to improve character detection speed and a CRNN model was used for character recognition. In the literature (Li et al., 2022) the code-spurting character detection algorithm uses YOLOv5 as the base network, combined with the ECA attention mechanism to improve its detection accuracy, and the character recognition model uses the CRNN network to finally deploy the algorithm to the NVIDIA TX2 embedded device. Similar methods have been used extensively in industrial production (Florea, 2020; Gong et al., 2021; Shanthini et al., 2021). These methods use common object detection algorithms for text detection. But some texts vary greatly in length. Common object detection methods are less effective in detecting text with large differences in length.

As the demand for OCR has increased many open-source OCR methods have emerged, such as Tesseract OCR, Keras OCR, Easy OCR, etc. A Tesseract OCR-based application is presented in (Hosozawa et al., 2018) and its performance is compared with two other open-source OCR engines: NHocr and OCRopus (Breuel, 2008). Tesseract has the most applicable character types and the best recognition accuracy, but it must be pre-processed to remove the influence of the background to have good recognition results in complex backgrounds. The open-source OCR method is used in (Kamisetty et al., 2022) to recognise the invoice content, where the paper first performs image preprocessing, after which three different OCRs are tested: the Keras OCR, Easy OCR and Tesseract OCR, where the Tesseract OCR gives the best recognition accuracy. But these open-source OCR engines are mainly used for character recognition of simple backgrounds like electronic documents. Complex pre-processing operations are required for images with background patterns, and the recognition results are not satisfactory.

To train such models, it is vital to obtain high-quality training data. There are currently very few retail merchandise datasets available. The only dataset that supports the training of OCR models on retail products is Unitail-OCR (Chen et al., 2022). To train a highly robust model, the amount of data in Unitail-OCR is still too small, which increases the risk of overfitting. The work in (Gupta et al., 2016; Liu et al., 2018) provides excellent solutions for the preparation of manually generated training datasets. We use a manually generated dataset for pre-training, followed by tuning using Unitail-OCR to make the model learn real-world features. Unitail-OCR is primarily for retail product category identification, not for expiry dates. Therefore, we collected images of expiry dates for 162 food items, a total of 2,161 images. They were made into a dataset. This allows the model to be better applied to expiry date recognition.

3 Text extraction models

Based on the analysis and study of existing algorithms, two steps are used to perform OCR: detection and recognition. In the first step, we detect regions of the image that may contain text. In a second step, text recognition is performed, where, for each detected region, the words in that region are recognised and transcribed using a CNN. The detailed process is shown in Figure 1.

Figure 1. General architecture of character recognition.

3.1 Text detection model

The current deep learning methods in the field of scene text detection mainly include: proposal-based text detection and segmentation-based text detection. The DBNet used in this paper is a segmentation-based text detection method, and the segmentation-based method can describe the scene text of each shape more accurately. The basic principle framework of DBNet is shown in Figure 2.

Figure 2. Architecture of Text detection model.

The DBNet network represents the input image as a high-dimensional convolutional feature map through the feature pyramid backbone network(Usually based on the ResNet architecture (He et al., 2016). In this paper, CBAM is added to each residual module for enhancing the network’s extraction of character features in complex contexts), and then, uses the feature map to predict the probability map and threshold map, and then calculates the binarization map from the probability map and threshold map to finally obtain the text position. Among them, the post-processing of binarization is crucial for segmentation-based detection, which converts the probability map generated by the segmentation method into a binarized map of the text region and thus obtains the location where the text is located. A module called Differentiable Binarization (DB) is used in the DBNet network, which performs the binarization process in the segmented network. The process is adaptive binarization for each pixel. The binarization threshold is obtained by network learning. The binarization step is thoroughly added to the network for training. In this way, the final output image will be very robust to the threshold, which simplifies post-processing and improves the effect of text detection.

To enhance the representation of foreground text and suppress background noise, a convolutional attention module (CBAM) (Woo et al., 2018) is added to the network. The attention mechanism is a resource allocation mechanism with the idea of finding intrinsic correlations in the raw data, highlighting strong correlations and ignoring some weakly correlated feature information. In this paper, we want the network to focus only on the character features, so the attention module is added to the network for improving the accuracy of the character detection algorithm.

The basic framework of CBAM is shown in Figure 3. CBAM consists of two parts: channel attention and spatial attention. The channel attention uses the average pooling and maximum pooling operations to aggregate the spatial information of the feature maps to generate the average pooling feature $F_{avg}^c$ and the maximum pooling feature $F_{max}^c$. These two features are then fed into a shared network consisting of MLPs with a single hidden layer. Finally the two parts are summed to generate the channel attention Mc. The channel attention is computed as: (1) $\begin{array}{rl}{\text{M}}_c(F)& =\sigma (MLP(avgPool(F))+MLP(maxPool(F)))\\ & =\sigma (W_1(W_0(F_{avg}^c))+W_1(W_0(F_{max}^c)))\end{array}$(1) After the spatial attention is computed in the feature map by channel attention, the average pooling and maximum pooling operations are applied along the channel axis, by which the channel information of the feature map can be aggregated to generate two 2D feature maps and stitch them together, and then a standard convolutional layer is used to generate the spatial attention Ms. The spatial attention is computed as: (2) $\begin{array}{rl}{M}_s(F)& =\sigma (f^{7\times 7}([avgPool(F);maxPool(F)]))\\ & =\sigma (f^{7\times 7}(F_{avg}^s;F_{max}^s))\end{array}$(2) The CBAM is added to each residual fast in the backbone as shown in Figure 4 to form a base module. Because information features are extracted by mixing channel information and spatial information in convolutional operations. Therefore, the CBAM module applies the channel and spatial attention modules in turn to enhance the meaningful feature information along the two main dimensions of channel and spatial axis to improve the feature extraction capability of convolutional networks. And CBAM is a lightweight module, so the overhead of parameter calculation has little impact on the speed of the network.

Figure 3. CBAM module.

Figure 4. Basicblock.

Table 1 gives the parameters of each layer of the text detection network. The backbone of the network is a feature pyramid structure, based on resnet18, which fuses feature layers of different scales through FPN. Pyramid features are upsampled to the same scale and cascaded to produce features F, which are then used to predict probability maps (P) and threshold maps (T) from the feature maps. After that, the approximate binary mapping (B) is calculated by P and F. The calculation process is carried out by means of differentiable binarization. The formula is as follows: (3) $B_{i,j}=\frac{1}{1+e^{-k(P_{i,j}-T_{i,j})}}$(3) where B is the approximate binary mapping; T is the adaptive threshold map learned from the network; and k denotes the amplification factor. k is set to 50 empirically. This approximate binarization function is differentiable and thus can be optimised with the segmentation network during training.

Table 1. Parameters of each layer of text detection model.

input	operator	output size	output layer name
640^2*3	Conv2d 7*7	160^2*64	layer1
160^2*64	Basicblock*2	160^2*64	layer2
160^2*64	Basicblock*2	80^2*128	layer3
80^2*128	Basicblock*2	40^2*256	layer4
40^2*256	Basicblock*2	20^2*512	layer5
160^2*64(layer2), 80^2*128(layer3), 40^2*256(layer4), 20^2*512(layer5)	FPN	160^2*256	feature map
160^2*256(feature map)	Conv2d 3*3	160^2*64	–
	ConvTranspose2d 2*2	320^2*64	–
	ConvTranspose2d 2*2	640^2*1	probability map
160^2*256(feature map)	Conv2d 3*3	160^2*64	–
	ConvTranspose2d 2*2	320^2*64	–
	ConvTranspose2d 2*2	640^2*1	threshold map
640^2*1(probability map), 640^2*1(threshold map)	DB	640^2*1	binary map

3.2 Text recognition model

The character recognition model is a fully convolutional model. The architecture of the model is shown in Figure 5. The convolutional body is based on the ResNet-18 architecture, and the CBAM module is added in the first residual black. Where the input image is passed through the convolutional body and then the most likely character at each image text position is predicted by the last convolution. In this paper, we do not use recurrent neural networks (such as LSTM or GRU) because recurrent networks are poorly parallel computed. Due to its chained structure, the sequence modelling process is closely related to the length of the input sequence, making the recursive network computation more time consuming (Gao et al., 2017). The CNN can acquire the features of each character in the image and simulate the interaction between characters very well (Borisyuk et al., 2018), and it is more computationally efficient.

Figure 5. Architecture of Text Recognition model.

This model is trained using the CTC loss, which calculates the conditional probability of a label given the prediction by marginalising over the set of all possible alignment paths, and that can be efficiently computed using dynamic programming. As shown in Figure 5, every column of the feature map corresponds to the probability distribution of all characters in the alphabet at that position of the image. These predictions may contain duplicate characters or blank characters (-) to finally get the ground truth label. For example, in Figure 5 we show that for an input image with the characters 20211203, the model may generate the sequence of characters “2-00-22-1-122-00-3-”, which includes blank characters and repeated characters. In inference, the characters with the highest probability of prediction in each column of the feature map are extracted and blank characters and duplicate characters are removed to obtain the final prediction results.

Equipment problems such as vibration of the printhead and unstable movement of the conveyor belt can cause some of the characters to be detected to appear skewed. To address this problem, this paper performs an affine transformation on the localised character region before entering the recognition network, and corrects the skewed characters before feeding them into the character recognition network.

The affine transformation is a linear transformation from two-dimensional coordinates to two-dimensional coordinates, maintaining the “straightness” and “parallelness” of the graph. The affine transformation allows you to translate and rotate the image. The transformation rule calculates a transformation matrix from the four predicted coordinate points on the original image and the corresponding preset position coordinates, and then transforms the original image into a new image by the transformation matrix. The affine transformation can be expressed by Equation (4). (4) $\left[\begin{array}{c}{x}^{\mathrm{\prime }}\\ y^{\mathrm{\prime }}\\ 1\end{array}\right]=\left[\begin{array}{ccc}{a}_1& a_2& t_x\\ a_3& a_4& t_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$(4) where (t_x, t_y) represents the amount of translation, while the parameter reflects a_i the changes in image rotation, scaling, etc.

In this paper, we judge the coordinate information of the text box after detecting the character area, and when the tilt angle of the text box exceeds 3°, the character is considered to be tilted excessively, and the affine transformation is used to map the tilted character area back to the correct position. Figure 6 shows the result of an affine transformation.

Figure 6. Affine transformation to corrected characters.

In most text recognition methods, the positioned text area is cropped out and resized to 32 × 100 pixels without preserving the aspect ratio. This results in the same characters having different features depending on the length of the text in the image, which requires a larger model capacity for accurate representation. Therefore, at training time, we adjust the image to 32 × 128 pixels. The height of the image is reduced to 32 pixels wide for proportional scaling. When the width is greater than 128, the aspect ratio is not retained, and the image is directly scaled to 32 × 128 pixels. Otherwise, the edge value of the image is used to fill to 128 pixels width on the right side. In this way, on the one hand, the original features of the characters can be retained, and on the other hand, can effectively use batches of images for training. In the experiments, as can be seen from Table 3 in Section 4, this method has superior accuracy than resizing the image without preserving the aspect ratio.

4 Experiments

4.1 Experimental evaluation indexes

Precision and Recall are used as evaluation metrics in the text detection model. Where Precision and Recall are calculated as follows: (5) $\begin{array}{rl}P& =\frac{TP}{TP+FP}\end{array}$(5) (6) $\begin{array}{rl}R& =\frac{TP}{TP+FN}\end{array}$(6)

Where TP, TN, FP and FN are respectively: True Positive, True Negative, False Positive, False Positive.

For the recognition model, accuracy and Normal Edit Distance were used as performance measures. Where accuracy is calculated as follows: (7) $Accuracy=\frac{TP+TN}{TP+TN+FP+FN}$(7) For accuracy calculation, a prediction is recorded as correct only when the predicted word matches the real word exactly. The accuracy metric provides information on the percentage of words correctly identified, but it does not provide any information on the degree of error in the incorrect predictions. In an input image, incorrectly recognising one character or incorrectly recognising five characters have the same impact on accuracy, zero. To obtain more information about the degree of misidentification, the normal edit distance is used, which calculates the minimum number of steps required between the conversion of a predicted string into a real string by editing operations (substitution, insertion, deletion) and can be considered as the degree of difference between the two strings. This paper uses normal edit distance to evaluate the similarity between the predicted and the ground truth.

4.2 Model training and datasets

The character detection network and the character recognition network are trained and tested respectively. Table 2 shows the parameters set for the model training. The character detection network optimisation algorithm uses Adam optimiser. The batch size is 16, the epoch is 120, and the initialisation learning rate is set to 1e-3. Use Warm up for learning rate changes. The character recognition network optimisation algorithm uses Adam optimiser. The batch size is 32, the epoch is 100, and the initialisation learning rate is set to 1e-4. The learning rate decreases by 20% every two epochs. The training process of the character recognition model is shown in Figure 7.

Figure 7. Curve of parameters during training.

Table 2. Model training parameters settings.

Set parameters	detection model	recognition model
optimiser	adam	adam
batch_size	16	32
epoch	120	100
lr	0.001	0.0001

Having high quality training and testing datasets is important for building robust supervised machine learning models. We first pre-trained using the artificial dataset and then tuned using the unitail-OCR dataset to make the model learn real-world features. Unitail-OCR is an OCR dataset for retail product identification which contains 1454 product categories. This dataset was benchmarked on a variety of state-of-the-art. However, Unitail-OCR does not match the data distribution of the image of the date printed on the package. We therefore manually annotated a dataset with thousands of images of packages with print dates and used it to fine-tune the model, which improved the results considerably.

4.3 Results and analysis

4.3.1 Text detection model

Table 3 shows the experimental results of the text detection model on the open source dataset Unitail-OCR and our manually collected dataset. The→ denotes finetuning, i.e. A→ B means train on A and then finetune on B. We can see that the method used in this paper improves the feature extraction ability of the original model’s backbone, and improves the precision and recall.

Table 3. Comparison of text detection models on different datasets.

Model	Datasets	Precision	Recall	hmean
DBNet	Unitail-OCR	87.5	67.2	76
DBNet + CBAM	Unitail-OCR	88.7	72.8	80
DBNet	Unitail-OCR→Expiry Data	95.6	81.6	88
DBNet + CBAM	Unitail-OCR→Expiry Data	97.7	83.8	90.2

The experimental results of different detection algorithms are compared in Table 4. The test set used was a manual collection of food packaging images. The CPTN (Tian et al., 2016) cuts the text into smaller proposals of fixed width along the text direction, and merges the small text boxes belonging to the same text part to get a complete text box. However, CPTN does not consider the text angle and is only applicable to horizontal text, which is not effective in detecting tilted text. Both EAST (Zhou et al., 2017) and TextSnake (Long et al., 2018) consider the text angle. Among them, EAST uses two-stage text detection methods: full convolutional network (FCN) and non-maximal suppression (NMS). The shape of the detection can be either a normal quadrilateral or a rotating rectangle. TextSnake obtained 7 feature maps after the basic feature extraction network, of which 4 channels predict TR (Text Region) and TCL (Text Center Line), and the other 3 channels predict r (radius), cosθ and sinθ. The region where the text is located is calculated from these 7 feature maps. However, these methods have high model complexity, slow detection speed and are not suitable for deployment on low computing power devices. The DBNet model used in this paper is of low complexity, and high accuracy can be obtained by using a simpler feature extraction network. And the CBAM module is added to the feature extraction network to improve the network’s effectiveness in localising characters in complex backgrounds. In Table 4, we can see that the final method achieves the best performance compared to the other models.

Table 4. Comparison of different detection algorithms.

Methods	Precision	Recall	hmean
CPTN	94.2	80.1	86.6
EAST	94.7	81.8	87.8
TextSnake	96.3	83	89.2
DBNet	95.6	81.6	88
DBNet + CBAM	97.7	83.8	90.2

4.3.2 Text recognition model

In production, the common method used for text recognition is CRNN. However, the recurrent neural network (RNN) structure in CRNN networks has a large time overhead in embedded devices. According to the study in (Borisyuk et al., 2018 Gao et al., 2017;) for horizontal text can be done well without using RNN structure and convolution can be done well for character recognition. Therefore, this paper uses a full convolutional recognition network and further adds CBAM to the network to improve the recognition effect. From Table 5, we can see that the improved algorithm has higher recognition accuracy and a lower probability of making mistakes.

Table 5. Comparison of the effect of the improved recognition algorithm.

Methods	Datasets	Accuracy/%	Normal Edit Distance
CRNN	Unitail-OCR	51.1	0.263
Resnet + CTC	Unitail-OCR	50.2	0.291
Resnet + CBAM + CTC	Unitail-OCR	56.7	0.226
CRNN	Expiry Data	97.5	0.016
Resnet + CTC	Expiry Data	96.5	0.021
Resnet + CBAM + CTC	Expiry Data	98.1	0.014

The input image of the character recognition algorithm is usually scaled directly to the size of 32 × 128, which will lead to changes in the features of some characters, as shown in Figure 8. In particular, direct resizing for dot matrix characters will cause the originally circular dots to be stretched into ellipses. The result is that some of the same characters appear with different features affecting the recognition effect of the network. In this paper, we try to maintain the aspect ratio of the original image while changing the image to 32 × 128 pixels size to ensure the character features remain unchanged, which can effectively improve the accuracy of the algorithm recognition. As shown in Table 6.

Figure 8. Resize.

Table 6. Comparison of resize results.

Resize	accuracy	Normal Edit Distance
b	94.4	0.031
c	98.1	0.014

5 Model deployment to embedded system

To save hardware costs and facilitate quick embedding in factory production lines for use. Deploy the improved algorithms to the NVIDIA Jetson nano embedded platform and accelerate inference with Tensorrt8.0. The configuration of Jetson nano is shown in Table 7.

Table 7. Jetson nano configuration.

Platform	CPU	GPU	Computing power/TFLOPS	Memory/GB	Power consumption
Jetson nano	Quad-core ARM A57	128-core	0.5	2	5–10 W
		NVIDIA
		Maxwell

Table 8 shows the parameters and computation of several models and the inference time of the models on the embedded platform. The parameters and calculation amount of one of the common convolutional layers are calculated as follows.

Table 8. Comparison of model complexity and inference time.

Method	Model	Params	FLOPs	time/ms
Text detection	DBNet	12.1 M	37.97 G	303.4
	DBNet + CBAM	12.5 M	37.98 G	331.6
Text recognition	CRNN	6.1 M	1.59 G	94.3
	Resnet + CBAM + CTC	11.2 M	1.04 G	39.4

Parameters: (8) $C_{out}\times C_{in}\times K\times K$(8) Amount of calculation: (9) $H\times W\times C_{out}\times (2\times C_{in}\times K\times K-1)$(9) Where H, W is the width and height of the input feature map, C is the number of channels of the input or output of the feature map, K is the size of the convolution kernel, and the above equations are all in the case of ignoring bias.

The parameter is mainly used to describe the size degree of the model, which is similar to the space complexity in algorithms and is related to the hardware frame buffer size. The amount of calculation is similar to the time complexity in an algorithm and is related to the floating point operations per second (FLOPS) of the hardware. The inference time of the model is mainly related to the amount of calculation. From rows 2 and 4 of Table 8, we can see that the two models have similar parameters, but the difference in the amount of calculation and therefore the time is also very different.

In addition, the CBAM module is lightweight as can be seen in the text detection section of the table. It has little impact on the parameters and amount of calculation of the model, and can effectively improve the detection accuracy of the model. From the text recognition part of the table, it can be found that although the model we used is larger than the CRNN parameters, the calculation amount is smaller than the CRNN. And because the CRNN network uses a recurrent neural network (RNN) structure, the GPU has poor acceleration for the RNN. The model we used is full convolution and is better when accelerated using the GPU, so it is twice as fast as CRNN.

We used a Hikvision industrial camera to capture images at a size of 640 × 480 as input, and we used Tensorrt in the jetson nano platform to call the GPU to accelerate inference. As shown in Figure 9 and 10, the improved algorithm can recognise fonts commonly used for printing valid dates, including dot matrix characters. In addition, we deploy the model in an embedded device, which offers huge advantages in terms of power consumption and cost compared to using an expensive industrial control machine.

Figure 9. Dot matrix character recognition effect.

Figure 10. Other print character recognition effect.

6 Conclusion

We have combined DBNet with the CBAM module based on the natural scene character recognition approach in deep learning, which effectively improves the accuracy of the algorithm in localising character regions in complex backgrounds. The intercepted character region images after localisation are processed to the input size required for the character recognition model while preserving the character aspect ratio, and the accuracy is improved by 3.5% compared to the unprocessed method. The character recognition method we use does not use the commonly used RNN structure and uses a full convolutional recognition algorithm, which improves the subsequent deployment and recognition speed on embedded devices. And the CBAM module is also added to the recognition algorithm to improve the feature extraction of dot matrix characters by the network, which improves the accuracy by 1.2% up to 97.9%. Finally, the improved algorithm is deployed on jetson nano to implement embedded edge detection. It makes the OCR system more portable and convenient for quick application in actual industrial production lines and reduces the labour cost for character verification. The next step will be to continue to optimise the algorithm to improve the recognition accuracy of characters, and to try to train more character datasets to meet the application of more character products with different types of fonts to make the algorithm more universal.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Special Fund for Education and Research of Fujian Provincial Department of Finance, China [grant number GY-Z21001].