Deep Learning Technique Based Surveillance Video Analysis for the Store

ABSTRACT

AI technology has developed so fast, and it has been applied to the commercial area. In order to predict the customer preference and adjust the placement of product or advertisement, etc., the intelligent surveillance video analysis technique has been proposed to gather the sufficient customer information and realize crowd counting and density map drawing. In this paper, a series of deep learning techniques are adopted to realize surveillance video analysis. This work covers different subproblems such as object detection, tracking and human identification. A skeleton recognition algorithm is adopted instead of object detection algorithm to overcome the severe occlusion problem. A multiple human tracking algorithm combing the human re-identification technology is adopted to realize the human tracking and counting. Finally, the density map and statistics information are obtained which can be used to evaluate and adjust the current business plan. A real store surveillance video is analyzed by the algorithm, and the results show the advantage of the algorithm.

Introduction

As the rapid development of high technology, AI has applied to kinds of area, including the commercial area. How can we make accurate and reliable predictions about actions or behaviors of consumer? The merchants deploy knowledge management and business intelligence systems to help them organize data, and gain insights from the data in order to make predictions about sales, operating costs, customer preferences, etc. (Nagar and Malone 2011). Historically, before the computer age, such forecasts were done by human experts (working individually or in teams). With application of computer technology, the merchants have made use of predictive algorithms to detect and predict customer preferences based on relevant data collected by employees. However, the data, such as customer preferences in fashion-driven businesses, are often unstructured which is problematic to obtain and realize in computer (Negash and Gray 2013).

In the past decade, automated scene analysis has already attracted much research attention in the computer vision community. The intelligent surveillance in replace of the traditional passive video surveillance is an important application (Ratre and Pankajakshan 2018). In the intelligent surveillance analysis system, it has accurate data processing and efficient information fusion with fewer human operators (Guo et al. 2016). In reality, it has a great advantage compared to the traditional manually observation which require a high human resource cost to constantly monitor surveillance cameras (Saleh et al. 2015). It has been used for automatic detecting and counting people, and also detecting of anomalies crowd automatically.

Nowadays, video surveillance system is an essential part of the store. Surveillance video contains abundant information, such as customer preferences, etc. The intelligent surveillance video analysis technique can be used for crowd estimation and gathering intelligence for further analysis and inference such as gauge people’s interest in a product of a store and this information can be used for appropriate product placement, etc. Therefore, the commercial prediction problem can be realized based on the analysis of earlier surveillance video of the store. Estimating the customer preference and drawing density maps by the surveillance video are a crowd counting and density estimation problem. Crowd counting aims to count the number of people in a crowd image and the density estimation aims to map the corresponding density map of crowd image which indicates the density of people at certain position (Sindagi and Patel 2018).

Crowd estimating, tracking and density maps drawing from crowd images have a wide range of applications such as video surveillance, traffic monitoring, public safety, disaster management, intelligence gathering and analysis, forensic search and urban planning. The complexity of the problem together with the wide range of applications has led to an increased focus of the researchers in the recent years. This challenge work covers different subproblems such as object detection, localization and tracking, etc. This task is also riddled with many challenges such as occlusions, nonuniform distribution of people and illumination, intrascene and interscene variations in appearance, scale and perspective, etc. (Sindagi and Patel 2017).

At the earlier stage, researchers have attempted to use a variety of approaches to address the issue of crowd estimation and density drawing, such as detection-based, clustering-based and regression-based methods. Generally, the problem of human counting and density estimation of crowd can be handled by two types of approaches: direct and indirect approaches (Saleh, Suandi, and Ibrahim 2015). The indirect approach is called feature-based method, human counting is achieved by the learning algorithms using some features of the crowd (Zhang and Li 2012). The feature-based methods usually extract several local and global features of the groups of people in foreground image. For this reason, many feature extraction methods for the foreground have been proposed such as foreground area, texture features, edge features to realize crowd counting and estimating by a regression function, like Gaussian probability, neural network, etc. However, it is often limited to the presence of the occlusions and perspective problem. The direct approach is also called object detection-based method which tries to segment and detect each human in the crowd and then count them. In this method, counting people is relied on the correct segment of human which is more complex when a severe crowd or occlusions occurred (Hou and Pang 2011). In early time, the indirect methods are more efficient because feature extraction is easier than detecting persons due to the slow developing object detection algorithm.

The detection-based approaches attempt to determine the number of people by identifying single persons and their locations simultaneously. Human detection is a more complex task in the presence of crowded and occlusion conditions. A sliding window is adopted as detector to count the number of people in the image. The detection is usually realized either in the monolithic or parts-based style. Monolithic detection approaches (Enzweiler and Gavrila 2009; Li et al. 2008) usually extracted features of the whole body for training classifier by Haar wavelets (Viola and Jones 2004), edgelet (Wu and Nevatia 2007) and shapelet (Sabzmeydani and Mori 2007) information. Various classifiers have been adopted with varying degree of success such as support vector machines, boosting (Jones and Viola 2007), random forest (Gall et al. 2011), etc. However, these methods are adversely affected by the presence of high-density crowds. The part-based detection methods (Felzenszwalb et al. 2010; Lin et al. 2001; Wu and Nevatia 2007) construct the classifiers based on the specific parts of human such as the head, hand and shoulder to estimate the people counts (Zhang, Li, and Nevatia 2008). The parts-based methods were also not successful at the situation of extremely dense crowds and high background clutter. In order to overcome these issues, the regression methods learn to map the features extracted from local image patches to their counts (Chen et al. 2012). These methods are independent of detectors which is easy to capture the global feature of the image. Some other local feature extraction method is combined, such as foreground features, edge features, texture and gradient features, to further improve the performance of regression models. Once these global and local features are extracted, different regression techniques, such as linear regression, piecewise linear regression, ridge regression, Gaussian process regression and neural network, are used to map low-level feature to realize crowd count. While the regression methods ignored the important spatial information as they were based on the global features. Some researches (Lempitsky and Zisserman 2010) combined the local patch features with corresponding object spatial information in the learning process to improve the detection performance. Zhan et al. (2008) point out that conventional computer vision processing technology may be ineffective when dealing with the analysis of crowded video sequences. Indeed, in a high-density situation the presence of severe occlusions consistently limits the performance of traditional methods for object detection and visual tracking (Jacques Junior et al. 2010). Although, it is often limited to small variations in crowd density and scales, the human information can be obtained by the direct way.

Over the last few years, the object detection-based crowd estimation has developed the ability to perform on a wide range of scenarios successfully. The success of crowd estimation methods can be largely attributed to the development of deep learning (Babaee et al. 2017; Brahimi et al. 2017) and publications of challenging datasets, especially the convolutional neural network (CNN). Some famous object detection algorithms are proposed, such as Faster RCNN, SSD, YOLO, etc. (Liu et al. 2018). The CNN has been applied to numerous computer vision tasks successfully (Dolata et al. 2018; Ferreira et al. 2017; Li et al. 2018), and it inspired researchers to exploit their nonlinear learning abilities for estimating the corresponding density maps or counting from the crowd image. The CNN-based approaches have demonstrated significant advantages over previous conventional object detection methods. Therefore, more researchers are drawn into exploring CNN-based approaches further for crowd analysis-related problems. Some CNN-based advanced methods (Boominathan et al. 2016; Hu et al. 2016; Jing et al. 2017; Oñoro-Rubio and López-Sastre 2016; Shao et al. 2015; Walach 2016; Zhang et al. 2016) and some new challenging crowd datasets are proposed (Zhang et al. 2016; Zhang et al. 2015). CNN-based approaches have achieved drastically lower error rates, and the creation of new datasets has made the models more generalized. Although, the deep learning technology has accelerated the development of the object detection algorithm, as mentioned previously, conventional pipeline of human detection and the subsequent tracking of those detections can hardly perform accurate human detection and tracking due to the increasing density of the crowd, the severe scene clutter and severe occlusions (Zhou et al. 2011).

Therefore, a special scheme is adopted for human detection in this paper. A skeleton detection algorithm is used to recognize the human skeleton which has a better performance in human detection than the object detection algorithm. The outline of human can be determined according to the skeleton data. And then, this paper makes use of some compound deep learning technique to realize crowd counting and density maps drawing based on the surveillance video of the store. After that, the merchants can predict the customer preference and adjust the goods arrangement according to the customer information.

Method

The object detection-based crowd counting and density map drawing for the surveillance video usually can be categorized into three main steps: (1) detection, (2) tracking and (3) analysis. The problem of human detection aims to localize all humans in the given image or video sequence. The human tracking aims to following the human motion trail to predict the motion or for drawing density map. There are different researches for different tasks at the analysis steps, such as customer preference prediction for this paper or density map analysis according to the previous steps. In this paper, the flowchart of the application is shown in Figure 1.

Figure 1. The flowchart of the system

Figure 2. The diagram of openpose

Figure 3. The flowchart of human tracking

Figure 4. The diagram of ReID

Figure 5. The principle of ReID

Human detection

Object detection is a basic and important problem in the computer vision, including the human detection. In crowd counting of surveillance video, one of the key tasks is to detect and identify humans in crowd. With the development of deep learning technology, CNN-based object detection methods have achieved great success. There are two categorized CNN-based object detection methods. Region-based methods make use of the candidate regions to potentially cover object, such as RCNN (Girshick et al. 2014), SPPNet (Zhang et al. 2015), Fast RCNN (Girshick 2015) and Faster RCNN (Ren et al. 2017). Regression-based methods have better real-time characteristic, such as YOLO (Redmon et al. 2016; Redmon and Farhadi 2017, 2018) and SSD (Liu et al. 2016). These methods have got good results than the conventional object detection methods. Although these algorithms have been developed to detect, track and understand the behaviors of various objects in video, they were mainly designed for common scenes with a low density of population (Morris and Trivedi 2008). The performance of these methods is modest at the situation of heavy occlusion.

In fact, the density of people substantially affects their appearance in video sequences. Especially in dense crowds, people occlude each other and only some parts of each individual are visible (Fradi and Dugelay 2015). Therefore, it is a big challenge for the CNN-based object detection algorithm. However, in this paper, the people must be detected accurately for counting and density map drawing. Human skeleton recognition is a special application which is designed for human (Hbali et al. 2018; Manzi et al. 2018). It has a robust recognition ability for human. Therefore, a human 2D pose estimation algorithm (openpose) (Cao et al. 2017; Wei et al. 2016) is adopted to detect human instead of object detection algorithm. The skeleton recognizing algorithms can recognize the human skeleton despite of occlusion occurring. Therefore, the skeleton recognition algorithm is adopted to detect human which has better performance than conventional object detection algorithm. The bounding box will be obtained when the skeleton data are gathered. The openpose algorithm adopts part of VGG19 as the backbone in Figure 2. The image coming from the surveillance video inputs the ten layers of the CNN of VGG19 and additional two convolutional layers, and then generates the featuremaps for two handling branches of the first stage. The confidence maps $S^1$ and part affinity fields $L^1$ are generated by the CNNs $Cov1$ and $Cov2$.

At stage t, the original featuremaps (F) of VGG19, $S^{t-1}$ and $L^{t-1}$ are the input of the branches. Then, the confidence map and part affinity vector are generated by Equations (1)(1) $S^t=Cov1\left(F,{\mathrm{S}}^{t-1},L^{t-1}\right),t\ge 2$(1) and (2(2) $L^t=Cov2\left(F,S^{t-1},L^{t-1}\right),t\ge 2$(2) ).

(1) $S^t=Cov1\left(F,{\mathrm{S}}^{t-1},L^{t-1}\right),t\ge 2$(1)

(2) $L^t=Cov2\left(F,S^{t-1},L^{t-1}\right),t\ge 2$(2)

And then, a loss value f is calculated according to $S^t$ and $L^t$ to train the network.

(3) $f=\sum f_S^t+f_L^t$(3)

(4) $f_S^t=\sum _{j=1}^J\sum _{p}W\left(p\right){∥S_j^t\left(p\right)-S_j^{\ast }\left(p\right)∥}_2^2$(4)

(5) $f_L^t=\sum _{c=1}^C\sum _{p}W\left(p\right){∥L_c^t\left(p\right)-L_c^{\ast }\left(p\right)∥}_2^2$(5)

where $S_j^{\ast }$ is the groundtruth part of confidence map, $L_c^{\ast }$ is the groundtruth part of affinity vector field. W is a binary mask with W(p) = 0 when annotation is missing at an image location p. f is the overall objective.

The openpose outputs a vector with 19 key points (1: nose, 2: neck, 3: right shoulder, 4: right elbow, 5: right wrist, 6: left shoulder, 7: left elbow, 8: left wrist, 9: right hip, 10: right knee, 11: right ankle, 12: left hip, 13: left knee, 14: left, 15: left eye, 16: right eye, 17: left ear, 18: right ear, 19: background).

According to these parameters, the bounding box can be generated by Equation (6)(6) $\left[x,y,h,w\right]=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\left(\mathrm{s}\mathrm{k}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\right)$(6) .

(6) $\left[x,y,h,w\right]=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{t}\left(\mathrm{s}\mathrm{k}\mathrm{e}\mathrm{l}\mathrm{e}\mathrm{t}\mathrm{o}\mathrm{n}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\right)$(6)

where $x,y,h,w$ are the central coordinates and size information of bounding box.

Making use of the skeleton data, a bounding box parameter (x,y,h,w) of the human can be obtained. The human object can be detected by this way even though there is a problem of occlusion. This parameter will be used as the input of the multiple human tracking algorithm to select the corresponding human for counting and tracking.

Multiple human tracking

Automatic tracking of people in video sequence is one of the most fundamental and challenging tasks in the research of video analysis and its results lay the foundations of a wide range of applications such as video surveillance, behavior analysis, security applications and traffic control. The task of multiple human tracking aims to locate multiple human, maintain their identities and individual trajectories according to the given video sequence. Visual tracking has achieved a great progress in recent years (Beyan and Temizel 2012; Tesfaye et al. 2016; Xie et al. 2013). Many tracking algorithms use the tracking-by-detection method which estimates the tracks of multiple objects based on previous object detections. The detection-based tracking algorithm has become the leading paradigm in multiple object tracking (Wojke et al. 2017). Generally speaking, it is largely attributed to the development of object detection methods which is based on deep learning technique. They have also significantly advanced the state-of-the-art of object tracking.

Traditional multiple human tracking methods (Fortmann et al. 2003; Reid 1979) perform a frame-by-frame data association on which have been improved by a tracking-by-detection way (Kim et al. 2015; Rezatofighi et al. 2015) and shown promising results. However, the computation and implement is complex. The flow network (Berclaz et al. 2011; Pirsiavash et al. 2011; Zhang et al. 2008) and probabilistic graphical models (Milan et al. 2013; Nevatia 2012; Roth 2012; Yang and Nevatia 2012) are popular frameworks based on the tracking-by-detection method. However, these methods are not applicable in online system due to the batch processing.

Simple online and real-time tracking (SORT) (Bewley et al. 2016) is a much simpler framework which has the advantage of high frame rates. However, SORT returns a relatively huge number of identity switches while achieving good precision and accuracy performance in tracking. Therefore, SORT has a deficiency in tracking at the situation of occlusions as they typically appear in frontal-view camera scenes. The deepSORT (Wojke, Bewley, and Paulus 2017) integrates the motion and appearance information to improve the performance of SORT. It is also easy to implement and apply to online system efficiently. Therefore, the deepSORT method is adopted to realize multiple human tracking in this paper, and some improvements are carried out in order to improve the performance of deepSORT. The object detection section of deepSORT is replaced by the openpose (above mentioned) in order to overcome the occlusion problem. After obtaining the bounding box data by openpose, the motion matching according to the Kalman filter and appearance matching are combined to build the association problem. The parameters of tracker are defined by a vector [$x,y,w,h,\dot{x},\dot{y},\dot{w},\dot{h}$]. [$x,y$] is the bounding box center, $w$ is the aspect ratio, $h$ is the height size and [$\dot{x},\dot{y},\dot{w},\dot{h}$] is the velocity of these parameters. The standard Kalman filter with constant velocity motion and linear observation model is adopted to observe the state of [$x,y,w,h$]. In the association of predicted Kalman states and newly detected status, the motion and appearance information are integrated as a comprehensive metrics. The Mahalanobis distance is adopted in the motion metrics between predicted Kalman states and newly detected human by following equation.

(7) $dm\left(i,j\right)=(d_j-{\theta }_i{)}^T{S}_i^{-1}(d_j-{\theta }_i)$(7)

where $dm\left(i,j\right)$ is the matching degree of motion information between jth detection and ith tracker. $S_i$ is the covariance matrix of ith tracker in the current observation space by Kalman filter. $d_j$ and ${\theta }_i$ are the tracker status [$x,y,w,h$] by detection and prediction.

While the Mahalanobis distance is suitable for lower motion uncertainty, unaccounted camera motion, the Kalman filtering provides only a rough estimate of the object location. Therefore, the second metric appearance descriptor for each bounding box is calculated. The smallest cosine distance between the ith tracker and jth detection in the appearance space is the second metric,

(8) $da\left(i,j\right)=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}\{1-{\vartheta }_j^T{\vartheta }_k^i|{\vartheta }_k^i\in {\mathcal{R}}_i\}$(8)

${\vartheta }_j$ is the appearance descriptor with $||{\vartheta }_j||=1$, and a buffer of the last $L_k$ appearance descriptors ${\mathcal{R}}_k=\{{\vartheta }_k^i{\}}_{k=1}^{L_k}$ for each constructed tracker k. And then, the two metrics are combined by a weighted sum.

(9) $c\left(i,j\right)=\lambda \cdot dm\left(i,j\right)+\left(1-\lambda \right)\cdot da\left(i,j\right)$(9)

A max sequent frame number N is set to determine the object is out of the vision or should be tracked continually. If the object cannot be found in the following N frames, it can be deleted from the tracker.

In the human tracking process, the human re-identification is an important part (Chen et al. 2016). In order to further improve the tracking performance of the deepSORT, a person re-identification (ReID) algorithm AlignedReID (Zhang et al. 2017) is integrated after the “IOU assignment” step in this paper. The whole process of the deepSORT algorithm is shown as Figure 3.

AlignedReID adopts ImageNet pretrained Resnet50 and Resnet50-Xception (Resnet-X) as the backbone as Figure 4. The Resnet output the feature maps of the selected human with size of 2048*7*7. Then, these features are handled through two branches, a local feature processing channel and a global processing channel. The local and global branches together discriminate two images in the learning stage. The global feature is extracted by a global pooling operation. A horizontal pooling is adopted in local feature process, and then the feature map with size of 2048*7*1 is obtained. A 1*1 convolution is followed to reduce the channel number to 128*7.

In order to deal with the alignment problem for two people with different size, the algorithm adopts a special matching method as Figure 5.

Analysis of customer information

In order to attract, retain and satisfy customers and maintain its competitiveness and business growth, the merchant must determinate and establish the customer’s preferences and expectations in today’s markets. The surveillance video contains sufficient information about the customer.

After successfully tracking the customer of the surveillance video, the information about the customer in the store can be recorded, such as gender and age estimation, track route, stopping time at certain position, etc. Therefore, we can analyze the whole trajectory of customers according to the tracking route, and then adjust the placement of the product and advertising, and also the density map can be drawn. Based on the accumulation of gender and age information, the age distribution of customers and preferences for different products at different ages can be estimated.

Experiments

The proposed approach is applied to a real application for analyzing the customer preferences of certain store in China. In the system, some improvements are adopted to improve the performance of the algorithm, including the human detection and tracking techniques.

Human detection

Human detection is the base of this project. Therefore, in order to demonstrate the effectiveness of the proposed human detection method, we compare our results to the conventional objection algorithm (SSD and Faster RCNN). In particular, the occlusion problem is tested for these algorithms. One frame of the surveillance video is shown in Figure 6. The marked area has a severe occlusion problem which can testify the performance the algorithms.

Figure 6. One image of the surveillance video

As shown in Figures 7 and 8, the human detection results are worse using conventional object detection algorithm. Although, the Faster RCNN adopts different threshold values to detect human and a smaller threshold value may detect more person, only six persons are detected with small threshold from the nine persons and a large proportion of occlusion is tactless as shown in Figure 7. Although, the SSD algorithm performs better than Faster RCNN especially for the occlusion region, some persons are also undetected or some persons are marked twice when different threshold values are adopted. That is why, we consider adopting skeleton recognition method to detect human.

Figure 7. The detection results of Faster RCNN under different threshold: (a) threshold = 0.8 and (b) threshold = 0.5

Figure 8. The detection results of SSD under different threshold: (a) threshold = 0.17 and (b) threshold = 0.1

The skeleton recognition algorithm can detect the skeleton of human even at the situation of severe occlusion as shown in Figure 9(a), and then the person can be marked according to the skeleton data as shown in Figure 9(b). Accurate human detection is the base of the system. The detected human can be clipped and used for human ReID, tracking or counting, etc.

Figure 9. Human detection based on skeleton recognition: (a) skeleton recognition result and (b) human detection result based on skeleton

Human tracking

After the detection of human, lots of works can be achieved, such as human counting, tracking and heat map drawing, etc. Figure 10 shows the human tracking results. Figure 10(a) shows the initial stage of human tracking. We can see that the human are detected and marked by detection algorithm, and then the tracking algorithm will track the corresponding person and combine the ReID algorithm. Figure 10(a) and (b) are the middle frames which we can see that the persons are tracked and the trajectories are marked by different color lines. Figure 10(d) shows a comprehensive human tracking result. According to Figure 10, we can see the human can be tracked well by the algorithm.

Figure 10. Human tracking results: (a) the initial frame, (b) the middle frame, (c) the latter frame and (d) the last frame

Others

After the detection of human, the human countering can be achieved and the heat map also can be draw. Figure 11 shows the heat map diagram. Figure 11(a) shows the heat map of certain scene as given in Figures 9 and 10. Figure 11(b) shows the heat map corresponding to the scene. The upper area of the figure is a passageway, therefore the human number is larger than other area and it is a hot area. According to the heat map diagram, the area where customers often walk can be obtain which can be used for adjusting advertisement placement.

Figure 11. The heat map diagram

According to the statistical result, the customer constitution is shown as Table 1.

Table 1. The customer constitution

Customer constitution	Percentage
Percentage of male	38%
Percentage of female	62%
Percentage of adult	89%
Percentage of child	11%

Conclusions

Intelligent surveillance video analysis has applied kinds of area, such as traffic, security, commerce, etc. The key point of video analysis is the object detection, especially the human detection. Some famous object detection algorithms have been proposed, such as Faster RCNN, SSD, etc. However, they cannot handle the severe occlusion problem. Therefore, a skeleton recognition algorithm is adopted to detect the human according to skeleton data which has better performance than conventional algorithm. After the detection of human, human counting and tracking can be achieved. A multiple human tracking algorithm combing the ReID is adopted to realize human tracking. According to the statistical data, the heat map can be drawn. According to the statistical results, the customer constitution and preference can be achieved which can be used for product placement adjustment.

Acknowledgments

This work was supported by the National Nature Science Foundation of China under Grant (61973184, 61803227, 61603214 and 61573213); Shandong Provincial Key Research and Development Plan under Grant (2018GGX101039).

Additional information

Funding

This work is supported by the National Natural Science Foundation of China [61573213, 61803227, 61603214]; National Key Research and Development Plan of China under Grant 2017YFB1300205, Shandong Province Key Research and Development Plan under Grants [2018GGX101039,2016ZDJS02A07] and China Postdoctoral Science Foundation under Grant 2018M630778.