Visualization Techniques for Surveillance: Visualizing What Cannot Be Seen and Hiding What Should Not Be Seen

Click to increase image sizeClick to decrease image size

Summary

This paper gives an introduction to some of the problems of modern camera surveillance, and how these problems are, or can be, addressed using visualization techniques. The paper is written from an engineering point of view, attempting to communicate visualization techniques invented in recent years to the non-engineer reader. Most of these techniques have the purpose of facilitating for the surveillance operator to recognize or detect relevant events (such as violence), while, in contrast, some have the purpose of hiding information in order to be less privacy-intrusive. Furthermore, there are also cameras and sensors that produce data that have no natural visible form, and methods for visualizing such data are discussed as well. Finally, in a concluding discussion an attempt is made to predict how the discussed methods and techniques will be used in the future.

Jörgen Ahlberg

Department of Electrical Engineering

Linköping University

SE-581 83 Linköping

Sweden

E-mail: [email protected]

Notes

1. Rafael C. Gonzalez and Richard E. Woods, Digital Image Processing, Upper Saddle River, N.J.: Prentice Hall, 2008.

2. Milan Sonka, Vaclav Hlavac and Roger Boyle, Image Processing, Analysis, and Machine Vision, Toronto: Thompson Learning, 2008.

3. Richard Hartley and Andrew Zisserman, Multiple View Geometry in Computer Vision, Cambridge: Cambridge University Press, 2004.

4. Richard Szeliski, Computer Vision: Algorithms and Applications, London: Springer, 2011. Available at szeliski.org/Book/.

5. Christopher M. Bishop, Pattern Recognition and Machine Learning, New York: Springer, 2006.

6. Simon J.D. Prince, Computer Vision: Models, Learning and Inference, Cambridge: Cambridge University Press, 2012. Available at www.computervisionmodels.com.

7. Gary Bradski and Adrian Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library, Sebastopol, CA: O’Reilly Media, 2008.

8. Jan Erik Solem, Programming Computer Vision with Python: Tools and Algorithms for Analyzing Images, Sebastopol, CA: O’Reilly Media, 2012.

9. Rafael C. Gonzalez, Richard E. Woods and Steven L. Eddins, Digital Image Processing Using MATLAB, Gatesmark Publishing, 2009.

10. Katsushi Ikeuchi, Computer Vision – A Reference Guide, Boston, MA: Springer, 2014.

11. Ingemar Ragnemalm, Polygons Feel No Pain, 2014. Available at http://www.computer-graphics.se.

12. »Suspect in JFK Airport Security Breach Arrested«, Fox News, 18 January 2010, Associated Press.

13. »Terminal at Newark Airport Evacuated after Man Enters Secure Area through Exit«, NJ News, 5 April 2010.

14. According a subway system security manager interviewed by the author, falls in stairs were, contrary to public belief, a more common reason of injury than violence. If this can be generalized to other subway systems is beyond the knowledge of the author.

15. YingLi Tian, »Robust Detection of Abandoned and Removed Objects in Complex Surveillance Videos«, in IEEE Transactions on Systems, Man, and Cybernetics, Vol. 41, Issue 5, 2011, pp. 565–576.

16. James Ferryman, David Hogg, Jan Sochman, Ardhendu Behera, José A. Rodriguez-Serrano, Simon Worgan, Longzhen Li, Valerie Leung, Murray Evans, Philippe Cornic, Stéphane Herbin, Stefan Schlenger and Michael Dose, »Robust Abandoned Object Detection Integrating Wide Area Visual Surveillance and Social Context«, Pattern Recognition Letters, Vol. 34, Issue 7, 2013, pp. 789–798.

17. Abandoned luggage detection was also the theme of the 9th IEEE International Workshop on Performance Evaluation of Tracking Systems (PETS), New York, 2006.

18. A related problem is that most people leaving their luggage are not bombers, so even if automatic detection could be done, bomb squads cannot run to the place each time a piece of abandoned luggage is detected. For example, in a major subway system such as the London underground, there can be close to one hundred forgotten bags on a single day.

19. Alex Stedmon, Sarah Harris, Anthony Carse, Sarah Sharples and John Wilson, Tracking a Suspicious Person Using CCTV: But What Do We Mean by Suspicious Behaviour? Contemporary Ergonomics 2008, London: Taylor & Francis, 2008.

20. Tom Troscianko, Alison Holmes, Jennifer Stillman, Majid Mirmehdi, Daniel Wright and Anna Wilson, »What Happens Next? The Predictability of Natural Behaviour Viewed through CCTV Cameras«, Perception, No 33, 2004, pp. 87–101.

21. Julian F.P. Kooij, Gwenn Englebienne and Dariu M. Gavrila, »A Non-parametric Hierarchical Model to Discover Behavior Dynamics from Tracks«, European Conference on Computer Vision, 2012.

22. Daniel Kuettel, Michael D. Breitenstein, Luc Van Gool and Vittorio Ferrari. »What’s going on? Discovering Spatio-temporal Dependencies in Dynamic Scenes«, International Conference on Computer Vision and Pattern Recognition (CVPR), 2010.

23. An Internet search on »abnormal behavior detection« gives numerous examples of currently ongoing research.

24. Surveillance of unattended baggage and the identification and tracking of the owner (SUBITO), the European Union’s Security Research Programme, Topic SEC-2007-2.3-01, 2009–2011.

25. Suspicious and abnormal behaviour monitoring using a network of cameras & sensors for situation awareness enhancement (SAMURAI), the European Union’s Security Research Programme, Topic SEC-2007-2.3-04, 2008–2011.

26. Automatic detection of abnormal behaviour and threats in crowded spaces (ADABTS), the European Union’s Security Research Programme, Topic SEC-2007-2.3-04, 2009–2013. www.adabts-fp7.eu.

27. Future Attribute Screening Technology (FAST), United States Department of Homeland Security.

28. Jörgen Ahlberg, »Intelligent Surveillance: Towards More Efficient Crime Prevention while Still Considering Aspects of Privacy«, in Jerker Hellström, Mikael Eriksson and Niklas Granholm (eds.), Strategic Outlook 2011, Stockholm: FOI Swedish Defence Research Agency, 2011.

29. Andrew A. Adams and James M. Ferryman, »The Future of Video Analytics for Surveillance and Its Ethical Implications«, Security Journal, January 2013, pp. 1–18.

30. Fatih Porikli, Francois Bremond, Shiloh L. Dockstader, James Ferryman, Anthony Hoogs, Brian C. Lovell, Sharath Pankanti, Bernhard Rinner, Peter Tu and Peter L. Venetianer, »Video Surveillance: Past, Present, and now the Future«, IEEE Signal Processing Magazine, Vol. 30, Issue 3, 2013, pp. 190–198.

31. Kinect is an accessory to the game console XBox analyzing the movements of the gamers.

32. Piotr Dollar, Christian Wojek, Bernt Schiele and Pietro Perona, »Pedestrian Detection: An Evaluation of the State of the Art«, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, Issue 4, 2012, pp. 743–761.

33. For commercially available systems, see, for example, www.autoliv.com or www.bmw.com.

34. The KTH Human Action Database. www.nada.kth.se/cvap/actions.

35. UCF Action Recognition Data Set. crcv.ucf.edu/data/UCF101.php.

36. Marcin Marszalek, Ivan Laptev and Cordelia Schmid, »Actions in Context«, IEEE Conference on Computer Vision and Pattern Recognition, 2009. Available at www.di.ens.fr/~laptev/actions/hollywood2"> .

37. Martijn C. Liem and Dariu M. Gavrila, »A Comparative Study on Multi-person Tracking Using Overlapping Cameras«, Proceedings of the International Conference on Computer Vision Systems (ICVS), Vol. 7963 in Lecture Notes in Computer Science, pp. 203–212, 2013.

38. Harpreet S. Sawhney, Aydin Arpa, Rakesh Kumar, Supun Samarasekera, Manoj Aggarwal, Steven C. Hsu, David Nister and Keith J. Hanna, »Video Flashlights – Real Time Rendering of Multiple Videos for Immersive Model Visualization«, Proceedings of the 13th Eurographics Workshop on Rendering (EGRW’02), pp. 157–168, Aire-la-Ville, Switzerland.

39. Hedvig Sidenbladh, Jörgen Ahlberg and Lena Klasén, New Systems for Urban Surveillance, Linköping: FOI Swedish Defence Research Agency, 2005.

40. Kristoffer Gunnartz, Välkommen till övervakningssamhället, Stockholm: Forum/Bokförlaget DN, 2007. Available at www.kristoffergunnartz.com.

41. Leon Hempel and Eric Töpfer, CCTV in Europe, Urban Eye Paper No. 15, 2004. Available at www.urbaneye.net.

42. Privacy Preserving Perimeter Protection Project (P5), the European Union’s Security Research Programme, Topic SEC-2012.2.3-1, 2013-2016. Available at www.p5-fp7.eu.

43. »Ökad övervakning – ökad integritet«, Aftonbladet, 20 September 2005.

44. »JK välkomnar smart övervakningskamera«, Dagens Nyheter, 13 September 2005.

45. »Kräv intelligent övervakning«, Svenska Dagbladet, 20 September 2005.

46. If the object is really hot, it will emit visual light as well. The sun is a well-known example.

47. FLIR One. Available at www.flir.com/flirone.

48. A wavelength band is a range of wavelengths, for example light seen as red has a wavelength in the range of 600–700 nm.

49. The human visual system (HVS) is typically modeled as sensing a combination of red, green, and blue (RGB) light, that is, the dimensionality of the color space equals three. Combining these, all visible colors are created. Common digital cameras thus have sensor elements for these three colors, and images to be displayed on a computer screen are represented as numbers telling the amounts of RGB in each point. That is, even if the color space dimensionality of the HVS would be higher than three (which it arguably is), a computer screen cannot reproduce more than three color dimensions anyway.

50. James V. Stone, Vision and Brain: How We Perceive the World, Cambridge, MA: MIT Press, 2012.

51. John M. Findlay and Iain D. Gilchrist, Active Vision: The Psychology of Looking and Seeing, Oxford: Oxford University Press, 2003.

52. Dirk Borghys, Ingebjørg Kåsen, Véronique Achard and Christiaan Perneel, »Hyperspectral Anomaly Detection: Comparative Evaluation in Scenes with Diverse Complexity«, Journal of Electrical and Computer Engineering, No 2012, 2012, pp. 1–6.

53. J.v.d. Sande et al., Sound Source Localization and Analysis, ADABTS21 deliverable 5.2, 2012.