An overview of optical flow-based approaches for motion segmentation

References

• Bideau P, Learned-Miller E. A detailed rubric for motion segmentation. arXiv preprint arXiv:1610.10033, 2016.
   Google Scholar
• Ochs P, Malik J, Brox T. Segmentation of moving objects by long term video analysis. IEEE Trans Pattern Anal Machine Intell. 2014;36(6):1187–1200.
   PubMed Web of Science ®Google Scholar
• Lu X, Manduchi R. Fast image motion segmentation for surveillance applications. Image Vision Comput. 2011;29(2–3):104–116.
   Web of Science ®Google Scholar
• Courtney JD. Automatic video indexing via object motion analysis. Pattern Recogn. 1997;30(4):607–625.
   Web of Science ®Google Scholar
• Wang L, Hu W, Tan T. Recent developments in human motion analysis. Pattern Recogn. 2003;36(3):585–601.
   Web of Science ®Google Scholar
• Hu W, Tan T, Wang L, et al. A survey on visual surveillance of object motion and behaviors. IEEE Trans Syst Man Cybern – Part C: Appl Rev. 2004;34(3):334–352.
   PubMed Web of Science ®Google Scholar
• Zapella L, Llado X, Salvi J. Motion segmentation: a review. Proceedings of the 11th International Conference of the Catalan Association for Artificial Intelligence; 2008; Spain: IOS Press; 2008. p. 398–407.
   Google Scholar
• Zhang D, Lu G. Segmentation of moving objects in image sequence: A review. Circuits Syst Signal Process. 2001;20(2):143–183.
   Web of Science ®Google Scholar
• Baker S, Scharstein D, Lewis JP, et al. A database and evaluation methodology for optical flow. Int J Comput Vision. 2011;92(1):1–31.
   Web of Science ®Google Scholar
• Farneback G. Two-frame motion estimation based on polynomial expansion. Proceedings of 13th Scandinavian Conference on Image Analysis; 2003 June 29–July 02; Halmstad, Sweden: Springer-Verlag; 2003. p. 363–370.
   Google Scholar
• Lucas BD, Kanade T. An iterative image registration technique with and application to stereo vision. Proceedings of 7th International Joint Conference on Artificial Intelligence. 1981; Vancouver, British Columbia, 1981, Morgan Kaufmann Publishers Inc. 1981. p. 674–679.
   Google Scholar
• Horn BKP, Schunck BG. Determining optical flow. Artif Intell. 1981;17(1–3):185–203.
   Web of Science ®Google Scholar
• Beauchemin SS, Barron JL. The computation of optical flow. ACM Comput Surveys. 1995;27(3):433–466.
   Web of Science ®Google Scholar
• Anandan P. A Computational framework and an algorithm for the Measurement of visual motion. Int J Comput Vision. 1989;2(3):283–310.
   Web of Science ®Google Scholar
• Singh A. An estimation-theoretic framework for image flow computation. Proceedings of Third International Conference on Computer Vision; 1990 Dec 04–07; Osaka, Japan: IEEE Explore; 2002. p. 168–177.
   Google Scholar
• Heeger DJ. Optical flow using Spatiotemporal filters. Int J Comput Vision. 1988;1(4):279–302.
   Web of Science ®Google Scholar
• Memin E, Perez P. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans Image Process. 1998;7(5):703–719.
   PubMed Web of Science ®Google Scholar
• Wedel A, Cremers D, Pock T, et al. Structure- and motion-adaptive regularization for high accuracy optic flow. Proceedings of IEEE 12th International Conference on Computer Vision; 2009 29 Sept 29–Oct 02; Kyoto, Japan. IEEE Explore; 2010. p. 1663–1668.
   Google Scholar
• Brox T, Bruhn A, Papenberg N, et al. High accuracy optical flow estimation based on a theory of warping. In: Pajdla T and Matas J, editors. Proceedings of 8th European Conference on Computer Vision; Prague, Czech Republic Springer LNCS 3024, Vol. 4; May 2004. p. 25–36.
   Google Scholar
• Brox T, Malik J. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Machine Intell. 2011;33(3):500–513.
   PubMed Web of Science ®Google Scholar
• Weinzaepfel P, Revaud J, Harchaoui Z, et al. DeepFlow: large displacement optical flow with deep matching. Proceedings of the IEEE International Conference on Computer Vision; 2013 Dec 01–08; Sydney, NSW, Australia. IEEE Explore; 2014. p. 1385–1392.
   Google Scholar
• Revaud J, Weinzaepfel P, Harchaoui Z, et al. EpicFlow: Edge-preserving interpolation of correspondences for optical flow. Proceedings of the CVPR; 2015. p. 1164–1172.
   Google Scholar
• Zweig S, Wolf L. InterpoNet. A brain inspired neural network for optical flow dense interpolation. CVPR, pages 4563–4572, 2017.
   Google Scholar
• Geiger A, Lenz P, Stiller C, et al. Vision meets robotics: the KITTI dataset. Int J Robot Res. 2013;32(11):1231–1237.
   Web of Science ®Google Scholar
• Butler DJ, Wulff J, Stanley GB, et al. A naturalistic open source movie for optical flow evaluation. European Conference on Computer Vision, Vol. 4; 2012. p. 611–625.
   Google Scholar
• Sun D, Roth S, Lewis JP, et al. Learning optical flow. Proceedings of European Conference on Computer Vision; 2008 Oct 12–18; Marseille, France, LNCS Vol. 5304; 2008. Springer-Verlag. p. 83–97.
   Google Scholar
• Rosenbaum D, Zoran D, Weiss Y. Learning the local statistics of optical flow. NIPS, Nevada, USA; 2013. p. 2373–2781.
   Google Scholar
• Roth S, Black MJ. On the spatial statistics of optical flow. Int J Comput Vision. 2007;74(1):33–50.
   Web of Science ®Google Scholar
• Wulff J, Black MJ. Efficient sparse-to-dense optical flow estimation using a learned basis and layers. Proceedings of Computer Vision and Pattern Recognition; 2015. p. 120–130.
   Google Scholar
• Dosovitskiyz A, Fischer P, Ilg E, et al. FlowNet: learning optical flow with convolutional networks. IEEE International Conference on Computer Vision; 2015 Dec 07–13; Santiago, Chile. IEEE Explore; 2016. p. 2758–2766.
   Google Scholar
• Mayer N, Ilg E, Hausser P, et al. A large dataset to train convolutional networks for disparity. Optical Flow, And Scene Flow Estimation Proceeding Computer Vision and Pattern Recognition; 2017. 2462–2470.
   Google Scholar
• Ranjan A, Black MJ. Optical flow estimation using a spatial pyramid network. Proceedings of Computer Vision and Pattern Recognition; 2017. p. 4161–4170.
   Google Scholar
• Hu P, Wang G, Tan Y-P. Recurrent spatial pyramid CNN for optical flow estimation. IEEE Trans Multimedia. 2018;20(10):2814–2823.
   Web of Science ®Google Scholar
• Ilg E, Mayer N, Saikia T, et al. FlowNet2.0: evolution of optical flow estimation with deep networks. Proceedings of the Computer vision and Pattern Recognition; 2017. p. 2462–2470.
   Google Scholar
• Hui T, Tang X, Loy C. Liteflownet: a lightweight convolutional neural network for optical flow estimation. Proceedings of the Computer Vision and Pattern Recognition; 2018. p. 8981–8989.
   Google Scholar
• Xu J, Ranftl R, Koltun V. Accurate optical flow via direct cost volume processing. Proceedings of the Computer Vision and Pattern Recognition; 2018. p. 1289–1297.
   Google Scholar
• Sun D, Yang X, Liu M, et al. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. Proceedings of the Computer Vision and Pattern Recognition; 2018. p. 8934–8943.
   Google Scholar
• Ahmadi A, Patras I. Unsupervised convolutional neural networks for motion estimation. Proceedings of the ICIP; 2016. p. 1629–1633.
   Google Scholar
• Yu JJ, Harley AW, Derpanis KG. Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. Proceedings of the ECCVW; 2016. p. 3–10.
   Google Scholar
• Lai W, Huang J, Yang M. Semi-supervised learning for optical flow with generative adversarial networks. Proceedings of the NIPS; 2017.
   Google Scholar
• Yang Y, Soatto S. Conditional prior networks for optical flow. Proceedings of the ECCV. 2018: 271–287.
   Google Scholar
• Zitova B, Flusser J. Image registration methods: a survey. Image Vision Comput. 2003;21(11):977–1000.
   Web of Science ®Google Scholar
• Yoo DW, Won DY, Tahk MJ. Optical flow based collision avoidance of multi-rotor UAVs in urban environments. Int J Aeronaut Space Sci. 2011;12(3):252–259.
   Google Scholar
• Souhila K, Karim A. Optical flow based robot obstacle avoidance. Int J Adv Robot Syst. 2007;4(1):13–16.
   Google Scholar
• Eresen A, Imamoglu N, Efe MO. Autonomous quadrotor flight with vision-based obstacle avoidance. Expert Syst Appl. 2012;39(1):894–905.
   Web of Science ®Google Scholar
• Agrawal P, Ratnoo A, Ghose D. A composite guidance strategy for optical flow based UAV navigation. Proceedings of Third International Conference on Advances in Control and Optimization of Dynamical Systems, Kanpur, India; 2014.
   Google Scholar
• Szeliski R. Image alignment and stitching: a tutorial. Found Trends Comput Graphics Vision. 2006;2(1):1–104.
   Google Scholar
• Rebiere N, Flavie M, Fortier A. Image Mosaicing using local optical flow registration. Proceedings of19th IEEE International Conference on Pattern Recognition; 2008. p. 1–5.
   Google Scholar
• Chen T, Lu S. Object-level motion detection from moving cameras. IEEE Trans Circuits Syst Video Technol. 2017;27(11):2333–2343.
   Web of Science ®Google Scholar
• Adiv G. Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE Trans Pattern Anal Machine Intell. 1985;7(4):384–401.
   PubMed Web of Science ®Google Scholar
• Liu T, Shen L. Fluid flow and optical flow. J Fluid Mech. 2008;614:253–291.
   Web of Science ®Google Scholar
• Guo DA, van de Ven L, Zhou X. RBC tracking using optical flow methods. IEEE J Biomed Health Inform. 2014;18(3):991–998.
   PubMed Web of Science ®Google Scholar
• Weng N, Yang Y-H, Pierson R. Three-dimensional surface reconstruction using optical flow for medical imaging. IEEE Trans Med Imag. 1997;16(5):630–641.
   PubMed Web of Science ®Google Scholar
• Loncaric S, Majcenic Z. Optical flow algorithm for cardiac motion estimation. Proceedings of the 22nd Annual IEEE International Conference of Engineering in Medicine and Biology Society, Chicago, United States; 2000. p. 415–417.
   Google Scholar
• Marzban C, Sandgathe S. Optical flow for verification. Weather Forecast. 2010;25:1479–1494.
   Web of Science ®Google Scholar
• Enkelmann W. Obstacle detection by evaluation of optical flow fields from image sequences. Image Vision Comput. 1991;9(3):160–168.
   Web of Science ®Google Scholar
• Hariyono J, Hoang V.-D, and Jo K.-H. Motion segmentation using optical flow for pedestrian detection from moving vehicle. Proceedings of ICCCI 2014: Computational Collective Intelligence, Technologies and Applications. p. 204–213.
   Google Scholar
• Smith SM, Brady JM. ASSET 2 real time motion segmentation and shape tracking. IEEE Trans Pattern Anal Machine Intell. 1995;17(8):814–820.
   Web of Science ®Google Scholar
• Thompson WB, Mutch KM, Berzins VA. Dynamic occlusion analysis in optical flow fields. IEEE Trans Pattern Anal Machine Intell. 1985;7(4):374–383.
   PubMed Web of Science ®Google Scholar
• Overington I. Gradient-based flow segmentation and location of the focus of expansion. In John Frisby, editors. Proceedings of the Alvey Vision Conference, Alvey Vision Club; 1987 Sept 1987. p. 23.1–23.10.
   Google Scholar
• Spoerri A, Ullman S. The early detection of motion boundaries. Proceedings of 1st International Conference on Computer Vision; 1987. p. 209–218.
   Google Scholar
• Clocksin WF. Perception of surface slant and edge labels from optical flow: a computational approach. Perception. 1980;9(3):253–269.
   PubMed Web of Science ®Google Scholar
• Schunck BG. Image flow segmentation and estimation by constraint line clustering. IEEE Trans Pattern Anal Machine Intell. 1989;11(10):1010–1027.
   Web of Science ®Google Scholar
• Ullman S, Hildreth EC. The measurement of visual motion. In: Braddick OJ, Sleigh AC, editors. Physical and biological processing of images. Springer Series in Information Sciences, Vol. 11. Berlin: Springer; 1985.
   Google Scholar
• Nagel H.-H, Socher G, Kollnig H, et al. Motion Boundary detection in image sequences by local stochastic test. Proceedings of the 3rd European Conference on Computer Vision, Stockholm; May 1994. p. 305–314.
   Google Scholar
• Papazoglou A, Ferrari V. Fast object segmentation in unconstrained video. IEEE International Conference on Computer Vision, Sydney. Australia; 2013. p. 1777–1784.
   Google Scholar
• Brox T, Malik J. Object segmentation by long term analysis of point trajectories. Proceedings of the 3rd European Conference on Computer Vision, Greece; 2010. p. 282–295.
   Google Scholar
• Ochs P, Brox T. Higher order motion models and spectral clustering. IEEE International Conference on Computer Vision and Pattern Recognition; 2012.
   Google Scholar
• Ng AY, Jordan MI, Weiss Y. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems (NIPS), 2002.
   Google Scholar
• Fragkiadaki K, Zhang G, Shi J. Video segmentation by tracing discontinuities in a trajectory embedding. IEEE Conference on Computer Vision and Pattern Recognition, Providence, United States; 2012. p. 1846–1853.
   Google Scholar
• Keuper M, Andres B, Brox T. Motion trajectory segmentation via minimum cost multicuts. IEEE International Conference on Computer Vision, Santiago, Chile; 2015. p. 3271–3279.
   Google Scholar
• Keuper M. Higher-order minimum cost lifted multicuts for motion segmentation. IEEE International Conference on Computer Vision, Venice, Italy; 2017. p. 4252–4260.
   Google Scholar
• Shen J, Peng J, Shao L. Submodular trajectories for better motion segmentation in videos. IEEE Trans Image Process. 2018;27(6):2688–2700.
   Web of Science ®Google Scholar
• Fujishige S. Submodular functions and optimization, annals of discrete mathematics, 2nd edition. Amsterdam: Elsevier; 2005.
   Google Scholar
• Torr PHS. Geometric motion segmentation and model selection. Philos Trans R Soc London A: Math Phys Eng Sci. 1998;356(1740): 1321–1340.
   Web of Science ®Google Scholar
• Black MJ, Fleet DJ. Probabilistic detection and tracking of motion boundaries. Int J Comput Vision. 2000;38(3):231–245.
   Web of Science ®Google Scholar
• Narayana M, Hanson A, Learned-Miller E. Coherent motion segmentation in moving camera videos using optical flow orientations. IEEE International Conference on Computer Vision, Sydney, Australia; 2013. p. 1577–1584.
   Google Scholar
• Bideau P, Learned-Miller E. It’s moving! A probabilistic model for causal motion segmentation in moving camera videos. European Conference on Computer Vision, Amsterdam, Netherlands; 2016. p. 433–449.
   Google Scholar
• Sevilla-Lara L, Sun D, Jampani V, et al. Optical flow with semantic segmentation and localized layers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016. p. 3889–3898.
   Google Scholar
• Vertens J, Valada A, Burgard W. SMSnet: Semantic motion segmentation using deep convolutional neural networks. Proceedings of International Conference on Intelligent Robots and Systems, Vancouver. Canada; 2017. p. 582–589.
   Google Scholar
• Jain S, Xiong B, Grauman K. FusionSeg: learning to combine motion and appearance for fully automatic segmentation of generic objects in videos. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, United States; 2017. p. 2117–2126.
   Google Scholar
• Tokmakov P, Alahari K, Schmid C. Learning motion patterns in videos. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, United States; 2017. p. 531–539.
   Google Scholar
• Tokmakov P, Alahari K, Schmid C. Learning video object segmentation with visual memory. IEEE International Conference on Computer Vision, Venice, Italy; 2017. p. 4491–4500.
   Google Scholar
• Jang W-D, Kim C-S. Online video object segmentation via convolutional trident network. IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, United States; 2017. p. 7474–7483.
   Google Scholar
• Perazzi F, Pont-Tuset J, McWilliams B, et al. A benchmark dataset and evaluation methodology for video object segmentation. IEEE Conference on Computer Vision and Pattern Recognition, Nevada, United States; 2016. p. 724–732.
   Google Scholar
• Bideau P, RoyChowdhury A, Menon RR, et al. The best of both worlds: combining CNNs and geometric constraints for hierarchical motion segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2018 June 18–22; Salt Lake City, Utah, United States. p. 508–517.
   Google Scholar
• Bideau P, Menon RR, and Learned-Miller E. MoA-Net: self-supervised motion segmentation. In: Leal-Taixé L, Roth S, editors. Computer Vision – ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science, vol. 11134. Cham: Springer.
   Google Scholar
• Verri A, Uras S, De Micheli E. Motion Segmentation from optical flow. Proceedings of the 5th Alvey Vision Conference; 1989. p. 209–214.
   Google Scholar
• Rowley HA, Rehg JM. Analyzing articulated motion using expectation-maximization. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, United States; 1997. p. 935–941.
   Google Scholar
• Bregler C. Learning and recognizing human dynamics in video sequences. IEEE Conference on Computer Vision and Pattern Recognition, Puerto Rico, United States; 1997. p. 568–574.
   Google Scholar
• Meyer D, Denzler J, Niemann H. Model based extraction of articulated objects in image sequences. Proceedings of the Fourth International Conference on Image Processing; 1997.
   Google Scholar
• Wedel A, Meißner A, Rabe C, et al. Detection and segmentation of independently moving objects from dense scene flow. Energy Minimization Methods in Computer Vision and Pattern Recognition, Bonn, Germany; 2009. p. 14–27.
   Google Scholar
• Ogale AS, Fermuller C, Aloimonos Y. Motion segmentation using occlusions. IEEE Trans Pattern Anal Machine Intell. 2005;27(6):988–992.
   PubMed Web of Science ®Google Scholar
• Dey S, Reilly V, Saleemi I, et al. Detection of independently moving objects in non-planar scenes via multi-frame monocular epipolar constraint. European Conference on Computer Vision, Florence, Italy; 2012. p. 860–873.
   Google Scholar
• Namdev RK, Kundu A, Krishna KM, et al. Motion segmentation of multiple objects from a freely moving monocular camera. IEEE International Conference on Robotics and Automation, Saint Paul, United States; 2012. p. 4092–4099.
   Google Scholar
• Talukder A, Matthies L. Real-time detection of moving objects from moving vehicles using dense stereo and optical flow. Proceedings of IEEE/RSJ International Conference on Intelligent Robots and System, Japan; 2004. p. 3718–3725.
   Google Scholar
• Gao P, Sun X, Wang W. Moving object detection based on kirsch operator combined with optical flow. IEEE International Conference on Image Analysis and Signal Processing, Zhejiang, China; 2010. p. 620–624.
   Google Scholar
• Royden CS, Connors EM. The detection of moving objects by moving observers. Vision Res. 2010;50(11):1014–1024.
   PubMed Web of Science ®Google Scholar
• Royden CS, Moore KD. Use of speed cues in the detection of moving objects by moving observers. Vision Res. 2012;59:17–24.
   PubMed Web of Science ®Google Scholar
• Patel MP, Parmar SK. Moving object detection with moving background using optic flow. Proceedings of IEEE International Conference on Recent Advances and Innovations in Engineering, Jaipur, India; 2014. p. 1–6.
   Google Scholar
• Royden CS, Holloway MA. Detecting moving objects in an optic flow field using direction- and speed-tuned operators. Vision Res. 2014;98:14–25.
   PubMed Web of Science ®Google Scholar